Main classes related to the structure extraction via lines classification

Here described the main classes used for features extraction from textual lines, line classifier training and classifier usage for lines’ types prediction.

Features extraction

class dedoc.structure_extractors.feature_extractors.abstract_extractor.AbstractFeatureExtractor

_before_special_line(document: List[LineWithMeta], find_special_line: cov) → List[float]

Find “distance” to the closest special line in the document. If line is located before the special line, then distance is negative, if line is after special line then distance is positive, for special line distance is 0. If there is no special line in the document then distance is 0 for every line.

Parameters:

• document – document in form of list of lines

• find_special_line – function that finds some special line, see __find_toc for example

Returns:

list of distances from given line to first special line

static _can_be_prev_element(this_item: str | None, prev_item: str | None) → bool

Check if prev_item can be the previous element of this_item in the correct list For example, “2” can be previous element of “3”, “2.1.” can be previous element of “2.1.1” If prev_item is None then this_item is the first item in the list and should be 1

Parameters:

• this_item – text of the current list item

• this_item – text of the previous list item

Returns:

True if prev_item can be the previous element of this_item

static _get_bold(line: LineWithMeta) → float

Parameters:

line – document line

Returns:

indicator if there are some parts of the line bold

static _get_bold_percent(line: LineWithMeta) → float

Parameters:

line – document line

Returns:

the percent of bold characters in a line

static _get_color(line: LineWithMeta) → Iterable[Tuple[str, float]]

Parameters:

line – document line

Returns:

color values (R,G,B+color dispersion) with features names

static _get_features_quantile(feature_column: Series) → Series

Parameters:

feature_column – column with one feature

Returns:

column with feature’s quantiles

static _get_indentation(line: LineWithMeta) → float

Parameters:

line – document line

Returns:

indentation value (distance between line and document margin)

static _get_italic(line: LineWithMeta) → float

Parameters:

line – document line

Returns:

indicator if there are some parts of the line italic

static _get_size(line: LineWithMeta) → float

Parameters:

line – document line

Returns:

font size value

static _get_spacing(line: LineWithMeta) → float

Parameters:

line – document line

Returns:

spacing value (distance between two lines)

static _get_underlined(line: LineWithMeta) → float

Parameters:

line – document line

Returns:

indicator if there are some parts of the line underlined

static _is_first_bold(line: LineWithMeta) → float

Parameters:

line – document line

Returns:

indicator if the beginning of the line is bold

_list_features(lines: List[LineWithMeta], list_item_regexp: Pattern = re.compile('(^\\s*\\d{1,2}(\\.\\d{1,2})*)(\\s|$|\\)|\\}|\\.([A-ZА-Яa-zа-яё]|\\s))'), end_regexp: Pattern = re.compile('([A-ZА-Яa-zа-яё]|\\s|( )*)$')) → List[float]

For each line, the indicator is computed if the line is a list item. The indicator is obtained using regular expressions and checking, if list items form the correct list.

Parameters:

• lines – list of document lines

• list_item_regexp – regular expression for a list item

• end_regexp – regular expression for a text after the list item

Returns:

list of indicators if a line is a list item

static _next_line_features(feature_matrix: ndarray, n: int) → ndarray

Parameters:

• feature_matrix – matrix where rows are lines, columns are features

• n – the number of next lines, those features is needed to add to the current line’s features

Returns:

feature matrix where lines were shifted on n lines upwards (feature matrix for n next lines, for the last n lines zeros are added)

static _normalize_features(feature_column: Series) → Series

new_feature = (feature - mean) / (max - min) if max = min else 0

Parameters:

feature_column – column with one feature

Returns:

normalized feature vector [-1; 1]

static _prev_line_features(feature_matrix: ndarray, n: int) → ndarray

Parameters:

• feature_matrix – matrix where rows are lines, columns are features

• n – the number of previous lines, those features is needed to add to the current line’s features

Returns:

feature matrix where lines were shifted on n lines downwards (feature matrix for n previous lines, for the first n lines zeros are added)

_start_regexp(line: str, regexps: List[Pattern], suffix: str | None = None) → Iterable[Tuple[str, float]]

Apply regular expressions to the given line, calculate the length of the match and number of matches in the line

Parameters:

• line – document line

• regexps – list of regular expressions that will be applied to the line

• suffix – the additional info for feature naming

Returns:

feature name + value, values are length of the match and number of matches

abstract parameters() → dict

Returns the dictionary with parameters for the __init__ method of the feature extractor.

prev_next_line_features(matrix: DataFrame, n_prev: int, n_next: int) → DataFrame

Add features of previous and next lines to the input feature matrix. For each document line, n_prev features of previous lines and n_next features of next lines will be added. For lines that don’t have previous/next lines, zeros are added as features

Parameters:

• matrix – matrix where rows are lines, columns are features

• n_prev – the number of previous lines, those features is needed to add to the current line’s features

• n_next – the number of next lines, those features is needed to add to the current line’s features

Returns:

feature matrix with added features

abstract transform(documents: List[List[LineWithMeta]], toc_lines: List[List[TocItem]] | None = None) → DataFrame

Extract a list of float features for each document line.

Parameters:

• documents – list of documents in form of list of lines

• toc_lines –

Returns:

matrix of features extracted for the document

Training utilities

class scripts.train.trainers.data_loader.DataLoader(dataset_dir: str, label_transformer: Callable[[str], str], logger: Logger, data_url: str, *, config: dict)

Class for downloading data from the cloud, distributing lines into document groups and sorting them. Returns data in form of document lines with their labels.

__init__(dataset_dir: str, label_transformer: Callable[[str], str], logger: Logger, data_url: str, *, config: dict) → None

Parameters:

• dataset_dir – path to the directory where to store downloaded dataset

• label_transformer – function for mapping initial data labels into the labels for classifier training

• logger – logger for logging details of dataset loading

• data_url – url to download data from

• config – any custom configuration

get_data(no_cache: bool = False, loading_with_annotation_update: bool = False) → List[List[LineWithLabel]]

Download data from a cloud at self.data_url and sort document lines.

Parameters:

• no_cache – whether to use cached data (if dataset is already downloaded) or download it anyway

• loading_with_annotation_update – True if the dataset is out of date (lines’ annotations in dataset are out of date. Not recommended method of loading dataset.

Returns:

list of documents, which are lists of lines with labels of the training dataset

class scripts.train.trainers.dataset.LineClassifierDataset(dataframe: DataFrame, feature_list: List[str], group_name: str, label_name: str, text_name: str)

Dataset of lines in form of a feature matrix (with already extracted features)

__init__(dataframe: DataFrame, feature_list: List[str], group_name: str, label_name: str, text_name: str) → None

Parameters:

• dataframe – pandas dataframe with features and metadata, columns are features (and metadata), rows are documents’ lines

• feature_list – list of feature columns’ names

• group_name – name of the group column, e.g., “document” (dataset is split into groups, one group of lines is related to one document)

• label_name – name of the label column, e.g., “label”

• text_name – name of the column with lines’ text, e.g., “text”

property features: DataFrame

pandas dataframe with features, columns are features, rows are documents’ lines

property labels: Series

a vector of labels for each line

save(path: str = '/tmp', csv_only: bool = False) → str

Parameters:

• path – path to the directory where the dataset will be saved

• csv_only – whether to save only csv-file instead of saving dataset as a directory with csv, pkl and json files

Returns:

path to the directory where the dataset is saved

class scripts.train.trainers.base_sklearn_line_classifier.BaseClassifier(**kwargs: dict)

Bases: XGBClassifier

Base class for a classifier. See documentation of XGBClassifier to get more details.

class scripts.train.trainers.base_sklearn_line_classifier.BaseSklearnLineClassifierTrainer(data_url: str, logger: Logger, feature_extractor: AbstractFeatureExtractor, path_out: str, path_scores: str | None = None, path_features_importances: str | None = None, tmp_dir: str | None = None, train_size: float = 0.75, classifier_parameters: dict | None = None, label_transformer: Callable[[str], str] | None = None, random_seed: int = 42, get_sample_weight: Callable[[LineWithLabel], float] | None = None, n_splits: int = 10, *, config: dict)

Base class for training BaseClassifier. It provides data loading and saving, classifier training and cross-validation. During cross-validation, classifier errors are saved into errors directory inside tmp_dir.

__get_labels(data: List[List[LineWithLabel]]) → List[str]

Parameters:

data – list of documents, which are lists of lines with labels of the training dataset

Returns:

a flattened list of lines’ labels

__init__(data_url: str, logger: Logger, feature_extractor: AbstractFeatureExtractor, path_out: str, path_scores: str | None = None, path_features_importances: str | None = None, tmp_dir: str | None = None, train_size: float = 0.75, classifier_parameters: dict | None = None, label_transformer: Callable[[str], str] | None = None, random_seed: int = 42, get_sample_weight: Callable[[LineWithLabel], float] | None = None, n_splits: int = 10, *, config: dict) → None

Parameters:

• data_url – url to download training data for DataLoader

• logger – logger for logging details of classifier training

• feature_extractor – feature extractor for making feature matrix from document lines

• path_out – full path (with filename) to save a trained model

• path_scores – full path to the JSON file to save the scores (accuracy) of a trained classifier (won’t be saved if None)

• path_features_importances – full path to the XLSX file to save information about most important features for classifier (won’t be saved if None)

• tmp_dir – path to the directory where to save downloaded training data and the classifier’s errors, “/tmp” if None is provided

• train_size – proportion of the training data, value can be in the diapason (0;1) or (0;100)

• classifier_parameters – parameters for the classifier initialization

• label_transformer – function for mapping initial data labels into the labels for classifier training, labels identity if None is provided

• random_seed – seed for the classifier initialization

• get_sample_weight – function for sample_weight calculating: LineWithLabel->weight [0,1] for setting line importance during classifier training, LineWithLabel->1 if None is provided

• n_splits – number of data splits for cross-validation

• config – any custom configuration

__save_dataset_lines(data: List[List[LineWithLabel]], path: str = '/tmp', csv_only: bool = False) → str

Save training dataset in form of a feature matrix (LineClassifierDataset).

Parameters:

• data – list of documents, which are lists of lines with labels of the training dataset

• path – path to the directory where the dataset will be saved

• csv_only – whether to save only csv-file instead of saving dataset as a directory with csv, pkl and json files

Returns:

path to the directory where the dataset is saved

_cross_val(data: List[List[LineWithLabel]], save_errors_images: bool) → dict

Cross-validate the classifier and save its errors on validation data

Parameters:

• data – list of documents, which are lists of lines with labels of the training dataset

• save_errors_images – whether to visualize errors line classification (ErrorsSaver)

Returns:

dictionary with classifier scores during cross-validation

abstract _get_classifier() → BaseClassifier

Initialize the classifier.

Returns:

classifier instance for training

_save_features_importances(cls: BaseClassifier, feature_names: List[str]) → None

Save information about most important features for classifier during training into a file with path self.path_features_importances.

Parameters:

• cls – classifier trained on the features with names feature_names

• feature_names – column names of the feature matrix, that was used for classifier training

fit(no_cache: bool = False, cross_val_only: bool = False, save_dataset: bool = False, save_errors_images: bool = False) → None

Fit the line classifier with cross-validation and errors statistics saving.

Parameters:

• no_cache – whether to use cached training data (DataLoader)

• cross_val_only – whether to execute only cross-validation, without training and saving a resulting classifier

• save_dataset – whether to save training dataset in form of a feature matrix (LineClassifierDataset)

• save_errors_images – whether to visualize errors line classification (ErrorsSaver)

class scripts.train.trainers.xgboost_line_classifier_trainer.XGBoostLineClassifierTrainer(data_url: str, logger: Logger, feature_extractor: AbstractFeatureExtractor, path_out: str, path_scores: str | None = None, path_features_importances: str | None = None, tmp_dir: str | None = None, train_size: float = 0.75, classifier_parameters: dict | None = None, label_transformer: Callable[[str], str] | None = None, random_seed: int = 42, get_sample_weight: Callable[[LineWithLabel], float] | None = None, n_splits: int = 10, *, config: dict)

Bases: BaseSklearnLineClassifierTrainer

Trainer of XGBoost line classifier. See documentation of XGBClassifier to get more details.

_get_classifier() → XGBClassifier

Initialize the XGBClassifier.

Returns:

XGBClassifier instance for training

_save_features_importances(cls: XGBClassifier, feature_names: List[str]) → None

Save information about most important features for XGBClassifier using xgbfir library.

Parameters:

• cls – XGBClassifier trained on the features with names feature_names

• feature_names – column names of the feature matrix, that was used for classifier training

class scripts.train.trainers.errors_saver.ErrorsSaver(errors_path: str, dataset_path: str, logger: Logger, *, config: dict)

Class for saving line classifier’s errors during training. Errors are saved in the directory with a path errors_path as a list of TXT files. Each TXT file contains information about lines with similar errors (same substitution true_type -> predicted_type).

In addition, images can be saved for misclassified lines visualization (misclassified lines are highlighted by a bounding box).

save_errors(error_cnt: Counter, errors_uids: List[str], csv_path: str, save_errors_images: bool = False) → None

Parameters:

• error_cnt – counter of label pairs (y_true, y_pred) where y_true != y_pred

• errors_uids – list of lines’ uids corresponding misclassified lines

• csv_path – path to the csv-file with dataset in form of a feature matrix (LineClassifierDataset)

• save_errors_images – whether to save images with highlighted misclassified lines, if True images will be saved in “errors_images.zip”

Classification

class dedoc.structure_extractors.line_type_classifiers.abstract_line_type_classifier.AbstractLineTypeClassifier(*, config: dict | None = None)

Abstract class for lines classification with predict method.

abstract predict(lines: List[LineWithMeta]) → List[str]

Predict the line type according to some domain. For this purpose, some pretrained classifier may be used.

Parameters:

lines – list of document lines

Returns:

list predicted labels for each line

class dedoc.structure_extractors.line_type_classifiers.abstract_pickled_classifier.AbstractPickledLineTypeClassifier(*, config: dict | None = None)

Bases: AbstractLineTypeClassifier, ABC

Abstract class for lines classification with functionality of loading and saving a classifier.

load(classifier_type: str, path: str) → Tuple[XGBClassifier, dict]

Load the pickled classifier with parameters for a feature extractor.

Parameters:

• classifier_type – name of the classifier to load from huggingface in case path does not exist (https://huggingface.co/dedoc/line_type_classifiers)

• path – path from where to load the pickled classifier

Returns:

loaded XGBClassifier and parameters for a feature extractor

static save(path_out: str, classifier: XGBClassifier, parameters: dict) → str

Save the classifier (with initialization parameters for a feature extractor) into the .zip file with path=`path_out`

• classifier -> classifier.json

• parameters -> parameters.json

Parameters:

• path_out – path (with file name) where to save the object

• classifier – classifier to save

• parameters – feature extractor parameters to save

Returns:

the resulting path of the saved file