Lines classification

See also

Training classifier is performed on the custom dataset described in Dataset creation. Before classification, Features extraction to feed the classifier should be done.

Here we describe two steps:

1. Writing and running training script to get weights of a trained classifier.

2. Adding a classifier class in code of Dedoc for further usage.

Classifier training

Training a line classifier is quite typical, training scripts for different domains can be found in scripts/train. But during the process of adjusting the extracted features and classifier’s hyperparameters for enhancing classifier’s accuracy, a lot of problems may be encountered.

There are several classes, that are used during training, in particular, for analysis of a training process:

• DataLoader downloads data from cloud and transforms json-lines into Python classes.

• LineClassifierDataset represents dataset of document lines in form of a feature matrix.

• BaseClassifier is a wrapper of XGBClassifier.

• BaseSklearnLineClassifierTrainer – base class for training BaseClassifier.

• XGBoostLineClassifierTrainer is a trainer for XGBClassifier with ability to save importance of dataset features.

• ErrorsSaver is used for saving line classifier’s errors during training.

With this functionality, one may train classifiers of document lines, analyse the most important features for classifiers, visualize and analyse errors made by classifiers. After the analysis, training data or feature extraction process may be changed in order to improve classification results.

Example: training a classifier for English articles

To train a classifier for English articles, we will use the class XGBoostLineClassifierTrainer – please read the documentation of the class (and its base class) to learn its main parameters.

Below there is a code of trainer initialization and running a training process. As a result, the trained classifier will be saved in classifier_path, its scores and importance of used features – in path_scores and path_feature_importances correspondingly.

# trainer initialization
trainer = XGBoostLineClassifierTrainer(
    data_url="url with training dataset",
    logger=config.get("logging", logging.getLogger()),
    feature_extractor=feature_extractor,
    path_out=classifier_path,
    path_scores=path_scores,
    path_features_importances=path_feature_importances,
    tmp_dir="/tmp",
    label_transformer=skip_labels,
    classifier_parameters=classifier_parameters,
    random_seed=42,
    config=config,
)

# run training of the classifier
trainer.fit(cross_val_only=False, save_errors_images=False, no_cache=False)

The label_transformer parameter allows to rename labels of the initial labeled data. It may be useful when we want to merge two labels into one, or to ignore some labels completely – in this case, the label_transformer should return None. In our example, let’s ignore lines that have a label other. The rest lines will have their labels unchanged.

def skip_labels(label: str) -> Optional[str]:
    """
    Function for filtering `other` lines and do not train the classifier on them
    """
    if label == "other":
        return None
    return label

We also need to do path configuration for saving resulting files:

classifier_name = "article_classifier"

# configure path for saving a trained classifier
classifier_directory_path = os.path.join(os.path.expanduser("~"), ".cache", "dedoc", "resources", "line_type_classifiers")
os.makedirs(classifier_directory_path, exist_ok=True)
classifier_path = os.path.join(classifier_directory_path, f"{classifier_name}.zip")

# configure paths for saving scores and features importances (this is not obligatory)
resources_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "resources"))
assert os.path.isdir(resources_path)
path_scores = os.path.join(resources_path, "benchmarks", f"{classifier_name}_scores.json")
path_feature_importances = os.path.join(resources_path, "feature_importances", f"{classifier_name}_feature_importances.xlsx")

Now we have almost everything we need, let’s initialize the remaining parameters required by a trainer:

# features extractor for the classifier
feature_extractor = ArticleFeatureExtractor()

# parameters of the XGBClassifier (https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.XGBClassifier)
classifier_parameters = dict(learning_rate=0.5, n_estimators=600, booster="gbtree", tree_method="hist", max_depth=3, colsample_bynode=0.8)

# dedoc configuration (just in case)
config = get_config()

To run the script without errors, one should provide data_url parameter to download the training dataset. Please, see training scripts in scripts/train where urls to the existing datasets are provided to get some examples. Obtaining a training dataset is described in Dataset creation in more details.

Thus, writing a training script is finished. The file with the script should be placed in the scripts/train directory. The full training script may be downloaded here, or you may copy the code below.

import logging
import os
from typing import Optional

from article_feature_extractor import ArticleFeatureExtractor

from dedoc.config import get_config
from scripts.train.trainers.xgboost_line_classifier_trainer import XGBoostLineClassifierTrainer


def skip_labels(label: str) -> Optional[str]:
    """
    Function for filtering `other` lines and do not train the classifier on them
    """
    if label == "other":
        return None
    return label


classifier_name = "article_classifier"

# configure path for saving a trained classifier
classifier_directory_path = os.path.join(os.path.expanduser("~"), ".cache", "dedoc", "resources", "line_type_classifiers")
os.makedirs(classifier_directory_path, exist_ok=True)
classifier_path = os.path.join(classifier_directory_path, f"{classifier_name}.zip")

# configure paths for saving scores and features importances (this is not obligatory)
resources_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "resources"))
assert os.path.isdir(resources_path)
path_scores = os.path.join(resources_path, "benchmarks", f"{classifier_name}_scores.json")
path_feature_importances = os.path.join(resources_path, "feature_importances", f"{classifier_name}_feature_importances.xlsx")

# features extractor for the classifier
feature_extractor = ArticleFeatureExtractor()

# parameters of the XGBClassifier (https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.XGBClassifier)
classifier_parameters = dict(learning_rate=0.5, n_estimators=600, booster="gbtree", tree_method="hist", max_depth=3, colsample_bynode=0.8)

# dedoc configuration (just in case)
config = get_config()

# trainer initialization
trainer = XGBoostLineClassifierTrainer(
    data_url="url with training dataset",
    logger=config.get("logging", logging.getLogger()),
    feature_extractor=feature_extractor,
    path_out=classifier_path,
    path_scores=path_scores,
    path_features_importances=path_feature_importances,
    tmp_dir="/tmp",
    label_transformer=skip_labels,
    classifier_parameters=classifier_parameters,
    random_seed=42,
    config=config,
)

# run training of the classifier
trainer.fit(cross_val_only=False, save_errors_images=False, no_cache=False)

Classifier adding

Since we trained a new classifier and saved its weights, we can use it in dedoc for inference. Do do this, an inheritor of the AbstractPickledLineTypeClassifier should be implemented. In this class, the predict() method is the most important part. This method is used for classification of the list of input LineWithMeta related to one document. As a result, the list of line types is obtained.

The commonplace pipeline in the predict() method works as follows:

• Forming a feature matrix for the input list of document lines using a feature extractor;

• Calling a classifier’s prediction method (classes probabilities also may be obtained);

• If needed, some post-processing of probabilities can be done (optional).

Example: implementation of a line classifier for English articles

In the features extraction tutorial, we used an example domain of English articles and implemented ArticleFeatureExtractor. Here we continue this example and implement ArticleLineTypeClassifier.

In the __init__ method of the class, let’s load classifier weights and initialize a feature extractor:

    def __init__(self, path: str, *, config: Optional[dict] = None) -> None:
        super().__init__(config=config)
        self.classifier, feature_extractor_parameters = self.load("article", path)
        self.feature_extractor = ArticleFeatureExtractor()

Let’s implement the aforesaid commonplace pipeline in the predict method.

Forming a feature matrix:

        features = self.feature_extractor.transform([lines])

Calling a classifier’s prediction method:

        labels_probability = self.classifier.predict_proba(features)

Executing some custom post-processing of the predicted classes probabilities:

        # set empty lines as raw_text
        raw_text_id = list(self.classifier.classes_).index("raw_text")
        empty_line = [line.line.strip() == "" for line in lines]
        labels_probability[empty_line, :] = 0
        labels_probability[empty_line, raw_text_id] = 1

        # work with a title
        labels = [self.classifier.classes_[i] for i in labels_probability.argmax(1)]
        first_non_title = 0
        for i, line in enumerate(lines):
            if "Abstract" in line.line or labels[i] not in ("title", "raw_text"):
                first_non_title = i
                break

        # probability=1 for title before the body, probability=0 for title after body of document has begun
        title_id = list(self.classifier.classes_).index("title")
        labels_probability[:first_non_title, :] = 0
        labels_probability[:first_non_title, title_id] = 1
        labels_probability[first_non_title:, title_id] = 0

        labels = [self.classifier.classes_[i] for i in labels_probability.argmax(1)]

The file with the line classifier should be placed in dedoc/structure_extractors/line_type_classifiers. The resulting file with the classifier for English articles (our example) can be downloaded here.

Or you may copy the code below.

from typing import List, Optional

from article_feature_extractor import ArticleFeatureExtractor

from dedoc.data_structures.line_with_meta import LineWithMeta
from dedoc.structure_extractors.line_type_classifiers.abstract_pickled_classifier import AbstractPickledLineTypeClassifier


class ArticleLineTypeClassifier(AbstractPickledLineTypeClassifier):

    def __init__(self, path: str, *, config: Optional[dict] = None) -> None:
        super().__init__(config=config)
        self.classifier, feature_extractor_parameters = self.load("article", path)
        self.feature_extractor = ArticleFeatureExtractor()

    def predict(self, lines: List[LineWithMeta]) -> List[str]:
        if len(lines) == 0:
            return []

        features = self.feature_extractor.transform([lines])
        labels_probability = self.classifier.predict_proba(features)

        # set empty lines as raw_text
        raw_text_id = list(self.classifier.classes_).index("raw_text")
        empty_line = [line.line.strip() == "" for line in lines]
        labels_probability[empty_line, :] = 0
        labels_probability[empty_line, raw_text_id] = 1

        # work with a title
        labels = [self.classifier.classes_[i] for i in labels_probability.argmax(1)]
        first_non_title = 0
        for i, line in enumerate(lines):
            if "Abstract" in line.line or labels[i] not in ("title", "raw_text"):
                first_non_title = i
                break

        # probability=1 for title before the body, probability=0 for title after body of document has begun
        title_id = list(self.classifier.classes_).index("title")
        labels_probability[:first_non_title, :] = 0
        labels_probability[:first_non_title, title_id] = 1
        labels_probability[first_non_title:, title_id] = 0

        labels = [self.classifier.classes_[i] for i in labels_probability.argmax(1)]
        assert len(labels) == len(lines)
        return labels