Lines post-processing

See also

The description of the whole pipeline for structure extraction using classification. Before post-processing, Lines classification should be done.

The last step of the structure extraction pipeline is post-processing and filling lines’ hierarchy levels. For this purpose, one should implement a structure extractor – an inheritor of the class AbstractStructureExtractor.

Classifiers of line types sometimes make mistakes, some lines can be split or merged – all this may be done during post-processing. A specific logic can be described during implementation of a method extract(). The main purpose of a structure extractor is to fill hierarchy_level attribute of the LineMetadata class. Hierarchy level for document lines and documentation of the HierarchyLevel may help in implementation of a custom structure extractor.

Example: implementation of a structure extractor for English articles

In the lines classification tutorial, we used an example domain of English articles and implemented ArticleLineTypeClassifier. Here we continue this example and implement ArticleStructureExtractor.

In the __init__ method of the class, let’s initialize a classifier and define domain-specific keywords:

    def __init__(self, *, config: Optional[dict] = None) -> None:
        super().__init__(config=config)
        path = os.path.abspath(os.path.dirname(__file__))  # path to the directory where the classifier weights are located
        self.classifier = ArticleLineTypeClassifier(path=os.path.join(path, "article_classifier.zip"), config=self.config)

        self.named_item_keywords = ("abstract", "introduction", "related work", "conclusion", "references", "appendix", "acknowledgements")

Then we should define the extract() method and fill hierarchy_level attribute according to the following assumptions:

• title lines will be merged and form a document root;

• named_item lines are children of the root, they may be hierarchical if there is a numeration of article chapters;

• author, affiliation, reference are less important than named_item and title but more important the rest lines.

With the help of the description of hierarchy level peculiarities, we decided to use the following values of level_1 and level_2 of HierarchyLevel:

• title → level_1=0, level_2=0

• named_item → level_1=1,2, level_2=1,2,3...

• author, affiliation, reference → level_1=3, level_2=1

• other line types → level_1=None, level_2=None

The implementation of the extract() method can be the following:

    def extract(self, document: UnstructuredDocument, parameters: Optional[dict] = None) -> UnstructuredDocument:
        predictions = self.classifier.predict(document.lines)
        assert len(predictions) == len(document.lines)

        for line, line_type in zip(document.lines, predictions):

            if line_type == "title":
                # for root, level_1=0, level_2=0, can_be_multiline=True
                line.metadata.hierarchy_level = HierarchyLevel.create_root()
                continue

            if line_type == "named_item":
                # for named_item, level_1=1,2, can_be_multiline=True
                line.metadata.hierarchy_level = self.__handle_named_item(line=line, prediction=line_type, level_1=1)

            if line_type in ("author", "affiliation", "reference"):
                line.metadata.hierarchy_level = HierarchyLevel(level_1=3, level_2=1, can_be_multiline=False, line_type=line_type)
                continue

            # for caption and raw_text, level_1=None, level_2=None, can_be_multiline=True
            line.metadata.hierarchy_level = HierarchyLevel.create_raw_text()
            line.metadata.hierarchy_level.line_type = line_type

        return document

Note

In the example, we didn’t fix any mistakes of the classifier in order to simplify the code. In reality, a lot of post-processing work should be done in order to get good results of structure parsing.

The whole code of the ArticleStructureExtractor can be found below.

The file with the structure extractor should be placed in dedoc/structure_extractors/concrete_structure_extractors. The resulting file with the structure extractor for English articles (our example) can be downloaded here.

Or you may copy the code below.

import os
from typing import Optional

from article_classifier import ArticleLineTypeClassifier

from dedoc.data_structures.hierarchy_level import HierarchyLevel
from dedoc.data_structures.line_with_meta import LineWithMeta
from dedoc.data_structures.unstructured_document import UnstructuredDocument
from dedoc.structure_extractors.abstract_structure_extractor import AbstractStructureExtractor
from dedoc.structure_extractors.feature_extractors.list_features.list_utils import get_dotted_item_depth


class ArticleStructureExtractor(AbstractStructureExtractor):
    """
    This class is used for extraction structure from English articles
    """
    document_type = "article"

    def __init__(self, *, config: Optional[dict] = None) -> None:
        super().__init__(config=config)
        path = os.path.abspath(os.path.dirname(__file__))  # path to the directory where the classifier weights are located
        self.classifier = ArticleLineTypeClassifier(path=os.path.join(path, "article_classifier.zip"), config=self.config)

        self.named_item_keywords = ("abstract", "introduction", "related work", "conclusion", "references", "appendix", "acknowledgements")

    def extract(self, document: UnstructuredDocument, parameters: Optional[dict] = None) -> UnstructuredDocument:
        predictions = self.classifier.predict(document.lines)
        assert len(predictions) == len(document.lines)

        for line, line_type in zip(document.lines, predictions):

            if line_type == "title":
                # for root, level_1=0, level_2=0, can_be_multiline=True
                line.metadata.hierarchy_level = HierarchyLevel.create_root()
                continue

            if line_type == "named_item":
                # for named_item, level_1=1,2, can_be_multiline=True
                line.metadata.hierarchy_level = self.__handle_named_item(line=line, prediction=line_type, level_1=1)

            if line_type in ("author", "affiliation", "reference"):
                line.metadata.hierarchy_level = HierarchyLevel(level_1=3, level_2=1, can_be_multiline=False, line_type=line_type)
                continue

            # for caption and raw_text, level_1=None, level_2=None, can_be_multiline=True
            line.metadata.hierarchy_level = HierarchyLevel.create_raw_text()
            line.metadata.hierarchy_level.line_type = line_type

        return document

    def __handle_named_item(self, line: LineWithMeta, prediction: str, level_1: int) -> HierarchyLevel:
        text = line.line.strip().lower()

        if text in self.named_item_keywords:
            # for standard headers like Introduction, Conclusion, etc.
            return HierarchyLevel(level_1=level_1, level_2=1, can_be_multiline=True, line_type=prediction)

        # get list depth for numerated headers, e.g. 1.1->2, 1.1.1->3, for lines without numeration -1 is returned
        item_depth = get_dotted_item_depth(text)

        if item_depth == -1:
            # for non-standard headers without numeration or continuation of multiline headers
            return HierarchyLevel(level_1=level_1 + 1, level_2=1, can_be_multiline=True, line_type=prediction)

        # for headers with numeration
        return HierarchyLevel(level_1=level_1, level_2=item_depth, can_be_multiline=True, line_type=prediction)