Adding implemented extractor to the main pipeline

See also

Before doing this step, implementation of a structure extractor should be done. Here there is an instruction how to add a new structure extractor.

All implemented document handlers are linked in dedoc/manager_config.py You may edit this file or create your own manager_config with dedoc handlers you need and your custom handlers directly in your code.

Let’s consider an example. In the tutorial about structure extractor implementation, we implemented the class ArticleStructureExtractor for handling English scientific articles in PDF format. Example of a manager config with PDF handlers and the implemented structure extractor:

from article_structure_extractor import ArticleStructureExtractor

from dedoc import DedocManager
from dedoc.attachments_handler import AttachmentsHandler
from dedoc.converters import ConverterComposition
from dedoc.metadata_extractors import MetadataExtractorComposition, PdfMetadataExtractor
from dedoc.readers import PdfAutoReader, ReaderComposition
from dedoc.structure_constructors import StructureConstructorComposition, TreeConstructor
from dedoc.structure_extractors import StructureExtractorComposition

manager_config = dict(
    converter=ConverterComposition(converters=[]),
    reader=ReaderComposition(readers=[PdfAutoReader()]),
    structure_extractor=StructureExtractorComposition(extractors={ArticleStructureExtractor.document_type: ArticleStructureExtractor()},
                                                      default_key=ArticleStructureExtractor.document_type),
    structure_constructor=StructureConstructorComposition(constructors={"tree": TreeConstructor()}, default_constructor=TreeConstructor()),
    document_metadata_extractor=MetadataExtractorComposition(extractors=[PdfMetadataExtractor()]),
    attachments_handler=AttachmentsHandler(),
)

manager = DedocManager(manager_config=manager_config)

After initialization of manager, it can be used for document parsing, document_type parameter is equal to the ArticleStructureExtractor.document_type attribute.

result = manager.parse(file_path=file_path, parameters={"document_type": "article"})