Understanding GLiFormer’s Benchmarks and Schema-Driven Extraction

gliformer-large-v1 is a large 575.6-million-parameter model from knowledgator designed for schema-driven information extraction and text representation. This model, which is licensed under Apache-2.0, uses a shared DeBERTa encoder with task-specific heads for named entity recognition (NER), classification, joint relation extraction, structured record extraction, and text embeddings.

The layout-aware architecture of this model allows it to process both text and document-layout inputs, such as PDFs. However, the published examples and evaluation results mainly focus on English text. To run gliformer, one can use the gliformer Python library or the GLiFormer source framework in Python 3.10 or newer.

The model's performance varies significantly depending on the task and dataset. On NER datasets, the model reaches a mean strict entity F1 of 50.91 across 26 datasets and a mean macro-F1 of 75.03 across 13 classification datasets. However, joint relation extraction shows subpar performance on several benchmarks. The strongest performance is seen in structured record extraction, with an 91.10 score on 500 examples, but this score utilizes order-free, boundary-tolerant JSON evaluation rather than exact JSON matching.

The model is most useful for schema-driven entity extraction from English text. It is particularly beneficial when entity types vary by project and labels need to be provided at inference time instead of training a separate fixed-label NER model. Additionally, text classification with custom candidate classes can be performed using the model, which supports named groups such as separate sentiment and topic label sets.

Structured record extraction from semi-structured prose is another area where the model shines, producing Python dictionaries from simple field lists or nested Pydantic schemas.

The model can also be used for joint relation extraction with a closed schema, extracting entities and relations in one configured task, such as (Alice, works_at, Acme). Finally, the model can be used to embed short text for similarity workflows, returning 1,024-dimensional vectors that can be compared using cosine similarity. This functionality can be used in semantic search prototypes, duplicate detection, clustering, and retrieval features.

However, there are several limitations to the model, including its inability to guarantee extraction of every fact and its weak performance on relation extraction. The model also lacks an open relation head, requiring use of the joint_relations method instead. Lastly, the model's structure makes it challenging to attach records to the wrong parent, omit fields, or produce incorrect values, with the 91.10 score not representing an exact JSON match.