Identify, Locate, Link: End-to-End Key-Value Extraction from Document Images

Authors:A. Said Gurbuz (1 and 2), Ahmed Nassar (1), Christoph Auer (1), Maksym Lysak (1), Lucas Morin (1), Matteo Omenetti (1), Tim Strohmeyer (1), Panagiotis Vagenas (1), Nikolaos Livathinos (1), Michele Dolfi (1), Peter Staar (1) ((1) IBM Research Zurich, (2) ETH Zurich)

Abstract:Document processing pipelines traditionally cascade optical character recognition (OCR) engines with downstream models for structured information extraction, leading to multi-stage error propagation. We fine-tune SmolDocling, a compact 256M-parameter vision-language model (VLM), to perform end-to-end key-value extraction directly from document images, jointly solving identification, localization, and association in a single pass without OCR preprocessing. We extend DocTags with specialized key, value, region, and link tags, enabling many-to-many relationships in a unified output sequence. To address data limitations, we design an augmentation pipeline combining synthetic form filling and graph-based crops that preserve complete key-value subgraphs. We further introduce a layout-aware evaluation framework extending text matching with spatial bounding box verification. On FUNSD, XFUND, and a large-scale private dataset, our model outperforms larger zero-shot VLM baselines under layout-aware evaluation, while being 27 times smaller than Qwen2.5-VL (7B) and over 5 times faster at inference. The model weights will be released publicly after publication.

Accepted at ICDAR 2026. 17 pages, 6 figures, 7 tables

Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)

arXiv:2608.20868 [cs.CV]

(or arXiv:2608.20868v1 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2608.20868 arXiv-issued DOI via DataCite (pending registration)