The most comprehensive benchmark to date for evaluating document understanding capabilities of Vision-Language Models (VLMs).

What is it?
A unified evaluation suite covering 6 core IDP tasks across 16 datasets and 9,229 documents:

• Key Information Extraction (KIE)
• Visual Question Answering (VQA)
• Optical Character Recognition (OCR)
• Document Classification
• Table Extraction
• Long Document Processing (LongDocBench)
• (Coming soon: Confidence Score Calibration)

Each task uses multiple datasets, including real-world, synthetic, and newly annotated ones.

Highlights from the Benchmark

• Gemini 2.5 Flash leads overall, but surprisingly underperforms its predecessor on OCR and classification.
• All models struggled with long document understanding – top score was just 69.08%.
• Table extraction remains a bottleneck — especially for long, sparse, or unstructured tables.
• Surprisingly, GPT-4o's performance decreased in the latest version (gpt-4o-2024-11-20) compared to its earlier release (gpt-4o-2024-08-06).
• Token usage (and thus cost) varies dramatically across models — GPT-4o-mini was the most expensive per request due to high token usage.

Why does this matter?
There’s currently no unified benchmark that evaluates all IDP tasks together — most leaderboards (e.g., OpenVLM, Chatbot Arena) don’t deeply assess document understanding.

Document Variety
We evaluated models on a wide range of documents: Invoices, forms, receipts, charts, tables (structured + unstructured), handwritten docs, and even diacritics texts.

Get Involved
We’re actively updating the benchmark with new models and datasets.

This is developed with collaboration from IIT Indore and Nanonets.

Leaderboard: https://idp-leaderboard.org/
Release blog: https://idp-leaderboard.org/details/
GithHub: https://github.com/NanoNets/docext/tree/main/docext/benchmark

Feel free to share your feedback!