OCR in 2026: Classical Pipelines, VLMs, and Document AI

Viacheslav Dubrov

2026-03-04 · Updated 2026-08-10 · 12 min · Retrieval and Language Systems

OCR leaderboards disagree because they test different documents, outputs, and judges. An early-2026 snapshot of OmniDocBench and OCR Arena produced sharply different model orderings; the scores are not interchangeable, but the disagreement is useful. A production choice needs documents and metrics from the actual workload.

Vision-language models (VLMs) can handle layout, handwriting, tables, and degraded images that break a plain text-recognition pipeline. Traditional engines remain competitive on clean print, especially when CPU latency and operating cost matter. The dated snapshot below includes PaddleOCR-VL 1.6 and dots.mocr, with different hardware, privacy, and output trade-offs. Check each project before making a current choice.

Companion repo: The OCR Gauntlet contains three notebooks. Its main notebook compares up to five OCR engines on five downloaded samples and reports CER, WER, ANLS, latency, and estimated cost. The checked-in run contains Tesseract, Docling + Tesseract, Mistral OCR v3, and a Gemini run; dots.ocr was unavailable. Do not use the Gemini row for model comparison: the runner requests gemini-2.5-flash, but the checked-in notebook labels that output Gemini 3 Flash.

This guide is for engineers choosing an OCR or document-AI pipeline for real workloads where text, layout, tables, or extracted fields matter.

After reading it, you should be able to choose a baseline, compare models on task-specific metrics, and route uncertain or high-risk fields to validation or review.

For the compact model-selection version, see Best OCR Models in 2026: Classical OCR, PaddleOCR-VL, VLMs.

OCR now sets downstream quality

OCR has long powered archives, postal systems, accessibility tools, and document management. RAG and document agents made its failure modes visible to a wider group of engineers: a downstream model cannot recover text or table structure that extraction discarded.

Your RAG system’s retrieval quality is capped by OCR quality. If extraction garbles a table, misreads a date, or drops a paragraph, later chunking and embedding changes cannot recover the missing information. Those errors can hide a contract clause, change an invoice total, or corrupt a medical record.

OCR is therefore part of retrieval and agent infrastructure, alongside parsing, chunking, embedding, and indexing. Its errors need their own evaluation rather than being absorbed into a single end-to-end score.

What OCR means in the age of foundation models

OCR converts text in an image into machine-readable characters. Document AI is the broader system around it: layout analysis, table and formula parsing, field extraction, semantic reasoning, provenance, and validation. Some papers use “OCR-2.0” for end-to-end models that combine several of those stages, but that label should not erase the distinction between recognition and document understanding.

The traditional OCR pipeline has three core stages:

1. Text detection: locate regions containing text (e.g., CRAFT, DBNet).
2. Text recognition: convert detected regions into character sequences (e.g., CRNN).
3. Post-processing: spell-checking and language-model correction.

This works well for clean documents, but detection, recognition, and post-processing errors can compound. Measure both character accuracy and downstream field accuracy so a readable page does not hide a wrong total or identifier.

OCR-2.0 collapses more of that pipeline into a vision encoder plus language decoder. Models such as GOT-OCR 2.0 can emit text and structure together, while general VLMs can also map fields to a requested schema. The trade-offs are workload-specific latency, GPU or API cost, and the risk of plausible text that is absent from the image.

A practical caveat: don’t let the “end-to-end” label fool you. In production, OCR-2.0 unifies model inference, not the entire document pipeline. You still need PDF rasterization to produce images, image normalization (deskew, DPI adjustment) for consistent quality, and output parsing to pull structured fields out of the model’s text. The pipeline got shorter, not extinct.

What OCR benchmarks measure and miss

The following datasets illustrate how OCR tasks produce different metrics. Dataset sizes and metrics describe the named dataset version; use each project’s current leaderboard for model scores.

The “benchmark vs. arena” gap

Automated benchmark rankings can conflict with human preference because the input distribution and judging criteria differ.

In the OCR Arena, users vote blindly on head-to-head outputs. On 2026-08-09 its live ordering placed Gemini 3 Flash above GLM-OCR and DeepSeek-OCR, while the versioned OmniDocBench table later in this article ranked specialized document models above Gemini 3 Flash. Arena values change after new battles, so the ordering is directional evidence rather than a reproducible benchmark snapshot. The two leaderboards should not be combined into one score because one uses dataset metrics and the other preference votes.

The likely drivers include document mix, output formatting, language coverage, and judge criteria. Published numbers are useful for screening, but final selection needs a held-out set from the target workload.

Traditional OCR engines: still relevant

If traditional engines are worse on complex data, why use them? Because they’re fast and cheap on clean structured data.

Traditional engines are useful baselines because they can run locally on CPU. Their latency and accuracy depend on the selected model, page resolution, language, preprocessing, and hardware, so benchmark them on the same labeled pages used for VLM evaluation.

Tesseract for clean print

Tesseract (v5.5.x, Apache 2.0) is a mature CPU-only engine with 100+ language packs. Clean-print accuracy can be high after appropriate rasterization and preprocessing, but handwriting, scene text, and complex layouts need separate testing. Its main advantage is a small, local CPU deployment rather than a universal accuracy lead.

EasyOCR

EasyOCR pairs a CRAFT detector with a CRNN recognizer. With full PyTorch GPU acceleration, it’s a fast option for quick prototyping and scene text.

The snippet requires pip install easyocr, PyTorch, EasyOCR’s downloaded model weights, and a local receipt.jpg. It is syntax-checked but not executed by the repository’s Markdown runner.

import easyocr

# Three lines for complete OCR
reader = easyocr.Reader(["en"])
result = reader.readtext("receipt.jpg")
# Returns: [(bbox, text, confidence), ...]

PaddleOCR 3

PaddleOCR 3 packages maintained OCR, document parsing, and deployment pipelines. The current quick start uses the predict() API and explicit orientation settings. Pin the paddleocr package and selected pipeline because 2.x examples using .ocr(..., cls=True) do not match the 3.x API.

The snippet requires pip install "paddleocr>=3,<4", a compatible PaddlePaddle runtime, downloaded model weights, and a local receipt.jpg. It is syntax-checked but not executed by the repository’s Markdown runner.

from paddleocr import PaddleOCR

ocr = PaddleOCR(
    use_doc_orientation_classify=False,
    use_doc_unwarping=False,
    use_textline_orientation=False,
)

for result in ocr.predict("receipt.jpg"):
    result.print()
    result.save_to_json("output")

Specialized and general VLM options

Crooked receipts, skewed product labels, handwriting, and dense layouts are where specialized or general VLMs become worth testing against traditional engines.

The specialized OCR wave

The model list in this section is the snapshot checked on 2026-08-09. It is not a current ranking. Specialized document-parsing models published from 2024 through 2026 include:

• PaddleOCR-VL 1.6: A two-stage pipeline that performs layout analysis, then uses a 0.9B VLM component on detected regions. PaddleOCR reports 109 languages and 96.3 on OmniDocBench v1.6; keep that vendor result attached to the named pipeline and benchmark version.
• dots.mocr (3B): The March 2026 successor to dots.ocr-1.5 parses text and structured graphics, including an SVG-oriented variant. The original dots.ocr remains a separate 2025 model.
• GOT-OCR 2.0: A 580M-parameter unified model that emits plain text and formatted outputs such as Markdown and LaTeX. Its official repository does not publish a minimum VRAM figure, so measure peak memory with the chosen runtime, precision, image size, and output limit.
• DeepSeek-OCR2: The checkpoint documented in the official repository at the snapshot date, succeeding the original 3B-class DeepSeek-OCR model that introduced “contextual optical compression.” Treat throughput figures for either generation as hardware- and dataset-specific.
• Mistral OCR 4.0 (mistral-ocr-4-0), snapshot 2026-08-09: The proprietary text-and-structure extraction service documented by Mistral. OCR 3 (mistral-ocr-2512) remains available for existing integrations and is the version used by the companion notebook. The companion uses OCR 3 intentionally for reproducibility, not because OCR 3 is newer. Pricing and benchmark numbers change, so check the current model card and provider terms when comparing them.

Frontier VLMs

General VLMs are another option when the task combines extraction with visual or semantic reasoning. The examples below reflect the article’s early-2026 snapshot, not a current ranking:

• Qwen3-VL (Alibaba): An open-weight VLM family with several sizes and long-context variants.
• Gemini 3 Flash (Google): A hosted multimodal model that ranked highly in the cited OCR Arena snapshot.
• Claude Opus 4.6 (Anthropic): A hosted general VLM with structured-output support.
• GPT-5.2 (OpenAI): A hosted general VLM for mixed visual and text inputs.

Measure latency by tier

Measure page latency with the actual resolution, batch size, hardware or provider region, and output length. Keep preprocessing and retries in the total; a model-only timing cannot price a successful page.

Metrics: measuring what matters

Pick a metric that matches the output type:

• CER and WER for plain text. Character and Word Error Rate depend on normalization choices such as case, whitespace, and punctuation, so fix the comparison protocol before comparing models.
• EMR and Field F1 for forms and receipts. Exact Match Rate is binary, which is what you want for tax IDs and totals. Field F1 balances precision and recall per field type.
• TEDS for tables. Tree-Edit-Distance-based Similarity compares predicted and reference HTML trees, catching structural and cell-content errors that CER hides.
• ANLS for document VQA. Average Normalized Levenshtein Similarity gives partial credit for answers with minor OCR errors.

For implementations: jiwer handles CER/WER out of the box, and TEDS implementations live in the OmniDocBench repo.

Testing VLMs with OpenRouter

OpenRouter provides an OpenAI-compatible gateway to models from several providers. Model IDs and supported request features change, so verify them against the gateway’s current catalog before running the example.

The snippet requires pip install openai, an OPENROUTER_API_KEY, network access, a local receipt.jpg, and model IDs still supported by OpenRouter. It is syntax-checked but not executed by the repository’s Markdown runner.

import base64
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

def extract_text(image_path: str, model: str) -> str:
    with open(image_path, "rb") as f:
        image_b64 = base64.b64encode(f.read()).decode()

    response = client.chat.completions.create(
        model=model,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract all text from this image, preserving layout as markdown."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
            ],
        }],
        max_tokens=4096,
    )
    return response.choices[0].message.content

# Compare models by changing one string
models = [
    "google/gemini-3-flash-preview",
    "anthropic/claude-sonnet-4.5",
    "qwen/qwen3-vl-8b-instruct",
]
for model in models:
    print(f"\n--- {model} ---\n{extract_text('receipt.jpg', model)[:200]}...")

These model IDs were checked against the OpenRouter catalog on 2026-08-09. Check the catalog again before running the example.

For structured extraction, use response_format with a JSON schema when the selected model and gateway support it. This can make the response parseable; it does not validate extracted values against the image. The following block repeats its setup so it is independently readable. It still requires pip install openai, an OPENROUTER_API_KEY, network access, a local receipt.jpg, and a model that supports JSON Schema; the repository runner only syntax-checks it.

import base64
import json
import os
from openai import OpenAI

with open("receipt.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

response = client.chat.completions.create(
    model="google/gemini-3-flash-preview",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Extract the receipt fields from this image."},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
        ],
    }],
    max_tokens=4096,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "receipt",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "vendor": {"type": "string"},
                    "date": {"type": "string"},
                    "items": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "properties": {
                                "description": {"type": "string"},
                                "amount": {"type": "number"},
                            },
                            "required": ["description", "amount"],
                            "additionalProperties": False,
                        },
                    },
                    "total": {"type": "number"},
                },
                "required": ["vendor", "date", "items", "total"],
                "additionalProperties": False,
            },
        },
    },
)
receipt = json.loads(response.choices[0].message.content)

Benchmark results: what the numbers actually show

The table below is one versioned snapshot: OmniDocBench v1.6_full, official README at commit 09ba2b606662695b16aafe5f5e36b7ef020e11a8, published 2026-04-10 and accessed 2026-08-09. All four rows come from that pinned table. The table does not mix values from earlier paper tables or other leaderboards.

The conclusion is limited to this OmniDocBench version: model size alone does not predict document-parsing score.

The companion notebook computes CER, WER, ANLS, latency, and estimated cost on its five downloaded samples. Exclude its mislabeled Gemini row unless the model identity is corrected and the samples are rerun.

Deploying OCR in production

A tiered architecture can separate CPU preprocessing from GPU or API inference and reserve expensive paths for documents that need them.

Note on orchestration: Tools such as Docling can coordinate conversion and batch processing. Retry policy still belongs in the surrounding application or service, and routing still needs its own quality labels and thresholds.

The tiered fallback pattern

Start with the cheapest path that meets the quality target, then calibrate routing on labeled pages:

0. Check for embedded text (Tier 0). For PDFs, inspect the text layer with PyMuPDF or pdfplumber before rasterizing, but validate that the layer is complete and correctly ordered.
1. Attempt with a fast model. Use a traditional engine for document classes on which it meets the target.
2. Evaluate calibrated confidence. Combine model confidence with document class, field criticality, and validation rules.
3. Escalate to a stronger model. Route uncertain pages to a specialized or general VLM.
4. Escalate high-risk failures to a human. Human review is a separate tier for values whose cost of error exceeds the automation benefit.

Note on confidence: Raw character probabilities are not automatically calibrated to field correctness. The area-weighted function below is a baseline for page-level aggregation, not a universal router. Calibrate it against labeled pages, and give critical fields their own rules because a page average can hide a wrong ID or total.

def area_weighted_confidence(page):
    """Compute area-weighted confidence from a PaddleOCR 3.x result."""
    total_area, weighted_sum = 0, 0
    for (x_min, y_min, x_max, y_max), score in zip(
        page["rec_boxes"], page["rec_scores"]
    ):
        area = (x_max - x_min) * (y_max - y_min)
        weighted_sum += score * area
        total_area += area
    return weighted_sum / total_area if total_area > 0 else 0

Cost analysis at scale

There is no universal page-volume break-even between an API and self-hosting. Build the comparison from the same workload:

Use a shared formula: monthly pages × cost per successful page + review cost + fixed operating cost. A “successful page” must meet the same text, table, and field criteria on both paths. Provider prices and GPU rentals change too quickly to embed as a durable procurement estimate.

Error handling: the hallucination problem

VLM errors can be contextually plausible and factually wrong. A receipt total of “$42.50” might become “$45.20”: syntactically valid, but invisible to a spell-checker.

Synthetic failure example: A VLM extracts three receipt line items and a stated total that agree with one another, but one digit differs from the image. Internal arithmetic passes even though the extraction is wrong. This is why validation needs image-grounded labels or an independent review path, not only consistency checks.

A few practical mitigations:

• Arithmetic reconciliation. When the schema exposes them, verify subtotal + tax + fees + shipping - discounts, within the currency’s rounding tolerance, against the stated total. Route missing components or mismatches to review.
• Regex sanity checks for dates (no month 13), phone numbers (correct digit count), and currency formats.
• Cross-model verification. Run critical fields through two different models and flag disagreements.
• Independent OCR cross-check. Run a second extraction path on critical figures and flag disagreements. Agreement raises confidence only when the two paths have sufficiently different failure modes; it is not proof of correctness.

Key takeaways

1. Match the model tier to a labeled document class. Traditional engines can be sufficient for clean text; specialized and general VLMs should earn their added cost on harder pages.
2. Do not merge unlike leaderboards. OmniDocBench metrics and OCR Arena preferences answer different questions.
3. Calibrate routing. Confidence thresholds, document classes, field criticality, and human-review policy belong in one evaluation.
4. Validate plausible output. Schema conformance and internal arithmetic cannot prove that a value appears in the image.
5. Price successful pages. Include retries, review, fixed operations, and quality gates when comparing APIs with self-hosting.

Preprocessing and detection still matter, but production OCR now also requires routing, task-specific evaluation, and defenses against plausible extraction errors.

References

• OCR Arena Leaderboard - Crowdsourced, head-to-head model battles
• The OCR Gauntlet repo - Runnable notebooks for comparing OCR engines, inspecting Docling output, and estimating cost
• OmniDocBench - End-to-end document parsing eval
• dots.ocr - About 3B total parameters, including a 1.7B language model
• PaddleOCR - Traditional OCR toolkit and models
• OpenRouter - Unified access gateway for A/B testing models