doceval — Extraction Eval Harness

Open-source eval harness for LLM extraction. Point it at your extractor and a labeled dataset: field-level accuracy, a failure taxonomy and cost per document.

Overview

In plain terms: if you’re using AI to pull data out of invoices, contracts, forms — or emails, web pages and call transcripts — doceval measures how accurate it is. A number you can show a client, an auditor, or your own team, instead of “it looks about right.”

For the technical side: you’ve built an LLM-based extractor. It seems to work. But how accurate is it? Which fields fail most — and why? Did accuracy change when you updated the prompt?

Without answers, “seems to work” is all you have. That’s not good enough for production.

doceval is an open-source eval harness that gives you those answers. Point it at your extraction function and a labeled dataset, and it produces:

Works with any extractor (Claude, GPT, regex, rules-based) and any schema. It also works on any input — what it scores is whether the extracted fields are right, which never depended on the source being a document. PDFs and scans, yes; equally emails, HTML pages, transcripts or feed dumps.

View on GitHub

How It Works

Write an extractor — a Python function that takes (doc_bytes, filepath) and returns a dict:

def extract(doc_bytes: bytes, filepath: str) -> dict:
    # call Claude, GPT, or any extraction logic
    return {"vendor": "Acme", "total": "1234.56", "date": "2026-01-15"}

Add one JSON label file per document:

{
  "vendor": "Acme Corp",
  "total": "1234.56",
  "date": "2026-01-15"
}

Run the eval:

pip install doceval

doceval run \
  --docs    ./dataset/docs \
  --labels  ./dataset/labels \
  --extractor my_module:extract

Failure Mode Taxonomy

Every mismatch is classified into one of four modes:

Mode Meaning
missed_field Label has a value; extractor returned empty
hallucination Extractor returned a value; label is empty
wrong_format Both non-empty; numeric or date values differ
wrong_value Both non-empty; string values differ

doceval handles numeric normalisation ($1,234.56 = 1234.56 = 1.234,56) and date normalisation (Nov 15 2012 = 2012-11-15) before comparison, so format differences don’t inflate your error count.

Optional Cost Tracking

Return a (dict, cost_usd) tuple from your extractor and doceval tracks cost automatically:

def extract(doc_bytes: bytes, filepath: str) -> tuple[dict, float]:
    response = client.messages.create(...)
    cost = response.usage.input_tokens / 1e6 * 0.80
    return result_dict, cost

Try the Example

The repo includes a working 20-document invoice dataset with labels and a Claude Haiku extractor you can run immediately:

git clone https://github.com/dave8172/doceval
cd doceval
pip install -e ".[examples]"
export ANTHROPIC_API_KEY=sk-ant-...

doceval run \
  --docs    examples/invoices/docs \
  --labels  examples/invoices/labels \
  --extractor examples.invoices.extractor:extract

The same harness on something that isn’t a document

examples/leads/ runs over eight inbound sales emails — a forwarded thread where the lead is the original sender rather than the forwarder, a terse RFQ, a web-form dump, an automated tender notice with no contact details, and one enquiry in Italian.

doceval run \
  --docs    examples/leads/docs \
  --labels  examples/leads/labels \
  --extractor examples.leads.extractor:extract

Three of those emails carry null-heavy labels on purpose. An extractor that invents a quantity or a budget for them scores worse, not better — which is the point of measuring fields rather than output shape.

MCP Server — Use It From an AI Agent

doceval also ships as an MCP server, so an AI coding agent (Claude Code, Claude Desktop, Cursor) can score extraction accuracy directly, mid-conversation, without shelling out to the CLI.

pip install "doceval[mcp]"

Two tools:

Point any MCP client at the doceval-mcp command over stdio:

{
  "mcpServers": {
    "doceval": { "command": "doceval-mcp" }
  }
}

Tech Stack

← Back home