vellum

# Vellum ### Coding you can prove. **An auditable agentic ICD-10 autocoder whose work you can prove, and the moment it cannot.** Five Claude Opus roles read a hospital discharge summary and propose the billable codes. Four model-free, network-free engines then prove or refuse every one of them. The point of difference is that Vellum catches the model inventing its own evidence: when the model quotes a sentence that is not in the record, a substring check with no model and no network finds it and holds the code back.

                       

**[▶ Watch the three-minute demo](https://www.youtube.com/watch?v=FBezih99aBI)**  ·  Built with Claude: Life Sciences (Builder track)

For judges

Contents

• The problem
• What Vellum is
• The trust mechanism
• The cockpit
• How it works
• Our approach
• Experiments and results
• Data
• Getting started
• Command reference
• Project structure
• How Claude was used
• Testing
• Limitations and honest scope
• Technical report
• License and acknowledgments

The problem

Clinical coding turns a free-text discharge summary into the ICD-10-CM diagnosis codes and ICD-10-PCS
procedure codes a hospital bills for the admission. It is a character-level target: there are 98,186 current
diagnosis codes and roughly 79,000 procedure codes, and a single wrong character is a different diagnosis
and a different bill. I50.23 is acute on chronic systolic heart failure; I50.33, one character away, is
the diastolic form. Certified human inpatient coders reach only about 55 to 65 percent full-record accuracy,
so this is not a task a language model trivially clears.

A language model is fluent but not exact, and the failure that matters is quiet. Asked to justify a code, a
fluent model will sometimes invent its citation: it quotes a sentence that reads like the record but
appears nowhere in it, usually by splicing a section header onto a distant line. The code can be real,
current, billable, and plausibly correct, so no code validator objects. But the sentence offered as its
proof was never written. A more capable model makes that fabrication more convincing, not less. A coding
system that cannot tell a grounded citation from a constructed one will eventually bill on a sentence nobody
wrote, and for a research-cohort builder a few false members poison a study. Grounding is a property of the
evidence, not of the answer.

What Vellum is

Vellum is a glass-box agentic coder. The whole workflow happens in front of the user, and every
deterministic check fires where it can be watched.

• Five Claude Opus roles do all of the clinical reasoning: they read the note, resolve each finding to a
  current code through live terminology tools, challenge one another, and sign off a chart.
• Four deterministic engines, each with no model and no network, verify and ground every code. No model
  participates in any decision an engine makes.
• Every code carries a receipt: the sentence the model quoted, and a three-valued verdict on whether
  that sentence exists in the note.
• A purpose-built ICD-10 terminology server (sixteen tools over a self-updating ICD-10-CM/PCS data
  layer) is how every code is resolved and verified, never recalled from memory.
• Seven surfaces tell the story end to end, from the live pipeline to the measured study with its
  limitations as first-class content.

The trust mechanism

The most defensible property is a mechanism, not a score, because it needs no gold standard: a citation
either appears in the note or it does not.

On the shipped cardiogenic-shock note, Claude proposes six valid, current codes. The deterministic gate
auto-accepts four and holds two for review, and one of those two is the principal diagnosis, the code
that drives reimbursement. The model cited "PAST MEDICAL HISTORY: - Hypertension". The note does say
PAST MEDICAL HISTORY:, and it does list - Hypertension several lines further down, after other bullets.
The patient really is hypertensive, so the fact is not wrong. What the model produced is a constructed
quotation: a section header spliced onto a line that never followed it. A substring check with no model,
no network, and no credential finds that the sentence is not in the record and routes the code to review
rather than accepting it on the system's own authority.

The cockpit

One command builds the interface and serves it with the API on one port. The demo replays a recorded run of
the pipeline as a paced event stream, so it is fast, reproducible, and free, and it needs no API key and no
data build. Every recorded run is a run on a synthetic note.

Theater. The pipeline as a live circuit: the five roles and the two gates as stations, each lighting up
as the work reaches it, the note highlighting its harvested spans, and every terminology tool call scrolling
past.

Chart. The auditable chart, with a provenance banner that states the gap between a chart that cites a
sentence for every code and a chart whose every citation was proven. Here it reads: two fabricated citations
caught, do not sign until corrected.

Graph. The coded encounter as a knowledge graph: the principal diagnosis anchored at the center, the
official code-first and use-additional sequencing between codes, and the fabricated codes marked.

Review. A human-in-the-loop worklist over the codes the pipeline did not auto-accept, with accept,
flag, and query, and a signed-receipt export.

Explorer. A study workbench over the terminology tools: validate a code, search descriptions, walk the
hierarchy, read the Alphabetic Index and the instructional notes, and check severity and grouping, all
against the live built code set with no model and no network.

Science. The measured results as a study: the open-source ceiling on real MIMIC-IV data, the controlled
head-to-head against the deployed pipeline on identical synthetic notes, and the limitations as first-class
content.

The Ledger (shown above) is the seventh surface, and a signed Receipt export completes the workflow.

How it works

The pipeline is five model roles and four deterministic engines, in a fixed order, with the deterministic
gates interposed between the model steps.

The five Claude Opus roles. An abstractor reads the note and lists every codeable condition, each
with an assertion (present, absent, historical, family history) and a quoted citation. A completeness
critic looks for conditions the abstractor missed. A terminology coder resolves each concept to a
valid, current code using the live tools. An adversarial verifier attempts to refute each proposed code
against the note. A lead assembles, sequences, and signs off the chart. The completeness critic and the
adversarial verifier are Claude turned against Claude's own first pass.

The four deterministic engines. The terminology layer decides existence, billability, CMS severity,
and more-specific siblings. The rulebook applies Excludes1 conflicts and code-first or use-additional
sequencing. The evidence gate checks, on a bare interpreter, whether each quoted sentence is a verbatim
substring of the note. Triage makes the fail-closed disposition: only a code that clears every hard
check auto-accepts, and a verifier that did not run is treated exactly like one that ran and rejected the
code.

The three-valued verdict. Every code carries verified, not_in_note, or nothing_to_check, plus a per-code
confidence, so an unproven code can never render as proven. A fabricated citation is a grounding failure,
which is not proof the diagnosis is wrong, so flagged codes go to needs-review, not rejected.

Our approach

The model proposes and a deterministic check decides. Claude does the hard part, which is reading unstructured
clinical prose and proposing the codes; ordinary Python, over published tables, decides whether each code may
be trusted. The guarantees the system offers therefore do not depend on trusting the model that produced the
codes.

The multi-agent design is deliberately adversarial. Two of the five roles exist to attack the first pass:
one hunts for conditions that were missed, and one tries to refute the codes that were proposed. That design
is what surfaced the fabricated-citation failure mode in the first place, and the deterministic trust layer
is what defeats it in production.

Experiments and results

The evaluation is split into two stages by a hard data-handling rule, and the split turns that constraint
into a controlled experiment. Credentialed MIMIC-IV data is processed only by self-hosted, open-weight
models on infrastructure under our control, so it never reaches a third-party API. A hosted model (Claude)
only ever sees synthetic, de-identified notes.

Stage 1, the open-source ceiling on real data. A self-hosted Qwen3-8B coder is crossed with two
retrieval augmentations (SapBERT concept-to-code candidates, and guideline-RAG) on 90 stratified MIMIC-IV
discharge notes. Only aggregate metrics leave the machine.

An open coder alone hallucinates most of its codes (73 percent, F1 0.02). SapBERT candidates ground it,
cutting hallucination to 17 percent for the largest single lift, but the best open configuration still
reaches only F1 0.16. The 0.223 recall-ceiling is the structural cause: even a perfect selector over the
retrieved candidates could recover at most about 22 percent of the gold codes. Retrieval-augmented open
models plateau far below usable accuracy on real inpatient coding.

Stage 2, the deployed pipeline against the same open-source arms, identical notes. Four short but
deliberately dense synthetic discharge summaries (15 to 19 codeable conditions each) were hand-authored to
mirror worst-case inpatient complexity, and their gold sets were validated code-by-code against the live
terminology. Every pipeline coded the identical notes.

The strongest open-source configuration reaches 0.57; the deployed pipeline reaches 0.93 on the identical
notes, about 1.6 times the exact F1, and it is the only pipeline that is 100 percent valid and current,
100 percent grounded in verified citations, and provably free of fabricated citations. Because the inputs
are shared, the gap is a controlled comparison rather than an artifact of different data. At four notes it
is directional, not a population estimate. Per-note Opus exact F1 ranged 0.83 to 1.00.

Why the deployed pipeline is lean. It reaches 0.93 with no GPU-backed layers at all. The optional
retrieval augmentations, evaluated as levers on the deployed pipeline, did not clear it by a margin that
survives run-to-run noise, so they ship off by default behind feature flags, each failing soft to the
deterministic keyword path. The shipped combination is Opus, keyword grounding, the Alphabetic Index, the
rulebook, and the two deterministic gates: simpler, cheaper, fully reproducible, and GPU-free.

Data

• Public-domain terminology, pinned by digest. ICD-10-CM (CDC), ICD-10-PCS and the official guidelines
  and CMS severity list (CMS), and the AHRQ CCSR groupings are works of the United States federal government
  in the public domain. Each source is verified against a recorded SHA-256 before it is built, and the
  built stores are reproducible from the pinned manifest. Copyrighted sources (AHA Coding Clinic, WHO
  ICD-10) are excluded from the repository.
• Which model saw which data. Credentialed MIMIC-IV data is processed only by self-hosted open-weight
  models on infrastructure under our control and never leaves it. The hosted model (Claude) codes only
  synthetic, de-identified notes, each headed "SYNTHETIC DISCHARGE SUMMARY. NOT REAL PATIENT DATA."
• Nothing credentialed is published. MIMIC note text, note ids, and per-note traces are git-ignored;
  only PHI-free aggregate metrics and the fully synthetic artifacts appear in the repository. Stage 1 is
  reproducible by any credentialed researcher who runs the shipped harness on their own MIMIC copy.
• UMLS is reached live under the operator's own license through the umls_search tool and is never
  redistributed. Full detail is in DATA_SOURCES.md.

Getting started

The core is standard-library only. The optional layers install as extras. None of the shipped pipeline needs
a GPU.

Prerequisites. Python 3.11 or newer (the package requires 3.11). Ubuntu 24.04 ships 3.12; on Ubuntu
22.04 install python3.11 from the deadsnakes PPA. Ubuntu also needs the matching venv package before
python3 -m venv will work. For the web cockpit, Node 20 or newer and pnpm 10 (corepack enable
provisions the pinned pnpm, or npm install -g pnpm). No C toolchain is required; every Python dependency
installs from a prebuilt wheel.

# Ubuntu only: install the venv package (24.04 -> 3.12; 22.04 -> 3.11 via deadsnakes)
sudo apt install -y python3.12-venv          # or python3.11-venv on 22.04
python3 -m venv .venv && source .venv/bin/activate   # 22.04: python3.11 -m venv .venv | Windows: py -3 -m venv .venv; .venv\Scripts\activate
python -m pip install --upgrade pip
pip install -e .                                     # the standard-library core needs no third-party deps

Prove a citation in ten seconds, with no data and no key. The evidence gate is standard-library only.

from icd10_coder.provenance.evidence import verify_citation

note = "Assessment: acute systolic heart failure, managed with diuresis."
print(verify_citation("acute systolic heart failure", note))   # EvidenceVerdict.VERIFIED
print(verify_citation("acute diastolic heart failure", note))  # EvidenceVerdict.NOT_IN_NOTE

Run the cockpit. The demo replays the recorded run with no API key and no data build.

pip install -e ".[web,agents]"     # core, the web server, and the agent layer
cd web && pnpm install             # or: make web-install
cd .. && make run                  # builds web/dist and serves http://127.0.0.1:8000

Then open http://127.0.0.1:8000 in a browser and press Run the demo: the pipeline replays and the
deterministic gate catches the fabricated citation for the principal diagnosis. No API key, no data build.

On macOS this works as written (install Node and Python with Homebrew). On native Windows there is no
make, so either use WSL2 and follow the Linux commands above, or run the two underlying steps directly:
cd web; pnpm install; pnpm build, then icd10-coder-web from the repository root (make run only chains
those two). Activate the virtualenv with .venv\Scripts\activate and set the key with $env:ANTHROPIC_API_KEY.

Build the live terminology layer (about 122 MB of SQLite, from roughly 40 MB of public-domain
downloads). This makes the Explorer query live and lets the coder resolve and validate real codes.

pip install -e ".[build]"
icd10-coder bootstrap

Code a real note. Build the terminology layer first (above) so the coder resolves and validates against
real, current codes rather than degrading to needs-review. The code command reads a note from a file. It
calls the Anthropic API and therefore costs money, so every paid command prints the model and the roles it
will run and refuses to start without an explicit --yes.

export ANTHROPIC_API_KEY=...                       # never logged, never placed in an exception
printf 'Assessment: acute systolic heart failure with cardiogenic shock.\n' > note.txt
icd10-coder code note.txt --yes --out chart.json
icd10-coder audit chart.json --note note.txt       # re-audit for free, by anyone, no key and no model

For hot-reload development, run the two processes together: icd10-coder-web on :8000 and, in a second
terminal, cd web && pnpm dev on :5173. The terminology tool server for a Claude client is
icd10-coder serve-mcp.

Command reference

Every command that spends money is marked. The others need neither a key nor the network once the stores are
built.

Project structure

icd10-coder/
├── src/
│   ├── icd10_coder/
│   │   ├── agents/         # the five Opus roles, the team orchestrator, the client bridge
│   │   ├── terminology/    # the code stores, the query layer, the rulebook engine
│   │   ├── provenance/     # the evidence gate, triage, and the chart builder
│   │   ├── mcp/            # the sixteen-tool terminology server
│   │   ├── data/           # the source manifest and its SHA-256 pins
│   │   ├── eval/           # the Stage 1 ablation and Stage 2 head-to-head harnesses
│   │   └── observability/  # the typed event stream the cockpit replays
│   └── icd10_coder_web/    # the FastAPI application that serves the cockpit
├── web/                    # the React cockpit (Vite, Tailwind, TypeScript)
├── demo/                   # the synthetic notes, the recorded chart, and the result figures
├── fixtures/               # the recorded pipeline run the demo replays
├── tests/                  # 24 test files
├── docs/                   # the technical report and the images in this README
├── infra/ services/        # optional GPU provisioning and the self-hosted model servers
├── METHODOLOGY.md          # the judge-facing methodology and results
├── DATA_SOURCES.md         # every source, its license, and its digest
└── Makefile

How Claude was used

• Claude Code built the entire system end to end: the five-agent coding team, the sixteen-tool ICD-10
  terminology server, the four deterministic verification engines, the observability layer, and the web
  cockpit.
• Claude Opus 4.8 does all of the clinical reasoning, as a five-role adversarial team in which a
  completeness critic and an adversarial verifier are Claude turned against its own first pass, driving
  tool-use loops over the terminology server.
• Claude Design took the built application, produced a self-contained offline build of the whole
  cockpit, and recorded the demo video from it.

Claude mattered most in two places: the adversarial multi-agent design, and the deterministic trust layer
that surfaced, and then defeated, the model's fabricated-citation failure mode.

Testing

make test        # the offline suite: no key, no network, no built stores required
make test-all    # the full suite

The suite is 214 tests across 24 files. A dedicated test runs the pipeline with and without an observer and
asserts the coded output is identical, so the observability layer that the cockpit depends on cannot change
a result.

Limitations and honest scope

These are properties of the system, and the Science surface states them as loudly as the numbers.

• Auto-accept does not mean correct. It means the code is real and current, its quoted sentence is
  provably in the note, an adversarial reviewer examined it, and it breaks no published rule. Whether it is
  the right code for this patient is a clinical judgment a human makes.
• The accuracy figures are a small, controlled comparison, not a population estimate. Stage 2 is four
  hand-authored synthetic notes; the head-to-head is directional.
• Only one kind of note has been evaluated: adult general-medicine and intensive-care inpatient
  discharge summaries.
• The deployed coder is nondeterministic (it accepts no fixed seed), so per-run scores vary; the
  guarantees come from the deterministic gates, not from a single run.
• This is not a grouper. It returns no diagnosis-related group and no payment amount. The CMS severity
  layer reports an upper-bound complication tier only.
• Three ICD tables are not loaded: the Table of Drugs, the Table of Neoplasms, and the External Cause
  index, so oncology, poisoning, and injury external-cause coding are structurally weak.
• The retained GPU, retrieval, and funnel code contributes to no shipped metric. It is kept behind
  flags, off by default.
• Vellum is a research prototype evaluated on hard cases, not a validated clinical device.

Technical report

The full methodology, the diagrams, and the complete results are in the technical report:

• docs/vellum-technical-report.pdf
• docs/vellum-technical-report.md
• METHODOLOGY.md for the in-repo methodology, and SUBMISSION.md for the
  submission packet.

License and acknowledgments

Released under the MIT License; see LICENSE. The official terminology it downloads is published by
the CDC, CMS, and AHRQ and is in the United States public domain. The clinical reasoning runs on Claude Opus
by Anthropic. The package processes only the note text it is given and never retains it.

Vellum. Coding you can prove. Built with Claude Code.