Halyk AI Challenge — covenant compliance

Reads a corpus of unlabelled financial PDFs plus a transaction ledger, and decides for
every covenant of every borrower whether it is met, what the metric's actual value is,
and which single transaction caused any breach.

Score against the provided answer key: 36.000 / 36 (100%), with all 9 evidence
transactions identified correctly. 276 tests.

Approach

Deterministic core, LLM at the edges.

Every number is computed in Python with Decimal. The LLM's only job is turning covenant
prose into a validated structure — it never performs arithmetic and never decides a
verdict. The competition scores actual on a scale that reaches zero at 5% relative
error, so the arithmetic has to be exact; the prose varies too much to parse with rules
alone. Each tool does the half it is good at.

Documents are routed by exact identifiers and authority rules, not similarity search.
Of 200 documents about 55 carry authoritative facts. The hard problem is not finding
candidates but deciding which of two documents is in force — a retriever cannot tell a
current credit agreement from the superseded revision beside it, because they are nearly
identical and differ only in the threshold.

documents/  ──► extract (pdftotext, per-page OCR fallback)
                  │
                  ├─► route      200 docs → the ~55 that are authoritative
                  │              (drops stale revisions, draft memos, 145 noise files)
                  │
                  └─► facts      covenant clauses, related parties + per-borrower
                                 thresholds, accepted reclassifications, ledger patches,
                                 period exclusions, FX rates, document scalars
ledger.csv  ──► categorise ──►   context ──► DSL evaluate ──► status / actual
                                                   │
                                                   └─► counterfactual ──► evidence
                                                                              │
                                            submission.json  +  audit.json ◄──┘

A covenant's metric is data, not code: a small expression language (Ratio, Sum,
Add, DocValue, Conditional, Period, …) that the evaluator executes with exact
decimals. That is what lets the same pipeline handle twelve borrowers whose clauses share
no common shape — and what lets an LLM emit a covenant it has never seen.

Running it

python -m venv .venv
.venv/Scripts/python.exe -m pip install -r requirements.txt
.venv/Scripts/python.exe scripts/bootstrap.py     # fetches tesseract language data
.venv/Scripts/python.exe main.py                  # writes out/submission.json + out/audit.json
.venv/Scripts/python.exe -m pytest -q             # 277 tests

pdftotext (poppler) and tesseract must be on PATH. On Windows run from Git Bash —
poppler ships in /mingw64/bin there but is not on PATH in PowerShell.

A ready submission.json is committed at out/submission.json;
main.py reproduces it byte for byte from a cold cache.

Two ways to get the covenant specifications

Both score 36/36. The LLM path is provider-agnostic — it picks OpenAI or Anthropic
from whichever key is in the environment — and emits the expression tree through
schema-constrained tool use, never a number. Copy .env.example to .env and fill in a
key to use it. Full measurement, per-clause agreement and the failure analysis are in
docs/llm-extraction-notes.md.

Layout

What made this hard

The corpus is adversarial by design. Twenty-one traps were found; the ones that cost the
most to find, all documented with their resolution in
docs/covenant-catalogue.md:

• The OCR trigger has to be per page. Three PDFs are mostly born-digital with one
  image page, and that page is where the table lives. A document-level check catches the
  one fully-scanned file and silently loses the other three.
• One audit note states no conclusion and defers to a separate report by number. That
  referenced report is the only place its borrower's reclassification is recorded.
• FX is not tabulated. The auditor's note says explicitly that no rate table is kept,
  then discloses one invoice-and-payment pair. The rate is that quotient.
• A capital-expenditure figure is never written down. It has to be solved out of a
  property, plant and equipment roll-forward whose additions line was removed.
• Related-party sums count outflows only. One borrower's related party is also its
  customer; summing both directions overstates by a factor of thirty. Eleven of twelve
  borrowers reconcile without the sign filter, so the bug hides behind the statistics.
• Two ledger rows have a blank amount, and those borrowers' covenants say in as many
  words that the ledger is incomplete.

One trap was not in the data at all. For most of the project P4 6.3 computed BREACH where
the key said COMPLIANT, and no rule over (ratio, threshold) could produce both that
verdict and the other 35 cells'. Proving no such rule existed is what made the question
worth putting to the organisers — who confirmed the training PDF renders that clause's
threshold as 0.04x when the real figure is 0.045x. The pipeline was reading the document
correctly; the document was wrong. covenant/specs_data.py carries 0.045 with a comment
marking it as the one value that deliberately disagrees with its source, on the issuer's
instruction.

Documentation

• docs/covenant-catalogue.md — all 36 clauses, every trap
  in the corpus, and how each was resolved
• docs/cell-recipes.md — how each cell is computed and the
  preprocessing order that matters
• docs/llm-extraction-notes.md — how the LLM path was
  built and measured, including what it still gets wrong
• out/audit.json — per cell: the clause text, the expression, every contributing
  transaction with its amount and category, the source document for every fact, and the
  counterfactual candidates considered