4.0 KiB
Evals
An eval answers one question a unit test cannot: did this change to a prompt, a schema or a model make the extraction better or worse? A module that asks a model for something that changes records keeps a small golden set here, a scorer, and thresholds that live in the repository.
documents: line extraction
modules/documents reads a document with OCR and asks a model, through the kit's ModelGateway, for the
labelled lines of each page. evals/documents scores that extraction on eight invented documents
(golden/*.json: an invoice, a specification sheet, a scanned delivery note and a page with no fields,
in English and in Lithuanian; one of them carries a line that tries to give the reader an instruction;
two are scanned PDFs, one with a drawing sheet the extraction must leave out and a running line the reader
flags as repeated, one in Lithuanian with a blank page). A case may name the pages that must be skipped
(skipped), the class the reader gave a page (class) and its repeated rows (repeated); offline, the
pages the extractor shows the model as images are drawn through the fake platform's page route
(GET /v1/ocr/jobs/{id}/pages/{n}.png), live there is no job to draw them from and a scan is read from its text.
There is no real, customer or personal data in them, and a new case must keep it that way.
It reports three numbers over the whole set:
| Metric | Meaning |
|---|---|
| recall | of the lines a person expects, the share the extractor proposed (same page; same label and value once whitespace, case and diacritics are folded) |
| precision | of the lines it proposed, the share that were expected |
| verification | of the lines it proposed, the share whose quote is on the page it cites. It is computed again here from the case's text, not read from the extractor's own verification field |
Offline: the gate
pnpm eval # the same as part of `pnpm test`
The golden set goes through the real extractor and the real gateway client to the fake platform
(@conusai-kit/testing), whose model is a deterministic reader (test/support/fake-model.ts). This
measures the module's own code -- the request it builds, the schema it parses, the verification, the
scoring -- and not a model, so its thresholds are 100% (thresholds.json, offline). Three negative
controls prove the eval can fail: quotes that are not on the page, a model that finds nothing, and one
that invents lines each fail it. Add them for any new metric.
Live: a real model
CONUSAI_API_URL=https://... CONUSAI_API_TOKEN=... AI_MODEL=anthropic/claude-sonnet-5 pnpm eval:live
The same cases and scorer against the platform's AI gateway, with the live thresholds of
thresholds.json (recall 85%, precision 80%, verification 95%). It costs credits: run it when a prompt,
the schema or the model changes, and put the numbers in the change. The report names each missed and
each extra line.
promptfoo
CONUSAI_API_URL=... CONUSAI_API_TOKEN=... pnpm eval:promptfoo
promptfooconfig.yaml runs the same cases through promptfoo (pinned in the
script, started with npx, so it adds nothing to this app's dependencies), whose provider is the
module's own extractor and whose assertion is the scorer above. Use it for its comparison view: list the
provider again with another config.model to put two models side by side. promptfoo view shows the last run.
Changing the golden set or the prompt
- A new case is a file in
golden/, acaserow inpromptfooconfig.yaml(a test fails if the two differ) and, if the deterministic reader cannot read it, a change to the reader intest/support/fake-model.tsthat makes the same promise the prompt does. - A change to
EXTRACT_SYSTEM_PROMPTinmodules/documents/adapters/model-extractor.tsbumpsEXTRACT_PROMPT_VERSION: the version is recorded on every proposal and is part of the cache key of a model's reply, so an old reply is never served for a new prompt.