# Evals An eval answers one question a unit test cannot: *did this change to a prompt, a schema or a model make the extraction better or worse?* A module that asks a model for something that changes records keeps a small golden set here, a scorer, and thresholds that live in the repository. ## documents: line extraction `modules/documents` reads a document with OCR and asks a model, through the kit's `ModelGateway`, for the labelled lines of each page. `evals/documents` scores that extraction on eight invented documents (`golden/*.json`: an invoice, a specification sheet, a scanned delivery note and a page with no fields, in English and in Lithuanian; one of them carries a line that tries to give the reader an instruction; two are scanned PDFs, one with a drawing sheet the extraction must leave out and a running line the reader flags as repeated, one in Lithuanian with a blank page). A case may name the pages that must be skipped (`skipped`), the class the reader gave a page (`class`) and its repeated rows (`repeated`); offline, the pages the extractor shows the model as images are drawn through the fake platform's page route (`GET /v1/ocr/jobs/{id}/pages/{n}.png`), live there is no job to draw them from and a scan is read from its text. There is no real, customer or personal data in them, and a new case must keep it that way. It reports three numbers over the whole set: | Metric | Meaning | |---|---| | recall | of the lines a person expects, the share the extractor proposed (same page; same label and value once whitespace, case and diacritics are folded) | | precision | of the lines it proposed, the share that were expected | | verification | of the lines it proposed, the share whose quote is on the page it cites. It is computed again here from the case's text, not read from the extractor's own `verification` field | ### Offline: the gate ```bash pnpm eval # the same as part of `pnpm test` ``` The golden set goes through the real extractor and the real gateway client to the fake platform (`@conusai-kit/testing`), whose model is a deterministic reader (`test/support/fake-model.ts`). This measures the module's own code -- the request it builds, the schema it parses, the verification, the scoring -- and not a model, so its thresholds are 100% (`thresholds.json`, `offline`). Three negative controls prove the eval can fail: quotes that are not on the page, a model that finds nothing, and one that invents lines each fail it. Add them for any new metric. ### Live: a real model ```bash CONUSAI_API_URL=https://... CONUSAI_API_TOKEN=... AI_MODEL=anthropic/claude-sonnet-5 pnpm eval:live ``` The same cases and scorer against the platform's AI gateway, with the `live` thresholds of `thresholds.json` (recall 85%, precision 80%, verification 95%). It costs credits: run it when a prompt, the schema or the model changes, and put the numbers in the change. The report names each missed and each extra line. ### promptfoo ```bash CONUSAI_API_URL=... CONUSAI_API_TOKEN=... pnpm eval:promptfoo ``` `promptfooconfig.yaml` runs the same cases through [promptfoo](https://www.promptfoo.dev) (pinned in the script, started with `npx`, so it adds nothing to this app's dependencies), whose provider is the module's own extractor and whose assertion is the scorer above. Use it for its comparison view: list the provider again with another `config.model` to put two models side by side. `promptfoo view` shows the last run. ### Changing the golden set or the prompt - A new case is a file in `golden/`, a `case` row in `promptfooconfig.yaml` (a test fails if the two differ) and, if the deterministic reader cannot read it, a change to the reader in `test/support/fake-model.ts` that makes the same promise the prompt does. - A change to `EXTRACT_SYSTEM_PROMPT` in `modules/documents/adapters/model-extractor.ts` bumps `EXTRACT_PROMPT_VERSION`: the version is recorded on every proposal and is part of the cache key of a model's reply, so an old reply is never served for a new prompt.