ConusAI vendor v0.1.0-dev.20261027+9b1131fb

2026-10-27 18:05:17 +00:00
commit 3102273ec7
538 changed files with 80203 additions and 0 deletions
@@ -0,0 +1,71 @@
+<!-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0 -->
+# Evals
+
+An eval answers one question a unit test cannot: *did this change to a prompt, a schema or a model make
+the extraction better or worse?* A module that asks a model for something that changes records keeps a
+small golden set here, a scorer, and thresholds that live in the repository.
+
+## documents: line extraction
+
+`modules/documents` reads a document with OCR and asks a model, through the kit's `ModelGateway`, for the
+labelled lines of each page. `evals/documents` scores that extraction on eight invented documents
+(`golden/*.json`: an invoice, a specification sheet, a scanned delivery note and a page with no fields,
+in English and in Lithuanian; one of them carries a line that tries to give the reader an instruction;
+two are scanned PDFs, one with a drawing sheet the extraction must leave out and a running line the reader
+flags as repeated, one in Lithuanian with a blank page). A case may name the pages that must be skipped
+(`skipped`), the class the reader gave a page (`class`) and its repeated rows (`repeated`); offline, the
+pages the extractor shows the model as images are drawn through the fake platform's page route
+(`GET /v1/ocr/jobs/{id}/pages/{n}.png`), live there is no job to draw them from and a scan is read from its text.
+There is no real, customer or personal data in them, and a new case must keep it that way.
+
+It reports three numbers over the whole set:
+
+| Metric | Meaning |
+|---|---|
+| recall | of the lines a person expects, the share the extractor proposed (same page; same label and value once whitespace, case and diacritics are folded) |
+| precision | of the lines it proposed, the share that were expected |
+| verification | of the lines it proposed, the share whose quote is on the page it cites. It is computed again here from the case's text, not read from the extractor's own `verification` field |
+
+### Offline: the gate
+
+```bash
+pnpm eval          # the same as part of `pnpm test`
+```
+
+The golden set goes through the real extractor and the real gateway client to the fake platform
+(`@conusai-kit/testing`), whose model is a deterministic reader (`test/support/fake-model.ts`). This
+measures the module's own code -- the request it builds, the schema it parses, the verification, the
+scoring -- and not a model, so its thresholds are 100% (`thresholds.json`, `offline`). Three negative
+controls prove the eval can fail: quotes that are not on the page, a model that finds nothing, and one
+that invents lines each fail it. Add them for any new metric.
+
+### Live: a real model
+
+```bash
+CONUSAI_API_URL=https://... CONUSAI_API_TOKEN=... AI_MODEL=anthropic/claude-sonnet-5 pnpm eval:live
+```
+
+The same cases and scorer against the platform's AI gateway, with the `live` thresholds of
+`thresholds.json` (recall 85%, precision 80%, verification 95%). It costs credits: run it when a prompt,
+the schema or the model changes, and put the numbers in the change. The report names each missed and
+each extra line.
+
+### promptfoo
+
+```bash
+CONUSAI_API_URL=... CONUSAI_API_TOKEN=... pnpm eval:promptfoo
+```
+
+`promptfooconfig.yaml` runs the same cases through [promptfoo](https://www.promptfoo.dev) (pinned in the
+script, started with `npx`, so it adds nothing to this app's dependencies), whose provider is the
+module's own extractor and whose assertion is the scorer above. Use it for its comparison view: list the
+provider again with another `config.model` to put two models side by side. `promptfoo view` shows the last run.
+
+### Changing the golden set or the prompt
+
+- A new case is a file in `golden/`, a `case` row in `promptfooconfig.yaml` (a test fails if the two
+  differ) and, if the deterministic reader cannot read it, a change to the reader in
+  `test/support/fake-model.ts` that makes the same promise the prompt does.
+- A change to `EXTRACT_SYSTEM_PROMPT` in `modules/documents/adapters/model-extractor.ts` bumps
+  `EXTRACT_PROMPT_VERSION`: the version is recorded on every proposal and is part of the cache key of a
+  model's reply, so an old reply is never served for a new prompt.