72 lines
4.0 KiB
Markdown
72 lines
4.0 KiB
Markdown
<!-- SPDX-License-Identifier: LicenseRef-ConusAI-Template-1.0 -->
|
|
# Evals
|
|
|
|
An eval answers one question a unit test cannot: *did this change to a prompt, a schema or a model make
|
|
the extraction better or worse?* A module that asks a model for something that changes records keeps a
|
|
small golden set here, a scorer, and thresholds that live in the repository.
|
|
|
|
## documents: line extraction
|
|
|
|
`modules/documents` reads a document with OCR and asks a model, through the kit's `ModelGateway`, for the
|
|
labelled lines of each page. `evals/documents` scores that extraction on eight invented documents
|
|
(`golden/*.json`: an invoice, a specification sheet, a scanned delivery note and a page with no fields,
|
|
in English and in Lithuanian; one of them carries a line that tries to give the reader an instruction;
|
|
two are scanned PDFs, one with a drawing sheet the extraction must leave out and a running line the reader
|
|
flags as repeated, one in Lithuanian with a blank page). A case may name the pages that must be skipped
|
|
(`skipped`), the class the reader gave a page (`class`) and its repeated rows (`repeated`); offline, the
|
|
pages the extractor shows the model as images are drawn through the fake platform's page route
|
|
(`GET /v1/ocr/jobs/{id}/pages/{n}.png`), live there is no job to draw them from and a scan is read from its text.
|
|
There is no real, customer or personal data in them, and a new case must keep it that way.
|
|
|
|
It reports three numbers over the whole set:
|
|
|
|
| Metric | Meaning |
|
|
|---|---|
|
|
| recall | of the lines a person expects, the share the extractor proposed (same page; same label and value once whitespace, case and diacritics are folded) |
|
|
| precision | of the lines it proposed, the share that were expected |
|
|
| verification | of the lines it proposed, the share whose quote is on the page it cites. It is computed again here from the case's text, not read from the extractor's own `verification` field |
|
|
|
|
### Offline: the gate
|
|
|
|
```bash
|
|
pnpm eval # the same as part of `pnpm test`
|
|
```
|
|
|
|
The golden set goes through the real extractor and the real gateway client to the fake platform
|
|
(`@conusai-kit/testing`), whose model is a deterministic reader (`test/support/fake-model.ts`). This
|
|
measures the module's own code -- the request it builds, the schema it parses, the verification, the
|
|
scoring -- and not a model, so its thresholds are 100% (`thresholds.json`, `offline`). Three negative
|
|
controls prove the eval can fail: quotes that are not on the page, a model that finds nothing, and one
|
|
that invents lines each fail it. Add them for any new metric.
|
|
|
|
### Live: a real model
|
|
|
|
```bash
|
|
CONUSAI_API_URL=https://... CONUSAI_API_TOKEN=... AI_MODEL=anthropic/claude-sonnet-5 pnpm eval:live
|
|
```
|
|
|
|
The same cases and scorer against the platform's AI gateway, with the `live` thresholds of
|
|
`thresholds.json` (recall 85%, precision 80%, verification 95%). It costs credits: run it when a prompt,
|
|
the schema or the model changes, and put the numbers in the change. The report names each missed and
|
|
each extra line.
|
|
|
|
### promptfoo
|
|
|
|
```bash
|
|
CONUSAI_API_URL=... CONUSAI_API_TOKEN=... pnpm eval:promptfoo
|
|
```
|
|
|
|
`promptfooconfig.yaml` runs the same cases through [promptfoo](https://www.promptfoo.dev) (pinned in the
|
|
script, started with `npx`, so it adds nothing to this app's dependencies), whose provider is the
|
|
module's own extractor and whose assertion is the scorer above. Use it for its comparison view: list the
|
|
provider again with another `config.model` to put two models side by side. `promptfoo view` shows the last run.
|
|
|
|
### Changing the golden set or the prompt
|
|
|
|
- A new case is a file in `golden/`, a `case` row in `promptfooconfig.yaml` (a test fails if the two
|
|
differ) and, if the deterministic reader cannot read it, a change to the reader in
|
|
`test/support/fake-model.ts` that makes the same promise the prompt does.
|
|
- A change to `EXTRACT_SYSTEM_PROMPT` in `modules/documents/adapters/model-extractor.ts` bumps
|
|
`EXTRACT_PROMPT_VERSION`: the version is recorded on every proposal and is part of the cache key of a
|
|
model's reply, so an old reply is never served for a new prompt.
|