VasanthandClaude Opus 5.5
9dcdd8f572
- templates/web/pricing.py: pricing table with source and as-of date; cost_for_call
(input rate + (output + thinking) at output rate, long-context above threshold;
unknown model raises naming the known models).
- Cost tab: coverage pill, total/this-month/tokens tiles, month-by-month table,
per-run tokens and cost ("not priced", never $0), by-hand panel with editable
assumptions labelled as such.
- Contract: RoiRun and GET /roi; optional GET /models and POST /runs/{id}/retry.
- SKILL.md: cost page wiring, model picker and retry-on-other-model as optional sections.
- Tests: test_pricing.py self-check, page_check_ai/test_ai_cost.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
53 lines
3.0 KiB
Markdown
53 lines
3.0 KiB
Markdown
# Tests
|
|
|
|
Run everything from the repo root:
|
|
|
|
```
|
|
uv run --with pytest --with playwright pytest tests
|
|
```
|
|
|
|
- `test_*.py`: assert-based self-checks for template logic (lint of unfilled Python
|
|
templates, the custom palette, presets in step with the template).
|
|
- `page_check/`: **Seam 1.** Loads a dashboard page in headless Chromium with no
|
|
backend, pushes contract envelopes through `window.__pyMsg`, and asserts on what
|
|
is visible. Every case runs in light, dark and narrow (800px) mode.
|
|
- By default it checks the no-AI template, filled with test values. To check a
|
|
generated page instead, set `PAGE_CHECK_HTML=path/to/dashboard.html`.
|
|
- The page sees a fake Qt bridge. Every action the page sends is recorded in
|
|
`window.__calls` as `[slot, ...args]`.
|
|
- `PAGE_CHECK_WIRED=1`: the page is wired to a bot (sample data off). Sample-data
|
|
cases skip, and a case checks that the page's actions go to the bot. With no
|
|
`PAGE_CHECK_HTML`, the template is checked with `SAMPLE_DATA` switched off.
|
|
- `PAGE_CHECK_REPLAY=run.jsonl`: replay a real bot run (the JSON lines that
|
|
`python dashboard/bot_runner.py` prints) instead of the built-in run.
|
|
- To add a case, write a `test_*.py` file in `page_check/` that uses the `page`
|
|
and `push(event, data)` fixtures from `conftest.py`.
|
|
- `page_check_ai/`: **Seam 1, AI web page.** Builds the React starter with npm
|
|
(filled with test values, in a temp folder that keeps `node_modules`), serves it with
|
|
its own `server_stub.py`, and checks bands, band filters, drafts, the approve gate
|
|
"page X of Y" and the cost page (spend tiles, coverage pill, month breakdown,
|
|
by-hand assumptions), plus every no-AI case. Tests set run and review state on the stub
|
|
directly (the `stub` fixture). To check a generated dashboard folder instead, set
|
|
`PAGE_CHECK_AI_DIR=path/to/dashboard`. Needs npm on PATH, else it skips.
|
|
- `test_pricing.py`: the AI cost function (normal, long-context, thinking tokens,
|
|
unknown model).
|
|
- `test_reference_scan.py`: no patient data, internal IPs, credentials or keys in any
|
|
bundled file, and every Superbill extract carries an `EXAMPLE ONLY` banner.
|
|
- `evals/`: **Seam 2.** Runs the skills on a fixture project with `claude -p`,
|
|
then grades the reply and runs the page check on the page it built:
|
|
|
|
```
|
|
uv run --with pytest --with playwright python tests/evals/run_eval.py a-empty
|
|
```
|
|
|
|
Results go to `evals/results/<fixture>.json`. To add a fixture, add an entry to
|
|
`FIXTURES` in `run_eval.py`. An entry with a `"fixture"` key reuses another
|
|
fixture folder with different answers: `c-keep` and `c-restyle` both run on
|
|
`c-web-existing-frontend`, and also check what changed in the files it already had.
|
|
An `"ai": True` entry (`d-ai-extraction`) expects the AI skill's React starter and
|
|
runs `page_check_ai/` on the folder it built.
|
|
For a bot fixture (`b-pyqt-bot`), the eval also dry-runs the generated
|
|
`bot_runner.py`, saves its events to `evals/results/<fixture>.run.jsonl`, and
|
|
replays them in the page check.
|
|
- `fixtures/<letter>-<name>/`: small test projects for the evals.
|