# Tests

Run everything from the repo root:

```
uv run --with pytest --with playwright pytest tests
```

- `test_*.py`: assert-based self-checks for template logic (lint of unfilled Python
  templates, the custom palette, presets in step with the template).
- `page_check/`: **Seam 1.** Loads a dashboard page in headless Chromium with no
  backend, pushes contract envelopes through `window.__pyMsg`, and asserts on what
  is visible. Every case runs in light, dark and narrow (800px) mode.
  - By default it checks the no-AI template, filled with test values. To check a
    generated page instead, set `PAGE_CHECK_HTML=path/to/dashboard.html`.
  - The page sees a fake Qt bridge. Every action the page sends is recorded in
    `window.__calls` as `[slot, ...args]`.
  - `PAGE_CHECK_WIRED=1`: the page is wired to a bot (sample data off). Sample-data
    cases skip, and a case checks that the page's actions go to the bot. With no
    `PAGE_CHECK_HTML`, the template is checked with `SAMPLE_DATA` switched off.
  - `PAGE_CHECK_REPLAY=run.jsonl`: replay a real bot run (the JSON lines that
    `python dashboard/bot_runner.py` prints) instead of the built-in run.
  - To add a case, write a `test_*.py` file in `page_check/` that uses the `page`
    and `push(event, data)` fixtures from `conftest.py`.
  - Optional-section cases (`test_optional_sections.py`) skip on a generated page
    built without that section. The template must carry all of them. A page with
    the login section is signed in (`login_result ok`) before each case.
- `test_sections.py`: the optional-section strip tool keeps only the chosen
  sections, and a page with none of them still passes the page check.
- `page_check_ai/`: **Seam 1, AI web page.** Builds the React starter with npm
  (filled with test values, in a temp folder that keeps `node_modules`), serves it with
  its own `server_stub.py`, and checks bands, band filters, drafts, the approve gate
  "page X of Y" and the cost page (spend tiles, coverage pill, month breakdown,
  by-hand assumptions), plus every no-AI case. Tests set run and review state on the stub
  directly (the `stub` fixture). To check a generated dashboard folder instead, set
  `PAGE_CHECK_AI_DIR=path/to/dashboard`. Needs npm on PATH, else it skips.
- `page_check_ai_desktop/`: **Seam 1, AI desktop page.** The no-AI `dashboard.html` with
  `ai_sections.html` pasted in, loaded from `file://` with a fake Qt bridge whose
  `ai_request` slot goes to `server_stub.dispatch()` in this process (what `ai_bridge.py`
  does). Runs the AI web page's review and cost cases and the no-AI cases unchanged, plus
  a tampered page calling the slot directly (still refused). To check a generated folder,
  set `PAGE_CHECK_AI_DESKTOP_DIR=path/to/dashboard`; `PAGE_CHECK_WIRED=1` as above.
  `test_ai_fetch.py` runs the review cases on the same page with no Qt bridge (a plain-HTML
  web project): the AI sections fetch `/api/...`, routed to `server_stub.dispatch()`.
- `page_check_common.py`: what the three page checks share (modes, test fill, fake Qt
  bridge, stub loader, page opener, the list of no-AI cases the AI pages rerun).
- `test_pricing.py`: the AI cost function (normal, long-context, thinking tokens,
  unknown model).
- `test_reference_scan.py`: no patient data, internal IPs, credentials or keys in any
  bundled file, and every Superbill extract carries an `EXAMPLE ONLY` banner.
- `evals/`: **Seam 2.** Runs the skills on a fixture project with `claude -p`,
  then grades the reply and runs the page check on the page it built:

  ```
  uv run --with pytest --with playwright python tests/evals/run_eval.py a-empty
  ```

  Results go to `evals/results/<fixture>.json`. To add a fixture, add an entry to
  `FIXTURES` in `run_eval.py`. An entry with a `"fixture"` key reuses another
  fixture folder with different answers: `c-keep` and `c-restyle` both run on
  `c-web-existing-frontend`, and also check what changed in the files it already had.
  An `"ai": True` entry (`d-ai-extraction`) expects the AI skill's React starter and
  runs `page_check_ai/` on the folder it built. An `"ai_desktop": True` entry
  (`e-pyqt-ai`) expects the desktop AI page and runs `page_check/` (with the bot replay)
  and `page_check_ai_desktop/` on it.
  For a bot fixture (`b-pyqt-bot`), the eval also dry-runs the generated
  `bot_runner.py`, saves its events to `evals/results/<fixture>.run.jsonl`, and
  replays them in the page check.
- `fixtures/<letter>-<name>/`: small test projects for the evals.
- `page_check_run_tab/` + `test_run_tab.py`: **Run tab skill.** The page filled with a test
  config and driven through `window.__pyMsg`: Trigger states, sheet preview with masked
  passwords, checkpoints driving steps/boxes/progress, green/amber/red endings, the no-input
  mode, the dark toggle and no horizontal scroll, each in light, dark and narrow mode. The
  Python side covers line parsing, endings, the sample workbook and `palette.py --write`.
  The end-to-end fake-bot check lives in the skill itself: `python skills/rpa-run-tab/templates/run_tab.py --selfcheck`.