0 Fork 0

Files

T

VasanthandClaude Opus 5.5 8951ac05a5 AI skill: SKILL.md, AI contract, scrubbed references, eval fixture (d)

- rpa-dashboard-ai/SKILL.md: trigger description and the shared question flow;
  points at the no-AI frontend.md, themes and palette instead of copying them.
- templates/contract.md: runs, review rows, endpoints and the server approve gate.
- Superbill extracts: patient names and the client name removed, every file
  carries an EXAMPLE ONLY banner. tests/test_reference_scan.py checks both skills
  for patient data, internal IPs, credentials and keys.
- tests/fixtures/d-ai-extraction and its eval entry, which runs page_check_ai
  on the built folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-16 15:13:55 +05:30

2.9 KiB

Raw Blame History

Tests

Run everything from the repo root:

uv run --with pytest --with playwright pytest tests

test_*.py: assert-based self-checks for template logic (lint of unfilled Python templates, the custom palette, presets in step with the template).
page_check/: Seam 1. Loads a dashboard page in headless Chromium with no backend, pushes contract envelopes through window.__pyMsg, and asserts on what is visible. Every case runs in light, dark and narrow (800px) mode.
- By default it checks the no-AI template, filled with test values. To check a generated page instead, set PAGE_CHECK_HTML=path/to/dashboard.html.
- The page sees a fake Qt bridge. Every action the page sends is recorded in window.__calls as [slot, ...args].
- PAGE_CHECK_WIRED=1: the page is wired to a bot (sample data off). Sample-data cases skip, and a case checks that the page's actions go to the bot. With no PAGE_CHECK_HTML, the template is checked with SAMPLE_DATA switched off.
- PAGE_CHECK_REPLAY=run.jsonl: replay a real bot run (the JSON lines that python dashboard/bot_runner.py prints) instead of the built-in run.
- To add a case, write a test_*.py file in page_check/ that uses the page and push(event, data) fixtures from conftest.py.
page_check_ai/: Seam 1, AI web page. Builds the React starter with npm (filled with test values, in a temp folder that keeps node_modules), serves it with its own server_stub.py, and checks bands, band filters, drafts, the approve gate and "page X of Y", plus every no-AI case. Tests set run and review state on the stub directly (the stub fixture). To check a generated dashboard folder instead, set PAGE_CHECK_AI_DIR=path/to/dashboard. Needs npm on PATH, else it skips.
test_reference_scan.py: no patient data, internal IPs, credentials or keys in any bundled file, and every Superbill extract carries an EXAMPLE ONLY banner.
evals/: Seam 2. Runs the skills on a fixture project with claude -p, then grades the reply and runs the page check on the page it built:
```
uv run --with pytest --with playwright python tests/evals/run_eval.py a-empty
```
Results go to evals/results/<fixture>.json. To add a fixture, add an entry to FIXTURES in run_eval.py. An entry with a "fixture" key reuses another fixture folder with different answers: c-keep and c-restyle both run on c-web-existing-frontend, and also check what changed in the files it already had. An "ai": True entry (d-ai-extraction) expects the AI skill's React starter and runs page_check_ai/ on the folder it built. For a bot fixture (b-pyqt-bot), the eval also dry-runs the generated bot_runner.py, saves its events to evals/results/<fixture>.run.jsonl, and replays them in the page check.
fixtures/<letter>-<name>/: small test projects for the evals.

2.9 KiB Raw Blame History

Tests

2.9 KiB

Raw Blame History