Rpa_Skills/tests/README.md

Tests

Run everything from the repo root:

uv run --with pytest --with playwright pytest tests

• test_*.py: assert-based self-checks for template logic (lint of unfilled Python templates, the custom palette, presets in step with the template).

• page_check/: Seam 1. Loads a dashboard page in headless Chromium with no backend, pushes contract envelopes through window.__pyMsg, and asserts on what is visible. Every case runs in light, dark and narrow (800px) mode.

  • By default it checks the no-AI template, filled with test values. To check a generated page instead, set PAGE_CHECK_HTML=path/to/dashboard.html.
  • The page sees a fake Qt bridge. Every action the page sends is recorded in window.__calls as [slot, ...args].
  • PAGE_CHECK_WIRED=1: the page is wired to a bot (sample data off). Sample-data cases skip, and a case checks that the page's actions go to the bot. With no PAGE_CHECK_HTML, the template is checked with SAMPLE_DATA switched off.
  • PAGE_CHECK_REPLAY=run.jsonl: replay a real bot run (the JSON lines that python dashboard/bot_runner.py prints) instead of the built-in run.
  • To add a case, write a test_*.py file in page_check/ that uses the page and push(event, data) fixtures from conftest.py.
  • Optional-section cases (test_optional_sections.py) skip on a generated page built without that section. The template must carry all of them. A page with the login section is signed in (login_result ok) before each case.

• test_sections.py: the optional-section strip tool keeps only the chosen sections, and a page with none of them still passes the page check.

• page_check_ai/: Seam 1, AI web page. Builds the React starter with npm (filled with test values, in a temp folder that keeps node_modules), serves it with its own server_stub.py, and checks bands, band filters, drafts, the approve gate "page X of Y" and the cost page (spend tiles, coverage pill, month breakdown, by-hand assumptions), plus every no-AI case. Tests set run and review state on the stub directly (the stub fixture). To check a generated dashboard folder instead, set PAGE_CHECK_AI_DIR=path/to/dashboard. Needs npm on PATH, else it skips.

• page_check_ai_desktop/: Seam 1, AI desktop page. The no-AI dashboard.html with ai_sections.html pasted in, loaded from file:// with a fake Qt bridge whose ai_request slot goes to server_stub.dispatch() in this process (what ai_bridge.py does). Runs the AI web page's review and cost cases and the no-AI cases unchanged, plus a tampered page calling the slot directly (still refused). To check a generated folder, set PAGE_CHECK_AI_DESKTOP_DIR=path/to/dashboard; PAGE_CHECK_WIRED=1 as above.

• test_pricing.py: the AI cost function (normal, long-context, thinking tokens, unknown model).

• test_reference_scan.py: no patient data, internal IPs, credentials or keys in any bundled file, and every Superbill extract carries an EXAMPLE ONLY banner.

• evals/: Seam 2. Runs the skills on a fixture project with claude -p, then grades the reply and runs the page check on the page it built:

  uv run --with pytest --with playwright python tests/evals/run_eval.py a-empty
  

  Results go to evals/results/<fixture>.json. To add a fixture, add an entry to FIXTURES in run_eval.py. An entry with a "fixture" key reuses another fixture folder with different answers: c-keep and c-restyle both run on c-web-existing-frontend, and also check what changed in the files it already had. An "ai": True entry (d-ai-extraction) expects the AI skill's React starter and runs page_check_ai/ on the folder it built. An "ai_desktop": True entry (e-pyqt-ai) expects the desktop AI page and runs page_check/ (with the bot replay) and page_check_ai_desktop/ on it. For a bot fixture (b-pyqt-bot), the eval also dry-runs the generated bot_runner.py, saves its events to evals/results/<fixture>.run.jsonl, and replays them in the page check.

• fixtures/<letter>-<name>/: small test projects for the evals.