Rpa_Skills/tests/README.md

Tests

Run everything from the repo root:

uv run --with pytest --with playwright pytest tests

• test_*.py: assert-based self-checks for template logic (lint of unfilled Python templates, the custom palette, presets in step with the template).

• page_check/: Seam 1. Loads a dashboard page in headless Chromium with no backend, pushes contract envelopes through window.__pyMsg, and asserts on what is visible. Every case runs in light, dark and narrow (800px) mode.

  • By default it checks the no-AI template, filled with test values. To check a generated page instead, set PAGE_CHECK_HTML=path/to/dashboard.html.
  • The page sees a fake Qt bridge. Every action the page sends is recorded in window.__calls as [slot, ...args].
  • PAGE_CHECK_WIRED=1: the page is wired to a bot (sample data off). Sample-data cases skip, and a case checks that the page's actions go to the bot. With no PAGE_CHECK_HTML, the template is checked with SAMPLE_DATA switched off.
  • PAGE_CHECK_REPLAY=run.jsonl: replay a real bot run (the JSON lines that python dashboard/bot_runner.py prints) instead of the built-in run.
  • To add a case, write a test_*.py file in page_check/ that uses the page and push(event, data) fixtures from conftest.py.

• evals/: Seam 2. Runs the skills on a fixture project with claude -p, then grades the reply and runs the page check on the page it built:

  uv run --with pytest --with playwright python tests/evals/run_eval.py a-empty
  

  Results go to evals/results/<fixture>.json. To add a fixture, add an entry to FIXTURES in run_eval.py. An entry with a "fixture" key reuses another fixture folder with different answers: c-keep and c-restyle both run on c-web-existing-frontend, and also check what changed in the files it already had. For a bot fixture (b-pyqt-bot), the eval also dry-runs the generated bot_runner.py, saves its events to evals/results/<fixture>.run.jsonl, and replays them in the page check.

• fixtures/<letter>-<name>/: small test projects for the evals.