- templates/bot_runner.py (new, no Qt): runs the bot as a child process (unbuffered, UTF-8), tails its output, keeps raw lines in bot_raw.log, and turns lines into step/log/status events. Step markers and friendly rules use specific substrings, first match wins; unmatched error lines are always shown, other unmatched lines hidden. Dry run: python dashboard/bot_runner.py prints the events the page would get. - bridge.py: delegates to bot_runner, one run at a time, restores the saved dark mode on every page load (fixes sample-data mode in PyQt), adds load_latest_report / download_report slots the page calls. - SKILL.md: skip the stack question for a PyQt project; desktop wiring steps from the bot's real output; verify steps for the host. - tests: assert-based self-check for the line logic and run loop; Seam 1 run replay (status dot, Run button, steps, log) and wired-page mode; fixture b-pyqt-bot and its Seam 2 eval (all 14 checks pass). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests
Run everything from the repo root:
uv run --with pytest --with playwright pytest tests
-
test_*.py: assert-based self-checks for template logic (lint of unfilled Python templates, the custom palette, presets in step with the template). -
page_check/: Seam 1. Loads a dashboard page in headless Chromium with no backend, pushes contract envelopes throughwindow.__pyMsg, and asserts on what is visible. Every case runs in light, dark and narrow (800px) mode.- By default it checks the no-AI template, filled with test values. To check a
generated page instead, set
PAGE_CHECK_HTML=path/to/dashboard.html. - The page sees a fake Qt bridge. Every action the page sends is recorded in
window.__callsas[slot, ...args]. PAGE_CHECK_WIRED=1: the page is wired to a bot (sample data off). Sample-data cases skip, and a case checks that the page's actions go to the bot. With noPAGE_CHECK_HTML, the template is checked withSAMPLE_DATAswitched off.PAGE_CHECK_REPLAY=run.jsonl: replay a real bot run (the JSON lines thatpython dashboard/bot_runner.pyprints) instead of the built-in run.- To add a case, write a
test_*.pyfile inpage_check/that uses thepageandpush(event, data)fixtures fromconftest.py.
- By default it checks the no-AI template, filled with test values. To check a
generated page instead, set
-
evals/: Seam 2. Runs the skills on a fixture project withclaude -p, then grades the reply and runs the page check on the page it built:uv run --with pytest --with playwright python tests/evals/run_eval.py a-emptyResults go to
evals/results/<fixture>.json. To add a fixture, add an entry toFIXTURESinrun_eval.py. For a bot fixture (b-pyqt-bot), the eval also dry-runs the generatedbot_runner.py, saves its events toevals/results/<fixture>.run.jsonl, and replays them in the page check. -
fixtures/<letter>-<name>/: small test projects for the evals.