0 Fork 0

Files

T

VasanthandClaude Opus 5.5 9dcdd8f572 AI cost/ROI page: pricing table, cost per call, Cost tab, ROI endpoint

- templates/web/pricing.py: pricing table with source and as-of date; cost_for_call
  (input rate + (output + thinking) at output rate, long-context above threshold;
  unknown model raises naming the known models).
- Cost tab: coverage pill, total/this-month/tokens tiles, month-by-month table,
  per-run tokens and cost ("not priced", never $0), by-hand panel with editable
  assumptions labelled as such.
- Contract: RoiRun and GET /roi; optional GET /models and POST /runs/{id}/retry.
- SKILL.md: cost page wiring, model picker and retry-on-other-model as optional sections.
- Tests: test_pricing.py self-check, page_check_ai/test_ai_cost.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-16 15:23:52 +05:30

3.0 KiB

Raw Blame History

Tests

Run everything from the repo root:

uv run --with pytest --with playwright pytest tests

test_*.py: assert-based self-checks for template logic (lint of unfilled Python templates, the custom palette, presets in step with the template).
page_check/: Seam 1. Loads a dashboard page in headless Chromium with no backend, pushes contract envelopes through window.__pyMsg, and asserts on what is visible. Every case runs in light, dark and narrow (800px) mode.
- By default it checks the no-AI template, filled with test values. To check a generated page instead, set PAGE_CHECK_HTML=path/to/dashboard.html.
- The page sees a fake Qt bridge. Every action the page sends is recorded in window.__calls as [slot, ...args].
- PAGE_CHECK_WIRED=1: the page is wired to a bot (sample data off). Sample-data cases skip, and a case checks that the page's actions go to the bot. With no PAGE_CHECK_HTML, the template is checked with SAMPLE_DATA switched off.
- PAGE_CHECK_REPLAY=run.jsonl: replay a real bot run (the JSON lines that python dashboard/bot_runner.py prints) instead of the built-in run.
- To add a case, write a test_*.py file in page_check/ that uses the page and push(event, data) fixtures from conftest.py.
page_check_ai/: Seam 1, AI web page. Builds the React starter with npm (filled with test values, in a temp folder that keeps node_modules), serves it with its own server_stub.py, and checks bands, band filters, drafts, the approve gate "page X of Y" and the cost page (spend tiles, coverage pill, month breakdown, by-hand assumptions), plus every no-AI case. Tests set run and review state on the stub directly (the stub fixture). To check a generated dashboard folder instead, set PAGE_CHECK_AI_DIR=path/to/dashboard. Needs npm on PATH, else it skips.
test_pricing.py: the AI cost function (normal, long-context, thinking tokens, unknown model).
test_reference_scan.py: no patient data, internal IPs, credentials or keys in any bundled file, and every Superbill extract carries an EXAMPLE ONLY banner.
evals/: Seam 2. Runs the skills on a fixture project with claude -p, then grades the reply and runs the page check on the page it built:
```
uv run --with pytest --with playwright python tests/evals/run_eval.py a-empty
```
Results go to evals/results/<fixture>.json. To add a fixture, add an entry to FIXTURES in run_eval.py. An entry with a "fixture" key reuses another fixture folder with different answers: c-keep and c-restyle both run on c-web-existing-frontend, and also check what changed in the files it already had. An "ai": True entry (d-ai-extraction) expects the AI skill's React starter and runs page_check_ai/ on the folder it built. For a bot fixture (b-pyqt-bot), the eval also dry-runs the generated bot_runner.py, saves its events to evals/results/<fixture>.run.jsonl, and replays them in the page check.
fixtures/<letter>-<name>/: small test projects for the evals.

3.0 KiB Raw Blame History

Tests

3.0 KiB

Raw Blame History