AI desktop path: plain-HTML Review and Cost tabs over the Qt bridge, gate in Python

ai_sections.html pastes into the no-AI dashboard.html; ai_bridge.py adds one
ai_request slot that hands every AI contract request (incl. GET /roi) to the
review backend's dispatch(). Desktop page check reruns the web review and cost
cases plus a tampered-page gate case. Eval fixture e-pyqt-ai added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

rpa-dashboard-ai/SKILL.md

@@ -15,7 +15,7 @@ means `../rpa-dashboard-no-ai/templates/`.
 | `templates/web/pricing.py` | The pricing table (with its source and as-of date) and `cost_for_call`. The backend imports it. |
 | `templates/web/server_stub.py` | Sample-data backend for runs, review rows and the approve gate. Standard library only. |
 | `templates/web/bands.py` | The band cut points and rules. The one source of truth: the backend imports it. |
-| `templates/desktop/ai_sections.html` | Desktop: the Review tab (band legend, review split pane, drafts, Verify, approve gate) in plain HTML, pasted into `<no-ai>dashboard.html`. |
+| `templates/desktop/ai_sections.html` | Desktop: the Review tab (band legend, review split pane, drafts, Verify, approve gate) and the Cost tab in plain HTML, pasted into `<no-ai>dashboard.html`. |
 | `templates/desktop/ai_bridge.py` | Desktop: the no-AI `Bridge` plus the `ai_request` slot that hands every AI request to the Python review backend. |
 | `templates/contract.md` | The AI contract: runs, review rows, endpoints and the gate, over HTTP or the Qt bridge. Extends `<no-ai>contract.md`. |
 | `templates/reference/` | Superbill Review extracts, marked EXAMPLE ONLY. Use them to port extra polish, never copy their project names. |
@@ -71,9 +71,9 @@ pane, the approve gate and the cost page. Ask which extras to add (choose any, o
 the model per batch), **retry on another model** (for failed runs). Build only the
 chosen ones at their `OPTIONAL:` markers, and delete the markers.
 - **Model picker:** a `<select>` filled from `GET /models` (the pricing table's keys), at
-  the marker in `Cost.tsx` or next to the Run button; the chosen model goes with the
+  the marker in `Cost.tsx` (desktop: `ai_sections.html`) or next to the Run button; the chosen model goes with the
   batch's start action. Remember the last choice per user.
-- **Retry on another model:** on a `failed` run, at the marker in `Review.tsx`: a model
+- **Retry on another model:** on a `failed` run, at the marker in `Review.tsx` (desktop: `ai_sections.html`): a model
   `<select>` (from `GET /models`, defaulting to the run's own model) and a Retry button
   that sends `POST /runs/{job_id}/retry {model}`, then polls the run as usual.
 Both need the endpoints in `templates/contract.md`; add them to the backend (and the stub).
@@ -106,11 +106,11 @@ Both need the endpoints in `templates/contract.md`; add them to the backend (and
 1. Build the no-AI desktop page and host first, exactly as steps 6 and 7 of
    `../rpa-dashboard-no-ai/SKILL.md` say: `dashboard/dashboard.html` (theme, placeholders,
    `SAMPLE_DATA = false` once the bot is wired), `bridge.py` and `bot_runner.py`.
-2. Paste the whole of `templates/desktop/ai_sections.html` into `dashboard/dashboard.html`,
+2. Paste the whole of `templates/desktop/ai_sections.html` (Review and Cost tabs) into `dashboard/dashboard.html`,
    just before `</body>`. Change nothing in it: it uses the page's theme tokens, and its
    band colours are fixed.
-3. Copy `templates/desktop/ai_bridge.py`, `templates/web/server_stub.py` and
-   `templates/web/bands.py` into `dashboard/`. Fill `{{APP_NAME}}` in `ai_bridge.py`.
+3. Copy `templates/desktop/ai_bridge.py`, `templates/web/server_stub.py`,
+   `templates/web/bands.py` and `templates/web/pricing.py` into `dashboard/`. Fill `{{APP_NAME}}` in `ai_bridge.py`.
 4. The app now starts with `python dashboard/ai_bridge.py` (not `bridge.py`).
 5. Search the output for `{{`. None may be left.
 

rpa-dashboard-ai/templates/contract.md

@@ -78,6 +78,9 @@ The same endpoints and rules travel over the Qt bridge instead of HTTP:
   path above without `{API_BASE}`, `body` is a JSON string (`"{}"` when there is none).
 - Python to page: one envelope, `{event: "ai_response", data: {id, status, body}}`, with the
   same `status` and JSON `body` the web endpoint would answer.
+- Every endpoint above uses this pair, the cost page's `GET /roi` included (the page sends
+  `ai_request(id, "GET", "/roi", "{}")` and gets `{runs: [RoiRun], pricing: {source, as_of}}`
+  back as the `body`). No other AI events or slots exist.
 - `GET /runs/{job_id}/document` answers an SVG string or a URL the page can load (for
   example a `file:///` path to the PDF).
 

rpa-dashboard-ai/templates/desktop/ai_sections.html

@@ -34,6 +34,9 @@ body{--clean:#51a68f;--verify:#e39b34;--review:#de5243;--absent:#6b6580}
 .rv-fields .btn{width:92px}
 .rv-save{display:flex;align-items:center;gap:10px;border-top:1px solid var(--border);padding:10px 16px}
 .rv-save .rv-unsaved{font-size:12px;font-weight:700;color:var(--warn)}
+.co-tile{padding:16px 18px;border-radius:16px;background:var(--glass-strong);border:1px solid var(--border)}
+.co-tile .kpi-val{margin-top:4px}
+.co-r{text-align:right}
 @media (max-width:820px){.rv-split{flex-direction:column}.rv-filters{margin-left:0}}
 </style>
 
@@ -43,6 +46,8 @@ body{--clean:#51a68f;--verify:#e39b34;--review:#de5243;--absent:#6b6580}
       <label>Document <select id="rv-doc-pick" aria-label="Document"></select></label>
       <span id="rv-state"></span>
       <div class="rv-right" id="rv-actions"></div>
+      <!-- OPTIONAL: retry on another model (failed runs) goes here -->
+
       <p class="rv-alert hidden" id="rv-alert" role="alert"></p>
     </div>
     <div class="card rv-bar">
@@ -61,6 +66,29 @@ body{--clean:#51a68f;--verify:#e39b34;--review:#de5243;--absent:#6b6580}
   </div>
 </template>
 
+<template id="ai-cost-tab">
+  <div id="tab-cost" class="hidden">
+    <div class="toolbar"><span class="pill" id="co-cover"></span><span class="rv-muted" id="co-source"></span></div>
+    <div class="kpi-row" id="co-tiles"></div>
+    <div class="card"><div class="c-title">Month by month</div>
+      <div class="tbl-wrap"><table aria-label="Spend by month"><thead><tr><th>Month</th><th>Documents priced</th><th class="co-r">Spend</th></tr></thead><tbody id="co-months"></tbody></table></div>
+    </div>
+    <section class="card" aria-labelledby="co-by-hand">
+      <div class="c-title" id="co-by-hand">Against doing it by hand</div>
+      <p class="rv-muted" style="margin-bottom:12px">These are your assumptions, not a measurement. Change them to match the team.</p>
+      <div class="toolbar">
+        <label class="rv-muted">Minutes per document <input id="co-min" type="number" min="0" value="5" style="width:80px"></label>
+        <label class="rv-muted">Hourly rate <input id="co-rate" type="number" min="0" value="20" style="width:80px"></label>
+      </div>
+      <p id="co-hand" style="font-size:13px"></p>
+    </section>
+    <!-- OPTIONAL: model picker (per batch) goes here -->
+    <div class="card"><div class="c-title">Cost per run</div>
+      <div class="tbl-wrap"><table aria-label="Cost per run"><thead><tr><th>Document</th><th>Date</th><th>Model</th><th class="co-r">Input</th><th class="co-r">Output</th><th class="co-r">Thinking</th><th class="co-r">Cost</th></tr></thead><tbody id="co-runs"></tbody></table></div>
+    </div>
+  </div>
+</template>
+
 <script>
 (()=>{
 // Legend order, label and colour token. Cut points come from bands.py on the Python side.
@@ -71,18 +99,21 @@ const BUSY=['queued','running'];   // polled every 1.5s while in one of these
 const colourOf=b=>BANDS.find(x=>x[0]===b)[2];
 const settled=r=>r.corrected_value??r.value??'';
 
-// ----- the tab: nav button after Report, section in the content area -----
+// ----- the tabs: nav buttons after Report, sections in the content area -----
 TABS.review=['Review','Check extracted fields, then approve'];
+TABS.cost=['Cost','What the AI cost, run by run'];
 document.querySelector('.nav-item[data-tab="report"]').insertAdjacentHTML('afterend',
-  '<button class="nav-item" data-tab="review" title="Review"><svg viewBox="0 0 24 24"><path d="M9 11l3 3 8-8"/><path d="M20 12v7a1 1 0 0 1-1 1H5a1 1 0 0 1-1-1V5a1 1 0 0 1 1-1h11"/></svg><span>Review</span></button>');
-document.querySelector('.nav-item[data-tab="review"]').onclick=()=>showTab('review');
-document.querySelector('.content').append($('ai-review-tab').content.cloneNode(true));
+  '<button class="nav-item" data-tab="review" title="Review"><svg viewBox="0 0 24 24"><path d="M9 11l3 3 8-8"/><path d="M20 12v7a1 1 0 0 1-1 1H5a1 1 0 0 1-1-1V5a1 1 0 0 1 1-1h11"/></svg><span>Review</span></button>'
+  +'<button class="nav-item" data-tab="cost" title="Cost"><svg viewBox="0 0 24 24"><path d="M12 2v20M17 6H9.5a3.5 3.5 0 0 0 0 7h5a3.5 3.5 0 0 1 0 7H6"/></svg><span>Cost</span></button>');
+['review','cost'].forEach(t=>{ document.querySelector(`.nav-item[data-tab="${t}"]`).onclick=()=>showTab(t);
+  document.querySelector('.content').append($(`ai-${t}-tab`).content.cloneNode(true)); });
 const baseShow=showTab;
 showTab=t=>{
   const onReview=!$('tab-review').classList.contains('hidden');
   if(onReview && t!=='review' && !leaveOk()) return;
   baseShow(t);
   if(t==='review') loadRuns();
+  if(t==='cost') loadRoi();
 };
 window.addEventListener('beforeunload',e=>{ if(dirty().length) e.preventDefault(); });
 
@@ -192,5 +223,42 @@ function renderSave(){
     <button class="btn" id="rv-discard" style="margin-left:auto">Discard</button><button class="btn primary" id="rv-save-btn">Save</button>` : '';
   if(n){ $('rv-discard').onclick=()=>{ drafts={}; render(); }; $('rv-save-btn').onclick=save; }
 }
+
+// ----- cost: measured (tokens and cost per run, priced in Python) kept apart from assumed (typed here) -----
+// A run with no recorded token usage is "not priced", never $0, and the coverage pill says how many.
+let roi=[];
+const money=n=>`$${n.toFixed(n>0&&n<0.01?4:2)}`, num=n=>n==null?'—':n.toLocaleString('en-US');
+const monthOf=iso=>iso.slice(0,7);   // 'YYYY-MM', UTC
+const monthLabel=ym=>new Date(`${ym}-01T00:00:00Z`).toLocaleString('en-US',{month:'short',year:'numeric',timeZone:'UTC'});
+const sum=rs=>rs.reduce((n,r)=>n+(r.total_cost??0),0);
+const tileHtml=(name,value,note)=>`<div class="co-tile" role="group" aria-label="${name}"><div class="kpi-lbl">${name}</div><div class="kpi-val">${value}</div><div class="kpi-lbl">${esc(note)}</div></div>`;
+async function loadRoi(){
+  let d; try{ d=await ai('GET','/roi'); }catch(e){ $('co-source').textContent=`Can't load costs: ${e.message}`; return; }
+  roi=d.runs; const p=d.pricing||{};
+  $('co-source').innerHTML=`Rates from ${esc(p.source)}, as of ${esc(p.as_of)}`;
+  renderCost();
+}
+function renderCost(){
+  const priced=roi.filter(r=>r.total_cost!=null), thisMonth=new Date().toISOString().slice(0,7);
+  const tok=k=>priced.reduce((n,r)=>n+(r[k]??0),0);
+  $('co-cover').textContent=`${priced.length} of ${roi.length} documents priced`;
+  $('co-cover').className='pill '+(priced.length===roi.length?'success':'attention');
+  $('co-tiles').innerHTML=tileHtml('Total spend',money(sum(priced)),`${priced.length} priced runs`)
+    +tileHtml('This month',money(sum(roi.filter(r=>monthOf(r.created_at)===thisMonth))),monthLabel(thisMonth))
+    +tileHtml('Tokens',num(tok('input_tokens')+tok('output_tokens')+tok('thinking_tokens')),`in ${num(tok('input_tokens'))} · out ${num(tok('output_tokens'))} · thinking ${num(tok('thinking_tokens'))}`)
+    +tileHtml('Per document',priced.length?money(sum(priced)/priced.length):'—','average, priced runs');
+  $('co-months').innerHTML=[...new Set(roi.map(r=>monthOf(r.created_at)))].sort().reverse().map(m=>{
+    const rs=roi.filter(r=>monthOf(r.created_at)===m), n=rs.filter(r=>r.total_cost!=null).length;
+    return `<tr><td>${monthLabel(m)}${m===thisMonth?' <span class="rv-muted">this month</span>':''}</td><td>${n} of ${rs.length}</td><td class="co-r">${n?money(sum(rs)):'—'}</td></tr>`; }).join('');
+  $('co-runs').innerHTML=roi.map(r=>`<tr><td>${esc(r.document)}</td><td>${esc(r.created_at.slice(0,10))}</td><td>${esc(r.model)}</td>
+    <td class="co-r">${num(r.input_tokens)}</td><td class="co-r">${num(r.output_tokens)}</td><td class="co-r">${num(r.thinking_tokens)}</td>
+    <td class="co-r">${r.total_cost==null?'<span class="rv-muted">not priced</span>':money(r.total_cost)}</td></tr>`).join('');
+  renderByHand();
+}
+function renderByHand(){
+  const n=roi.filter(r=>r.total_cost!=null).length, min=+$('co-min').value||0, rate=+$('co-rate').value||0;
+  $('co-hand').innerHTML=`By hand, ${n} documents would take ${num(Math.round(n*min/6)/10)} h, about <b>${money(n*min/60*rate)}</b>. The AI spent <b>${money(sum(roi))}</b> on them.`;
+}
+$('co-min').oninput=$('co-rate').oninput=renderByHand;
 })();
 </script>

tests/README.md

@@ -29,6 +29,12 @@ uv run --with pytest --with playwright pytest tests
   by-hand assumptions), plus every no-AI case. Tests set run and review state on the stub
   directly (the `stub` fixture). To check a generated dashboard folder instead, set
   `PAGE_CHECK_AI_DIR=path/to/dashboard`. Needs npm on PATH, else it skips.
+- `page_check_ai_desktop/`: **Seam 1, AI desktop page.** The no-AI `dashboard.html` with
+  `ai_sections.html` pasted in, loaded from `file://` with a fake Qt bridge whose
+  `ai_request` slot goes to `server_stub.dispatch()` in this process (what `ai_bridge.py`
+  does). Runs the AI web page's review and cost cases and the no-AI cases unchanged, plus
+  a tampered page calling the slot directly (still refused). To check a generated folder,
+  set `PAGE_CHECK_AI_DESKTOP_DIR=path/to/dashboard`; `PAGE_CHECK_WIRED=1` as above.
 - `test_pricing.py`: the AI cost function (normal, long-context, thinking tokens,
   unknown model).
 - `test_reference_scan.py`: no patient data, internal IPs, credentials or keys in any
@@ -45,7 +51,9 @@ uv run --with pytest --with playwright pytest tests
   fixture folder with different answers: `c-keep` and `c-restyle` both run on
   `c-web-existing-frontend`, and also check what changed in the files it already had.
   An `"ai": True` entry (`d-ai-extraction`) expects the AI skill's React starter and
-  runs `page_check_ai/` on the folder it built.
+  runs `page_check_ai/` on the folder it built. An `"ai_desktop": True` entry
+  (`e-pyqt-ai`) expects the desktop AI page and runs `page_check/` (with the bot replay)
+  and `page_check_ai_desktop/` on it.
   For a bot fixture (`b-pyqt-bot`), the eval also dry-runs the generated
   `bot_runner.py`, saves its events to `evals/results/<fixture>.run.jsonl`, and
   replays them in the page check.

tests/evals/run_eval.py

@@ -80,6 +80,17 @@ FIXTURES = {
         "page_has": ["--accent:#ff3049"],
         "ai": True,
     },
+    # (e) PyQt bot with an LLM extraction call: the AI skill's desktop path, no web server.
+    "e-pyqt-ai": {
+        "prompt": "Build me an RPA dashboard for this project. I can't answer questions while you "
+                  "work, so here are my answers to anything the skill asks. Colours: Tebra Sky. "
+                  "Optional sections: none.",
+        "frontend": r"^\s*(none|no\b)",
+        "asks": [r"colou?r", r"section"],
+        "never_asks": [r"\bkeep\b|restyle", r"desktop|web"],
+        "bot": True,
+        "ai_desktop": True,
+    },
 }
 # A line a restyle may change: a custom property, a comment, a blank, or a web-font link.
 TOKEN_LINE = r"^\s*(--[\w-]+\s*:.*|/\*.*\*/|<link[^>]*fonts\.googleapis\.com.*|@import.*fonts\.googleapis\.com.*)?\s*$"
@@ -171,6 +182,11 @@ def main(fixture):
             html += css.read_text(encoding="utf-8") if css.exists() else ""
             checks["AI path: review starter and server stub built"] = all(
                 (pages[0].parent / f).exists() for f in ["server_stub.py", "bands.py", "src/components/Review.tsx"])
+        if cfg.get("ai_desktop"):  # the no-AI page with the plain-HTML AI sections, hosted by ai_bridge.py
+            host = pages[0].with_name("ai_bridge.py")
+            checks["AI desktop path: sections, ai_bridge and review backend built"] = 'id="ai-review-tab"' in html and all(
+                pages[0].with_name(f).exists() for f in ["ai_bridge.py", "server_stub.py", "bands.py", "pricing.py"])
+            checks["no placeholders left in ai_bridge.py"] = host.exists() and "{{" not in host.read_text(encoding="utf-8")
         shutil.copy(pages[0], results / f"{fixture}.dashboard.html")
         checks["no placeholders left"] = "{{" not in html
         if cfg.get("brand"):
@@ -186,8 +202,11 @@ def main(fixture):
             env.update(bot_checks(project, html, checks, results / f"{fixture}.run.jsonl"))
         if cfg.get("ai"):
             env["PAGE_CHECK_AI_DIR"] = str(pages[0].parent)
-        check_dir = ROOT / "tests" / ("page_check_ai" if cfg.get("ai") else "page_check")
-        pc = subprocess.run([sys.executable, "-m", "pytest", str(check_dir), "-q"],
+        check_dirs = [ROOT / "tests" / ("page_check_ai" if cfg.get("ai") else "page_check")]
+        if cfg.get("ai_desktop"):
+            env["PAGE_CHECK_AI_DESKTOP_DIR"] = str(pages[0].parent)
+            check_dirs.append(ROOT / "tests" / "page_check_ai_desktop")
+        pc = subprocess.run([sys.executable, "-m", "pytest", *map(str, check_dirs), "-q"],
                             env=env, capture_output=True, text=True)
         checks["page check passes"] = pc.returncode == 0
         page_check = pc.stdout.strip().splitlines()[-1:] if pc.stdout else [pc.stderr[-500:]]

tests/fixtures/e-pyqt-ai/README.md

@@ -0,0 +1,11 @@
+# Invoice intake bot
+
+A desktop RPA bot. It reads each scanned invoice in `inbox/` with an LLM
+(`extractor.py`) and queues the fields for a person to review.
+The operator window will be a PyQt6 app.
+
+Run the bot on its own:
+
+```
+python -m bot
+```

tests/fixtures/e-pyqt-ai/bot.py

@@ -0,0 +1,31 @@
+"""Invoice intake bot: picks up scanned invoices and extracts their fields with an LLM.
+Start it with:  python -m bot"""
+import time
+from pathlib import Path
+
+INBOX = Path(__file__).with_name("inbox")
+
+
+def say(line):
+    print(line, flush=True)
+    time.sleep(0.2)
+
+
+def main():
+    say("[step 1] Initializing")
+    docs = sorted(INBOX.glob("*.pdf"))
+    say(f"[inbox] {len(docs)} invoices waiting")
+    say("[step 2] Extracting fields")
+    for i, doc in enumerate(docs, 1):
+        try:
+            from extractor import extract_invoice  # the LLM call
+            rows = extract_invoice(doc.read_bytes(), "application/pdf")
+            say(f"[extract] {doc.name}: {len(rows)} fields ({i}/{len(docs)})")
+        except Exception as e:  # one bad document must not stop the batch
+            say(f"ERROR: extraction failed for {doc.name}: {type(e).__name__}")
+    say("[step 3] Queueing for review")
+    say(f"[all done] {len(docs)} invoices processed")
+
+
+if __name__ == "__main__":
+    main()

tests/fixtures/e-pyqt-ai/extractor.py

@@ -0,0 +1,34 @@
+"""The LLM extraction call: one scanned invoice in, its fields out, each with word scores."""
+import json
+
+from google import genai
+from google.genai import types
+
+MODEL = "gemini-2.5-flash"
+FIELDS = {  # field name -> mandatory
+    "Invoice number": True,
+    "Invoice date": True,
+    "Vendor": True,
+    "PO number": False,
+    "Due date": True,
+    "Total": True,
+}
+PROMPT = (
+    "Read this invoice. Return JSON: a list of {field, value, word_scores}, one per field in "
+    + ", ".join(FIELDS)
+    + ". word_scores is your 0-1 legibility score for each word of the value. Use an empty "
+    "value and no scores when a field isn't on the page."
+)
+
+
+def extract_invoice(document: bytes, mime_type: str) -> list[dict]:
+    client = genai.Client()  # credentials come from the environment
+    response = client.models.generate_content(
+        model=MODEL,
+        contents=[types.Part.from_bytes(data=document, mime_type=mime_type), PROMPT],
+        config=types.GenerateContentConfig(response_mime_type="application/json"),
+    )
+    rows = json.loads(response.text)
+    for row in rows:
+        row["mandatory"] = FIELDS.get(row["field"], False)
+    return rows

tests/fixtures/e-pyqt-ai/inbox/invoice-2201.pdf

@@ -0,0 +1,2 @@
+%PDF-1.4
+% sample invoice placeholder

tests/fixtures/e-pyqt-ai/inbox/invoice-2202.pdf

@@ -0,0 +1,2 @@
+%PDF-1.4
+% sample invoice placeholder

tests/fixtures/e-pyqt-ai/requirements.txt

@@ -0,0 +1,3 @@
+PyQt6
+PyQt6-WebEngine
+google-genai

tests/page_check_ai_desktop/conftest.py

@@ -63,7 +63,7 @@ def site(tmp_path_factory):
     sections = (AI / "desktop" / "ai_sections.html").read_text(encoding="utf-8")
     out = tmp_path_factory.mktemp("desktop")
     (out / "dashboard.html").write_text(html.replace("</body>", sections + "\n</body>"), encoding="utf-8")
-    for f in ["server_stub.py", "bands.py"]:
+    for f in ["server_stub.py", "bands.py", "pricing.py"]:
         (out / f).write_bytes((AI / "web" / f).read_bytes())
     return out
 

tests/page_check_ai_desktop/test_ai_desktop.py

@@ -16,6 +16,14 @@ from test_ai_review import (  # noqa: E402,F401
     test_page_x_of_y_updates_while_running,
 )
 from test_ai_review import FIELDS, FLAGGED, open_review  # noqa: E402
+from test_ai_cost import (  # noqa: E402,F401
+    cost_page,
+    test_by_hand_panel_is_labelled_assumptions_and_recomputes,
+    test_coverage_pill_counts_priced_of_all_documents,
+    test_month_breakdown_groups_runs_by_month,
+    test_runs_show_tokens_and_cost_and_unpriced_is_not_zero,
+    test_spend_tiles_sum_the_priced_runs,
+)
 
 sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "page_check"))
 from test_contract import (  # noqa: E402,F401