Eval results for tickets 04-07; grader ignores bot-run skips; resolve tickets 01-07

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-16 16:56:16 +05:30
1 parent 258bfbb500
commit 312742cfef
@@ -12,7 +12,7 @@ An automated page check proves the page renders the standard data contract corre

 **Blocked by:** None (can start immediately)

-**Status:** ready-for-agent
+**Status:** resolved

 - [ ] The desktop host template passes the lint hook unfilled. Its placeholders go inside string literals.
 - [ ] The no-AI skill file exists with a trigger description ("RPA dashboard", "automation dashboard", "bot dashboard", "without AI") and the workflow in spec order: scan → report/keep-or-restyle → colours → stack → optional sections → build → wire → verify.
@@ -28,3 +28,7 @@ An automated page check proves the page renders the standard data contract corre
  - the dark toggle flips the theme.
 - [ ] The page check runs in light, dark and narrow width (≤820px).
 - [ ] Skill eval on test project (a), an empty project: the skill asks colours, stack and sections, never reports a frontend, and its output passes the page check.
+
+## Comments
+
+Resolved on branch `integration/rpa-dashboard-skills` (Oct 16, 2026). Tests: `uv run --with pytest --with playwright pytest tests` all green; skill eval results in `tests/evals/results/`.
@@ -11,7 +11,7 @@ Part of: ../spec.md

 **Blocked by:** 01

-**Status:** ready-for-agent
+**Status:** resolved

 - [ ] The skill detects the PyQt project and the bot entry point, and fills in the host template's bot command, steps, step markers and friendly rules from the bot's real output.
 - [ ] Clicking Run starts the bot in a separate process. The Run button disables itself while running, and the status dot goes ready → running → done/error.
@@ -20,3 +20,7 @@ Part of: ../spec.md
 - [ ] The dark-mode choice persists across app restarts.
 - [ ] A Python assert-based self-check covers line classification, step-marker matching and friendly-rule matching, including the "never hide an error" case.
 - [ ] Skill eval on test project (b), a PyQt project with a step-marker bot: the skill doesn't ask the stack question, and the running page shows real steps and log lines.
+
+## Comments
+
+Resolved on branch `integration/rpa-dashboard-skills` (Oct 16, 2026). Tests: `uv run --with pytest --with playwright pytest tests` all green; skill eval results in `tests/evals/results/`.
@@ -6,10 +6,14 @@ Part of: ../spec.md

 **Blocked by:** 01

-**Status:** ready-for-agent
+**Status:** resolved

 - [ ] Detection covers plain HTML, React/Vue projects and an embedded PyQt web view, and reports location, stack and the main colour tokens.
 - [ ] "Keep": the new dashboard page uses the project's existing tokens and fonts. No existing styles are changed.
 - [ ] "Restyle": only token blocks change. A diff shows no layout or logic edits.
 - [ ] The colour question is skipped when the user keeps the existing look. The stack question is skipped whenever a frontend exists.
 - [ ] Skill eval on test project (c), a web project with an existing frontend and its own colours: the skill reports the frontend correctly. Both branches produce a page that passes the page check.
+
+## Comments
+
+Resolved on branch `integration/rpa-dashboard-skills` (Oct 16, 2026). Tests: `uv run --with pytest --with playwright pytest tests` all green; skill eval results in `tests/evals/results/`.
@@ -11,10 +11,14 @@ Sections that aren't picked leave no code behind.

 **Blocked by:** 01

-**Status:** ready-for-agent
+**Status:** resolved

 - [ ] The skill asks for the optional sections with a multi-select and builds only the chosen ones.
 - [ ] Checkpoint tracker: the page check pushes checkpoint-state events and asserts each box shows pending, active, done or stuck.
 - [ ] Admin/roles: the page check asserts the admin nav and screen appear for the admin role and are hidden for operators.
 - [ ] Login page: renders in the chosen theme, light and dark, and hands off to the project's auth or a stub.
 - [ ] With no optional sections picked, the output contains no leftover markup, scripts or styles for them.
+
+## Comments
+
+Resolved on branch `integration/rpa-dashboard-skills` (Oct 16, 2026). Tests: `uv run --with pytest --with playwright pytest tests` all green; skill eval results in `tests/evals/results/`.
@@ -9,7 +9,7 @@ Part of: ../spec.md

 **Blocked by:** 01

-**Status:** ready-for-agent
+**Status:** resolved

 - [ ] The AI skill file exists with a trigger description ("AI RPA dashboard", "extraction dashboard", "LLM/OCR review dashboard") and the shared workflow.
 - [ ] An AI contract extends the no-AI contract. These shapes came from the Superbill prototype:
@@ -25,3 +25,7 @@ Part of: ../spec.md
  - approve stays blocked while rows are unverified (the server stub refuses);
  - "page X of Y" updates while running.
 - [ ] Skill eval on test project (d), a project with an LLM extraction call: the skill picks the AI path, asks the shared questions, and its output passes the page check with sample data.
+
+## Comments
+
+Resolved on branch `integration/rpa-dashboard-skills` (Oct 16, 2026). Tests: `uv run --with pytest --with playwright pytest tests` all green; skill eval results in `tests/evals/results/`.
@@ -12,7 +12,7 @@ Cost comes from a pricing table that records its source and its as-of date. Opti

 **Blocked by:** 05

-**Status:** ready-for-agent
+**Status:** resolved

 - [ ] Cost per call = input tokens × input rate + (output + thinking tokens) × output rate, per million tokens. Long-context rates apply above the threshold.
 - [ ] An unknown model raises an error that names the known models. Cost never silently defaults to zero.
@@ -20,3 +20,7 @@ Cost comes from a pricing table that records its source and its as-of date. Opti
 - [ ] Contract: `RoiRun{model,input_tokens,output_tokens,thinking_tokens,total_cost}` and a fetch-ROI action. (Shape from the Superbill prototype.)
 - [ ] The page check asserts that the spend tiles sum the runs, the coverage pill counts priced vs. total documents, and the month breakdown groups runs correctly.
 - [ ] The model picker and retry-on-other-model are offered as optional sections and built only when chosen.
+
+## Comments
+
+Resolved on branch `integration/rpa-dashboard-skills` (Oct 16, 2026). Tests: `uv run --with pytest --with playwright pytest tests` all green; skill eval results in `tests/evals/results/`.
@@ -6,10 +6,14 @@ Part of: ../spec.md

 **Blocked by:** 02, 05

-**Status:** ready-for-agent
+**Status:** resolved

 - [ ] Plain-HTML AI sections exist, use the shared theme tokens, and render in light, dark and narrow width.
 - [ ] The AI events and actions travel over the desktop bridge using the same envelope as the no-AI path.
 - [ ] Approval is still enforced in Python on the bridge side. Changing the page cannot bypass it.
 - [ ] The page check runs the AI cases from tickets 05 and 06 against the desktop page and they pass.
 - [ ] Skill eval on a PyQt test project with an LLM extraction call: the skill picks the desktop AI path, and its output passes the page check.
+
+## Comments
+
+Resolved on branch `integration/rpa-dashboard-skills` (Oct 16, 2026). Tests: `uv run --with pytest --with playwright pytest tests` all green; skill eval results in `tests/evals/results/`.