Files
Rpa_Skills/tests/evals/results/handoff/h-web-portal-full-r2.json
T
VasanthandClaude Opus 5.5 3942fad6cd Add rpa-bot-handoff skill: ops guide for bot handover
The skill scans a bot with subagents (never runs it) and writes
HANDOVER.md, an 11-section ops guide in the operator's order: where is
what, how to run, how long to wait, after the run, how to see an
exception, exception list, how you will know, how to tell us, don'ts,
issues seen before. Dev material (check against 12 failure classes from
Tebra Auto Post, gaps with file:line, incidents) goes to HANDOVER-DEV.md.

Also adds ADR-0004, glossary terms, five fixture bots and a handoff eval
runner (--resume, --regrade) with results from 47 runs of the earlier
13-section version. The grader still expects that structure and must be
updated before the next eval run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 15:01:42 +05:30

32 lines
3.1 KiB
JSON

{
"case": "h-web-portal/full",
"rep": 2,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"no-history case says so": true,
"misleading status flagged": true,
"session re-login limit": true,
"screenshots folder": true,
"daily schedule": true
},
"cost_usd": 1.090367,
"subagents": 1,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-full-6wiwxpom\\project",
"reply": "I wrote `HANDOVER.md` in the repo root. No code was changed and the bot was never run. It's based on reading the code, config and log only, so nothing in it has been tested against a live run.\n\n| Item | Count |\n|---|---|\n| Sections | 13 (all from the template) |\n| Exceptions catalogued | 10 (E01\u2013E10) |\n| Statuses documented | all 8 that the code writes |\n| Failure classes | all 12 covered, none marked not applicable |\n| Fix before handover | 19 items |\n\n**Fix before handover \u2014 the important ones:**\n- **Maintenance shown as success:** the row says \"Downloaded\" in green, but no file was downloaded (#1).\n- **Any new CSV counts as success:** the file isn't checked against the provider (#2).\n- **One crash drops the rest:** if a re-login fails or the Export button is missing, the run stops and every later provider is missing from the report (#4, #5).\n- **No reason in the log:** the log never says why a provider failed, and the cause of a crash is lost (#6, #16).\n- **No report at all on early failures:** if the input file is open or a folder is missing, today's report is never written (#7, #8).\n- **Plain-text secrets:** the portal password sits in the input Excel, and an unused `api_token` sits in `config.yaml`. Remove the token and get it rotated (#14, #15).\n- The rest: no watchdog, no retries, a same-day re-run overwrites the report, and no version number.\n\n**Gaps you should know about:**\n- **No history:** there's no git history or changelog, so I skipped the history scan. Known incidents has only one entry, from the single run in `logs/bot.log` (2026-09-02).\n- **Folders don't exist:** the `data/*` folders aren't in the repo and the code never creates them, so the doc lists them as a pre-run check.\n- **Task Scheduler:** your access list doesn't include it, so for stopping or re-running the bot the doc tells ops to ask in #rpa-support if they don't have rights.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}