Skills/Rpa_Skills
8
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add rpa-bot-handoff skill: ops guide for bot handover

Browse Source 
The skill scans a bot with subagents (never runs it) and writes
HANDOVER.md, an 11-section ops guide in the operator's order: where is
what, how to run, how long to wait, after the run, how to see an
exception, exception list, how you will know, how to tell us, don'ts,
issues seen before. Dev material (check against 12 failure classes from
Tebra Auto Post, gaps with file:line, incidents) goes to HANDOVER-DEV.md.

Also adds ADR-0004, glossary terms, five fixture bots and a handoff eval
runner (--resume, --regrade) with results from 47 runs of the earlier
13-section version. The grader still expects that structure and must be
updated before the next eval run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
VasanthandClaude Opus 5.5 committed 2026-10-28 15:01:42 +05:30
1 parent 3d2db6aca6
commit 3942fad6cd
143 files changed
+11815 -2
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CLAUDE.md
+1 -1
View File
@@ -1,6 +1,6 @@
# RPA SKILLS
Shared Claude skills for RPA bots, all in `skills/`: `rpa-run-tab` (Tebra-style Run tab), `rpa-dashboard-no-ai` and `rpa-dashboard-ai`. See `README.md` and `PLAN.md`.
Shared Claude skills for RPA bots, all in `skills/`: `rpa-run-tab` (Tebra-style Run tab), `rpa-dashboard-no-ai`, `rpa-dashboard-ai` and `rpa-bot-handoff` (ops guide for handover). See `README.md` and `PLAN.md`.
## Agent skills
GLOSSARY.md
+10
View File
@@ -36,3 +36,13 @@ It is hidden for bots that take no input file.
- **failed**: crash, no summary, or stuck (red)
**Fake bot**: a bundled script that prints the Run tab protocol lines, used to verify the page. The skill never runs the real bot.
**Ops guide**: the `HANDOVER.md` the `rpa-bot-handoff` skill writes in a bot's repo, for the operator who runs the bot and does not read code. 11 sections in the operator's order: where is what, how to run, how long to wait, after the run, how to see an exception, exception list, how you will know, how to tell us, don'ts, issues seen before. See ADR-0004. _Avoid_: "handover doc" for the dev notes.
**Dev notes**: the `HANDOVER-DEV.md` written next to the ops guide, for developers only: the failure-class check, the gaps to fix before handover (with file:line) and the full incident list.
**Exception list**: section 7 of the ops guide: one numbered row (`E01`...) per message the operator can see, with where it appears, what it means, what to do, and whether to wait for the bot's retries or call the dev team.
**Business exception**: the data says no (cutoff date passed, negative balance, already posted). **System exception**: the bot or the target app failed (window not found, timeout, crash).
**Failure class**: one of the 12 kinds of problem Tebra Auto Post hit after handover (`F1`-`F12` in `skills/rpa-bot-handoff/templates/failure-classes.md`). Every bot's dev notes account for all 12.
README.md
+1
View File
@@ -8,5 +8,6 @@ Claude skills for RPA projects. Each folder in `skills/` is one skill: copy it i
| `rpa-dashboard-no-ai` | Builds a full dashboard (run panel, overview, report, optional checkpoints, admin and login) for a bot with no AI. Desktop or web. |
| `rpa-dashboard-ai` | The same dashboard, plus confidence bands, review-and-approve and a cost/ROI page, for a bot that uses an LLM or OCR. Install it together with `rpa-dashboard-no-ai`. |
| `sql-review` | Reviews SQL code (SQL Server + pyodbc) against 10 rules through a subagent and writes every issue with a before/after fix plan to `query_tracker.md`. Read-only. |
| `rpa-bot-handoff` | Writes the ops guide `HANDOVER.md` for a bot being handed over: which tab shows what, how to run, how long to wait, where the report is, how to see an exception, how to tell the dev team. Also writes `HANDOVER-DEV.md` with a check against 12 failure classes from Tebra Auto Post. |
Run all tests: `uv run --with pytest --with playwright pytest tests`
docs/adr/0004-handover-is-an-ops-guide.md
+13
View File
@@ -0,0 +1,13 @@
# The handover is an ops guide in the operator's order, with dev notes kept apart
`rpa-bot-handoff` writes two files:
- `HANDOVER.md` is for the operator who runs the bot and does not read code. Its 11 sections follow the operator's own questions, in the order they ask them: where is what, how do I run it, how long do I wait, how do I read the result and the report, how do I see an exception, how will I know, how do I tell the dev team, what must I not do, what happened before.
- `HANDOVER-DEV.md` is for developers. It holds the failure-class check and the gaps to fix before handover, with file:line.
Every exception row tells ops whether to wait (the bot retries by itself), act, or call the dev team. Every wait is a number of seconds or minutes taken from the code. The skill changes no code.
**Why:** the first version was one document sorted by architecture and exception type, with dev sections mixed in. Reviewing it, the user said ops need "which tab shows what, how to run, how many seconds to wait, which tab has the report, how to see an exception, how to tell us". In Tebra Auto Post's first months, most ops questions were exactly these. Ops re-ran items the bot was still retrying. They edited rows that needed a check in Tebra first ("Payment not confirmed after Save" had really posted). They couldn't tell why a batch stayed Pending. Architecture, failure classes and file:line refs don't help an operator, and they bury what does.
**Lessons as a checklist:** Tebra's incidents are bundled as 12 generic failure classes. Each new bot is checked against every class in `HANDOVER-DEV.md`, and an unused class is written down as "not applicable" with a reason, so nothing is dropped silently. A scrubbed Tebra example shows both files.
**Subagents are required:** in the first 47 eval runs, 8 wrote the doc without starting the scan subagents. The skill now says to always start them.
skills/rpa-bot-handoff/README.md
+37
View File
@@ -0,0 +1,37 @@
# rpa-bot-handoff
A Claude skill that writes the **ops guide** for an RPA bot that is being handed over to an ops team.
## What you get
**`HANDOVER.md`**: the ops guide, in plain words, for operators who don't read code:
1. What this bot does
2. Where is what: which tab, screen, file or folder shows what
3. How to run: checklist and steps with the real button names
4. How long to wait at each step, in seconds or minutes, and when to step in
5. After the run: what each ending means, where the report is
6. How to see an exception
7. Exception list: message → where → meaning → what you do → wait or call us
8. How you will know something went wrong
9. How to tell the dev team, with a ready message template
10. Don'ts, and the right way to stop the bot
11. Issues seen before
**`HANDOVER-DEV.md`**: for developers. The check against 12 failure classes learned from Tebra Auto Post, the gaps to fix before handover (file:line), and the incident list.
## Install
Copy this folder to `~/.claude/skills/rpa-bot-handoff/`.
## Use
In your bot project, ask Claude: *"write the ops guide for this bot"* or *"document this bot for handover"*. The skill:
1. Scans the code and history with subagents. It never runs the bot.
2. Asks up to three things: how ops reaches the dev team, what ops can open, and any unclear statuses.
3. Writes both files and checks them.
## Files
| File | What it is |
|---|---|
| `SKILL.md` | Steps Claude follows |
| `templates/HANDOVER.md` | The ops guide skeleton |
| `templates/HANDOVER-DEV.md` | The dev notes skeleton |
| `templates/failure-classes.md` | 12 failure classes from Tebra's first months with ops |
| `templates/reference/tebra-auto-post.md` | Both files filled in for Tebra, scrubbed |
skills/rpa-bot-handoff/SKILL.md
+89
View File
@@ -0,0 +1,89 @@
---
name: rpa-bot-handoff
description: Writes the ops guide for an RPA bot that is being handed over to an ops/support team. Scans the bot's code and history with subagents (never runs it) and writes HANDOVER.md, a plain-language guide that answers what an operator asks - which tab or screen shows what, how to run the bot step by step, how long to wait at each step, how to read the result and the report, how to see an exception and what each message means, how they will find out something went wrong, and exactly how to tell the dev team. It also writes HANDOVER-DEV.md for developers - a check against 12 failure classes learned from Tebra Auto Post and the gaps to fix before handover. Use when the user asks to "hand over a bot", "document a bot for ops", "write a handover doc", "ops guide", "runbook", "support doc", "operator guide" or "how ops should handle bot errors" for an RPA or automation project.
---
# RPA bot handoff
Write two files in the bot's repo:
- **`HANDOVER.md`**: the ops guide. The reader is an **operator who does not read code**. It answers their questions in the order they ask them: where is what, how do I run it, how long do I wait, how do I check the result, how do I see an exception, how will I know, how do I tell the dev team.
- **`HANDOVER-DEV.md`**: for the dev team only. It holds the failure-class check and the gaps to fix before handover, with file:line.
| File | What it is |
|---|---|
| `templates/HANDOVER.md` | The ops guide skeleton: 11 sections. Keep their order. |
| `templates/HANDOVER-DEV.md` | The dev notes skeleton. |
| `templates/failure-classes.md` | 12 failure classes from Tebra Auto Post. Every one must be checked against the bot. |
| `templates/reference/tebra-auto-post.md` | A filled example of both files for Tebra, scrubbed. Copy its depth and wording, not its content. |
Rules:
- **Never run the bot.** Real bots act on live systems. Read code, config, logs and history only.
- **Use the operator's words.** Name the real tab, button, column, colour and message the operator sees, exactly as the app shows them ("Report tab → File Status column → Error"). No file:line, no class or function names, no SQL in `HANDOVER.md`. File:line refs belong in `HANDOVER-DEV.md` only.
- **Every wait is a number.** Take timeouts, retries and watchdog limits from the code, in seconds or minutes. Take normal durations from the logs when they show them; otherwise write "not measured".
- **Write what the code does, not what it should do.** If a status is misleading (an item needing follow-up shows green), say so plainly in the ops guide and add it to the gaps in `HANDOVER-DEV.md`.
- **Never copy** patient data (PHI), customer names, credentials, internal IPs, keys or `.env` values into either file. Logs are read for message types and durations only.
- Change no code. The only output is the two files.
## 1. Scan (subagents are required)
**Always start the subagents.** Never write the guide from your own reading of the code. Send subagent A, and subagent B when there is history, in **one message** so they run in parallel. Tell each one: read-only, never run the bot, no PHI/credentials/IPs in the report, give file:line refs, under 1500 words.
**Subagent A: what the operator sees and does.** Ask it to report:
1. **Screens:** every window, tab, page and button the operator uses, with its exact label and what it shows. Output files and folders ops can open, with name patterns and columns.
2. **How to run:** how the bot starts (shortcut, exe, launcher, scheduler, button), what input it needs (file, sheets, columns, rules), and what the screen shows while it runs (progress, steps, checkpoint boxes, log).
3. **Waits:** every timeout, retry count, restart limit and watchdog limit, in seconds, with the step it belongs to. Typical durations seen in the logs.
4. **Result:** how a run ends (banner, colours, summary), and every status, comment and message the bot writes per item, with the colour it shows on screen.
5. **Exceptions:** every place the bot detects a problem: what it checks, the exact message the operator sees, where it appears (screen, column, file), whether the bot retries by itself and how often, and what happens after the last retry.
6. **Alerts:** whether the bot sends any email, popup, sound or notification when something fails. If not, say so.
7. **Rules:** anything the operator must not do while it runs (use the machine, close an app, open other windows), and how to stop it safely.
**Subagent B: history.** Skip it only when there is no git history, bug log, changelog or release notes. Ask it to mine `git log` (fix/bug/hotfix commits), bug logs, changelogs, PROGRESS / RELEASE / DEPLOY notes, regression tests and recurring messages in `logs/`. For each real incident: what the operator saw, the cause, the fix, and **what the operator should do if it happens again**.
While they run, read `templates/failure-classes.md` and `templates/reference/tebra-auto-post.md`.
## 2. Ask once (one message, at most 3 questions)
Send a 5-line summary of what the scan found, then ask:
1. **Escalation:** how ops tells the dev team: a chat channel, ticket queue or email group (a team or role, never a personal phone number), and how fast to expect an answer.
2. **Ops access:** which places ops can open: the app's screens, the output folder, the log folder, the database. The guide sends ops only to places they can open. The rest go to the gaps in `HANDOVER-DEV.md`.
3. **Unknowns:** any status or message whose meaning the code doesn't make clear. Skip this when there are none.
Default everything else: title from the project name, today's date, version from the project's version string.
## 3. Write `HANDOVER.md` (the ops guide)
Copy `templates/HANDOVER.md` to the bot's repo root and fill every `{{PLACEHOLDER}}`. Section notes:
1. **What this bot does:** at most 2 lines.
2. **Where is what:** one row per tab, screen, file or folder the operator uses: what it shows and when to open it. Only places ops can open (question 2).
3. **How to run:** a short "before you start" checklist (prerequisites, input rules with one bad and one good example), then numbered steps with the real button labels, then what a normal run looks like on screen.
4. **How long to wait:** one row per step: normal time, how long the bot waits at most, what the bot does after that, and what the operator does. End with the line that answers "when do I step in?".
5. **After the run:** what each ending colour or banner means, then exactly where the report is (tab, filter, columns) and how to export it.
6. **How to see an exception:** the click path from the report to the message, then "find the message in section 7".
7. **Exception list:** one row per message. Number them `E01`, `E02`... so ops can quote them.
- *Message you see*: the exact text.
- *Where*: tab and column, screen, or file.
- *What it means*: one plain sentence.
- *What you do*: numbered steps when there are several. Put any "check X **before** re-running" warning in bold.
- *Bot retries / call us when*: "the bot retries 3 times, wait" or "call us if it fails again".
8. **How you will know:** the alerts the bot sends, or "there is no alert: check the screen after every run", and when to check.
9. **How to tell us:** the channel from question 1, what to send (run/batch ID, time, E-number, screenshot, file name), what never to send (passwords, patient or customer data), and a ready-to-copy message template.
10. **Don'ts:** the rules from subagent A item 7, each with a short reason, plus the one right way to stop the bot.
11. **Issues seen before:** from subagent B, newest first: what you saw, what to do now. One line each for incidents fixed for good.
## 4. Write `HANDOVER-DEV.md` (dev notes)
Copy `templates/HANDOVER-DEV.md` and fill it:
- **Failure-class check:** walk `templates/failure-classes.md` one class at a time. Each class must end up in at least one of: a row in the ops guide (say which section or E-number), a gap in this file, or `F<n> not applicable: <reason>`. Never drop a class silently.
- **Gaps to fix before handover:** each with why it hurts ops and file:line. Include every misleading status and every cause ops cannot see.
- **Incidents:** the full list from subagent B with commit refs.
## 5. Verify
1. Every status and message from the scan appears in section 7 or section 5 of the ops guide.
2. Every tab, button, column, file and folder named in the ops guide exists in the code or is created by it.
3. Every row in section 4 has numbers, and every row in section 7 has all columns filled.
4. Search both files for PHI, customer names, credentials, IPs, keys and `.env` values. There must be none.
5. The ops guide has no file:line, class or function names, or SQL.
6. `HANDOVER-DEV.md` accounts for all of F1-F12.
7. Report to the user: sections filled, number of exceptions listed, failure classes covered vs not applicable, and the gaps for the dev team.
skills/rpa-bot-handoff/templates/HANDOVER-DEV.md
+19
View File
@@ -0,0 +1,19 @@
# {{BOT_NAME}}: dev notes for the handover
Version {{VERSION}} · Written {{DATE}} · The ops guide is `HANDOVER.md`. This file is for developers only.
## Failure-class check
{{ONE LINE PER CLASS F1-F12: "F<n> covered in HANDOVER.md section <s> / E<nn>", "F<n> gap <#> below", or "F<n> not applicable: <reason>"}}
## Gaps to fix before handover
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | {{GAP}} | {{IMPACT}} | {{REF}} |
## Incidents
| When / version | What ops saw | Cause | Fix | Ref |
|---|---|---|---|---|
| {{DATE / VERSION}} | {{SYMPTOM}} | {{CAUSE}} | {{FIX}} | {{COMMIT}} |
skills/rpa-bot-handoff/templates/HANDOVER.md
+92
View File
@@ -0,0 +1,92 @@
# {{BOT_NAME}}: ops guide
Version {{VERSION}} · Written {{DATE}} · Questions or problems: {{ESCALATION_CHANNEL}} (see section 9)
## 1. What this bot does
{{AT_MOST_2_LINES: what it does, in which app, from which input}}
## 2. Where is what
| Where | What you see there | Open it when |
|---|---|---|
| {{TAB / SCREEN / FILE / FOLDER}} | {{WHAT IT SHOWS}} | {{WHEN}} |
## 3. How to run
**Before you start**
- [ ] {{PREREQUISITE: e.g. the target app is installed and you can sign in}}
- [ ] {{INPUT RULE: e.g. one clinic per row. Wrong: "North, South". Right: "North" and "South" on two rows.}}
- [ ] The app shows version {{VERSION}}.
**Steps**
1. {{e.g. Open "<app name>" from the Desktop.}}
2. {{e.g. On the Run tab, drop the input Excel on the upload box.}}
3. {{e.g. Check the preview: every sheet has rows.}}
4. {{e.g. Click "Trigger automation".}}
**What a normal run looks like:** {{e.g. the 7 checkpoint boxes turn green one by one; the live log shows each item}}
## 4. How long to wait
| Step | Normal time | The bot waits at most | After that the bot | What you do |
|---|---|---|---|---|
| {{STEP}} | {{e.g. 1 min, or "not measured"}} | {{e.g. 5 min}} | {{e.g. restarts the app, up to 3 times}} | {{e.g. nothing, wait}} |
**When do you step in?** {{e.g. only when the run ends red, or a box stays red after 3 restarts}}
## 5. After the run
| Ending | Meaning | What you do |
|---|---|---|
| {{e.g. green "Completed"}} | {{MEANING}} | {{ACTION}} |
**The report:** {{e.g. Report tab → choose the batch → File Status column. Export with the Excel button.}}
| Status | Colour | Meaning | What you do |
|---|---|---|---|
| {{STATUS}} | {{COLOUR}} | {{MEANING}} | {{ACTION or "nothing"}} |
## 6. How to see an exception
1. {{e.g. Report tab → filter File Status = Error.}}
2. {{e.g. Read the Comments column.}}
3. Find that message in section 7.
## 7. Exception list
| ID | Message you see | Where | What it means | What you do | Bot retries / call us when |
|---|---|---|---|---|---|
| E01 | {{EXACT MESSAGE}} | {{TAB → COLUMN, or FILE}} | {{ONE SENTENCE}} | {{STEPS}} | {{e.g. "Bot retries 3 times, wait. Call us if it fails again."}} |
## 8. How you will know
{{e.g. "There is no email or alert. Check the Dashboard after every run, and every morning for the night run."}}
## 9. How to tell us
- **Where:** {{ESCALATION_CHANNEL}}. Expect an answer within {{RESPONSE_TIME}}.
- **Send:** {{e.g. batch ID, time of the problem, E-number from section 7, a screenshot of the screen, the report row}}
- **Never send:** passwords, the settings file, or patient or customer data in chat.
Copy this message:
```
Bot: {{BOT_NAME}} v{{VERSION}}
Batch / run ID:
Time:
E-number and message:
What I already did:
Screenshot: attached
```
## 10. Don'ts
- {{DON'T: e.g. "Don't use the mouse or keyboard while it runs. The bot's clicks land in the wrong window."}}
- **To stop the bot:** {{THE ONE RIGHT WAY}}
## 11. Issues seen before
| When / version | What you saw | What to do now |
|---|---|---|
| {{DATE / VERSION}} | {{SYMPTOM}} | {{ACTION, or "fixed in vX, nothing to do"}} |
skills/rpa-bot-handoff/templates/failure-classes.md
+63
View File
@@ -0,0 +1,63 @@
# Failure classes
Twelve kinds of problem Tebra Auto Post hit in its first months with ops. Check every new bot against each one. For each class: what to look for in the code, and where the ops guide (`HANDOVER.md`) or the dev notes (`HANDOVER-DEV.md`) must cover it. Section numbers are the ops guide's.
## F1. Missing prerequisite on the ops machine
**Tebra:** a missing database driver gave a cryptic error only on some laptops; a missing settings file made the duplicate check fail silently; a helper app's shortcut wasn't where the bot searched; some bugs appeared only in the packaged exe, never under `uv run`.
**Look for:** drivers, `.env` / settings files, shortcuts or install paths the bot searches, `sys.frozen` branches.
**Doc:** the "Before you start" checklist in section 3, and the exact message each missing item produces in section 7.
## F2. The bot grabs the wrong window, or can't take focus
**Tebra:** it matched its own window, and a database tool whose title held the target app's name, then closed the real app. Windows' foreground lock meant clicks landed elsewhere when someone used the machine during a run.
**Look for:** window matching by title substring, focus / foreground calls, exclude lists.
**Doc:** section 10: which windows must not be open, and hands off the machine.
## F3. The target app is slower, or different, than the bot expects
**Tebra:** a screen took about 98 s under load against a 90 s timeout; two screens shared a header and were confused; controls drawn without accessibility elements; a long list put the target off-screen; a wrapped name was clicked between its lines on a small screen.
**Look for:** every timeout value, screen-identification checks, "element not found" paths, screen-size assumptions.
**Doc:** a section 7 row for each "not found / did not load" message, and the wait from section 4. Screen-resolution rules go in section 3.
## F4. Input data the bot reads differently than ops writes it
**Tebra:** names containing commas were split into pieces and silently skipped; a `&` in a name broke matching; Excel dates arrived as serial numbers; a new client with no mapping entry got the wrong file prefix.
**Look for:** split / separator logic, name matching, date parsing, mapping or config files ops must maintain.
**Doc:** input rules in section 3, each with one wrong and one right example. Point at the sample input file if the bot has one.
## F5. Marked done when nothing was done
**Tebra:** a location was marked Success with 0 items when the bot couldn't confirm it was on the right page; a company was marked Success after a crash.
**Look for:** where success / complete is set, and whether it checks a positive count or a confirmation first.
**Doc:** if a guard exists, a section 7 row for the "left Pending" message. If not, a gap in `HANDOVER-DEV.md`.
## F6. The action happened but the bot couldn't confirm it
**Tebra:** "Payment not confirmed after Save" - the save took longer than the wait, so the row showed Error while the payment really had posted. Re-running risked posting twice.
**Look for:** waits after a commit action (save, submit, send), and what the bot writes when the confirmation times out.
**Doc:** a section 7 row that tells ops, in bold, to **check in the target app before re-running**, and how to correct the record afterwards.
## F7. A business exception shown as success
**Tebra:** "ERA Errors on posting" and "negative balance - not posted" rows showed green (Completed) although a person had to follow up.
**Look for:** status mapping tables; any comment that means "not done" stored under a success status.
**Doc:** mark these "not done, needs follow-up" in the section 5 status table, give each a section 7 row, and add the mapping to the gaps in `HANDOVER-DEV.md`.
## F8. Endless re-runs, or killed by the watchdog
**Tebra:** practices with no items re-ran the same company forever; already-processed rows were re-walked slowly after a restart until the watchdog killed the run at exactly 300 s, again and again.
**Look for:** auto-rerun loops and their limits, watchdog timeouts, "0 items" handling, heartbeats.
**Doc:** every limit in section 4, and in section 7 the signs of a loop ops should report ("the same company restarts more than N times").
## F9. Stopping the bot the wrong way
**Tebra:** closing the app window didn't stop the bot, which kept driving the target app; closing the target app doesn't stop it either.
**Look for:** how the child process ends (process-tree kill, parent-death watch), and any Stop button.
**Doc:** the one right way to stop it, in section 10.
## F10. One stuck item blocks the queue
**Tebra:** one Error row kept a whole batch Pending for weeks; a run that died in its first seconds locked the upload all day. Ops didn't know which row blocked it, or that yesterday's Pending batch never blocks today's upload.
**Look for:** completion gates (what must be true for a batch / company to complete), locks on new uploads, resume logic.
**Doc:** in section 7: what blocks what, how to find the blocking item (section 6), and how to release it (edit, discard, clear).
## F11. The cause is hidden from ops
**Tebra:** the real cause was only in a database column the Report tab didn't show, or in the raw log; the friendly on-screen log hid errors; some app messages went to the screen only, never to a file.
**Look for:** columns written but never displayed, log filters, messages printed only to the UI.
**Doc:** sections 2 and 6 point only at places ops can open (question 2). Every hidden cause becomes a gap in `HANDOVER-DEV.md`.
## F12. The update didn't arrive, or an old build undid a new one
**Tebra:** the launcher failed silently on a self-signed certificate, so users stayed on an old version; the server reported a deploy as done without swapping the container; a stale zip was shipped; each start re-deployed database procedures, so a machine on an old build overwrote newer ones.
**Look for:** update / launcher code, version display, anything the app writes to shared resources on start.
**Doc:** section 3 (check the version before a run) and section 11. Add "every machine must run the same version" when shared resources are rewritten on start.
skills/rpa-bot-handoff/templates/reference/tebra-auto-post.md
+171
View File
@@ -0,0 +1,171 @@
# Worked example: Tebra Auto Post (v2.5.3), scrubbed
What the two files look like for a real bot. It is shortened: the real guide lists every status and every message. Client names, IDs, amounts, credentials and server addresses are removed, and the channel name is an example. **Copy the depth and the plain wording, not the content.**
---
# File 1: `HANDOVER.md`
# Tebra Auto Post: ops guide
Version 2.5.3 · Questions or problems: #rpa-support Teams channel (see section 9)
## 1. What this bot does
Posts insurance ERA payments into the Tebra desktop app, company by company and practice by practice, from the input Excel. At the end it turns each downloaded ERA into a PDF with EasyPrint.
## 2. Where is what
| Where | What you see there | Open it when |
|---|---|---|
| **Run tab** | Upload box, sheet preview, **Trigger automation** button, live log, 7 checkpoint boxes | To start a run, and to watch it |
| **Dashboard** | Every batch with its status (Pending / Completed / Cleared). Click a batch to see each company and practice | After a run, to see if it finished |
| **Report tab** | One row per ERA: File Status (colour), Comments, Check #, Insurance. Search by Batch ID. Exports to Excel | To see which ERA failed and why |
| `data/files/Need_attention/` | ERAs whose PDF could not be made | When PDF Converted = No |
| `data/files/Xcel/ERA_Summary_<time>.xlsx` | Per-file PDF result: Status, Failure Reason | To see why a PDF failed |
## 3. How to run
**Before you start**
- [ ] EasyPrint shortcut is on the Desktop, the Public Desktop or the Start Menu.
- [ ] Input Excel has the sheets Login, Company, Location, Settings. Download the sample from the Run tab if unsure.
- [ ] **One practice per row.** Wrong: `North Clinic, South Clinic` in one cell (the bot splits it and skips both). Right: each on its own row.
- [ ] Practice names spelled exactly as in Tebra, including `&`.
- [ ] A new practice has a Client ID. Ask us to add it before the first run.
- [ ] The title bar shows v2.5.3.
**Steps**
1. Open **Ecare Tebra Automation Tool** from the Desktop.
2. Run tab → drop the input Excel on the upload box.
3. Check the preview: Login, Company and Location sheets all have rows.
4. Click **Trigger automation**.
5. Leave the machine alone until the run ends.
**What a normal run looks like:** the checkpoint boxes turn green one by one. The live log shows each company, practice and ERA. Tebra opens and closes by itself for every company.
## 4. How long to wait
| Step | Normal time | The bot waits at most | After that the bot | What you do |
|---|---|---|---|---|
| Any checkpoint box | — | 5 min (10 min for the ERA list) with no progress | Marks the box red **stuck**, closes Tebra, restarts. Up to 3 times | Nothing. Wait for the 3 restarts |
| Open one ERA | not measured | 90 s, 3 tries, then 3 Tebra restarts | Writes the row as Error, moves to the next ERA | Nothing during the run |
| Open the payment screen | not measured | 120 s, 4 tries, then 3 Tebra restarts | Pauses the run | See E07 |
| Save one payment | about 12 s (slowest seen 27 s) | 45 s + 30 s + 30 s | Checks the ERA list. If the ERA is gone, counts it as posted. If not, Error | See E05 |
| One company | depends on ERAs | 3 attempts | Leaves the company Pending | See E08 |
| PDF step | not measured | 60 min | Puts the file in `Need_attention/` | See E09 |
**When do you step in?** Only when the run has ended, or the log says "Stuck - automatic restarts exhausted. Click Resume.". Restarts before that are normal.
## 5. After the run
| Ending | Meaning | What you do |
|---|---|---|
| Batch **Completed** on the Dashboard | Every practice finished | Check the Report tab for amber/grey rows that need follow-up |
| Batch **Pending** | At least one practice is not finished | Section 6, then E08 |
| "Stuck - automatic restarts exhausted. Click Resume." | The bot gave up after 3 restarts | Click **Resume** once. If it stops again, tell us |
**The report:** Report tab → search the Batch ID → read File Status and Comments. Export it to Excel from the same tab.
| Status - Comments | Colour | Meaning | What you do |
|---|---|---|---|
| Completed - "Posted successfully" | green | Posted and saved | nothing |
| Completed - "ERA Errors on posting" | green | **Not posted.** Tebra rejected it | Review it in Tebra by hand |
| Completed - "negative balance account … not posted" | green | **Not posted on purpose** | Follow the negative-balance process |
| Completed - "Payment method '…' - not posted" | green | Not ACH, skipped | nothing |
| Deferred | grey | Check date after the cutoff or in the future | nothing, a later run picks it up |
| Error | red | Failed after all retries | Section 7 |
**Read the Comments column on every green row.** Three different results all show green.
## 6. How to see an exception
1. Report tab → search the Batch ID → sort by File Status → look at the red **Error** rows.
2. Read the **Comments** column.
3. Find that message in section 7.
4. If no row is red but the batch is Pending, open the batch on the Dashboard and find the practice that is still Pending.
## 7. Exception list
| ID | Message you see | Where | What it means | What you do | Bot retries / call us when |
|---|---|---|---|---|---|
| E01 | "[login] FAILED for company …" | Live log; company Pending on the Dashboard | Tebra's sign-in window or fields never appeared | Check the username and password in the Login sheet, then Resume | Bot tries the company 3 times. Call us if Resume fails too |
| E02 | "Wrong practice opened … leaving Pending" | Live log; practice Pending | The practice Tebra opened has a different name from the input | Fix the spelling in the input to match Tebra exactly (`&`, commas). Resume | Call us if the spelling is already exact |
| E03 | "Clearinghouse Reports not found after retries" | Live log; practice Pending | Tebra's menu didn't show up, even after restarts | Resume once | Call us if it happens again |
| E04 | "Could not open the ERA to view it" | Report tab, Comments, red | That ERA wouldn't open after 3 tries and 3 restarts | Resume. The bot continues at that ERA | Call us if the same ERA fails again |
| E05 | "Payment not confirmed after Save" | Report tab, Comments, red | The bot clicked Save but never saw "saved". **It may have posted** | 1. **Check the payment in Tebra before anything else.** 2. If it posted, edit the row in the Report tab to Completed. 3. Only if it did not post, Resume. **Re-running without checking can post it twice** | Call us if it is not posted and fails again |
| E06 | "Save button element not found" | Report tab, Comments, red | The payment screen had no Save button | Nothing. Tell us | Always call us |
| E07 | "… still failing after 3 Tebra restarts" | Live log; run paused | The payment screen won't open | Click Resume once | Call us if it pauses again |
| E08 | Batch stays **Pending** | Dashboard | One red or unfinished row keeps its practice open | Find the row (section 6). Fix or edit it, or request a Clear (an admin approves it). Yesterday's Pending batch does **not** block today's upload | Call us if you can't find the row |
| E09 | File in `Need_attention/` | `ERA_Summary_<time>.xlsx`, Failure Reason | EasyPrint couldn't make the PDF | Nothing. The next run tries again | Call us if it is still there after 2 runs |
## 8. How you will know
There is **no email or alert**. The bot only shows results on screen. Check the Dashboard after every run: a batch that is still Pending needs you.
## 9. How to tell us
- **Where:** #rpa-support Teams channel. Expect an answer the same working day.
- **Send:** Batch ID, time of the problem, E-number, a screenshot of the Run tab and of Tebra, the Report row.
- **Never send:** the Login sheet, passwords, the settings file, or patient names in chat.
```
Bot: Tebra Auto Post v2.5.3
Batch ID:
Time:
E-number and message:
What I already did:
Screenshot: attached
```
## 10. Don'ts
- **Don't use the mouse or keyboard while it runs.** Windows blocks the bot from taking focus, and its clicks land in the wrong window.
- **Don't keep SQL Server Management Studio, or any window with "Tebra" in its title, open.** The bot can grab it instead of Tebra.
- **Don't close Tebra to stop the bot.** The bot keeps going.
- **To stop the bot:** close the **Ecare Tebra Automation Tool** window.
## 11. Issues seen before
| When / version | What you saw | What to do now |
|---|---|---|
| v2.5.1 | Red "Payment not confirmed" rows that had really posted | Partly fixed. Always check in Tebra first (E05) |
| v2.5.0-2.5.3 | Batches Pending for weeks | Request a Clear (E08) |
| v2.5.1 | Stayed on 2.5.0 after an update | Check the title bar version after every update. Tell us if it didn't change |
| v2.5.2 | Some practices skipped on every run | One practice per row, no commas |
| v1.0.x | Bot drove the database tool, then closed Tebra | Close other "Tebra" windows (section 10) |
---
# File 2: `HANDOVER-DEV.md`
## Failure-class check (excerpt)
- F1 covered in HANDOVER.md section 3 (EasyPrint shortcut); gap 6 (ODBC driver not enforced)
- F2 covered in section 10
- F4 covered in section 3 (one practice per row); gap 5 (`&` in names)
- F6 covered in E05
- F7 covered in section 5 (three green results); gap 1
- F9 covered in section 10
- F10 covered in E08
- F11 gap 2, gap 3
- F12 covered in section 11; gap 4
## Gaps to fix before handover (excerpt)
| # | Gap | Why it hurts ops | Where |
|---|---|---|---|
| 1 | "ERA Errors" and "negative balance" rows are stored as Completed (green) | Ops thinks they are done | `src/automation/era_log.py` status map |
| 2 | The developer cause (TechComments) is in the DB only, not on the Report tab | Ops must ask dev for every red row | `src/db/report.py` |
| 3 | The app's own watchdog messages go to the screen only | Lost after the run | `src/ui/main_window.py` `_log()` |
| 4 | `ERA_Summary_*.xlsx` is named by time only | Can't be tied to a batch | `src/automation/remittance_pdf.py` |
| 5 | `&` in a practice name breaks the title match (fixed in the TVL fork only) | Practice abandoned after 13 tries | `src/automation/orchestrator.py` practice check |
| 6 | No check for ODBC Driver 17/18 at start | Cryptic "HYC00" error mid-run | `src/db/database.py` |
## Incidents (excerpt)
| When / version | What ops saw | Cause | Fix | Ref |
|---|---|---|---|---|
| v2.5.1 | "Payment not confirmed" rows that had posted | Save slower than the 20 s wait | 45/30/30 s + grid re-check | bug-206, bug-234 |
| v2.5.1 | Users on 2.5.0 for a month | Server marked the deploy done but kept the old container | Stop + Start on the server | bug-243 |
| v2.5.2 | 3 practices skipped every run | Split on commas | Split on newline/semicolon only | bug-233 |
| v1.0.x | Launcher never updated | Self-signed certificate | Retry without verification | bug-080 |
tests/evals/results/handoff/h-api-sync-full-r1.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-api-sync/full",
"rep": 1,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"retry limit": true,
"console-only crash output flagged": true
},
"cost_usd": 1.4335914,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-api-sync-full-hx49oaht\\project",
"reply": "I wrote `HANDOVER.md` in the repo root. I didn't run the bot, and no code was changed. The skill's questions were answered from your message, so I didn't stop to ask.\n\n| Item | Result |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 11 (E01\u2013E11) |\n| Failure classes | 11 covered, 1 not applicable (F2: the bot has no window and drives no app) |\n| Fix before handover | 18 items |\n\nFix before handover-la mukkiyamaanavai:\n- **The `data/output/` folder doesn't exist and the code never creates it.** Fresh machine-la ovvoru run-um first row-ku munnadiye stop aagum.\n- **Crash, network error ellaam console-ku mattum dhaan pogudhu**, log-ku illa. Failed row-la reason (expired key, billing down, rate limit) record aagala.\n- **Billing sila lines reject panna invoice-um \"Synced\"-nu dhaan kaattudhu.** \"Created (confirmed after retry)\" create aagadha invoice-kum varalaam.\n- **Billing-la irundhu oru unreadable reject vandha, illa duplicate check-la network fail aanaa, whole run crash aagum.** Andha invoice input file-la irukkira varai ovvoru hour-um ippadi dhaan.\n- **Bot version-a engeyum kaattala.** Andha 2 \"fix\" commits-la oru comment line mattum dhaan add aagirukku, so andha fixes git history-la trace panna mudiyala.\n- **Report-la customer names copy aagudhu.** Adhanaala section 11-la \"file name and invoice numbers mattum anuppunga\" nu ezhudhiyirukken.\n- Meedhi: missing key/URL silent-a pogudhu, retries pointless-a nadakkudhu, runs overlap aagalaam, BOM issue, `nan` amount validation-a pass aagudhu, Task Scheduler access, tests illa.\n\nStopping and run history-ku Task Scheduler venum. Ops-ku adhu access irukka-nu confirm aagala, so andha part fix list-la potturukken (#13).\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions whose meaning the code doesn't make clear actually mean?",
"stderr": ""
}
tests/evals/results/handoff/h-api-sync-full-r1.md
+188
View File
@@ -0,0 +1,188 @@
# Invoice sync - ops handover
Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every hour, Windows Task Scheduler starts the bot. It reads the finance export file `data/input/invoices.csv`, which has one approved invoice per row. It checks each row, then sends the invoice to the billing system over the internet.
For every run it writes:
- a report `data/output/sync_report_<YYYYMMDD_HHMM>.csv`, with one line per invoice and a Status and Message for each;
- a short line per invoice in `logs/sync.log`.
The bot has no window and drives no desktop app. It talks to the billing system directly, so you can use the machine while it runs.
## 2. How it works
```
Windows Task Scheduler (every hour)
└─ starts → Invoice sync (no window) ──sends invoices──> Billing system (online)
│
├─ reads → data/input/invoices.csv
├─ writes → data/output/sync_report_<YYYYMMDD_HHMM>.csv
└─ writes → logs/sync.log
```
- **Task Scheduler** starts the bot once an hour. Nothing else starts it.
- **Invoice sync** is a background program with no screen. It works through the input file from top to bottom, one invoice at a time.
- **Billing system** receives each invoice. It can accept it, reject it, say it already has it, or not answer.
- **invoices.csv** is the finance export. The bot never moves, renames or empties it, so every hour it sends the whole file again.
- **sync_report_….csv** is the only place where the reason for each result is written.
- **sync.log** keeps the invoice number and status for every run, plus a "run done" line when a run finishes.
**One run, step by step:**
1. Create a new report file named after the current date and time.
2. For each row of the input file:
- Check the row: required fields filled in, and the amount is a number above 0. If the check fails, the row is **Invalid** and the bot moves to the next row.
- Send the invoice to billing.
- If billing does not answer, or answers with a temporary error, check whether billing already has the invoice. If it does, the row is **Synced**. If not, wait and try again, up to 5 tries in total.
- Write the result to the report and the log.
3. Write `run done -> sync_report_<time>.csv` to the log.
## 3. Before a run
- [ ] The bot folder contains these three items. Without them the bot stops at the very start and writes no report.
- The folder `data/output/`. The bot does not create it.
- The folder `logs/`.
- The settings file `.env`.
- [ ] The settings file has both the billing address and the billing key filled in. If the key is missing or expired, every invoice takes about 1 minute and then ends as **Failed** (E08). Nothing on screen says the key is the problem.
- [ ] The input file is at `data/input/invoices.csv`.
- [ ] The first line of the input file is exactly: `invoice_no,customer_id,customer_name,amount,currency`. If a heading is renamed, every row comes back **Invalid - missing …** (E01, E09).
- [ ] One invoice per row. Each row has an invoice number, a customer ID, an amount and a currency.
- [ ] Amounts are plain numbers above 0.
- Good: `1200.50`
- Bad: `1,200.50` or `$1200` (Invalid - "is not a number"), `0` or `-50` (Invalid - "must be positive").
- [ ] Only **one** machine runs this bot. Two machines would send the same invoices twice every hour.
- [ ] The bot does not show its version anywhere. Confirm with dev that 0.9.1 is installed (section 10).
## 4. During a run
- You can use the machine. The bot has no window and clicks nothing.
- Don't replace or edit `invoices.csv` while a run is in progress. Wait until the log shows the `run done` line for that hour.
- You can open a report file to read it. Don't save changes to the report of a run that is still in progress.
- To stop the bot: in Windows Task Scheduler, right-click the invoice sync task and choose **End**. To stop the hourly runs, choose **Disable**. Ops access to Task Scheduler is not confirmed, so ask #rpa-support if you can't do it.
- If you stop it mid-run, the report for that run is incomplete, and the invoice it was sending at that moment may or may not have reached billing. The next run reports that invoice as **Duplicate** if it did.
- A run looks stuck when `logs/sync.log` gets no new line for more than **6 minutes**. One invoice can take up to about 6 minutes when billing does not answer at all.
- There is no watchdog: the bot never gives up on a run by itself.
- If many invoices fail, one run can take longer than an hour and overlap the next hourly run (E11).
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `logs/sync.log`. Look for `run done -> sync_report_<time>.csv` for that hour. If the line is missing, the run stopped partway (E10). |
| Did the run start at all? | `data/output/`. There should be one `sync_report_<YYYYMMDD_HHMM>.csv` per hour. A missing hour means the bot did not start or stopped at the very start (E10). |
| Which invoices failed? | Open that hour's report. Look at every row where Status is **Invalid** or **Failed**, and every **Synced** row whose Message is "Created (some lines rejected)". |
| Why did one invoice fail? | The **Message** column of the report. The log only shows the status, never the reason. |
| What happened to one invoice over several runs? | Search `logs/sync.log` for the invoice number. Each hit gives the run time, which points you to the matching report file. |
| Did the bot already retry? | Not recorded directly. "No success after 5 tries" means it tried 5 times. "Created (confirmed after retry)" means the first try failed and billing turned out to have the invoice. Other results came from the first try. |
| Why did the whole run stop? | Not visible to ops. The reason goes only to a console window nobody sees. Send it to dev (E10). |
| Which version is running? | Not shown anywhere ops can see. Ask dev (section 10). |
## 6. Statuses
The report is a plain CSV file, so statuses have no colour. Read the **Status** and **Message** columns together.
| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | Billing accepted the invoice | nothing |
| Synced | Created (some lines rejected) | **Needs follow-up.** Billing created the invoice but dropped some of its lines. | Check the invoice in billing and fix the missing lines (E06) |
| Synced | Created (confirmed after retry) | The first send failed or got no answer. Billing turned out to have the invoice, so the bot did not send it again. | Usually nothing. Spot-check the amount in billing (E07). |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number | Normally nothing: the same file is re-sent every hour (E05) |
| Invalid | missing `<field names>` | A required field is empty in the input | Fix the input row (E01) |
| Invalid | amount must be positive | Amount is 0 or below | Fix the input row (E02) |
| Invalid | amount '`<value>`' is not a number | The amount has commas, a currency sign, text, etc. | Fix the input row (E03) |
| Invalid | billing's own explanation, or "Rejected by billing" | Billing refused the invoice and gave a reason | Act on the reason (E04) |
| Failed | No success after 5 tries | Billing never accepted it in 5 tries, and the bot could not find the invoice in billing | Check billing first, then see E08 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing invoice_no" (or customer_id, amount, currency) | Before sending, it checks that these four fields are not empty | Report: Status and Message. Log: status only. | Business | Ops | Fill in the field in the finance export. The next hourly run picks the row up. | The field is filled in but still reported missing (see E09) |
| E02 | Invalid - "amount must be positive" | The amount is 0 or below | Report | Business | Ops | Correct the amount, or remove the row if it is a credit note. The bot does not send zero or negative invoices. | Credit notes need to go through. That is a change request to dev. |
| E03 | Invalid - "amount '1,200.50' is not a number" | It cannot read the amount as a number | Report | Business | Ops | Write the amount as a plain number with a dot for decimals, e.g. `1200.50`. No thousands separators, currency signs or spaces. | The amount already looks plain |
| E04 | Invalid - billing's message (e.g. unknown customer), or "Rejected by billing" | Billing refuses the invoice and explains why | Report: Message holds billing's reason | Business | Ops | Fix the data that billing's message names, usually in the finance export or in billing's customer records. | The message is "Rejected by billing" with no reason, or you can't tell what to fix. Send the invoice number, report file name and message. |
| E05 | Duplicate - "Invoice already exists in billing" | Billing says it already has this invoice number | Report | Business | Ops | Check earlier reports (search the log for the invoice number). If an earlier run shows Synced, this is normal: the bot re-sends the whole file every hour. If no run ever synced it, check in billing that the existing invoice is the right one, because the number may be used twice. | Two different invoices share a number |
| E06 | Synced - "Created (some lines rejected)" | Billing creates the invoice but rejects part of it | Report: Message. The Status still says Synced. | Business | Ops | Open the invoice in billing, find the missing lines and add or correct them by hand. The bot will not resend them: re-runs only get Duplicate. | Lines are rejected for many invoices in the same run |
| E07 | Synced - "Created (confirmed after retry)" | The first send failed or got no answer. Before retrying, it asked billing whether the invoice exists and got yes. | Report | System | Bot | Nothing usually. The bot cannot tell whether it created the invoice this run or the invoice already existed, so spot-check the amount in billing now and then. | The invoice in billing has the wrong amount or customer |
| E08 | Failed - "No success after 5 tries" | 5 sends, with waits of 2, 4, 8, 16 and 32 s between them. After each failed send it checks whether billing already has the invoice. | Report. The real cause (billing down, busy, key expired, wrong address) is not recorded anywhere. | System | Ops, then Dev | 1. **Search billing for the invoice number first.** It may have been created even though no confirmation came back. 2. If it isn't there, do nothing: the next hourly run tries again while the row is still in the input file. 3. If it is there, nothing more is needed. The next run will show Duplicate. | **Every** row in a run is Failed (likely the billing key or address). The same invoice is Failed in 2 runs in a row. Send the report file name, invoice numbers and run time. |
| E09 | **Every** row Invalid - "missing invoice_no", although the file looks fine | The first heading isn't read as `invoice_no`. Some Excel "CSV UTF-8" saves add an invisible mark before the first heading. | Report | Business | Ops, then Dev | Check the first line spelling exactly (section 3). If it is exact, re-export the file once. | It still happens. Send the file name and time, not the file itself. |
| E10 | No report for an hour, a report with fewer rows than the input, or no `run done` line in the log | The bot stops the whole run. Known causes: `data/output/` or `logs/` is missing, the input file is missing, billing sends back an unreadable rejection, or billing can't be reached while the bot checks for a duplicate. | Nothing ops can see. The reason goes only to a console window. The report stops at the last finished invoice. | System | Ops, then Dev | 1. Check that `data/output/`, `logs/` and `data/input/invoices.csv` exist. 2. Wait for the next hourly run. It starts again from the top, and already-sent invoices come back Duplicate. | The next run stops too, especially if it stops right after the same invoice. Send the run times, the last invoice number in the partial report, and `logs/sync.log`. |
| E11 | A run takes over an hour, two runs' lines are mixed in the log, or a report is overwritten | Many Failed invoices, each taking 1-6 minutes. The bot has no lock against two runs at once. | `logs/sync.log` (timestamps); `data/output/` | System | Dev | Treat it as E08 for the invoices. Tell dev runs are overlapping. | Always. Send the log and the times. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing doesn't answer, is busy, or returns a temporary error | Checks whether billing already has the invoice. If not, waits and sends again. | 5 tries, waits of 2+4+8+16+32 s. About 1 minute per invoice; up to about 6 minutes if billing doesn't answer at all (30 s wait per call). | Status Failed, moves to the next invoice (E08) |
| Billing created the invoice but the confirmation got lost | The existence check above finds it and does not send it again | Same as above | Synced - "Created (confirmed after retry)" (E07) |
| An invoice was not synced this hour | The next hourly run sends it again, because the input file is never emptied | As long as the row stays in the finance export | If finance removes the row, nobody retries it |
| The whole run stopped | No recovery inside the run | - | The next hourly run starts again from row 1 (E10) |
| Invalid input row | No retry | - | Stays Invalid every hour until the input is fixed |
## 9. Known incidents
The history records no dates or versions for these incidents. Both were fixed before 0.9.1.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| before 0.9.1 | The same invoice appeared twice in billing | Billing timed out but had created the invoice; the bot sent it again | Fixed: the bot now checks billing before every retry. Still **search billing before re-processing any Failed invoice** (E08). Duplicates are less likely but still possible when billing is slow to show a new invoice. | Partly |
| before 0.9.1 | The hourly run stopped partway, report incomplete | Billing's "too many requests" limit was not handled | Fixed: up to 5 tries with waits. If a run still stops, follow E10. | Partly |
## 10. Releases and version check
- How an update arrives: there is no launcher or auto-update. A developer replaces the bot files in the bot folder by hand.
- How to confirm it applied: the bot does not show or log its version. Ask in #rpa-support for the version installed after each update. The current version is 0.9.1.
- If the version did not change: report it in #rpa-support. Don't copy files yourself.
- Only one machine may run the bot (section 3).
## 11. Escalating to dev
Post in **#rpa-support** with:
- the exception ID from section 7;
- the invoice number(s);
- the report file name, e.g. `sync_report_20261023_1400.csv`, and the run time;
- `logs/sync.log`, which holds invoice numbers and statuses only.
Never send:
- the `.env` settings file or the billing key;
- rows copied out of the report or the input file. They contain customer names, so give the file name and invoice numbers instead.
## 12. Failure-class check
- F1 (missing prerequisite) covered in section 3 and E10. The missing output folder is also Fix #1.
- F2 not applicable: the bot has no window and drives no desktop app. It only sends data to billing online.
- F3 (target slower than expected) covered in E08 and section 8: 30 s wait per call, 5 tries. Fix #9 covers errors that are retried pointlessly.
- F4 (input read differently) covered in section 3 input rules and E01-E03, E09. Also Fix #12 and #15.
- F5 (marked done when nothing was done) covered in E07. "Confirmed after retry" can't tell whether this run created the invoice. Fix #6.
- F6 (action happened but not confirmed) covered in E08: search billing before re-processing. Incident 1. Fix #7.
- F7 (business exception shown as success) covered in section 6 and E06: partially rejected invoices show Synced. Fix #5.
- F8 (endless re-runs / watchdog) covered in section 4 (no watchdog), section 8, E05 (hourly Duplicates) and E11 (overlapping runs). Fix #10.
- F9 (stopping the wrong way) covered in section 4: End the task in Task Scheduler. Fix #13 covers the unconfirmed ops access.
- F10 (one stuck item blocks the queue) covered in E10. One bad invoice can stop the whole run and, if it stays in the file, every hourly run after it. Fix #4 and #7.
- F11 (cause hidden from ops) covered in section 5 and E08, E10. Fix #2 and #3.
- F12 (update didn't arrive) covered in section 10. Fix #11.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | `data/output/` is not in the repo and is never created. `open(report, "w")` raises FileNotFoundError. | On a fresh machine every run stops before row 1, and no report or log line is written | `sync.py:68-69` |
| 2 | Network errors and crashes go to `print()` (stdout) only, never to `sync.log` | Ops can't see why a run stopped (E10). Task Scheduler discards stdout. | `sync.py:52`, `sync.py:84` |
| 3 | Failed message is fixed text, and neither the last HTTP code nor the error is kept. The log line is status only. | An expired key, a billing outage and a rate limit all look the same (E08) | `sync.py:63`, `sync.py:75` |
| 4 | A non-JSON 422 body makes `r.json()` raise, which crashes the run | One bad invoice stops the whole run every hour while it stays in the file | `sync.py:59` |
| 5 | HTTP 207 (partial) is stored as Status `Synced` | Ops filtering on Status miss rejected lines (F7) | `sync.py:54-55` |
| 6 | `exists()` returning 200 is reported as "Created (confirmed after retry)" even if the invoice existed before this run, e.g. after a 400 | Report claims a creation that may not have happened (F5) | `sync.py:60-61` |
| 7 | `exists()` has no try/except, and any non-200 counts as "not found" | A network outage crashes the run instead of retrying. A 429/5xx on the check leads to a re-POST, so duplicates are still possible (F6). | `sync.py:41-43`, `sync.py:60` |
| 8 | Missing `BILLING_API_KEY` sends an empty Bearer token silently. Missing `BILLING_API_URL` silently falls back to `billing.example.test`. | Bad settings look like ordinary Failed rows, about 1 minute each | `sync.py:18-19` |
| 9 | 400/401/403/404 are retried 5 times, `Retry-After` is ignored, and the bot sleeps 32 s after the last attempt | Slow runs, wasted rate-limit budget, overlapping hourly runs (E11) | `sync.py:60-62` |
| 10 | Input is never archived. There is no lock against overlapping runs. Report name has minute precision and is opened with `"w"`. | Duplicate noise every hour hides real duplicates. Overlapping runs can overwrite a report (E11). | `sync.py:68`, `sync.py:72` |
| 11 | Version exists only in README. It is not logged or shown, and there is a stray line "chore: bump to 0.9.1" in the README. | Ops can't confirm an update applied (F12) | `README.md:5-6` |
| 12 | Input opened as `utf-8`, not `utf-8-sig` | An Excel BOM makes every row "missing invoice_no" (E09) | `sync.py:69` |
| 13 | The Task Scheduler task definition is not in the repo, and ops access to Task Scheduler is not confirmed | Ops can't stop the bot or see run history (F9) | n/a: add the task export and agree access |
| 14 | Report copies `customer_name` into the output folder ops open | Customer personal data is spread to every report file | `sync.py:71`, `sync.py:76` |
| 15 | `nan` / `inf` amounts pass validation | Bad amounts reach billing | `sync.py:33-37` |
| 16 | Retries are not logged | Ops can't tell whether, or how often, the bot retried | `sync.py:48-62` |
| 17 | The two "fix" commits only add comment lines. The retry and check-before-retry code came with the initial import, and there are no regression tests. | Fixes can't be traced or protected from regressing | `sync.py:86-87` |
| 18 | A `BOT_WAS_RUN` marker is written to the project root every run and never read | Clutter. It shows whether the bot ran, not whether it finished. | `sync.py:67` |
tests/evals/results/handoff/h-api-sync-full-r2.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-api-sync/full",
"rep": 2,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"retry limit": true,
"console-only crash output flagged": true
},
"cost_usd": 1.401677,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-api-sync-full-k38td_tl\\project",
"reply": "`HANDOVER.md` ready, repo root-la irukku. Bot-a run pannala, code-um maathala. Neenga sonna answers-a apdiye use pannen: escalation #rpa-support, ops-ku DB access illa, clear illatha statuses-ku code-oda best reading.\n\n| Item | Result |\n|---|---|\n| Sections | 13 (template order apdiye) |\n| Exceptions catalogued | 13 (E01\u2013E13) |\n| Failure classes | 11 covered, 1 not applicable (F2: bot entha window-um drive pannala, API mattum dhaan) |\n| Fix before handover | 21 items |\n\n**Fix-before-handover-la mukkiyamaanavai:**\n- Crash reason, network errors console-ku mattum dhaan pogudhu, log-ku illa. **Failed** aana invoice-ku reason engayum record aagala.\n- `data/output` folder repo-la illa, code-um create pannala. Fresh deploy-la first run-ae stop aagidum.\n- Oru bad row (or existence check timeout) full run-a stop pannidum. Andha row input-la irundha, ovvoru mani run-um adhae row-la stop aagum.\n- 2 \"fix\" commits-um oru comment line mattum dhaan add pannirukku. Retry code initial import-laye irundhadhu.\n- 207 (sila lines reject aanadhu) **Synced**-nu kaattudhu. Ellaa invoice-um Failed aanaalum exit code 0, so Task Scheduler success-nu kaattum.\n- Version README-la mattum dhaan irukku, ops-ku theriyaadhu. Report file-la customer names irukku.\n\n**Also noticed:** `data/output` repo-la illa, adhanaala verify step-la \"path exists or the code creates it\" check adhukku fail aagudhu. Adhai doc-la prerequisite-aa podirukken (section 3, fix item 8). Ops-ku Task Scheduler access irukka-nu neenga sollala, so stop procedure-la \"ops-ku access illana #rpa-support-la kelunga\"-nu potten.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-api-sync-full-r2.md
+170
View File
@@ -0,0 +1,170 @@
# Invoice sync - ops handover
Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every hour, Windows Task Scheduler starts the bot. It reads the finance export file `data/input/invoices.csv` (one approved invoice per row), checks each row, and sends each valid invoice to the billing system over its web API. For each run it writes one report file, `data/output/sync_report_<date>_<time>.csv`, with a Status and a Message for every invoice, and adds a few lines to `logs/sync.log`. It has no screen and drives no desktop app. The only system it talks to is the billing system.
## 2. How it works
```
Windows Task Scheduler (every hour)
└─ starts → Invoice sync script ──reads──> data/input/invoices.csv
│
├─ sends each invoice ──> Billing system (web API)
├─ writes → data/output/sync_report_<date>_<time>.csv (one row per invoice)
└─ writes → logs/sync.log (one line per invoice + "run done")
```
- **Task Scheduler** starts the bot on the hour. The bot has no window of its own.
- **Invoice sync script** does all the work, one invoice at a time, top to bottom.
- **Input file** is the finance export. The bot reads the whole file on every run.
- **Billing system** is where invoices are created. The bot never deletes or changes an invoice there.
- **Report file** is the main place to see what happened to each invoice in that run. It contains customer names, so treat it as confidential.
- **Log file** records only the invoice number and its status, plus a "run done" line at the end of each complete run.
**One run, step by step:**
1. Open the input file and create a new report file for this run.
2. For each row: check that the invoice number, customer ID, amount and currency are filled and that the amount is a positive number.
3. If the row passes, send it to billing. If billing doesn't give a clear yes or no, ask billing whether the invoice now exists. If it doesn't, wait and send again, up to 5 tries.
4. Write the result to the report and the log, then go to the next row.
5. After the last row, write `run done -> <report name>` in the log.
Each run handles the whole input file again. Invoices that were already sent come back as **Duplicate**, which is normal.
## 3. Before a run
- [ ] The folder `data/output` exists. **The bot does not create it.** If it is missing, every run stops before the first invoice and writes no report (E11).
- [ ] The folder `logs` exists.
- [ ] The settings file (`.env`) is in the bot folder with the billing address and billing key. The dev team maintains it. Ops never opens or copies it. If the key is missing or expired, every invoice ends up **Failed** (E09).
- [ ] The input file is at `data/input/invoices.csv`, with the header row `invoice_no, customer_id, customer_name, amount, currency`.
- [ ] One invoice per row. Invoice number, customer ID, amount and currency must be filled.
- [ ] Amount is a plain number with a dot for decimals. Good: `1250.00`. Bad: `1,250.00`, `$1250`, `1250 USD` (each becomes **Invalid** E03).
- [ ] Amount is above zero. Credit notes or zero-amount rows become **Invalid** (E02).
- [ ] Version: ops cannot see which version is running (see section 10). Ask the dev team to confirm 0.9.1.
## 4. During a run
- Don't save over or edit the input file while a run is in progress. The bot reads it row by row as it goes.
- Don't delete or move the newest report file until the log shows `run done` for it.
- **To stop the bot:** whoever has access to Task Scheduler on the bot machine selects the invoice sync task and chooses **End**. To stop future hourly runs as well, choose **Disable**. If ops don't have Task Scheduler access, ask in #rpa-support. A stopped run leaves a short report and no `run done` line. Running it again is safe: invoices already created come back as **Duplicate**.
- **A run looks stuck when** the newest report file stops growing for several minutes. When billing is down, **each invoice can take up to about 6 minutes** before it is marked Failed (5 tries with waits). There is no automatic stop or restart. Wait for that. Escalate when a run is still going when the next hourly run is due, or when the report hasn't grown for more than 10 minutes.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `logs/sync.log`: look for `run done -> sync_report_<date>_<time>.csv` for that hour. No such line means the run stopped partway (E10) or never started (E11). |
| Which invoices failed? | The report for that run in `data/output`: Status column is anything other than **Synced**. Also check **Synced** rows whose Message says "some lines rejected" (E06). |
| Why did one invoice fail? | The same report, Message column. For **Failed** rows the real reason is **not recorded anywhere ops can open** (fix item 3). Escalate with the invoice number. |
| When was an invoice last tried? | `logs/sync.log`: search for the invoice number. Each line has the date, time and status. |
| Did the bot already retry? | **Failed** always means 5 tries were made. "Created (confirmed after retry)" means a retry happened and worked. Nothing else records retries. |
| Why did a whole run stop? | Not visible to ops. The reason is only shown on a console nobody sees (fix item 1). Escalate (E10). |
| Which version is running? | Not visible to ops (fix item 15). Ask the dev team. |
## 6. Statuses
The report is a plain CSV file, so there are no colours. The Status column holds one of four values. The Message column explains it.
| Status - Message | Colour | Meaning | What ops does |
|---|---|---|---|
| Synced - "Created" | none (plain text) | Invoice created in billing | Nothing |
| Synced - "Created (confirmed after retry)" | none | Billing didn't answer clearly the first time, but the bot checked and the invoice is there | Nothing |
| Synced - "Created (some lines rejected)" | none | **Needs follow-up.** The invoice was created, but billing refused some of its lines. Shown as Synced anyway. | Check the invoice in billing and add the missing lines by hand (E06) |
| Duplicate - "Invoice already exists in billing" | none | Billing already has this invoice number. Normal for invoices sent in an earlier hour. | Nothing, unless billing shows it twice (E13) |
| Invalid - "missing <field names>" | none | A required field in the input row is empty | Fix the row in the input (E01) |
| Invalid - "amount must be positive" | none | Amount is zero or negative | Check the row with finance (E02) |
| Invalid - "amount '<value>' is not a number" | none | Amount isn't a plain number | Fix the amount format (E03) |
| Invalid - billing's own text, or "Rejected by billing" | none | Billing looked at the invoice and refused it | Read the message and fix the data with finance (E04) |
| Failed - "No success after 5 tries" | none | Billing couldn't be reached or kept failing. Not created. | See E08 / E09 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing amount, currency" (lists the empty fields) | Before sending, it checks that invoice number, customer ID, amount and currency are filled | Report: Status, Message. Log: invoice number + Invalid | Business | Ops | Ask finance to fill the fields in the export. The next hourly run picks the row up. | Never, unless the fields are filled and it still says missing (check the header row spelling) |
| E02 | Invalid - "amount must be positive" | Amount is zero or below | Report; log | Business | Ops | Check with finance. Zero and credit invoices can't go through this bot. | Finance needs credits to sync (change request) |
| E03 | Invalid - "amount '1,250.00' is not a number" | The amount can't be read as a plain number | Report; log | Business | Ops | Remove thousands separators, currency signs and text from the amount | The amount is already plain digits |
| E04 | Invalid - with billing's own explanation, or "Rejected by billing" | Billing replied that the invoice data is wrong | Report; log | Business | Ops | Read the message; fix the customer, currency or amount with finance | The message doesn't make sense, or a correct invoice is rejected again |
| E05 | Duplicate - "Invoice already exists in billing" every hour for the same invoices | Billing replied that the invoice number already exists | Report; log | Business | Ops | Normal for invoices already sent: the export keeps old rows. Only check billing when you don't expect the invoice to exist yet. | The invoice is **not** in billing although the bot says Duplicate |
| E06 | Synced - "Created (some lines rejected)" | Billing created the invoice but said some lines were refused | Report Message column only. The log just says Synced. | Business | Ops | Open the invoice in billing, compare with the export, add the missing lines by hand | Every invoice of a run shows this |
| E07 | Synced - "Created (confirmed after retry)" | Billing timed out or errored, so the bot asked billing whether the invoice now exists, and it did | Report; log says Synced | System | Bot | Nothing. The bot avoided sending it twice. | Never |
| E08 | Failed - "No success after 5 tries" on some invoices | Five tries with no clear answer from billing, and billing doesn't have the invoice. Causes: billing slow/down, too many requests, network trouble. | Report; log. **The real cause isn't recorded.** | System | Bot, then Dev | Nothing at first: if the invoice is still in the input file, the next hourly run tries it again | The same invoice is Failed in 2 runs in a row. Send: invoice number, report file name, the log lines for it. |
| E09 | Failed on **every** invoice of a run | Same as E08, but for all rows. Usually the billing key/address in the settings file is wrong or expired, or billing is down. | Report; log | System | Dev | Don't edit anything. Escalate straight away. | Always. Send: report file name, run time, the `logs/sync.log` lines for that hour. |
| E10 | Report ends early; no `run done` line in the log for that hour; later invoices never appear | An unexpected error ended the whole run, e.g. billing replied in a form the bot can't read, or the network dropped while it was checking whether an invoice exists. The bot does not skip the row; it stops. | Report is cut short; log has no `run done`. Reason only on a console nobody sees. | System | Dev | Note the last invoice in the report; the stop happened on the **next** row of the input. If the same row stops every hourly run, nothing after it is ever synced. | Always. Send: report file name, last invoice number in the report, run time. |
| E11 | No new report and no log lines for an hour | The run stopped before the first invoice: `data/output` folder missing, input file missing or renamed, or the scheduled task didn't start | Nowhere ops can see | System | Ops, then Dev | Check that `data/output` exists (create it if not) and that `data/input/invoices.csv` is there | Both are in place and still no report next hour. Send: the hours with no report. |
| E12 | Two reports a few minutes apart, or one hour's report missing | A slow run (billing down, large file) was still going when the next hourly run started. Nothing stops two runs overlapping. Reports from two runs started in the same minute overwrite each other. | Report file names and times in `data/output`; log lines from two runs mixed | System | Dev | Don't delete anything | Always. Send: the report file names and times. |
| E13 | Billing (or finance) reports the same invoice twice | Billing created it but the reply was lost. The bot checks before re-sending, but if that check also fails it sends again. Usually billing then refuses it as Duplicate, so the report shows **Duplicate** for an invoice this run actually created. | Report shows Synced or Duplicate; nothing marks it | System | Dev | Don't delete in billing yourself unless that is your normal billing process | Always. Send: invoice number, both report files around it. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing times out, is busy (too many requests), errors, or the network fails | Asks billing whether the invoice now exists; if not, waits and sends again. Waits are 2, 4, 8, 16, 32 s. Each request can take up to 30 s. | 5 tries, about 1 to 6 minutes per invoice | Failed - "No success after 5 tries", next invoice |
| Billing created the invoice but its reply was lost | Finds it with the "does it exist" check | Each try | Synced - "Created (confirmed after retry)" |
| An invoice not synced this hour | Nothing during the run. The next hourly run reads the whole input file again. | Every hour, as long as the row stays in the input file | - |
| Run stopped partway (E10, E11) | **Nothing.** No restart until the next hour. | - | If the cause is the same row, every hourly run stops at that row |
There is no watchdog, no stuck detection and no automatic restart.
## 9. Known incidents
Dates aren't recorded. Both were fixed before version 0.9.1.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | The same invoice twice in billing | Billing created it but timed out; the bot sent it again | Fixed: the bot checks before re-sending. Not fully: if the check fails too, it can still re-send (E13). | Partly |
| Before 0.9.1 | The hourly run died partway; later invoices weren't sent | Billing said "too many requests" and the bot stopped | Fixed: now retried 5 times. Not fully: other errors can still stop the run (E10). Compare the report row count with the input; escalate. | Partly |
## 10. Releases and version check
- How an update arrives: the dev team copies the new code into the bot folder on the bot machine. There is no launcher or automatic update.
- How to confirm it applied: **ops can't.** The version is written only in the project's README, not in the log, the report or anywhere ops can open. Ask the dev team in #rpa-support to confirm the version after each update (fix item 15).
- If the version did not change: escalate in #rpa-support.
## 11. Escalating to dev
Post in **#rpa-support** (Teams). Send: the exception ID from section 7, the report file name (it carries the run date and time), the invoice number(s), and the matching lines from `logs/sync.log`.
Never send: the settings file (`.env`), billing keys or passwords, or the report file itself. It contains customer names, so give the file name and the dev team opens it on the machine.
## 12. Failure-class check
- F1 covered in section 3 (output folder, settings file, input path) and E09, E11.
- F2 not applicable: the bot drives no windows; it only talks to the billing web API.
- F3 covered in section 8 (30 s per request, 5 tries) and E07, E08, E12.
- F4 covered in section 3 (amount format, good/bad examples) and E01-E03.
- F5 covered in fix items 9 and 10 (a run where every invoice failed still looks successful to Task Scheduler) and E09.
- F6 covered in E07, E13 and fix item 14.
- F7 covered in section 6 ("some lines rejected" shown as Synced), E06 and fix item 9.
- F8 covered in section 8 (no watchdog, hourly re-read of the whole file), E05, E12 and fix item 11.
- F9 covered in section 4 (End the scheduled task) and fix item 21.
- F10 covered in E10: one bad row stops that run and every later run at the same row. Fix item 7.
- F11 covered in section 5 and fix items 1, 2, 3.
- F12 covered in section 10 and fix item 15.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Network errors and the crash reason go only to the console (`print`), never to `sync.log` | Under Task Scheduler the reason for a stopped run is lost | `sync.py:52`, `sync.py:84` |
| 2 | Log line has only invoice + status, always INFO; no message, no WARNING/ERROR | Ops can't search the log for problems or reasons | `sync.py:75` |
| 3 | Failed rows don't record the last HTTP status or error | Ops can't tell billing-down from bad key from bad data | `sync.py:60-63` |
| 4 | Every non-handled status is retried, including 400/401/403/404 (docstring says only 429/5xx) | A bad key costs ~1 min per invoice and shows only as "Failed" | `sync.py:47`, `sync.py:60-62` |
| 5 | The `exists()` check has no error handling | A timeout during the check kills the whole run (same symptom as the 429 incident) | `sync.py:41-43`, `sync.py:60` |
| 6 | 422 handling calls `r.json()` without a guard | A non-JSON 422 body kills the run | `sync.py:59` |
| 7 | No per-row error handling in the main loop | One bad row stops the run and, if it stays in the input, every later hourly run | `sync.py:72-76` |
| 8 | `data/output` isn't created and isn't in the repo | A fresh deploy fails on every run with no report | `sync.py:68-69` |
| 9 | HTTP 207 (some lines rejected) is reported as Synced | Partly created invoices look done | `sync.py:54-55` |
| 10 | Exit code is 0 even when every invoice Failed | Task Scheduler shows success for a fully failed run | `sync.py:80-85` |
| 11 | No lock against overlapping runs; report name only to the minute | Overlapping runs; reports overwrite each other | `sync.py:68` |
| 12 | Sleeps 32 s after the final attempt too | Wastes time per Failed invoice, more overrun risk | `sync.py:62` |
| 13 | `Retry-After` header from billing is ignored on 429 | Retries may hit the rate limit again | `sync.py:62` |
| 14 | If the existence check returns non-200 for any reason, the bot re-posts; a 409 then reports **Duplicate** for an invoice this run created. Relies on billing returning 409 (unconfirmed). | Misleading status; double-post risk if billing doesn't reject duplicates | `sync.py:43`, `sync.py:56-57` |
| 15 | Version only in README (plus a stray commit-message line); not in log or report | Ops can't confirm which version runs | `README.md:5-6` |
| 16 | The two "fix" commits (6be93ce, 28da8f3) only add comment lines; the retry/check code was in the initial import | Can't tell when the fixes went live, or that the deployed copy has them | `sync.py:86-87` |
| 17 | The report includes `customer_name` | Personal data sits in the output folder ops share | `sync.py:76` |
| 18 | `BOT_WAS_RUN` marker written to the project root on every run, read by nothing | Confusing file; remove or document | `sync.py:67` |
| 19 | `nan` / `inf` amounts pass validation | Garbage amounts sent to billing | `sync.py:33-37` |
| 20 | Missing billing key or URL isn't checked at start; falls back to a placeholder URL / empty key | Shows up only as every row Failed (E09) | `sync.py:18-19` |
| 21 | The scheduled task isn't in the repo; console output not redirected; ops have no listed Task Scheduler access | Ops can't stop the bot or see crash output themselves | `sync.py:3` |
tests/evals/results/handoff/h-api-sync-full-r3.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-api-sync/full",
"rep": 3,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"retry limit": true,
"console-only crash output flagged": true
},
"cost_usd": 1.3519728,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-api-sync-full-rul4dkzn\\project",
"reply": "`HANDOVER.md` ready, repo root la irukku: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-api-sync-full-rul4dkzn\\project\\HANDOVER.md`. Code, logs and git history mattum padichen. Bot-ah run pannala, code edhuvum change pannala.\n\n| Item | Result |\n|---|---|\n| Sections | 13 (template order apdiye) |\n| Exceptions | 10 (E01\u2013E10) |\n| Failure classes | 10 covered, 2 not applicable (F2: window illa, API mattum; F10: ovvoru invoice-um thaniya process aagudhu) |\n| Known incidents | 2: 429 crash, 504 duplicate. Exact dates git-la illa. Rendum 0.9.1 ku munnadi nadandhadhu. |\n| Fix before handover | 14 items |\n\n**Fix before handover \u2014 mukkiyamanavai:**\n- 207 partial response \"Synced\" nu varudhu. Sila lines reject aanaalum success maadhiri theriyum.\n- Crash reason-um network error-um screen-la mattum print aagudhu. Ops paakkura entha file-layum varadhu.\n- `data/output/` folder-ah code create pannala. Pudhu machine-la first run-e crash aagum.\n- Duplicate check fragile: network error vandha run crash aagum, outage-la duplicate invoice varalaam.\n- 400/401/403/404 errors-um retry aagudhu. Key expire aana, ellaa row-um \"Failed\" nu thaan varum. Key problem-nu theriyadhu.\n- Overlapping runs-ah thadukka lock illa. Version ops-ku engayum theriyadhu.\n\nMattha 8 items (log-la reason illadhadhu, Invalid rendu artham-la use aagradhu, tests illadhadhu, etc.) doc-oda section 13 la irukku.\n\n**Also noticed:**\n- Bot-ku screen kidaiyadhu. Ops paakka mudiyuradhu output folder-um logs folder-um mattum.\n- Task Scheduler access ops-ku irukka-nu theriyala. Stop panna Task Scheduler la End pannanum. Access illana #rpa-support la kekka sollirukken.\n- History scan-la git commands run aagala. Commit messages-um ippo irukura code-um vechu thaan incidents ezhudhinen. Pazhaiya diffs paakkala.\n\nQUESTIONS: Escalation: who does ops call for a dev issue, and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions mean where the code doesn't make it clear?",
"stderr": ""
}
tests/evals/results/handoff/h-api-sync-full-r3.md
+168
View File
@@ -0,0 +1,168 @@
# Invoice sync - ops handover
Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel #rpa-support
## 1. What the bot does
Every hour it takes the approved invoices in the finance export file (`data/input/invoices.csv`) and creates them in the billing system through billing's web API. It checks each invoice first: required fields filled in and a valid, positive amount. It then sends the invoice to billing. If billing is busy or slow, it retries. For each run it writes one report file in `data/output/`, listing every invoice with a Status and a Message, and adds a few lines to `logs/sync.log`. It has no window or screen of its own and does not drive any desktop app or browser. It talks to billing directly over the network.
## 2. How it works
```
Windows Task Scheduler (every hour)
└─ starts → Invoice sync script ──sends invoices──> Billing system (web API)
│
├─ reads → data/input/invoices.csv
├─ writes → data/output/sync_report_<YYYYMMDD_HHMM>.csv
└─ writes → logs/sync.log
```
- **Task Scheduler** starts the bot every hour on the bot machine. Nobody needs to click anything.
- **The Invoice sync script** is the bot. It runs with no window, works through the input file from top to bottom, then stops.
- **The billing system** is where invoices are created. The bot only creates invoices and checks whether they exist. It never edits or deletes them.
- **The input file** is the finance export. The bot reads it and never changes it. If the file is not replaced, the next run sends every row again (see Duplicate in section 6).
- **The report file** is one new file per run with columns `invoice_no`, `customer`, `Status` and `Message`. This is the main place to look.
- **The log file** gets one line per invoice (invoice number and status) and a closing `run done` line when the run finishes.
**One run, step by step:**
1. Open the input file and create a new report file for this run.
2. For each row, check that invoice number, customer ID, amount and currency are filled in, and that the amount is a positive number. If not, the row is marked **Invalid** and nothing is sent to billing.
3. Send the invoice to billing.
4. If billing is busy, slow or unreachable, check whether the invoice was created anyway, wait, and try again (up to 5 tries; section 8).
5. Write the outcome to the report and the log, then move to the next row.
6. Write `run done -> sync_report_<time>.csv` to the log.
## 3. Before a run
- [ ] The folder `data/output/` exists next to the bot. **The bot does not create it.** If it is missing, the run stops before processing any invoice (E09).
- [ ] The settings file `.env` is in the bot folder and contains the billing address and the billing access key. Ops never opens or edits this file. If it is missing or the key has expired, every invoice ends up **Failed** (E08).
- [ ] The input file `data/input/invoices.csv` is present and has the columns `invoice_no, customer_id, customer_name, amount, currency`.
- [ ] Input rules, one row per invoice:
- Invoice number, customer ID, amount and currency must all be filled in. A blank in any of them gives **Invalid - missing ...**.
- Amount must be a plain positive number. Use a dot for decimals and no thousands separator or currency symbol.
- Bad: `1,250.00`, `$120`, `-50`, `0`
- Good: `1250.00`, `120`
- The bot cannot send credit notes or zero-value invoices, because the amount must be above 0.
- [ ] Any version change is confirmed by the dev team (section 10). The bot does not show its version anywhere ops can see.
## 4. During a run
- There is nothing to watch. A normal run takes seconds to a few minutes.
- Don't edit, rename or move the input file while a run is in progress. Replace it between runs.
- Don't open the newest report file in Excel while the run is still writing it. Wait for the `run done` line in the log.
- **To stop the bot:** in Task Scheduler on the bot machine, right-click the task, choose **End**, and then **Disable** it if it must not start again next hour. If you don't have access to Task Scheduler, ask in #rpa-support. Stopping halfway is safe: the next run re-sends the file, and invoices already in billing come back as **Duplicate**.
- **A run looks stuck** when the newest report file stops getting new rows and there is no `run done` line in the log. The bot has no stuck-detection of its own. Each invoice can take up to about 6 minutes when billing is not responding (section 8), so wait 10 minutes. If a run is still going when the next hourly run starts, escalate (E10).
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `logs/sync.log`: a line `run done -> sync_report_<time>.csv` for that hour. No such line means the run stopped partway (E09). **"run done" only means the run reached the end, not that the invoices were created.** |
| Which invoices failed? | The newest `data/output/sync_report_<YYYYMMDD_HHMM>.csv`. Filter Status for **Failed** and **Invalid**, and filter Message for **some lines rejected**. |
| Why did one invoice fail? | The **Message** column of the same report. The log shows only the status, never the reason. |
| Did the bot already retry? | **Failed** always means it tried 5 times. **Synced** with "Created (confirmed after retry)" means a retry found the invoice already in billing. Nothing else records the retries ops can see. |
| When did it fail? | `logs/sync.log`: search for the invoice number. Each line starts with date and time. |
| Why did the whole run stop? | Ops cannot see this. The reason is not written to any file ops can open. Escalate (E09). |
| Which version is running? | Ops cannot see this. Ask in #rpa-support. |
## 6. Statuses
The report is a plain CSV file, so there are no colours. The Status column holds one of four values. Read it together with the Message column.
| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | Invoice created in billing | nothing |
| Synced | Created (some lines rejected) | **Invoice created, but billing refused some of its lines. Needs follow-up.** | See E05 |
| Synced | Created (confirmed after retry) | Billing was slow or busy, but on a retry the bot found the invoice in billing | nothing |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number. Normal for rows that were sent in an earlier run. | See E06 |
| Invalid | missing *field names* | A required field is empty in the input. Not sent to billing. | See E01 |
| Invalid | amount must be positive | Amount is 0 or negative. Not sent to billing. | See E02 |
| Invalid | amount '*value*' is not a number | Amount is not a plain number. Not sent to billing. | See E03 |
| Invalid | Rejected by billing, or billing's own explanation text | Billing looked at the invoice and refused it | See E04 |
| Failed | No success after 5 tries | Billing never confirmed it after 5 tries. **It may still have been created.** | See E07 and E08 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing customer_id" (or other field names) | Before sending, it checks that invoice number, customer ID, amount and currency are not blank | Report: Status, Message. Log: invoice number + Invalid | Business | Ops | Ask finance to fill the field in the export. The next hourly run picks the row up again. | The field is filled in the file and it still says missing. Send the invoice number and report file name. |
| E02 | Invalid - "amount must be positive" | The amount is 0 or less | Report: Status, Message | Business | Ops | Check with finance. The bot cannot send zero or negative amounts (credit notes); these must be entered in billing by hand. | Finance says this kind of invoice must be synced automatically. |
| E03 | Invalid - "amount '1,250.00' is not a number" | The amount cannot be read as a number (comma, currency symbol, text) | Report: Status, Message | Business | Ops | Ask finance to export the amount as a plain number, e.g. `1250.00`. | The amount looks plain and is still rejected. |
| E04 | Invalid - "Rejected by billing" or another explanation text | Billing replied that the invoice data is not acceptable | Report: Status, Message (billing's own text) | Business | Ops | Read the message (e.g. unknown customer, wrong currency) and get the data fixed at the source. Retrying unchanged data gives the same result. | The message is unclear, or the data looks right. Send the invoice number, run time and the exact message. |
| E05 | Synced - "Created (some lines rejected)" | Billing replied that it created the invoice but refused some lines | Report: Status, Message. The log shows only "Synced". | Business | Ops | **This is not done.** Open the invoice in billing (or ask the billing team), find the missing lines, and get them added by hand or fixed by finance. The bot will not resend them, because the invoice now exists. | You cannot tell which lines are missing. Send the invoice number and run time. |
| E06 | Duplicate - "Invoice already exists in billing" | Billing replied that this invoice number already exists | Report: Status, Message | Business | Ops (usually nothing) | If the invoice was sent in an earlier run, do nothing. This is normal when the input file is not replaced. If this is the **first** time you see this invoice number, check in billing that the existing invoice is the same one (same customer and amount). | The existing invoice in billing is a different one. Send the invoice number and run time. |
| E07 | Failed - "No success after 5 tries" on one or a few rows | Billing was busy, slow (no reply within 30 s), down, or gave an unexpected reply on all 5 tries. Before each retry it checked whether the invoice existed and did not find it. | Report: Status, Message. Log: invoice number + Failed | System | Bot, then Ops | **First check in billing whether the invoice was created anyway.** If it is there, nothing more is needed; the next run will show Duplicate. If it is not there and the row is still in the input file, the next hourly run tries again. | The same invoice is Failed in 3 runs in a row. Send the invoice number, run times and the report file names. |
| E08 | Failed on **all or almost all** rows, and the run takes much longer than usual | Same as E07, but for every row. This usually means the billing access key is missing or expired, the billing address is wrong, or billing is down. | Report: every Status Failed. Log: long gaps between lines. | System | Dev | Nothing ops can fix. Don't change the input file. | Always. Send the run time, the report file name and the `logs/sync.log` file. |
| E09 | No `run done` line for that hour; the report is missing, empty, or stops partway | The run stopped early. Known causes: `data/output/` folder missing, input file missing, network failure while the bot was checking whether an invoice exists, or an unexpected reply from billing. | `logs/sync.log` (the closing line is missing). The cause is not in any file ops can open. | System | Dev | Check that `data/output/` and the input file exist (section 3). Rows written to the report before the stop are real results. Rows after it were not processed; the next hourly run re-sends them. | It happens in 2 runs in a row, or the folders are present. Send the run time, the last report file name and `logs/sync.log`. |
| E10 | Two report files close together for the same hour, or a run still going when the next one starts | A run took more than an hour (many slow or failed rows). Nothing stops two runs at once. | Report file names and log times | System | Dev | Check first in billing for any invoice that shows Failed, as in E07. | Always. Send both report file names and the log. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing busy (too many requests), down, slow (no reply in 30 s), or an unexpected reply for one invoice | Checks whether the invoice was created anyway. If not, waits and tries again. Waits grow: 2, 4, 8, 16, 32 s. | 5 tries per invoice. About 1 minute of waiting, and up to about 6 minutes per invoice if every call times out. | Invoice marked Failed (E07), next invoice |
| The invoice was created but billing's reply got lost | The check before each retry finds it | Same 5 tries | Synced - "Created (confirmed after retry)" |
| Invoice sent again in a later run | Billing answers "already exists" | none | Duplicate (E06). Nothing is created twice. |
| Run stopped partway, or Failed / not-yet-sent rows | No restart within the hour. Task Scheduler's next hourly run starts fresh and re-sends the whole input file. | Once per hour | If the same problem repeats over 2-3 runs, escalate (E07, E09) |
Before acting on a Failed row, wait for at least one more hourly run. The bot will usually have retried it already.
## 9. Known incidents
Exact dates are not recorded. Both incidents happened before version 0.9.1.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | The same invoice appeared twice in billing. Finance or billing noticed it; the bot reported nothing. | Billing created the invoice but timed out before replying, and the bot's retry sent it again | Fixed in 0.9.1: the bot checks billing before each retry. The check can still miss when billing is down, so **for any Failed row, check billing before re-sending it by hand** (E07). | Partly |
| Before 0.9.1 | Hourly run died partway through when billing said "too many requests". Invoices after that row were not sent. | The bot treated "too many requests" as fatal | Fixed in 0.9.1: up to 5 tries with waits (section 8). A run can still stop for other reasons (E09). | Partly |
## 10. Releases and version check
- **How an update arrives:** the dev team copies the new version into the bot folder on the bot machine. There is no launcher or auto-update. The next hourly run uses the new code.
- **How to confirm it applied:** ops cannot see the version. The bot writes it to no file ops can open. Ask the dev team to confirm in #rpa-support after every update.
- **If the version did not change:** report it in #rpa-support. Don't copy files yourself.
## 11. Escalating to dev
Post in the Teams channel **#rpa-support** (RPA dev team). Send:
- the exception ID from section 7 (e.g. E08)
- the invoice number(s)
- the run time (from `logs/sync.log`)
- the report file name (`sync_report_<YYYYMMDD_HHMM>.csv`)
- the `logs/sync.log` file
Never send: the `.env` settings file or anything from it, the billing access key, or customer names and amounts pasted into chat. Refer to rows by invoice number.
## 12. Failure-class check
- F1 covered in section 3 (output folder, settings file, input file) and E08, E09.
- F2 not applicable: the bot has no windows. It talks to billing over its web API and drives no desktop app.
- F3 covered in E07 and section 8 (30 s reply limit, 5 tries).
- F4 covered in section 3 input rules (bad and good amount examples) and E01-E04.
- F5 covered in section 5 ("run done" is not success) and E08. Fix #4 and #5 in section 13.
- F6 covered in E07 (check billing before re-sending) and section 9. Fix #5 in section 13.
- F7 covered in section 6 and E05 ("some lines rejected" shows as Synced). Fix #1 in section 13.
- F8 covered in E10 and section 8. There is no automatic re-run loop; the only repeat is the hourly schedule. Fix #8 in section 13.
- F9 covered in section 4 (End the task in Task Scheduler).
- F10 not applicable: every invoice is handled on its own. No batch waits for all rows, and nothing locks the next run. A slow row only delays the run (see F8).
- F11 covered by fixes #2, #3 and #10 in section 13. The reason a run stopped and network errors are not written to any file ops can open, and the log has no reason text.
- F12 covered in section 10 and fix #9 in section 13. The version is not visible to ops.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | HTTP 207 (partial) is written as Status `Synced` | Ops filtering on Status thinks the invoice is complete when lines are missing | `sync.py:54-55` |
| 2 | Crash reason (`CRASH: ...`) and network errors are only `print`ed to stdout, never logged | Ops cannot find why a run stopped (E09). Lost unless Task Scheduler captures stdout. | `sync.py:52`, `sync.py:84` |
| 3 | Log line has status only, no Message | Ops must open the report to learn why | `sync.py:75` |
| 4 | `data/output/` is not in the repo and never created | Fresh machine: the first run crashes before any invoice | `sync.py:68-69` |
| 5 | `exists()` is not in a try block, and any non-200 counts as "not there" | A network error during the check crashes the run. During an outage it can re-POST an invoice that was already created (possible duplicate). | `sync.py:41-43`, `sync.py:60` |
| 6 | A 422 with a non-JSON body makes `r.json()` raise | Whole run crashes on one bad reply | `sync.py:59` |
| 7 | Every non-success code is retried (400/401/403/404 too, not only 429/5xx as the docstring says), and `Retry-After` is ignored | A bad or expired key shows as generic `Failed` on every row and takes about 1-6 min per row (E08) | `sync.py:47`, `sync.py:54-62` |
| 8 | No lock against overlapping runs. The report name has minute resolution, so a same-minute run overwrites the report. It also sleeps 32 s after the last try for nothing. | Slow runs overlap the next hour (E10), and reports can be lost | `sync.py:62`, `sync.py:68` |
| 9 | Version only in `README.md` (plus a stray "chore: bump to 0.9.1" line). Not logged or shown anywhere. | Ops cannot confirm an update applied | `README.md:5-6` |
| 10 | `Invalid` is used both for local checks and for billing's 422 rejection | The status alone doesn't say whether to fix the file or ask billing | `sync.py:59`, `sync.py:74` |
| 11 | The report stores customer names. No retention policy, and `logs/sync.log` is never rotated. | Sensitive data piles up in `data/output/`. The log grows forever. | `sync.py:24`, `sync.py:71`, `sync.py:76` |
| 12 | A `BOT_WAS_RUN` marker file is written every run and never read | Unclear purpose. Confirm or remove. | `sync.py:67` |
| 13 | Task Scheduler task definition (and whether stdout is captured) is not in the repo | Can't rebuild the schedule on a new machine | `sync.py:3` |
| 14 | No tests cover the two incident fixes (429 retry, check-before-retry) | Regressions would come back unnoticed | whole repo |
tests/evals/results/handoff/h-api-sync-full-r4.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-api-sync/full",
"rep": 4,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"retry limit": true,
"console-only crash output flagged": true
},
"cost_usd": 1.259045,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-api-sync-full-mf49pa7c\\project",
"reply": "I've written `HANDOVER.md` in the repo root. Your three answers were used, so I didn't ask anything. I only read the code and history; the bot was never run, and no code was changed.\n\n| Item | Count |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 13 (E01\u2013E13) |\n| Failure classes | 10 covered, 2 not applicable (F2: the bot drives no windows; F10: one bad invoice never blocks the others) |\n| Fix before handover items | 14 |\n\n**Fix before handover, main items:**\n1. The reason an invoice failed is only printed to the screen, never logged. Every Failed row just says \"No success after 5 tries\".\n2. Duplicate invoices can still happen. The 504 fix still has a timing gap, and there is no idempotency key.\n3. An error while the bot checks \"does this invoice exist\" crashes the whole run, and the crash isn't saved anywhere. A run where every invoice failed still exits as success.\n4. The \"some lines rejected\" case is shown as **Synced**, so those rows hide when ops filter on Status.\n5. Every row is resent every hour, and nothing stops two runs at once. Ops will see Duplicate noise every hour, and overlapping runs risk duplicates.\n6. The code doesn't create the `data/output/` folder, and it isn't in the repo. A fresh machine crashes before the first invoice.\n7. Other items are in the doc: an expired key is retried slowly instead of failing fast, input checks have gaps, the version isn't shown anywhere, the Task Scheduler job isn't documented, the report holds customer names, and an unused `BOT_WAS_RUN` file is written in the bot folder.\n\n**Checks on the doc:**\n- All 4 statuses and 9 messages are listed.\n- No file:line references or code terms outside section 13.\n- No customer data or keys; I never opened the `.env` file or the input file.\n- Every Dev row says what to send.\n- The one path that doesn't exist yet is `data/output/`; it's item 6 above.\n\n**Assumptions to confirm:**\n- The bot has no screens, so for \"ops access\" I took the output and logs folders. The doc doesn't assume ops can open Task Scheduler.\n- On a re-run, I read **Duplicate** as \"already synced in an earlier hour\".\n- The git diff commands were denied, so the incident causes come from the commit messages and the current code, not the actual diffs.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-api-sync-full-r4.md
+161
View File
@@ -0,0 +1,161 @@
# Invoice sync - ops handover
Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every hour it takes the approved invoices from the finance export file and sends each one to the billing system. Input: the CSV file `data/input/invoices.csv`, one invoice per row, with an invoice number, a customer ID, a customer name, an amount and a currency. Output: one report file per run in `data/output/` with a status and a message for each invoice, and a line per invoice in `logs/sync.log`. It has no screens and drives no desktop apps. It talks to the billing system over the network in the background. It starts by itself from Windows Task Scheduler.
## 2. How it works
```
Windows Task Scheduler (every hour)
└─ starts → Invoice sync (background, no window)
│ reads → data/input/invoices.csv
├─ sends each invoice ──> Billing system (online)
├─ writes → data/output/sync_report_<date>_<time>.csv
└─ writes → logs/sync.log
```
- **Task Scheduler** starts the bot once an hour. Nobody has to click anything.
- **Invoice sync** is the bot. It runs without a window and stops by itself when it reaches the end of the file.
- **The input file** is read from top to bottom, every row on every run.
- **The billing system** receives each invoice and answers "created", "already exists", "rejected" or nothing at all.
- **The report file** is the only place that says *why* an invoice did not go through (the Message column).
- **sync.log** gets one line per invoice with its status only, and a final "run done" line when the run finished cleanly.
**One run, step by step:**
1. Open a new report file named after the date and minute the run started.
2. For each row: check the row (invoice number, customer ID, amount and currency present; amount a positive number).
3. If the row is fine, send it to billing. If billing doesn't answer properly, ask billing whether the invoice is already there, wait, and try again. Up to 5 tries.
4. Write the result to the report and the log.
5. After the last row, write "run done -> <report name>" in the log.
## 3. Before a run
These are set up once by the dev team. Ops can't check most of them, but the message each one gives is in section 7.
- [ ] The folders `data/output/` and `logs/` exist. If either is missing, the run stops before it does anything (E11).
- [ ] The settings file next to the bot has the billing address and the billing key. If the key is missing or has expired, every invoice ends **Failed** and the run is very slow (E09).
- [ ] The finance export is saved as `data/input/invoices.csv` with the header row `invoice_no, customer_id, customer_name, amount, currency`.
- [ ] Input rules:
- **Amount is a plain number.** Good: `1200.50`. Bad: `1,200.50`, `$1200`, `1200.50 USD`. These all end **Invalid** "is not a number" (E03).
- **Amount above zero.** Credit notes or zero amounts end **Invalid** (E02).
- **Invoice number, customer ID, amount and currency are never blank.** Otherwise the row ends **Invalid** "missing …" (E01). A cell that holds only spaces is *not* caught, so check those by eye.
- **Each invoice number once per file.** The bot doesn't spot repeats in the same file.
- [ ] Remove invoices from the file once they are synced, if the finance process allows it. The bot sends **every row on every hourly run**, so old rows come back as **Duplicate** each hour (E05).
## 4. During a run
- Don't edit, move or save over `data/input/invoices.csv` while a run is going. Wait until the hour's "run done" line appears in `logs/sync.log`.
- Don't open the current report file in Excel while the run is still writing it. Open a copy, or wait for "run done".
- To stop the bot: end the "Invoice sync" task in Windows Task Scheduler. If ops can't open Task Scheduler on the bot machine, ask in #rpa-support. Invoices already sent stay in billing. Stopping it does not undo them.
- A run is normally finished within a few minutes. One stubborn invoice can hold the run for **up to about 6 minutes** while the bot retries it. A run looks stuck when `logs/sync.log` gets no new line for more than 10 minutes and there is no "run done" line. There is no automatic watchdog. Escalate (E13).
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `logs/sync.log`: the run ends with a line "run done -> sync_report_<date>_<time>.csv". No such line after that hour's invoice lines means the run crashed (E10). |
| Which invoices failed? | The run's report in `data/output/` (named `sync_report_YYYYMMDD_HHMM.csv`): filter the **Status** column for anything except Synced. |
| Why did one invoice fail? | The same report, **Message** column. The log has no reasons, only the status. |
| Which run handled this invoice? | Search `logs/sync.log` for the invoice number. The time on that line matches the report's file name. |
| Did the bot already retry? | Not recorded anywhere ops can see. **Failed** always means 5 tries were made. **Synced - "Created (confirmed after retry)"** means at least one retry happened. |
| Which version is running? | Not shown by the bot. Ask in #rpa-support (see section 10). |
## 6. Statuses
The report is a plain CSV file, so there are no colours. Read the **Status** column first, then the **Message** column.
| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | The invoice is in billing. | Nothing |
| Synced | Created (confirmed after retry) | Billing didn't answer at first. The bot then found the invoice in billing. It may also have been there from an earlier run. | Nothing. If billing shows it twice, see E12. |
| Synced | Created (some lines rejected) | **Needs follow-up.** The invoice was created, but billing refused some of its lines. | Check the invoice in billing and fix the missing lines (E06) |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number. On an hourly re-run this usually just means it was synced in an earlier run. | First time you see it: check in billing (E05). Otherwise nothing. |
| Invalid | missing <field names> | Required fields were blank. The row was not sent. | Fix the input row (E01) |
| Invalid | amount must be positive | The amount is zero or negative. Not sent. | Fix the input row (E02) |
| Invalid | amount '<value>' is not a number | The amount has commas, a currency sign or text. Not sent. | Fix the input row (E03) |
| Invalid | any other text, or "Rejected by billing" | Billing refused the invoice and gave this reason. Not created. | Fix the data as the message says (E04) |
| Failed | No success after 5 tries | Billing never gave a clear answer in 5 tries. **The invoice may or may not be in billing.** | Check in billing first (E08) |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing customer_id, amount" (or other fields) | Before sending, it checks that invoice number, customer ID, amount and currency are filled in | Report: Status, Message. Log: status only | Business | Ops | Fill the blank fields in the finance export. The next hourly run picks the row up again. | Never, unless the fields are filled and it still says missing |
| E02 | Invalid - "amount must be positive" | Amount is zero or below | Report: Status, Message | Business | Ops | Check with finance. Credit notes and zero invoices can't go through this bot. | Finance needs credit notes synced |
| E03 | Invalid - "amount '1,200.50' is not a number" | The amount can't be read as a number | Report: Status, Message | Business | Ops | Write the amount as a plain number, e.g. `1200.50`: no thousands commas, no currency sign | Never |
| E04 | Invalid - with a message from billing, or "Rejected by billing" | Billing answers that the invoice data is not acceptable | Report: Status, Message | Business | Ops | Fix the data as billing's message says (e.g. unknown customer, bad currency) | The message is unclear, or the data is already correct |
| E05 | Duplicate - "Invoice already exists in billing" | Billing answers that this invoice number already exists | Report: Status, Message | Business | Ops | **On re-runs this is normal.** The invoice was synced in an earlier hour. Search older reports for the same invoice number with Synced. If there's no earlier Synced, open it in billing: if it's a different invoice with the same number, give it to finance. | The invoice was never synced before and billing doesn't show it |
| E06 | Synced - "Created (some lines rejected)" | Billing answers "created, but part of it was refused" | Report: Message only. **Status says Synced.** | Business | Ops | Open the invoice in billing, compare its lines with the export, and add or fix the missing lines there | You can't see which lines were refused |
| E07 | Synced - "Created (confirmed after retry)" | Billing didn't answer clearly. The bot asked billing whether the invoice exists, and it does. | Report: Message | System | Bot | Nothing. The bot recovered. | Billing shows the invoice twice (E12) |
| E08 | Failed - "No success after 5 tries" | 5 tries with waits of 2, 4, 8, 16 and 32 s, and billing never confirmed the invoice | Report: Status, Message. The real reason is not saved. | System | Ops, then Dev | **1. Check in billing first.** The invoice may have been created even though the bot didn't get an answer. 2. If it's there, nothing to do: the next run will show it as Duplicate. 3. If it isn't, wait for the next hourly run, which tries again by itself. | The same invoice is Failed in 2 runs in a row. Send the invoice number and both report file names. |
| E09 | Most or all invoices in one run are **Failed**, and the run takes very long (about a minute or more per invoice) | Same as E08, for every invoice. Usually billing is down, or the billing key in the settings file is missing or expired. | Report: every row Failed | System | Dev | Don't edit the input. Check billing yourself if you can log in. | Always. Send the report file name and the time of the run. |
| E10 | The hour's invoice lines in `logs/sync.log` stop and there is **no "run done" line**. The report has fewer rows than the input. | The bot hit an unexpected error and stopped. Known triggers: billing unreachable while the bot checks whether an invoice exists, or an odd answer from billing. | Only the missing "run done" line. The error itself is not saved anywhere ops can see. | System | Dev | Invoices before the stop were handled as the report shows. The rest are tried again next hour. If it happens two hours in a row, escalate. | 2 runs in a row. Send the report file name, the last invoice number in the log, and the time. |
| E11 | No new report in `data/output/` for an hour and no new lines in `logs/sync.log` | The run stopped before the first invoice: the input file, the output folder or the logs folder is missing, or Task Scheduler didn't start it | Nothing is recorded | System | Ops, then Dev | Check that `data/input/invoices.csv` exists and has the right name | The input file is there. Send the hours with no report. |
| E12 | The same invoice appears **twice in billing** | Not detected by the bot. Billing finished creating the invoice after the bot gave up waiting, and the bot sent it again. Or two runs overlapped. | Not recorded. Report shows Synced, Failed or Duplicate. | System | Dev (and finance) | Don't delete anything in billing yourself. Collect the invoice number and the report rows for it. | Always |
| E13 | A run is still going after 10 minutes with no new log line, or two runs' lines are mixed in the log | Not detected. There is no watchdog and nothing stops two runs at once. | Log timestamps only | System | Dev | Don't start another run by hand. | Always. Send the time the run started and the last log line. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing doesn't answer, is busy, or answers with an error | Asks billing whether the invoice already exists. If yes: Synced. If not: waits, then sends it again. | 5 tries. Waits of 2, 4, 8, 16 and 32 s (about 1 minute), plus up to 30 s per try waiting for billing. Up to about 6 minutes for one invoice. | Failed - "No success after 5 tries", next invoice (E08) |
| An invoice ended Invalid or Failed | Nothing in that run. **The next hourly run reads the whole file again and retries it.** | Every hour, as long as the row is in the file | Stays Invalid until the input is fixed |
| Run crashed | Nothing. There is no resume. The next hourly run starts from the top of the file again. | - | E10 |
Ops rule of thumb: for one Failed invoice, wait one hour for the next run before acting. For a missing "run done" line or a whole run of Failed, escalate straight away.
## 9. Known incidents
Dates aren't reliable in the history. Both fixes are in version 0.9.1.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| before 0.9.1 | The same invoice twice in billing | Billing created the invoice but timed out before answering, and the bot sent it again | Partly fixed: the bot now asks billing whether the invoice exists before retrying. It can still happen when billing is slow (E12). For **Failed** rows, check billing before anything else (E08). | Partly |
| before 0.9.1 | Hourly run stopped partway, with no "run done" line | Billing said "too many requests" and the bot stopped | Partly fixed: the bot now waits and retries. It can still stop if billing is unreachable during the "does it exist" check (E10). | Partly |
## 10. Releases and version check
- How an update arrives: the dev team replaces the bot's files on the bot machine. There is no automatic updater.
- How to confirm it applied: the bot doesn't show or log its version. Ask in #rpa-support for the version on the machine. The version is only written in the bot's README file.
- If the version did not change: report it in #rpa-support. Ops has nothing to re-install.
## 11. Escalating to dev
Post in **#rpa-support** (Teams) and send: the exception ID from section 7, the run time, the report file name (`sync_report_YYYYMMDD_HHMM.csv`), the invoice number(s), and the matching lines from `logs/sync.log`.
Never send: the billing key, the settings file, or customer names and amounts in chat. Refer to invoices by invoice number only.
## 12. Failure-class check
- F1 covered in section 3 (folders, settings file, input file name) and E09 / E11; gaps in section 13 #7, #8.
- F2 not applicable: the bot drives no windows. It only talks to billing over the network.
- F3 covered in E08 / E09 and section 8 (30 s wait per try, 5 tries). No screens, so screen-size rules don't apply.
- F4 covered in section 3 input rules (amount format, blanks, repeated invoice numbers) and E01-E04; gaps in section 13 #9.
- F5 covered in E06 and E10. A crashed run leaves a short report with no "run done" line. Gap in section 13 #3.
- F6 covered in E08 and E12 (check billing before acting on Failed); gap in section 13 #2.
- F7 covered in section 6 and E06 ("some lines rejected" shows Synced); gap in section 13 #4.
- F8 covered in E05 / section 3 (every row resent every hour) and E13 (overlapping runs); gaps in section 13 #5, #6.
- F9 covered in section 4 (end the task in Task Scheduler); gap in section 13 #11.
- F10 not applicable as a queue: one bad invoice never blocks the others, and nothing locks the next run. The nearest case is a slow invoice holding a run for about 6 minutes (section 4, E13).
- F11 covered in section 5; the hidden causes are section 13 #1, #3.
- F12 covered in section 10; gap in section 13 #10.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The reason for Failed (HTTP code, network error) is printed to stdout only. The log has the status but no message. | Ops can't tell an expired key from billing being down. Everything reads "No success after 5 tries". | `sync.py:52`, `sync.py:63`, `sync.py:75` |
| 2 | The exists-check runs straight after the timeout, with no idempotency key on the POST. A slow create is retried, giving a duplicate or a misleading 409 "Duplicate". | Duplicate invoices in billing (E12). Ops chases false duplicates. | `sync.py:50`, `sync.py:60` |
| 3 | `exists()` and `r.json()` are unprotected, there is no per-row try/except, and the crash goes to stdout only. Exit code 0 when every row Failed. | A whole run dies with no saved cause (E10). A run where everything failed looks successful to Task Scheduler. | `sync.py:42`, `sync.py:59`, `sync.py:72-76`, `sync.py:84` |
| 4 | HTTP 207 (some lines rejected) is mapped to Synced | Rows needing follow-up are hidden when the report is filtered on Status (F7) | `sync.py:54-55` |
| 5 | Every row is resent every hour. Nothing records what was already synced. | Hourly Duplicate noise. If billing ever stops returning 409, it creates duplicates. | `sync.py:72-74` |
| 6 | No single-run lock. Report name is to the minute, opened with "w". | Overlapping runs (E13) can double-send and overwrite each other's report | `sync.py:68-69` |
| 7 | `data/output/` and `logs/` are not created by the code. `logs/` missing fails at import. | A fresh machine produces nothing and no error ops can see (E11) | `sync.py:24`, `sync.py:68` |
| 8 | A missing key becomes an empty Bearer token. 400/401/403/404 are retried 5 times like 429/5xx, and Retry-After is ignored. | An expired key turns one run into ~1 min+ per invoice, all Failed (E09) | `sync.py:19`, `sync.py:60-62` |
| 9 | Spaces-only fields and `nan`/`inf` amounts pass validation. Currency isn't checked. Duplicate invoice numbers within a file aren't checked. | Bad rows go to billing instead of being flagged as Invalid | `sync.py:30`, `sync.py:34` |
| 10 | Version isn't logged or shown, and README has a stray "chore: bump to 0.9.1" line | Ops can't confirm an update applied | `README.md:5-6`, `sync.py:77` |
| 11 | The Task Scheduler job is not in the repo or documented (name, account, output redirection), and ops access to it is not confirmed | Ops can't stop a stuck run or see the last result themselves | n/a (scheduler config) |
| 12 | The report's "customer" column holds the customer name, and the whole row (including the name) is sent to billing | Customer names sit in plain CSV files in the output folder that ops shares around | `sync.py:50`, `sync.py:76` |
| 13 | Writes an unused `BOT_WAS_RUN` marker file in the bot folder | Confusing leftover. Ops may think it's a lock. | `sync.py:67` |
| 14 | The "network error" print and the "CRASH" print are the only signal for network problems, and the sleep after the 5th try is wasted (32 s) | Slower runs, no record | `sync.py:52`, `sync.py:62` |
tests/evals/results/handoff/h-api-sync-full-r5.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-api-sync/full",
"rep": 5,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"retry limit": true,
"console-only crash output flagged": true
},
"cost_usd": 1.3559016000000002,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-api-sync-full-h9gb7d_4\\project",
"reply": "I wrote `HANDOVER.md` in the repo root (UNVERIFIED in the sense that, as the skill says, I didn't run the bot; it's all from reading the code, git history and the log).\n\n| Item | Count |\n|---|---|\n| Sections | 13 (template order apdiye) |\n| Exceptions catalogued | 13 (E01\u2013E13) |\n| Failure classes | 11 covered, 1 not applicable (F2: bot entha window-um drive pannala, it only calls the billing web API) |\n| Fix before handover | 18 items |\n\nUnga answers use pannen: escalation #rpa-support, and \"Where to look\" only points at the output and logs folders. Bot-ku screens ethuvum illa, so screens row skip pannen. Database-um illa.\n\n**Fix before handover (dev team):**\n1. The `.env` file (API key) is committed to git. Rotate the key, then untrack the file.\n2. The `data/output` folder isn't in the repo and the code doesn't create it, so a fresh install crashes every run.\n3. A network drop while the bot checks billing after a failed send crashes the whole run.\n4. If billing sends a reply the bot can't read, the run crashes, and it crashes at the same invoice every hour, so every invoice after it is blocked.\n5. Crash and network error messages only go to the console, so they're lost under Task Scheduler.\n6. \"Some lines rejected\" (partly rejected) invoices are stored as Synced.\n7. \"Confirmed after retry\" only checks the invoice number, so an older invoice with the same number can hide a send that never happened.\n8. Wrong-key and bad-request errors are retried 5 times like temporary errors.\n9. A missing API key or URL fails silently instead of stopping with a clear error.\n10. An Excel \"CSV UTF-8\" export makes every row show \"missing invoice_no\".\n11. Nothing stops two runs from overlapping, and the report file name only goes to the minute, so one can overwrite the other.\n12. `sync.log` doesn't record the reason, error code or number of tries.\n13. Ops can't see the version anywhere, and stray commit messages are pasted into the README and the code.\n14. The Task Scheduler setup isn't in the repo.\n15. Nobody gets an alert when an invoice fails or the run crashes.\n16. The bot waits 32 s after the 5th try before giving up.\n17. The purpose of the `BOT_WAS_RUN` marker file isn't documented.\n18. There are no regression tests for the two incident fixes.\n\n**Also noticed:** rendu \"fix\" commits-um code-la oru comment mattum thaan add pannudhu. Retry logic already first import-laye irundhuchu.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-api-sync-full-r5.md
+180
View File
@@ -0,0 +1,180 @@
# Invoice sync - ops handover
Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every hour, the bot reads the finance export of approved invoices (`data/input/invoices.csv`) and sends each invoice to the billing system over its web API. It writes one report file per run in `data/output/` with a Status and a Message for every invoice, and adds one line per invoice to `logs/sync.log`. It has no window or screen of its own and does not drive any desktop app or website. It is a background job that Windows Task Scheduler starts.
## 2. How it works
```
Windows Task Scheduler (every hour)
└─ starts → Invoice sync (background job, no window)
│ reads ← data/input/invoices.csv (finance export)
├─ sends → Billing system (web API, over the internet)
├─ writes → data/output/sync_report_<YYYYMMDD_HHMM>.csv
├─ writes → logs/sync.log
└─ writes → BOT_WAS_RUN (marker file in the bot folder, "a run started")
```
- **Task Scheduler** starts the job once an hour. The sample log shows runs at the top of the hour.
- **Invoice sync** is the bot. It runs and exits by itself, with nothing on screen.
- **The input file** is the finance export. The bot only reads it and never moves, renames or marks it.
- **The billing system** is where invoices get created. The bot never deletes or changes an invoice there.
- **The report file** is the only place that says *why* an invoice did not sync.
- **`sync.log`** shows when each invoice was handled and its status, plus a "run done" line at the end.
- **`BOT_WAS_RUN`** only shows that a run started. Ignore it.
**One run, step by step:**
1. Write the `BOT_WAS_RUN` marker and create a new report file named after the current date and minute.
2. Read the input file one row (one invoice) at a time.
3. Check the row: invoice number, customer ID, amount and currency must be filled in, and the amount must be a number above 0. If the row fails a check, it is marked **Invalid** and the bot goes to the next row.
4. Send the invoice to billing. The bot reads billing's answer and marks the row Synced, Duplicate or Invalid. If billing does not answer or answers with an error, the bot retries (section 8).
5. Write the row's result to the report and `sync.log`.
6. After the last row, write `run done -> sync_report_<time>.csv` to `sync.log`.
**Every hour the bot sends the whole input file again.** Invoices that already went through come back from billing as **Duplicate**. This is normal and does not create a second invoice.
## 3. Before a run
- [ ] The folder `data/output` exists inside the bot folder. **If it is missing, every run crashes before doing anything** (E10). The bot does not create it.
- [ ] The folder `logs` exists inside the bot folder.
- [ ] The settings file (`.env`) in the bot folder contains the billing address and API key. Dev maintains this file. If the key is missing or wrong, every invoice ends **Failed** (E09).
- [ ] The input file has exactly this header row: `invoice_no,customer_id,customer_name,amount,currency`.
- [ ] Input rules for whoever prepares the export:
- One invoice per row, with each invoice number used once.
- **Amount** is a plain number with no currency sign and no thousands separator.
- Bad: `$1,200.00`, `1.200,00`, `-50`
- Good: `1200.00`
- **Currency** is filled in, e.g. `USD`.
- **Customer ID** is filled in. The customer name is not checked.
- Save it as a plain CSV. If every row comes back "missing invoice_no" although the column is filled, the file was saved in a format the bot can't read (E12).
- [ ] The bot has no version display. See section 10.
## 4. During a run
- There is nothing to watch and nothing to click. Don't open, edit or move the input file while a run is going on.
- Don't open the newest report file in Excel while the run is still writing it. Wait for the "run done" line in `sync.log`.
- **To stop the bot:** it has no Stop button. End the running "Invoice sync" task in Windows Task Scheduler. If you don't have access to Task Scheduler, ask in #rpa-support. A stopped run leaves a partial report and no "run done" line. That is safe: the next run sends everything again, and invoices that already went through come back Duplicate.
- **A run looks stuck** when `sync.log` gets no new line for several minutes. One invoice that billing keeps rejecting can take **up to about 6 minutes** before the bot gives up on it (section 8). Wait 10 minutes before escalating. A run that is still going when the next hour starts is E13.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `logs/sync.log`: the run should end with a line `run done -> sync_report_<time>.csv`. If there is no such line for an hour, the run crashed or is still going (E10, E13). |
| Which invoices failed? | Newest file in `data/output/` (`sync_report_<YYYYMMDD_HHMM>.csv`). Filter the **Status** column for anything other than Synced and Duplicate. Also filter **Message** for "some lines rejected" and "confirmed after retry" (E06, E07). |
| Why did one invoice fail? | The same report file, **Message** column on that invoice's row. `sync.log` shows only the time and status, never the reason. |
| Did the bot already retry? | Only partly visible. "No success after 5 tries" means it retried 5 times. "Created (confirmed after retry)" means at least one retry happened. A gap of 30+ seconds between two lines in `sync.log` also means retries. Each retry is not logged. |
| When was one invoice last handled? | Search `logs/sync.log` for the invoice number, then open the report file whose time matches that line. |
| Which version is running? | Not visible to ops. Ask in #rpa-support (section 10). |
The report files contain customer names. Keep them inside the output folder, and don't paste their contents into chat or tickets.
## 6. Statuses
The report file is plain text, so no status has a colour.
| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | Billing created the invoice. | Nothing. |
| Synced | Created (some lines rejected) | **Needs follow-up.** Billing created the invoice but rejected some of its lines, so the invoice is incomplete. | Check the invoice in billing and follow up (E06). |
| Synced | Created (confirmed after retry) | Sending failed, but billing then showed an invoice with this number. Usually the bot's own earlier attempt went through. It can also be an older invoice with the same number. | Spot-check the invoice in billing (E07). |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number. It is usually an invoice an earlier run already sent. | Nothing, unless the number was reused for a different invoice (E05). |
| Invalid | missing `<field names>` | A required field is empty on that row. | Fix the input (E01). |
| Invalid | amount must be positive | The amount is 0 or negative. | Fix the input (E02). |
| Invalid | amount '`<value>`' is not a number | The amount has a symbol, separator or text in it. | Fix the input (E03). |
| Invalid | billing's own reason text, or "Rejected by billing" | Billing refused the invoice's content. | Fix the data as billing says (E04). |
| Failed | No success after 5 tries | Billing never accepted it after 5 tries over about a minute or more. | See E08. If every row says this, see E09. |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing customer_id, amount" (or other field names) | Before sending, it checks that invoice number, customer ID, amount and currency are filled in. | Report: Status, Message. Log: invoice number + Invalid. | Business | Ops | Get the export corrected. The next hourly run picks the row up again. | Never, unless the fields are filled in and it still says missing (then see E12). |
| E02 | Invalid - "amount must be positive" | The amount is 0 or below. | Report: Status, Message | Business | Ops | Credit notes and zero invoices are not supported. Get the row removed or corrected in the export. | Finance needs credit notes synced. |
| E03 | Invalid - "amount '…' is not a number" | The amount cannot be read as a plain number. | Report: Status, Message | Business | Ops | Fix the amount format, e.g. `1200.00` instead of `$1,200.00`. | Never. |
| E04 | Invalid - billing's own reason, or "Rejected by billing" | Billing answers that the invoice content is not acceptable. | Report: Status, Message (billing's text) | Business | Ops | Read the message and fix that data in the export, e.g. an unknown customer or currency. | The message isn't understandable, or the data looks right. |
| E05 | Duplicate - "Invoice already exists in billing" | Billing answers that the invoice number already exists. | Report: Status, Message | Business | Ops (usually nothing) | Normal for invoices sent in an earlier hour. Act only if this invoice number was **never** synced before (search older reports and `sync.log`). Then two different invoices share a number, so check in billing. | Billing holds a different invoice under that number. |
| E06 | Synced - "Created (some lines rejected)" | Billing answers "created, but some lines refused". | Report: Message only. The log shows plain "Synced". | Business | Ops | Open the invoice in billing, see which lines are missing and add them or follow up with finance. **The bot will not resend it.** Every later run shows it as Duplicate. | Billing gives no way to see the rejected lines. |
| E07 | Synced - "Created (confirmed after retry)" | Sending failed, so before retrying the bot asked billing whether this invoice number exists, and billing said yes. | Report: Message | System | Ops | Spot-check in billing that the invoice under this number has the right customer and amount. Usually it is the bot's own earlier attempt. If it is an older, different invoice, this one was **not** created. | The invoice in billing doesn't match the export. |
| E08 | Failed - "No success after 5 tries" | Billing didn't answer, timed out, said "too many requests", had a server error, or answered with an error the bot doesn't recognise, 5 times in a row. | Report: Status, Message. The log shows "Failed" with no reason. | System | Bot, then Ops | **Check billing first** for this invoice number before entering anything by hand. The next hourly run tries again if the invoice is still in the export. It then shows Synced, Duplicate (already there) or Failed again. | The same invoice is Failed in 3 runs in a row. |
| E09 | **Every** invoice in a run is Failed, and the run is much slower than usual | Every send is refused: wrong or missing API key, wrong billing address, or billing down. The bot can't tell these apart and retries each one. | Report: every row Failed. Log: every row Failed. | System | Dev | Nothing to fix on the ops side. Post in #rpa-support right away. | Always. |
| E10 | No new report file, or a report that stops part-way, and **no "run done" line** in `sync.log` for that hour | The bot crashed. Known causes: the output folder is missing; the network dropped while the bot was checking billing after a failed send; billing sent an answer the bot couldn't read; the input file is missing or unreadable. | Only the missing "run done" line and the partial report. **The crash reason is not written to any file ops can open.** | System | Dev | Check that `data/output` exists, and create it if missing. Otherwise escalate. Invoices after the crash point were not sent. | The output folder exists and the next hourly run also has no "run done". |
| E11 | Every hour the report stops at **the same invoice**, and invoices after it never get a row | One invoice makes billing send an answer the bot can't read. The bot crashes there each hour (as in E10), so every row after it is blocked. | Partial report, ending just before the same invoice each hour. No "run done". | System | Ops, then Dev | Note the first invoice that is missing from the report. Ask whoever prepares the export to take that row out for now, so the rest can sync. | Always: send the invoice number. |
| E12 | Every row Invalid - "missing invoice_no" although the column is filled | The file was saved with a hidden marker at the start (Excel's "CSV UTF-8" option), so the bot doesn't recognise the first column name. | Report: every row Invalid | Business | Ops, then Dev | Re-save the export as plain "CSV (Comma delimited)" and wait for the next run. | Re-saving doesn't help, or names with accents then make the run crash (E10). |
| E13 | A run is still going when the next hour starts, or two report files with close times overlap | Each invoice that keeps failing can take up to about 6 minutes, so a file with many such invoices runs past the hour. Nothing stops a second run from starting. | `sync.log`: lines from two runs mixed together. A report file may be overwritten if two runs start in the same minute. | System | Bot, then Dev | Wait for both to finish. Results are still safe because Duplicate protects against double invoices. | It happens 2 hours in a row, or E09 is the cause. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing doesn't answer within 30 seconds, the network drops, billing says "too many requests", or billing gives any error the bot doesn't recognise | Asks billing whether the invoice already exists. If yes: Synced (E07). If not: waits and sends again. | 5 tries. Waits 2, 4, 8, 16 and 32 s between them (about 1 minute), plus up to 30 s per call. **Worst case about 6 minutes per invoice.** | Row = Failed (E08). The bot goes to the next invoice. |
| Billing timed out but had actually created the invoice | The existence check above finds it, so no second invoice is created. | Same as above | Synced - "Created (confirmed after retry)" |
| An invoice failed this hour | Nothing is resent within this run. The **next hourly run** sends the whole input file again. | Only while the invoice is still in the export | Shows Duplicate once it has gone through |
| The bot crashed | **Nothing.** There is no restart, resume or alert. The next run starts on the hour as usual. | - | See E10 / E11 |
## 9. Known incidents
Both incidents are recorded only in the version history. Their dates were not recorded, but both were fixed before 0.9.1.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | The same invoice appeared twice in billing. | Billing created the invoice but answered "timed out", so the bot sent it again. | Fixed: the bot now checks whether the invoice exists before every retry (E07). Still spot-check "confirmed after retry" rows. Removing a duplicate in billing needs billing or dev access. | Partly |
| Before 0.9.1 | The hourly run died part-way. Invoices after that point got no report row, and there was no "run done" line. | Billing answered "too many requests" and the bot had no retry. | Fixed: up to 5 tries with waits (section 8). A network drop during the existence check can still crash the run the same way (E10). | Partly: ops could see the missing "run done" and wait for the next hour. |
## 10. Releases and version check
- **How an update arrives:** the bot has no updater. The RPA dev team replaces the files in the bot folder on the machine.
- **How to confirm it applied:** the bot shows no version anywhere ops can see. Ask dev in #rpa-support to confirm the version (currently 0.9.1) after every update.
- **If the version did not change:** report it in #rpa-support. Ops can't update the bot themselves.
## 11. Escalating to dev
Post in **#rpa-support** (Teams) with:
- the exception ID from section 7 (e.g. E10);
- the date and hour of the run;
- the report file name (e.g. `sync_report_20260903_0900.csv`), the name only;
- the invoice numbers involved;
- the matching lines from `logs/sync.log`, or say "no run done line".
**Never send:** the `.env` settings file, API keys or passwords. Don't paste or attach report files either, because they contain customer names. Give the file name and dev will open it.
## 12. Failure-class check
- **F1** Missing prerequisite: covered in section 3 (output folder, logs folder, settings file), E09 and E10, and Fix #2 and #9.
- **F2** not applicable: the bot drives no windows. It calls the billing web API with no screen, mouse or keyboard.
- **F3** Target slower than expected: covered in E08 and E13, section 8 (30 s wait, 5 tries) and the section 4 "stuck" rule.
- **F4** Input read differently: covered in the section 3 input rules (amount format, header, CSV type) and E01-E03 and E12, and Fix #10.
- **F5** Marked done when nothing was done: covered in E07 ("confirmed after retry" can be an older invoice) and Fix #7.
- **F6** Action happened but not confirmed: covered in E07 and E08 ("check billing first"), and incident 1.
- **F7** Business exception shown as success: covered in Statuses and E06 (partly rejected invoice shown as Synced), and Fix #6.
- **F8** Endless re-runs / watchdog: there is no watchdog or auto-rerun. The hourly re-send of the whole file is covered in sections 2 and 8, and the overlap risk in E13 and Fix #11.
- **F9** Stopping the wrong way: covered in section 4 (end the Task Scheduler task; a partial report is safe).
- **F10** One item blocks the queue: covered in E11 (one unreadable billing answer crashes the run at the same invoice every hour), and Fix #3 and #4.
- **F11** Cause hidden from ops: covered in sections 5 and E10. Crash and network messages are not saved to any file, and the log has no reason. See Fix #5 and #12.
- **F12** Update didn't arrive: covered in section 10 (no version visible), and Fix #13.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | `.env` (API key) is committed to git | The key is exposed to anyone with repo access. Rotate the key and untrack the file. | `.env` (tracked); loaded at `sync.py:17-19` |
| 2 | `data/output/` is not in the repo and not created by the code | A fresh install crashes every run before processing anything (E10) | `sync.py:68-69` |
| 3 | `exists()` has no error handling and is called right after network errors | A network drop during the duplicate check crashes the whole run (E10). This is the 429 incident again by another route. | `sync.py:41-43`, `sync.py:60` |
| 4 | `r.json()` on a 422 with a non-JSON body is unguarded | One bad billing answer crashes the run, at the same row every hour (E11) | `sync.py:59` |
| 5 | Network errors and `CRASH:` go to `print()` only | The cause of a crash is lost under Task Scheduler. Ops only sees a missing "run done". | `sync.py:52`, `sync.py:84-85` |
| 6 | HTTP 207 (some lines rejected) is stored as Synced | Ops treats an incomplete invoice as done (E06) | `sync.py:54-55` |
| 7 | "Created (confirmed after retry)" only checks that the invoice number exists, and also fires after non-retryable errors such as 400 | A pre-existing or different invoice is reported as Synced, and this one was never created (E07) | `sync.py:60-61` |
| 8 | 400/401/403/404 are retried 5 times like temporary errors | A bad key makes every row wait about 1 min and then Fail. This drives runs past the hour (E09, E13). | `sync.py:60-62` |
| 9 | A missing API key silently sends an empty token. A missing URL silently falls back to a built-in default address. | Misconfiguration looks like E09 instead of a clear startup error | `sync.py:18-19` |
| 10 | Input is read as `utf-8`, not `utf-8-sig` | An Excel "CSV UTF-8" export makes every row "missing invoice_no" (E12). A cp1252 export with accents crashes. | `sync.py:69` |
| 11 | No guard against overlapping runs, and the report name only goes down to the minute | Two runs can overlap and overwrite each other's report (E13) | `sync.py:68` |
| 12 | `sync.log` holds only invoice number + status, with no message, HTTP code or attempt count | Ops can't answer "did it retry / why" from the log. "Synced" hides 207 rows. | `sync.py:75` |
| 13 | No version anywhere ops can see. A stray commit message in the README and two in the code. | Ops can't confirm an update (section 10) | `README.md:5-6`, `sync.py:86-87` |
| 14 | The Task Scheduler task is not in the repo (schedule, "do not start a new instance", stdout capture) | No one can check how the bot is actually started or stopped | `sync.py:3` |
| 15 | No alert on Failed rows or a crash | Problems are found only if someone opens the report or log | `sync.py:63`, `sync.py:80-85` |
| 16 | Sleeps 32 s after the 5th try before giving up | Adds 32 s to every Failed invoice for nothing | `sync.py:62-63` |
| 17 | `BOT_WAS_RUN` marker is undocumented | Unclear purpose. Ops may delete it or worry about it. | `sync.py:67` |
| 18 | No regression tests for the 429 and 504 fixes. Both fix commits only add a comment. | The incident fixes can regress unnoticed, and history doesn't show what changed | `sync.py:46-63` |
tests/evals/results/handoff/h-api-sync-screens-r1.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-api-sync/screens",
"rep": 1,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"retry limit": true,
"console-only crash output flagged": true
},
"cost_usd": 1.3940436000000003,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-api-sync-screens-8tutneru\\project",
"reply": "I wrote `HANDOVER.md` in the repo root. I read the code and git history only; the bot was not run and no code was changed. The verification was done by reading through the doc, so it's not a test of the bot.\n\n| Item | Result |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 12 (E01\u2013E12) |\n| Failure classes | 11 covered, 1 not applicable (F2: the bot has no windows, it only calls a web service) |\n| Statuses | All 4 (Synced / Duplicate / Invalid / Failed) and all 9 message strings are covered |\n| PHI, keys, IPs | None in the doc; there are no file:line refs outside section 13 |\n\n**Your access answer vs the code:**\n- The bot has **no screens**. It is a background script that Task Scheduler starts every hour. So for ops, only `data/output/` is usable.\n- There **is** a log, `logs/sync.log`. I read your answer as \"ops cannot open it\" and did not point ops at it anywhere.\n\n**Most important \"Fix before handover\" items (17 in total):**\n1. The duplicate check before a retry has no error handling, so one network error crashes the whole run. That is the same symptom as the old 429 incident.\n2. The live billing API key is in `.env`, which is committed to git. It needs to be rotated.\n3. Nothing in the output folder shows whether a run finished or crashed. The \"run done\" line only goes to the log.\n4. A partial create (some lines rejected) still shows as \"Synced\".\n5. A wrong API key is retried 5 times on every invoice and gives no reason, so runs can overlap.\n6. The `data/output/` folder is not created by the code, so a fresh install crashes on every run.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: are there any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-api-sync-screens-r1.md
+177
View File
@@ -0,0 +1,177 @@
# Invoice sync - ops handover
Version 0.9.1 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Every hour it takes the approved invoices from the finance export file (`data/input/invoices.csv`) and creates each one in the billing system through the billing system's web service. For every invoice in the file it writes one line to a report file in `data/output/`, saying whether the invoice was created, was already there, was rejected, or failed. It has **no window and no screens**: it runs in the background from Windows Task Scheduler, and it drives no desktop apps or websites. Ops works from the report files in the output folder.
## 2. How it works
```
Windows Task Scheduler (every hour)
└─ starts → Invoice sync (background script, no window)
│
├─ reads ← data/input/invoices.csv (finance export)
├─ sends ──> Billing system web service (creates invoices)
├─ writes → data/output/sync_report_<date>_<time>.csv (ops can open)
└─ writes → logs/sync.log (ops cannot open)
```
- **Task Scheduler** starts the bot once an hour. Ops has no access to it.
- **Invoice sync** reads the whole export from top to bottom, one invoice at a time.
- **The finance export** is the input. Every invoice still in it is sent again every hour.
- **The billing web service** creates the invoice, or answers "already exists", "rejected" or with an error.
- **The report file** has one line per invoice: invoice number, customer, Status, Message. A new file is made every run, named by date and time (for example `sync_report_20261023_1400.csv` = 23 Oct 2026, 14:00).
- **The log file** holds only the invoice number and status, plus a "run done" line. Ops cannot open it, and it doesn't hold the reasons anyway.
**One run, step by step:**
1. Task Scheduler starts the bot on the hour.
2. The bot creates a new, empty report file in `data/output/`.
3. For each invoice in the export, it checks that the invoice number, customer ID, amount and currency are filled in and that the amount is a positive number. If not, the line is **Invalid** and it moves on.
4. It sends the invoice to billing. If billing doesn't answer clearly, it checks whether billing has the invoice anyway, waits and tries again, up to 5 tries (section 8).
5. It writes the result line to the report and goes to the next invoice.
6. After the last invoice the run ends. Nothing in the report marks the end (see E11).
## 3. Before a run
Ops cannot change the bot machine. These items are for whoever sets it up; ops only needs to know them so the symptom makes sense.
- [ ] The bot folder has a `.env` settings file with the billing address and access key. If it is missing or the key is wrong or expired, every invoice ends **Failed** (E10).
- [ ] The folders `data/output/` and `logs/` exist in the bot folder. The bot does not create them. If one is missing, every run crashes and no report appears (E11).
- [ ] Python is installed on the bot machine with the two packages listed in `requirements.txt`.
- [ ] The finance export follows these rules (if it doesn't, ask whoever produces the export to fix it):
| Rule | Bad | Good |
|---|---|---|
| Column headers exactly, lower case: `invoice_no, customer_id, customer_name, amount, currency` | `Invoice No` | `invoice_no` |
| Invoice number, customer ID, amount and currency are always filled in (customer name may be empty) | amount empty | `1200.50` |
| Amount is a plain number: dot for decimals, no thousands separator, no currency sign | `1,200.50` or `$1200.50` | `1200.50` |
| Amount is above zero. Zero and negative amounts (credit notes) are never synced | `-50.00` | `50.00` |
| The file comes straight from the finance system. **Don't re-save it in Excel**: Excel's "CSV UTF-8" option makes every line Invalid "missing invoice_no", and other Excel CSV options can crash the run when names have accents | re-saved in Excel | original export |
| No extra columns. The bot sends every column to billing, so an extra column can be rejected or leak data into billing | an added "notes" column | the five columns only |
Sample of the layout: `data/input/invoices.csv` (headers only - don't copy its rows).
## 4. During a run
- Don't move, rename or delete files in `data/output/` while a run is going. The newest report is still being written.
- Don't open the newest report until its "modified" time has stopped changing for a few minutes.
- **To stop the bot:** there is no Stop button. It can only be stopped in Task Scheduler, which ops cannot reach. Raise a ticket. Stopping it mid-invoice can leave that one invoice created in billing but missing from the report.
- **A run looks stuck** when the newest report's modified time hasn't changed for more than 6 minutes. One invoice can legitimately take up to about 6 minutes while billing is not answering (section 8). The bot has no watchdog: it never gives up on the run as a whole, it only gives up on one invoice at a time. A run with many failing invoices can last longer than an hour (E12).
## 5. Where to look when something goes wrong
Ops can open only the output folder `data/output/`. Everything below uses it.
| Question | Where to look |
|---|---|
| Did the run start? | A report file exists for that hour: `sync_report_<date>_<hour><minute>.csv`. No file for the hour = the run did not start or crashed at once (E11). |
| Did the run finish? | **Ops cannot tell for sure.** The report has no end line. If it has fewer invoices than the export, or stops at the same invoice every hour, treat it as a crash (E11). |
| Which invoices failed? | Open the report, filter the **Status** column for `Failed`, `Invalid`, and `Synced` lines whose Message says "some lines rejected" or "confirmed after retry". |
| Why did one invoice fail? | The **Message** column on that invoice's line. That is the only reason the bot gives ops. For `Failed` there is no further detail anywhere ops can see. |
| Did the bot already retry? | Yes, always, for every `Failed` line (5 tries). A `Synced` line with "confirmed after retry" also went through a retry. The next hourly run tries again any invoice still in the export. |
| Which version is running? | Ops cannot see it. Ask in the ticket (section 10). |
## 6. Statuses
The report is a plain CSV file: it has **no colours**. The Status is the third column, the Message the fourth.
| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | Invoice created in billing | nothing |
| Synced | Created (confirmed after retry) | The first send had no clear answer; the bot then found the invoice in billing | **Check billing for a second copy** (E07) |
| Synced | Created (some lines rejected) | **Needs follow-up.** Invoice created, but billing refused some of its lines | Fix the missing lines in billing (E08) |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number. Normal for every invoice synced in an earlier hour | nothing, unless the invoice is new (E06) |
| Invalid | missing <field names> | A required field is empty in the export | Fix the export (E01, E02) |
| Invalid | amount must be positive | Amount is zero or negative | Business decision (E03) |
| Invalid | amount '<value>' is not a number | Amount is written in a way the bot can't read | Fix the export (E04) |
| Invalid | billing's own reason text, or "Rejected by billing" | Billing checked the invoice and refused it | Read the reason, fix the data (E05) |
| Failed | No success after 5 tries | Billing never confirmed it after 5 tries. **It may still have been created** | Check billing first (E09, E10) |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing customer_id" (or another field name) on some lines | Before sending, it checks that invoice number, customer ID, amount and currency are not empty | Report, Status and Message | Business | Ops | Ask the export owner to fill the field. The next hourly run picks the invoice up. | The field is filled in the export but still reported missing |
| E02 | Invalid - "missing invoice_no" (often with other fields) on **every** line | Same check, but the bot can't find the invoice number column at all: the file was re-saved in Excel as "CSV UTF-8", or the headers were renamed | Report, Status and Message | Business | Ops, then Dev | Ask for the file straight from the finance system with the exact headers (section 3). | The file is an untouched export and every line is still Invalid. Send: report file name, E02. |
| E03 | Invalid - "amount must be positive" | Amount is zero or below | Report, Status and Message | Business | Ops | Zero and negative invoices are never synced by this bot. Handle them in billing by hand if the business needs them. | The amount in the export is above zero |
| E04 | Invalid - "amount '1,200.50' is not a number" (shows the value) | The amount can't be read as a number: comma separator, currency sign, text | Report, Status and Message | Business | Ops | Ask for the amount as a plain number, e.g. `1200.50`. | The value shown looks like a plain number |
| E05 | Invalid - billing's own reason (for example about the customer), or "Rejected by billing" | Billing answered that the invoice data is not acceptable. The bot does not retry this | Report, Status and Message | Business | Ops | Read the reason. Fix the data in the export, or in billing (for example a customer ID billing doesn't know). | The reason is unclear, or every line has it. Send: report file name, invoice numbers, E05. |
| E06 | Duplicate - "Invoice already exists in billing" | Billing answered that the invoice number already exists. No retry | Report, Status and Message | Business | Ops | Normally nothing: every invoice synced in an earlier hour shows this every hour after. If the invoice is **new**, look it up in billing: another invoice may already use that number. | Billing shows no invoice with that number. Send: invoice number, report file name, E06. |
| E07 | Synced - "Created (confirmed after retry)" | The first send timed out or billing gave an error. Before trying again, the bot looked the invoice number up in billing and found it | Report, Status and Message | System | Bot, then Ops | Look the invoice up in billing and make sure there is **only one** copy (see the 504 incident, section 9). Remove an extra copy using the billing team's process. | Two copies found. Send: invoice number, report file name, E07. |
| E08 | Synced - "Created (some lines rejected)" | Billing created the invoice but said part of it was refused. The bot records this as Synced anyway | Report, Status and Message | Business | Ops | **Not finished despite "Synced".** Open the invoice in billing, compare its lines with finance, add the missing lines with the finance team. The report doesn't say which lines were refused. | You can't work out which lines are missing. Send: invoice number, report file name, E08. |
| E09 | Failed - "No success after 5 tries" on a **few** lines | Billing didn't answer, timed out, or gave an error 5 times in a row (section 8), and the invoice number was never found in billing | Report, Status and Message | System | Bot, then Ops | **Check billing first.** The invoice may have been created even though the bot couldn't confirm it. Found: nothing to do, the next run will show Duplicate. Not found: wait for the next hourly run, which tries again if the invoice is still in the export. | The same invoice is Failed in 2 runs in a row. Send: invoice number, both report file names, E09. |
| E10 | Failed - "No success after 5 tries" on **all or most** lines | Same as E09, for every invoice. Usually billing is down, or the bot's billing address or access key is wrong or expired. The bot can't tell these apart and gives no reason | Report, Status and Message | System | Dev | Raise a ticket at once. Expect the run to be slow: each invoice takes over a minute (E12). | Always. Send: report file name, number of Failed lines, time noticed, E10. |
| E11 | No report for the hour, or a report that stops part-way (fewer invoices than the export, often stopping at the same invoice every hour) | The run crashed. Known causes: the billing look-up after a timeout failed too; billing refused an invoice with an answer the bot couldn't read; the `data/output` or `logs` folder is missing; the export file is missing or saved in an unexpected encoding. The cause is not recorded anywhere ops can see | Nothing ops can see. The report simply ends | System | Dev | Raise a ticket. Don't re-run anything: the next hourly run starts again from the top and will probably stop in the same place. Check billing for the invoice **after** the last line in the report: it may have been created. | Always. Send: report file name (or "no report for <hour>"), the last invoice number in the report, E11. |
| E12 | The newest report is still growing after the next hour has started, or an hour's report is missing while the previous one is still being written | The run is longer than an hour, usually because many invoices are failing slowly (E10). Nothing stops two runs from overlapping | Report file times in `data/output/` | System | Dev | Raise a ticket. Two runs at once can send the same invoice twice. | Always. Send: the report file names and their times, E12. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing doesn't answer, times out, is busy, or gives an error (this includes a wrong access key) | Waits up to 30 s for an answer. Then checks whether billing has the invoice anyway; if it does, records Synced "confirmed after retry". If not, waits and sends again | 5 tries. Waits of 2, 4, 8, 16 and 32 s between tries. Worst case about **6 minutes for one invoice** | Line = Failed "No success after 5 tries", next invoice |
| Billing says the invoice already exists | No retry | - | Line = Duplicate, next invoice |
| Billing refuses the invoice data | No retry | - | Line = Invalid with billing's reason, next invoice |
| Invoice not synced this hour (Failed, Invalid, or after a crash) | The next hourly run sends every invoice in the export again, from the top | Every hour, for as long as the invoice is in the export | Already-created invoices come back as Duplicate |
| Run crash | **No recovery inside the run.** The rest of the export is not processed until the next hour | - | Next hourly run starts from the top (E11) |
There is no watchdog and no automatic restart. A Failed line is final for that run; ops waits for the next hourly report.
## 9. Known incidents
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | The same invoice appeared twice in billing | Billing timed out after it had already created the invoice; the bot sent it again and billing accepted the second copy | Fixed: the bot now looks the invoice up before every retry. Still check every "confirmed after retry" and every Failed line in billing (E07, E09). | Partly |
| Before 0.9.1 | The hourly run died part-way; invoices after that point got no report line | Billing limited how fast the bot could send (rate limit) and the bot didn't retry | Fixed for the send: it now retries with waits. The look-up after a timeout can still crash the run the same way (E11, section 13 item 1). | Partly |
## 10. Releases and version check
- How an update arrives: not handled by the bot. There is no launcher or auto-update; a developer replaces the bot's files on the bot machine.
- How to confirm it applied: ops can't. The version (0.9.1) is written only in the project's `README.md`, which ops can't open. Ask the developer to confirm the version in the ticket.
- If the version did not change: raise a ticket in RPA Support.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue.
Send: the exception ID from section 7, the report file name (it carries the date and time of the run), the invoice numbers affected, and the time you noticed. For a crash (E11), the last invoice number in the report.
Never send: the `.env` settings file, the billing access key, or customer names and amounts pasted into chat. The report files contain customer names: give the file name and let the developer open it.
## 12. Failure-class check
- F1 covered in section 3 (settings file, folders, packages) and E10 / E11.
- F2 not applicable: the bot has no windows and drives no desktop app; it only talks to a web service.
- F3 covered in E07, E09, E10 and section 8 (30 s wait, 5 tries).
- F4 covered in section 3 input rules and E01-E04; also section 13 items 8 and 9.
- F5 covered in E07 (Synced only when billing confirms or the invoice is found by number). Remaining gap: section 13 item 16.
- F6 covered in E09 and E11 (check billing before assuming not created); section 13 items 1 and 16.
- F7 covered in section 6 and E08 ("some lines rejected" shown as Synced); section 13 item 5.
- F8 covered in section 8 and E12: no loop limit beyond 5 tries per invoice, no watchdog, runs can overlap; section 13 items 3 and 10.
- F9 covered in section 4: no Stop button, Task Scheduler only; section 13 item 17.
- F10 covered in E11: one invoice that crashes the run blocks every invoice after it, every hour; section 13 items 1 and 2.
- F11 covered in section 5 and section 13 items 4 and 6: reasons for Failed and for crashes are not shown anywhere ops can open.
- F12 covered in section 10: no auto-update and no version ops can see; section 13 item 12.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The existence check has no error handling. A timeout or connection error there crashes the whole run: the same symptom as the 429 incident, on the second call | Run stops mid-file, every hour, at the same invoice (E11) | `sync.py:41-43`, called at `sync.py:60` |
| 2 | A 422 with a non-JSON body (e.g. HTML error page) raises in `r.json()` and crashes the run | Same as above (E11) | `sync.py:59` |
| 3 | Every non-success status is retried, including 400/401/403/404; the docstring says only 429/5xx. A bad key makes every invoice take ~62 s+ and end Failed; the run doesn't stop early | Slow runs that overlap the next hour, no clue it's the key (E10, E12) | `sync.py:47`, `sync.py:60-62` |
| 4 | Failed message has no HTTP status or reason; the log records only invoice + status; network errors and the CRASH line go to stdout, lost under Task Scheduler | Ops (and dev) can't tell a rate limit from an auth failure from an outage | `sync.py:52`, `sync.py:63`, `sync.py:75`, `sync.py:84` |
| 5 | HTTP 207 (partial) is stored as Status `Synced` | Ops filtering on Status misses invoices that need follow-up (E08) | `sync.py:54-55` |
| 6 | The "run done" line goes only to `logs/sync.log` (no ops access); the `BOT_WAS_RUN` marker is in the bot folder and holds only "1". Nothing in the output folder says a run finished or crashed | Ops can't tell a finished run from a crashed one. Suggest a final summary line or a `.done` / `.failed` file in `data/output/` | `sync.py:67`, `sync.py:77`, `sync.py:80-85` |
| 7 | `data/output/` and `logs/` are not created by the code and `data/output/` isn't in the repo. A fresh install crashes every run | No report at all (E11) | `sync.py:24`, `sync.py:68-69` |
| 8 | Input opened as `utf-8`, not `utf-8-sig`. A BOM (Excel "CSV UTF-8") makes every line Invalid "missing invoice_no"; a non-UTF-8 file crashes the run | E02, E11 | `sync.py:69` |
| 9 | The whole CSV row, including any extra columns and `customer_name`, is posted as JSON | Extra columns can be rejected or leak into billing | `sync.py:50` |
| 10 | No lock against overlapping runs; report name is to the minute, so two runs in one minute overwrite one file. Task Scheduler's "if already running" setting isn't documented | Same invoice sent twice at once; lost report (E12) | `sync.py:68`; Task Scheduler task |
| 11 | `Retry-After` is ignored on 429; the 32 s sleep after the 5th try is wasted | Longer runs | `sync.py:62` |
| 12 | Version shown only in README; nothing ops can see shows it. README also has a stray commit-message line | Can't confirm an update applied | `README.md:5-6` |
| 13 | `.env` with the live billing key is committed (initial import) | Key exposed in git history: rotate it, remove `.env` from the repo, add to `.gitignore` | `.env`, commit 88b2057 |
| 14 | Customer names are copied into every report; no retention or access rule for `data/output/` | Customer data piles up in a folder ops opens daily | `sync.py:76` |
| 15 | Both fix commits only add comments; there are no tests | Nothing proves the 429 / 504 fixes work or keeps them from regressing | `sync.py:86-87` |
| 16 | Every hourly run re-posts every invoice in the export and relies on billing answering 409. The 504 incident shows billing accepted a second copy at least once; the existence check only matches by invoice number | Duplicate invoices remain possible (E07, E09) | `sync.py:72-74`, `sync.py:41-43` |
| 17 | No stop mechanism ops can use; killing mid-POST leaves the invoice unrecorded | Ops must open a ticket to stop a runaway run | `sync.py:80-85`; Task Scheduler task |
tests/evals/results/handoff/h-api-sync-screens-r2.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-api-sync/screens",
"rep": 2,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"retry limit": true,
"console-only crash output flagged": true
},
"cost_usd": 1.5065946000000001,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-api-sync-screens-456f_fnz\\project",
"reply": "I've written `HANDOVER.md` in the repo root, following the skill. Bot run pannala, code change edhuvum illa; written file `HANDOVER.md` mattum dhaan.\n\n| Check | Result |\n|---|---|\n| Sections | 13, template order-la |\n| Exceptions catalogued | 13 (E01\u2013E13) |\n| Failure classes | 11 covered, 1 not applicable (F2: the bot drives no windows, it just calls the billing API) |\n| Statuses | All 9 status/message combinations the bot writes are listed. CSV report, so no colours |\n| Where to look | Only the output folder and report files, as you said. The log is marked dev-only |\n| Secrets / customer data | Doc-la `.env` values, keys, customer names, IPs edhuvum illa |\n\n**Fix before handover \u2014 16 items. Top ones:**\n1. The `.env` file with the live API key was committed to git in the first commit, and there's no `.gitignore`. **Rotate the key.**\n2. When billing rejects only some lines of an invoice, the row still shows \"Synced\", and which lines were dropped isn't recorded.\n3. If billing times out during the \"does this invoice already exist?\" check, the whole run stops. The report just ends partway, with no reason.\n4. Duplicate invoices in billing are still possible.\n5. Crashes and network errors go only to the console, so ops can't tell a finished run from a stopped one.\n6. `data/output` isn't in the repo and the code doesn't create it, so the first run on a new machine stops with no report.\n\n**Also noticed:** the two \"fix\" commits only added comment lines; the actual retry and duplicate-check code was already in the first commit. So git history doesn't show when those fixes really went in.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions mean whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-api-sync-screens-r2.md
+179
View File
@@ -0,0 +1,179 @@
# Invoice sync - ops handover
Version 0.9.1 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Every hour, the bot sends approved invoices from the finance export file (`data/input/invoices.csv`) into the billing system. It goes through the file one invoice at a time. First it checks that the invoice has an invoice number, customer, amount and currency, and that the amount is a positive number. Then it sends the invoice to billing. For every invoice it writes one row in a report file, `data/output/sync_report_<date>_<time>.csv`, giving the result and the reason. The bot has no window and no screens. It runs in the background, started by Windows Task Scheduler. It does not drive any desktop app or website. It talks to the billing system directly over the network.
## 2. How it works
```
Windows Task Scheduler (every hour)
└─ starts → Invoice sync (background, no window)
│
├─ reads → data/input/invoices.csv (finance export)
├─ sends → Billing system (over the network)
├─ writes → data/output/sync_report_<date>_<time>.csv ← ops reads this
└─ writes → logs/sync.log (dev only)
```
- **Task Scheduler** starts the bot at the top of every hour.
- **Invoice sync** is the bot itself. It has no window, so you never see it running.
- **The finance export** is the bot's only input. Each run reads the **whole** file again from the top.
- **The billing system** is where invoices end up. You can look up an invoice there to see whether it really arrived.
- **The report file** gets one new file per run, with one row per invoice. This is the only place ops can see results.
- **The log** holds one short line per invoice with no reason. Ops cannot open it; dev uses it.
**One run, step by step:**
1. The bot starts and creates an empty report for this hour.
2. It reads the next invoice from the finance export.
3. It checks the invoice. If a field is missing or the amount is wrong, the row is **Invalid** and it moves on.
4. It sends the invoice to billing.
5. If billing gives no clear answer, the bot first checks whether billing already has the invoice. If billing does not have it, the bot waits and tries again, up to 5 times.
6. It writes the result (Synced, Duplicate, Invalid or Failed) and a message into the report.
7. It repeats steps 2-6 until the file ends, then the run is over.
## 3. Before a run
These are mostly for dev or whoever sets up the machine and prepares the export. Ops should know them, because each one has a symptom in section 7.
- [ ] The settings file (`.env`) is in the bot's folder and holds the billing address and a **valid, current** API key. If the key is missing or expired, the bot does not warn. Every invoice ends as **Failed** (E10).
- [ ] The folders `data/output` and `logs` exist in the bot's folder. If either is missing, the run stops at once and **no report file appears** (E01).
- [ ] Python and the bot's two add-on packages are installed on the machine. If not, no report file appears (E01).
- [ ] Only **one** machine has the hourly task. Two machines running it would send the same invoices twice.
- [ ] Input rules for the finance export:
- **Export the file straight from the finance system. Do not open it and save it again in Excel.** A file re-saved by Excel can make every row show "missing invoice_no" (E03).
- **Amounts are plain numbers, no thousands separator and no currency sign.** Bad: `1,250.00`, `₹500`. Good: `1250.00`, `500`.
- **Amounts must be above zero.** Credit notes and zero invoices are rejected as Invalid (E04). The bot does not handle them.
- Every row needs invoice_no, customer_id, amount and currency filled in. customer_name is copied into the report but is not checked.
## 4. During a run
- **Ops does not need to do anything while a run is going on.** There is no window to watch and nothing to click.
- The current hour's report is written while the bot works. If you open it during the run you may see only part of it. **Judge an hour's report only once the next hour's report has appeared.**
- **To stop the bot:** ops cannot stop it. If it must stop (for example, it is sending wrong invoices), raise an **urgent** ticket in RPA Support. Dev disables the hourly task. Closing a report file or the billing system does **not** stop it.
- **How long a run takes:** usually a few minutes. Each invoice that billing does not answer can take **up to about 6 minutes** before it is marked Failed. A file with many failures can run past the hour.
- **When to worry:** no new report file in `data/output` for **2 hours in a row**. See E01.
## 5. Where to look when something goes wrong
Ops can open only the output folder `data/output` and the report files in it. Everything else is dev-only.
| Question | Where to look |
|---|---|
| Did the run happen? | `data/output`: is there a `sync_report_<date>_<time>.csv` for that hour? No file means the run did not start, or it stopped at once (E01). |
| Did the run finish? | **You cannot tell for sure from the report.** If a report has far fewer rows than the hour before, the run probably stopped partway (E11). See Fix before handover #7. |
| Which invoices failed? | Open the report and filter the **Status** column for `Failed` and `Invalid`. |
| Why did one invoice fail? | The **Message** column in the same row. |
| Did the bot already retry? | Every `Failed` row has **already** been tried 5 times. A `Synced` row with "confirmed after retry" was also retried. Every invoice still in the export is tried again in the next hourly run. |
| Did the invoice really reach billing? | Search for the invoice number in the billing system. |
| Which version is running? | Ops cannot see this. Ask in the ticket (Fix before handover #12). |
## 6. Statuses
The report is a plain CSV file, so there are no colours. Status is the **Status** column and the message is the **Message** column.
| Status - Message | Colour | Meaning | What ops does |
|---|---|---|---|
| Synced - "Created" | none | Billing created the invoice | nothing |
| Synced - "Created (confirmed after retry)" | none | The first send got no clear answer. The bot then checked and found the invoice in billing | nothing (see E09) |
| **Synced - "Created (some lines rejected)"** | none | **Needs follow-up.** Billing created the invoice but **dropped some of its lines**. The report does not say which ones | Check the invoice in billing against the export (E08) |
| Duplicate - "Invoice already exists in billing" | none | Billing already has this invoice number. Normal for invoices sent in earlier hours, because each run re-sends the whole export | nothing, unless the invoice is new (E07) |
| Invalid - "missing <field names>" | none | A required field is empty in the export. Not sent | Fix the export (E02 / E03) |
| Invalid - "amount must be positive" | none | Amount is zero or negative. Not sent | Handle by hand (E04) |
| Invalid - "amount '<value>' is not a number" | none | Amount has commas, a currency sign or text. Not sent | Fix the export (E05) |
| Invalid - "Rejected by billing", or billing's own reason text | none | Billing refused the invoice and said why. Not sent | Fix per the message (E06) |
| Failed - "No success after 5 tries" | none | Billing never gave a usable answer after 5 tries. **Not sent** | See E10 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | No report file in `data/output` for an hour | The bot stops before writing anything. Causes: the output or log folder is missing, the export file is missing, Python is not set up, or the hourly task did not run | Nowhere ops can see (dev: console output only) | System | Dev | Wait for the next hour. A missing hour is caught up by the next run, because each run re-reads the whole export | No report for **2 hours in a row**. Send: the hours with no report and a screenshot of the `data/output` folder |
| E02 | Invalid - "missing customer_id" (or other field names) on some rows | Before sending, the bot checks that invoice_no, customer_id, amount and currency are filled in | Report: Status, Message | Business | Ops | Ask finance to fill the field in the source system. The next hourly run picks the row up | The field is filled in the finance system but still reported missing |
| E03 | **Every** row is Invalid - "missing invoice_no" (and the invoice_no column in the report is empty) | The bot cannot read the first column heading, usually because the export was re-saved in Excel | Report: Status, Message | Business | Ops, then Dev | Ask finance to re-export the file straight from the finance system, not through Excel | It still happens after a clean export. Send: the report file name |
| E04 | Invalid - "amount must be positive" | The amount is zero or negative | Report: Status, Message | Business | Ops | The bot does not send credit notes or zero invoices. Handle them by hand in billing per your process, or ask finance to remove them from the export | Never (by design) |
| E05 | Invalid - "amount '<value>' is not a number" | The amount cannot be read as a number (commas, currency sign, text) | Report: Status, Message | Business | Ops | Ask finance to export the amount as a plain number, e.g. `1250.00` | The amount already looks like a plain number |
| E06 | Invalid - billing's own reason, or "Rejected by billing" | Billing refused the invoice and sent a reason | Report: Status, Message | Business | Ops | Read the message, then fix the data in finance (e.g. unknown customer, wrong currency) | The message is unclear or the data looks right. Send: report file name and invoice number |
| E07 | Duplicate - "Invoice already exists in billing" for an invoice that is **new** this hour | Billing says the invoice number is already in use | Report: Status, Message | Business | Ops | Search billing for the number. If it is our own invoice from a moment ago, nothing to do. If it is a **different** invoice with the same number, ask finance to renumber | Billing shows no invoice with that number |
| E08 | Synced - "Created (some lines rejected)" | Billing accepted the invoice but said some lines were rejected | Report: Status, Message. **Which lines is not recorded** | Business | Ops | Open the invoice in billing, compare its lines with the export, add or fix the missing lines by hand | You cannot work out which lines are missing. Send: report file name and invoice number |
| E09 | Synced - "Created (confirmed after retry)" | The first send timed out or got an error. Before trying again, the bot asked billing and found the invoice | Report: Status, Message | System | Bot | Nothing. If you want to be sure, check that billing has only **one** copy of the invoice (see E13) | Billing has two copies |
| E10 | Failed - "No success after 5 tries" | Billing gave no usable answer 5 times in a row (down, too busy, or the API key is wrong or expired) | Report: Status, Message | System | Bot, then Ops / Dev | **First check billing for the invoice.** If it is there, nothing to do. If not, wait: the next hourly run tries it again | **Every** row in a report is Failed (likely the API key), or the same invoice is Failed in **2 hourly reports in a row**. Send: report file names and invoice numbers |
| E11 | Report stops partway: far fewer rows than the hour before, and invoices you expected are missing from it | The run stopped on an error it does not handle. Typical cause: billing timed out while the bot was checking whether an invoice exists, or billing sent back an unreadable rejection | Nowhere ops can see. The report just ends | System | Bot, then Dev | Wait for the next hour. The next run re-reads the whole export and continues; invoices already sent show Duplicate | The report stops at the **same** invoice 2 hours in a row. Send: report file names and the last invoice number in each |
| E12 | The same invoice appears in every hourly report, never Synced | The export still contains it, and each run sends the whole file again | Report: one row per hour | Business / System | Ops, then Dev | For Invalid rows, fix the data (E02-E06). For Failed rows, see E10 | Failed for 2 hours or more. Send: the report file names |
| E13 | An invoice appears **twice** in billing | Billing created it, but its answer was lost, and billing's lookup did not show it yet, so the bot sent it again. The bot cannot catch this case | Not recorded. Seen only in billing | System | Ops, then Dev | Void the extra copy in billing per your billing process | Always. Send: invoice number, both report file names where it appears |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing does not answer, is busy, or answers with an error | Before each new try, checks whether billing already has the invoice (if yes: Synced, "confirmed after retry"). Otherwise waits and sends again | 5 tries, waiting 2, 4, 8, 16, 32 s between them. **Up to about 6 minutes per invoice** | Row = Failed, bot moves to the next invoice |
| An invoice was Failed, Invalid or skipped by a stopped run | Nothing during the run. The **next hourly run** reads the whole export again and retries every invoice in it | No limit. It is retried every hour for as long as it stays in the export | Same result every hour until the data or billing is fixed (E12) |
| The run stops partway (E11) | No restart within the hour | n/a | The next hourly run starts again from the top |
There is no watchdog, no automatic restart and no "resume". The hourly schedule is the only recovery.
## 9. Known incidents
Both incidents happened before v0.9.1. Git history does not hold real dates.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | Invoices appeared **twice** in billing | Billing timed out after it had already created the invoice, and the bot sent it again | Fixed: the bot now checks billing before each retry. It can still happen in rare cases (E13). Void the extra copy | Partly. Ops can find and void duplicates. Only code could prevent them |
| Before 0.9.1 | Hourly run stopped partway when billing said "too many requests"; later invoices not sent that hour | No retry for a busy billing system | Fixed: retries with waiting (section 8). A similar stop can still happen if billing times out during the check (E11) | Partly. Waiting for the next hour is safe |
## 10. Releases and version check
- How an update arrives: dev replaces the bot files on the machine by hand. There is no launcher and no automatic update.
- How to confirm it applied: ops cannot see the version. Ask dev to confirm the version in the ticket after any update.
- If behaviour did not change after an update: raise a ticket. Only one machine should run the bot, so there is only one place to update.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue.
Send:
- the exception ID from section 7 (e.g. E10)
- the report file name(s), e.g. `sync_report_<date>_<time>.csv`
- the invoice number(s)
- the time you noticed it
- for E01, a screenshot of the `data/output` folder
Never send: the `.env` settings file, API keys, or customer names and amounts in chat. Attach the report file to the ticket instead.
## 12. Failure-class check
- F1 covered in section 3 (settings file, folders, Python), E01, E10, and Fix #1, #6, #8.
- F2 not applicable: the bot drives no windows. It talks to billing over the network.
- F3 covered in E10, E11 and section 8. A slow billing system is handled by the 30 s wait and 5 retries; a timeout during the existence check is Fix #3.
- F4 covered in section 3 input rules, E03 and E05, and Fix #9.
- F5 covered in E08, and Fix #2 and #4.
- F6 covered in E09, E10 ("check billing first") and E13, and Fix #4.
- F7 covered in section 6 ("Created (some lines rejected)" needs follow-up) and E08, and Fix #2.
- F8 covered in E12 and section 8 (every invoice in the export is retried every hour, without limit), and Fix #10.
- F9 covered in section 4: ops raises an urgent ticket and dev disables the task. Fix #11.
- F10 covered in section 4 and E10: failing invoices slow the run (up to 6 min each) but never block other invoices. Fix #11.
- F11 covered in section 5 and E01 / E11: the real cause is often only in console output or the dev-only log. Fix #7.
- F12 covered in section 10. The version is not visible to ops. Fix #12.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The settings file with the live API key was committed to git in the first commit, and there is no `.gitignore` | Credential exposure. Rotate the key and remove the file from history | `.env`, commit `88b2057` |
| 2 | A partial reject from billing (207) is stored as Synced, and the rejected lines are not recorded | Ops sees "Synced" and assumes it is done (F7) | `sync.py:54-55` |
| 3 | The existence check has no error handling. A timeout there stops the whole run | Report stops partway with no reason (E11), the same symptom as the 429 incident | `sync.py:41-43`, `sync.py:60` |
| 4 | The existence check treats any non-200 (429, 5xx, 401) as "not there" and sends again. There is no idempotency key | Duplicates in billing are still possible (E13) | `sync.py:43`, `sync.py:50` |
| 5 | A 422 with a non-JSON body crashes the run | Report stops partway (E11) | `sync.py:59` |
| 6 | Every non-success code is retried, including 400/401/403. A missing API key silently becomes empty | An expired key shows as a generic "Failed" on every row, about a minute per row, with no hint | `sync.py:19`, `sync.py:60-63` |
| 7 | Network errors and crashes go to the console only. The log holds no reason, and the report has no "run finished" row. Ops cannot open the log | Ops cannot tell a finished run from a stopped one, or see why it stopped (F11) | `sync.py:52`, `sync.py:75`, `sync.py:77`, `sync.py:84` |
| 8 | `data/output` is not in the repo and not created by the code. `logs/` is also assumed to exist | On a fresh machine the first run stops with no report (E01) | `sync.py:24`, `sync.py:68-69` |
| 9 | The input is read as `utf-8`, not `utf-8-sig`. `nan` and `inf` amounts pass validation | An Excel-saved export makes every row Invalid (E03); bad amounts reach billing | `sync.py:69`, `sync.py:34` |
| 10 | The whole export is re-sent every hour, with no archive or processed marker | Hourly "Duplicate" noise; Failed rows retried forever (E12) | `sync.py:72` |
| 11 | No run lock and no time budget, and the report name only goes down to the minute | A long run can overlap the next one; two runs in the same minute overwrite each other's report | `sync.py:66-68` |
| 12 | The version exists only in the README (which also has a stray commit line) and is not shown anywhere ops can see | Ops cannot confirm an update applied (F12) | `README.md:5-6` |
| 13 | The two "fix" commits only added comment lines. The retry and existence-check code was already in the initial import | History does not show when the fixes actually landed | `sync.py:86-87`, commits `83a380f`, `b2dcd70` |
| 14 | A `BOT_WAS_RUN` marker file is written to the repo root on every run | Unexplained file in the bot folder | `sync.py:67` |
| 15 | The bot sleeps after the 5th failed try and ignores `Retry-After` | Adds 32 s to every Failed row and may keep hitting the rate limit | `sync.py:62` |
| 16 | The task definition for the hourly schedule is not in the repo | Rebuilding the machine depends on memory | n/a (missing file) |
tests/evals/results/handoff/h-api-sync-screens-r3.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-api-sync/screens",
"rep": 3,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"retry limit": true,
"console-only crash output flagged": true
},
"cost_usd": 1.2465458,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-api-sync-screens-xj0v_9bb\\project",
"reply": "`HANDOVER.md` ready: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-api-sync-screens-xj0v_9bb\\project\\HANDOVER.md`. Code edhuvum maathala, bot-ah run pannala.\n\n| Item | Count |\n|---|---|\n| Sections | 13 |\n| Exceptions | 14 (E01\u2013E14) |\n| Failure classes | 11 covered, 1 not applicable (F2: bot oru window-um drive pannala, API mattum) |\n| Fix before handover | 13 items |\n\n**Fix before handover \u2014 mukkiyamaanadhu:**\n- `data/output/` folder repo-la illa, code-um adha create pannala. Pudhu machine-la first run-e crash aagum, report varaadhu.\n- Crash reason console-ku mattum dhaan pogudhu. Ops log folder-ah open panna mudiyaadhu, so ops-ku run yen ninnuchu-nu theriyaadhu.\n- \"Created (some lines rejected)\" (partial) item-um Synced-nu dhaan kaattudhu.\n- Billing-oda \"exists\" check fail aanaalo, 422 reply JSON illanaalo, full run ninnudum. Adhukku apram ulla invoices ovvoru hour-um block aagum.\n- Ellaa error-um 5 vaati retry aagudhu, expired key (401) kooda. Failed message-la HTTP code illa.\n- Excel CSV BOM irundha ella line-um Invalid aagum.\n- Run lock illa, input archive aagala.\n- Version ops-ku engayum theriyaadhu.\n- Report-la customer name irukku.\n- `.env` git-la commit aayirukku (key-ah rotate pannunga).\n- Billing 409 sollum-nu nambi dhaan duplicate protection irukku, aana incident 2-la duplicate vandhuchu. Billing unique invoice number-ah enforce pannudhaa-nu confirm pannunga.\n- `BOT_WAS_RUN` marker file ovvoru run-um project root-la ezhudhapadudhu.\n\n**Verification:**\n- Ella status/message string-um section 6-la irukku.\n- PHI, credentials, IP edhuvum illa.\n- file:line section 13-la mattum dhaan.\n- `data/output/` repo-la illa, code-um create pannala. Adhanaala doc-la adha \"Before a run\" pre-requisite-aa pottirukken.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel)? | Ops access: which of the bot's places ops can open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-api-sync-screens-r3.md
+177
View File
@@ -0,0 +1,177 @@
# Invoice sync - ops handover
Version 0.9.1 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Every hour, the bot takes the approved invoices in the finance export file (`data/input/invoices.csv`) and creates each one in the billing system through the billing system's web API. It has no window and drives no desktop app or website. It only talks to the billing API in the background. For each run it writes one report file to `data/output/`, with one line per invoice saying what happened to it.
## 2. How it works
```
Windows Task Scheduler (every hour)
└─ starts → Invoice sync (no window) ──sends each invoice──> Billing API
│
├─ reads → data/input/invoices.csv
├─ writes → data/output/sync_report_<YYYYMMDD_HHMM>.csv (ops can open)
└─ writes → logs/sync.log (dev only)
```
- **Windows Task Scheduler** starts the bot at the top of every hour. Nobody clicks anything.
- **Invoice sync** is a background program. It has no screen.
- **Billing API** is the billing system. Invoices the bot creates appear there.
- **The input file** is the finance export. The bot never moves or empties it, so every run reads the whole file again.
- **The report file** is the only record ops can open. Each run makes a new file, named after the date and time the run started, for example `sync_report_20261023_1400.csv` for the 14:00 run.
- **The log** holds one line per invoice (number and status only) plus an end-of-run line. Ops has no access to it.
**One run, step by step:**
1. Task Scheduler starts the bot.
2. The bot creates a new, empty report file in `data/output/`.
3. For each line of the input file, in order:
1. It checks the line: invoice number, customer ID, amount and currency must be filled in, and the amount must be a number above 0. If the line fails the check, the bot marks it **Invalid** and moves on without sending it.
2. It sends the invoice to billing.
3. If billing gives no clear answer, it asks billing "does this invoice exist now?". If yes, it marks the invoice **Synced**. If not, it waits and tries again, up to 5 times (section 8).
4. It writes the result as one line in the report.
4. When every line is done, it writes "run done" to the log. Ops cannot see this. To check a run finished, see section 5.
## 3. Before a run
- [ ] The folder `data/output/` exists next to the bot. **The bot does not create it.** If it is missing, every run stops at once and no report appears.
- [ ] The folder `logs/` exists next to the bot. If it is missing, the bot stops at once.
- [ ] The settings file (`.env`) with the billing address and key is next to the bot. If it is missing or the key has expired, every invoice ends **Failed** after about a minute of retries each (E09).
- [ ] The input file is at `data/input/invoices.csv` and keeps its header line exactly: `invoice_no, customer_id, customer_name, amount, currency`.
- [ ] Input rules, one line per invoice:
| Field | Rule | Bad | Good |
|---|---|---|---|
| invoice_no, customer_id, currency | Must not be empty | (blank) | `INV-1001` |
| amount | A plain number above 0. No currency sign, no thousands separator. | `$1,200.50`, `0`, `-5` | `1200.50` |
| File format | Save as **CSV UTF-8** without the "BOM" marker. Some Excel "CSV UTF-8" exports add an invisible marker at the start, and then **every** line comes back **Invalid - missing invoice_no** (E04). | Excel "CSV UTF-8" with marker | Plain CSV / UTF-8 without marker |
- [ ] Only one copy of the bot is scheduled. Nothing stops two runs from overlapping (section 4).
## 4. During a run
- There is nothing to watch. The bot has no window. Leave the input file alone while a run is going. Don't open it in Excel and save over it at the top of the hour.
- **To stop the bot:** whoever has access to the bot machine ends the "Invoice sync" task in Task Scheduler, and disables it to stop future runs. Ops has no access to that machine, so raise a ticket. Stopping mid-run is safe: invoices already created come back as **Duplicate** on the next run, not as second copies.
- **A run looks stuck or unfinished when:**
- the newest report has fewer lines than the input file (not counting the header) more than an hour after it started, or
- no new report appears for an hour.
- The bot has no watchdog. Each invoice can take up to about **6 minutes** when billing is slow or down: 5 tries, each waiting up to 30 s to send and 30 s to check, plus 62 s of pauses. A run that is behind can still be going when the next hourly run starts. Both then write their own report.
## 5. Where to look when something goes wrong
Ops can open only the `data/output/` folder and its report files. The bot has no screens.
| Question | Where to look |
|---|---|
| Did the run start? | `data/output/` has a `sync_report_<date>_<time>.csv` for that hour |
| Did the run finish? | That report has one line per input line (not counting the header). Fewer lines means the run stopped part way (E10-E13). |
| Which invoices failed? | Report → **Status** column: anything other than `Synced` or `Duplicate`, plus `Synced` lines whose **Message** is "Created (some lines rejected)" (E07) |
| Why did one invoice fail? | Report → **Message** column on that invoice's line |
| Did the bot already retry? | Status `Failed` with Message "No success after 5 tries" means it retried 5 times. Message "Created (confirmed after retry)" means at least one try had no clear answer. Otherwise it did not retry. |
| Why did the whole run stop? | Not visible to ops. Raise a ticket (section 11). |
| Which version is running? | Not visible to ops (section 10). |
## 6. Statuses
The report is a plain CSV file, so nothing has a colour. These are every Status and Message the bot writes.
| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | The invoice was created in billing | nothing |
| Synced | Created (some lines rejected) | **Needs follow-up.** Billing created the invoice but rejected some of its lines. The invoice in billing is incomplete. | Open the invoice in billing and fix the missing lines (E07) |
| Synced | Created (confirmed after retry) | Billing did not answer clearly, but the invoice exists in billing now. It may have existed before this run (E08). | nothing, unless the invoice in billing looks wrong |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number, usually from an earlier hourly run. This is normal for every invoice that stays in the input file. | nothing. Look into it only if the invoice is new and was never sent before (E06). |
| Invalid | missing `<field names>` | A required field is empty on that line | Fix the input line (E01). If every line says "missing invoice_no", see E04. |
| Invalid | amount must be positive | The amount is 0 or negative | Fix the amount (E02) |
| Invalid | amount '`<value>`' is not a number | The amount has a currency sign, comma or text | Write it as a plain number (E03) |
| Invalid | (billing's own explanation), or "Rejected by billing" | Billing refused the invoice's content | Read the message, fix the input (E05) |
| Failed | No success after 5 tries | Billing failed 5 times and the invoice is not in billing | See E09 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing customer_id" (or other field names) | Before sending, it checks that invoice number, customer ID, amount and currency are filled in | Report, Status and Message | Business | Ops | Fill in the field in the finance export. The next hourly run picks it up. | Never |
| E02 | Invalid - "amount must be positive" | The amount is 0 or below | Report, Status and Message | Business | Ops | Correct the amount, or remove credit notes from the export | Never |
| E03 | Invalid - "amount '1,200.50' is not a number" | The amount cannot be read as a number | Report, Status and Message | Business | Ops | Write amounts as plain numbers: `1200.50` | Never |
| E04 | **Every** line Invalid - "missing invoice_no" (and maybe other fields) | It cannot find the invoice-number column because the file starts with an invisible marker, or the header was renamed | Report, Status and Message | Business | Ops | Re-save the input as plain CSV/UTF-8 without the marker. Check the header line spelling. | It keeps happening with a clean file |
| E05 | Invalid - billing's own explanation, or "Rejected by billing" | Billing answered "content not acceptable" | Report, Status and Message | Business | Ops | Read the message, fix the data in the finance export | The message makes no sense, or billing rejects data that is correct |
| E06 | Duplicate - "Invoice already exists in billing" | Billing answered "this invoice number already exists" | Report, Status and Message | Business | Ops (usually nothing) | Normal for invoices left in the input from earlier runs. For a **new** invoice, check in billing whether its number is already used by a different invoice. | A new invoice is blocked by someone else's invoice with the same number |
| E07 | **Synced** - "Created (some lines rejected)" | Billing answered "partly accepted" | Report, Status and Message. **Status still says Synced.** | Business | Ops | Open the invoice in billing and complete or correct the rejected lines. Filter the report on Message, not only Status. | The cause of the rejection is not clear |
| E08 | Synced - "Created (confirmed after retry)" | Sending got no clear answer or an error, so the bot asked billing whether the invoice exists, and it does | Report, Status and Message | System | Bot | Nothing. If the invoice already existed before this run (for example, someone typed it in by hand), the bot still writes this message. | The invoice in billing is not the one in the input |
| E09 | Failed - "No success after 5 tries" | 5 tries without success. Each time, billing was unreachable, timed out (30 s), was busy, or refused the key, and the invoice still was not in billing. | Report, Status and Message. The real reason (busy, down, wrong key) is **not** shown. | System | Bot, then Ops, then Dev | 1. If only a few invoices failed, wait. The next hourly run tries them again. 2. Before re-entering anything by hand, **search billing for the invoice number**: a timed-out send may have created it anyway. | **Every** invoice in a run is Failed (likely an expired key or billing down), or the same invoice fails 2 runs in a row |
| E10 | No new report for an hour | The bot stops before it opens the report: the `data/output/` or `logs/` folder is missing, the input file is missing, or the computer did not run the task | Nothing ops can see | System | Dev | Raise a ticket with the hour that has no report | Always |
| E11 | The report stops part way and later lines never appear. The **same line** is the last one hour after hour. | While checking "does this invoice exist?" billing was unreachable, and the bot stopped the whole run | Report is cut short. The cause is visible to dev only. | System | Bot, then Dev | Wait one run. If billing was only briefly down, the next run gets past it. | The report stops at the same place 2 runs in a row |
| E12 | The report stops part way, at an invoice billing rejected | Billing rejected the invoice with an answer the bot could not read, and the bot stopped the whole run | Report is cut short. The cause is visible to dev only. | System | Dev | Note the invoice number on the line **after** the last line in the report. That is the one that stopped the run. Every invoice after it is blocked every hour until it is fixed or removed from the input. | Always |
| E13 | Report cut short for any other reason | Any unexpected error stops the run | Report is cut short. Details are visible to dev only. | System | Dev | Raise a ticket | Always |
| E14 | The same invoice appears **twice** in billing | A send timed out after billing had created it, and the check-then-retry did not catch it | Not recorded by the bot | System | Ops, then Dev | Void the extra copy in billing following the finance process | Always. It means the duplicate protection did not work (incident 2). |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing slow, busy, down, unreachable, or refusing the key | Checks whether the invoice was created anyway, then waits and tries again. The waits are 2, 4, 8, 16 and 32 s. | 5 tries per invoice, up to about 6 minutes per invoice | Invoice marked **Failed**, the bot moves to the next invoice (E09) |
| Send timed out after billing actually created the invoice | Before each retry, it asks billing whether the invoice exists | Same 5 tries | Marked **Synced - Created (confirmed after retry)** (E08) |
| Invoice not processed in this run (Failed, Invalid, or the run stopped) | Nothing within the run. The next hourly run reads the whole input file again. | Every hour, without end, while the line stays in the input | Fixed invoices go through. Invoices already created show **Duplicate**. |
| The whole run stops | Nothing. No restart, no resume. | - | The next hourly run starts again from the first line |
## 9. Known incidents
Newest first. The history has no dates. Both were fixed before or at 0.9.1.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | The same invoice appeared twice in billing | Billing timed out after it had already created the invoice. The bot thought it failed and sent it again. | Fixed: the bot now checks whether the invoice exists before each retry. That check can still stop the run if billing is down (E11). If doubles appear again: void the extra copy and raise a ticket (E14). | Partly. The clean-up, yes. Prevention needed a code change. |
| Before 0.9.1 | The hourly run stopped part way | Billing said "too many requests" and the bot had no retry | Fixed: 5 tries with waits. Re-running is safe because invoices already created come back as Duplicate. | Partly |
## 10. Releases and version check
- **How an update arrives:** a developer replaces the bot's files on the bot machine. There is no auto-update.
- **How to confirm it applied:** ops cannot see the version. The bot does not show it anywhere ops can reach. Ask dev to confirm the version in the ticket that announced the update.
- **If the version did not change:** reply on the same ticket in the RPA Support queue.
- Only one machine runs this bot. It writes nothing shared at start-up, so an old copy cannot overwrite a new one. But **two copies scheduled on two machines would both send the same invoices**. Keep it to one.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue with:
- the exception ID from section 7 (for example, E11)
- the report file name, for example `sync_report_20261023_1400.csv`. It gives the run date and time.
- the invoice number(s) affected, and the last invoice number in a cut-short report
- whether the previous hour's run showed the same problem
Never send: the settings file (`.env`), the billing key, or the report file itself in chat. The report contains customer names, so reference it by name.
## 12. Failure-class check
- F1 (missing prerequisite) covered in section 3 (output folder, logs folder, settings file, input file), E09, E10, and Fix before handover #1.
- F2 not applicable: the bot drives no windows. It only calls a web API in the background.
- F3 (target slower than expected) covered in E09, E11, section 8 (30 s timeout, 5 tries) and section 4 (6 minutes per invoice).
- F4 (input read differently) covered in section 3 input rules, E01-E04, and Fix before handover #7.
- F5 (marked done when nothing was done) covered in E08 and Fix before handover #5.
- F6 (action happened, not confirmed) covered in E08, E09 (search billing before re-entering), E14, and Fix before handover #4.
- F7 (business exception shown as success) covered in E07, section 6, and Fix before handover #3.
- F8 (endless re-runs / watchdog) covered in section 4 (no watchdog, overlapping runs), section 8 (whole file every hour), and Fix before handover #6 and #8.
- F9 (stopping the wrong way) covered in section 4 (end the task in Task Scheduler, via ticket).
- F10 (one stuck item blocks the queue) covered in E11 and E12 (one invoice can stop every later invoice, every hour) and Fix before handover #4.
- F11 (cause hidden from ops) covered in section 5 and Fix before handover #2.
- F12 (update did not arrive) covered in section 10 and Fix before handover #9.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | `data/output/` is not in the repo and the code never creates it. `logs/` must also exist before `logging.basicConfig` opens the file. | A fresh install fails every run with no report and nothing ops can see | `sync.py:24`, `sync.py:68-69` |
| 2 | Run crashes and network errors go only to `print()` (console), which Task Scheduler throws away. `sync.log` holds the status but never the Message. Ops cannot open `logs/`. | Ops cannot tell why a run stopped (E10-E13). Even dev has no traceback. | `sync.py:52`, `sync.py:75`, `sync.py:84` |
| 3 | HTTP 207 (partial) is stored as Status `Synced` | Incomplete invoices look done when filtering by Status (E07) | `sync.py:54-55` |
| 4 | `exists()` has no try/except, and the 422 `r.json()` is not guarded. Neither is caught per row. | One bad invoice or a short outage stops the whole run, and every later invoice is blocked every hour (E11, E12) | `sync.py:42`, `sync.py:59`, `sync.py:72-76` |
| 5 | `exists()` returning 200 is reported as "Created (confirmed after retry)", even on the first attempt and even if the invoice existed before this run | Synced does not prove this run created it (E08) | `sync.py:60-61` |
| 6 | Every status other than 200/201/207/409/422 is retried, including 400/401/403/404. The docstring says only 429/5xx. `Retry-After` is ignored, and the bot sleeps after the last try too. The Failed message has no HTTP code. | An expired key costs about a minute per invoice and shows only "No success after 5 tries" (E09) | `sync.py:47`, `sync.py:60-63` |
| 7 | Input read as `utf-8`, not `utf-8-sig`. The amount check accepts `nan`/`inf`. | An Excel export with a BOM turns every line Invalid (E04) | `sync.py:34`, `sync.py:69` |
| 8 | Input is never archived and there is no run lock | The whole file is re-sent every hour (noisy Duplicates), and slow runs can overlap the next hourly run | `sync.py:66-77` |
| 9 | Version exists only in README text, with a stray `chore: bump to 0.9.1` line. Nothing ops can see shows the version. | Ops cannot confirm an update applied (F12) | `README.md:5-6` |
| 10 | The report includes `customer_name` | Personal data in a folder ops can open. Confirm this is allowed. | `sync.py:76` |
| 11 | `.env` with the billing key is committed to git | Credential exposure. Rotate the key and untrack the file. | `.env` (commit `88b2057`) |
| 12 | Duplicate protection relies on billing answering 409, yet incident 2 produced doubles | Confirm billing enforces unique invoice numbers. Otherwise E14 can recur. | `sync.py:56-57` |
| 13 | `BOT_WAS_RUN` marker file is written into the project root every run | Leftover file with no purpose for ops | `sync.py:67` |
tests/evals/results/handoff/h-api-sync-screens-r4.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-api-sync/screens",
"rep": 4,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"retry limit": true,
"console-only crash output flagged": true
},
"cost_usd": 1.3588046,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-api-sync-screens-s_dvo5sy\\project",
"reply": "I wrote `HANDOVER.md` in the repo root. Bot-a run pannala; code, config, git history and the log mattum dhaan padichen. Code-ah change pannala. Unga 3 answers apdiye use pannen: escalation-ku RPA Support ticket, ops-ku output folder mattum dhaan access, and unclear statuses-ku code-oda best reading.\n\n| Item | Result |\n|---|---|\n| Sections | 13 (template-oda ella sections-um) |\n| Exceptions catalogued | 11 (E01\u2013E11) |\n| Failure classes | 11 covered, 1 not applicable (F2: bot entha window-um drive pannadhu, API calls mattum dhaan) |\n| Fix before handover | 19 items |\n\nFix before handover-la top items:\n- **Billing API key `.env`-la git-ku commit aagiruku.** Key-ah rotate pannanum, file-ah history-la irundhu remove pannanum.\n- Duplicate check-ku error handling illa. Billing slow-aa irundha full run-um nindhurum. Ops paakra report-la reason record aagaadhu.\n- Billing \"partly created\" nu sonnaalum report-la **Synced**-nu dhaan varudhu. Follow-up venum, aana success maadhiri theriyum.\n- Wrong or expired key-ah bot temporary error maadhiri treat pannudhu. Ovvoru invoice-um 5 tries pannitu Failed aagum.\n- `data/output` folder-ah code create pannadhu. Fresh machine-la first run-e crash aagum.\n- Overlapping runs-ku lock illa. Ovvoru hour-um full file-ah thirumba anupudhu.\n- Version README-la mattum dhaan iruku, ops-ku theriyaadhu.\n- Rendu \"fix\" commits-um oru comment line mattum dhaan add pannudhu. Real code changes git-la record aagala.\n\nOps-ku logs access illa, so oru run complete aachaa nu ops-aala uruthiya solla mudiyaadhu. Idhu doc-la Fix item 9-aa iruku.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the unclear statuses and exceptions mean?",
"stderr": ""
}
tests/evals/results/handoff/h-api-sync-screens-r4.md
+175
View File
@@ -0,0 +1,175 @@
# Invoice sync - ops handover
Version 0.9.1 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Every hour, the bot takes the approved invoices from the finance export file (`data/input/invoices.csv`) and creates each one in the billing system through the billing system's web API. It reads the whole file on every run. For each invoice it writes one line to a report file in `data/output/`, named `sync_report_<date>_<time>.csv`, with the result (Status) and the reason (Message). It has no screen and drives no desktop app or website. Windows Task Scheduler starts it.
## 2. How it works
```
Windows Task Scheduler (every hour)
└─ starts → Invoice sync (no window)
├─ reads → data/input/invoices.csv (finance export)
├─ sends → Billing system API (over the network)
├─ writes → data/output/sync_report_<date>_<time>.csv ← ops reads this
└─ writes → logs/sync.log (dev only)
```
- **Task Scheduler** starts the bot once an hour. The task's settings are not stored with the bot.
- **Invoice sync** runs in the background. There is nothing on screen to watch or click.
- **The finance export** is the input. Every row is one invoice.
- **The billing system** is where invoices are created. The bot only talks to it over the network.
- **The report file** is the only place ops can see results. A new one is written every run.
- **The log file** holds only the invoice number and status. Ops cannot open it.
**One run, step by step:**
1. Open the finance export and start a new report file.
2. For each invoice, check the row: invoice number, customer ID, amount and currency must be filled in, and the amount must be a number above zero. If the check fails, the row is marked **Invalid** and the bot does not send it.
3. Send the invoice to billing.
4. If billing says "created", the row is **Synced**. If billing says it already has the invoice, the row is **Duplicate**. If billing rejects the data, the row is **Invalid**.
5. If billing gives any other answer, or does not answer within 30 seconds, the bot asks billing whether the invoice exists now. If it does, the row is **Synced**. If not, it waits and tries again, up to 5 times. After 5 tries the row is **Failed**.
6. Write the row to the report and go to the next invoice.
7. After the last invoice, the bot writes "run done" to its log. Ops cannot see this line.
## 3. Before a run
- [ ] The folder `data/output/` exists next to the bot. The bot does not create it. Without it, the run stops before the first invoice and no report appears.
- [ ] The settings file (`.env`) is next to the bot and holds the billing address and key. Without the key, every invoice ends as **Failed** after about a minute each (see E08).
- [ ] The finance export is at `data/input/invoices.csv`, and its first line has the column names `invoice_no, customer_id, customer_name, amount, currency`, spelled exactly like that.
- [ ] Every row has an invoice number, customer ID, amount and currency.
- [ ] The amount is a plain number with a dot for decimals:
- Bad: `1,250.00`, `$1250`, `1250,00`
- Good: `1250.00`
- [ ] Do not open the export in Excel and save it again. Excel can change the file so that every row fails as "missing invoice_no", or so that the run stops.
- [ ] Only **one** machine runs the hourly task. Two machines on the same file can create the same invoice twice.
- [ ] The bot is version 0.9.1 (see section 10).
## 4. During a run
- There is nothing on screen. Do not start the bot by hand while the hourly task may be running. The bot has no protection against two runs at once.
- To stop the bot: in **Task Scheduler**, right-click the task → **End**. To stop the hourly runs, **Disable** the task. If ops has no access to Task Scheduler, raise a ticket.
- A run can be slow. Each invoice that billing does not accept can take **up to about 6 minutes** (5 tries, 30 s wait for each answer, pauses of 2, 4, 8, 16 and 32 s). Many such invoices can make one run last longer than an hour, so the next hourly run may start on top of it.
- The bot has **no** stuck detection and no automatic restart. If no new report has appeared for more than 2 hours, raise a ticket.
- Expect every run to list **every** invoice in the export again. An invoice synced in an earlier hour shows as **Duplicate** in every later report. That is normal.
## 5. Where to look when something goes wrong
Ops can open only the output folder `data/output/` and its report files.
| Question | Where to look |
|---|---|
| Did the run happen? | `data/output/`: a new `sync_report_<date>_<time>.csv` for that hour. No new file means the run did not start or stopped at once (E09). |
| Did the run finish? | **Ops cannot tell for certain.** A report that has fewer rows than the previous hour's report, or that ends earlier than usual, means the run stopped partway (E09). The "run done" line is in the log only (Fix item 9). |
| Which invoices failed? | The report file, **Status** column: `Failed`, `Invalid`, and `Synced` with Message "Created (some lines rejected)" |
| Why did one invoice fail? | The report file, **Message** column, on that invoice's row |
| Did the bot already retry? | Yes, always: every `Failed` row was tried 5 times. Every invoice is also tried again in the next hourly run. |
| Did an earlier run already sync it? | Open the earlier report files in `data/output/` and look for the invoice number with Status `Synced` |
| Which version is running? | Ops cannot see it (Fix item 15). Ask in a ticket. |
## 6. Statuses
The report is a plain CSV file, so it has no colours. The Status column holds one of these values.
| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | Billing created the invoice | Nothing |
| Synced | Created (confirmed after retry) | Billing did not confirm the first send, but the bot then found the invoice in billing | Nothing |
| Synced | Created (some lines rejected) | **Needs follow-up.** Billing created the invoice but dropped some of its lines. It is shown as a success, but it is not complete. | Raise a ticket (E06) |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number. This is usually the bot's own earlier run. | Nothing, if an earlier report shows it as Synced. Otherwise see E05. |
| Invalid | missing `<column names>` | A required column is empty in the export | Fix the export (E01) |
| Invalid | amount must be positive | Amount is zero or negative | Fix the export (E02) |
| Invalid | amount '`<value>`' is not a number | The amount could not be read as a number | Fix the export (E03) |
| Invalid | billing's own reason, or "Rejected by billing" | Billing refused the invoice data | Read the reason, then fix the export (E04) |
| Failed | No success after 5 tries | Billing did not accept it after 5 tries | Wait for the next hour (E08) |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing invoice_no" (or customer_id, amount, currency) | Before sending, it checks that these four columns are filled in | Report, Status + Message | Business | Ops | Ask finance to fill in the column in the export. The next hourly run picks it up. | **Every** row says "missing invoice_no": the export's format changed (often after it was saved in Excel). Send the report file name. |
| E02 | Invalid - "amount must be positive" | Before sending, it checks that the amount is above zero | Report, Status + Message | Business | Ops | Ask finance to correct the amount, or remove credit notes from the export | Never; this is a data issue |
| E03 | Invalid - "amount '…' is not a number" | Before sending, it tries to read the amount as a number | Report, Status + Message | Business | Ops | Ask finance to write the amount as a plain number, e.g. `1250.00` | Many rows at once after a change to the export |
| E04 | Invalid - billing's reason, or "Rejected by billing" | Billing answers that the invoice data is wrong | Report, Status + Message | Business | Ops | Read the Message, fix the row with finance, and wait for the next hour | The Message is unclear, or the same row is rejected after the fix. Send the invoice number, report file name and Message. |
| E05 | Duplicate - "Invoice already exists in billing" | Billing answers that the invoice number already exists | Report, Status + Message | Business | Ops | Search earlier reports for the invoice. If one shows it as Synced, do nothing. | No earlier report shows it as Synced. Someone may have created it in billing by hand, or the number is reused. Send the invoice number and report file names. |
| E06 | Synced - "Created (some lines rejected)" | Billing answers "partly created" | Report, Status + Message only | Business | Dev | Raise a ticket. Ops cannot see which lines were dropped. | Always. Send the invoice number and report file name. |
| E07 | Synced - "Created (confirmed after retry)" | The send failed or timed out, then the bot asked billing and found the invoice | Report, Status + Message | System | Bot | Nothing. The bot did not send it again. | Finance reports the invoice twice in billing (E11) |
| E08 | Failed - "No success after 5 tries" | Billing gave no usable answer (busy, down, too many requests, wrong key) or no answer in 30 s, 5 times in a row | Report, Status + Message. The real reason is not recorded. | System | Bot, then Ops | Wait for the next hourly run; it tries again. Check the next report for this invoice. | The same invoice is Failed in **2 reports in a row**, or **most rows** in one report are Failed (billing is down or the key is wrong). Send the report file names and the times. |
| E09 | No new report for an hour, or a report that stops partway | The run stopped. Known causes: billing did not answer while the bot was checking for a duplicate; billing sent an unreadable rejection; the output folder or export file is missing. | **Nowhere ops can see.** The reason is lost (Fix items 2-4). | System | Dev | Raise a ticket. Invoices after the stop point were not handled in that run. | Always. Send the hour, the report file name (if any) and the **last invoice number** in it. If the same report stops at the same invoice every hour, say so: that invoice is blocking every run. |
| E10 | Report has rows for the same hour twice, or a report is overwritten | Two runs ran at the same time (a slow run overlapped the next one, or the bot was started by hand) | Report files in `data/output/` | System | Dev | Raise a ticket. Two runs at once can create duplicates in billing. | Always. Send the report file names and times. |
| E11 | Finance or billing reports the same invoice created twice | The bot sent it again because billing timed out after creating it, and the check before the retry did not see it | Nowhere in the bot. The report shows Synced or Failed. | System | Dev | Raise a ticket. Billing must void the extra copy. | Always. Send the invoice number and the report file names that list it. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing does not answer | Waits 30 s for each answer | 30 s per request | Counts as one failed try |
| Billing busy, down, "too many requests", or any unexpected answer | Asks billing whether the invoice already exists. If not, pauses and sends again. | 5 tries; pauses of 2, 4, 8, 16, 32 s; up to about 6 min per invoice | Row = **Failed**, then the next invoice |
| Invoice created but the bot did not get the confirmation | The check before each retry finds it, and the bot does not send it again | Once per failed try | Row = **Synced** "confirmed after retry" |
| Failed or Invalid invoice | The next hourly run sends every invoice in the export again | Every hour, for as long as the invoice is in the export | Never gives up; it shows in every report |
| Run stopped (E09) | Nothing. There is no restart within the hour. | - | The next hourly run starts again from the top |
## 9. Known incidents
Both were fixed before version 0.9.1. Their dates are not recorded.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | Invoices created **twice** in billing | Billing timed out after it had already created the invoice, so the bot sent it again | Partly fixed: the bot now checks billing before each retry. If that check itself fails to get an answer, the run stops (E09), and a duplicate is still possible (E11). Raise a ticket so billing can void the extra copy. | Partly (clean-up only) |
| Before 0.9.1 | Hourly run stopped when billing answered "too many requests" | Billing's rate limit was not retried | Fixed: up to 5 tries with pauses. If many rows are Failed, see E08. | Partly |
## 10. Releases and version check
- How an update arrives: there is no automatic update. A developer copies the new files to the bot machine.
- How to confirm it applied: the version is written only in the bot's README file, which ops cannot open. Ask the developer to state the version in the ticket when they finish the update (Fix item 15).
- If the version did not change: raise a ticket. Only one machine runs the hourly task, so there is no risk of an old machine undoing a new one, unless a second machine is set up (section 3).
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue. Send:
- the exception ID from section 7 (e.g. E09)
- the hour of the run
- the report file name from `data/output/`
- the invoice numbers involved
Never send: the API key, the `.env` settings file, customer names, or whole report files in chat. The report contains customer names; attach it to the ticket only.
## 12. Failure-class check
- F1 covered in section 3 (output folder, settings file, export location), E08 and E09.
- F2 not applicable: the bot drives no windows. It only talks to billing over the network.
- F3 covered in E08, E09 and section 8: the 30 s answer limit and 5 tries.
- F4 covered in section 3 input rules, E01-E03, and Fix item 12 (Excel-saved file).
- F5 covered: the bot writes Synced only when billing confirms the invoice. A stopped run writes no success for the rows it did not reach (E09). The partial-success case is under F7.
- F6 covered in E07, E09 and E11, and Fix items 2 and 14.
- F7 covered in section 6 (Synced "some lines rejected"), E06 and Fix item 6.
- F8 covered in section 4 (the whole file is re-sent every hour, no stuck detection), E10, and Fix items 10 and 11.
- F9 covered in section 4: stop it in Task Scheduler.
- F10 covered in E09: an invoice that makes the run stop blocks every invoice after it, every hour. Fix items 2 and 3.
- F11 covered in section 5 and Fix items 4, 5 and 9: the reasons are in the log or lost, not in the report.
- F12 covered in section 10 and Fix item 15.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The billing API key is committed to git in `.env` | The key is exposed to anyone with the repo. Rotate the key and remove the file from history. | `.env` (commit 76b9600) |
| 2 | The duplicate check has no error handling. A timeout or network error on it stops the whole run. | Run stops (E09) exactly when billing is slow, which is the case the 504 fix was meant for | `sync.py:41-43`, `sync.py:60` |
| 3 | A 422 reply without JSON stops the whole run, and the same row then stops every hourly run | One invoice blocks all later invoices, every hour (E09) | `sync.py:59` |
| 4 | The crash reason and network errors go to stdout only, which Task Scheduler discards. Nothing reaches the log or the report. | The cause of E09 is lost | `sync.py:52`, `sync.py:81-85` |
| 5 | The log holds only invoice number and status: no HTTP code, attempt count or error. "Failed" never says why. Ops cannot open the log anyway. | Ops cannot tell a wrong key from an outage from a rate limit (E08) | `sync.py:63`, `sync.py:75` |
| 6 | HTTP 207 (partial) is written as Status "Synced" | A partly created invoice looks like a success (F7) | `sync.py:54-55` |
| 7 | 400, 401, 403 and 404 are retried 5 times like temporary errors, with no abort for the whole run | A wrong or expired key makes every invoice take about 1-6 min and end as Failed; the run can overlap the next | `sync.py:60-62` |
| 8 | `data/output/` is not created by the code | The first run on a new machine stops with no report | `sync.py:68-69` |
| 9 | No end-of-run marker that ops can see ("run done" is in the log only) | Ops cannot tell a finished run from a stopped one | `sync.py:77` |
| 10 | No lock against overlapping runs. The report name has minute precision and is opened for overwrite. The Task Scheduler task settings are not in the repo. | Overlapping runs can create duplicates and overwrite reports (E10) | `sync.py:68-69` |
| 11 | No resume and no memory of synced invoices: every row is posted again every hour | Report fills with "Duplicate" rows, it uses up the rate limit, and the risk of duplicates grows | `sync.py:72-74` |
| 12 | The export is read as `utf-8`, not `utf-8-sig`. A file saved by Excel with a BOM would make every row "missing invoice_no" (possible, not seen yet). | Whole run Invalid after one re-save | `sync.py:69` |
| 13 | `Retry-After` is ignored, a duplicate-check GET is added to every failed POST, and the bot sleeps 32 s after the final attempt | Slower runs and more rate-limit hits | `sync.py:60-62` |
| 14 | The duplicate check treats any non-200 (429, 5xx) as "not there" and posts again | Duplicates are still possible (E11). Needs an idempotency key or a billing-side unique constraint. | `sync.py:43` |
| 15 | The version exists only in the README (with a stray "chore: bump to 0.9.1" line), not in the report or log | Ops cannot confirm which version is running | `README.md:5-6` |
| 16 | The report copies customer names | The output folder holds customer data | `sync.py:76` |
| 17 | `nan` and `inf` pass the amount check, and currency is not checked against a list | Bad rows reach billing and come back as Invalid or Failed | `sync.py:33-37` |
| 18 | Both "fix" commits contain only a comment line, and there are no regression tests | The incident fixes were never reviewed as code changes | `sync.py:86-87`, commits 244fb15 and 8656aa3 |
| 19 | The `BOT_WAS_RUN` marker file is written and never read | Clutter, and it can be mistaken for a lock | `sync.py:67` |
tests/evals/results/handoff/h-api-sync-screens-r5.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-api-sync/screens",
"rep": 5,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": false,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"retry limit": true,
"console-only crash output flagged": true
},
"cost_usd": 0.721561,
"subagents": 0,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-api-sync-screens-0b_hw7xu\\project",
"reply": "I've written `HANDOVER.md` at the repo root (Thanglish summary). Code edhuvum maathala, bot-ah run pannala \u2014 code, git history, logs mattum padichen. Repo-la oru script (`sync.py`) mattum dhaan irundhudhu, so skill sonna rendu subagents anuppama naane direct-ah padichen.\n\n| Item | Count |\n|---|---|\n| Sections | 13 (template order-la) |\n| Exceptions | 10 (E01\u2013E10) |\n| Failure classes | 12-la 10 cover aachu; F2 (windows drive pannala), F5 (billing confirm pannina apram dhaan Synced) not applicable |\n| Fix before handover | 11 items |\n\n**Fix before handover \u2014 mukkiyamaanavai:**\n- Billing \"partly accepted\" (207) invoice-um **Synced**-nu kaattudhu, but sila lines reject aagirukkum, yaarum follow-up panna maattaanga.\n- Network error, crash reason ellam console-ku mattum print aagudhu; log-la status mattum dhaan irukku, so cause kaanaama poidudhu.\n- Excel \"CSV UTF-8\"-ah save pannina, ella row-um Invalid \"missing invoice_no\"-nu varum.\n- Oru network blip vandhaa run naduvula nikkum; report-la \"run mudinjudhu\"-nu oru line-um illa.\n- `data/output` folder-ah code create pannaadhu, so folder illaati report-e varaadhu.\n- `.env` (billing API key) git-la commit aagirukku. Key-ah rotate pannanum, history-la irundhu remove pannanum. Doc-la value podala.\n- Matha items: 401/403 error billing outage maadhiri retry aagudhu, rendu hourly run overlap aagalaam, version ops-ku theriyaadhu, last try-ku apram veena 32 s wait.\n\nVerify pannadhu: ella status string-um Section 6-la irukku. File:line references Section 13-la mattum. PHI, credentials, IP edhuvum illa. Doc-la sonna ella path-um repo-la irukku illa code create pannum \u2014 `data/output` mattum illa, adhu Fix #9-la irukku.\n\nUnga answers apply pannadhu: escalation-ku RPA Support queue ticket. Ops-ku output folder mattum dhaan access, so log-la mattum theriyura causes ellam Fix list-ku poyiduchu. Status meaning code-la irundhu naane decide panninen.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel)? | Ops access: which of the bot's places ops can open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-api-sync-screens-r5.md
+158
View File
@@ -0,0 +1,158 @@
# Invoice sync - ops handover
Version 0.9.1 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Every hour, the bot reads the finance export of approved invoices (one invoice per row) and sends each invoice to the billing system over its web connection (API). It does not drive any desktop app or website screen. For every run it writes one report file in the output folder that lists each invoice with a **Status** and a **Message**. That report is the only place ops can see the result.
## 2. How it works
```
Windows Task Scheduler (every hour)
└─ starts → Invoice sync bot ──sends each invoice──> Billing system (web API)
│
├─ reads → finance export file (input, one invoice per row)
├─ writes → data/output/sync_report_<date>_<time>.csv (ops can open)
└─ writes → a log file in the bot folder (dev only)
```
- **Task Scheduler** starts the bot once an hour. There is no app window and nothing to click.
- **The bot** checks each invoice, sends it to billing, and retries by itself when billing is busy or down.
- **The billing system** is where the invoice is created. It is the final truth: if in doubt, check there.
- **The report file** gets one row per invoice: invoice number, customer, Status, Message.
- **The log file** is for developers only. Ops cannot open it.
**One run, step by step:**
1. Create a new report file, named with the date and time the run started (for example `sync_report_20260903_0900.csv` = 3 Sep 2026, 09:00).
2. Read the input file, top to bottom.
3. For each invoice: check that the required fields are filled and the amount is a positive number. If not, mark it **Invalid** and move on, without sending it.
4. Send it to billing. If billing says yes, mark it **Synced**. If billing says it already exists, mark it **Duplicate**. If billing rejects the data, mark it **Invalid** with billing's reason.
5. If billing is busy, down or does not answer: check whether the invoice was created anyway, wait, and try again, up to 5 times. Then mark it **Failed**.
6. After the last invoice, the run ends. The report does not show an "end of run" line (see section 13).
## 3. Before a run
- [ ] The `data/output` folder exists in the bot folder. **The bot does not create it.** Without it the run stops at once and no report appears at all (E09).
- [ ] The bot's settings file (billing address and access key) is in the bot folder. Without it, the run stops at the first invoice or every invoice ends **Failed** (E07, E09). Only dev can check this file.
- [ ] Input file: every row has **invoice_no, customer_id, amount and currency** filled. customer_name is optional for the bot but appears in the report.
- [ ] Amounts are plain numbers with a dot for decimals, no thousands separator and no currency sign.
- Bad: `1,200.00` · `$1200` · `-50` · `0` → Invalid
- Good: `1200.00`
- [ ] Save the input as plain CSV. If Excel saves it as "CSV UTF-8", every row can come out **Invalid - "missing invoice_no"** (E02).
- [ ] The version is 0.9.1 (see section 10 - ops cannot check this yourselves today).
## 4. During a run
- There is nothing to watch on screen. The bot has no window and does not use the mouse or keyboard, so you can use the machine as normal.
- To stop the bot: end the "Invoice sync" task in Windows Task Scheduler (or ask whoever manages that machine). Do not delete or move the report file while a run is going. Invoices already sent stay in billing. Invoices not yet reached are tried again in the next hourly run.
- A run looks slow when billing is having problems. Each invoice that fails can take **about 1 minute, and up to about 6 minutes**, before the bot gives up and marks it Failed. A file with many failing invoices can take longer than an hour. Wait for that before you act.
- The same invoice can show up in more than one hourly report. An invoice that was **Synced** in an earlier run shows **Duplicate** the next time. That is normal.
## 5. Where to look when something goes wrong
Ops can open only the **output folder** (`data/output`) and the report files in it.
| Question | Where to look |
|---|---|
| Did the run happen? | Output folder: there is a `sync_report_<date>_<time>.csv` for that hour. No file for an hour means the run did not start or stopped at once (E09). |
| Did the run finish? | **This cannot be seen in the report today.** A report that has fewer rows than invoices in the input, or that is empty below the header, means the run stopped part-way (E08). Raise a ticket. |
| Which invoices failed? | The report → filter the **Status** column: anything other than Synced. Also check Synced rows whose Message is "Created (some lines rejected)". |
| Why did one invoice fail? | The report → the **Message** column on that row. Then use section 7. |
| Did the bot already retry? | Message "No success after 5 tries" = yes, 5 times. Message "Created (confirmed after retry)" = it retried and the invoice is in billing. Any other message = no retry was needed. |
| Which version is running? | Ops cannot see it. Ask in the ticket (section 13). |
## 6. Statuses
The report is a plain CSV file, so there are **no colours**. Read the Status and Message columns together.
| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | Billing created the invoice | nothing |
| Synced | Created (confirmed after retry) | Billing did not answer at first, but the bot then confirmed the invoice exists in billing | nothing |
| Synced | Created (some lines rejected) | **Needs follow-up.** The invoice was created, but billing refused some of its lines. Shown as Synced. | Check the invoice in billing and fix the missing lines (E05) |
| Duplicate | Invoice already exists in billing | Billing already has an invoice with this number | nothing, if it was Synced in an earlier run. Otherwise check it (E04) |
| Invalid | missing <field names> | A required field is empty in the input. Not sent. | Fix the input row (E01, E02) |
| Invalid | amount must be positive | Amount is 0 or negative. Not sent. | Fix the input row (E03) |
| Invalid | amount '<value>' is not a number | Amount has commas, a currency sign or text. Not sent. | Fix the input row (E03) |
| Invalid | <billing's reason>, or "Rejected by billing" | The bot sent it and billing refused the data | Read billing's reason and fix the input row (E06) |
| Failed | No success after 5 tries | Billing was busy, down, unreachable or refused access 5 times in a row | See E07 |
## 7. Exception catalogue
Quote the ID (E01, E02...) in your ticket.
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing customer_id" (or currency, amount, invoice_no) on some rows | Before sending, it checks the four required fields are not empty | Report, Status + Message | Business | Ops | Fill the field in the finance export. The next hourly run picks it up. | Never - this is an input fix |
| E02 | **Every** row Invalid - "missing invoice_no", although the numbers are in the file | Same check. The first column name in the file is not read as "invoice_no", usually because Excel saved it as "CSV UTF-8" | Report, Status + Message | Business | Ops, then Dev | Re-save the input as plain "CSV (comma delimited)" and wait for the next run | Still every row Invalid after re-saving. Send the report file name and the time. |
| E03 | Invalid - "amount must be positive" or "amount '...' is not a number" | Before sending, it reads the amount as a number and checks it is above 0 | Report, Status + Message | Business | Ops | Write the amount as a plain number, e.g. `1200.00`. Credit notes / negative amounts cannot go through this bot. | A real negative amount must be billed - dev must decide how |
| E04 | Duplicate - "Invoice already exists in billing" | Billing answers that the invoice number already exists | Report, Status + Message | Business | Ops | If the invoice was Synced in an earlier hourly report, nothing to do. If it never synced before, open it in billing and check it is the right invoice (same customer, same amount). | The invoice in billing belongs to a different customer or amount |
| E05 | **Synced** - "Created (some lines rejected)" | Billing answers "partly accepted" | Report, Status + Message. Which lines were refused is **not** recorded anywhere ops can see | Business | Ops | Treat as **not done**. Open the invoice in billing, compare its lines with the export, and fix the missing lines in billing. | You cannot tell which lines are missing. Send the invoice number and report file name. |
| E06 | Invalid - with a reason from billing, or "Rejected by billing" | Billing refuses the data as wrong (e.g. unknown customer, bad currency) | Report, Status + Message | Business | Ops | Read the reason, fix the input row, wait for the next run | The reason is "Rejected by billing" with no detail, or the data looks correct |
| E07 | Failed - "No success after 5 tries" | Billing is busy, down or does not answer within 30 s. Before each new try the bot checks whether billing created the invoice anyway. After 5 tries it gives up. | Report, Status + Message | System | Bot, then Ops / Dev | One or a few rows: leave them. The next hourly run tries again, and if the invoice was created after all it will show Duplicate (safe). | **Every** row in a report is Failed (billing down, or the bot's access key is wrong or expired), or the same invoice is Failed in 3 hourly runs in a row. Send the report file name(s) and the time. |
| E08 | The report stops part-way: fewer rows than the input, or only the header row | The bot stopped unexpectedly (for example, it lost the network connection while checking billing, or billing sent an answer the bot could not read). The invoices after that point were not tried. | Report is shorter than the input. The reason is only in the developer log. | System | Dev | Raise a ticket. The next hourly run starts from the top again, so missing invoices may still go through. Check the next report. | Always - send the report file name, the time, and the last invoice number in the report |
| E09 | No report file at all for an hour (or for every hour) | The run stopped before writing anything: the output folder is missing, the settings file is missing, or the task did not start | Nothing ops can see - the absence of the file is the sign | System | Dev | Raise a ticket | Always - send the hour(s) with no report |
| E10 | An invoice appears twice in billing | Before 0.9.1, a timeout after billing had already created the invoice made the bot create it again. Fixed in 0.9.1, but two runs running at the same time can still send the same invoice. | Billing only | System | Ops, then Dev | Void the extra copy in billing following the normal finance process | It happens on 0.9.1. Send both invoice numbers/IDs from billing and the report file names. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing is busy ("too many requests") | Waits and tries again: 2, 4, 8, 16, 32 seconds | 5 tries per invoice, about 1 minute | Row = Failed, next invoice |
| Billing is down or errors | Checks whether the invoice was created anyway; if yes, marks Synced. Otherwise waits and tries again. | 5 tries per invoice, about 1 to 6 minutes | Row = Failed, next invoice |
| Billing does not answer | Waits up to 30 s per attempt, then same as above | 5 tries per invoice, up to about 6 minutes | Row = Failed, next invoice |
| Whole run stopped (E08, E09) | No restart within the hour. The next hourly run starts from the top of the input. | 1 hour | Next hourly run |
The bot never retries Invalid or Duplicate rows by itself within a run. They come back in the next hourly run only if the row is still in the input.
## 9. Known incidents
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | Some invoices created twice in billing | Billing timed out after it had already created the invoice; the bot sent it again | Fixed in 0.9.1: the bot checks billing before each retry. If it still happens, see E10. | Partly - ops could spot and void duplicates, not prevent them |
| Before 0.9.1 | Hourly run crashed when billing was busy | Billing's "too many requests" answer was not handled | Fixed in 0.9.1 (retries with waits). | No |
## 10. Releases and version check
- How an update arrives: a developer replaces the bot files on the machine. There is no automatic update.
- How to confirm it applied: ops cannot see the version today (no screen, not in the report). Ask dev to confirm the version in the ticket.
- If the version did not change: raise a ticket in the RPA Support queue.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue.
Send: the exception ID from section 7, the report file name (e.g. `sync_report_20260903_0900.csv`), the time you noticed it, and the invoice number(s) involved.
Never send: passwords or access keys, the bot's settings file, or customer data in chat. Attach the report file to the ticket instead of pasting rows.
## 12. Failure-class check
- F1 covered in section 3 (output folder, settings file) and E09, E07.
- F2 not applicable: the bot drives no windows; it talks to billing over its web API.
- F3 covered in E07 and section 8 (30 s wait per attempt, 5 tries). No screens, so no screen-size rules.
- F4 covered in section 3 input rules, E01, E02, E03; gaps in section 13 #3.
- F5 not applicable: Synced is only written after billing itself confirms the invoice exists.
- F6 covered in E07 (bot checks billing before retrying; re-run is safe, shows Duplicate) and section 13 #6.
- F7 covered in section 6 and E05; section 13 #1.
- F8 covered in section 8 and section 4 (long runs); overlap risk in section 13 #7.
- F9 covered in section 4 (end the task in Task Scheduler).
- F10 covered in E08 (one bad answer stops the rest of the run) and section 13 #4.
- F11 covered in section 5 and section 13 #2, #5.
- F12 covered in section 10 and section 13 #8.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Partly accepted invoices (billing 207) are written as Synced | Ops reads them as done; refused lines are never followed up. Which lines were refused is not recorded. | sync.py:54-55 |
| 2 | Network errors and crash reasons go to the console only (`print`), which Task Scheduler throws away; the log keeps only invoice + status, not the message | The cause of E07/E08/E09 is lost, even for dev | sync.py:52, sync.py:75, sync.py:84 |
| 3 | Input read as `utf-8`, not `utf-8-sig`; amount parsing rejects `1,200.00` | Excel "CSV UTF-8" files make every row Invalid (E02); common amount formats are rejected | sync.py:69, sync.py:34 |
| 4 | `exists()` has no error handling, and `r.json()` on a 422 is not guarded | One network blip or non-JSON reply stops the whole run; later invoices are skipped (E08) | sync.py:41-43, sync.py:59-60 |
| 5 | No end-of-run marker or summary in the report | Ops cannot tell a finished run from one that stopped part-way | sync.py:77 (log only) |
| 6 | 401/403/404 and other 4xx are retried like outages and end as "Failed - No success after 5 tries" | A wrong/expired access key looks like a billing outage, and each invoice burns ~1-6 min | sync.py:54-62 |
| 7 | No guard against two hourly runs overlapping; report named only to the minute | Long runs can overlap and send the same invoice twice (E10); two runs in the same minute overwrite one report | sync.py:68, Task Scheduler settings |
| 8 | Version is only in README; not shown in the report or log | Ops cannot confirm an update applied | README.md:5 |
| 9 | `data/output` is not created by the code | Fresh install or a cleared folder = no report, every hour (E09) | sync.py:68-69 |
| 10 | The `.env` settings file (billing access key) is committed to git | Credential exposed to anyone with repo access; rotate the key and remove it from history | .env (in git since 9fa3ae1) |
| 11 | After the 5th failed try the bot still waits 32 s before giving up | Slows runs with no benefit | sync.py:62 |
tests/evals/results/handoff/h-desktop-poster-full-r1.json
+34
View File
@@ -0,0 +1,34 @@
{
"case": "h-desktop-poster/full",
"rep": 1,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"comma rule before a run": true,
"how to stop": true,
"not-confirmed row: check ClaimDesk first": true,
"hidden tech comments flagged": false,
"Pending batch blocks a new upload": true
},
"cost_usd": 1.7011209999999997,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-desktop-poster-full-qgejgjlt\\project",
"reply": "`HANDOVER.md` ready, repo root-la irukku. Code onnum maathala, bot-a run pannala. Ungaloda answers-a (#rpa-support, ops access, unclear statuses) apdiye use pannen, adhanaala unga kitta question ethuvum ketkala.\n\n| Check | Result |\n|---|---|\n| Sections | 13 (template-la irukka order-laye) |\n| Exceptions | 18 (E01 to E18) |\n| Failure classes | 12/12 covered, \"not applicable\" onnum illa |\n| Statuses | Code ezhudhura ella status-um comment-um Section 6-la irukku |\n| PHI / credentials | Doc-la edhuvum illa; file:line refs section 13-la mattum |\n\n**Fix before handover:** dev team-ku 18 items. Mukkiyamanavai:\n1. **Oru red row (or failed/killed run) vandha batch eppavum open-a irukkum, next upload block aagum.** Database-a kaila edit pannaa dhaan clear aagum. Ops-ku database access illa, so ovvoru failure-kum dev-a wait pannanum.\n2. \"Rejected by app\", \"Skipped\" rows-um green Completed-a dhaan kaatum. Post aagaadha remittance kooda done maadhiri theriyum.\n3. Window-a title-la \"ClaimDesk\" irukka-nu mattum paarthu edukkudhu. \"Fixed\"-nu commit irundhaalum code maaralai, matching apdiye dhaan irukku. Launcher window-um idhula match aagum.\n4. Bot oru remittance-um process pannaama 5 min aanaa, watchdog full run-aiyum kill pannidum; slow-a irukkra healthy run-um kill aagidum.\n5. `logs/run_20260901_101500.log` line 3-la patient name irukku, ops-ku andha folder access irukku. Delete pannalaam-a-nu decide pannunga.\n6. Bot-a safe-a stop panna vazhi illa. Launcher code repo-la illa, adhanaala update fix-um check panna mudiyala.\n\nDoc-la sila points code-a padichu naan purinjukittadhu mattum dhaan, adhai confirm pannala:\n- Wrong password vandha ella clinic-um \"not found in list\"-nu fail aagum.\n- Launcher close panna bot stop aagumaa-nu theriyala.\n\nRepo-la bug fix commits nalu irukku. Adhula save wait 10s-la irundhu 20s aanadhu mattum dhaan current code-la theriyudhu.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions mean whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-desktop-poster-full-r1.md
+190
View File
@@ -0,0 +1,190 @@
# ClaimDesk Poster - ops handover
Version 1.4.2 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
ClaimDesk Poster posts remittance payments into the **ClaimDesk** desktop app, one clinic at a time. Ops start it from the **ClaimDesk Poster** launcher window with an input Excel that has three sheets: **Login** (ClaimDesk user name and password), **Clinics** (the clinics to work) and **Settings** (an optional cutoff date). For every clinic the bot opens ClaimDesk, signs in, opens the clinic, and works through every unposted remittance in the list: ACH payments are posted and saved; other payment methods are skipped; checks dated after the cutoff are deferred. The result of every remittance is shown on the launcher's **Report screen**. The bot writes one run log per run in the `logs` folder. It drives one app: **ClaimDesk**.
## 2. How it works
```
ClaimDesk Poster launcher (the window you use, Report screen)
└─ starts → posting bot (runs in the background) ──drives──> ClaimDesk desktop app
│ (do not touch while it runs)
├─ writes → bot database (ops cannot open it; the Report screen reads it)
└─ writes → logs/run_<date>_<time>.log
```
- **Launcher**: where you start the run, see the Report screen, and read the version in the title bar.
- **Posting bot**: a background process the launcher starts. It has a built-in watchdog that kills the whole run if no remittance is processed for 5 minutes.
- **ClaimDesk**: the bot starts it fresh for every clinic, and force-closes it after every clinic, whether that clinic worked or not. This closes **every** ClaimDesk window on the machine, including one a person has open.
- **Bot database**: holds the batch, the clinics and one row per remittance. Ops see it only through the Report screen.
- **Run log**: one file per run, named by the start time, e.g. `logs/run_20260901_101500.log`.
**One run, step by step:**
1. Check that no earlier batch is still open. If one is, the bot stops at once (E01).
2. Read the Excel: Login, Clinics, Settings. Create a new batch with every clinic set to Pending.
3. For each clinic: start ClaimDesk → sign in → click the clinic in the list → wait for the remittance list.
4. For each remittance in the list: skip / defer / post → Save → wait up to 20 s for "Payment saved". One row goes to the Report screen.
5. Close ClaimDesk. If anything in step 3 failed, try the whole clinic again (up to 3 attempts).
6. After the last clinic, the batch is marked Completed **only if** no row is Error and no clinic is still Pending. Otherwise it stays open and blocks the next upload (E01).
## 3. Before a run
- [ ] ClaimDesk is installed on this machine and the bot's settings file (`.env`, next to the bot) has the path to it. Without it every clinic fails (E03). Only the dev team edits this file.
- [ ] The last batch finished as Completed. If it did not, the next run will not start (E01) and you need the dev team first.
- [ ] **Login sheet**: user name in cell A2, password in cell B2. Both must be filled, and the password must be current. A wrong password does not give a clear message; every clinic then fails with "not found in list" (E05).
- [ ] **Clinics sheet**: column A, from row 2 down, **one clinic per cell, no commas and no semicolons anywhere**. The bot splits a cell at every comma or semicolon.
- Bad: `Clinic A, Clinic B` in one cell → the bot looks for two names it may not find. Bad: a clinic whose real name contains a comma (`Smith, Jones Family Care`) → split into `Smith` and `Jones Family Care`, both fail. Bad: a trailing comma `Clinic A,` → an extra empty clinic that fails.
- Good: `Clinic A` in A2, `Clinic B` in A3, spelled exactly as in the ClaimDesk clinic list.
- A clinic whose real name contains a comma cannot be run today. Tell the dev team (see section 13).
- [ ] **Settings sheet**: cell B1 is the cutoff date, or empty for no cutoff. Type it in the **same format** the ClaimDesk remittance list shows dates. The bot compares the two as text, not as dates, so a different format defers or posts the wrong checks (E13).
- [ ] All three sheets exist, named exactly Login, Clinics, Settings. A missing sheet stops the run at once (E02).
- [ ] The launcher title bar shows version 1.4.2 (see section 10).
## 4. During a run
- **Hands off this machine.** Don't use the mouse or keyboard. The bot clicks on screen; your clicks move the focus and its clicks land in the wrong place.
- **Close every other window whose title contains "ClaimDesk"** before the run, and don't open one during it: a database browser, a File Explorer folder, a browser tab, a document. The bot takes the first window with "ClaimDesk" in its title as the app, so it can drive the wrong window and then close the real ClaimDesk (E15).
- **Don't use ClaimDesk yourself on this machine during a run.** The bot force-closes all ClaimDesk windows after every clinic.
- Closing ClaimDesk does **not** stop the bot. It treats it as a failure, starts ClaimDesk again and retries the clinic.
- **To stop the bot**: there is no Stop button in the bot. Closing the launcher is not confirmed to stop it. If you must stop it, ask in #rpa-support first. Any stop leaves the batch open, and the next run will not start until the dev team clears it (E01).
- **A run looks stuck** when the newest run log in `logs` has no new line for more than 5 minutes. The bot's watchdog kills the run after 5 minutes with no remittance processed; the last line is then `[WATCHDOG] no progress for 300s - killing run` (E14). Wait 6 minutes before escalating.
- One clinic can take up to about 6-7 minutes when it fails all 3 attempts (each attempt waits up to 60 s for sign-in and up to 60 s for the remittance list). A clinic with many remittances takes longer; that is normal as long as new lines keep appearing.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | Newest file in `logs`: the last line is `[RUN SUMMARY] batch <n> done`. If the last line is `[WATCHDOG] ...`, the run was killed (E14). If the file is empty, the run never started (E01 / E02). |
| Did the batch complete? | Not visible to ops today (section 13, item 3). Rule of thumb: if the Report screen shows any red Error row, or any clinic has no rows at all, the batch is still open and the next run will be blocked. |
| Which remittances failed? | Report screen → the batch → rows in **red** (File Status Error). |
| Why did one remittance fail? | Report screen → **Comments** column for that check number. |
| Which clinics failed completely? | A clinic with **no rows** on the Report screen. Then search the run log for the clinic name: lines `[ERROR] [ERROR] Clinic '<name>' : <reason>` and `[DB] Clinic '<name>' left Pending after 3 attempts`. |
| Did the bot already retry? | Run log: `[AUTO-RERUN] attempt 2 for '<name>'` and `attempt 3`. Single remittances are never retried. |
| Was a green row really posted? | Comments column. Only **"Posted successfully"** means posted. Green rows with other comments were not posted (section 6). |
| Which version is running? | The launcher title bar. |
Note: error lines in the log start with `[ERROR] [ERROR]` (the word appears twice). That is normal for this version.
## 6. Statuses
The Report screen shows **File Status** (as a colour) and **Comments**. Read both: three different outcomes share the green colour.
| File Status | Colour | Comments | Meaning | What ops does |
|---|---|---|---|---|
| Completed | green | Posted successfully | Payment posted and saved in ClaimDesk | nothing |
| Completed | green | Remittance Errors on posting | **Not posted.** ClaimDesk showed its Remittance Errors screen; the bot cancelled. **Needs follow-up.** | Review the remittance in ClaimDesk by hand (E11) |
| Completed | green | Payment method '<method>' - not posted | **Not posted on purpose**: not an ACH payment | nothing, unless the client wants non-ACH posted (E12) |
| Deferred | grey | Check date <date> is after the cutoff - deferred | Not posted: dated after the cutoff in Settings B1 | It stays unposted in ClaimDesk and is picked up by a later run. Check the date format if many rows defer (E13) |
| Error | red | Payment not confirmed after Save | Saved, but "Payment saved" did not appear in 20 s. **It may have posted.** | **Check ClaimDesk before doing anything** (E09) |
| Error | red | Save button not found | Not posted: the Save button was missing on the payment screen | Escalate (E10) |
| Pending | amber | (none) | Listed on the screen, but the current bot never writes it | If you see it, escalate |
| (no rows for a clinic) | - | - | The clinic failed all 3 attempts, or it had nothing to post | See E08 and E18 |
Statuses kept in the bot database that ops cannot see: each clinic is **Pending** or **Success**, and each batch is **Pending** or **Completed**. A clinic is set to Success even when it had zero remittances (E18). A batch stays Pending if any row is Error or any clinic is Pending.
## 7. Exception catalogue
Quote the ID (E01, E02...) in #rpa-support.
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Run stops at once with "A Pending batch exists - finish or clear it before a new upload". The new run log is empty. | At start it checks for an earlier batch that never completed. Any batch with one red Error row, one failed clinic, or a killed run stays open for good. | On screen only; not in the run log | System | **Dev** | Nothing ops can do: ops cannot clear a batch. Don't retry. | Always. Send: the time, the previous run's log file, and which earlier batch had red rows or a killed run. |
| E02 | Run stops at once with "'Login' sheet not found", "UserName/Password missing in Login sheet (row 2)", or a technical error naming Clinics or Settings. The new run log is empty. | The Excel has no Login sheet, A2 or B2 is empty, or the Clinics / Settings sheet is missing. | On screen only | Business | Ops | Fix the Excel (section 3) and start again. No batch was created. | The sheets are correct and it still stops. Send the error text and the time (not the Excel, it holds the password). |
| E03 | Every clinic fails, log lines `[ERROR] [ERROR] Clinic '<name>' : 'CLAIMDESK_EXE'` or `... : Login window not found` for every clinic | The bot can't find where ClaimDesk is installed, or ClaimDesk doesn't open its sign-in window within 60 s. | Run log | System | Bot, then Dev | The bot retries each clinic 3 times. Check that ClaimDesk opens normally by hand on this machine (after the run). | Every clinic shows it. Send the log file. The batch is now open → E01. |
| E04 | Log line `... : Login window not found` for some clinics only | ClaimDesk's sign-in window did not appear within 60 s (slow start). | Run log | System | Bot | Wait; the bot retries the clinic up to 3 times. | The clinic is left Pending (E08). |
| E05 | Log line `... : Clinic '<name>' not found in list`; the clinic has no rows on the Report screen | After sign-in, the bot looks 10 s for a link with exactly that clinic name. | Run log | Business | Ops | Check the spelling against ClaimDesk, and that the cell has no comma or semicolon. **If every clinic shows this, the password in the Login sheet is probably wrong.** | Spelling and password are correct. |
| E06 | Log line `... : Wrong clinic opened, leaving Pending: <name>` | Right after clicking the clinic, the ClaimDesk title does not contain the clinic name. It checks with no wait, so a slow screen or a name with `&` can fail even when the right clinic opened. | Run log | System | Bot, then Dev | Wait for the 3 attempts. Check the spelling. | It fails all 3 attempts with the spelling exact. Send the log file and the clinic name. |
| E07 | Log line `... : Remittance grid did not load` | The remittance list did not appear within 60 s after opening the clinic. | Run log | System | Bot, then Dev | Wait for the 3 attempts. | The same clinic fails it on two runs. Send the log file. |
| E08 | Log line `[DB] Clinic '<name>' left Pending after 3 attempts`; the clinic has no rows on the Report screen (or only rows from an earlier attempt) | All 3 attempts failed. | Run log | System | Ops, then Dev | Read the `[ERROR]` lines just above it for the reason (E03-E07) and act on that one. The batch is now open → E01. | Always, to get the batch cleared. Send the log file and the clinic name. |
| E09 | Red row, Comments "Payment not confirmed after Save" | After Save the bot waits 20 s for "Payment saved". It never appeared. | Report screen | System | **Ops** | **Check the payment in ClaimDesk first** (the check number). If it posted: do not post again; note it for the dev team. If it did not post: post it by hand or leave it for the next run. Re-running or re-posting without checking can post it twice. The row stays red (ops can't edit it) and the batch stays open → E01. | Always, to get the row corrected and the batch cleared. Send the check number, what ClaimDesk shows, and the log file. |
| E10 | Red row, Comments "Save button not found" | The payment screen opened but its Save button did not appear within 10 s. | Report screen | System | Dev | Check in ClaimDesk that the payment was not posted. | Always. Send the check number, the time and the log file. |
| E11 | Green row, Comments "Remittance Errors on posting" | ClaimDesk showed its Remittance Errors screen within 5 s of Post Payment; the bot pressed Cancel. | Report screen | Business | Ops | **Not posted.** Review the remittance in ClaimDesk by hand and resolve it there. | The same remittance keeps coming back on every run. |
| E12 | Green row, Comments "Payment method '<method>' - not posted" | The remittance's method is not ACH. | Report screen | Business | Ops | Nothing; the bot posts ACH only. Post by hand if the client wants it. It reappears on every run while unposted. | Never. |
| E13 | Grey row, Comments "Check date <date> is after the cutoff - deferred" | The check date is compared with Settings B1 **as text**, not as a date. | Report screen | Business | Ops | Normal when the date really is after the cutoff. If checks dated before the cutoff were deferred (or later ones posted), fix the B1 format to match how ClaimDesk shows dates. | B1 matches ClaimDesk's format and it's still wrong. |
| E14 | Run stopped in the middle; last log line `[WATCHDOG] no progress for 300s - killing run`; later clinics never ran | No remittance was processed for 5 minutes. Sign-in and list waits don't count as progress, so 3 slow attempts in a row can trigger it. | Run log | System | Dev | Don't restart: the batch is open → E01. ClaimDesk may still be open; leave it. | Always. Send the log file and the time. |
| E15 | The bot types or clicks in the wrong window, or a ClaimDesk you had open suddenly closes | The bot takes the first window with "ClaimDesk" in its title, and force-closes all ClaimDesk windows after every clinic. | Not recorded; you see it on screen | System | Ops, then Dev | Follow the section 4 rules: no other "ClaimDesk" windows, no one using ClaimDesk on this machine. | It happens with no other ClaimDesk window open. Send the time and the log file. |
| E16 | The same check number appears twice or three times on the Report screen | Each retry of a clinic writes rows again for everything still unposted. | Report screen | System | Ops | Count each check once. Skipped / deferred / rejected duplicates are harmless. If one copy is red and another green "Posted successfully", check ClaimDesk (E09). | A red copy keeps the batch open → E01. |
| E17 | The launcher title still shows an old version after an update | The launcher keeps the old folder until it is fully restarted. | Launcher title bar | System | Ops | Close the launcher completely and start it again. Check the title bar. | Still the old version after a restart. |
| E18 | A clinic that should have work shows no rows at all, and the log has no error for it | The remittance list loaded but was empty when read (possibly not filled in yet). The bot marks the clinic done with nothing posted. | Nowhere ops can see | System | Ops, then Dev | Open the clinic in ClaimDesk (after the run) and check for unposted remittances. | Remittances are there. Send the clinic name, time and log file. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Sign-in, clinic, or remittance list fails, or any click fails (E03-E07) | Closes ClaimDesk, starts it again, signs in, retries the whole clinic | 3 attempts per clinic, no pause between; each up to ~2 min | Clinic left Pending (E08), next clinic. Batch stays open. |
| "Payment saved" doesn't appear | Waits | 20 s, **no retry** | Row Error (E09), next remittance |
| Save button missing | Waits | 10 s, **no retry** | Row Error (E10), next remittance |
| Remittance Errors screen | Presses Cancel | waits 5 s for it | Row green "Remittance Errors on posting" (E11) |
| No remittance processed for a while | Watchdog kills the whole run | 5 min (checked every 10 s) | Run ends; nothing restarts it; batch open (E14, E01) |
The bot never retries a single remittance, never resumes a killed run, and never clears an open batch.
## 9. Known incidents
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Up to 1.3.0 | Users stayed on 1.3.0 after an update | The launcher kept the old folder until it was restarted | Restart the launcher fully and check the title bar (E17). Fix not visible in this repo. | Yes |
| Before 1.4.2 | Red rows "Payment not confirmed after Save" that had really posted | ClaimDesk took longer than the 10 s wait | Wait raised to 20 s. Still check ClaimDesk before re-posting (E09). | Partly |
| Before 1.4.2 | Clinics with a comma in the name were split and skipped | The bot splits cells at commas | One clinic per row, no commas (section 3). Still open for names that really contain a comma. | Partly |
| Before 1.4.2 | The bot drove a database browser whose title contained "ClaimDesk", then closed the real ClaimDesk | Window found by title text | Close other "ClaimDesk" windows (section 4). Marked fixed, but the code still matches by title. | Partly |
| 2026-09-01 run log | Clinic with `&` in its name: "Wrong clinic opened, leaving Pending" | Not diagnosed (no wait before the title check, or `&` shown differently) | Escalate if spelling is exact (E06) | No |
| 2026-09-01 run log | Run killed by the watchdog after auto-reruns | Retries without progress add up to 5 min | Escalate (E14) | No |
## 10. Releases and version check
- **How an update arrives**: through the launcher. The launcher's update steps are not in this repo; ask #rpa-support when a new version is due.
- **How to confirm it applied**: close the launcher completely, start it again, and read the version in its title bar.
- **If the version did not change**: restart the launcher once more. If it is still old, post in #rpa-support with the version you see. Don't run on the old version.
- Every machine that runs the bot must be on the same version.
## 11. Escalating to dev
Post in **#rpa-support** (RPA dev team). Send:
- the exception ID from section 7 (E01...),
- the date and time of the failure,
- the clinic name and check number when it is about one remittance,
- the run log file `logs/run_<date>_<time>.log` (attach the file),
- a screenshot of the Report screen or of the ClaimDesk message.
Never send: passwords, the input Excel (its Login sheet holds the password), the `.env` settings file, or patient data in chat. Don't paste log lines that contain patient names (older logs may have them; see section 13).
## 12. Failure-class check
- F1 covered in section 3 (ClaimDesk path, sheets) and E02, E03.
- F2 covered in section 4 and E15; root cause in section 13 item 6.
- F3 covered in E04, E06, E07, E09, E10 and section 8. No screen-size dependency found in the code.
- F4 covered in section 3 (commas, cutoff format, exact spelling) and E05, E13; section 13 items 8, 9.
- F5 covered in E18; section 13 item 5.
- F6 covered in E09.
- F7 covered in section 6 and E11, E12; section 13 item 4.
- F8 covered in E14 and section 8; section 13 item 7. No endless loop: a clinic stops after 3 attempts.
- F9 covered in section 4 (no Stop button, closing ClaimDesk doesn't stop it); section 13 item 10.
- F10 covered in E01, E08, E09, E16; section 13 items 1, 2.
- F11 covered in section 5 and E01, E02, E18; section 13 items 3, 11.
- F12 covered in section 10 and E17; section 13 item 12. The bot writes nothing to shared resources at start.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | An open batch blocks every new upload, and nothing can clear it except a hand edit of the database | Ops is blocked after any failure and must wait for dev every time | `bot.py:167-168` |
| 2 | A batch can never complete once any row is Error or any clinic is Pending; nothing re-checks it later, and retries leave the old Error rows behind | One red row = the next run is blocked (E01) | `bot.py:150-154`, `bot.py:177-179`, `bot.py:139` |
| 3 | Batch and clinic status are in the database only, not on the Report screen | Ops can't see whether a batch completed or which clinic failed without the log | `report_screen.py:6`, `bot.py:140`, `bot.py:178` |
| 4 | "Rejected by app" and "Skipped" are stored as Completed (green), same as Posted | Unposted remittances look done; ops must read every comment | `bot.py:27-33` |
| 5 | A clinic is marked Success after reading the remittance list with no positive count; an empty or not-yet-filled list counts as done | Work silently not done (E18) | `bot.py:96-101`, `bot.py:140` |
| 6 | Window found by "ClaimDesk" anywhere in the title, which also matches the launcher "ClaimDesk Poster" and "ClaimDesk Login"; the commit "fix: bot grabbed the SQL browser window" did not change this code | Bot drives the wrong window (E15); a wrong password looks like "clinic not found" | `bot.py:66-70`, `bot.py:83` |
| 7 | Watchdog heartbeat only on each remittance; sign-in and list waits (up to 3 × ~130 s) and empty clinics never reset it; kill leaves ClaimDesk open and the batch open | Healthy-but-slow runs get killed (E14) | `bot.py:39-49`, `bot.py:137` |
| 8 | Clinic cells split at `,` and `;`; real names with a comma can't be run; a trailing comma creates an empty clinic | Clinics silently fail (E05) | `bot.py:62` |
| 9 | Cutoff date compared as text with the grid's date text | Wrong checks deferred or posted (E13) | `bot.py:107` |
| 10 | No Stop button / clean stop; launcher code not in the repo, so whether closing it stops the bot is unknown | Ops has no safe way to stop a run | `bot.py:157-180` |
| 11 | Start-up errors (open batch, Excel problems) are not written to the run log, only to the screen | Lost once the window closes; empty log | `bot.py:167-169`, `bot.py:52-63` |
| 12 | Launcher/update code is not in this repo; the 1.3.0 update fix can't be checked | Version drift; the 2026-09-01 log has lines this code never writes, so it came from another build | `report_screen.py:8`, `report_screen.py:14` |
| 13 | The existing log `logs/run_20260901_101500.log` contains a patient name (line 3). Ops can open the logs folder. Current code doesn't write that line. | PHI exposed to anyone with log access; could be pasted into chat | `logs/run_20260901_101500.log:3` |
| 14 | "Wrong clinic opened" checked immediately after the click, with no wait; names with `&` fail | Clinics left Pending for no real reason (E06) | `bot.py:90-92` |
| 15 | Force-close of all ClaimDesk instances after every clinic, including successful ones | Closes a person's ClaimDesk on the same machine | `bot.py:146` |
| 16 | "Payment not confirmed after Save" has no re-check in ClaimDesk (e.g. is the check still in the unposted list?) and no way for ops to correct the row | Double-posting risk; red row blocks the batch (E09) | `bot.py:117-118` |
| 17 | Error log lines carry a doubled `[ERROR] [ERROR]` prefix | Confusing when searching logs | `bot.py:144`, `bot.py:165` |
| 18 | ClaimDesk password stored in plain text in the input Excel | Excel can't be shared for support | `bot.py:56` |
tests/evals/results/handoff/h-desktop-poster-full-r2.json
+34
View File
@@ -0,0 +1,34 @@
{
"case": "h-desktop-poster/full",
"rep": 2,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": false,
"asks ops access": false,
"known incidents from history": true,
"misleading status flagged": true,
"comma rule before a run": true,
"how to stop": true,
"not-confirmed row: check ClaimDesk first": true,
"hidden tech comments flagged": true,
"Pending batch blocks a new upload": true
},
"cost_usd": 1.8916888000000003,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-desktop-poster-full-fbmcbuke\\project",
"reply": "Handover already mudinjiduchu, innum seyya vendiyadhu edhuvum illa.\n\n- `HANDOVER.md` repo root-la irukku: 13 sections, 19 exceptions (E01\u2013E19), 24 fix-before-handover items.\n- File commit pannala.\n- Ovvoru item-um mela irukkura en previous reply-la irukku.\n\nAdhula edhavadhu maathanum na sollunga.",
"stderr": ""
}
tests/evals/results/handoff/h-desktop-poster-full-r2.md
+221
View File
@@ -0,0 +1,221 @@
# ClaimDesk Poster - ops handover
Version 1.4.2 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
ClaimDesk Poster posts remittance payments into the **ClaimDesk** desktop app, one clinic at a time. It reads the input Excel file (sheets **Login**, **Clinics** and **Settings**). For every clinic it signs in to ClaimDesk, opens that clinic, goes through the list of unposted remittances, and posts each ACH payment that is dated on or before the cutoff date. Every remittance it looks at becomes one row on the **Report screen** of the "ClaimDesk Poster" launcher. It runs when ops start it from the launcher. It drives one desktop app: **ClaimDesk**.
## 2. How it works
```
ClaimDesk Poster launcher (the window you use; title bar shows the version)
└─ starts → Posting bot ──drives──> ClaimDesk desktop app
│ (do not touch it while the bot runs;
│ the bot force-closes it after every clinic)
├─ writes → Report screen (one row per remittance)
├─ writes → logs/run_<date>_<time>.log (one file per run)
└─ watchdog: stops the whole run after 5 minutes with no new row
```
- **Launcher:** where you start a run and read the Report screen. The title bar shows the version.
- **Posting bot:** a background program with no window of its own. It does all the clicking.
- **ClaimDesk:** the bot opens it, signs in, and force-closes it again after every clinic, whether the clinic worked or not.
- **Report screen:** one row per remittance: Clinic, Check No, File Status (coloured), Comments. It can be filtered by batch.
- **Run log:** one text file per run in the `logs` folder. It is the only place that shows clinic-level problems, retries and watchdog stops.
- **Watchdog:** if no new Report row appears for 5 minutes, it stops the whole bot at once.
**One run, step by step:**
1. The bot checks that no earlier batch is still open (Pending). If one is, it refuses to start (E01).
2. It reads the input Excel file and creates a new batch. Each clinic in it starts as Pending.
3. For each clinic, it opens ClaimDesk and signs in with the Login sheet's username and password.
4. It clicks the clinic's name in ClaimDesk's clinic list and checks that the window title now shows that clinic.
5. It waits for the remittance list (grid) to load.
6. For each row in the grid:
- not ACH → **Skipped**
- dated after the cutoff → **Deferred**
- otherwise it clicks Post Payment, then Save, and waits for "Payment saved"
7. It writes one Report row per remittance, closes ClaimDesk, and moves on to the next clinic. If anything goes wrong at clinic level, it retries that clinic up to 3 times in total.
8. At the end the batch is marked Completed only if every clinic finished and no Report row is red (Error) or amber (Pending). Otherwise the batch stays **Pending**, and that blocks the next run (E01, E10).
## 3. Before a run
- [ ] **Check the previous batch finished.** Open the last run log. Its last line must be `[RUN SUMMARY] batch <n> done`, and it must have no `left Pending` line. Then on the Report screen, that batch must have no red Error rows. If it fails any of these checks, the new run will refuse to start (E01). Raise it with dev first.
- [ ] The settings file (`.env`) is in the bot's folder. You never need to open or edit it. Without it, every clinic fails (E03).
- [ ] The input Excel file is the one the launcher uses (`data/input/input.xlsx`) and has all three sheets:
- **Login:** username in A2, password in B2.
- **Clinics:** clinic names in column A, starting at row 2.
- **Settings:** the cutoff date in B1.
- [ ] **One clinic per row in the Clinics sheet. No commas or semicolons anywhere in the cell.** The bot splits a cell at every comma and semicolon.
- Bad: `North Clinic, South Clinic` in one cell, or a clinic whose real name contains a comma, like `Smith, Jones Family Care`.
- Good: `North Clinic` in A2 and `South Clinic` in A3.
- A clinic whose real ClaimDesk name contains a comma **cannot be posted by this version** (Fix item 7). Post it by hand.
- [ ] Clinic names are spelled exactly as ClaimDesk's clinic list shows them, including `&`, dots and spaces.
- Bad: `St Marys & Co` when ClaimDesk shows `St. Mary's & Co.`
- Good: `St. Mary's & Co.`
- [ ] The cutoff date in Settings B1 is a real date. The bot compares dates as text, so check a few Deferred rows after the run (E15).
- [ ] The machine stays unlocked for the whole run, and the screen saver and sleep are off. The bot moves the real mouse.
- [ ] The launcher title bar shows version **1.4.2** (see section 10).
## 4. During a run
- **Hands off this machine.** Don't use the mouse or keyboard. The bot clicks with the real mouse, so your clicks and its clicks get mixed up.
- **Close every window whose title contains "ClaimDesk"** before you start, except the launcher: SQL or database browsers, a second ClaimDesk, documents or emails with "ClaimDesk" in their title. The bot takes the first window whose title contains "ClaimDesk", and it has driven the wrong one before (section 9).
- **Don't open or use ClaimDesk yourself on this machine during a run.** The bot force-closes ClaimDesk after every clinic, and anything unsaved is lost.
- **To stop the bot:** the code has no Stop button, and closing the launcher or ClaimDesk is not confirmed to stop it.
- If you must stop it, end the Python process running the bot in Task Manager, then close ClaimDesk.
- The batch then stays Pending and blocks the next run, so tell dev (E01).
- **A run looks stuck** when no new Report row and no new log line appears for several minutes. Don't act yet:
- A clinic that keeps failing can take up to about 7-8 minutes: 3 attempts, each with up to 60 s for the login window, 10 s for the clinic link and 60 s for the grid.
- The watchdog stops the whole run after **5 minutes with no new Report row**.
- Wait at least 10 minutes before you treat it as stuck. Then check the log for `[WATCHDOG]` (E16).
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | Last line of `logs/run_<date>_<time>.log` is `[RUN SUMMARY] batch <n> done`. **"done" only means the run reached the end. It does not mean everything posted.** If the log stops without that line, the run died (E16). |
| Which batch number was it? | The `[RUN SUMMARY] batch <n>` line in the log. Use that number as the batch filter on the Report screen. |
| Which remittances failed? | Report screen, filter by batch: File Status **red (Error)**. Also read the green rows whose Comments are not "Posted successfully" (see section 6). |
| Which clinics failed completely? | **Not on the Report screen.** Search the run log for `left Pending`. A failed clinic may have no Report rows at all. |
| Why did one remittance fail? | Report screen, Comments column. |
| Why did one clinic fail? | Run log: search for `Clinic '<name>'`. The reason is on the `[ERROR]` line. |
| Did the bot already retry? | Run log: `[AUTO-RERUN] attempt 2` / `attempt 3` lines for that clinic. |
| Why did the run refuse to start, or end at once? | The new log file is empty. See E01 and E02. |
| Which version is running? | Launcher title bar. |
| Is the batch Pending or Completed? | **Ops cannot see this** (it is in the database only). Use the first-row check in section 3, or ask dev. |
## 6. Statuses
Report screen, File Status column, with the Comments text that goes with it:
| Status (File Status - Comments) | Colour | Meaning | What ops does |
|---|---|---|---|
| Completed - "Posted successfully" | green | Payment posted and ClaimDesk showed "Payment saved". | nothing |
| Completed - "Remittance Errors on posting" | green | **NOT posted.** ClaimDesk showed its Remittance Errors screen and the bot cancelled. **Needs follow-up despite the green colour.** | Review and post it in ClaimDesk by hand (E13). |
| Completed - "Payment method '<method>' - not posted" | green | Not ACH, so skipped on purpose. **Not posted.** | Nothing, unless the client wants non-ACH payments posted by hand (E14). |
| Deferred - "Check date <date> is after the cutoff - deferred" | grey | Dated after the cutoff in Settings B1. Not posted. | Nothing. A later run picks it up. Spot-check the date (E15). |
| Error - "Save button not found" | red | The payment screen had no Save button. Not saved. **Blocks the batch.** | E11 |
| Error - "Payment not confirmed after Save" | red | Save was clicked but "Payment saved" did not appear within 20 s. **It may really have posted.** Blocks the batch. | E12. Check in ClaimDesk first. |
| Pending (any comment) | amber | Fallback for a result the bot does not recognise. The current version never writes it. Blocks the batch. | Send to dev. |
Clinic and batch statuses (stored in the database, **not shown to ops**):
| Status | Meaning | How ops can tell |
|---|---|---|
| Clinic Pending | The clinic failed 3 attempts or the run died before reaching it. It blocks the batch. | Log line `[DB] Clinic '<name>' left Pending after 3 attempts`, or no rows for that clinic on the Report screen |
| Clinic Success | The clinic was processed. It can have 0 rows if ClaimDesk listed nothing. | No `left Pending` line for it |
| Batch Pending | At least one Pending clinic, or a red or amber row. **The next run refuses to start.** | Section 3 first check; E01 |
| Batch Completed | Everything finished with no red or amber row | Section 3 first check passes |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | You start a run, nothing happens, and the new log file is empty. | Before reading the Excel file, the bot looks for any earlier batch still Pending and refuses to start if it finds one: "A Pending batch exists - finish or clear it before a new upload". | Not in the log. The message only goes to the bot's console. | System | **Dev** | First rule out E02: check the input file. If the input is fine, the previous batch is blocking. Ops cannot clear it. | Always, once E02 is ruled out. Send: the previous run's log file name and its batch number, the time of the failed start, "E01". |
| E02 | Same as E01: nothing happens and the new log file is empty. | The input file is missing the Login sheet, the username or password in Login row 2, or the Clinics or Settings sheet. Messages: "'Login' sheet not found" or "UserName/Password missing in Login sheet (row 2)". | Not in the log. Console only. | Business | Ops | Fix the input file (section 3) and start again. | The input file is correct and it still fails. Treat it as E01. |
| E03 | Every clinic fails. The log shows `[ERROR] Clinic '<name>' : 'CLAIMDESK_EXE'` | The bot cannot find where ClaimDesk is installed: the settings file is missing or incomplete. | Run log; every clinic `left Pending` | System | Ops, then Dev | Check the `.env` file is in the bot's folder. Don't open it or send it. | The file is there. Send: log file name, "E03". |
| E04 | Log: `[ERROR] Clinic '<name>' : Login window not found` | ClaimDesk's login window did not appear within 60 s of starting ClaimDesk. | Run log | System | Bot, then Ops | The bot retries the clinic up to 3 times. After that, open ClaimDesk by hand once to check it starts and signs in, close it, and tell dev the clinic needs a re-run. | Every clinic fails like this. Send: log file name, batch number, "E04". |
| E05 | Log: `[ERROR] Clinic '<name>' : ClaimDesk window not found` | After sign-in, no window title contained "ClaimDesk". | Run log | System | Bot, then Dev | Wait for the 3 attempts. | It ends in `left Pending`. Send: log file name, batch number, "E05". |
| E06 | Log: `[ERROR] Clinic '<name>' : Clinic '<name>' not found in list` | The clinic's name did not appear as a link in ClaimDesk's clinic list within 10 s. | Run log; after 3 attempts, `left Pending` | Business (usually) | Bot, then Ops | Check the Clinics sheet against ClaimDesk: exact spelling, one clinic per row, no commas or semicolons. If **every** clinic shows this, check the Login username and password: a failed sign-in shows up as this message. | The spelling is exact and the clinic is visible in ClaimDesk. Send: log file name, batch number, "E06". |
| E07 | Log: `[ERROR] Clinic '<name>' : Wrong clinic opened, leaving Pending: <name>` | After clicking the clinic, the ClaimDesk title bar did not contain the clinic name exactly. Names with `&` or other special characters have caused this. | Run log | System | Bot, then Ops | Compare the name in the Clinics sheet with ClaimDesk's title bar for that clinic. | The spelling is already exact. Send: log file name, batch number, "E07". |
| E08 | Log: `[ERROR] Clinic '<name>' : Remittance grid did not load` | The remittance list did not appear within 60 s of opening the clinic. | Run log | System | Bot, then Ops | The bot retries 3 times. ClaimDesk may be slow. Tell dev the clinic needs a re-run. | The same clinic fails like this in 2 runs. Send: log file name, batch number, "E08". |
| E09 | Log: `[ERROR] Clinic '<name>' : <any other text>` | An unexpected failure inside the clinic, for example the Post Payment button was not found. | Run log | System | Bot, then Dev | Wait for the 3 attempts. | It ends in `left Pending`. Send: log file name, batch number, the full `[ERROR]` line, "E09". |
| E10 | Log: `[DB] Clinic '<name>' left Pending after 3 attempts`. The clinic has few or no rows on the Report screen. | All 3 attempts for the clinic failed (E03-E09). | Run log only. **Not on the Report screen.** | System | Dev | Read the `[ERROR]` lines above it for the reason, and fix input causes (E06, E07). The batch stays Pending and **blocks the next run**. Ops cannot release it. | Always. Send: log file name, batch number, clinic name, the reason from the `[ERROR]` line, "E10". |
| E11 | Report: red **Error** - "Save button not found" | After Post Payment, no Save button appeared within 10 s. Not retried. | Report screen | System | Ops, then Dev | Check in ClaimDesk that the payment was **not** posted. If it needs posting, post it by hand. The red row blocks the batch. | Always, so dev can correct the row. Send: batch number, clinic, check number, time, log file name, "E11". |
| E12 | Report: red **Error** - "Payment not confirmed after Save" | The bot clicked Save and waited 20 s for "Payment saved", which never appeared. Not retried. **The payment may have posted anyway.** | Report screen | System | **Ops**, then Dev | **Check the payment in ClaimDesk before doing anything.** If it posted, do not post it again: a second post creates a duplicate payment. If it did not post, post it by hand. Either way the red row blocks the batch. | Always, so dev can correct the row. Send: batch number, clinic, check number, "posted: yes/no", "E12". |
| E13 | Report: green **Completed** - "Remittance Errors on posting" | ClaimDesk showed its "Remittance Errors" screen within 5 s of Post Payment. The bot pressed Cancel. | Report screen | Business | Ops | **Not posted, despite the green colour.** Review the remittance's errors in ClaimDesk and post it by hand or follow the client's process. | ClaimDesk shows no error for it. Send: batch number, clinic, check number, "E13". |
| E14 | Report: green **Completed** - "Payment method '<method>' - not posted" | The method in ClaimDesk is not ACH. | Report screen | Business | Ops | Nothing, unless the client wants non-ACH payments posted. Then post them by hand. | Never. This is by design. |
| E15 | Report: grey **Deferred** - "Check date <date> is after the cutoff - deferred" | The check date is later than the cutoff in Settings B1. The bot compares the dates **as text**. | Report screen | Business | Ops | Nothing; the next run picks it up. Spot-check a few rows: if a date clearly on or before the cutoff was deferred, or a later one posted, report it. | A deferred or posted row has the wrong date. Send: batch number, clinic, check number, the cutoff value, "E15". |
| E16 | The run stops part-way. The log's last line is `[WATCHDOG] no progress for 300s - killing run`, with no `[RUN SUMMARY]` line. ClaimDesk may still be open. | No new Report row for 5 minutes. A clinic retrying, or several clinics in a row with no remittances, can trigger it even when the bot is still working. | Run log | System | Dev | Close ClaimDesk. The batch stays Pending and **blocks the next run**. Clinics after the stop were never processed. | Always. Send: log file name, batch number, time, last clinic in the log, "E16". |
| E17 | Log: `[AUTO-RERUN] attempt 2 for '<name>'` (or attempt 3). ClaimDesk closes and reopens. | A clinic-level failure (E03-E09) triggers a fresh ClaimDesk start and sign-in. | Run log | System | Bot | Wait. At most 3 attempts per clinic. | The same clinic shows more than 3 attempts in one run. Send: log file name, "E17". |
| E18 | The same check number appears twice for one clinic on the Report screen. | When a clinic fails part-way and is retried, remittances still in ClaimDesk's list are written again. | Report screen | System | Ops, then Dev | Count each check once. Use the latest row. A red row from the first attempt still blocks the batch even if the retry posted it. | There is a red row from an earlier attempt. Send: batch number, clinic, check number, "E18". |
| E19 | The bot clicks in a window other than ClaimDesk, or ClaimDesk you had open closes suddenly. | The bot takes the first window whose title contains "ClaimDesk". It also force-closes ClaimDesk after every clinic. | Not recorded. Usually shows up as E06 or E09 in the log. | System | Ops | Follow section 4: close other "ClaimDesk" windows and keep hands off. Let the run finish, then check the Report screen. | It happens with only the launcher and ClaimDesk open. Send: log file name, a screenshot of what was open, "E19". |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Any clinic-level failure (E03-E09) | Force-closes ClaimDesk, restarts it, signs in again and redoes the clinic from the start | 3 attempts per clinic, each up to about 2.5 min of waiting | Clinic left Pending (E10), next clinic. The batch stays Pending. |
| No progress | Watchdog stops the whole run | 5 min without a new Report row | Run ends, batch Pending, ClaimDesk may stay open (E16) |
| Save button missing / save not confirmed | **No retry** | - | Row red Error, next remittance (E11, E12) |
| ClaimDesk "Remittance Errors" screen | Presses Cancel | waits 5 s for the screen | Row green "Remittance Errors on posting" (E13) |
| Run died or was stopped | **No resume.** A new run starts a new batch, but only after dev releases the old Pending batch. | - | E01 |
## 9. Known incidents
Newest first. The git history has no real dates, so dates are unknown unless a log shows one.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| After 1.3.0 | The launcher still showed and ran 1.3.0 after an update | The launcher kept the old program folder until it was restarted | Fully close and reopen the launcher after every update, then check the title bar (section 10). The fix is not in this repo, so it is unverified. | Yes |
| Before 1.4.2 | Red rows "Payment not confirmed after Save" that had really posted | Save took longer than the 10 s wait | Wait raised to 20 s. A slower save still produces this. **Always check ClaimDesk first** (E12). | Partly |
| Before 1.4.2 | Clinics never processed; "not found in list" for pieces of a name | Names with commas were split into pieces | One clinic per row, no commas. A name that really contains a comma still can't be posted (Fix 7). | Partly |
| Before 1.4.2 | The bot drove a SQL browser window instead of ClaimDesk, then closed the real ClaimDesk | That window's title contained "ClaimDesk" | Close every other "ClaimDesk" window before a run (section 4). The code is **not** fixed. | Partly |
| 2026-09-01 run log (older build) | A clinic whose name contains `&` failed "Wrong clinic opened", was re-run, and the run was then stopped by the watchdog | Exact title match on a special-character name; the watchdog fired during retries | E07, E16 | Partly |
| 2026-09-01 run log (older build) | "could not bring window to foreground" warning | Someone used the machine, or another window held focus | Hands off the machine (section 4) | Yes |
## 10. Releases and version check
- **How an update arrives:** through the launcher. The update code is not in this repo, so dev must confirm the exact steps.
- **How to confirm it applied:** after an update, **close the launcher completely and reopen it**. The title bar must show the new version number. This is version 1.4.2.
- **If the version did not change:** close the launcher again and make sure no launcher window or process is left running, then reopen it. If it still shows the old number, tell #rpa-support with the version you see and the version expected.
- Each machine keeps its own batch history. A Pending batch on one machine blocks only that machine.
## 11. Escalating to dev
Post in the Teams channel **#rpa-support** (RPA dev team).
Send:
- the exception ID from section 7
- the batch number (from the `[RUN SUMMARY]` line or the Report screen filter)
- the time of the failure
- the run log file name `logs/run_<date>_<time>.log`
- the clinic and check number if it is about one row
- a screenshot of the Report screen or of ClaimDesk if it helps
Never send: the Login sheet or its password, the `.env` settings file, or patient data. Logs can contain clinic names, so attach the file rather than pasting lines into chat.
## 12. Failure-class check
- **F1** Missing prerequisite: covered in section 3 (settings file, input sheets) and in E02 and E03. Fix 13 and 15 cover the unlogged startup errors and missing database tables.
- **F2** Wrong window or focus: covered in section 4, E19 and section 9. Fix 5 and 6.
- **F3** Slow or different target app: covered in E04-E09, E11 and E12, with timeouts in section 8. Fix 4 and 18.
- **F4** Input read differently: covered in section 3 (commas, spelling, cutoff date) and in E06, E07 and E15. Fix 7, 8 and 14.
- **F5** Marked done when nothing was done: the clinic name is checked against the title bar (E07). But a clinic is marked Success with 0 rows if the grid is empty or not yet filled. Fix 16.
- **F6** Action happened but not confirmed: covered in E12 ("check in ClaimDesk first"). Fix 4.
- **F7** Business exception shown as success: covered in section 6 and in E13 and E14. Fix 2.
- **F8** Endless re-runs or watchdog kills: covered in section 8 and in E16 and E17. Fix 9.
- **F9** Stopping the wrong way: covered in section 4. Fix 17.
- **F10** One stuck item blocks the queue: covered in E01, E10, E11, E12 and E18. Fix 1.
- **F11** Cause hidden from ops: covered in section 5 (clinic failures are in the log only). Fix 10, 11, 12 and 13.
- **F12** Update didn't arrive: covered in section 10. Fix 19. Not applicable to shared resources: nothing is written to a shared database or server on start; each machine uses its own local database.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Any Pending batch blocks every new run. There is no Clear, Resume or row-edit option, and ops cannot open the DB. | One red row or one watchdog kill stops all posting until a dev edits the DB | bot.py:167-168, bot.py:150-154 |
| 2 | "Rejected by app" (not posted) is stored as Completed, green | Ops reads unposted payments as done | bot.py:29 |
| 3 | "Skipped" non-ACH is also green Completed | Not posted, but looks done; at least needs a distinct colour | bot.py:30 |
| 4 | "Payment not confirmed" does no second check (e.g. re-read the grid). 20 s is still a fixed wait. | False red rows, a duplicate-posting risk, and the batch is blocked | bot.py:117-118, bot.py:21 |
| 5 | Window match is by substring "ClaimDesk". It also matches the launcher ("ClaimDesk Poster") and "ClaimDesk Login". The SQL-browser "fix" is only a comment. | The bot can drive the wrong window and kill the real ClaimDesk | bot.py:66-70, bot.py:185 |
| 6 | No wait after clicking Sign in, so the bot may grab the login window. A failed sign-in surfaces as "Clinic not found in list". | Misleading cause, 3 wasted attempts per clinic | bot.py:82-83 |
| 7 | Clinic cells are split on `,` and `;`. A real name with a comma can never be posted. The fix was README text only. | Clinics silently skipped | bot.py:62, README.md:8 |
| 8 | The clinic check needs the exact name in the title bar. `&` and other special characters have failed it. | Clinics left Pending | bot.py:91 |
| 9 | The watchdog heartbeat only runs per remittance row. Login, grid waits, retries and 0-row clinics never beat. A kill leaves ClaimDesk open and the batch Pending. | Healthy runs killed; next run blocked | bot.py:39-49, bot.py:137 |
| 10 | Clinic-level failures write no Report row | Failed clinics are invisible on the Report screen | bot.py:143-147 |
| 11 | The technical-detail column is written but not shown on the Report screen | Ops lacks the dev-level cause | report_screen.py:6 |
| 12 | The batch status is not shown anywhere ops can see. `[RUN SUMMARY] ... done` prints even when the batch stays Pending. | Ops can't tell if a run really completed | bot.py:177-180 |
| 13 | Startup failures (Pending batch, input sheets, missing DB tables) are raised before or outside logging, so the log file is empty | Ops sees "nothing happened" | bot.py:161-169, bot.py:52-63 |
| 14 | A clinic retry re-logs rows already written, so duplicates appear. A stale Error row from attempt 1 keeps blocking. | Confusing Report screen; batch blocked | bot.py:136-139 |
| 15 | DB tables are never created by the code. A fresh machine fails at the Pending check. | A new install can't run, and nothing is logged | bot.py:166-167 |
| 16 | A clinic is marked Success with 0 rows without confirming the grid finished loading | A clinic could be "done" with nothing posted | bot.py:96-101, bot.py:140 |
| 17 | No Stop mechanism. The launcher-to-bot process link and its behaviour on close are unknown (launcher code not in repo). | Ops has to kill processes in Task Manager | bot.py:157-180 |
| 18 | The "Remittance Errors" popup is only waited for 5 s. A slower popup ends as "Save button not found" with the popup left open. | Wrong cause; the next row may misfire | bot.py:110 |
| 19 | Launcher/updater code is not in this repo. The 1.3.0 "fix" is only a comment and can't be verified. | Update failures can't be diagnosed | report_screen.py:8, report_screen.py:14 |
| 20 | `.env` is committed to git and holds an unused password-fallback key and an unused alert setting. | Credential exposure; no alerts actually sent | .env |
| 21 | The committed run log (older build) contains a patient name | PHI in the repo; purge it from history | logs/run_20260901_101500.log:3 |
| 22 | The cutoff is compared as text, not as a date | Wrong Deferred/Posted decisions | bot.py:107 |
| 23 | The Login password is stored in plain text in the input Excel | The credential sits in an ops-handled file | bot.py:56 |
| 24 | The log prefix is doubled: `[ERROR] [ERROR]` | Noise when ops searches the log | bot.py:144 |
tests/evals/results/handoff/h-desktop-poster-full-r3.json
+34
View File
@@ -0,0 +1,34 @@
{
"case": "h-desktop-poster/full",
"rep": 3,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"comma rule before a run": true,
"how to stop": true,
"not-confirmed row: check ClaimDesk first": true,
"hidden tech comments flagged": true,
"Pending batch blocks a new upload": true
},
"cost_usd": 2.0747394,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-desktop-poster-full-nuvq4d1l\\project",
"reply": "`HANDOVER.md` ready in the repo root. Bot-a run pannala, code-um git history-um mattum padichen. Code edhuvum maathala.\n\n| Item | Count |\n|---|---|\n| Sections | 13 (template order) |\n| Exceptions catalogued | 20 (E01\u2013E20) |\n| Failure classes | 12/12 covered, 0 \"not applicable\" |\n| Fix before handover | 17 items |\n\n**Mukkiyamaana findings:**\n- **\"Fix\"-nu label panna 4 commits-um comment mattum add pannudhu.** Window grab, save wait, comma split, 1.3.0 update: naalum innum open. Save wait import-la irundhe 20s dhaan.\n- **Green \u2260 posted.** \"Rejected by app\" and \"Skipped\" rows-um green Completed-aa dhaan kaatum.\n- **Oru red row, illa watchdog kill aana oru run, next upload-a block pannidum.** Ops-ku DB access illa, so ovvoru dhadavaiyum dev dhaan batch-a clear pannanum.\n- **Watchdog 5 min.** Login, retries edhuvum progress-aa count aagadhu, so normal retries-um run-a kill pannidum.\n- **Repo-la PHI irukku.** `.env` file-um, oru patient name irukura oru run log-um git-la tracked. Adhu Fix item 16-la irukku; doc-la copy pannala.\n- **Launcher code repo-la illa.** Adhanala \"launcher-a close panna bot stop aagumaa\" and update steps unknown-nu mark panni irukken.\n\n**Fix before handover (17):**\n1. Status map rejected/skipped rows-a green-aa kaatudhu\n2. Save-confirm ClaimDesk-la re-check pannala\n3. Title substring-la window match aagudhu\n4. taskkill ella ClaimDesk-ayum kill pannudhu\n5. Comma/semicolon split\n6. Cutoff date text-aa compare aagudhu\n7. Watchdog heartbeat payment-ku mattum\n8. Zero rows irundhaalum clinic Success; sign-in check illa\n9. DB/schema-va code create pannala\n10. Start-up errors log-la varala; unused .env keys\n11. Stop button illa\n12. Open batch upload-a block pannudhu, resume illa\n13. Hidden columns, clinic/batch status kaatala\n14. Row-la timestamp illa, \"[ERROR] [ERROR]\", RUN SUMMARY misleading, retry-la duplicate rows\n15. Launcher/update code repo-la illa\n16. .env and PHI log git-la; password plain-text Excel-la\n17. Rendu runs ore nerathula start aaga lock illa\n\nUngal answers apply panniten: escalation #rpa-support, \"Where to look\" Report screen + logs mattum, DB items \"Fix before handover\"-la. Andha moonu questions-ku neenga munnadiye answer koduthadhaala, ask step-a skip panniten.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions whose meaning the code doesn't make clear actually mean?",
"stderr": ""
}
tests/evals/results/handoff/h-desktop-poster-full-r3.md
+214
View File
@@ -0,0 +1,214 @@
# ClaimDesk Poster - ops handover
Version 1.4.2 · Written 2026-10-23 · Escalation: RPA dev team, #rpa-support Teams channel
## 1. What the bot does
The bot posts remittance payments into the **ClaimDesk** desktop app, one clinic at a time. Ops starts it from the "ClaimDesk Poster" launcher window with an input Excel file that has three sheets: **Login** (the ClaimDesk user name and password in row 2), **Clinics** (one clinic per row in column A, starting at row 2) and **Settings** (the cutoff date in cell B1). For each clinic, the bot opens ClaimDesk, signs in, opens the clinic and goes through the remittance list:
- It posts and saves every ACH payment dated on or before the cutoff.
- It skips payments that are not ACH.
- It defers payments dated after the cutoff.
It writes one result row per payment to the **Report screen** in the launcher, and a run log to the `logs` folder. It writes no other output files. It drives one app: **ClaimDesk**.
## 2. How it works
```
"ClaimDesk Poster" launcher (the window you use; title bar shows the version)
└─ starts → posting bot (runs in the background) ──drives──> ClaimDesk desktop app
│ (do not touch while it runs)
├─ writes → bot database ──read by──> Report screen in the launcher
└─ writes → logs/run_<date>_<time>.log
```
- **Launcher:** where you pick the input file, start the run, see the version and open the Report screen.
- **Posting bot:** a separate background program that does the work. It has no window of its own.
- **ClaimDesk:** the bot starts it, signs in and clicks through it. Before each new clinic, and after each failed try, it force-closes **every** ClaimDesk window on the machine.
- **Bot database:** holds the batch, the clinic list and one result row per payment. Ops cannot open it directly. The Report screen shows part of it.
- **Run log:** one file per run, in local time. It is the only place that shows clinic-level failures and retries.
**One run, step by step:**
1. The bot checks that no earlier batch is still open. If one is, it stops at once (E01).
2. It reads the input Excel and creates a new batch with every clinic set to Pending.
3. For each clinic, in input order:
1. Start ClaimDesk and sign in.
2. Click the clinic's link and check that the window title now shows the clinic name.
3. Wait for the remittance list.
4. For each payment in the list: skip it, defer it, or click Post Payment, then Save, then wait for "Payment saved". One Report row is written per payment.
5. Close ClaimDesk.
4. If a clinic fails, the bot tries it again, up to 3 tries in total. After 3 failed tries the clinic stays Pending and the bot moves on to the next clinic.
5. At the end, the batch is marked Completed only if no Report row is red and no clinic is still Pending. Otherwise the batch stays open, and **the next upload is refused** (E01).
6. The log ends with "[RUN SUMMARY] batch N done". This line appears even when the batch did **not** complete.
## 3. Before a run
- [ ] On this machine, the RPA dev team has set the ClaimDesk program path for the bot. If it is missing or wrong, every clinic fails (E07).
- [ ] The bot's `data` folder exists next to the bot. If it doesn't, the bot cannot start; escalate (E01).
- [ ] No earlier batch is still open. If you are unsure, check the last run log for "left Pending", "[WATCHDOG]" or red rows on the Report screen (see E01).
- [ ] **Input Excel, Login sheet:** the user name goes in A2 and the password in B2. Both must be filled in.
- [ ] **Input Excel, Clinics sheet: one clinic per row.** Never put commas or semicolons in a cell. The bot cuts a cell at every comma and every semicolon.
- Bad: `Clinic A, Clinic B` in one cell.
- Good: `Clinic A` in A2 and `Clinic B` in A3.
- A clinic whose **real name** contains a comma or a semicolon cannot be run at all. Leave it out and tell dev (E03).
- [ ] Clinic names are spelled **exactly** as the clinic link in ClaimDesk, including capitals and `&`. Names containing `&` have failed before (E06).
- [ ] **Settings sheet:** put the cutoff date in B1. The bot compares the date as text, not as a calendar date. After the first run, check that the Deferred rows really are later than the cutoff (E13).
- [ ] The Login, Clinics and Settings sheets all exist and are spelled exactly like that.
- [ ] The launcher title bar shows version 1.4.2 (see section 10).
## 4. During a run
- **Don't use the mouse or keyboard on this machine.** The bot clicks on the screen, and Windows may stop it from bringing ClaimDesk to the front. The log then shows "could not bring window to foreground".
- **Don't open ClaimDesk yourself on this machine**, and keep any other window whose title contains "ClaimDesk" closed. This includes SQL or database tools, browser tabs and Explorer folders named ClaimDesk. The bot uses the first window it finds with "ClaimDesk" in its title, and between clinics it force-closes every ClaimDesk window (E17).
- **To stop the bot:** there is no Stop button and no safe way to stop it mid-run.
- Closing ClaimDesk does **not** stop it. The bot counts that as a failed try and starts ClaimDesk again.
- The launcher code is not part of this handover, so it is unknown whether closing the launcher stops the background bot. Ask dev before relying on it.
- Any stop mid-run leaves the batch open, and the next upload is refused (E01). Let the run finish if you can.
- **A run looks stuck** when ClaimDesk sits on the same screen and no new line has been added to the run log for several minutes.
- The bot gives up by itself when **no payment has been processed for 5 minutes**. It then ends the whole run and writes "[WATCHDOG]" in the log (E16).
- Wait those 5 minutes before you act.
- Signing in and waiting for screens do **not** count as progress. A clinic that keeps failing can therefore trigger the watchdog before its 3 tries are used up.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | The newest file in `logs` (`run_<date>_<time>.log`): the last line should be "[RUN SUMMARY] batch N done". If the last line is "[WATCHDOG]", or the file is empty, the run did not finish. |
| Did the batch complete, or is it blocking the next upload? | Ops cannot see this directly. A batch is still open if the Report screen shows any **red** row, or the run log has a "left Pending" or "[WATCHDOG]" line. |
| Which payments failed? | Report screen for that batch: the red rows (File Status = Error). |
| Why did one payment fail? | Report screen, Comments column. Then search the run log for the clinic name. |
| A clinic has no rows on the Report screen | Search the run log for the clinic name. "left Pending after 3 attempts" means the clinic failed: see the message on the "[ERROR] Clinic" lines just above it. If there is no error line, the clinic had nothing to post. |
| Did the bot already retry? | Run log: lines "[AUTO-RERUN] attempt 2" or "attempt 3" for that clinic. |
| Which version is running? | The launcher title bar. |
| Output files | None. The bot writes no output files. Its results are only on the Report screen and in the run log. |
Note: Report rows have no time stamp. To match a row to a log line, use the clinic name and the order of events.
## 6. Statuses
The Report screen shows a File Status (with a colour) and a Comment for each payment. **Green does not always mean posted.** Always read the Comment.
| Status (File Status - Comment) | Colour | Meaning | What ops does |
|---|---|---|---|
| Completed - "Posted successfully" | green | Payment posted and saved in ClaimDesk | Nothing |
| Completed - "Remittance Errors on posting" | green | **Not posted.** ClaimDesk showed its "Remittance Errors" screen, and the bot pressed Cancel. Needs follow-up. | Review and post it in ClaimDesk by hand (E11) |
| Completed - "Payment method '<method>' - not posted" | green | **Not posted on purpose.** The payment is not ACH. | Nothing, unless the client wants non-ACH payments posted (E12) |
| Deferred - "Check date <date> is after the cutoff - deferred" | grey | Not posted. Its date is later than the Settings cutoff. | Nothing. A later run with a later cutoff picks it up. Check the date really is after the cutoff (E13). |
| Error - "Save button not found" | red | Post Payment was clicked, but the payment screen had no Save button. Not saved. **Blocks the batch.** | E14 |
| Error - "Payment not confirmed after Save" | red | Save was clicked, but "Payment saved" did not appear within 20 seconds. **It may have posted.** **Blocks the batch.** | E15: check ClaimDesk first, never re-post blindly |
| Pending | amber | The bot never writes this today. It would mean a result the bot did not recognise. | Escalate (Dev) |
| Clinic Pending / Success | not shown | Kept in the bot database only. Success means the clinic's list was walked, **even if some rows are red or there were no payments**. Pending means all 3 tries failed. | See the run log for "left Pending" |
| Item status Posted / Rejected by app / Skipped / Deferred / Error | not shown | A second, more precise status the bot stores for each row, in a column the Report screen hides. Posted, Rejected by app and Skipped all show as green Completed. | Tell them apart by the Comment |
| Batch Pending / Completed | not shown | Kept in the bot database only. Pending blocks every new upload. | E01 |
## 7. Exception catalogue
Clinic-level errors (E03-E10) appear in the run log as `[ERROR] [ERROR] Clinic '<name>' : <message>`. The word ERROR appears twice; that is normal.
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Upload refused: "A Pending batch exists - finish or clear it before a new upload". The new run log is empty. | At start, the bot checks for an earlier batch that is still open. An earlier batch stays open if it has a red row, a clinic left Pending, or a run that was killed or stopped. | Launcher message only (if the launcher shows it). The new run log is empty. | System | Dev | Find the last run log and note why that batch did not complete (red rows, "left Pending", "[WATCHDOG]"). Ops cannot clear a batch. Ask dev to clear it. | Always. Send the batch number from the last "[RUN SUMMARY]" line (if there is one), the last run log file, and the time of the refused upload. |
| E02 | The run stops at once with "'Login' sheet not found", "UserName/Password missing in Login sheet (row 2)", or an error naming "Clinics" or "Settings". The run log is empty. | The bot reads the input Excel before it creates the batch. A missing sheet, or an empty A2 or B2 on the Login sheet, stops it. | Launcher message only (if shown). The run log is empty. No batch is created. | Business | Ops | Fix the input Excel (section 3) and start again. Nothing is blocked. | The Excel is correct and it still fails. Send the time, the error text and the empty log file name, **not** the Excel (it holds the password). |
| E03 | Log shows "Clinic '<part of a name>' not found in list" for names that look cut in half, or for an empty name '' | The bot cut a Clinics cell at a comma or semicolon (or a trailing comma). Each piece is looked for in ClaimDesk and is not found. | Run log, 3 tries per piece, then "left Pending after 3 attempts". No Report rows. Batch stays open. | Business | Ops, then Dev | Put one clinic per row with no commas or semicolons. The batch is now blocked: ask dev to clear it (E01). | Always, to clear the batch. If the clinic's real name contains a comma, it cannot be run until dev fixes the bot. Send the batch number, run log and clinic row number. |
| E04 | "Clinic '<name>' not found in list" for one clinic | The bot waits 10 seconds for a link with exactly that name in ClaimDesk, and none appears. | Run log, 3 tries, then "left Pending after 3 attempts". Batch stays open. | Business | Ops, then Dev | Check the spelling against ClaimDesk exactly (capitals, `&`, spaces) and whether this user can see the clinic. Fix the input. Ask dev to clear the batch (E01). | Always, to clear the batch. Also escalate when the spelling is already exact. Send the batch number, run log and time. |
| E05 | "not found in list" for **every** clinic in the run | Most likely the sign-in failed (wrong user name or password). The bot never checks that sign-in worked, so it looks for clinics on the login screen. | Run log; every clinic "left Pending". Batch stays open. | Business | Ops, then Dev | Check the user name and password in Login A2/B2 by signing in to ClaimDesk by hand when no run is going. Ask dev to clear the batch. | The login is correct. Send the batch number, run log and time. Never send the password. |
| E06 | "Wrong clinic opened, leaving Pending: <name>" | After clicking the clinic link, the ClaimDesk title bar does not contain the clinic name exactly. This has happened with names containing `&`, and when the bot could not bring the window to the front. | Run log, 3 tries, then "left Pending". Batch stays open. | System | Bot, then Dev | Wait for the 3 tries. Check that nothing else was used on the machine during the run (section 4). | It fails after 3 tries. Send the batch number, run log, clinic row number and a screenshot of ClaimDesk if it is still open. |
| E07 | "Login window not found", or every clinic fails with an error naming the ClaimDesk program path | ClaimDesk was started, but its "ClaimDesk Login" window did not appear within 60 seconds, or the program path is not set. | Run log, 3 tries per clinic. Batch stays open. | System | Bot, then Dev | Wait for the 3 tries. Check that ClaimDesk opens by hand on this machine when no run is going. | It fails after 3 tries. Send the batch number, run log and time. |
| E08 | "ClaimDesk window not found" | Right after Sign in, no window title contains "ClaimDesk". | Run log, 3 tries. Batch stays open. | System | Bot, then Dev | Wait for the 3 tries. | It fails after 3 tries. Send the batch number, run log and time. |
| E09 | "Remittance grid did not load" | The remittance list did not appear within 60 seconds after the clinic opened. | Run log, 3 tries. Batch stays open. | System | Bot, then Dev | Wait for the 3 tries. ClaimDesk may just be slow; try again at a quieter time. | It fails after 3 tries, or on two runs in a row. Send the batch number, run log and time. |
| E10 | Any other "[ERROR] Clinic '<name>' : <message>" (for example "list index out of range", or a message about an element or a click) | An unexpected screen or click failure while the clinic was open, for example clicking Post Payment. The whole clinic is retried from the top. | Run log, 3 tries. Batch stays open. Rows already written stay on the Report screen. | System | Bot, then Dev | Wait for the 3 tries. Check the Report screen for duplicate rows (E20). | It fails after 3 tries. Send the batch number, run log, time and the exact message. |
| E11 | Green row, "Remittance Errors on posting" | After Post Payment, ClaimDesk showed its "Remittance Errors" screen within 5 seconds. The bot pressed Cancel. | Report screen (green, **not posted**) | Business | Ops | Open the payment in ClaimDesk, review the errors, and post it by hand or follow the client's process. | ClaimDesk shows no error for that payment. Send the batch number and check number. |
| E12 | Green row, "Payment method '<method>' - not posted" | The payment method in the list is not exactly "ACH". | Report screen (green) | Business | Ops | Nothing, unless the client wants that method posted. Then post it by hand. | An ACH payment was skipped. Send the batch number and check number. |
| E13 | Grey row, "Check date <date> is after the cutoff - deferred". Or payments posted or deferred on the wrong side of the cutoff. | The bot compares the payment's date with the Settings cutoff **as text, not as a calendar date**. Different date formats can give a wrong answer. | Report screen (grey) | Business | Ops | Check a few Deferred and Posted dates against the cutoff. If they look right, nothing to do: deferred payments are picked up by a later run. | Dates are on the wrong side of the cutoff. Send the batch number, two example check numbers and the cutoff you entered. |
| E14 | Red row, "Save button not found" | After Post Payment, no Save button appeared within 10 seconds. The bot moves on without saving. | Report screen (red). Batch stays open. | System | Ops, then Dev | Check the payment in ClaimDesk. It should be unposted. Post it by hand if needed. Ask dev to correct the row and clear the batch. | Always (to clear the batch). Send the batch number, check number and run log. |
| E15 | Red row, "Payment not confirmed after Save" | After Save, the bot waits 20 seconds for "Payment saved". If it does not appear, it writes Error. **It does not check whether the payment really posted.** | Report screen (red). Batch stays open. | System | **Ops**, then Dev | **Check the payment in ClaimDesk before doing anything.** In the past these had often really posted. If it posted, do **not** post it again. If it did not post, post it by hand. Ops cannot edit the row, so ask dev to correct it and clear the batch. | Always (to correct the row). Send the batch number, check number, whether it posted in ClaimDesk, and the run log. |
| E16 | The run ends early. The last log line is "[WATCHDOG] no progress for 300s - killing run", with no "[RUN SUMMARY]" line. ClaimDesk may be left open. | No payment was processed for 5 minutes. Signing in, opening clinics and waiting for lists do not count as progress. | Run log. Batch stays open. Remaining clinics are not run. | System | Dev | Close ClaimDesk. Do not start a new run: it will be refused (E01). | Always. Send the batch number, the run log, and which clinic was being retried (from the last "[ERROR]" or "[AUTO-RERUN]" lines). |
| E17 | ClaimDesk closes unexpectedly, or the bot clicks in a database tool or another window with "ClaimDesk" in its title | The bot takes the first window with "ClaimDesk" anywhere in its title. After every clinic it force-closes every ClaimDesk program on the machine. | Nothing specific. Often followed by E04, E06 or E08 in the run log. | System | Ops | Close every other window with "ClaimDesk" in its title before the run, and don't use ClaimDesk on this machine during a run (section 4). | It happens with no such window open. Send the batch number, run log and the names of the windows that were open. |
| E18 | Log shows "could not bring window to foreground" | Windows refused to bring ClaimDesk to the front, usually because someone used the machine. | Run log (warning). Usually followed by E06. | System | Ops | Keep hands off the machine during a run. | It happens with nobody using the machine. Send the run log. |
| E19 | A clinic has no rows on the Report screen | Either the clinic had nothing to post (the bot still marks it done), or all 3 tries failed (see E03-E10). | Run log: "left Pending" if it failed. No line if it was empty. | Business | Ops | Search the run log for the clinic name. If there is no error, confirm in ClaimDesk that the clinic really has nothing to post. | ClaimDesk shows payments, but the bot wrote no rows and no error. Send the batch number, run log and clinic name. |
| E20 | The same check number appears more than once for a clinic on the Report screen | When a clinic is retried, the bot walks its list again from the top and writes new rows. It never checks for a row that already exists. | Report screen | System | Ops, then Dev | Check the payment in ClaimDesk before acting, to see whether it posted once, twice or not at all. An old red row keeps the batch open even if a later row is green. | A payment posted twice, or a red duplicate blocks the batch. Send the batch number, check number and run log. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Any clinic-level failure (E04-E10) | Force-closes ClaimDesk, then starts it again, signs in again and walks the clinic's list from the top | 3 tries per clinic. Each try can wait up to about 2 minutes (60 s for login + 10 s for the clinic link + 60 s for the list). | Clinic left Pending ("left Pending after 3 attempts"), and the bot moves to the next clinic. The batch will not complete. |
| "Remittance Errors" screen after Post Payment | Presses Cancel | Waits 5 s for the screen | Row green "Remittance Errors on posting" (not posted), next payment |
| Save button missing / no "Payment saved" | **No retry** | 10 s / 20 s | Row red, next payment |
| No progress at all | Watchdog ends the whole run | 5 minutes without a processed payment | The run stops, the batch stays open and ClaimDesk may stay open (E16) |
| Earlier batch still open | **No recovery or resume** | - | Every new upload is refused until dev clears it (E01) |
How long to wait: a clinic that keeps failing shows up to 3 "[AUTO-RERUN]" lines over about 5-7 minutes. If the log has had no new line for 5 minutes, the watchdog will end the run. Act after that, not before.
## 9. Known incidents
The repository shows a "fix" for incidents 1-4, but **none of them changed how the bot behaves**. Treat all of them as still open. Dates are unknown unless a log shows one; all were recorded on 2026-10-23.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Date unknown, launcher 1.3.0 → 1.4.x | Users stayed on version 1.3.0 after an update | The launcher kept using the old program folder until it was restarted | After every update, fully close and reopen the launcher, then check the title bar (section 10) | Yes |
| Seen 2026-09-01 and before | Red rows "Payment not confirmed after Save" for payments that had really posted | ClaimDesk sometimes takes longer than the bot's 20 s wait to save | E15: check ClaimDesk first, never re-post blindly. Not fixed. | Yes |
| Date unknown | Some clinics never processed, or "not found in list" for pieces of names | The bot cuts clinic cells at commas and semicolons | One clinic per row (section 3, E03). Names that really contain a comma still can't be run. | Partly |
| Date unknown | The bot drove a SQL browser window and then closed the real ClaimDesk | The window title of the SQL browser contained "ClaimDesk" | Close other "ClaimDesk" windows and don't use ClaimDesk during a run (section 4, E17). Not fixed. | Partly |
| 2026-09-01 | "could not bring window to foreground", then "Wrong clinic opened" for a clinic whose name contains `&`, then "[AUTO-RERUN]", then "[WATCHDOG] … killing run" | Window focus lost, `&` in the name (likely), and the watchdog does not count retries as progress. The batch was left open. | Hands off the machine (section 4). Check `&` names (E06). Ask dev to clear the batch (E01, E16). | Partly |
## 10. Releases and version check
- **How an update arrives:** through the launcher. The launcher's update code is not part of this repository, so dev must confirm the exact steps.
- **How to confirm it applied:** fully close and reopen the launcher, then check that its title bar shows the new version (today: 1.4.2).
- **If the version did not change:** close the launcher completely, reopen it and check again. If it still shows the old version, escalate to dev with the machine name and the version shown.
- Each machine keeps its own bot database and logs, so one machine on an old version does not overwrite another. Still, keep every ops machine on the same version.
## 11. Escalating to dev
Post in the **#rpa-support** Teams channel (RPA dev team).
Send:
- the exception ID from section 7 (for example "E15")
- the batch number, from the "[RUN SUMMARY] batch N" line in the log (a run killed by the watchdog has no such line; send the log file name instead)
- the time of the failure
- the run log file `logs/run_<date>_<time>.log`
- the clinic name and check number
- a screenshot of ClaimDesk, when it is still on the failing screen
Never send:
- passwords
- the input Excel (its Login sheet holds the ClaimDesk password)
- the bot's settings file
- patient or customer data, typed into chat
## 12. Failure-class check
- **F1 Missing prerequisite:** section 3 (ClaimDesk path, `data` folder), E07, E01. The settings and database setup are also in Fix before handover, items 9 and 10.
- **F2 Wrong window / focus:** section 4, E17, E18. Fix before handover, items 3 and 4.
- **F3 App slower or different:** E07, E08, E09, E10, E14, section 8. Fix before handover, item 7.
- **F4 Input read differently:** section 3 (one clinic per row, exact spelling, `&`, cutoff), E03, E04, E06, E13. Fix before handover, items 5 and 6.
- **F5 Marked done when nothing was done:** E19. Fix before handover, item 8.
- **F6 Action happened but not confirmed:** E15, E20. Fix before handover, item 2.
- **F7 Business exception shown as success:** section 6 (green "Remittance Errors" and "not posted" rows), E11, E12. Fix before handover, item 1.
- **F8 Endless re-runs / watchdog kill:** section 8, E16. Fix before handover, item 7.
- **F9 Stopping the bot the wrong way:** section 4 (no safe stop). Fix before handover, item 11.
- **F10 One stuck item blocks the queue:** E01, section 5. Fix before handover, item 12.
- **F11 Cause hidden from ops:** section 5 points only at the Report screen and the logs. Fix before handover, items 1, 13 and 14.
- **F12 Update didn't arrive:** section 10, section 9. Fix before handover, item 15.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | "Rejected by app" and "Skipped" are stored with File Status Completed (green) | Ops reads green as posted, and rejected payments are missed | `bot.py:27-33` (STATUS_MAP) |
| 2 | "Payment not confirmed after Save" never re-checks ClaimDesk. The commit "raise save wait 10s to 20s" changed nothing (already 20 s at import). | Real posts show red, re-posting causes duplicates, and the batch is blocked | `bot.py:21`, `bot.py:117-118`, `bot.py:186` |
| 3 | The window search matches any title containing "ClaimDesk" (SQL tools, the launcher itself, the login window). The commit labelled "fix" only added a comment. It is also called straight after Sign in, with no wait. | The bot drives the wrong window | `bot.py:66-70`, `bot.py:83`, `bot.py:185` |
| 4 | `taskkill /im ClaimDesk.exe /f` kills every ClaimDesk on the machine | A user's own ClaimDesk session is closed | `bot.py:146` |
| 5 | Clinic cells are split on `,` and `;` | Real names containing a comma cannot be run, and blank pieces are created from trailing commas | `bot.py:62` |
| 6 | The cutoff is compared as text (`check_date > str(cutoff)`) | Payments are posted or deferred wrongly when the formats differ | `bot.py:107` |
| 7 | The watchdog heartbeat runs only per payment. Login, opening the clinic and list waits do not count, and `_last_beat` is set when the program loads. A kill skips cleanup. | Normal retries trigger the 300 s kill. ClaimDesk is left open and the batch stays open. | `bot.py:36`, `bot.py:39-49`, `bot.py:137` |
| 8 | A clinic is set to Success with no count check, even with 0 rows or red rows. Sign-in success is never checked. | Empty or failed clinics look done, and a wrong password looks like "clinic not found" | `bot.py:83`, `bot.py:140` |
| 9 | `data/poster.db` and its tables are never created by the code, and the `data` folder is not in the repo | A fresh machine crashes before any log line | `bot.py:20`, `bot.py:166-167` |
| 10 | Input and start-up errors (a missing sheet, the Pending-batch refusal) are not written to the run log, and `.env` keys `ALERT_SMTP` and `CLAIMDESK_PASSWORD_FALLBACK` are unused | Ops sees an empty log, and there are no alerts | `bot.py:54-63`, `bot.py:167-169` |
| 11 | No Stop button and no clean shutdown. It is unknown whether closing the launcher stops the bot. | Any stop leaves the batch open and blocks uploads | `bot.py:157-180`; launcher not in repo |
| 12 | An open batch blocks every new upload, there is no resume, and there is no way for ops to clear a batch or correct a row | One red row or one killed run stops all posting until dev edits the database | `bot.py:150-154`, `bot.py:167-168` |
| 13 | The Report screen hides `status` and `tech_comments`, and clinic and batch status are not shown anywhere ops can open | Ops cannot see whether a batch is blocking, or the developer cause | `report_screen.py:6` |
| 14 | No time stamp on Report rows. The log prints "[ERROR] [ERROR]". "[RUN SUMMARY] … done" is written even when the batch is not Completed. A clinic retry writes duplicate rows. | Rows can't be matched to the log, and "done" is misleading | `bot.py:122-126`, `bot.py:144`, `bot.py:180` |
| 15 | The launcher and update code are not in the repo, and the 1.3.0 "fix" is a comment only | Update failures can't be checked or fixed | `report_screen.py:8`, `report_screen.py:14` |
| 16 | `.env` and a run log are tracked in git. That run log contains a patient name. The ClaimDesk password sits in plain text in the input Excel. | PHI and credential exposure | `logs/run_20260901_101500.log:3`, `.env`, `bot.py:56` |
| 17 | `BOT_WAS_RUN` is written on every start and never read, and there is no lock against two runs at once | Two runs started together can both pass the open-batch check | `bot.py:161`, `bot.py:167` |
tests/evals/results/handoff/h-desktop-poster-full-r4.json
+34
View File
@@ -0,0 +1,34 @@
{
"case": "h-desktop-poster/full",
"rep": 4,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"comma rule before a run": true,
"how to stop": true,
"not-confirmed row: check ClaimDesk first": true,
"hidden tech comments flagged": true,
"Pending batch blocks a new upload": true
},
"cost_usd": 1.9490990000000001,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-desktop-poster-full-5atujyru\\project",
"reply": "I wrote `HANDOVER.md` in the repo root. Bot-a run pannala, code change edhuvum illa, verum read-only scan mattum dhaan.\n\n| Item | Count |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 19 (E01\u2013E19) |\n| Failure classes | 12/12 covered, not-applicable 0 |\n| Fix before handover | 20 items |\n\n**Fix before handover \u2014 mukkiyamanavai:**\n- Oru red Error row irundhaale andha batch Pending-laye nikkum. Pending batch iruntha next upload-um refuse aagum. Release panna ops-ku screen illa; ovvoru thadavaiyum dev dhaan venum.\n- \"Rejected by app\" and \"Skipped\" rows green Completed-aa kaattudhu, aana avai post aagala.\n- Watchdog (5 min la stop pannum timer) payment-ku mattum dhaan reset aagudhu. Normal-aa slow sign-in or retry nadakkumbodhu kooda run-a stop pannidum. Appo ClaimDesk open-aave irukkum, batch Pending aagum.\n- Wrong-window bug (\"fix\" commit d9a2483) and update bug (d84ff92): rendu commit-um oru comment line mattum dhaan add pannirukku, code change illa. Rendum innum open.\n- Bot ella ClaimDesk window-um force-close pannudhu, user-oda sondha session-um serthu.\n- Clinic name-la comma or semicolon irundha andha name-a split pannidum. README-la \"one clinic per row\" nu sonnalum andha split nadakkum.\n- `&` name ulla clinic-um fail aagudhu.\n- Cutoff date-a text-aa compare pannudhu, adhanaala date order thappa varalaam.\n- `.env`-um, oru patient name ulla run log-um git-la commit aagi irukku. Andha log-a history-la irundhu purge pannanum.\n\nUnga answers-a use pannen:\n- **Escalation:** RPA dev team, #rpa-support channel.\n- **Ops access:** Report screen and logs folder mattum. DB-la mattum irukka causes ellam Fix list-ku pochu.\n- **Unclear statuses:** code padi ezhudhinen. \"Pending\" file status-a current code ezhudhaadhu.\n\nQUESTIONS: Escalation: who does ops call for a dev issue, and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions mean whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-desktop-poster-full-r4.md
+215
View File
@@ -0,0 +1,215 @@
# ClaimDesk Poster - ops handover
Version 1.4.2 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
ClaimDesk Poster posts remittance payments into the **ClaimDesk** desktop app, one clinic at a time. It reads one input Excel file with three sheets: **Login** (the ClaimDesk user name and password), **Clinics** (the clinics to work on) and **Settings** (the cutoff date). For every clinic it signs in to ClaimDesk, opens the clinic, and works through the remittance list. ACH payments dated on or before the cutoff are posted and saved. Non-ACH payments are skipped, later-dated payments are deferred, and payments ClaimDesk rejects are cancelled. Every payment gets one row on the **Report screen** of the ClaimDesk Poster launcher. Each run is one **batch**. It drives one app: **ClaimDesk**.
## 2. How it works
```
ClaimDesk Poster launcher (the window you use; title shows the version)
└─ starts → Posting process ──drives──> ClaimDesk desktop app
│ (do not touch while it runs;
│ the bot force-closes it after every clinic)
├─ writes → Report screen (one row per payment)
├─ writes → logs/run_<date>_<time>.log (one file per run)
└─ watchdog: ends the run if no payment is handled for 5 minutes
```
- **ClaimDesk Poster launcher**: where you start a run and open the Report screen. The title bar shows the version.
- **Posting process**: runs in the background with no window of its own. It does the clicking in ClaimDesk.
- **ClaimDesk**: the target app. The bot starts it, signs in, and **force-closes every ClaimDesk window on the machine** when each clinic finishes, including one you opened yourself.
- **Report screen**: shows Clinic, Check No, File Status (coloured) and Comments for every payment the bot touched.
- **Run log**: a text file in the `logs` folder, named by the start time. Clinic-level problems are recorded **only here**.
- **Watchdog**: a timer inside the bot. If no payment is handled for 300 seconds, it ends the whole run.
**One run, step by step:**
1. The bot checks that no earlier batch is still unfinished. If one is, it refuses to start (E01).
2. It reads the input Excel: Login, Clinics and Settings sheets.
3. It creates a new batch and marks every clinic as not yet done.
4. For each clinic: it starts ClaimDesk and signs in, opens the clinic from the clinic list, and checks that the window title shows that clinic's name.
5. It reads the remittance list and, for each payment, either skips it (not ACH), defers it (after the cutoff), cancels it (ClaimDesk shows "Remittance Errors"), or clicks Post Payment → Save and waits for "Payment saved".
6. Each payment gets a Report screen row straight away.
7. When the whole list is done, the clinic counts as done. If anything fails before that, the bot closes ClaimDesk and retries the clinic from sign-in, up to 3 attempts in total.
8. At the end, the batch is finished **only if** no row is red (Error) and no clinic was left undone. The log always ends with "[RUN SUMMARY] batch <n> done", **even when the batch is not finished**.
## 3. Before a run
- [ ] The launcher title bar shows version **1.4.2** (see section 10).
- [ ] **The last batch is finished.** On the Report screen for the last batch there is no red (Error) row, and the last run log has no "left Pending" line. Otherwise the new run is refused (E01), and you need the dev team to release the old batch (E17).
- [ ] The settings file (`.env`) is in the bot folder and the bot's database is in its `data` folder. Dev sets both up at install. Ops should not edit them. Without the settings file every clinic fails (E04).
- [ ] The input Excel has three sheets named exactly **Login**, **Clinics**, **Settings**.
- Login: user name in **A2**, password in **B2**. The password must be the current ClaimDesk password. A wrong password does not give a clear message: every clinic fails with "not found in list" (E07).
- Clinics: column A, from row 2 down. **One clinic per row.**
- Settings: the cutoff date in **B1**. If B1 is blank, there is no cutoff and every ACH payment is posted.
- [ ] **Clinic names must contain no comma and no semicolon.** The bot cuts every cell at commas and semicolons, even when the row holds one clinic.
- Bad: `North, East Clinic` → the bot looks for "North" and "East Clinic", finds neither, and leaves both undone.
- Good: `North East Clinic` (only if that is exactly how ClaimDesk spells it).
- A clinic whose real ClaimDesk name has a comma or semicolon **cannot be run by the bot**. Post it by hand and tell the dev team.
- [ ] Spell each clinic **exactly** as in ClaimDesk's clinic list: spaces, capitals, `&`. Clinics with `&` in the name are known to fail with "Wrong clinic opened" (E08). Plan to post those by hand until dev fixes it.
- [ ] Cutoff date: enter it as a real Excel date, not text.
- Bad: `30-Sep` typed as text.
- Good: a date cell showing 30/09/2026.
- Even then, spot-check the first Deferred and Posted rows after the run. The bot compares dates as text, so it can get the order wrong (E13).
- [ ] The machine stays unlocked with the screen on for the whole run (no lock screen or screensaver). The bot clicks on the screen like a person.
- [ ] Close every window whose title contains "ClaimDesk", except the ClaimDesk Poster launcher. That includes your own ClaimDesk session and SQL or database tools showing "ClaimDesk" in the title (E18).
## 4. During a run
- Don't use the mouse or keyboard on this machine. The bot clicks on screen, and your input can send its clicks to the wrong place.
- Don't open ClaimDesk on this machine. After every clinic the bot force-closes **all** ClaimDesk windows, yours included.
- Don't open any window with "ClaimDesk" in its title (SQL or database tools, a second launcher). The bot may grab it instead of the real ClaimDesk (E18).
- **To stop the bot:** the bot has no Stop button, and its code does not watch the launcher. Closing the launcher may leave it running. The safe way:
1. Open Task Manager and end the **Python** process that runs the bot.
2. Close ClaimDesk.
3. Tell the dev team. The batch is left unfinished and blocks the next run (E17).
- Closing ClaimDesk alone does **not** stop the bot. It treats this as a failure and starts ClaimDesk again, up to 3 attempts per clinic.
- **A run looks stuck** when no new rows appear on the Report screen and no new lines appear in the run log for 5 minutes. The watchdog ends the run itself after 300 seconds with no payment handled. Wait 6 minutes before acting. After a watchdog stop, see E16.
## 5. Where to look when something goes wrong
Ops can open the launcher and its Report screen and the `logs` folder. The bot writes no other output files or folders. It does not save screenshots, Excel output or emails.
| Question | Where to look |
|---|---|
| Did the run finish? | The run log (`logs/run_<date>_<time>.log`, newest file). The last line reads "[RUN SUMMARY] batch <n> done" when the bot reached the end. If the last line is "[WATCHDOG] …", the run was cut off (E16). If the file is **empty**, the run never started (E01-E03). |
| Is the batch fully finished? | No screen shows this. It is finished only if the Report screen shows **no red row** for the batch **and** the log has **no "left Pending" line**. |
| Which payments failed? | Report screen → the batch → File Status = **Error** (red). |
| Which payments were not posted but show green? | Report screen → File Status = Completed → read **Comments**. Only "Posted successfully" means posted (see section 6). |
| Which clinics failed completely? | The Report screen shows nothing for them. Run log → search for **"left Pending"**, then for **"[ERROR] Clinic '<name>'"** to see why. |
| Why did one payment fail? | Report screen → **Comments** column. The bot writes no per-payment lines in the log. |
| Did the bot already retry a clinic? | Run log → lines "**[AUTO-RERUN] attempt 2**" / "**attempt 3** for '<clinic>'". |
| Which version is running? | The launcher title bar. |
## 6. Statuses
The Report screen shows **File Status** (coloured) and **Comments**. The bot also stores an internal status per row, shown in brackets below. Ops cannot see it, but dev may quote it.
| Status | Colour | Meaning | What ops does |
|---|---|---|---|
| Completed - "Posted successfully" (internal: Posted) | green | Payment posted and ClaimDesk showed "Payment saved". | Nothing |
| Completed - "Remittance Errors on posting" (internal: Rejected by app) | green | **Not posted.** ClaimDesk showed its Remittance Errors screen and the bot cancelled. **Needs follow-up despite the green colour.** | Review and post it in ClaimDesk by hand (E11) |
| Completed - "Payment method '<method>' - not posted" (internal: Skipped) | green | **Not posted on purpose.** The bot only posts ACH. | Nothing, unless the client wants non-ACH posted (E12) |
| Deferred - "Check date <date> is after the cutoff - deferred" (internal: Deferred) | grey | Not posted. Dated after the Settings cutoff. It stays in ClaimDesk for a later run. | Spot-check the date against the cutoff (E13) |
| Error - "Save button not found" (internal: Error) | red | The payment screen opened but had no Save button. Not saved. **Blocks the batch.** | E14 |
| Error - "Payment not confirmed after Save" (internal: Error) | red | Save was clicked, but "Payment saved" did not appear within 20 s. **It may have posted.** **Blocks the batch.** | Check in ClaimDesk first (E15) |
| Pending | amber | The current version never writes this for a payment. It would mean an unknown result. | Report it to dev |
| Clinic: not done / done (internal: Pending / Success) | not shown | A clinic stays "not done" if all 3 attempts fail. The only trace is "left Pending" in the log. **Blocks the batch.** | E10 |
| Batch: unfinished / finished (internal: Pending / Completed) | not shown | Finished only when no red row and no undone clinic. An unfinished batch **blocks every new run**. | E17 |
## 7. Exception catalogue
Quote the ID (E01…) in every message to #rpa-support.
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Run does nothing; the new log file is empty; no new rows on the Report screen. (If the launcher shows the bot's message: "A Pending batch exists - finish or clear it before a new upload".) | At start, the bot looks for any earlier batch that is not finished and refuses to start. | **Not in the log file.** The message goes only to the launcher's output, if it shows it. | System | Dev | Check the last batch for red rows and "left Pending" lines (E17). There is no screen to release a batch. | Always. Send: time of the attempt, the empty log file name, the last batch number. |
| E02 | Run does nothing; empty log file. (Launcher may show "'Login' sheet not found" or "UserName/Password missing in Login sheet (row 2)".) | The Login sheet is missing, or A2 or B2 is empty. | Not in the log file. No batch is created. | Business | Ops | Fix the Login sheet (name exactly "Login", user in A2, password in B2) and run again. | The sheet is correct and it still fails. |
| E03 | Run does nothing; empty log file; no friendly message. | The Clinics or Settings sheet is missing or misnamed. The bot stops with a technical error. | Not in the log file. No batch is created. | Business | Ops | Check that the sheets are named exactly "Clinics" and "Settings", then run again. | All three sheets are correct and it still fails. Send the time and the empty log file name. |
| E04 | Every clinic fails. Log: "[ERROR] Clinic '<name>' : 'CLAIMDESK_EXE'", then AUTO-RERUN and "left Pending" for each clinic. | The bot cannot find the ClaimDesk program path in its settings file. | Run log | System | Dev | Don't re-run. The settings file is missing or incomplete. | Always. Send the log file name. |
| E05 | Log: "[ERROR] Clinic '<name>' : Login window not found" | The ClaimDesk sign-in window did not appear within 60 s of starting ClaimDesk. | Run log | System | Bot, then Ops | The bot retries the clinic up to 3 times. If it is left undone: open ClaimDesk by hand on this machine and check it starts and shows its sign-in window. | ClaimDesk starts fine by hand but the bot still gets this. Send log file name and time. |
| E06 | Log: "[ERROR] Clinic '<name>' : ClaimDesk window not found" | After sign-in, no window with "ClaimDesk" in its title exists. | Run log | System | Bot, then Dev | The bot retries up to 3 times. | It happens on 2 runs in a row. Send log file name and time. |
| E07 | Log: "[ERROR] Clinic '<name>' : Clinic '<name>' not found in list" | The clinic link with that exact name did not appear within 10 s. | Run log; clinic left undone (no Report rows) | Business | Ops | 1) Check the spelling matches ClaimDesk exactly. 2) Check the name has no comma or semicolon; if the log shows a piece of a name, that is the comma split (section 3). 3) If **every** clinic shows this, check the Login sheet password: a failed sign-in looks like this. | Spelling and password are correct. Send log file name, time, clinic name. |
| E08 | Log: "[ERROR] Clinic '<name>' : Wrong clinic opened, leaving Pending: <name>" | After clicking the clinic, the window title does not contain the clinic's name. Despite the wording, the bot still retries 3 times. | Run log; clinic left undone | System | Ops, then Dev | Check the spelling. Names with `&` are known to fail here: post that clinic by hand in ClaimDesk. | Spelling is exact. Send log file name, time, clinic name. |
| E09 | Log: "[ERROR] Clinic '<name>' : Remittance grid did not load" | The remittance list did not appear within 60 s after opening the clinic. | Run log | System | Bot, then Dev | The bot retries up to 3 times. If ClaimDesk is slow for everyone, run again later. | Same clinic on 2 runs. Send log file name and time. |
| E10 | Log: "[DB] Clinic '<name>' left Pending after 3 attempts". Few or no Report rows for that clinic. | All 3 attempts at the clinic failed. | Run log only (not on the Report screen) | System or Business | Ops | Read the "[ERROR] Clinic '<name>'" lines just above it and follow E04-E09. The batch stays unfinished (E17). | The cause is a Dev row above. |
| E11 | Green Completed, Comments "Remittance Errors on posting" | After Post Payment, ClaimDesk showed its "Remittance Errors" screen within 5 s. The bot clicked Cancel. | Report screen | Business | Ops | **It was not posted.** Review the remittance in ClaimDesk and post or correct it by hand. | Never, unless every payment shows it |
| E12 | Green Completed, Comments "Payment method '<method>' - not posted" | The payment method in the list is not ACH. | Report screen | Business | Ops | Nothing, unless the client wants that method posted. Then post by hand. | Never |
| E13 | Grey Deferred, Comments "Check date <date> is after the cutoff - deferred" | The check date is later than the Settings cutoff (compared as text). | Report screen | Business | Ops | Compare the date with the cutoff. If it is really before the cutoff, post it by hand. Do the same check in reverse: a Posted row dated after the cutoff. | A wrong Deferred or Posted is found. Send batch number, check number, cutoff. |
| E14 | Red Error, Comments "Save button not found" | The payment screen opened but no Save button appeared within 10 s. | Report screen | System | Ops, then Dev | Check that payment in ClaimDesk: it was **not** saved. Post it by hand if needed. The bot may have left the payment screen open, so the next payment in that clinic may also fail. | Always (the row blocks the batch). Send batch number, check number, log file name, time. |
| E15 | Red Error, Comments "Payment not confirmed after Save" | Save was clicked, but "Payment saved" did not appear within 20 s. | Report screen | System | **Ops** | **Check the payment in ClaimDesk before doing anything.** It has often really posted. Do **not** post it again by hand if it shows as posted. If it is not posted, post it by hand or leave it for the next run. Either way the red row stays, and only dev can correct it. | Always, so dev can correct the row and release the batch. Send batch number, check number, and whether it posted in ClaimDesk. |
| E16 | Run ended mid-way. Last log line "[WATCHDOG] no progress for 300s - killing run". No "[RUN SUMMARY]" line. ClaimDesk may still be open. | No payment was handled for 300 s. This also happens during normal slow sign-ins and retries, or when several clinics in a row have no payments. | Run log | System | Ops, then Dev | Close ClaimDesk. Note which clinics have no Report rows. The batch is left unfinished (E17). | Always. Send log file name and the clinic it was on. |
| E17 | The next run is refused (E01), or a batch never finishes | Any red Error row, or any clinic left undone, keeps the batch unfinished. An unfinished batch blocks every new run. | Report screen (red rows) + run log ("left Pending") | Business / System | Ops, then Dev | 1) List the red rows and "left Pending" clinics. 2) Settle each one in ClaimDesk (E10, E14, E15). 3) Ask dev to release the batch. Ops has no screen for this. | Always. Send batch number and the list from step 1. |
| E18 | ClaimDesk closes by itself, or your own ClaimDesk session closes; or the bot works in the wrong window | The bot takes the first window with "ClaimDesk" in its title (this can be a SQL tool, the launcher or the sign-in window). It force-closes every ClaimDesk window after each clinic. | Not recorded. May show as E07 or E08 in the log. | System | Ops | Close all other "ClaimDesk" windows before a run. Don't use ClaimDesk on this machine during a run. | It happens with no other ClaimDesk window open |
| E19 | The same check number appears twice or more for one clinic on the Report screen | When a clinic is retried, rows from the earlier attempt stay, and Skipped, Deferred or Rejected payments are recorded again. | Report screen; "[AUTO-RERUN]" in the log | System | Ops | Treat duplicates as one payment. Check ClaimDesk for the real state. | Two rows for one check disagree (e.g. one Posted, one Error) |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Any clinic-level failure (sign-in, clinic not found, wrong clinic, list didn't load, unexpected screen) | Force-closes ClaimDesk, starts it again, signs in and re-does the clinic from the start | 3 attempts per clinic. Each attempt can wait up to 60 s for sign-in, 10 s for the clinic link, 60 s for the list. | Log "left Pending". The bot moves to the next clinic and the batch stays unfinished. |
| One payment fails (Save button missing, save not confirmed) | **No retry.** Records a red Error row and moves to the next payment. | none | Red row; batch unfinished |
| ClaimDesk "Remittance Errors" screen | Clicks Cancel and moves on | waits 5 s for it | Green "Remittance Errors on posting" row |
| No progress | The watchdog ends the whole run | 300 s with no payment handled | Run stops. ClaimDesk is left open and the batch unfinished (E16). |
| Bot stopped or crashed | **No resume.** A new run starts a new batch from the Excel, and only after the old batch is released by dev. | none | E01 / E17 |
## 9. Known incidents
Dates were not kept in the project history. Versions are given where known. Each "fix" recorded in history added only a note to the code. The code change itself is not in this repository, so treat the risks as still open.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| before 1.4.2 | Users stayed on 1.3.0 after an update | The launcher kept using the old folder until it was restarted | After every update, close the launcher fully and reopen it. Check the title bar version (section 10). | Yes |
| before 1.4.2 | Red "Payment not confirmed after Save" rows for payments that had really posted | ClaimDesk took longer to save than the bot waited. The wait was raised from 10 s to 20 s but is still a fixed wait. | Always check in ClaimDesk first (E15) | Partly: ops can check, only dev can correct the row |
| before 1.4.2 | Clinics silently never posted | Clinic names with commas were cut into pieces | One clinic per row **and** no comma or semicolon in any name. Clinics whose real name has a comma are posted by hand (section 3). | Partly |
| before 1.4.2 | The bot drove a SQL tool window, then closed the real ClaimDesk | Any window with "ClaimDesk" in its title is grabbed | Close all other "ClaimDesk" windows before a run (E18). The code still matches this way. | Partly |
| 2026-09-01 run | A clinic with `&` in its name failed "Wrong clinic opened", re-ran, then the watchdog killed the run | `&` name check (E08), plus the watchdog counting retries as "no progress" (E16) | Post `&` clinics by hand. Escalate watchdog stops. | No |
## 10. Releases and version check
- **How an update arrives:** through the ClaimDesk Poster launcher. The launcher and updater are not in this repository; dev owns them.
- **How to confirm it applied:** the launcher title bar shows the new version (currently **1.4.2**).
- **If the version did not change:** close the launcher completely (check that it is gone from Task Manager) and reopen it. Still old? Post in #rpa-support with the machine name and the version shown.
- Each install keeps its own Report data, and nothing shared is rewritten at start. A machine on an old version does not damage another. Still keep all machines on the same version so the doc matches what you see.
## 11. Escalating to dev
Post in **#rpa-support** (Teams), to the RPA dev team.
Send:
- the exception ID from section 7 (e.g. E15)
- the batch number (from the "[RUN SUMMARY] batch <n>" line or the Report screen)
- the time of the failure
- the run log **file name** (`logs/run_<date>_<time>.log`) and the machine name
- the clinic name and check number
- for E15: whether the payment shows as posted in ClaimDesk
Never send:
- passwords
- the settings file
- the input Excel (its Login sheet holds the ClaimDesk password)
- patient or customer data in chat
Don't paste log lines into Teams: older run logs can contain patient names. Dev reads the log on the machine.
## 12. Failure-class check
- F1 (missing prerequisite) covered in section 3 (settings file, database, input sheets) and E02, E03, E04. The missing-database case → Fix #19.
- F2 (wrong window / focus) covered in sections 3 and 4 and E18. Code fix → Fix #6, #7.
- F3 (app slower or different than expected) covered in E05, E06, E07, E09, E11, E14, E15 with the timeouts in section 8; unlocked screen rule in section 3. The 5 s popup wait → Fix #15.
- F4 (input read differently) covered in section 3 (comma/semicolon split, `&`, cutoff date, password) and E07, E08, E13. Code → Fix #8, #9, #10.
- F5 (marked done when nothing was done) → Fix #13: a clinic counts as done with zero payments and no confirmation. Not visible to ops.
- F6 (action happened, not confirmed) covered in E15 and incident 2. Code → Fix #14.
- F7 (business exception shown as success) covered in section 6 and E11, E12. Mapping → Fix #3.
- F8 (endless re-runs / watchdog) covered in section 8 and E16 (3 attempts per clinic, so no endless loop). Watchdog heartbeat → Fix #5.
- F9 (stopping the wrong way) covered in section 4. No Stop button → Fix #18.
- F10 (one stuck item blocks the queue) covered in E01, E17 and section 3. No release tool → Fix #1.
- F11 (cause hidden from ops) covered in section 5 (log-only clinic failures, no visible batch status). → Fix #2, #4, #17.
- F12 (update didn't arrive) covered in section 10 and incident 1. Launcher not in repo → Fix #18.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Any Error row or undone clinic keeps the batch Pending forever, and any Pending batch blocks every new upload. There is no screen or tool to release a batch. | Ops are blocked after a single red row and need dev for every release (E17) | `bot.py:150-154`, `bot.py:167-168` |
| 2 | The "Pending batch exists" refusal and input-sheet errors are raised without being logged | Ops see an empty log file and no reason (E01-E03) | `bot.py:167-169`, `bot.py:52-63` |
| 3 | "Rejected by app" and "Skipped" map to Completed (green) | Unposted payments look done (E11, E12) | `bot.py:27-33` |
| 4 | Report screen hides `status` and `tech_comments`; clinic status and batch status are not shown anywhere; clinic failures produce no Report rows | Causes are only in the DB (ops can't open it) or the log | `report_screen.py:6`, `bot.py:140`, `bot.py:147` |
| 5 | The watchdog heartbeat beats only per payment, so it kills runs during normal sign-in, retries or empty clinics. `os._exit` skips cleanup, leaving ClaimDesk open and the batch Pending. | False kills, blocked next run (E16) | `bot.py:36-49`, `bot.py:137` |
| 6 | The ClaimDesk window is matched by title substring, which can match the launcher ("ClaimDesk Poster"), "ClaimDesk Login" or a SQL tool. Commit d9a2483 added only a comment; there is no code fix. | Wrong window driven (E18) | `bot.py:66-70`, `bot.py:83`, `bot.py:185` |
| 7 | `taskkill /im ClaimDesk.exe /f` after every attempt kills every ClaimDesk process on the machine | Users lose their own ClaimDesk work | `bot.py:146` |
| 8 | Clinic cells are split on `,` and `;`, even with one clinic per row. The README advice doesn't prevent it. | Clinics with commas can never be run (E07) | `bot.py:62`, `README.md:8` |
| 9 | The "wrong clinic" check needs the exact name in the window title. It fails for `&` names. | `&` clinics always fail (E08) | `bot.py:91` |
| 10 | The cutoff is compared as text (`check_date > str(cutoff)`) | Payments wrongly deferred or posted (E13) | `bot.py:107` |
| 11 | A failed sign-in is not detected. It surfaces later as "Clinic not found in list". | Misleading message (E07) | `bot.py:82-83` |
| 12 | Retries re-insert report rows. Rows carry no timestamp or attempt number. | Duplicate rows (E19) | `bot.py:122-139` |
| 13 | A clinic is marked Success with zero payments, with no positive confirmation | A clinic done on the wrong screen looks finished | `bot.py:140` |
| 14 | "Payment not confirmed" relies on a fixed 20 s wait with no re-check of the list. Ops cannot correct the row. | False red rows that block the batch (E15) | `bot.py:117-118`, `bot.py:21` |
| 15 | The Remittance Errors popup wait is only 5 s. After "Save button not found", the payment screen is left open. | A slow popup or open screen turns into Error rows or clinic retries (E14) | `bot.py:110-115` |
| 16 | The grid is read live while posting, header rows are not skipped, and a short row raises IndexError | Skipped or double-handled payments; spurious clinic retries | `bot.py:95-101` |
| 17 | "[RUN SUMMARY] batch N done" is logged even when the batch is not completed | Ops think the batch is finished | `bot.py:180` |
| 18 | No Stop button; the worker doesn't stop when the launcher closes. The launcher and updater code (incl. the d84ff92 fix) are not in this repo. | Ops must use Task Manager. The update fix can't be verified. | `bot.py:157-180`, `report_screen.py:14` |
| 19 | The `data/` folder and database schema are not in the repo or created by code | A fresh install fails with a database error | `bot.py:20`, `bot.py:166` |
| 20 | `.env` and a run log containing a patient name are committed to git. The ClaimDesk password sits in plain text in the input Excel. | Credential and PHI exposure. The log must be purged from history and old logs on ops machines checked. | `.env`, `logs/run_20260901_101500.log:3`, `bot.py:56` |
tests/evals/results/handoff/h-desktop-poster-full-r5.json
+34
View File
@@ -0,0 +1,34 @@
{
"case": "h-desktop-poster/full",
"rep": 5,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"comma rule before a run": true,
"how to stop": true,
"not-confirmed row: check ClaimDesk first": true,
"hidden tech comments flagged": true,
"Pending batch blocks a new upload": true
},
"cost_usd": 1.8817840000000001,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-desktop-poster-full-a_jp15nd\\project",
"reply": "`HANDOVER.md` ready \u2014 repo root la irukku: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-desktop-poster-full-a_jp15nd\\project\\HANDOVER.md`. Bot-a run pannala; code, git history, log mattum read pannen. Unga escalation, ops access answers use pannirukken; unclear statuses-ku code-oda best reading. Unga answers already irundhadhaala skill sonna \"ask once\" step-la questions ketkala.\n\n| Check | Result |\n|---|---|\n| Sections | 13 (template-oda ella sections-um) |\n| Exceptions catalogued | 16 (E01\u2013E16) |\n| Failure classes | 12/12 covered, onnum \"not applicable\" illa |\n| Statuses | Item status 5, file status 4, clinic 2, batch 2 \u2014 ellam section 6-la irukku |\n| Fix before handover | 20 items (file:line oda) |\n\n**Ops-ku romba mukkiyamaana 4 vishayam:**\n- **Oru red row irundhaale batch Pending-la ninnudum.** Adhukku apram vara ella upload-um refuse aagum. Ops-ku DB access illa, adhanaala clear panna dev thaan venum.\n- **Green rows ellam posted illa.** \"Remittance Errors on posting\" row green-a kaatum, aana post aagala. \"Skipped\" um green thaan.\n- **\"Payment not confirmed after Save\" red row-a ninaichu re-run pannadheenga.** Post aagirukkalaam; munnaadi ClaimDesk-la check pannunga.\n- **3 \"fix\" commits-la (d9a2483, c7215d0, d84ff92) comment mattum thaan add aagirukku, code maaralai.** Wrong-window bug innum open. Save wait 20 s already initial import-laye irundhudhu.\n\n**Fix before handover (dev team, 20 items):**\n1. Window title-a \"contains ClaimDesk\" nu match pannudhu \u2014 launcher, login window, SQL tools ellathayum pidikkum.\n2. \"Rejected by app\", \"Skipped\" green Completed-a kaatudhu; internal status column screen-la illa.\n3. Tech comments DB-la mattum irukku, screen-la kaatala.\n4. Fail aana clinic-ku Report row varadhu; clinic/batch status ops-ku engayum theriyaadhu.\n5. Oru red row illa failed clinic batch-a eppavume Pending-la vechirukkum; ops clear panna vazhi illa.\n6. Watchdog heartbeat item post aana mattum thaan; healthy run-a kill pannum, ClaimDesk open-a vittudum, resume illa.\n7. Save timeout-ku apram grid-a re-check pannala \u2014 double-post risk.\n8. Sign-in success check illa; wrong password \"Clinic not found\"-a kaatum.\n9. Zero items / ellam red-a irundhaalum clinic Success; wrong-clinic check click-ku apram udane, `&` problem.\n10. Clinic cell `,` / `;`-la split aagudhu; comma ulla name eppavume run aagaadhu.\n11. Cutoff date-a text-a compare pannudhu.\n12. Input errors, \"Pending batch exists\" refusal log-la varala.\n13. Batch Pending-a irundhaalum log \"[RUN SUMMARY] done\" nu sollum.\n14. Retry duplicate rows podum; taskkill machine-la irukka ella ClaimDesk-um close pannum.\n15. Stop button illa, safe stop vazhi illa; launcher code repo-la illa.\n16. Launcher/updater repo-la illa; 1.3.0 fix comment mattum.\n17. Alerting illa \u2014 `ALERT_SMTP` define aagirukku, use aagala.\n18. DB tables code create pannaadhu; ClaimDesk path illana cryptic error.\n19. Security: `.env` git-la commit aagirukku (password-fallback key um serthu), committed log-la patient data irukku, Excel-la plain-text password.\n20. Committed log-la current code ezhudhaadha messages irukku \u2014 production-la endha build nu confirm pannunga.\n\n**Also noticed:**\n- Ops-ku \"output folder\" access sonnenga, aana bot output Excel/folder edhuvum create pannaadhu. Report screen, `logs` folder mattum thaan.\n- \"Launcher close panna bot nikkuma\" nu check panna mudiyala (launcher code repo-la illa), adhanaala doc-la UNVERIFIED nu mark pannirukken.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role) and how (ticket, chat channel) | Ops access: which of the bot's places ops can open - the app's screens, the output folder, the log folder, the database | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear",
"stderr": ""
}
tests/evals/results/handoff/h-desktop-poster-full-r5.md
+215
View File
@@ -0,0 +1,215 @@
# ClaimDesk Poster - ops handover
Version 1.4.2 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
The bot posts remittance payments into the **ClaimDesk** desktop app, one clinic at a time. You start it from the **ClaimDesk Poster** launcher window and give it an input Excel with three sheets:
- **Login:** the ClaimDesk user name and password.
- **Clinics:** the clinics to process.
- **Settings:** the cutoff date.
For each clinic it signs in to ClaimDesk, opens the clinic and reads the list of unposted remittances. Then it handles each item:
- **ACH payments dated on or before the cutoff:** it posts them.
- **Items paid by any other method:** it skips them.
- **Items dated after the cutoff:** it defers them.
You see one result row per item on the launcher's **Report screen**. The bot also writes a run log in the `logs` folder. It drives only one app, **ClaimDesk**.
## 2. How it works
```
ClaimDesk Poster launcher (the window you use, shows the Report screen)
└─ starts → bot (automation process) ──drives──> ClaimDesk desktop app
│ (do not touch while it runs)
├─ writes → bot database (batch, clinics, one row per item)
│ └─ read by → Report screen
└─ writes → logs/run_<date>_<time>.log
```
- **ClaimDesk Poster launcher:** where you start a run and read the results on the Report screen. Its title bar shows the version.
- **Bot:** a background process with no window of its own. It moves the real mouse and types into ClaimDesk.
- **ClaimDesk:** the payment app. The bot starts it, signs in, and **force-closes it after every clinic**. That closes every ClaimDesk window on the machine, including one a person has open.
- **Bot database:** holds the batch, clinic and item results. Ops cannot open it; the Report screen shows part of it.
- **Run log:** one new file per run, with errors and retries.
**One run, step by step:**
1. The bot checks that no earlier batch is still unfinished. If one is, it refuses to start (E01).
2. It reads the input Excel and creates a new batch.
3. For each clinic:
1. It starts ClaimDesk and signs in.
2. It clicks the clinic in the clinic list.
3. It reads the unposted remittance list.
4. It handles each item: skip, defer, or post and save. One row per item goes to the Report screen.
5. It closes ClaimDesk.
6. If anything fails at clinic level, it repeats the whole clinic, up to 3 tries.
4. At the end it marks the batch Completed, but only when no red (Error) row and no unfinished clinic is left. Otherwise the batch stays Pending (see E01).
## 3. Before a run
- [ ] The previous batch is finished. If it isn't, the bot refuses to start (E01).
- [ ] Nobody is using ClaimDesk on this machine. The bot closes every ClaimDesk window.
- [ ] No other window whose title contains "ClaimDesk" is open: SQL tools, Explorer folders, browser tabs, a second launcher. The bot can grab the wrong one (E15).
- [ ] **Login sheet:** user name in cell A2, password in cell B2. Both must be filled.
- [ ] **Clinics sheet:** clinic names in column A, starting at row 2.
- **Put one clinic per row.** Never put two clinics in one cell.
- A name that itself contains a comma or a semicolon cannot be processed at all. Tell dev about such clinics.
- Bad: `North Clinic, South Clinic` in one cell. Good: `North Clinic` in A2, `South Clinic` in A3.
- Spell each name exactly as it appears in ClaimDesk's clinic list, including `&`, dots and spaces.
- [ ] **Settings sheet:** the cutoff date in cell B1.
- The bot compares dates **as text**, not as dates. So the cutoff must be written in the **same format ClaimDesk shows check dates in**.
- If you are not sure the format matches, ask dev before the first run.
- [ ] The ClaimDesk program path is set in the bot's settings file. Dev does this once per machine. If it is missing, every clinic fails (E06).
- [ ] The launcher title bar shows version 1.4.2 (see section 10).
## 4. During a run
- **Hands off this machine.** Don't use the mouse or keyboard: the bot moves the real mouse, so your clicks and its clicks collide.
- **Don't close or use ClaimDesk.** If you close it, the bot treats the clinic as failed and restarts it.
- **To stop the bot:** there is no Stop button.
- Best: let the run finish.
- If you must stop it, close the ClaimDesk Poster launcher, then watch ClaimDesk for one minute. If ClaimDesk keeps moving, the bot is still running; contact dev. (Whether closing the launcher stops the bot could not be checked; the launcher code is not in this repo.)
- **Any stop leaves the batch unfinished, and that blocks the next upload** (E01).
- **A run looks stuck** when ClaimDesk sits on the same screen and the newest log file stops growing.
- The bot gives up by itself after **5 minutes with no item posted**: the log then ends with a `[WATCHDOG]` line (E14).
- Wait those 5 minutes before you act.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | Newest file in the `logs` folder (`run_<date>_<time>.log`). Look at its last line. A `[RUN SUMMARY] batch N done` line means the run reached the end. **It does not mean every item worked**, so check the Report screen too. A `[WATCHDOG]` last line means the run was killed (E14). If neither line is there, the run is still going or crashed. |
| Is the batch still unfinished? | Ops can't see this directly. If the next upload is refused with "A Pending batch exists", it is still unfinished (E01). |
| Which items failed? | Report screen → that batch → rows in **red** (File Status = Error). Also read the comment on **green** rows, because some green rows were not posted (section 6). |
| Why did one item fail? | Report screen → **Comments** column. |
| Which clinics failed completely? | A clinic that failed completely has **no rows at all** on the Report screen. Search the run log for `left Pending after 3 attempts`; the clinic name is on that line. The `[ERROR] Clinic '<name>'` lines just above it give the reason. |
| Did the bot already retry? | Run log: lines starting `[AUTO-RERUN] attempt 2` or `attempt 3` for that clinic. |
| Which version is running? | The launcher title bar. |
The bot creates no output Excel or output files, and takes no screenshots. Everything ops can see is on the Report screen and in the `logs` folder.
## 6. Statuses
**On the Report screen (File Status colour plus Comment).** You can see only the colour and the comment, so **read the comment on green rows**.
| Status (internal name) | Colour / File Status | Comment shown | Meaning | What ops does |
|---|---|---|---|---|
| Posted | green, Completed | "Posted successfully" | Payment posted, and ClaimDesk showed "Payment saved". | Nothing. |
| Rejected by app | **green, Completed** | "Remittance Errors on posting" | **Not posted.** ClaimDesk showed its Remittance Errors screen and the bot pressed Cancel. | **Needs follow-up.** Review and post it in ClaimDesk by hand (E11). |
| Skipped | **green, Completed** | "Payment method '<method>' - not posted" | Not posted on purpose, because the bot only posts ACH payments. | Nothing, unless the client wants non-ACH posted (E09). |
| Deferred | grey, Deferred | "Check date <date> is after the cutoff - deferred" | Not posted, because the check is dated after the cutoff. | Picked up by a later run once the cutoff moves past the check date. Check the date really is later (E10). |
| Error | red, Error | "Save button not found" | Not posted. The payment screen had no Save button. | E12. |
| Error | red, Error | "Payment not confirmed after Save" | **Possibly posted.** The bot clicked Save but didn't see "Payment saved" within 20 seconds. | **Check in ClaimDesk before re-running** (E13). |
| Pending | amber, Pending | (any) | The code never writes this colour today; it is a fallback. | If you ever see it, report to dev. |
**Not shown anywhere ops can see.** These are kept in the bot database only.
| Status | Of what | Meaning | How ops can tell |
|---|---|---|---|
| Pending | Clinic | Not done yet, or it failed all 3 tries. | Run log: "left Pending after 3 attempts". |
| Success | Clinic | The bot went through the clinic's whole list. **This doesn't mean every item posted**; red rows can still exist. | Run log has no "left Pending" line for that clinic. |
| Pending | Batch | Unfinished. **Blocks every new upload.** | The next upload is refused (E01). |
| Completed | Batch | No red rows and no unfinished clinics. | The next upload is accepted. |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | The run refuses to start: "A Pending batch exists - finish or clear it before a new upload". | Before starting, it checks whether any earlier batch is still unfinished. A batch stays unfinished if it has **even one red row**, a clinic that failed all 3 tries, or a run that was killed or stopped. | Message on screen only. The new log file is empty. | System | **Dev** | Ops cannot clear it, because there is no button and ops has no database access. Find the cause first: red rows on the previous batch's Report screen, "left Pending" lines or a `[WATCHDOG]` line in the previous run's log. | Always. Send the previous run's log file name, its batch number, and the red rows / clinic names you found. |
| E02 | The run stops at once with an input message: "'Login' sheet not found", "UserName/Password missing in Login sheet (row 2)", or a technical error naming "Clinics" or "Settings". | It reads the Excel before doing anything else. | Message on screen only. The new log file is empty. No batch is created, so nothing is blocked. | Business | Ops | Fix the Excel: sheet names exactly Login, Clinics, Settings; user name in A2 and password in B2 of Login. Then start again. | The sheets are correct and it still stops. |
| E03 | **Every** clinic fails with "Clinic '<name>' not found in list", and each is left Pending after 3 attempts. | The bot does **not** check whether sign-in worked. After a wrong password it stays on the login screen and can't find any clinic. | Run log `[ERROR] Clinic ... not found in list` lines. No rows on the Report screen. Batch left unfinished. | Business | Ops, then Dev | Sign in to ClaimDesk by hand with the Login sheet's user name and password. If that fails, fix the password in the Excel. | Always for the unfinished batch (E01). Also when the password works by hand. |
| E04 | **One** clinic fails with "Clinic '<name>' not found in list". | The clinic's link doesn't appear in ClaimDesk's clinic list within 10 seconds. It is retried 3 times with a fresh sign-in. | Run log `[ERROR]` and "left Pending after 3 attempts". No rows on the Report screen for that clinic. | Business | Bot, then Ops | Compare the name in the Excel with ClaimDesk's list, letter by letter. A half-name (for example only the part before a comma) means two clinics were in one cell, or the name contains a comma. Fix the input and re-run just that clinic once the batch is cleared (E01). | The spelling is exact, or the real name contains a comma or semicolon. |
| E05 | "Wrong clinic opened, leaving Pending: <name>". | Right after clicking the clinic, it checks whether the ClaimDesk window title contains the clinic name. It checks instantly, so a slow ClaimDesk, or a name with characters like `&`, can trigger it falsely. | Run log `[ERROR]` and `[AUTO-RERUN]` lines. No Report rows for that clinic. | System | Bot, then Ops / Dev | Wait for the 3 tries. If the clinic still fails, check its spelling against ClaimDesk (as in E04). | The spelling is exact. Send the log file name and clinic name. |
| E06 | "Login window not found", "ClaimDesk window not found", or an error that just names the ClaimDesk program setting. | ClaimDesk's sign-in window doesn't appear within 60 seconds, ClaimDesk can't be found after sign-in, or the program path isn't set. | Run log `[ERROR] Clinic ...` lines, then "left Pending after 3 attempts". | System | Bot, then Ops / Dev | Wait for the 3 tries. Then open ClaimDesk by hand on this machine. If it doesn't open or is very slow, the problem is ClaimDesk or the machine. | ClaimDesk opens fine by hand, or the error names the program setting. Send the log file name. |
| E07 | "Remittance grid did not load". | The list of unposted remittances doesn't appear within 60 seconds after opening the clinic. | Run log `[ERROR]` and "left Pending" lines. | System | Bot, then Dev | Wait for the 3 tries. Open the clinic in ClaimDesk by hand to see whether the list loads at all. | It fails after 3 tries. Send the log file name and clinic name. |
| E08 | Any other `[ERROR] Clinic '<name>'` line, for example "list index out of range". | Something unexpected happened on the ClaimDesk screen while it was handling the clinic. | Run log. | System | Bot, then Dev | Wait for the 3 tries. | It fails after 3 tries. Send the log file name, clinic name and the error text. |
| E09 | Green row: "Payment method '<method>' - not posted". | The item's payment method is not ACH. | Report screen. | Business | Ops | Nothing. Post it by hand only if the client asks. | Never. |
| E10 | Grey row: "Check date <date> is after the cutoff - deferred". | The check date is "later" than the Settings cutoff, compared as text. | Report screen. | Business | Ops | Check that the date really is after the cutoff. If a date that is clearly earlier was deferred, or a later one posted, the date formats don't match (see section 3). | The formats match and the result is still wrong. |
| E11 | **Green** row: "Remittance Errors on posting". | After Post Payment, ClaimDesk shows its "Remittance Errors" screen within 5 seconds. The bot presses Cancel. | Report screen. | Business | **Ops** | **It was not posted, even though the row is green.** Open the item in ClaimDesk, read the error, and post it or follow up by hand. | Never, unless every item does this. |
| E12 | Red row: "Save button not found". | The payment screen has no Save button within 10 seconds. | Report screen. Batch left unfinished (E01). | System | Dev | Check the item in ClaimDesk: it should not be posted. | Always: this red row blocks the batch. Send the log file name, clinic name, check number and time. |
| E13 | Red row: "Payment not confirmed after Save". | It clicked Save, but ClaimDesk's "Payment saved" message did not appear within 20 seconds. **The payment may still have posted.** | Report screen. Batch left unfinished (E01). | System | **Ops**, then Dev | **Check the payment in ClaimDesk before doing anything else.** Note whether it posted. **Don't re-run it or post it by hand until you've checked; it can post twice.** | Always: ops can't correct the row, and it blocks the batch. Send the clinic name, check number and whether it posted. |
| E14 | The run stopped half-way. The log's last line is `[WATCHDOG] no progress for 300s - killing run`. ClaimDesk is often left open. Later clinics never ran. | No item was posted for 5 minutes. Signing in, opening a clinic and retries don't count as progress, so a slow clinic or a run of clinics with no items can trigger it. | Run log, last line. Clinics after that point have no Report rows. Batch left unfinished (E01). | System | Ops, then Dev | Close ClaimDesk. List the clinics that have no Report rows; they need a re-run once dev clears the batch. | Always (the batch must be cleared). Also when it happens on 2 runs in a row. Send the log file name and the list of clinics that didn't run. |
| E15 | The bot clicks in a window that isn't ClaimDesk (a SQL tool, a folder, the launcher), and then ClaimDesk closes. | It takes the first window whose title **contains** "ClaimDesk", which can be the wrong one. | Usually only seen on screen. The log then shows clinic errors like E04 or E05. | System | Ops, then Dev | Close every other window with "ClaimDesk" in its title, then re-run once the batch is cleared. | It happens with no other such window open. |
| E16 | The same check number appears more than once for a clinic on the Report screen. | After a clinic-level failure, the bot repeats the whole clinic, and items that are still in the unposted list get a new row. | Report screen. Run log shows `[AUTO-RERUN]` for that clinic. | System | Ops | Read the latest row for that check. If any copy says "Posted successfully", check ClaimDesk to make sure it posted only once. | A payment posted twice. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Any clinic-level failure (E03-E08, E15) | Closes ClaimDesk, starts it again, signs in again and repeats the whole clinic. No pause between tries. | 3 tries. Each try can take up to about 2 minutes of waiting. | Log: "left Pending after 3 attempts". The clinic gets no Report rows, the batch stays unfinished, and the bot moves to the next clinic. |
| Remittance Errors screen on one item | Presses Cancel. | Once, no retry. | Green "Remittance Errors on posting", next item. |
| Save button missing | None. | 10 seconds. | Red "Save button not found", next item. |
| No "Payment saved" message | None. | 20 seconds. | Red "Payment not confirmed after Save", next item. |
| Nothing posted for a long time | The watchdog kills the whole run. | 5 minutes. | Run ends, ClaimDesk may stay open, batch unfinished (E14). **The bot does not resume.** |
**How long to wait:** a clinic that is failing can take up to about 6-7 minutes over 3 tries. The watchdog may kill the run before the third try ends. If the log has not grown for more than 5 minutes and has no `[WATCHDOG]` line, escalate.
## 9. Known incidents
Dates are when the note was added to the code, not when the incident happened. In version 1.4.2, the three code "fixes" below are only notes: **the code itself didn't change.** Treat them as still open.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Noted 2026-10-23 (before 1.4.2) | Users stayed on 1.3.0 after an update. | The launcher kept the old program folder loaded until it was restarted. | After every update, fully close and reopen the launcher, then check the title bar (section 10). | Yes |
| Noted 2026-10-23 (before 1.4.2) | Red "Payment not confirmed after Save" rows that had really posted. | ClaimDesk saved more slowly than the bot waited, which was 10 seconds then and is 20 seconds now. | Always check in ClaimDesk first (E13). It can still happen when ClaimDesk takes more than 20 seconds. | Yes |
| Noted 2026-10-23 (before 1.4.2) | Clinics never processed; "not found in list" errors for half-names. | Clinic names were split at commas and semicolons. | One clinic per row (section 3). Names containing a comma still fail; report them to dev. | Partly |
| Noted 2026-10-23 (before 1.4.2) | The bot drove a SQL browser window, then closed the real ClaimDesk. | It picks any window whose title contains "ClaimDesk". **Still not fixed.** | Close other "ClaimDesk" windows before a run (E15). | Partly |
| 2026-09-01 run | "Wrong clinic opened" for a clinic with `&` in its name. During the retry the watchdog killed the run; the remaining clinics never ran and the batch stayed unfinished. | Sign-in and retries don't count as progress for the watchdog. The `&` may also have broken the name check. | E05, then E14. | Partly |
## 10. Releases and version check
- **How an update arrives:** through the ClaimDesk Poster launcher. The updater is not part of this code, so dev must confirm how updates are delivered.
- **How to confirm it applied:** the launcher title bar shows the new version. Today that is **1.4.2**.
- **If the version did not change:** fully close the launcher, open it again and check the title bar. If it still shows the old version, post in #rpa-support. Don't run the bot on an old version.
## 11. Escalating to dev
Post in Teams **#rpa-support**. Send:
- the exception ID from section 7 (for example E13)
- the batch number from the Report screen
- the time of the failure
- the run log **file name** from the `logs` folder, for example `run_20261023_093000.log`. Dev can open the folder; don't paste the log's contents.
- the clinic name and check number
- a screenshot of the Report screen
Never send passwords, the input Excel (it holds the ClaimDesk password), the bot's settings file, or patient data in chat.
## 12. Failure-class check
- F1 (missing prerequisite) covered in section 3 (ClaimDesk path set by dev) and E06. Fix-before-handover item 18.
- F2 (wrong window / focus) covered in sections 3 and 4 and E15. Fix-before-handover item 1.
- F3 (app slower or different than expected) covered in E05, E06, E07, E08 and E12, and section 8.
- F4 (input read differently) covered in section 3 input rules, E02, E04 and E10. Fix-before-handover items 10 and 11.
- F5 (marked done when nothing done) covered in section 6 (clinic Success) and fix-before-handover item 9.
- F6 (done but not confirmed) covered in E13 and fix-before-handover item 7.
- F7 (business exception shown as success) covered in section 6, E09 and E11. Fix-before-handover item 2.
- F8 (endless re-runs / watchdog) covered in E14, E16 and section 8. Fix-before-handover item 6.
- F9 (stopping the wrong way) covered in section 4. Fix-before-handover item 15.
- F10 (one stuck item blocks the queue) covered in E01, E12, E13 and E14. Fix-before-handover item 5.
- F11 (cause hidden from ops) covered in section 5. Fix-before-handover items 3, 4, 12 and 13.
- F12 (update didn't arrive) covered in section 10 and incident list. Fix-before-handover item 16.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The window is matched on any title *containing* "ClaimDesk". That also matches the launcher ("ClaimDesk Poster"), the login window and SQL tools. Commit d9a2483 says this was fixed, but only a comment was added. | The bot drives the wrong window and closes the real ClaimDesk. | bot.py:66-70 (line 68), bot.py:185 |
| 2 | "Rejected by app" and "Skipped" map to Completed (green). The internal status column isn't shown on the Report screen. | Unposted items look done. | bot.py:27-30, report_screen.py:6 |
| 3 | `tech_comments` is written but never shown. | The technical cause is invisible to ops, who have no database access. | bot.py:118, report_screen.py:6 |
| 4 | Clinic-level failures write no Report row. Clinic and batch status are not shown anywhere ops can reach. | Ops must read raw logs to find failed clinics and can't see whether a batch is finished. | bot.py:143-147, report_screen.py:11-13 |
| 5 | One red row, or one failed clinic, keeps the batch Pending for ever. Any Pending batch refuses every new upload. Ops has no way to edit or clear it. | One item stops all posting until dev edits the database. | bot.py:150-154, bot.py:167-168 |
| 6 | The heartbeat fires only when an item is posted. Sign-in, clinic retries and clinics with no items don't count, so the watchdog can kill healthy runs. The kill skips cleanup (ClaimDesk is left open) and leaves the batch Pending, with no resume. | Half-finished runs and a blocked queue. | bot.py:36-49, bot.py:137, bot.py:145-146 |
| 7 | After a Save timeout, the bot doesn't re-check the grid. | Red rows that actually posted, and a double-posting risk. | bot.py:117-118 |
| 8 | Sign-in success isn't checked. | A wrong password shows as "Clinic not found" for every clinic. | bot.py:80-83 |
| 9 | Clinic marked Success even with zero items or all-red items. The wrong-clinic check runs instantly after the click, with plain text matching (`&`). | False Success; false "Wrong clinic opened". | bot.py:90-92, bot.py:140 |
| 10 | Clinic cells are split on `,` and `;`. A trailing comma creates an empty clinic. | Names containing a comma can never run; the doc only works around this. | bot.py:62, README.md:8 |
| 11 | The cutoff is compared as text, not as dates. | Wrong items deferred or posted when date formats differ. | bot.py:107 |
| 12 | Input errors and the "Pending batch exists" refusal aren't written to the log. | The cause is lost once the screen closes. | bot.py:54-58, bot.py:167-169 |
| 13 | `[RUN SUMMARY] batch N done` is logged even when the batch stays Pending. Errors are logged as `[ERROR] [ERROR]`. | "Done" misleads ops. | bot.py:144, bot.py:180 |
| 14 | A clinic retry adds new Report rows for items already logged. `taskkill /im ClaimDesk.exe` kills every ClaimDesk on the machine. | Duplicate rows; a person's ClaimDesk session is closed. | bot.py:136-139, bot.py:146 |
| 15 | There's no Stop button and no confirmed safe way to stop. The launcher code isn't in this repo. | Ops can't stop a run safely. | bot.py:157-180 (launcher missing) |
| 16 | The launcher/updater isn't in the repo. The 1.3.0 fix (d84ff92) is only a comment. | Users can stay on old versions without noticing. | report_screen.py:8, report_screen.py:14 |
| 17 | No alerting: `ALERT_SMTP` is defined but never used. | Nobody is told when a run fails or is killed. | .env (key name only), bot.py (not referenced) |
| 18 | Database tables aren't created by the code. The ClaimDesk path missing from the settings file fails every clinic with a cryptic message. | A new machine fails in confusing ways. | bot.py:20, bot.py:74, bot.py:166 |
| 19 | Security: `.env` is committed to git, including an unused password-fallback key. A committed log contains patient-identifying data. The ClaimDesk password sits in plain text in the input Excel. | Leak of credentials and PHI. | .env, logs/run_20260901_101500.log:3, bot.py:56 |
| 20 | The committed log contains messages the current code never writes (for example "could not bring window to foreground"). | Ops can't map those log lines to this doc. Confirm which build is in production. | logs/run_20260901_101500.log:5 |
tests/evals/results/handoff/h-desktop-poster-screens-r1.json
+34
View File
@@ -0,0 +1,34 @@
{
"case": "h-desktop-poster/screens",
"rep": 1,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"comma rule before a run": true,
"how to stop": true,
"not-confirmed row: check ClaimDesk first": true,
"hidden tech comments flagged": true,
"Pending batch blocks a new upload": true
},
"cost_usd": 1.7118525999999998,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-desktop-poster-screens-5m9n9yvo\\project",
"reply": "I wrote `HANDOVER.md` in the repo root, the only file I changed. I never ran the bot, and I did not read the `.env` values.\n\n| Item | Result |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 19 (E01\u2013E19) |\n| Failure classes | All 12 covered, none marked not applicable |\n| Fix before handover | 22 items |\n\n**Ops-ku periya risk items (Fix before handover-la top):**\n- \"Rejected by app\" and \"Skipped\" rows are shown green as Completed, aana payment post aagala.\n- Oru clinic 3 attempts-um fail aana, Report screen-la row-e varaathu. Log mattum thaan record, adhu ops-ku access illa.\n- Oru red row, illa oru failed clinic irundhaale batch Pending-la nikkum. Adhu next upload-a block pannum. Clear panna function illa, so ovvoru time-um dev venum.\n- Rendu \"fix:\" commits code-la comment mattum thaan add pannirukku: wrong-window grab (7d468fa) and the 1.3.0 update issue (d840396). Window grab code innum maaralai, so that incident can happen again.\n- `.env` (password key irukku) and patient data irukura oru run log rendum git-la committed.\n\n**Also noticed:** the history scan couldn't open the git diffs, so it judged those \"fix:\" commits only from the current code. Dev `git show` panni confirm pannanum. Launcher and update code repo-la illa, so section 10 is partly from the commit note only.\n\nQUESTIONS: Escalation: who ops calls for a dev issue, and how (ticket, chat channel) | Ops access: which of the bot's places ops can open (app screens, output folder, log folder, database) | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear",
"stderr": ""
}
tests/evals/results/handoff/h-desktop-poster-screens-r1.md
+226
View File
@@ -0,0 +1,226 @@
# ClaimDesk Poster - ops handover
Version 1.4.2 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
The bot posts remittance (insurance) payments into the **ClaimDesk** desktop app one clinic at a time. Each run starts from the **ClaimDesk Poster** launcher window and works from one input Excel file, which has three sheets:
- **Login:** the ClaimDesk username and password
- **Clinics:** the clinic names
- **Settings:** the cutoff date
For every clinic, the bot does five things:
1. signs in to ClaimDesk;
2. opens that clinic;
3. reads the list of unposted payments;
4. posts every ACH payment dated on or before the cutoff;
5. writes one row per payment to the launcher's **Report screen**.
It writes no output file. The Report screen is the only result ops can see. It drives one app, **ClaimDesk**, and you must not touch ClaimDesk while the bot is running.
## 2. How it works
```
ClaimDesk Poster launcher (the window you use; title bar shows the version)
└─ starts → Bot process ──drives──> ClaimDesk desktop app
│ (do not touch while it runs)
├─ writes → the bot's database (one row per payment, plus batch and clinic status)
│ └─ read by → Report screen in the launcher
└─ writes → a run log in the bot's logs folder (dev only; ops has no access)
```
- **The launcher** is where you start a run and read the Report screen.
- **The bot process** does the work. It has no window of its own.
- **ClaimDesk** is the app the bot clicks through. The bot opens it and closes it again for every clinic.
- **The database** stores every result. Ops sees part of it on the Report screen; the rest is only available to dev.
- **The run log** records clinic-level failures and restarts. Only dev can read it.
**One run, step by step:**
1. **Refuse to start** if an earlier batch is still Pending (E01).
2. **Read the input Excel.** One batch is created, and every clinic in it starts as Pending.
3. **Process each clinic in order.** For each one the bot:
1. starts ClaimDesk and signs in;
2. opens the clinic;
3. waits for the payments list.
4. **Handle each payment in the list:**
- not ACH → skipped;
- dated after the cutoff → deferred;
- otherwise → the bot clicks Post Payment, then Save, then waits for "Payment saved".
5. **Close ClaimDesk** and mark the clinic done. If any step in 3 or 4 failed, the clinic is tried again, up to 3 times in total.
6. **Mark the batch Completed** only when there are no red rows and no clinic left undone. Otherwise the batch stays **Pending**, and that blocks the next upload (E01).
## 3. Before a run
- [ ] **The previous batch is Completed.** If it is still Pending, the new run will not start (E01) and dev must clear it first.
- [ ] **No other ClaimDesk is open on this machine**, not even one you opened yourself. The bot force-closes every ClaimDesk window after each clinic, so any work open in it will be lost.
- [ ] **No other window has "ClaimDesk" in its title.** That includes a SQL or database browser, a file explorer, or an email with that title. The bot may grab that window by mistake (E17).
- [ ] **The input Excel has the sheets Login, Clinics and Settings**, spelled exactly like that (E02).
- [ ] **Login sheet:** the username is in cell A2 and the password in B2. Before the run, check that they still work by logging in to ClaimDesk by hand. The bot cannot detect a wrong password (E05).
- [ ] **Clinics sheet: one clinic per row** in column A, starting at row 2. Spell each name exactly as ClaimDesk shows it.
- **Bad:** `North Clinic, South Clinic` in one cell.
- **Good:** `North Clinic` in A2 and `South Clinic` in A3.
- **Never use a comma or a semicolon in a cell.** The bot cuts the name at that point, so a clinic whose real name contains a comma cannot be processed at all. Post those clinics by hand (E07).
- A clinic with **"&"** in its name may fail as "wrong clinic" (E08). Watch these clinics on the Report screen.
- [ ] **Settings sheet:** put the cutoff date in cell B1. Payments dated after it are deferred. See E12 for a caution about date formats.
- [ ] **The launcher title bar shows version 1.4.2** (see section 10).
## 4. During a run
- **Don't use the mouse or keyboard on this machine.** Don't lock the screen and don't minimise ClaimDesk either. The bot clicks on screen; if another window comes to the front, its clicks land in the wrong place (E18).
- **Don't open ClaimDesk** or any window with "ClaimDesk" in its title.
- **To stop the bot:** the bot has no Stop button.
- **Closing ClaimDesk does not stop it.** The bot treats that as a failure and opens ClaimDesk again, up to 3 times per clinic.
- Ask dev before stopping a run. Any stop leaves the batch **Pending**, and dev then has to clear it before the next upload (E01).
- **When a run looks stuck:** the Report screen gets no new rows for 5 minutes and ClaimDesk sits idle.
- The bot ends the whole run by itself after **5 minutes with no new payment row** (E16). Wait until then.
- That stop is a kill, not a recovery. The batch stays Pending and dev must step in.
## 5. Where to look when something goes wrong
Ops can open only the launcher's screens. This version writes **no output file** and nothing to the output folder.
| Question | Where to look |
|---|---|
| Did the run finish? | The Report screen gets no more new rows and ClaimDesk is closed. The batch status (Pending or Completed) is **not shown to ops**, so if in doubt, raise a ticket. |
| Which payments failed? | **Report screen** for the batch. Look at the File Status column: red **Error** rows. |
| Which payments were *not* posted although green? | Report screen, green **Completed** rows. Read the Comments column: anything other than "Posted successfully" was **not** posted (E11, E13). |
| Why did one payment fail? | Report screen, **Comments** column. See section 6 and the catalogue. |
| A clinic from the input has **no rows at all** | Ops cannot see why. Possible causes: <br>- the clinic failed 3 times (E03-E10); <br>- the run was killed before it got there (E16); <br>- the clinic had nothing to post. <br>Raise a ticket and give the clinic name. |
| Did the bot already retry? | Not visible to ops; only the run log shows it. Assume it already tried each failing clinic 3 times. |
| Which version is running? | The launcher title bar. |
## 6. Statuses
What the Report screen shows: the **File Status** colour and the **Comments** text.
| File Status - Comments | Colour | Meaning | What ops does |
|---|---|---|---|
| Completed - "Posted successfully" | green | Payment posted and saved in ClaimDesk | Nothing |
| Completed - "Remittance Errors on posting" | green | **Needs follow-up. NOT posted.** ClaimDesk showed its Remittance Errors screen and the bot pressed Cancel. | Review and post it in ClaimDesk by hand (E13) |
| Completed - "Payment method '<method>' - not posted" | green | **Not posted on purpose**, because the method is not ACH | Nothing, unless the client wants it posted (E11) |
| Deferred - "Check date <date> is after the cutoff - deferred" | grey | Not posted. Its date is after the cutoff in Settings B1. | A later run picks it up (E12) |
| Error - "Save button not found" | red | Not posted. The payment screen had no Save button. | E14 |
| Error - "Payment not confirmed after Save" | red | **Might have posted.** Save was clicked, but "Payment saved" did not appear within 20 s. | **Check in ClaimDesk first** (E15) |
| Pending | amber | Not written by this version. If you see it, something unexpected happened. | Raise a ticket |
Statuses the bot stores but **does not show** on the Report screen (dev can read them):
| Item | Values | Meaning |
|---|---|---|
| Payment (detail) | Posted, Rejected by app, Skipped, Deferred, Error | "Rejected by app" and "Skipped" both appear on screen as green Completed |
| Clinic | Pending, Success | Success = the bot got through the clinic's list, **even if some rows are red or there were no payments at all** |
| Batch | Pending, Completed | Pending blocks the next upload (E01) |
| Technical comment | (empty), "RemitErrors screen shown", "Save button missing on payment screen", "no confirm text within 20s" | Dev detail for E13, E14 and E15 |
## 7. Exception catalogue
When a run fails, check the Report screen first. The **What you see** column describes the screen or ClaimDesk. In **Where it is recorded**, "run log" means the record is for dev only.
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | A new upload does nothing: no Report rows appear and ClaimDesk never opens | When it starts, the bot checks for an earlier batch still Pending and stops if it finds one. The message is "A Pending batch exists - finish or clear it before a new upload". | The bot's console output only. It is not in the log or on the Report screen. | System | Dev | Raise a ticket asking dev to clear the Pending batch. Ops cannot clear it. | Always. Send the date and start time of the previous run and the batch number shown in the launcher. |
| E02 | A new upload does nothing: no Report rows appear and ClaimDesk never opens | Before creating a batch, the bot reads the Excel and stops if any of these is missing: <br>- the Login, Clinics or Settings sheet; <br>- the username in Login A2; <br>- the password in Login B2. <br>Messages include "'Login' sheet not found" and "UserName/Password missing in Login sheet (row 2)". | The bot's console output only | Business | Ops | Check the three sheet names and Login A2/B2. Fix them and run again. No batch was created, so nothing is blocked. | The file looks right and it still does nothing. Send the input file **without the password**. |
| E03 | No clinic gets any Report rows, and ClaimDesk never opens | The setting that points to the ClaimDesk program is missing or wrong, so each clinic fails 3 times | Run log | System | Dev | Raise a ticket. Afterwards the batch is Pending (E01). | Always. Send the run start time. |
| E04 | ClaimDesk opens but stays on its start screen. The clinic has no Report rows. | The bot waits **60 s** for the "ClaimDesk Login" window; then the message is "Login window not found" | Run log | System | Bot, then Dev | Wait. The bot tries the clinic 3 times. | The clinic still has no rows after the run. Send the run start time and the clinic name. |
| E05 | Clinics have no Report rows. If you were watching, ClaimDesk showed a login error. | **Not detected.** A wrong or expired password shows up later as E06 or E07. | Run log, as E06 or E07 | Business | Ops | Log in to ClaimDesk by hand with the Login sheet details. If they fail, correct them. A dev must then clear the batch (E01). | The details work by hand but clinics still get no rows |
| E06 | The clinic has no Report rows | Right after signing in, the bot looks for a window whose title contains "ClaimDesk"; if none is found, the message is "ClaimDesk window not found" | Run log | System | Bot, then Dev | Wait for the 3 attempts | The clinic still has no rows after the run. Send the run start time and the clinic name. |
| E07 | The clinic has no Report rows | The bot waits **10 s** for the clinic name to appear as a link in ClaimDesk; then the message is "Clinic '<name>' not found in list". <br>Usual causes: <br>- a spelling mismatch; <br>- a comma in the name, which cut it in two; <br>- the clinic not being on this login; <br>- a wrong password (E05). | Run log | Business | Ops | Compare the spelling in the Excel with ClaimDesk, character for character. Make sure each clinic is on its own row with no comma or semicolon. A clinic whose real name contains a comma must be posted by hand. | The spelling matches exactly and it still fails. Send the clinic name and the run start time. |
| E08 | The clinic has no Report rows | After clicking the clinic link, the bot checks that the clinic name appears in ClaimDesk's title. If not, the message is "Wrong clinic opened, leaving Pending: <name>". Clinics with "&" in the name are suspected to fail here every time. | Run log | System | Dev | Post that clinic by hand for now | Always. Send the clinic name and the run start time. |
| E09 | The clinic has no Report rows | The bot waits **60 s** for the payments list; then the message is "Remittance grid did not load" | Run log | System | Bot, then Dev | Wait for the 3 attempts | It happens on 2 runs in a row. Send the clinic name and the run start time. |
| E10 | The clinic has no Report rows (or only some), and the batch stays Pending | The result of E03-E09 after 3 failed attempts on one clinic. The message is "Clinic '<name>' left Pending after 3 attempts". | Run log | System | Dev | Raise a ticket. Later clinics are still processed. | Always. Send the clinic name and the run start time. |
| E11 | Green Completed - "Payment method '<method>' - not posted" | The payment method in ClaimDesk's list is not exactly "ACH". The match is case-sensitive, so "Ach" is skipped too. | Report screen | Business | Ops | Nothing, unless this payment should have been posted. In that case post it by hand. | The method is ACH but it was still skipped. Send the clinic name and the check number. |
| E12 | Grey Deferred - "Check date <date> is after the cutoff - deferred" | The payment date is compared with Settings B1. The compare is done **as text**, not as a date, so different date formats can defer the wrong payments or post ones that should wait. | Report screen | Business | Ops | Check that the date really is after the cutoff. A later run picks it up. | A payment dated *before* the cutoff was deferred, or one dated after it was posted |
| E13 | Green Completed - "Remittance Errors on posting" | After Post Payment, ClaimDesk showed its "Remittance Errors" screen within 5 s. The bot pressed Cancel. **Nothing was posted.** | Report screen | Business | Ops | Open the payment in ClaimDesk, review the errors, and post it by hand or follow the client's process | Many rows in the same clinic show this |
| E14 | Red Error - "Save button not found" | On the payment screen the Save button did not appear within 10 s. There is no retry for this payment. | Report screen | System | Ops, then Dev | Check the payment in ClaimDesk. Post it by hand if needed. The batch now stays Pending (E01). | Always, so dev can clear the batch. Send the clinic name, the check number and the run start time. |
| E15 | Red Error - "Payment not confirmed after Save" | The bot clicked Save and waited **20 s** for "Payment saved". It did not see it, but **the payment may still have posted**. | Report screen | System | **Ops** | **Check this check number in ClaimDesk before doing anything else.** <br>- **If it posted:** do not post it again; raise a ticket so dev can correct the row. <br>- **If it did not post:** post it by hand, or let a later run pick it up. <br>Re-running without checking can post it twice. | Always, because the red row keeps the batch Pending (E01). Send the clinic name, the check number, the run start time, and whether it was posted in ClaimDesk. |
| E16 | No new Report rows for 5 minutes, then the bot stops. ClaimDesk is left open and idle, and later clinics have no rows. | No new payment row for **300 s**, so the bot kills the whole run. Signing in, opening a clinic and its retries do **not** count as progress, so a slow or failing clinic, or several clinics in a row with no payments, can trigger it. Message: "[WATCHDOG] no progress for 300s - killing run". | Run log (last line) | System | Dev | Close ClaimDesk. Raise a ticket. The batch is Pending, so the next upload is blocked (E01). | Always. Send the run start time and the last clinic with rows. |
| E17 | ClaimDesk suddenly closes while another "ClaimDesk" window stays open, or the bot clicks inside the wrong window | The bot uses the first window whose title *contains* "ClaimDesk". That can be: <br>- a database tool; <br>- the launcher itself ("ClaimDesk Poster"); <br>- the ClaimDesk login window. <br>After every clinic it force-closes **all** ClaimDesk windows. | Run log, usually as E06/E07 | System | Ops, then Dev | Before the next run, close every other window with "ClaimDesk" in its title (section 3) | It happens again with no other such window open. Send the run start time. |
| E18 | Clinics fail with no clear reason while someone was using the PC, or the screen was locked | Windows refused to bring ClaimDesk to the front, so the bot's clicks missed. The current code has no message of its own for this; older logs showed "could not bring window to foreground". | Run log | System | Ops | Keep hands off the machine and the screen unlocked (section 4). A dev must clear the batch (E01). | It happens with nobody using the machine |
| E19 | The same check number appears twice or more on the Report screen for one clinic | When a clinic is retried, the bot reads the clinic's list again and writes the Skipped, Deferred and Rejected rows a second time | Report screen | System | Ops | Count each check number once. A duplicate grey or green row is not a second payment. | A check number shows "Posted successfully" twice. Send the clinic name and the check number. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Any clinic-level failure (E03-E09, E17, E18) | Force-closes ClaimDesk, starts it again, signs in and redoes the whole clinic | 3 attempts per clinic. Each attempt can take up to about 2 minutes: 60 s for login, 10 s for the clinic, 60 s for the list. | Clinic left Pending (E10), batch stays Pending, next clinic starts |
| Payment-level failure (E14, E15) | **No retry** | - | Red Error row, next payment |
| No progress | Kills the whole run (not a recovery) | 5 minutes without a new payment row | Run ends, ClaimDesk left open, batch Pending (E16) |
| A Pending batch from an earlier run | **Nothing.** There is no resume and no clear. | - | Every new upload is refused until dev clears it (E01) |
Rules of thumb:
- Wait at least **7 minutes** after the last new Report row before treating a run as dead.
- **Any red row, any clinic without rows, or any killed run means dev must clear the batch** before the next upload.
## 9. Known incidents
Newest first. The repo history does not give exact dates.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 1.3.0 → 1.4.x | After an update the title bar still showed 1.3.0 | The launcher kept running from the old folder until it was restarted | After every update, close the launcher completely, reopen it, and check the title bar (section 10) | Yes |
| Before 1.4.2 | Red "Payment not confirmed after Save" rows for payments that had really posted | ClaimDesk took longer than the wait to show "Payment saved" | The wait was raised from 10 s to 20 s. A slow save can still cause it, so **always check in ClaimDesk first** (E15). | Partly |
| Before 1.4.2 | Some clinics were never processed; "not found in list" | Clinic names with commas were cut into pieces | One clinic per row, with no commas or semicolons. A real name containing a comma still cannot be processed, so post it by hand (E07). | Partly |
| Before 1.4.2 | The bot drove a database tool and then closed the real ClaimDesk | That tool's title contained "ClaimDesk". **The code still matches window titles the same way**, so this can happen again. | Close every other "ClaimDesk" window before a run (E17) | Yes |
| 2026-09-01 run | Run stopped mid-way; batch Pending; the next upload was refused | The bot's 5-minute no-progress stop (E16) | Raise a ticket (E16, E01) | Partly |
| 2026-09-01 run | A clinic with "&" in the name failed as "wrong clinic" | Suspected "&" title mismatch (not confirmed) | Post it by hand (E08) | Partly |
| 2026-09-01 run | Clicks missed while the PC was in use or locked | Windows blocked the bot from bringing ClaimDesk to the front | Hands off, screen unlocked (E18) | Yes |
## 10. Releases and version check
- **How an update arrives:** through the ClaimDesk Poster launcher. The update code is not in this repo, so dev must confirm the exact steps.
- **How to confirm it applied:** **fully close the launcher and open it again.** The title bar must show the new version, currently **1.4.2**.
- **If the version did not change:** close the launcher again, check that no launcher window is left open, and reopen it. If it is still old, raise a ticket with the version shown and the machine name. Do not run the bot on an old version.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue and include:
- the exception ID from section 7 (for example E15);
- the date and start time of the run, so dev can find the run log;
- the batch number, if the launcher shows it;
- the clinic name and the check number;
- a screenshot of the Report screen;
- for E15: whether the payment shows as posted in ClaimDesk.
Never send: passwords (including the Login sheet), the bot's settings file, or patient details in chat or in the ticket text.
## 12. Failure-class check
- **F1** (missing prerequisite): covered in E02, E03 and section 3. Missing database setup is in section 13, item 18.
- **F2** (wrong window or focus): covered in E17, E18, and the rules in sections 3 and 4. Also section 13, items 6 and 7.
- **F3** (target app slower than expected): covered in E04, E06-E09, E14 and section 8 (all timeouts). No screen-size rule is needed, because clicks are found by element, not by position.
- **F4** (input read differently): covered in section 3 input rules (comma/semicolon, "&", A2/B2, B1), E07, E11 and E12. Also section 13, items 8, 11 and 12.
- **F5** (marked done when nothing was done): there is no guard. A clinic is marked done with 0 payments or with red rows. See section 13, item 14.
- **F6** (action happened but not confirmed): covered in E15 (check ClaimDesk first). Also section 13, item 10.
- **F7** (business exception shown as success): covered in section 6 (green rows that need follow-up), E11 and E13. Also section 13, item 1.
- **F8** (re-runs and watchdog): covered in E16 and section 8. Retries are capped at 3 per clinic, so there is no endless loop. The watchdog problem is in section 13, item 5.
- **F9** (stopping the wrong way): covered in section 4. There is no Stop button and closing ClaimDesk does not stop the bot. See section 13, item 17.
- **F10** (one stuck item blocks the queue): covered in E01 and section 8. Any red row or failed clinic blocks the next upload. See section 13, items 3 and 4.
- **F11** (cause hidden from ops): covered in section 5. Clinic failures, retries, the batch status and the detailed status are hidden. See section 13, items 2, 3 and 15.
- **F12** (update didn't arrive): covered in section 10. The database lives on each machine, so nothing shared is overwritten at start. The launcher code is missing from the repo; see section 13, item 19.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | "Rejected by app" (Remittance Errors, cancelled) and "Skipped" (not ACH) are stored as Completed and shown green | Ops reads green as posted; unposted payments are missed | `bot.py:27-33` STATUS_MAP |
| 2 | The Report screen hides the detailed status and the technical comment | Ops can't tell Rejected from Posted except by reading the comment | `report_screen.py:6` COLUMNS |
| 3 | A clinic that fails all 3 attempts writes **no row** to the Report screen, only a log line; ops has no log access | A failed clinic is invisible; it looks the same as "no payments" | `bot.py:143-147` |
| 4 | A Pending batch blocks every new upload. There is no clear or resume function, and the refusal message goes to the console only. Any red row also keeps the batch Pending, even a payment that really posted (E15), and ops can't edit rows. | Every red row or failed clinic needs a dev before the next run | `bot.py:150-154`, `bot.py:167-168`, `report_screen.py:11-13` |
| 5 | The watchdog only counts a new payment row as progress. Sign-in, opening a clinic and retries don't count. It exits without cleanup, so ClaimDesk stays open and the batch stays Pending. | Slow clinics, or several empty clinics in a row, kill the whole run | `bot.py:44-49`, `bot.py:137` |
| 6 | Window match is "title contains ClaimDesk", first hit wins. It can also match the launcher ("ClaimDesk Poster") and the login window right after Sign in. Commit 7d468fa says fixed, but only a comment was added. | The bot drives the wrong window; the incident can recur | `bot.py:66-70`, `bot.py:83`, `bot.py:185` |
| 7 | After every clinic, `taskkill /im ClaimDesk.exe /f` kills every ClaimDesk process | A user's own ClaimDesk session is closed and their work is lost | `bot.py:146` |
| 8 | Clinic cells are split on "," and ";" and blank pieces are not removed | Clinics whose real names contain a comma can never be processed | `bot.py:60-62` |
| 9 | A wrong password is not detected after Sign in | It shows up as a misleading "clinic not found" | `bot.py:82-83` |
| 10 | When "Payment saved" times out, the bot does not re-check ClaimDesk | Posted payments show red; risk of posting twice | `bot.py:117-118` |
| 11 | The cutoff is compared as text (`check_date > str(cutoff)`) | Wrong payments can be deferred or posted when date formats differ | `bot.py:107` |
| 12 | The payment method must be exactly "ACH" (case-sensitive) | "Ach" and similar values are skipped | `bot.py:105` |
| 13 | A clinic retry re-walks the list and writes duplicate Report rows | Confusing duplicates on screen (E19) | `bot.py:130-139` |
| 14 | A clinic is set to Success with 0 payments, or with red rows | Nothing proves the bot was on the right, populated list | `bot.py:140` |
| 15 | Input sheet errors are not logged. A missing Clinics or Settings sheet raises a raw error with no friendly message. | Ops sees "nothing happened" | `bot.py:52-63`, `bot.py:169` |
| 16 | A missing ClaimDesk path setting causes a raw error for every clinic. The alert mail setting in `.env` is never used, so nobody is notified of anything. | Silent failures | `bot.py:74`, `.env` |
| 17 | There is no Stop button or clean stop. The launcher is not in the repo, so nobody can confirm whether closing it ends the bot process. | Ops has no safe way to stop a run | `bot.py:157-180`; launcher code missing |
| 18 | The code never creates the database tables | A new machine fails at the first query | `bot.py:166-167` |
| 19 | The launcher and update code are not in the repo. Commit d840396 (users stuck on 1.3.0) shows only a comment. | The update path can't be documented or checked | `report_screen.py:14` |
| 20 | Clinic names with "&" are suspected to fail the title check every time (not confirmed) | The clinic is never posted (E08) | `bot.py:91-92` |
| 21 | `.env` (with a password key) and a run log containing patient data are committed to git. The ClaimDesk password also sits in plain text in the input Excel. | Credential and PHI exposure | `.env`, `logs/run_20260901_101500.log:3`, `bot.py:56` |
| 22 | The committed log doesn't match the current code's messages | The incident evidence comes from a different build | `logs/run_20260901_101500.log` |
tests/evals/results/handoff/h-desktop-poster-screens-r2.json
+34
View File
@@ -0,0 +1,34 @@
{
"case": "h-desktop-poster/screens",
"rep": 2,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"comma rule before a run": true,
"how to stop": true,
"not-confirmed row: check ClaimDesk first": true,
"hidden tech comments flagged": true,
"Pending batch blocks a new upload": true
},
"cost_usd": 1.5344808,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-desktop-poster-screens-qbv9h8s2\\project",
"reply": "I've written `HANDOVER.md` in the repo root. Itha la, ops-ku code theriyama oru check fail aana: enna nadanthuchu, enga record aagum, bot retry pannucha, ops fix pannanuma illa dev-a koopidanuma - ellam irukku. Bot-a naan run pannala, code-um maathala.\n\n| Item | Result |\n|---|---|\n| Sections | 13 (template order) |\n| Exceptions catalogued | 18 (E01\u2013E18) |\n| Failure classes | 12/12 covered, 0 not applicable |\n| Statuses | 7 on the Report screen + 4 clinic/batch statuses that ops can't see |\n| Fix before handover | 20 items for dev, each with file:line |\n\n**Ops-ku romba mukkiyamana dev fixes:**\n- Oru red row irundha kooda, antha machine la next run start aagadhu. Dev database-a edit pannura varaikkum ippadiye nikkum. Clear pannura tool illa.\n- \"Remittance Errors on posting\" rows green-a theriyum, aana avanga post aagave illa.\n- 5 nimisham pudhu check row varalanna watchdog run-a kill pannidum. Login, retry, empty clinic ithellam progress-a count aagadhu.\n- Oru clinic full-a fail aana Report screen la row-e varadhu. Logs, database ops-ku access illa, adhanala ops-ku antha reason theriyave theriyadhu.\n- Commit 34cae4a \"fixed\"-nu sollum, aana bot innum title la \"ClaimDesk\" irukka mudhal window-a eduthukkum. Athu Poster launcher window-avum irukkalam.\n- Stop button illa. Launcher code repo la illa, adhanala Poster window-a close panna bot nikkuma-nu confirm panna mudiyala.\n- Sample log `logs/run_20260901_101500.log` la oru patient name irukku. Scrub pannanum.\n\n**Also noticed:** Git commands-a denied aachu, adhanala history `.git/logs/HEAD` file-ayum code la irukka commit comments-ayum vachu thaan padichen. Commit diffs-a naan paakkala.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role) and how (ticket, chat channel) | Ops access: which of the bot's places ops can open - the app's screens, the output folder, the log folder, the database | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear",
"stderr": ""
}
tests/evals/results/handoff/h-desktop-poster-screens-r2.md
+194
View File
@@ -0,0 +1,194 @@
# ClaimDesk Poster - ops handover
Version 1.4.2 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Posts remittance (insurance payment) checks into the **ClaimDesk** desktop app, one clinic at a time, from an input Excel file with three sheets: **Login** (ClaimDesk username and password), **Clinics** (the clinics to work) and **Settings** (a cutoff date). For every clinic it logs in to ClaimDesk, opens the clinic, and walks its list of unposted remittances. ACH payments dated on or before the cutoff are posted and saved; other payment methods are skipped and later dates are deferred. Every check it touches gets a row on the **Report screen** of the ClaimDesk Poster window. It runs when ops start it from the ClaimDesk Poster window; it does not run on a schedule. It drives one app: **ClaimDesk**.
## 2. How it works
```
ClaimDesk Poster window (what you use: start a run, Report screen)
└─ starts → Posting bot ──drives──> ClaimDesk desktop app
│ (do not touch while it runs)
├─ writes → bot database (one row per check, clinic and batch status)
└─ writes → a run log file (dev only)
```
- **ClaimDesk Poster window** - where you pick the input Excel, start a run and read the Report screen. Its title bar shows the version.
- **Posting bot** - the automation itself. It runs in the background with no window of its own.
- **ClaimDesk** - the app the bot clicks through. The bot opens it, logs in, and force-closes it after every clinic.
- **Bot database** - stores the result for every check, plus clinic and batch status. Ops see part of it through the Report screen only.
- **Run log** - one file per run with technical messages. Ops cannot open it; dev reads it.
**One run, step by step:**
1. The bot checks that no earlier batch is still open (Pending). If one is, it refuses to start.
2. It reads the input Excel and creates a new batch with one entry per clinic.
3. For each clinic: start ClaimDesk → log in → click the clinic's link → wait for the remittance list.
4. For each check in the list: skip it (not ACH), defer it (after the cutoff), or click Post Payment → Save → wait for "Payment saved". One Report-screen row per check.
5. ClaimDesk is force-closed. If anything in step 3-4 broke, the whole clinic is retried from login (up to 3 tries).
6. After the last clinic the batch is marked Completed - **only if** no check ended red and every clinic finished.
## 3. Before a run
- [ ] ClaimDesk is installed and the bot's settings on this machine point to it. If not, every clinic fails at login and no rows appear (E04).
- [ ] The input Excel has exactly these sheet names: **Login**, **Clinics**, **Settings**. A missing Clinics or Settings sheet makes the run stop at once with no rows (E03).
- [ ] **Login** sheet: username in A2, password in B2, both filled. Check they are correct - the bot never checks that the login worked (E05). Keep this file private: it holds the ClaimDesk password.
- [ ] **Clinics** sheet: **one clinic per row** in column A from row 2, spelled exactly as the link in ClaimDesk.
- Bad: `North Clinic, South Clinic` in one cell (or with `;`). Good: `North Clinic` in A2, `South Clinic` in A3.
- A clinic whose real name contains a comma (e.g. `Smith, Jones Clinic`) **cannot be run by this version** - the bot cuts it into two names that don't exist. Do those clinics by hand and tell dev (Fix #7).
- No trailing comma at the end of a cell.
- [ ] **Settings** sheet: cutoff date in B1. Leave B1 empty to post everything with no date check. After the run, spot-check a few Deferred and Posted rows against the cutoff - the date comparison is unreliable (E11).
- [ ] The last batch on this machine finished. If any row of the last run was red, the next run will refuse to start until dev clears it (E01, E15).
- [ ] The ClaimDesk Poster title bar shows version **1.4.2** (see section 10).
## 4. During a run
- **Hands off this machine.** No mouse, no keyboard. The bot clicks on screen positions, and your clicks can land in the wrong place.
- **Close every other window whose title contains "ClaimDesk"** before you start - database tools, Explorer folders, browser tabs, your own ClaimDesk session. The bot grabs the first window with "ClaimDesk" in its title and may work in the wrong one (E18).
- **Don't use ClaimDesk on this machine during a run.** After every clinic the bot force-closes *all* ClaimDesk windows, including one you opened yourself. Unsaved work in it is lost.
- **Don't close ClaimDesk to stop the bot.** The bot treats that as a failure and starts the clinic again from login.
- To stop the bot: this version has **no Stop button**, and it is not confirmed that closing the ClaimDesk Poster window stops it. If you must stop a run, raise an RPA Support ticket; until dev confirms the safe way, let the run finish (Fix #14).
- A run looks stuck when **no new rows appear on the Report screen for more than 5 minutes**. The bot ends the run by itself after 5 minutes with no new row (E16) - wait that long before raising a ticket.
## 5. Where to look when something goes wrong
Ops can open only the ClaimDesk Poster window (its Report screen and title bar), the bot's output files and the output folder. The bot writes **no output files** (no Excel, no screenshots) - the Report screen is the only result ops can read. Everything else below goes to dev.
| Question | Where to look |
|---|---|
| Did the run finish? | **Not shown on any ops screen in this version.** Signs it ended: ClaimDesk is closed and the Report screen has stopped growing. Whether the batch is Completed or still Pending is only visible to dev (Fix #4). |
| Which checks failed? | Report screen → pick the batch → File Status column red (Error). |
| Why did one check fail? | Report screen → the Comments column on that row. The technical reason is hidden; dev can read it (Fix #5). |
| Which clinic failed completely? | A clinic from your input with **no rows at all** on the Report screen. Either it had nothing to post, or the bot never got into it (E04-E09). Ops cannot tell which - check in ClaimDesk whether the clinic has unposted ACH checks; if it has, raise a ticket. |
| Did the bot already retry? | Not visible to ops. A clinic is always tried up to 3 times before it is given up. Repeated rows for the same check number are a sign of a retry (E17). |
| Which version is running? | ClaimDesk Poster title bar. |
## 6. Statuses
What you see on the Report screen (File Status + Comments):
| Status | Colour | Meaning | What ops does |
|---|---|---|---|
| Completed - "Posted successfully" | green | Payment posted and ClaimDesk showed "Payment saved". | Nothing. |
| Completed - "Remittance Errors on posting" | green | **NOT posted.** ClaimDesk showed its Remittance Errors screen and the bot pressed Cancel. Needs follow-up despite the green colour. | Review and post it by hand in ClaimDesk (E12). |
| Completed - "Payment method '<method>' - not posted" | green | Not ACH, so skipped on purpose. Not posted. | Nothing, unless the client wants non-ACH posted - then post by hand. |
| Deferred - "Check date <date> is after the cutoff - deferred" | grey | Dated after the cutoff; left unposted. A later run picks it up. | Spot-check the date against the cutoff (E11). |
| Error - "Save button not found" | red | Payment screen opened but had no Save button. Not posted. **Blocks the next run.** | E13. |
| Error - "Payment not confirmed after Save" | red | Save was clicked but "Payment saved" did not appear within 20 s. **It may have posted.** **Blocks the next run.** | Check in ClaimDesk first (E14). |
| Pending | amber | The screen supports this colour, but the current bot never writes it. | If you ever see it, raise a ticket. |
Statuses ops cannot see (dev reads them from the bot database):
| Status | Meaning |
|---|---|
| Clinic Success | The clinic's list was walked to the end. Set **even when some of its checks are red.** |
| Clinic Pending | The clinic failed 3 times, or the run was killed before it. It has no rows on the Report screen. |
| Batch Completed | No red rows and every clinic Success. |
| Batch Pending | Anything else. **A Pending batch stops every later run on this machine** until dev clears it. |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Run doesn't start; no new rows. The message (if the window shows it) reads "A Pending batch exists - finish or clear it before a new upload". | At start it finds an earlier batch still Pending (see E15, E16). | On screen only, if shown. Not in the log. | System | Dev | Raise a ticket. Ops has no way to clear a batch. | Always. Send: machine, time, the last batch number on the Report screen, and its red rows. |
| E02 | Run doesn't start; no new rows. Message "'Login' sheet not found" or "UserName/Password missing in Login sheet (row 2)". | It checks the Login sheet exists and A2/B2 are filled. | On screen only, if shown. | Business | Ops | Fix the Login sheet name and fill A2 and B2. Start again. | Sheet and cells are correct and it still fails. |
| E03 | Run doesn't start; no new rows; a short technical message naming "Clinics" or "Settings". | A sheet named Clinics or Settings is missing - the bot crashes with no friendly message. | On screen only, if shown. | Business | Ops | Rename the sheets exactly: Clinics, Settings. Start again. | Names are exact and it still fails. Send the input file **without the Login sheet**. |
| E04 | Every clinic has no rows; ClaimDesk never appears or never shows its login window. | It waits 60 s for the "ClaimDesk Login" window; if ClaimDesk can't be started or the window never appears, the clinic is retried. | Run log (dev): "Login window not found". Clinic left Pending. | System | Bot, then Dev | Check ClaimDesk opens by hand on this machine. If it does, raise a ticket - the bot's ClaimDesk setting on this machine is likely wrong. | All clinics fail. Send machine, time, batch number. |
| E05 | Every clinic has no rows; ClaimDesk opens and the login screen stays or shows an error. | The bot types the username and password and clicks Sign in **without checking it worked**. It then fails on the next step. | Run log (dev): "ClaimDesk window not found" or "Clinic ... not found in list". | Business | Ops | Check the username and password in the Login sheet by logging in to ClaimDesk by hand. Fix and start again (after dev clears the batch, E01). | Login works by hand and the bot still fails. |
| E06 | One clinic has no rows. | It waits 10 s for a link with exactly the clinic's name; if none, the clinic is retried, then given up. | Run log (dev): "Clinic '<name>' not found in list". Clinic left Pending. | Business | Ops | Compare the spelling in the Clinics sheet with the link in ClaimDesk, character by character (spaces, `&`, commas). One clinic per row. Fix, then re-run that clinic. | Spelling is exact and it still has no rows. |
| E07 | One clinic has no rows, though spelled correctly. | Right after clicking the link, the clinic's name is not in ClaimDesk's title. It checks without waiting, so a slow screen can trigger it. | Run log (dev): "Wrong clinic opened, leaving Pending: <name>". Clinic left Pending. | System | Bot, then Dev | Bot retries 3 times. Then raise a ticket. | Always after 3 tries. Send machine, time, batch number, clinic name. |
| E08 | One clinic has no rows; ClaimDesk was on the clinic but showed no list. | It waits 60 s for the remittance list; if it never appears, the clinic is retried. | Run log (dev): "Remittance grid did not load". Clinic left Pending. | System | Bot, then Dev | Bot retries 3 times. Check in ClaimDesk whether the clinic's list opens by hand. Then raise a ticket. | Always after 3 tries. Send machine, time, batch number, clinic name. |
| E09 | One clinic has no rows, or only some of its checks. | Any other unexpected problem in the clinic (e.g. an odd row in the list) restarts the clinic from login. | Run log (dev): "[ERROR] Clinic '<name>' : ..." then "[DB] Clinic '<name>' left Pending after 3 attempts". | System | Bot, then Dev | Wait for the run to end. Raise a ticket. | Always after 3 tries. Send machine, time, batch number, clinic name. |
| E10 | Green "Payment method '<method>' - not posted". | The check's payment method is not ACH. | Report screen. | Business | Ops (if needed) | Nothing - by design. Post by hand only if the client wants it. | Never. |
| E11 | Grey "Check date <date> is after the cutoff - deferred" - or checks posted that should have been deferred. | It compares the check date to the Settings cutoff **as text, not as a date**. Depending on how ClaimDesk shows dates, it can defer the wrong checks or never defer anything. | Report screen. | Business | Ops, then Dev | After a run with a cutoff, compare a few Deferred and Posted rows' dates with the cutoff. Deferred ones are retried on a later run. | Any row is on the wrong side of the cutoff. Send batch number and two example check numbers. |
| E12 | **Green** "Remittance Errors on posting". | After Post Payment, ClaimDesk showed its "Remittance Errors" screen within 5 s; the bot pressed Cancel. | Report screen (green, so easy to miss - filter on the Comments text). | Business | Ops | Open the check in ClaimDesk, resolve the remittance error, and post by hand. | ClaimDesk shows no error for it. |
| E13 | Red "Save button not found". | The payment screen opened but no Save button appeared within 10 s. Not posted. | Report screen. | System | Dev | Check in ClaimDesk that it is not posted. Raise a ticket - this row also blocks the next run (E15). | Always. Send batch number, check number, time. |
| E14 | Red "Payment not confirmed after Save". | It clicked Save and waited 20 s for "Payment saved"; it never appeared. **The payment may still have posted** (this happened before - see section 9). | Report screen. | System | **Ops**, then Dev | **Check the payment in ClaimDesk first. Do not post it by hand until you have.** If it posted: nothing more in ClaimDesk. If not: post by hand, or let the next run pick it up. Either way raise a ticket so dev can correct the row - ops cannot edit rows, and the red row blocks the next run (E15). | Always (to clear the row). Send batch number, check number, and whether it posted. |
| E15 | Yesterday's run had a red row; today's run won't start (E01). | A batch is only Completed when it has no red rows and every clinic finished. One red row, or one clinic given up, keeps it Pending for good; a Pending batch blocks all new runs. | Batch status (dev only). | Business | Dev | Handle the red rows (E13, E14) and clinics with no rows (E04-E09) first, then raise a ticket to clear the batch. | Always. Send batch number and the list of red rows / missing clinics. |
| E16 | The run stops halfway: ClaimDesk stays open, later clinics have no rows, nothing new for 5+ minutes. Next run won't start (E01). | A watchdog ends the run when no new check row has been written for 5 minutes. Logging in, waiting for screens, retries and clinics with nothing to post do not count as progress, so several slow or empty clinics in a row can trigger it. | Run log (dev): "[WATCHDOG] no progress for 300s - killing run". Batch left Pending. | System | Dev | Close ClaimDesk. Raise a ticket to clear the batch. Then re-run only the clinics that have no rows. | Always. Send machine, time, batch number, last clinic with rows. |
| E17 | The same check number appears twice in one batch (e.g. a red row and later a green one). | When a clinic is retried, rows already written are kept and the clinic is walked again. | Report screen. | System | Ops, then Dev | The latest row is the true result. Checks that posted drop out of ClaimDesk's unposted list, so they are not posted twice. If a stale red row remains, it still blocks the batch - raise a ticket. | A stale red row remains. Send batch number and check number. |
| E18 | The bot clicks in a database tool, Explorer, or its own Poster window instead of ClaimDesk; or your own ClaimDesk session closes. | It takes the first window whose title contains "ClaimDesk", and force-closes every ClaimDesk process after each clinic. | Nowhere. Usually shows up as clinics with no rows (E05-E09). | System | Ops | Before a run, close every other window with "ClaimDesk" in its title. Don't use ClaimDesk on this machine during a run. | It happens with no other such window open. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Anything goes wrong inside a clinic (login, clinic link, list, odd row, ClaimDesk closed) | Force-closes ClaimDesk, logs in again, starts the clinic from the top | 3 tries per clinic. One try can take up to about 2.5 minutes before it gives up. | Clinic given up (no rows / partial rows); next clinic. Batch stays Pending. |
| A single check fails (no Save button, save not confirmed) | **No retry.** Writes a red row. | - | Next check. The check stays unposted in ClaimDesk and is tried again on the next run. |
| ClaimDesk "Remittance Errors" screen | Clicks Cancel | Once | Green row "Remittance Errors on posting" - needs ops (E12). |
| No new check row for 5 minutes | Ends the whole run | 5 minutes | Run over. No restart, no resume. ClaimDesk left open. Batch Pending (E16). |
| Run stopped part-way | Nothing - there is no resume | - | Dev clears the batch; ops re-runs the missing clinics. |
**Waiting rule:** if the Report screen is still getting new rows, leave it. If nothing new for 5 minutes, the bot has ended or is about to - then act.
## 9. Known incidents
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 1.4.2 | Users still on 1.3.0 after an update was released | The launcher kept using the old program folder until it was restarted | After every update, fully close and reopen ClaimDesk Poster and check the title bar (section 10). | Yes |
| Before 1.4.2 | Red "Payment not confirmed after Save" rows that had really posted | ClaimDesk took longer than the bot's 10 s wait | Wait raised to 20 s. Still check in ClaimDesk before acting (E14). | Partly |
| 2026-09-01, batch 41 | Run stopped mid-way after a clinic retry; next upload refused | Watchdog ended the run after 5 min with no new check row (retries don't count as progress); batch stayed Pending | E16, then E01. Not fixed in code (Fix #3). | Partly |
| Before 1.4.2 | Clinics with a comma in the cell were "not found" and skipped every run | The bot splits clinic cells on commas | One clinic per row (section 3). Names that really contain a comma still can't run (Fix #7). | Yes |
| Before 1.4.2 | Bot worked in a database tool whose title contained "ClaimDesk", then closed the real ClaimDesk | Window picked by title text | Close other "ClaimDesk" windows before a run (section 4). Marked fixed, but the code still matches by title (Fix #6). | Partly |
## 10. Releases and version check
- How an update arrives: through the ClaimDesk Poster launcher. Its update code is not in this repo, so dev must confirm the exact steps.
- How to confirm it applied: **fully close ClaimDesk Poster, open it again**, and check the title bar shows the new version (today: 1.4.2). Without the restart the old version keeps running.
- If the version did not change: raise an RPA Support ticket with the machine name and the version shown. Don't run with an old version.
- The bot's database and logs are per machine, so one machine on an old build does not overwrite another. Still keep every machine on the same version so results match this document.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue. Send:
- the exception ID from section 7 (e.g. E14),
- machine name, date and start time of the run,
- batch number and, where relevant, clinic name and check number,
- what you checked in ClaimDesk (e.g. "payment did post"),
- a screenshot of the Report screen and, if relevant, of ClaimDesk.
Dev pulls the run log and database from the machine themselves - ops can't open those.
Never send: the ClaimDesk password, the input Excel's Login sheet, the bot's settings file, or patient details in chat or tickets.
## 12. Failure-class check
- **F1** covered in section 3 (ClaimDesk setting, sheet names, login cells) and E02-E05.
- **F2** covered in section 4 (close other "ClaimDesk" windows, hands off, don't use ClaimDesk) and E18; Fix #6.
- **F3** covered in E04, E06-E09, E13, E14 (60 s / 10 s / 5 s / 20 s waits, 3 clinic tries in section 8); Fix #9.
- **F4** covered in section 3 (one clinic per row, comma names, cutoff check) and E06, E11; Fix #7, #10.
- **F5** covered in section 6 (clinic Success even with red rows) and E05 (login never confirmed); Fix #8. The batch itself has a guard and is only Completed with no red rows and all clinics finished.
- **F6** covered in E14; Fix #11.
- **F7** covered in section 6 and E12 ("Remittance Errors on posting" shows green); Fix #1.
- **F8** covered in section 8 and E16 (5-minute watchdog, retries don't count as progress); Fix #3. No endless loop: retries stop at 3 per clinic.
- **F9** covered in section 4 (no Stop button; don't close ClaimDesk); Fix #14.
- **F10** covered in E01, E15, E16 (one red row blocks all later runs); Fix #2.
- **F11** covered in section 5 (only the Report screen is open to ops) and section 6 (hidden statuses); Fix #4, #5, #13.
- **F12** covered in section 10 and incident "Users still on 1.3.0"; Fix #17.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | "Rejected by app" (Remittance Errors, not posted) maps to Completed/green. | Unposted payments look done. | `bot.py:29` STATUS_MAP |
| 2 | Any Error row or Pending clinic keeps the batch Pending forever, and a Pending batch blocks every new run. No clear/discard/resume tool exists. | One red row stops the machine until dev edits the DB. | `bot.py:150-154`, `bot.py:167-168` |
| 3 | Watchdog heartbeat is only called per posted item; login, screen waits, retries and empty clinics don't beat. On kill (`os._exit`) the batch stays Pending and ClaimDesk stays open. | Healthy-but-slow runs get killed and then block the next run. | `bot.py:137`, `bot.py:44-49` |
| 4 | Clinic status and batch status are never shown to ops; a clinic that failed writes no Report row; the bot writes no output file ops can open. Clinic-level reasons exist only in the log. | Ops can't tell "nothing to post" from "never got in", or whether a run finished. | `bot.py:140`, `bot.py:147`, `bot.py:177-180`, `report_screen.py:6` |
| 5 | Report screen hides `status` and `tech_comments`. | Technical cause needs dev. | `report_screen.py:6` |
| 6 | Window chosen by title substring "ClaimDesk" - also matches the launcher "ClaimDesk Poster", "ClaimDesk Login" and any DB tool. Commit 34cae4a did not change the logic. Cleanup kills every ClaimDesk.exe. | Bot works in the wrong window; ops lose their own ClaimDesk session. | `bot.py:66-70`, `bot.py:146` |
| 7 | Clinic cells split on `,` and `;`; real names with commas can't run; trailing comma creates an empty clinic. | Clinics silently skipped. | `bot.py:62` |
| 8 | Login success never verified. | Wrong password shows up as unrelated clinic failures. | `bot.py:80-83` |
| 9 | Wrong-clinic title check runs right after the click with no wait. | False "wrong clinic" failures on a slow screen. | `bot.py:90-92` |
| 10 | Cutoff compared as text (`check_date > str(cutoff)`); an Excel date becomes `YYYY-MM-DD 00:00:00`. | Wrong checks deferred or posted with no warning. | `bot.py:107` |
| 11 | Save confirmation is a fixed 20 s wait with no follow-up check (e.g. re-read the unposted grid); no way for ops to correct a row afterwards. | False red rows that also block the batch. | `bot.py:116-118` |
| 12 | Clinic retry keeps earlier rows and re-inserts; stale Error rows keep the batch Pending. | Duplicate rows; batch blocked by already-resolved checks. | `bot.py:130-139`, `bot.py:122-126` |
| 13 | Startup failures (Pending batch, missing sheets/login) go to the console only, not the log; missing Clinics/Settings sheet crashes with a raw KeyError. | Ops see no reason, or a technical one. | `bot.py:53-63`, `bot.py:168` |
| 14 | No Stop mechanism; launcher code not in repo, so it's unknown whether closing the Poster window kills the bot. | Ops have no safe way to stop a run. | `bot.py:174-176` (launcher missing) |
| 15 | Sample log contains a patient name; the log lines in it don't match current code (older build logged item details at INFO). | PHI in log files. Scrub it and confirm no current build logs PHI. | `logs/run_20260901_101500.log:3` |
| 16 | Error lines are written as `[ERROR] [ERROR] ...`. | Breaks log searches for dev. | `bot.py:144` |
| 17 | Launcher/updater code not in repo; the 1.3.0 update fix can't be verified; version is a hard-coded string. | Machines may silently stay on old versions. | `report_screen.py:8`, `report_screen.py:14` |
| 18 | No notification on failure (an alert mail setting exists in `.env` but is unused). | Nobody hears about a killed or blocked run. | `bot.py` (none) |
| 19 | Odd/header rows in the remittance list raise an index error and restart the whole clinic. | Whole clinic retried for one bad row. | `bot.py:99-101` |
| 20 | No tests; none of the four fixes has a regression check. | Fixes can quietly regress. | repo root |
tests/evals/results/handoff/h-desktop-poster-screens-r3.json
+34
View File
@@ -0,0 +1,34 @@
{
"case": "h-desktop-poster/screens",
"rep": 3,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"comma rule before a run": true,
"how to stop": true,
"not-confirmed row: check ClaimDesk first": true,
"hidden tech comments flagged": true,
"Pending batch blocks a new upload": true
},
"cost_usd": 1.5256686000000002,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-desktop-poster-screens-nnl0sb1u\\project",
"reply": "**HANDOVER.md** ready aachu, repo root-la irukku: `HANDOVER.md`. Bot-a naan run pannala, code-um maathala.\n\n| Item | Count |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 18 (E01\u2013E18) |\n| Failure classes | 12/12 covered, not-applicable onnum illa |\n| Fix before handover items | 20 |\n\nNee kuduthha answers ippadi use pannen:\n- **Escalation:** RPA Support queue-la ticket.\n- **Ops access:** Report screen mattum dhaan ops paakka mudiyum. Log-la mattum irukkura causes ellam *Fix before handover*-ku poyiduchu.\n- **Unclear statuses:** code-oda best reading vechu fill pannen.\n\n**Mukkiyamaana Fix before handover items:**\n- Remittance Errors-la reject aana rows, non-ACH skip aana rows rendum green \"Completed\"-aa dhaan kaattudhu. Unmaiyila rendum post aagala.\n- \"Payment not confirmed after Save\" row red-aa kaattum, aana payment post aagiyirukkalaam. Thirumba post panna double posting risk.\n- Window grab bug (commit 6c4ca09) code-la innum fix aagala. Matching innum plain \"title-la ClaimDesk irukkaa\" check dhaan. Mel, bot ClaimDesk-a mudikkum bodhu machine-la irukkura ellaa ClaimDesk window-um force close aagum.\n- Batch Pending-la maatikittaa adutha upload-a block pannum. Adha clear panna tool illa, refusal message-um screen-la varaadhu.\n- 5-minute watchdog, clinic retry pannum bodhu kooda full run-a kill pannidum.\n- Comma irukkura clinic names split aagudhu. README \"split and skipped\"-nu sollum, aana unmaiyila 3 thadava retry aagi Pending-la nikkum.\n- Launcher, updater, Stop button code indha repo-la illa. Ops-ku safe-aa stop panna vazhi illa.\n\n**Also noticed:**\n- `logs/run_20260901_101500.log:3`-la oru patient name irukku (pazhaya build ezhudhinadhu). Doc-la adha copy pannala.\n- History agent-ku shell access kidaikkala. Git diffs paakka mudiyala, commit messages mattum dhaan paathuchu. Ippo code-la paatha, fix commits ellam file kadaisila comment line mattum add pannina maadhiri theriyudhu. `git show 6c4ca09 d93c7fb 9d39f25 2794ede` run panni confirm pannu.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-desktop-poster-screens-r3.md
+207
View File
@@ -0,0 +1,207 @@
# ClaimDesk Poster - ops handover
Version 1.4.2 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
ClaimDesk Poster posts remittance payments into the **ClaimDesk** desktop app, one clinic at a time. It takes one input Excel with three sheets: **Login** (the ClaimDesk user name and password), **Clinics** (the clinics to post) and **Settings** (the cutoff date). For each clinic it signs in to ClaimDesk, opens the clinic, and goes through every row in the remittance list. Each ACH payment dated on or before the cutoff is posted and saved. Other payments are skipped or deferred. Every row it handles appears on the **Report screen** of the ClaimDesk Poster window, with a colour and a comment. The only app it drives is **ClaimDesk**. Do not touch ClaimDesk while the bot runs.
## 2. How it works
```
ClaimDesk Poster (the window you use: start a run, Report screen)
└─ starts → posting bot (runs in the background) ──drives──> ClaimDesk desktop app
│ (do not touch while it runs)
├─ writes → the bot's own database (one row per remittance, clinic and batch status)
├─ writes → a run log (developers only)
└─ watchdog: ends the run if nothing moves for 5 minutes
```
- **ClaimDesk Poster** is the window ops uses. It starts a run and shows results on the Report screen. Its title bar shows the version.
- **The posting bot** runs in the background. It opens ClaimDesk, signs in, and clicks through it like a person would.
- **ClaimDesk** is the target app. The bot opens it fresh for each clinic and **force-closes every ClaimDesk window on the machine** when it finishes that clinic.
- **The database** feeds the Report screen. Ops cannot open it directly.
- **The run log** records clinic-level errors and retries. Only developers can open it.
- **The watchdog** ends the whole run if no remittance row has been processed for 5 minutes.
**One run, step by step:**
1. The bot refuses to start if the previous batch is still open (see E01).
2. It reads the input Excel and creates a new batch. Every clinic starts as Pending.
3. For each clinic, it opens ClaimDesk and signs in with the user name and password from the Login sheet.
4. It clicks the clinic's name in ClaimDesk's clinic list.
5. It waits for the remittance list, then handles each row in turn:
- not ACH → skip;
- check date after the cutoff → defer;
- otherwise → Post Payment, then Save, then wait for "Payment saved".
6. It force-closes ClaimDesk. If anything in steps 3-5 failed, it repeats the whole clinic, up to 3 attempts in total.
7. After the last clinic, it closes the batch. This only happens if no row is red and every clinic finished.
## 3. Before a run
- [ ] ClaimDesk is installed on this machine, and the bot's settings file points at it. Dev sets this up once per machine. If it is missing, every clinic fails (E06).
- [ ] ClaimDesk is **closed**, and no other window has "ClaimDesk" in its title. That includes SQL or database tools (see section 4).
- [ ] The machine stays signed in and unlocked, with the screen on, for the whole run. The bot clicks with the real mouse.
- [ ] **Login sheet:** user name in cell A2, password in cell B2. Both must be filled in (E03). The password must be current. A wrong password shows up as "clinic not found" for every clinic (E07).
- [ ] **Clinics sheet:** column A, starting at row 2, **one clinic per row**. Spell each name exactly as it appears in ClaimDesk's clinic list.
- Bad: `North Clinic, South Clinic` in one cell, or `North Clinic; South Clinic`.
- Good: `North Clinic` in A2, `South Clinic` in A3.
- **A clinic whose real name contains a comma cannot be posted by this bot.** It is always cut into two names that don't exist (E07). Post that clinic by hand.
- No comma at the end of a cell. `North Clinic,` creates an extra empty clinic that fails.
- [ ] **Settings sheet:** the cutoff date goes in cell B1, typed as a real Excel date. If B1 is empty, nothing is deferred and every ACH payment is posted, whatever its date.
- [ ] The previous run ended with **no red rows and no clinic missing** from the Report screen. Otherwise this upload is refused (E01, E16).
- [ ] The ClaimDesk Poster title bar shows version 1.4.2 (see section 10).
## 4. During a run
- **Hands off the machine.** Don't use the mouse or keyboard. The bot clicks with the real mouse, so your clicks and its clicks get mixed up.
- **Don't open ClaimDesk, or any window with "ClaimDesk" in its title, on this machine during a run.** That includes SQL or database tools. The bot picks the first window with "ClaimDesk" in its title, so it can end up working in the wrong one. At the end of each clinic it also force-closes **every** ClaimDesk window on the machine, including one you are using.
- **Closing ClaimDesk does not stop the bot.** The bot treats it as a failure, reopens ClaimDesk and tries that clinic again.
- **To stop the bot:** the code in this repo has no Stop button. Raise an RPA Support ticket if a run must be stopped. Do not just close windows. See *Fix before handover* item 15.
- **A run looks stuck** when no new rows appear on the Report screen for several minutes. The bot ends the run by itself after **5 minutes with no new row**, so wait at least that long. After that the batch stays open and the next upload is refused (E15).
## 5. Where to look when something goes wrong
Ops can open only the ClaimDesk Poster window, its Report screen, and the bot's output files and folder. The bot writes no output files today. Everything ops can see is on the Report screen. The run log and the database are for developers only.
| Question | Where to look |
|---|---|
| Did the run finish? | Ops cannot see the batch status. Compare the Report screen with your Clinics sheet: every clinic must have rows, and no row should be red. Otherwise raise a ticket. |
| Which rows failed? | Report screen → File Status = Error (red) |
| Which rows need a person even though they are green? | Report screen → Comments column: "Remittance Errors on posting" and "Payment method '…' - not posted" |
| Why did one row fail? | Report screen → Comments column, then the matching row in section 7 |
| A clinic has no rows at all | The clinic failed all 3 attempts (E05-E09). The reason is only in the run log, so raise a ticket naming the clinic. |
| Did the bot already retry? | Ops can't see this. Every clinic gets up to 3 attempts automatically. A row that appears twice for the same check number is a sign of a retry (E17). |
| Which version is running? | The ClaimDesk Poster title bar |
## 6. Statuses
What the Report screen shows. File Status is the colour; Comments is the text.
| Status (File Status - Comments) | Colour | Meaning | What ops does |
|---|---|---|---|
| Completed - "Posted successfully" | green | Payment posted and ClaimDesk showed "Payment saved" | nothing |
| Completed - "Remittance Errors on posting" | green | **NOT posted.** ClaimDesk showed its Remittance Errors screen, and the bot cancelled. Needs follow-up. | Review and post it in ClaimDesk by hand (E10) |
| Completed - "Payment method '<method>' - not posted" | green | **NOT posted on purpose.** Not an ACH payment. | Nothing, unless the client wants non-ACH payments posted; then post it by hand (E13) |
| Deferred - "Check date <date> is after the cutoff - deferred" | grey | Not posted. The check date is after the Settings cutoff. | Nothing. A later run with a later cutoff picks it up. Spot-check the date (E14). |
| Error - "Save button not found" | red | The payment screen had no Save button. Probably not posted. | E11 |
| Error - "Payment not confirmed after Save" | red | Save was clicked but "Payment saved" never appeared within 20 s. **It may have posted.** | Check in ClaimDesk first (E12) |
| Pending | amber | The code never writes this on a row today. If you see it, something unexpected happened. | Raise a ticket |
| Clinic: Pending / Success | not shown | Stored by the bot, not on any screen. A clinic left Pending has **no rows** on the Report screen. | See section 5 |
| Batch: Pending / Completed | not shown | The batch stays Pending while any row is red or any clinic is unfinished. A Pending batch blocks the next upload. | E16 |
For developers: the stored row status behind these is Posted, Rejected by app, Skipped, Deferred or Error. It is not shown to ops.
## 7. Exception catalogue
For any **Dev** row, the ticket must include:
- the exception ID;
- the date and time of the run (to the minute);
- the clinic name and check number from the Report screen;
- a screenshot of the Report screen, and of ClaimDesk if it is still open.
Dev finds the run log from the run time.
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | You start a run and nothing happens; no new rows appear | Before reading the Excel, the bot checks for an unfinished (Pending) batch and refuses to start. Message: "A Pending batch exists - finish or clear it before a new upload". | Nowhere ops can see. The message goes to the background window only. | System | Dev | Ops cannot clear the batch. Raise a ticket. | Always |
| E02 | The run ends at once; no new rows | The Excel has no sheet named "Login" | Nowhere ops can see | Business | Ops | Rename the sheet to exactly `Login`, then run again | It still fails with the sheet present |
| E03 | The run ends at once; no new rows | The Login sheet's A2 or B2 is empty ("UserName/Password missing in Login sheet (row 2)") | Nowhere ops can see | Business | Ops | Fill in A2 (user name) and B2 (password) | It still fails with both filled |
| E04 | The run ends at once; no new rows | The Excel has no sheet named "Clinics" or "Settings" | Nowhere ops can see | Business | Ops | Add the missing sheet with exactly that name | It still fails with all three sheets |
| E05 | A clinic has no rows on the Report screen | The ClaimDesk sign-in window did not appear within 60 s, on all 3 attempts | Run log only; clinic left Pending | System | Bot, then Dev | Wait for the run to end. Check that ClaimDesk opens normally on this machine by hand. | Every clinic has no rows, or it repeats on the next run |
| E06 | **No** clinic has any rows; the run ends quickly | ClaimDesk's location is missing from the bot's settings, so it cannot start ClaimDesk | Run log only; every clinic left Pending | System | Dev | Raise a ticket | Always |
| E07 | One clinic, or every clinic, has no rows | The bot clicks the clinic's name in ClaimDesk's list; the name was not there within 10 s ("Clinic '<name>' not found in list"). Usual causes: a spelling difference, a comma name that was cut in two, or a wrong or expired password (then **every** clinic fails this way). | Run log only; clinic left Pending after 3 attempts | Business | Ops | Compare the spelling with ClaimDesk's list exactly. Put one clinic per row. Check that the Login sheet password still works by signing in to ClaimDesk by hand. | Spelling and password are both correct |
| E08 | A clinic has no rows, or only some rows from a partial attempt | Right after clicking the clinic, the bot checks that ClaimDesk's title contains the clinic name ("Wrong clinic opened, leaving Pending"). The bot still retries it. Names with `&` or other special characters often fail this check. | Run log only; clinic left Pending after 3 attempts | System | Bot, then Ops | Post that clinic by hand in ClaimDesk this time | It happens to a clinic without special characters |
| E09 | A clinic has no rows | The clinic opened, but the remittance list did not appear within 60 s, on all 3 attempts ("Remittance grid did not load") | Run log only; clinic left Pending | System | Bot, then Dev | Check by hand whether that clinic's list opens in ClaimDesk; it may simply be slow | It repeats on the next run |
| E10 | Green row, Comments "Remittance Errors on posting" | After Post Payment, ClaimDesk showed its "Remittance Errors" screen within 5 s. The bot pressed Cancel. | Report screen | Business | **Ops** | **This payment was not posted.** Open it in ClaimDesk, resolve the remittance error, post it by hand. | ClaimDesk shows no error for it |
| E11 | Red row, "Save button not found" | After Post Payment, no Save button appeared within 10 s | Report screen | System | Ops, then Dev | Check the payment in ClaimDesk. It is most likely not posted. Post it by hand if needed. The red row keeps the batch open (E16). | It happens on more than one row |
| E12 | Red row, "Payment not confirmed after Save" | Save was clicked, but ClaimDesk's "Payment saved" message did not appear within 20 s | Report screen | System | **Ops** | **Check the payment in ClaimDesk before doing anything else. It may have posted.** If it posted, do not post it again. Raise a ticket so dev corrects the row; ops cannot edit it, and the red row keeps the batch open (E16). If it did not post, post it by hand. | Every time, so the row can be corrected |
| E13 | Green row, "Payment method '<method>' - not posted" | The remittance's method is anything other than ACH | Report screen | Business | Ops | Nothing, unless the client wants it posted; then post it by hand | - |
| E14 | Grey row, "Check date <date> is after the cutoff - deferred" | The check date is after the Settings B1 cutoff. The comparison is done on the date text, so unusual date formats can give the wrong answer. | Report screen | Business | Ops | Nothing for a real future date. If the date is plainly before the cutoff, raise a ticket. | A date before the cutoff was deferred, or a later date was posted |
| E15 | The run stops partway: later clinics have no rows; ClaimDesk may still be open | The watchdog saw no new row for 5 minutes and ended the whole run. It often fires when a clinic fails all 3 attempts, or has no remittances, because no rows are produced. | Run log only; the batch stays open | System | Dev | Close ClaimDesk. Raise a ticket listing the clinics with no rows. The next upload will be refused until dev clears the batch. | Always |
| E16 | The next upload is refused (E01) after a run that "finished" | The batch only closes when no row is red and every clinic finished. One red row or one failed clinic keeps it open, and an open batch blocks every new upload. | Not shown to ops | Business | Ops, then Dev | Find the red rows and the clinics with no rows on the Report screen. Handle each one by its ID above. Then raise a ticket asking dev to close or clear the batch. | Always: only dev can clear it |
| E17 | The same check number appears twice for one clinic | A clinic was retried after a failure partway through its list. The first attempt's rows stay, and the retry writes the rows again. | Report screen | System | Ops | Trust the **latest** row for that check. An old red row from the first attempt still keeps the batch open (E16). | The two rows disagree on whether it was posted |
| E18 | ClaimDesk you were using closes by itself, or the bot clicks inside a database tool or the Poster window | The bot takes the first window with "ClaimDesk" in its title, and force-closes all ClaimDesk windows after each clinic | Not recorded | System | Ops, then Dev | Close all other "ClaimDesk" windows before a run. Don't use ClaimDesk on this machine during a run. | It happens with no other such window open |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Anything fails while signing in, opening the clinic or walking its list (E05-E09, or a ClaimDesk button missing mid-list) | Force-closes ClaimDesk, signs in again and repeats the whole clinic | 3 attempts per clinic. Each attempt can take up to about 2-3 minutes (60 s sign-in, 10 s clinic list, 60 s remittance list). | Clinic left Pending with no new rows; the bot moves to the next clinic |
| One remittance row fails (E11, E12) | **No retry.** Writes the red row and moves to the next remittance. | - | Row stays red; batch stays open |
| No new row for 5 minutes | Ends the entire run | 5 minutes, checked every 10 s | Run over; batch open; ClaimDesk may be left open (E15) |
| Next run | **No resume.** A new upload is refused while the old batch is open. | - | Dev must clear the batch (E01, E16) |
So: **once a clinic has had 3 attempts, or the run has been quiet for 5 minutes, the bot will not try again by itself.** Wait for that, then act.
## 9. Known incidents
Newest first.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 1.4.2 | Users kept running 1.3.0 after an update | The launcher kept using the old program folder until it was restarted | After every update, close and reopen ClaimDesk Poster, and check the title bar shows the new version (section 10) | Yes |
| Before 1.4.2 | Red "Payment not confirmed after Save" rows that had really posted | ClaimDesk took longer to confirm than the bot waited (10 s) | The wait is now 20 s, but a slow ClaimDesk can still cause it. **Always check in ClaimDesk first** (E12). | Partly |
| Before 1.4.2 | Clinics with a comma in the cell were never posted | The bot cuts names at every comma or semicolon | One clinic per row. Clinics whose real name has a comma must be posted by hand (E07). | Partly |
| Before 1.4.2 | The bot worked inside a SQL tool whose title contained "ClaimDesk", then closed the real ClaimDesk | The bot takes the first window with "ClaimDesk" in its title | Close every other "ClaimDesk" window before a run (section 4, E18). The code still matches this way. | Partly |
| Earlier build (log) | Run stopped partway with no message on screen | Watchdog: 5 minutes with no new row | E15 | Partly |
| Earlier build (log) | A clinic with `&` in its name was retried and left without rows | The clinic-name check failed on the special character | E08 | Partly |
## 10. Releases and version check
- **How an update arrives:** through the ClaimDesk Poster launcher. **The launcher keeps the old version until it is closed and reopened.**
- **How to confirm it applied:** the ClaimDesk Poster title bar shows the version. Today that is 1.4.2.
- **If the version did not change:** close ClaimDesk Poster completely and reopen it, then check the title bar again. If it is still old, raise an RPA Support ticket with the version shown and the machine name.
- Every machine that runs the bot should show the same version.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue.
**Send:**
- the exception ID from section 7;
- the date and time of the run (to the minute) and the machine name;
- the clinic name and check number(s) from the Report screen;
- the version from the title bar;
- a screenshot of the Report screen, and of ClaimDesk if it is open.
**Never send:** the ClaimDesk password or the Login sheet, the bot's settings file, or patient details in the ticket or in chat.
## 12. Failure-class check
- F1 (missing prerequisite) covered in section 3, E06, and Fix 10, 16.
- F2 (wrong window / focus) covered in section 4, E18, and Fix 3, 4.
- F3 (target app slower or different) covered in E05, E08, E09, E11, E12, section 8, and the section 3 "unlocked, screen on" rule.
- F4 (input read differently) covered in section 3 input rules, E07, E08, E14, and Fix 5, 13.
- F5 (done when nothing done) covered in E15 and Fix 7. A clinic is marked finished even with 0 remittances, and that looks the same as a failed clinic on screen.
- F6 (action happened but not confirmed) covered in E12 and Fix 2.
- F7 (business exception shown as success) covered in section 6, E10, E13, and Fix 1.
- F8 (endless re-runs / watchdog) covered in section 8 and E15. Retries are capped at 3, so there is no endless loop. Fix 11.
- F9 (stopping the wrong way) covered in section 4 and Fix 15.
- F10 (one stuck item blocks the queue) covered in E01, E16, E17, and Fix 9, 12.
- F11 (cause hidden from ops) covered in section 5 and Fix 7, 8, 9, 10.
- F12 (update didn't arrive) covered in section 10, incident 1, and Fix 15, 17.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | "Rejected by app" (Remittance Errors, not posted) and "Skipped" (not ACH) are stored as Completed, so they show green | Ops thinks unposted payments are done | `bot.py:29-30` STATUS_MAP |
| 2 | A Save confirmation timeout is recorded as Error. There is no read-back in the grid to check whether the payment really posted. | Real posts show red; re-posting by hand risks double payment; the red row blocks the batch | `bot.py:117-118` |
| 3 | The window match is still a plain "title contains ClaimDesk", even though commit 6c4ca09 says it was fixed. It can grab the "ClaimDesk Poster" launcher, the "ClaimDesk Login" window or a SQL tool. | The bot drives the wrong window | `bot.py:66-70` |
| 4 | `taskkill /im ClaimDesk.exe /f` kills every ClaimDesk process on the machine | A user's own ClaimDesk session is closed | `bot.py:146` |
| 5 | Clinic cells are split on commas and semicolons. README:8 wrongly says such names are "split and skipped": they are actually retried 3 times, then left Pending. | Clinics with a comma in their name can never be posted; the batch is blocked | `bot.py:62`, `README.md:8` |
| 6 | The wrong-clinic check runs immediately after the click, with no wait, as a substring match. Its message says "leaving Pending" but the clinic is retried. | False failures on `&` names; a misleading message | `bot.py:91-92` |
| 7 | A clinic left Pending writes no Report row. Clinic and batch status are not on any screen. | Ops can't see failed clinics without the log | `bot.py:147`, `report_screen.py:6` |
| 8 | The raw status and tech_comments columns are stored but not shown on the Report screen | The developer-level cause is hidden from ops | `report_screen.py:6` |
| 9 | The "A Pending batch exists" refusal goes to stderr only, and there is no clear or resume tool for ops | The upload silently does nothing; only dev can unblock it | `bot.py:167-168` |
| 10 | Input errors (missing sheets, missing login) are raised before or without logging, and never reach a screen. A missing CLAIMDESK_EXE surfaces only as a clinic error in the log. | Ops sees "nothing happened" | `bot.py:52-63`, `bot.py:74`, `bot.py:169` |
| 11 | The watchdog heartbeat happens only per remittance row, so sign-in, clinic open and retries don't count. A clinic failing 3 times, or one with 0 rows, can exceed 300 s. Exiting with os._exit skips cleanup: ClaimDesk is left open and the batch stays Pending. | Whole runs die partway | `bot.py:39-49`, `bot.py:137` |
| 12 | A clinic retry re-inserts report rows without removing the first attempt's rows | Duplicate rows; a stale red row blocks the batch forever | `bot.py:130-139` |
| 13 | The cutoff is compared as text (`check_date > str(cutoff)`) | Wrong defer/post decisions when date formats differ | `bot.py:107` |
| 14 | The Remittance Errors popup is only waited for 5 s | A late popup is clicked past, and the row may end up as "Save button not found" | `bot.py:110` |
| 15 | The launcher and updater code are not in this repo. There is no Stop button, and it's unknown whether closing the launcher stops the bot. The 1.3.0 update fix has no code here. | Ops has no safe way to stop a run; the update behaviour can't be verified | `report_screen.py:14`, `bot.py:3` |
| 16 | The ClaimDesk user name and password are kept in plain text in the input Excel | Credentials get passed around with the input file | `bot.py:56` |
| 17 | The version is shown in the title bar only. It is not written to the run log. | The run that produced a log can't be matched to a build | `report_screen.py:8`, `bot.py:164` |
| 18 | The existing run log contains a patient name. It was written by an older build, but the log folder should be treated as PHI. | PHI exposure if logs are shared | `logs/run_20260901_101500.log:3` |
| 19 | Remittance outcomes are not written to the run log at all. The bot writes no output file, although ops' access was planned around an output folder. | Ops has no output to check outside the Report screen | `bot.py:136-139` |
| 20 | Sign-in success is never checked. With a wrong password, the window search returns the Login window, and every clinic fails as "not found in list". | Credential problems are misreported | `bot.py:80-83` |
tests/evals/results/handoff/h-desktop-poster-screens-r4.json
+34
View File
@@ -0,0 +1,34 @@
{
"case": "h-desktop-poster/screens",
"rep": 4,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"comma rule before a run": true,
"how to stop": true,
"not-confirmed row: check ClaimDesk first": true,
"hidden tech comments flagged": true,
"Pending batch blocks a new upload": true
},
"cost_usd": 1.5966884000000003,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-desktop-poster-screens-_c3yh3s7\\project",
"reply": "`HANDOVER.md` repo root-la ready. Code-um history-um mattum padichen; bot-a run pannala, code edhuvum maathala.\n\n| Item | Count |\n|---|---|\n| Sections | 13 (template-la irukka ella section-um) |\n| Exceptions catalogued | 20 (E01\u2013E20) |\n| Failure classes | 12/12 covered, \"not applicable\" onnum illa |\n| Fix before handover | 18 items |\n\n**Mukkiyamana findings:**\n- **Moonu \"fix\" commits-um (63f0a86, bb0f3b3, 4af55eb) code-a maathala:** rendu comment line mattum add pannudhu, onnu README line mattum. Wrong-window grab, comma split, \"Payment not confirmed\" moonum innum open. 4th fix (9945eb1, launcher update) launcher code repo-la illadhadhaala check panna mudiyala.\n- **Ops-ku ella batch problem-um dev ticket dhaan:** oru red row, fail aana clinic, illa watchdog kill aanaalum batch Pending-la nikkum, adutha run-a block pannum. Adha clear panna vazhi code-la illa.\n- **Green \u2260 posted:** \"Remittance Errors\" rows-um non-ACH rows-um green Completed-a kaatum, aana post aagala.\n- **Ops-ku clinic failures theriyaadhu:** clinic fail aana reason log-la mattum irukku, ops-ku log access illa. Screen-la andha clinic-ku rows illaama dhaan theriyum.\n- **Security:** `.env` git-la commit aagi irukku (fallback password key-oda). Committed log-la oru patient name irukku. Rendum Fix #1, #2-la irukku. Andha values-a doc-la copy pannala.\n\n**Unclear statuses** (unga instruction padi code padichu meaning ezhudhinen):\n- amber Pending: indha version ezhudhave ezhudhaadhu.\n- `&` irukkura clinic names \"wrong clinic\"-a fail aagum nu suspect mattum, confirm illa.\n\n**Also noticed** (fix pannala):\n- No Stop button repo-la illa.\n- Cutoff date-a text-a compare pannudhu, adhaala defer decisions thappa varalaam.\n\nQUESTIONS: Who does ops call for a dev issue (team or role), and how - ticket or chat channel? | Which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-desktop-poster-screens-r4.md
+206
View File
@@ -0,0 +1,206 @@
# ClaimDesk Poster - ops handover
Version 1.4.2 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
ClaimDesk Poster posts remittance payments into the **ClaimDesk** desktop app, one clinic at a time. It reads the input Excel, which has three sheets: **Login** (ClaimDesk user name and password), **Clinics** (the clinics to process) and **Settings** (the cutoff date). For each clinic it signs in to ClaimDesk, opens the clinic, and works through every unposted remittance in the list:
- ACH payments dated on or before the cutoff are posted and saved.
- Other payment methods are skipped.
- Payments dated after the cutoff are deferred.
Every remittance gets one row on the **Report screen** of the ClaimDesk Poster window. The bot writes no Excel or PDF output files, so the Report screen is its only output. It drives one desktop app: **ClaimDesk**.
## 2. How it works
```
ClaimDesk Poster window (what you use: start a run, Report screen)
└─ starts → automation process ──drives──> ClaimDesk desktop app
│ (do not touch while it runs)
├─ writes → one row per remittance → Report screen
└─ watchdog: stops the whole run after 5 min with no payment handled
```
- **ClaimDesk Poster window** is where you start a run, see the Report screen, and check the version in the title bar.
- **The automation process** does the work. It has no window of its own.
- **ClaimDesk** is opened, signed in to, and **force-closed** by the bot for every clinic. Any ClaimDesk window open on this machine is closed too.
- **The watchdog** stops the whole run if no remittance has been handled for 5 minutes.
**One run, step by step:**
1. The bot refuses to start if an earlier batch is still unfinished (E04).
2. It reads the input Excel and creates a new batch.
3. For each clinic, it does steps 4 to 8.
4. Open ClaimDesk and sign in.
5. Click the clinic in ClaimDesk's clinic list.
6. Wait for the remittance list to load.
7. For each remittance in the list, post it, skip it or defer it, and write one Report row.
8. Close ClaimDesk. If anything in steps 4 to 7 failed, start the clinic again from step 4 (3 tries in total).
9. After the last clinic, the batch is marked complete only if no row is red and no clinic is still unfinished.
## 3. Before a run
- [ ] **No unfinished batch.** If the last batch has any red row, or any clinic that never got rows, the next run is refused (E04). Raise a ticket first.
- [ ] **The dev team has set up this machine**: Python, the bot's packages, ClaimDesk installed, and the bot's settings file holding the ClaimDesk program path. If the path is missing, every clinic fails with no rows (E05).
- [ ] **Login sheet**: user name in A2, password in B2. Both must be filled in, or the run stops (E02).
- [ ] **Clinics sheet**:
- Put one clinic per row in column A, starting at row 2.
- Spell each name exactly as it appears in ClaimDesk's clinic list.
- **The bot splits a cell at every comma and semicolon:**
- Bad: `Example, Sample Dental` in one cell. The bot looks for "Example" and "Sample Dental" and finds neither.
- Good: `Example Sample Dental`. Only works if ClaimDesk spells it that way.
- A clinic whose real ClaimDesk name contains a comma or semicolon **cannot be run by this version**. Leave it out and post it by hand.
- Names containing `&` may be reported as "wrong clinic" (E09). Check those clinics on the Report screen after the run.
- [ ] **Settings sheet**: cutoff date in B1.
- The bot compares dates as text, not as real dates. On the first run after any change to the cutoff or the input file, check a few Deferred rows and a few posted rows against their check dates.
- If they look wrong, raise a ticket. Do not re-run.
- [ ] **All three sheets exist** and are named exactly `Login`, `Clinics` and `Settings` (E01, E03).
- [ ] The ClaimDesk Poster title bar shows version **1.4.2** (see section 10).
## 4. During a run
- **Don't use ClaimDesk on this machine.** The bot force-closes every ClaimDesk window after each clinic, including yours.
- **Close every other window whose title contains "ClaimDesk"**, for example a database tool with ClaimDesk data open, or a browser tab. The bot picks the first window with "ClaimDesk" in its title and can drive or close the wrong one (E20).
- Don't use the mouse or keyboard on this machine. The bot clicks on screen.
- **To stop the bot:** this version has no Stop button that we can confirm.
- Closing ClaimDesk does **not** stop the bot. It treats that as a failure, re-opens ClaimDesk and starts the clinic again.
- If you must stop a run, raise a ticket and the dev team will stop it.
- **A run looks stuck** when ClaimDesk sits on the same screen, or opens and closes again and again for the same clinic. The bot itself gives up after **5 minutes without handling a remittance**. It then stops the whole run and leaves ClaimDesk open (E17). Wait that long before acting.
## 5. Where to look when something goes wrong
You can open the Report screen of the ClaimDesk Poster window. The bot writes no output files, so there is nothing to check in the output folder. Its logs and database are not open to ops, so some answers below need a ticket.
| Question | Where to look |
|---|---|
| Did the run finish? | Not shown on screen today. Signs: every clinic in the input has rows on the Report screen, and the next run is not refused. Otherwise raise a ticket (dev reads the run log for "RUN SUMMARY" or "WATCHDOG"). |
| Which remittances failed? | Report screen → File Status = **Error** (red). |
| Why did one remittance fail? | Report screen → **Comments** column, then match it in section 7. |
| Which remittances were NOT posted although green? | Report screen → File Status = Completed, and Comments is **not** "Posted successfully" (see section 6). |
| A clinic is missing from the Report screen | The clinic failed all 3 tries, or the run was stopped before reaching it. The reason is only in the run log, so raise a ticket (E05 to E11, E17). |
| Did the bot already retry? | Not shown on screen. If you saw ClaimDesk open and close several times for one clinic, it retried. Dev can confirm from "[AUTO-RERUN]" lines in the log. |
| Which version is running? | ClaimDesk Poster title bar. |
## 6. Statuses
What you see on the Report screen is **File Status** (coloured) plus **Comments**. Read them together: green does not always mean posted.
| Status (File Status - Comments) | Colour | Meaning | What ops does |
|---|---|---|---|
| Completed - "Posted successfully" | green | Payment posted and saved in ClaimDesk | Nothing |
| Completed - "Remittance Errors on posting" | green | **NOT posted.** ClaimDesk showed its Remittance Errors screen and the bot pressed Cancel. Needs follow-up. | Review this remittance in ClaimDesk by hand (E14) |
| Completed - "Payment method '<method>' - not posted" | green | **NOT posted on purpose.** It is not an ACH payment. | Nothing, unless the client wants non-ACH payments posted (E12) |
| Deferred - "Check date <date> is after the cutoff - deferred" | grey | Not posted, because the check date is after the cutoff | A later run with a later cutoff picks it up. Spot-check the dates (E13). |
| Error - "Save button not found" | red | Payment screen opened, but there was no Save button. Not saved. | E15 |
| Error - "Payment not confirmed after Save" | red | The bot pressed Save but did not see "Payment saved" within 20 s. **It may have posted.** | Check in ClaimDesk first (E16) |
| Pending | amber | Never written by the current version | Raise a ticket if you ever see it |
| *(clinic has no rows)* | - | The clinic failed all 3 tries, or the run stopped first | E05 to E11, E17 |
| *(same check number twice)* | - | The clinic was retried, and the retry wrote its rows again | E18 |
Hidden statuses, not shown on screen (for reference when talking to dev):
- **Batch**: Pending or Completed. A Pending batch blocks the next run (E04).
- **Clinic**: Pending or Success. A clinic left Pending keeps the batch Pending.
## 7. Exception catalogue
Quote the ID (E01, E02...) in your ticket.
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Run ends at once, no new rows. Message may read "'Login' sheet not found". | The input Excel has no sheet named Login | Only in the process's own error output, not on the Report screen | Business | Ops | Add or rename the sheet to `Login`, then run again | It still fails with the sheet present. Send: time of the run, E01. |
| E02 | Run ends at once, no new rows. "UserName/Password missing in Login sheet (row 2)" | A2 or B2 of the Login sheet is empty | Process error output only | Business | Ops | Fill A2 (user name) and B2 (password), then run again | Both are filled in and it still fails. Send: time, E02. Never send the password. |
| E03 | Run ends at once, no new rows, no clear message | The Clinics or Settings sheet is missing, so the bot crashes while reading the input | Process error output only | Business | Ops | Check that `Clinics` and `Settings` exist with exactly those names, then run again | The sheets are correct. Send: time, E03. |
| E04 | Run refused: "A Pending batch exists - finish or clear it before a new upload" | An earlier batch is still unfinished: it has a red row, a clinic with no rows, or it was stopped by the watchdog | Process error output | System | Dev | You cannot clear it yourself. Note the red rows and missing clinics of the last batch from the Report screen. | Always. Send: time, E04, and the clinics or check numbers that are red or missing. |
| E05 | **No clinic** got any rows. ClaimDesk opened and closed many times. | Sign-in never reached the clinic list. Most likely a wrong password, or the ClaimDesk program path missing on this machine. A wrong password is **not detected directly**. | Run log only. Every clinic is left unfinished. | Business / System | Ops, then Dev | Check the user name and password in the Login sheet by signing in to ClaimDesk by hand. Then raise a ticket: the batch is now blocked (E04). | Always, because the batch must be cleared. Send: time, E05, and whether a manual sign-in works. |
| E06 | One clinic has no rows. ClaimDesk opened and closed 3 times for it. | The ClaimDesk Login window did not appear within 60 s | Run log: "Login window not found"; clinic left unfinished | System | Bot, then Dev | Wait. The bot tries the clinic 3 times. | The clinic still has no rows after the run. Send: time, clinic name, E06. |
| E07 | One clinic has no rows | After sign-in, no window titled ClaimDesk was found | Run log: "ClaimDesk window not found" | System | Bot, then Dev | Wait for the 3 tries | Still no rows. Send: time, clinic name, E07. |
| E08 | One clinic has no rows | The clinic name was not in ClaimDesk's clinic list within 10 s | Run log: "Clinic '<name>' not found in list" | Business | Ops | Compare the spelling in the Clinics sheet with ClaimDesk's list, letter for letter. Remove commas and semicolons (section 3). | The spelling is exact. Send: time, clinic name, E08. |
| E09 | One clinic has no rows | After clicking the clinic, its name is not part of ClaimDesk's window title | Run log: "Wrong clinic opened, leaving Pending: <name>" | System | Bot, then Dev | Check the spelling as in E08. Names with `&` are suspected to trigger this. | The spelling is exact. Send: time, clinic name, E09. |
| E10 | One clinic has no rows | The remittance list did not load within 60 s | Run log: "Remittance grid did not load" | System | Bot, then Dev | Wait for the 3 tries. Check that ClaimDesk opens that clinic quickly by hand. | Still no rows. Send: time, clinic name, E10. |
| E11 | A clinic has only some of its rows, or rows appear twice | Something unexpected failed part-way through a clinic. The bot starts the whole clinic again. | Run log: "[ERROR] Clinic '<name>' : ..." and "[AUTO-RERUN]" | System | Bot, then Dev | Wait for the 3 tries. Then treat duplicate rows as in E18. | The clinic still has missing rows. Send: time, clinic name, E11. |
| E12 | Green Completed - "Payment method '<method>' - not posted" | The payment method in ClaimDesk is not ACH | Report screen | Business | Ops | Nothing. Only ACH is posted. Post by hand if the client wants it. | Never |
| E13 | Grey Deferred - "Check date <date> is after the cutoff - deferred" | The check date is after the cutoff in Settings B1 | Report screen | Business | Ops | It is picked up by a later run. Spot-check that the date really is after the cutoff. | A Deferred row is dated before the cutoff, or a posted row after it. Send: time, check number, E13. |
| E14 | Green Completed - "Remittance Errors on posting" | ClaimDesk showed the Remittance Errors screen within 5 s of Post Payment. The bot pressed Cancel. | Report screen | Business | Ops | **It was not posted.** Open the remittance in ClaimDesk, resolve the error and post by hand. | Never, unless every remittance gets this |
| E15 | Red Error - "Save button not found" | No Save button appeared within 10 s on the payment screen | Report screen; technical detail only in the hidden database | System | Ops, then Dev | Check in ClaimDesk: the payment should not be posted. Post it by hand. This red row blocks the batch (E04). | Always, to clear the batch. Send: time, clinic, check number, E15. |
| E16 | Red Error - "Payment not confirmed after Save" | The bot pressed Save but "Payment saved" did not appear within 20 s | Report screen | System | **Ops**, then Dev | **Check in ClaimDesk first.** Look the check number up in that clinic. If it is posted, do nothing more in ClaimDesk. If not, post it by hand. Do not post it twice. This red row blocks the batch (E04). | Always, to clear the batch. Send: time, clinic, check number, whether it was posted in ClaimDesk, E16. |
| E17 | The run stops part-way. ClaimDesk is left open. Later clinics have no rows. The next run is refused (E04). | The watchdog: no remittance handled for 5 minutes. This also happens when one or more clinics keep failing their retries, or when several clinics in a row have nothing to post. | Run log: "[WATCHDOG] no progress for 300s - killing run" | System | Dev | Close ClaimDesk. List which clinics have rows. | Always. Send: time it stopped, last clinic with rows, E17. |
| E18 | The same check number appears more than once for a clinic | The clinic was retried, and the retry wrote rows again for remittances that were still in ClaimDesk's unposted list (skipped, deferred, rejected, failed) | Report screen | System | Ops | Read the rows together. If any copy says "Posted successfully", it is posted. Otherwise the latest outcome applies; check in ClaimDesk if unsure. | The copies disagree and ClaimDesk does not settle it. Send: clinic, check number, E18. |
| E19 | The title bar shows an older version after an update | The launcher keeps using the old folder until it is restarted | Title bar | System | Ops | Close the ClaimDesk Poster window completely and reopen it. Check the version again. | Still old after a restart. Send: the version shown, E19. |
| E20 | The bot types or clicks in another program, or ClaimDesk closes during a run | The bot uses the first window whose title contains "ClaimDesk". That can be a database tool, a browser tab, or the ClaimDesk Poster window itself. | Run log only; the clinic usually fails and is retried | System | Ops, then Dev | Before every run, close all other windows with "ClaimDesk" in the title (section 4) | It happens with no such window open. Send: time, clinic, the windows that were open, E20. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Any failure while a clinic is being processed: sign-in, clinic list, remittance list, or something unexpected | Closes ClaimDesk, signs in again, starts the clinic over | 3 tries per clinic. Each try can wait up to 60 s for sign-in, 10 s for the clinic link and 60 s for the remittance list. | Clinic left unfinished, with no new rows. The batch stays unfinished and blocks the next run (E04). The bot moves to the next clinic. |
| No remittance handled for a long time | The watchdog stops the **whole run** | 5 minutes, checked every 10 s | The run ends at once. ClaimDesk is left open, the remaining clinics are not done, and the batch blocks the next run (E17). Failing clinics often hit this before their 3rd try ends. |
| Save not confirmed, or Save button missing | **No retry.** The row is marked Error and the bot moves to the next remittance. | - | Red row; ops checks in ClaimDesk (E15, E16) |
| Remittance Errors screen in ClaimDesk | Presses Cancel and moves on | Waits 5 s for the screen to appear | Green row "Remittance Errors on posting", not posted (E14) |
**How long to wait:** one clinic that keeps failing takes up to about 7 minutes of retries. The watchdog ends the run if 5 minutes pass without a remittance being handled. So if ClaimDesk has shown no progress for more than 5 minutes, the run has stopped.
## 9. Known incidents
Newest first. Dates are not recorded; versions are from the commit history.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Up to 1.4.2 | The title bar still showed 1.3.0 after an update | The launcher kept using the old folder until restarted | Restart the ClaimDesk Poster window after every update and check the version (E19). No code fix is in this repo. | Yes |
| Up to 1.4.2 | Red "Payment not confirmed after Save" rows that had really posted | ClaimDesk saved more slowly than the wait | **Still open.** The wait is 20 s, but nothing re-checks the posting afterwards. Always check in ClaimDesk first (E16). | Partly: ops can check, but needs dev to clear the batch |
| Up to 1.4.2 | A clinic was never processed. The batch stayed unfinished. | Clinic names containing commas are split into pieces that don't exist | **Still open.** One clinic per row does not help when the name itself contains a comma. Leave such clinics out (section 3). | Partly |
| Up to 1.4.2 | The bot drove a database tool, then closed the real ClaimDesk | That tool's window title contained "ClaimDesk" | **Still open.** Close all other "ClaimDesk" windows before a run (E20). | Partly |
| Older version (from an old log) | "Wrong clinic opened", then the whole run stopped by the watchdog | Clinic-name check against the window title, plus retries with no progress, which hit the 5-minute limit | **Still open** (E09, E17) | Partly: ops can spot it, dev clears the batch |
## 10. Releases and version check
- **How an update arrives:** through the ClaimDesk Poster launcher. The update mechanism is not in this repo; ask the dev team for the details.
- **How to confirm it applied:** the ClaimDesk Poster title bar shows the version. Today it is **1.4.2**.
- **If the version did not change:** close the ClaimDesk Poster window completely, reopen it and check again. If it is still old, raise a ticket (E19).
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue. Include:
- the exception ID from section 7
- the time of the run, and the time it went wrong
- the clinic name and check number(s)
- a screenshot of the Report screen, and of ClaimDesk if relevant
- what you already checked in ClaimDesk
Never send passwords, the input Excel's Login sheet, the bot's settings file, or patient data in a ticket or chat.
## 12. Failure-class check
- F1 covered in section 3 (machine setup, ClaimDesk path) and E05; Fix #14.
- F2 covered in section 4 and E20; Fix #3.
- F3 covered in E06, E07, E08, E10, E15 and E16, and section 8.
- F4 covered in section 3 (commas, `&`, cutoff dates), E08, E09 and E13; Fix #4, #12 and #13.
- F5 covered in Fix #16: a clinic is marked done even when it had 0 remittances, with no check that the right list was shown.
- F6 covered in E16; Fix #5.
- F7 covered in section 6, E12 and E14; Fix #6.
- F8 covered in section 8 and E17; Fix #10.
- F9 covered in section 4 (no known Stop; closing ClaimDesk does not stop it); Fix #17.
- F10 covered in E04, E15, E16 and E17; Fix #9.
- F11 covered in section 5; Fix #7, #8 and #14.
- F12 covered in section 10 and E19; Fix #17.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | `.env` is committed to git, including a password-fallback key. ClaimDesk passwords are also kept in plain text in the input Excel. | Credential exposure | `.env` (tracked), `bot.py:56`, `bot.py:162` |
| 2 | The committed run log contains a patient name | PHI in the repo history | `logs/run_20260901_101500.log:3` |
| 3 | Window lookup takes the first window whose title *contains* "ClaimDesk". Commit 63f0a86 only added a comment. It can also match the launcher "ClaimDesk Poster" and the login window. | The bot drives or kills the wrong window (E20) | `bot.py:66-70`, `bot.py:185` |
| 4 | Every Clinics cell is split on `,` and `;`. Commit bb0f3b3 was README-only. | Clinics with commas can never run, and they block the batch | `bot.py:62` |
| 5 | Save confirmation is a 20 s wait with no re-check of the grid. Commit 4af55eb only added a comment; the value was already 20. | False Error rows, risk of double posting, blocked batch (E16) | `bot.py:21`, `bot.py:117-118`, `bot.py:186` |
| 6 | "Rejected by app" and "Skipped" map to green Completed | Ops thinks not-posted items are done | `bot.py:27-33` |
| 7 | `tech_comments` is written but not shown on the Report screen | The technical cause is hidden from ops | `report_screen.py:6`, `bot.py:123` |
| 8 | Clinic-level failures, retries and the watchdog kill are recorded only in the log. The Report screen has no clinic or batch view. Ops has no log access. No alerts (`ALERT_SMTP` unused). | Ops sees only "clinic missing" | `bot.py:144`, `bot.py:147`, `bot.py:48`, `report_screen.py:11-13` |
| 9 | Any Pending batch blocks every new upload, and there is no way to clear or resume it | Every red row or failed clinic needs a dev ticket before the next run (E04) | `bot.py:150-154`, `bot.py:167-168` |
| 10 | The watchdog heartbeat is only per remittance. Sign-in, clinic open, retries and empty clinics never beat. `os._exit` skips cleanup and leaves ClaimDesk open. | Healthy runs get killed; leftovers block the next run (E17) | `bot.py:39-49`, `bot.py:137`, `bot.py:146` |
| 11 | A wrong password is not detected; it surfaces as "Clinic not found" on every clinic | Ops chases spelling instead of credentials (E05) | `bot.py:80-83` |
| 12 | The wrong-clinic check is a substring test on the window title. It is suspected to fail on `&`, and "North Clinic" matches "North Clinic 2". | False "wrong clinic" failures, or a missed real one (E09) | `bot.py:91` |
| 13 | The cutoff is compared as text (grid text vs `str()` of the Excel cell) | Wrong post/defer decisions (E13) | `bot.py:107` |
| 14 | Input errors and a missing ClaimDesk path aren't logged or shown in the UI. A missing Clinics/Settings sheet crashes with no message. A missing `CLAIMDESK_EXE` is retried as a clinic failure. | Ops gets no message (E01-E03, E05) | `bot.py:55-63`, `bot.py:74`, `bot.py:169` |
| 15 | A clinic retry re-inserts report rows for items still unposted. Report rows have no timestamp. | Duplicate, ambiguous rows (E18) | `bot.py:136-139`, `bot.py:123` |
| 16 | A clinic is marked Success with 0 items, with no positive confirmation it was on the right list. The batch can complete with nothing posted. | Silent "done" | `bot.py:140`, `bot.py:150-154` |
| 17 | No Stop mechanism and no launcher/updater code in the repo. The 1.3.0 update fix (9945eb1) is a comment only. | Ops has no safe way to stop a run; the update fix can't be verified | `report_screen.py:14` |
| 18 | The run summary has no counts; the log shows doubled "[ERROR] [ERROR]" tags | Harder for dev to triage tickets | `bot.py:180`, `bot.py:144` |
tests/evals/results/handoff/h-desktop-poster-screens-r5.json
+34
View File
@@ -0,0 +1,34 @@
{
"case": "h-desktop-poster/screens",
"rep": 5,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"comma rule before a run": true,
"how to stop": true,
"not-confirmed row: check ClaimDesk first": true,
"hidden tech comments flagged": true,
"Pending batch blocks a new upload": true
},
"cost_usd": 1.8868500000000001,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-desktop-poster-screens-tyvh8gj0\\project",
"reply": "`HANDOVER.md` ready, repo root la irukku. Bot-a run pannala, code + git history + log mattum padichu ezhudhinen. Code change edhuvum illa.\n\n| Item | Count |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 13 (E01\u2013E13) |\n| Failure classes | 12/12 covered, 0 not applicable |\n| Fix before handover | 20 items |\n\n**Fix before handover la mukkiyamanavai:**\n- Neenga sonna \"output folder\" ku indha bot edhuvum ezhudhuradhu illa. Ops paakka mudinja ore idam launcher la irukkura Report screen dhaan.\n- ClaimDesk reject panna payment-um green \"Completed\" nu dhaan kaattum.\n- Ore oru red row irundha kooda batch Pending la nikkum, adutha ella run-um block aagum. Clear panna DB edit dhaan vazhi, adhanaala dev ticket venum.\n- Window-match fix (8fa51ac) git la commit irukku, aana code la apdiye dhaan irukku.\n- `.env` (plaintext password oda) and patient name irukkura log file rendum git la commit aagirukku. Password-a rotate pannunga.\n- Log la irukkura lines indha code ezhudhave mudiyaadhu. Adhanaala deploy aana build indha repo illa pola.\n\n**Assumption:** \"Ops access\" answer bot screens pathi mattum dhaan sonnuchu. Adhanaala ops ClaimDesk-a normal user-aa open pannuvaanga nu eduthukitten. E01/E02/E03 la \"ClaimDesk la check pannunga\" step adha nambi dhaan irukku. Ops ku ClaimDesk access illana sollunga, andha rows-a Dev ku maathidren.\n\nQUESTIONS: Escalation: who does ops call for a dev issue, and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open \u2014 the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions mean where the code doesn't make it clear?",
"stderr": ""
}
tests/evals/results/handoff/h-desktop-poster-screens-r5.md
+202
View File
@@ -0,0 +1,202 @@
# ClaimDesk Poster - ops handover
Version 1.4.2 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
> This document was written by reading the bot's code and history. The bot was not run to produce it.
## 1. What the bot does
ClaimDesk Poster posts insurance remittance payments into the **ClaimDesk** desktop app, one clinic at a time. Ops start it from the **ClaimDesk Poster** launcher window, which hands it the input Excel. The Excel has three sheets: **Login** (the ClaimDesk username and password), **Clinics** (the clinics to work) and **Settings** (the check-date cutoff). For each clinic the bot signs in to ClaimDesk, opens the clinic and goes down the remittance list. It posts and saves every ACH payment dated on or before the cutoff. It skips other payment methods and defers later dates. Each remittance it handles becomes one row on the **Report screen** of the launcher. The bot produces no Excel, PDF or email output. The Report screen is the only result ops can see.
## 2. How it works
```
ClaimDesk Poster launcher (the window you use; title bar shows the version)
└─ starts → Posting bot (runs in the background, no window of its own)
│
├──drives──> ClaimDesk desktop app
│ (opened, signed in and force-closed by the bot
│ for every clinic - do not touch while it runs)
│
├─ writes → bot database ──read by──> Report screen (in the launcher)
└─ writes → run log (developers only)
```
- **Launcher:** where ops start a run and open the Report screen. Its title bar shows the version.
- **Posting bot:** does the work. It has no window and no Stop button.
- **ClaimDesk:** the app the bot drives with the real mouse and keyboard. The bot opens it fresh for every clinic and force-closes it afterwards.
- **Bot database:** holds the batch, the clinics and one row per remittance. Ops see it only through the Report screen.
- **Run log:** one file per run, which only developers can open.
**One run, step by step:**
1. The bot refuses to start if the previous batch was never completed (see E08).
2. It reads the Excel and creates a new batch. Every clinic starts as Pending.
3. For each clinic, it opens ClaimDesk, signs in and clicks the clinic in the clinic list.
4. It waits for the remittance list, then handles each line. The result is one of: posted, skipped (not ACH), deferred (after the cutoff), rejected by ClaimDesk, or error. A row appears on the Report screen.
5. It force-closes ClaimDesk and moves on to the next clinic.
6. The batch becomes Completed only if every clinic finished and **no row is red**. Otherwise it stays Pending.
## 3. Before a run
- [ ] Nobody needs ClaimDesk on this machine during the run. The bot force-closes **every** ClaimDesk window on the machine after each clinic, including one a person has open.
- [ ] No other window has "ClaimDesk" in its title. That includes database or SQL tools, browser tabs and file explorer windows. The bot picks the first window it finds with "ClaimDesk" in its title.
- [ ] The machine stays unlocked, with the screen on, for the whole run. If the session is locked, minimised or disconnected, the bot's clicks have nowhere to land.
- [ ] **The previous batch is finished.** If any row from the last run is red, or a clinic did not finish, the new run is refused (E08). Clear that first through a ticket.
- [ ] **Login sheet:** username in A2 and password in B2, both filled in and correct. The bot does **not** detect a wrong password (E07). Each wrong attempt is a failed ClaimDesk login, and there can be 3 per clinic.
- [ ] **Clinics sheet:** **one clinic per row** in column A, starting at row 2. Never use commas or semicolons, because the bot splits on them.
- Bad: `Clinic A, Clinic B` in one cell.
- Bad: `Clinic C, Branch 2`. The bot reads this as two clinics, and neither exists.
- Good: `Clinic A` in A2 and `Clinic B` in A3.
- [ ] **Clinic spelling:** exactly as the clinic link reads in ClaimDesk, including `&`, capitals and spaces. No empty cells between clinics. Don't list a clinic twice, because the bot does not remove duplicates and will run it twice.
- [ ] **A clinic whose real name contains a comma cannot be run by this version.** Post it by hand and raise a ticket.
- [ ] **Settings sheet:** the cutoff date is in B1. The bot compares the cutoff as text, not as a date, so Deferred decisions near the cutoff can be wrong. Until dev fixes this, spot-check the first run after changing the cutoff (see E05).
- [ ] The launcher title bar shows version **1.4.2** (see section 10).
## 4. During a run
- **Hands off the machine.** The bot clicks with the real mouse. A click or keypress from a person can land anywhere, including on a payment screen.
- Don't open ClaimDesk, or any window with "ClaimDesk" in its title, on this machine.
- **To stop the bot:** this version has no Stop button.
- Closing ClaimDesk does **not** stop it. The bot treats that as a failed clinic and opens ClaimDesk again, up to 3 times per clinic.
- The code does not show whether closing the launcher stops the bot. After closing the launcher, watch for about 2 minutes. If ClaimDesk opens again, the bot is still running: raise a ticket.
- Any stop in the middle of a run leaves the batch Pending, and the next run will be refused (E08).
- **What normal progress looks like:** ClaimDesk opens, signs in, works through one clinic and closes. Then it opens again for the next clinic, and new rows appear on the Report screen.
- **When it looks stuck:**
- The bot kills itself after **5 minutes with no remittance handled** (E09). Wait at least 5 minutes before acting.
- After that, **ClaimDesk still open and not moving** means the run died: raise a ticket.
- **ClaimDesk closed and no longer reopening** means the run ended, whether it finished or failed.
## 5. Where to look when something goes wrong
Ops can open the launcher and its Report screen. The bot writes no output files, and the database and run logs are developer-only.
| Question | Where to look |
|---|---|
| Did the run finish? | Not shown on any screen ops can open (fix item 4). **Sign it has ended:** ClaimDesk has closed and does not reopen within 2 minutes. **Sign it ended badly:** ClaimDesk is left open, or the next run is refused (E08). |
| Which remittances failed? | Report screen → choose the batch → rows with File Status **Error** (red). |
| Which remittances were **not posted** although green? | Report screen → green rows whose Comments say **"Remittance Errors on posting"** (E03) or **"... - not posted"** (E04). |
| Why did one remittance fail? | Report screen → Comments column. Then use the exception catalogue (section 7). |
| Which clinic failed completely? | A failed clinic has **no rows at all** on the Report screen (E06). Compare the Report screen against your Clinics sheet. A clinic with no rows either had nothing to post or failed. Check ClaimDesk: unposted ACH remittances still listed for it means it failed. |
| Did the bot already retry? | Not visible to ops. A clinic is always tried 3 times before it is given up. Single remittances are never retried. |
| Which version is running? | The launcher title bar. |
## 6. Statuses
The Report screen shows **File Status** (with a colour) and **Comments**. Three different results all show as green Completed, so **always read the Comments column**.
| Status (File Status - Comments) | Colour | Meaning | What ops does |
|---|---|---|---|
| Completed - "Posted successfully" | green | Payment posted and ClaimDesk showed "Payment saved". | nothing |
| Completed - "Payment method '&lt;method&gt;' - not posted" | green | **Not posted on purpose.** Only ACH payments are posted. | nothing, unless the client wants that method posted by hand |
| Completed - "Remittance Errors on posting" | green | **Not posted. Needs follow-up.** ClaimDesk showed its Remittance Errors screen and the bot cancelled. | Review and post it by hand in ClaimDesk (E03) |
| Deferred - "Check date &lt;date&gt; is after the cutoff - deferred" | grey | Not posted. The check date is after the Settings cutoff. | nothing; a later run picks it up (see E05 for the caveat) |
| Error - "Save button not found" | red | The payment screen opened but had no Save button. Nothing was saved. | E02 |
| Error - "Payment not confirmed after Save" | red | The bot clicked Save but saw no "Payment saved" within 20 s. **It may really have posted.** | E01: check in ClaimDesk **before** anything else |
| Pending | amber | The screen supports this colour, but this version never writes it on a row. | If you see it, raise a ticket |
| Clinic Pending / Success | not shown | Kept per clinic in the database only. Pending means the clinic failed all 3 attempts or never ran. | See E06 |
| Batch Pending / Completed | not shown | Kept per batch in the database only. Pending means a red row or a failed clinic remains. A Pending batch blocks the next run. | See E08 |
## 7. Exception catalogue
Quote the ID (E01, E02, ...) in your ticket.
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Red row, Error - "Payment not confirmed after Save" | After clicking Save it waits 20 s for ClaimDesk's "Payment saved" message. If the message never appears, it writes Error and moves to the next remittance. It does not check again. | Report screen, Comments | System | **Ops, then Dev** | 1. **Check in ClaimDesk whether the payment posted.** 2. Do **not** post it again until you know. Re-posting can create a double payment. 3. If it did not post, post it by hand. 4. Ops cannot change the row, and this red row blocks the next run (E08). | Always. Raise a ticket so dev corrects the row. Send the batch ID, clinic, check number, and whether it posted in ClaimDesk. |
| E02 | Red row, Error - "Save button not found" | It waits 10 s for the Save button on the payment screen. If the button is missing, it writes Error and goes to the next remittance, leaving that screen open. | Report screen, Comments | System | **Ops, then Dev** | 1. In ClaimDesk, check that this remittance was not posted. 2. Check the **next few rows** of the same clinic as well. The bot carried on from the screen it left open, so they may be wrong. 3. Post by hand as needed. | Always. Send the batch ID, clinic, check numbers and time. |
| E03 | Green row, Completed - "Remittance Errors on posting" | ClaimDesk showed its "Remittance Errors" screen within 5 s of Post Payment. The bot clicked Cancel and moved on. | Report screen, Comments (the row is green, but nothing was posted) | Business | **Ops** | Open the remittance in ClaimDesk, review the errors and post it by hand, or follow the client's process. | The same remittance keeps coming back as rejected after it has been corrected. |
| E04 | Green row, Completed - "Payment method '&lt;method&gt;' - not posted" | The payment method in the remittance list is not ACH. | Report screen, Comments | Business | Bot | Nothing. The bot posts only ACH. Post it by hand if the client wants non-ACH posted. | ACH payments show up here. |
| E05 | Grey row, Deferred - "Check date ... is after the cutoff - deferred" | The check date text is compared with the cutoff in Settings B1 **as text, not as a date**. | Report screen, Comments | Business | Bot, then Ops | Nothing normally. A later run picks it up. After changing the cutoff, spot-check a few rows: a date before the cutoff should be posted, not Deferred, and a later date should not be posted. | Rows are deferred or posted on the wrong side of the cutoff. Send the batch ID, the cutoff value and two example rows. |
| E06 | A clinic has **no rows** on the Report screen, and ClaimDesk still lists unposted ACH remittances for it | The bot tried the clinic 3 times and every attempt failed at one of: ClaimDesk login window not shown within 60 s; clinic link not found within 10 s; a different clinic opened; remittance list not loaded within 60 s; ClaimDesk window not found. After 3 failures it skips the clinic and goes on to the next. | **Not on any ops screen.** The clinic stays Pending in the database, and the reason is in the run log. | Business (spelling) or System (ClaimDesk slow or down) | **Ops, then Dev** | 1. Check the clinic's spelling in the Clinics sheet against the link in ClaimDesk, including `&`. 2. Check for commas or semicolons (section 3). 3. If the spelling was wrong, fix it. The batch still needs clearing (E08). | The spelling is exact, or the clinic name contains a comma. Send the batch ID, clinic name and run start time. |
| E07 | ClaimDesk opens and closes again and again, but **no clinic gets any rows** | Not identified. The bot does not check whether sign-in worked. A wrong password shows up as "clinic not found" for every clinic, 3 attempts each. The same pattern appears if the bot's ClaimDesk path setting is missing on this machine. | Not on any ops screen; run log only | Business (password) or System (setup) | **Ops, then Dev** | 1. Stop the run (section 4). 2. Check the username and password in Login A2/B2 by signing in to ClaimDesk by hand. 3. Watch for an account lockout. | The login works by hand. Send the batch ID, run time and machine name. |
| E08 | A new run is refused, or does nothing. The bot's message is "A Pending batch exists - finish or clear it before a new upload" (where it shows depends on the launcher). | At start, the bot refuses if **any** earlier batch is still Pending. A batch stays Pending if it has any red row (E01, E02), a failed clinic (E06), or a run that was stopped or killed (E09). | Not on any ops screen; batch status is in the database only | System | **Dev** | Find the blocking items on the Report screen of the last batch: red rows and missing clinics. Sort out each one in ClaimDesk (E01, E02, E06), then raise a ticket to clear the batch. Ops have no button to clear it. | Always. Send the last batch ID and the list of red rows and missing clinics. |
| E09 | The run stops halfway. ClaimDesk is left **open and idle**, and later clinics have no rows. | The bot has a 5-minute no-progress limit, and the clock resets only when a remittance is handled. Slow sign-ins, clinics with nothing to post, and a clinic failing 3 times all count against it. When the limit passes, the bot kills itself at once, without closing ClaimDesk or finishing the batch. | Not on any ops screen; run log only | System | Bot, then **Dev** | 1. Close ClaimDesk. 2. On the Report screen, note the last clinic and check number that has a row. 3. In ClaimDesk, check that this remittance and the next one are not half-posted. 4. The batch is now Pending (E08). | Always. Send the batch ID, the time ClaimDesk stopped moving, and the last row shown. |
| E10 | Nothing happens after starting a run: ClaimDesk never opens and no batch appears | The Excel failed the bot's start checks: the Login sheet is missing, Login A2 or B2 is empty, or the Clinics or Settings sheet is missing. | Not on any ops screen and not in the run log. The launcher may show the message. | Business | **Ops** | Check that the Excel has the three sheets **Login**, **Clinics** and **Settings** with exactly those names, and that Login A2 and B2 are filled in. Start again. | The Excel is correct and the bot still does nothing. Send the time and a screenshot of the launcher. |
| E11 | A ClaimDesk window that a person had open on the bot machine suddenly closes | After every clinic the bot force-closes **all** ClaimDesk windows on the machine. | Nowhere | System | Ops | Don't use ClaimDesk on the bot machine during a run. Re-check any work that was open in ClaimDesk when it closed. | - |
| E12 | The bot types into or clicks in the wrong window (a database tool, a browser, the launcher), or ClaimDesk closes as soon as it opens | The bot attaches to the first window whose title contains "ClaimDesk". | Not on any ops screen; often appears as E06 | System | Ops, then **Dev** | Stop the run (section 4). Close every other window with "ClaimDesk" in its title, then check the batch (E08). | It happens with no other "ClaimDesk" window open. Send the time and a screenshot. |
| E13 | The same check number appears twice in one batch on the Report screen | When a clinic is retried, the bot goes through the list again and writes new rows for items it already skipped or deferred. | Report screen | System | Ops | Duplicate **Skipped** or **Deferred** rows are harmless. | The same check shows "Posted successfully" twice. Check ClaimDesk for a double payment and send the batch ID and check number. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| A clinic fails at any step: ClaimDesk won't start or sign in, the clinic is not found, the wrong clinic opens, the list won't load, or ClaimDesk is closed | Force-closes ClaimDesk, signs in again from scratch and retries the whole clinic | 3 attempts per clinic. Each attempt can take up to about 2.5 minutes of waiting (60 s sign-in + 10 s clinic + 60 s list). | Clinic skipped with no rows (E06), next clinic. The batch will stay Pending. |
| "Remittance Errors" screen | Clicks Cancel and moves on | 1 (no retry) | Green row "Remittance Errors on posting" (E03) |
| Save button missing / save not confirmed | Nothing. No retry. | - | Red row (E02, E01), next remittance |
| No progress at all | Kills the whole run | 5 minutes with no remittance handled, checked every 10 s. Can trigger during the clinic retries above. | Run ends with ClaimDesk open and the batch Pending (E09) |
| Run stopped or killed halfway | **No resume.** The next run will not continue where it left off. | - | The batch must be cleared by dev (E08) |
**How long to wait:** Act after **5 minutes** with no new row on the Report screen and ClaimDesk sitting still. Before that, the bot may still be retrying.
## 9. Known incidents
Dates are unknown for the four code-history items (they all came in one import). They are listed newest first by history order.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 1.3.0 → 1.4.x | Users still on 1.3.0 after an update | The launcher kept the old program folder until the machine was restarted | After an update, restart the machine and check the title bar (section 10) | Yes |
| before 1.4.2 (also in the 2026-09-01 run) | Red "Payment not confirmed after Save" rows that had really posted | ClaimDesk took longer than 10 s to show "Payment saved" | The wait was raised to 20 s, but it is still a fixed wait, so this can recur on a slow day. Always check in ClaimDesk first (E01). | Partly |
| before 1.4.2 | Clinics with a comma in the name were never processed | The bot splits clinic cells on commas and semicolons | One clinic per row. A real comma in a clinic name still can't be handled (section 3). | Partly |
| before 1.4.2 | The bot drove a SQL browser window, then closed the real ClaimDesk | Window match on "ClaimDesk" in the title | Close every other "ClaimDesk" window before a run. **Still open in the code** (E12). | Partly |
| 2026-09-01 | "Wrong clinic opened" for a clinic with `&` in its name. The bot retried, then the whole run was killed after 5 minutes without progress. The last clinic never ran. | The clinic check did not match a name with `&`, and the no-progress limit does not count sign-in and clinic retries as progress. | Not fixed. The batch stays Pending (E08, E09). Spell `&` names exactly as ClaimDesk shows them, and raise a ticket if it repeats. | Partly |
## 10. Releases and version check
- **How an update arrives:** through the launcher. Its update code is not part of this repository.
- **How to confirm it applied:** the launcher title bar shows the new version. This document is for **1.4.2**.
- **If the version did not change:** restart the machine, open the launcher and check again. If it still shows the old version, raise a ticket with the machine name and the version shown.
- Each machine keeps its own bot database. Batches and the Report screen are per machine, so check results on the machine that ran the batch.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue. Include:
- the exception ID from section 7 (for example E01)
- the batch ID (as chosen on the Report screen)
- the clinic name and check number(s)
- the time of the failure and the machine name
- a screenshot of the Report screen
- for E01, E02 and E09: what you found in ClaimDesk (posted or not)
Dev will read the run log and the database themselves.
**Never send:** passwords (including the Login sheet), the bot's settings file, or patient or payment details in chat. Refer to rows by batch ID, clinic and check number only.
## 12. Failure-class check
- **F1** Missing prerequisite: covered in section 3 (Excel sheets, Login cells, version), E07 (ClaimDesk path setting missing), E10 (missing sheets), fix items 14 and 18.
- **F2** Wrong window / focus: covered in sections 3 and 4, E11, E12, fix items 7 and 8.
- **F3** Target app slower or different: covered in E01, E02, E06, section 8 (every wait and limit), section 3 (keep the session unlocked), fix items 11 and 12.
- **F4** Input read differently: covered in section 3 (commas, `&`, duplicates, cutoff as text), E05, E06, fix items 9 and 10.
- **F5** Marked done when nothing was done: covered in fix items 15 and 20 (a clinic is marked Success with no positive check; a crash after Save leaves no row).
- **F6** Action happened but not confirmed: covered in E01 (check in ClaimDesk first; dev corrects the row), fix item 11.
- **F7** Business exception shown as success: covered in section 6 ("Remittance Errors on posting" and "not posted" are green), E03, E04, fix item 1.
- **F8** Endless re-runs / watchdog: no endless loop (3 attempts per clinic). The watchdog kill is covered in section 8, E09 and fix item 6.
- **F9** Stopping the wrong way: covered in section 4 (closing ClaimDesk does not stop it), fix item 19.
- **F10** One stuck item blocks the queue: covered in E08 (any red row or failed clinic blocks all later runs), fix item 5.
- **F11** Cause hidden from ops: covered in section 5, fix items 2, 3, 4 and 14.
- **F12** Update didn't arrive / old build undoes new: covered in section 10 and fix item 17. Nothing is written to shared resources on start, because each machine has its own database.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | "Rejected by app" (ClaimDesk rejected the remittance, nothing posted) and "Skipped" are stored as Completed (green) | Ops read green as done; rejected remittances get no follow-up | `bot.py:27-33` STATUS_MAP |
| 2 | Report screen hides the `status` and `tech_comments` columns | Rejected, skipped and posted rows look identical apart from the comment; the technical cause is invisible | `report_screen.py:6` |
| 3 | A clinic that fails all 3 attempts writes no report row, only a log line | Ops can't see failed clinics; they look the same as clinics with nothing to post | `bot.py:143-147` |
| 4 | Batch status and clinic status are not shown anywhere ops can open | Ops can't tell whether a run finished, or why the next run is refused | `bot.py:170,178`; `report_screen.py:11-13` |
| 5 | Any red row or Pending clinic keeps the batch Pending, and a Pending batch refuses every later run. There is no clear/edit function; the only fix is a manual DB edit. | One unconfirmed save stops all posting until dev intervenes | `bot.py:150-154`, `bot.py:167-168` |
| 6 | The watchdog clock resets only per item (and starts at import), so sign-in, clinic retries and zero-item clinics count as "no progress". `os._exit` skips cleanup, leaving ClaimDesk open and the batch Pending. | Whole runs are killed by one slow clinic; later clinics never run | `bot.py:36`, `bot.py:44-49`, `bot.py:137` |
| 7 | Window match is a title substring. It also matches the launcher "ClaimDesk Poster", the "ClaimDesk Login" window and SQL tools. The 8fa51ac fix is only a comment; the code is unchanged. | The bot can drive the wrong window | `bot.py:66-70`, `bot.py:185` |
| 8 | `taskkill /im ClaimDesk.exe /f` after every clinic kills every ClaimDesk on the machine | Users lose open work | `bot.py:146` |
| 9 | Clinic cells are split on `,` and `;`. There is no duplicate check, and empty names come from trailing commas. | Clinics with a comma in their name can never run; duplicates run twice | `bot.py:60-62` |
| 10 | The cutoff is compared as text (`str(cutoff)` vs the grid text) | Deferred/posted decisions near the cutoff can be wrong | `bot.py:107` |
| 11 | "Payment not confirmed" is a fixed 20 s wait with no re-check of the grid or payment afterwards | False red rows that really posted; risk of double-posting on manual re-post | `bot.py:117-118` |
| 12 | "Post Payment" is clicked without selecting the grid row. After Rejected/Error the screen is not reset before the next item, and the grid is read by column position, which a header row breaks. | A row's check number may not match what was actually posted; items after an error act on a stale screen | `bot.py:95-119` |
| 13 | A failed sign-in is not detected. It surfaces as "Clinic not found", with 3 sign-ins per clinic. | Wrong password causes many failed logins and a possible account lockout | `bot.py:73-83` |
| 14 | Start-up errors are not logged: missing sheets, an empty Login, and "A Pending batch exists". A missing `CLAIMDESK_EXE` becomes a per-clinic error. | Ops see "nothing happened" with no message on file | `bot.py:52-63`, `bot.py:74`, `bot.py:167-168` |
| 15 | The report row is written only after the post completes, so a crash or watchdog kill between Save and the write leaves no record | A payment can be posted with no trace on the Report screen | `bot.py:116-139` |
| 16 | A clinic retry re-writes rows for items already handled in the failed attempt | Duplicate rows on the Report screen | `bot.py:136-139` |
| 17 | The sample run log contains lines this code cannot write, including **patient names**, so the deployed build is not this repo's code. The bot also writes a `BOT_WAS_RUN` marker file into the program folder. | This doc may not match what is running; old logs hold PHI and must be handled as such. **That log is committed to git**, so the PHI is in the repo history. | `logs/run_20260901_101500.log`; `bot.py:161` |
| 18 | `.env` holds an unused plaintext password entry and an unused alert setting next to the code | Credential exposure: **`.env` is committed to git**, so the password is in the repo history and should be rotated. There is no alerting despite the setting. | `.env` (key names only; values not copied) |
| 19 | Launcher code (start, stop, update) is not in this repo. There is no Stop button, and it is unknown whether closing the launcher stops the bot. | Ops have no safe way to stop a run | `bot.py:3` (only reference) |
| 20 | The "right clinic" check is the clinic name as a substring of the window title (fails on `&`/encoding). A clinic is marked Success with no check that the list really loaded its items. | Correct clinics are retried and abandoned; an empty or half-loaded list is marked Success | `bot.py:91`, `bot.py:96-101`, `bot.py:140` |
tests/evals/results/handoff/h-mail-intake-full-r1.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-mail-intake/full",
"rep": 1,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"stuck Processing item blocks the queue": true,
"20 MB limit": true
},
"cost_usd": 1.3510176,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-mail-intake-full-oh8t63qd\\project",
"reply": "`HANDOVER.md` ready, repo root-la irukku. Bot-a run pannala, vera entha file-um change pannala.\n\n**Summary:**\n\n| Item | Count |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 8 (E01\u2013E08) |\n| Failure classes | 11 covered, 1 not applicable (F2: bot entha window-um drive pannala). F12-oda \"shared resources rewritten on start\" part-um not applicable |\n| Fix before handover | 20 items |\n\n**Mukkiyamaana findings:**\n- Moonu \"fix\" commits-um code-ah maathala, comment mattum add pannirukku. So moonu incidents-um innum open.\n - Stuck Processing item vandha ella run-um block aagum.\n - Encrypted PDF \"Done\"-nu kaattum, aana copy aagaathu.\n - Mailbox password expire aana, adhu log-la varaathu.\n- `.env` (credentials) and `logs/intake.log` git-la commit aagirukku. Log-la oru sender email address irukku. Mailbox password-a rotate pannanum.\n- Ops ungalukku new mail-a mailbox-la open panna, adhu read-nu mark aagidum. Appuram bot andha referral-a skip pannidum.\n\n**Fix before handover (20):**\n1. Stuck Processing item blocks every later run\n2. A damaged PDF or missing temp file crashes the run\n3. Encrypted PDF shows Done without being copied\n4. The final mailbox failure is never logged\n5. Crash tracebacks never reach the log\n6. A crash while mail is being fetched loses referrals\n7. Mail with no attachment, or mail someone opened first, leaves no record\n8. Attachments with the same name overwrite each other; file names are not sanitised\n9. Failed and Quarantined items are never retried, and re-queueing needs a database edit\n10. `queue_viewer.xlsx` is missing from the repo\n11. No version string\n12. No run start/end log line\n13. Credentials and the log are committed to git\n14. No timeouts on the mailbox connection or the share copy\n15. No lock against overlapping runs\n16. No setup step for folders and the queue table\n17. Signature images clutter the queue as Quarantined\n18. `data/tmp/` is never cleaned\n19. Hard-coded fallback share path\n20. Ops access to Task Scheduler not confirmed\n\n**Checks:**\n- Ellaa status values-um doc-la irukku.\n- Code refs (file:line) section 13-la mattum dhaan irukku.\n- PHI, IP, credentials doc-la illa.\n\n**Gaps:**\n- `queue_viewer.xlsx` repo-la illa. Adhanaala on-screen colours, viewer columns verify panna mudiyala.\n- Ops-ukku Task Scheduler access irukka-nu therila. Bot-a stop panradhu (section 4) and E06 rendum adha depend pannudhu (item 20).\n\nQUESTIONS: Who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Are there any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-mail-intake-full-r1.md
+183
View File
@@ -0,0 +1,183 @@
# Mail intake bot - ops handover
Version: none (the project has no version number - see section 13, item 11) · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every 10 minutes Windows Task Scheduler starts the bot. It reads every **unread** email in the referrals mailbox, saves each attachment, and checks that it is a PDF of 20 MB or less that can be opened. Good PDFs are copied to the **shared intake folder**. Every attachment becomes one item in the bot's queue with a status and a note, which ops read in the queue viewer workbook. The bot has no window of its own and does not drive any desktop app. It only talks to the mailbox and the shared intake folder.
## 2. How it works
```
Windows Task Scheduler (every 10 minutes)
└─ starts → Mail intake bot (no window, runs a few seconds to minutes)
├─ reads → referrals mailbox (unread mail only; marks it read)
├─ saves → data/tmp/ (a working copy of every attachment)
├─ copies → shared intake folder (good PDFs only)
├─ writes → the queue (one item per attachment: status + note)
│ └─ ops read it in queue_viewer.xlsx (Data > Refresh)
└─ writes → logs/intake.log
```
- **Task Scheduler** decides when the bot runs. Nothing runs between the 10-minute starts.
- **The bot** is a background script. There is no screen to watch while it works.
- **Referrals mailbox:** the bot only sees mail that is still unread. As soon as it reads a mail, the mail is marked read and the bot never looks at it again.
- **data/tmp/** holds a copy of every attachment the bot downloaded. Files are never deleted from here.
- **Shared intake folder:** where the good PDFs end up. The file keeps the name it had in the email.
- **The queue** is the bot's list of items. Ops see it through **queue_viewer.xlsx** (columns id, sender, subject, status, note).
- **logs/intake.log** is the bot's only log file. One line per finished item, plus warnings and errors.
**One run, step by step:**
1. The bot checks the queue. If any item is still **Processing** from an earlier run, it writes an error to the log and stops. It does **not** read mail (see E02).
2. It signs in to the mailbox. If that fails it tries 3 times, 30 seconds apart (see E01).
3. It reads every unread mail. Each attachment is saved to `data/tmp/` and added to the queue as **New**.
4. It takes each New item in order and sets it to **Processing**.
5. It checks the file: PDF, 20 MB or less, not password-protected.
6. It copies the file to the shared intake folder and sets the final status and note.
7. It writes one log line per item: `item <id> -> <status> (<note>)`.
The log has no "run started" or "run finished" line.
## 3. Before a run
The bot runs by itself, so "before a run" means "before you hand the machine over, and after any change to it".
- [ ] The scheduled task for the bot exists and is enabled in Task Scheduler, set to every 10 minutes.
- [ ] The settings file `.env` is in the bot's folder and has the mailbox server, mailbox user, mailbox password and shared-folder entries. If an entry is missing, every run crashes and **nothing** is written to the log (E06).
- [ ] The folders `data/`, `data/tmp/` and `logs/` exist in the bot's folder, and the queue file is in `data/`. The bot does not create them. If one is missing, every run crashes silently (E06).
- [ ] The machine can reach the shared intake folder. Open it in Explorer as the account the scheduled task runs as. If you can't, items end as Failed (E05).
- [ ] The mailbox password has not expired. When it expires, the log fills with "IMAP connect failed" warnings (E01). Ask the RPA dev team when it is next due.
- [ ] What the bot accepts. Ops can't control what senders send, but this tells you what to expect:
- Accepted: a PDF ending in `.pdf` (any case), 20 MB or less, with no password. Example: `referral_1023.pdf`, 2 MB.
- Quarantined: anything else, including Word files, photos, signature logos and PDFs over 20 MB. Example: `scan.jpg` or `image001.png` from an email signature.
- Lost or overwritten: two attachments with the **same file name**. The second one overwrites the first in the shared intake folder (E08).
## 4. During a run
- **Don't open new mail in the referrals mailbox.** Opening a mail in Outlook or webmail marks it read, and the bot then skips it for good (E07). If you opened one by mistake, mark it **unread** again. The next run picks it up.
- Don't move, rename or open files in `data/tmp/`. Each queue item points at its file there.
- Don't edit or rename files in the shared intake folder while the bot is copying. A run is usually short, but there is no "finished" signal.
- **To stop the bot:** in Task Scheduler, **disable** the bot's task. This stops future runs. Don't use "End" on a run that is in progress: stopping it mid-item leaves that item in Processing, and that blocks every later run (E02). If you don't have access to Task Scheduler, ask in #rpa-support.
- **A run looks stuck when:**
- the log shows `ERROR Previous item still Processing - aborting` again every 10 minutes, or
- the queue viewer shows an item in Processing for more than 10 minutes.
- **The bot does not recover from this by itself.** There is no watchdog and no time limit: the block lasts until a developer releases it. Escalate straight away (E02).
## 5. Where to look when something goes wrong
Ops can open: the queue viewer (`queue_viewer.xlsx`), the shared intake folder, and the `logs/` folder. Ops cannot open the queue database itself.
| Question | Where to look |
|---|---|
| Is the bot running at all? | `logs/intake.log`: the time of the last line. Runs with no new mail write nothing, so a quiet log alone is not an error. Send a test mail with a small PDF to the referrals mailbox. Within about 10 minutes, a new `item <id> -> Done` line should appear. |
| Did the last run finish? | There is no "finished" line. If no item in the queue viewer is in **Processing** and the log does not repeat `Previous item still Processing`, the last run finished. |
| Which items failed? | Queue viewer → Data > Refresh → status column: **Failed** and **Quarantined**, and also **Done** items whose note says `encrypted - not readable` (these were not delivered, E04). |
| Why did one item fail? | Queue viewer → note column. Then `logs/intake.log`, search for `item <id> ->`. |
| Did the bot already retry? | Mailbox sign-in: lines `IMAP connect failed (1/3)`, `(2/3)`, `(3/3)` in `logs/intake.log`. Items are **never** retried: Failed and Quarantined items stay that way. |
| Is the file really in the shared intake folder? | Open the shared intake folder and look for the file name from the item. Compare the page count with the note (`<n> pages saved`). |
| A referral never showed up in the queue | See E07. The bot keeps no record of mail it read without queueing anything. |
| Which version is running? | You can't see it. The project has no version number (section 10). |
## 6. Statuses
The bot sets no colours. The queue viewer shows the status as plain text in the status column.
| Status (note) | Colour | Meaning | What ops does |
|---|---|---|---|
| New (empty note) | none | Attachment saved and queued, not yet checked. | Nothing. It is handled in the same run or the next one. If it is still New after 20 minutes, see E02 and E01. |
| Processing (empty note) | none | The bot is working on this item right now. | Nothing for the first 10 minutes. **Still Processing after 10 minutes = the bot crashed on this item and every later run is blocked** → E02. |
| Done - "`<n>` pages saved" | none | Copied to the shared intake folder. | Nothing. |
| Done - "encrypted - not readable" | none | **Needs follow-up.** The PDF is password-protected. The status says Done, but the file was **not** copied to the shared intake folder. | E04: ask the sender for a copy without a password. |
| Failed - "share unreachable: `<reason>`" | none | The copy to the shared intake folder failed. The bot never tries again. | E05. |
| Quarantined - "not a PDF or over 20 MB" | none | Not a PDF, or larger than 20 MB. Not copied. The bot never tries again. | E03: usually nothing for signature logos. For a real referral, ask the sender for a PDF under 20 MB. |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | No new items arrive. The log shows `WARNING IMAP connect failed (1/3)`, `(2/3)`, `(3/3)` and then nothing more for that run. | Signing in to the mailbox fails. It tries 3 times, waiting 30 seconds after each try. After the third failure the run stops. That last failure is **not** written to the log. | `logs/intake.log` warning lines only. The text after the colon gives the reason: "getaddrinfo failed" means network or name lookup; an authentication or login message means a wrong or expired password, or a locked account. | System | Bot, then Dev | Wait for the next run (10 minutes later). A one-off network blip clears by itself. If the reason mentions login or authentication, the mailbox password has probably expired or the account is locked. | Three runs in a row (about 30 minutes) each end at `(3/3)`, or any login/authentication reason. Send: the time of the first `(1/3)` line, the reason text from the warning (never the password), and `logs/intake.log`. |
| E02 | Nothing new for a long time. The log shows `ERROR Previous item still Processing - aborting` every 10 minutes. One item in the queue viewer is stuck in **Processing**. | At the start of every run, the bot looks for any item still in Processing. If it finds one, it stops at once without reading mail. This happens after a crash in the middle of an item, for example on a damaged PDF, a missing file in `data/tmp/`, or someone ending the task. | `logs/intake.log` error line; the queue viewer status column. | System | **Dev** | The bot will **not** clear this by itself, and ops can't release it (it needs the database). Note the id, sender and subject of the Processing item. Check whether its file is already in the shared intake folder. Meanwhile, new mail waits safely in the mailbox as unread. | Always, as soon as you see the line twice. Send: the item id, the time the first error line appeared, whether the file is in the shared intake folder, and `logs/intake.log`. |
| E03 | Item **Quarantined**, note `not a PDF or over 20 MB`. | The file name does not end in `.pdf`, or the file is larger than 20 MB. | Queue viewer status and note; log line `item <id> -> Quarantined`. | Business | Ops | Look at the subject and sender. Signature logos and inline pictures (for example `image001.png`) also land here: ignore them. For a real referral, ask the sender to resend it as a PDF under 20 MB. The resend arrives as a new item. | The file really is a PDF under 20 MB and was still quarantined. Send: the item id and the file name. |
| E04 | Item **Done**, note `encrypted - not readable`, and the file is **not** in the shared intake folder. | The PDF is password-protected. The bot stops there and does not copy it, but it records the status as Done. | Queue viewer **note** column only. The status column alone looks like success. | Business | Ops | Ask the sender for a copy without a password. The new copy arrives as a new item. Check for these every day by filtering the note column on "encrypted". | Never needed for the item itself. The misleading status is on the dev list (section 13). |
| E05 | Item **Failed**, note `share unreachable: <reason>`. | Copying the file to the shared intake folder fails: folder offline, no permission, disk full, or a file with the same name is locked. | Queue viewer status and note; log line `item <id> -> Failed`. | System | Ops, then Dev | 1. Check that you can open the shared intake folder and that it has free space. 2. Look for the file there: a partial copy can be left behind. 3. The bot **never** retries a Failed item, so ask dev to put it back to New once the folder works again. **Don't** ask the sender to resend: that creates a duplicate. | Always, to re-queue it. Also escalate straight away if several items fail in a row. Send: the item ids, the reason text from the note, and the time. |
| E06 | The log stops completely: no new lines at all, not even warnings, although new mail with attachments is waiting. | The bot crashes at start-up because something it needs is missing: an entry in the settings file, the `data/` or `logs/` folder, or the queue file. It writes nothing to the log. | Nowhere ops can see. Only in Task Scheduler's "Last Run Result" for the task. | System | Dev | Check that the scheduled task is enabled (section 3). Send a small test PDF to the mailbox. If no `item` line appears within 20 minutes, escalate. | Always. Send: the time of the last log line, and the Task Scheduler "Last Run Result" if you can see it. |
| E07 | A sender says they sent a referral, but there is no item for it in the queue viewer. | No check. The bot only reads **unread** mail and marks it read straight away. A mail that someone opened first, a mail with no attachment, or a run that crashed while reading mail all leave no item and no log line. | Nowhere. | Business / System | Ops, then Dev | 1. If you can see the mailbox, find the mail. If it is there with an attachment, mark it **unread**: the next run picks it up. 2. If it has no attachment, ask the sender to attach the PDF. 3. Otherwise escalate. | The mail is unread with an attachment and still not queued after 20 minutes, or several referrals are missing. Send: the time sent and the subject (no patient details in chat). |
| E08 | Two items show **Done**, but the shared intake folder holds only one file, or the file doesn't match one of the senders. | No check. The bot saves files under the name from the email, so a second attachment with the same name overwrites the first, both in `data/tmp/` and in the shared intake folder. | Queue viewer (two items, same file name in the subject or sender context); shared intake folder. | System | Ops, then Dev | Ask the sender whose file was overwritten to resend it with a different file name. | Always. Send: both item ids and the file name. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Mailbox sign-in fails | Tries again after 30 seconds | 3 tries, about 90 seconds in total | The run stops silently. The next scheduled run tries again 10 minutes later, forever, with no alert (E01). |
| Run stopped before items were handled | New items left over are picked up | Next scheduled run | n/a |
| Unread mail not yet read | Stays unread in the mailbox | Until a run gets past sign-in | n/a |
The bot does **not** recover from any of these: an item stuck in Processing (E02), a Failed or Quarantined item (never retried), a crash at start-up (E06), or a missing referral (E07). Ops can wait out an E01 for about 30 minutes before escalating. For everything else, act as soon as you see it.
## 9. Known incidents
All three are recorded in the project history on 2026-10-23. Each was marked "fixed", but **the bot's behaviour did not change**: the fix records contain notes only. Treat all three as still open.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-10-23 / no version | No new referrals all day. No errors except the repeating "IMAP connect failed" warnings. | The mailbox password had expired. The bot only writes warnings and never logs that it gave up. | E01: three runs in a row ending at `(3/3)`, or any login reason, → escalate for a password reset. **Still not fixed.** | Partly: the warnings are visible, but nothing alerts anyone. |
| 2026-10-23 / no version | Referrals showed **Done** but never reached the shared intake folder (password-protected PDFs). | Password-protected PDFs are recorded as Done with the note "encrypted - not readable", and not copied. | E04: filter the note column on "encrypted" every day and ask senders for an unlocked copy. **Still not fixed.** | Partly: possible only if ops reads the note, not the status. |
| 2026-10-23 / no version | Nothing reached the shared intake folder for a whole day. The log repeated "Previous item still Processing - aborting". | The bot crashed in the middle of one item. That item stayed in Processing and blocked every later run. Ops reset it by hand. | E02: escalate on the second error line. **Still not fixed.** | Partly: ops can spot it quickly with this doc, but releasing it needs dev (database). |
## 10. Releases and version check
- **How an update arrives:** there is no updater. A developer replaces the bot's files in its folder, and the next scheduled run uses the new files.
- **How to confirm it applied:** ops can't. The bot has no version number and doesn't write one to the log. Ask the developer to confirm in #rpa-support which change went in, and check that the next runs write normal `item <id> -> ...` lines.
- **If the behaviour didn't change after an update:** report it in #rpa-support with the time of the update and a log line that shows the old behaviour.
- Only one machine should run the bot's scheduled task. Two machines reading the same mailbox would split the mail between them.
## 11. Escalating to dev
Post in Teams **#rpa-support**. Send:
- the exception ID from section 7 (for example "E02")
- the item id(s) from the queue viewer
- the time of the first related log line
- the `logs/intake.log` file (attach it; don't paste lines that contain sender addresses)
- whether the file is in the shared intake folder
**Never send:** passwords, the `.env` settings file, patient details, attachments, or the contents of referrals in chat.
## 12. Failure-class check
- **F1** Missing prerequisite: covered in section 3 (settings file, folders, queue file, folder access, password expiry), E06, and section 13 items 5 and 13.
- **F2** Wrong window / focus: not applicable. The bot drives no desktop app or window; it only talks to the mailbox and the file share.
- **F3** Target slower or different than expected: covered in E01 (mailbox, 3 tries x 30 s) and E05 (shared folder). There are no screen timeouts. Section 13 item 14 covers the lack of time limits.
- **F4** Input read differently: covered in section 3 (accepted and quarantined examples, same-name files), section 4 (opening mail marks it read), E03, E07 and E08.
- **F5** Marked done when nothing was done: covered in E04 (Done without copying) and E07 (mail marked read with no item). Section 13 items 3 and 7.
- **F6** Action happened but not confirmed: covered in E02 (file may already be in the shared folder; check before re-queueing) and E05 (partial copy; don't ask for a resend).
- **F7** Business exception shown as success: covered in section 6 (Done - "encrypted - not readable" = needs follow-up), E04, and section 13 item 3.
- **F8** Endless re-runs / watchdog: covered in section 8 (mailbox retries forever, every 10 minutes, with no alert), E01 and E02. There is no watchdog. Section 13 items 1 and 4.
- **F9** Stopping the wrong way: covered in section 4 (disable the task; never end a run mid-item).
- **F10** One stuck item blocks the queue: covered in E02, section 4 and section 6 (Processing). Section 13 item 1.
- **F11** Cause hidden from ops: covered in E06 (crash cause only in Task Scheduler) and E07. Section 13 items 4, 5, 9, 10 and 12.
- **F12** Update didn't arrive: covered in section 10 (no version, no updater; one machine only). Section 13 item 11. The part about shared resources being rewritten on start is not applicable: the bot rewrites nothing shared on start.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | One item left in `Processing` aborts every later run. There is no stale timeout, no auto-reset and no alert. Commit 4d3e78c only added a comment. | The bot stops for hours or days. Ops can't release it without the DB. | `intake.py:74-76`, `intake.py:78-80`, `intake.py:89` |
| 2 | `process()` has no handling for a damaged or unreadable PDF (`PdfReader`) or a missing temp file (`p.stat()`). The run crashes and leaves the row `Processing`, which causes #1. | Turns one bad attachment into a full outage. | `intake.py:59`, `intake.py:61` |
| 3 | An encrypted PDF returns `"Done"` without copying. It should be its own status (for example Quarantined or Failed). Commit 4beebcc only added a comment. | Undelivered referrals look like successes. | `intake.py:62-63`, `intake.py:90` |
| 4 | The final mailbox failure (`RuntimeError("Mailbox unreachable")`) is never caught or logged. A login failure is not told apart from a network failure. There is no alert. Commit 115f761 only added a comment. | Silent outage when the password expires (incident 1). | `intake.py:34-37`, `intake.py:77`, `intake.py:91` |
| 5 | Unhandled exceptions and tracebacks go to stderr only, not to `logs/intake.log`. A missing env var (`KeyError`), a missing folder, or a missing table crash with no log line. | The cause is visible only in Task Scheduler, which ops may not have access to (not confirmed). | `intake.py:23-24`, `intake.py:31-32`, `intake.py:73`, `intake.py:87-88` |
| 6 | Mail is fetched with `RFC822`, which marks it read on the server, but the rows are committed only after the whole loop. A crash during the fetch loses referrals with no trace. | Referrals vanish (E07). | `intake.py:44`, `intake.py:54` |
| 7 | Mail with no attachment, or mail read by a person first, leaves no record anywhere. | Ops can't prove whether a referral was received. | `intake.py:42`, `intake.py:46-49` |
| 8 | The attachment name is used as-is for `data/tmp` and the share. Same names overwrite each other, and the name is not sanitised (a path like `..\` can write outside the folder). | Lost or swapped referrals (E08); a security risk. | `intake.py:50`, `intake.py:65` |
| 9 | `Failed` and `Quarantined` are never retried, and re-queueing needs a DB edit that ops can't make. | Every share outage needs dev time (E05). Add a re-queue option in the viewer or a retry for `Failed`. | `intake.py:66-67`, `intake.py:78` |
| 10 | `queue_viewer.xlsx`, the ops screen named in the README, is not in the repo. Its columns and refresh source can't be checked. | Ops' only view of the queue is unversioned. | `README.md:6` |
| 11 | No version string anywhere, and none written to the log. | Ops can't confirm an update applied (F12). | `intake.py:1-2`, `intake.py:71` |
| 12 | No "run started / run finished" log line, and no count of mails read. | Ops can't tell a healthy quiet run from a dead bot. | `intake.py:71-84` |
| 13 | `.env` (mailbox credentials) and `logs/intake.log` are tracked in git since the initial import. The log contains a sender email address. Rotate the mailbox password, remove both files from history, and add them to `.gitignore`. | Credential and personal-data exposure. | `.env`, `logs/intake.log:6` |
| 14 | No timeout on the IMAP connection or on the share copy, beyond the OS defaults. | A hung run holds an item in `Processing` and causes #1. | `intake.py:31`, `intake.py:65` |
| 15 | No lock against overlapping runs. Two runs can pick up the same `New` rows if Task Scheduler starts one while another is still running. | Duplicate copies; confusing statuses. | `intake.py:74`, `intake.py:78` |
| 16 | Nothing creates `data/`, `data/tmp/`, `logs/` or the `queue` table. On a fresh machine, sqlite silently creates an empty `queue.db` and the run fails. | Setup on a new machine fails silently (E06). | `intake.py:17-18`, `intake.py:23`, `intake.py:73` |
| 17 | Every MIME part with a filename is queued, including inline signature images, which are then Quarantined. | Clutter that hides real quarantines. | `intake.py:46-53` |
| 18 | `data/tmp/` is never cleaned up. | The disk slowly fills with copies of referrals (personal data at rest). | `intake.py:50-51` |
| 19 | The share path falls back to a hard-coded default when `INTAKE_SHARE` is unset. | A misconfigured machine copies to an unexpected location. | `intake.py:19` |
| 20 | Confirm whether ops can open Task Scheduler (the task's history and the Disable option). Sections 4 and E06 depend on it. | Ops may not be able to stop the bot or see crash results. | n/a (access question) |
tests/evals/results/handoff/h-mail-intake-full-r2.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-mail-intake/full",
"rep": 2,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"stuck Processing item blocks the queue": true,
"20 MB limit": true
},
"cost_usd": 1.3097480000000001,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-mail-intake-full-oq0v6p0_\\project",
"reply": "`HANDOVER.md` ready, repo root la iruku \u2014 oru file mattum dhaan maathinen, code edhuvum touch pannala, bot-ah run pannala. Neenga sonna answers use panniten: escalation = RPA dev team on #rpa-support; ops-ku log folder and shared intake folder mattum, database illa; theliyaadha items-ku code-oda best reading.\n\n| Item | Count |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 8 (E01\u2013E08) |\n| Failure classes covered | 11 of 12 \u2014 F2 not applicable (indha bot entha window-um drive pannala) |\n| Fix before handover items | 19 |\n\n**Fix before handover \u2014 mukkiyamaanadhu:**\n- **Moonu \"fix\" commits-um oru comment mattum dhaan add pannirukku.** Stuck-in-Processing, encrypted-PDF, mailbox-password moonu incidents-um innum varum.\n- Oru item Processing-la stuck aana, ovvoru run-um stop aagum, forever. Ops-ku database access illa, so dev dhaan release pannanum.\n- Password potta PDF-ku status **Done** nu varudhu, aana adhu copy aagala.\n- Failed items-ah bot thirumba retry pannadhu.\n- \"Mailbox unreachable\" log-la ezhudhala. Mail fetch pannumbodhu crash aana, antha emails thadayame illaama kaanama pogum.\n- Ore file name-la rendu attachment vandha, rendaavadhu modhalaavadhai overwrite pannum. Path-oda vara file name-um check pannala.\n- `queue_viewer.xlsx` README-la irukku, aana repo-la illa. Ops-ku sender/subject theriyave theriyaadhu.\n- Version number engayum illa.\n- `.env` git-la commit aagirukku \u2014 credentials-ah rotate pannunga. Log-la oru sender email address-um irukku.\n\n**Rendu vishayam theriyala, best reading use panninen:**\n- `BOT_WAS_RUN` marker file edhukku-nu theriyala. Best reading: \"oru thadava run aachu\" flag.\n- Log-la oru line irukku, ippo irukura code adha ezhudhuradhu illa.\n\nQUESTIONS: Escalation: who does ops call for a dev issue, and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions mean where the code doesn't make it clear?",
"stderr": ""
}
tests/evals/results/handoff/h-mail-intake-full-r2.md
+173
View File
@@ -0,0 +1,173 @@
# Mail intake bot - ops handover
Version: not set (the bot has no version number yet, see section 13) · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every 10 minutes Windows Task Scheduler starts the bot. It reads the unread emails in the referrals mailbox and saves every attachment. It checks that each one is a PDF of 20 MB or less, then copies the PDF to the shared intake folder. It records every attachment as one item in its queue. It has no window or screen of its own. It works with three things: the **referrals mailbox** (it marks emails as read), the **shared intake folder**, where its output goes, and its own **queue database**, which ops cannot open. The bot's own log file is the only thing it writes for ops to read.
## 2. How it works
```
Windows Task Scheduler (every 10 min)
└─ starts → Mail intake bot (no window, runs in the background)
├─ reads + marks read → Referrals mailbox (INBOX, unread mail only)
├─ saves attachments → data/tmp/ (working folder)
├─ copies good PDFs → Shared intake folder (the output folder)
├─ records items → Queue database (data/queue.db - dev only)
└─ writes → logs/intake.log
```
- **Task Scheduler** starts the bot every 10 minutes. Nobody clicks anything.
- The **bot** runs once, from start to finish, then exits. The next run starts 10 minutes later.
- The **referrals mailbox** is where the bot finds its input. Once the bot fetches an email, the mailbox treats it as read.
- **data/tmp/** holds a raw copy of every attachment. Nothing cleans this folder.
- The **shared intake folder** is the output folder. Only good PDFs are copied there, and each keeps the file name the sender gave it.
- The **queue database** holds one row per attachment: sender, subject, file, status and note. Ops cannot open it.
- **logs/intake.log** gets one line per item and one line per mailbox problem. This is where ops looks.
**One run, step by step:**
1. The bot checks whether any item is still marked *Processing* from an earlier run. If one is, it writes "Previous item still Processing - aborting" to the log and stops. Nothing else happens in that run (see E02).
2. It signs in to the mailbox. If that fails it tries again, up to 3 tries with 30 seconds between them (see E01).
3. It fetches every unread email and saves each attachment. Each attachment becomes one item with status *New*.
4. It works through each *New* item: it sets the item to *Processing* and checks it, then copies the file to the shared intake folder.
5. It records the result (*Done*, *Quarantined* or *Failed*), then writes `item <number> -> <status> (<note>)` to the log.
## 3. Before a run
This bot runs by itself, so "before a run" means before the go-live and after any change to the machine. Dev sets these items up. Ops can confirm the ones marked (ops).
- [ ] The bot's settings file (`.env`) is in the bot folder and has the current mailbox password. If it is missing, or the password has expired, the bot logs "IMAP connect failed" or writes nothing at all (E01, E06).
- [ ] The folders `data/`, `data/tmp/` and `logs/` exist, and so does the queue database `data/queue.db`. The bot does not create them. If one is missing, the bot stops at the very start and writes nothing (E06).
- [ ] (ops) You can open the shared intake folder from this machine. If it is unreachable, items become *Failed* and stay that way (E05).
- [ ] (ops) `logs/intake.log` has gained new lines in the last 20 minutes. That shows the scheduled task is running.
- [ ] Input rules for the people who send referrals (ops passes these on):
- Send a **PDF**. A Word file or a photo is not accepted. Bad: `referral.docx`, `scan.jpg`. Good: `referral.pdf`.
- Keep each file **20 MB or smaller**.
- **No password on the PDF.** A protected PDF is not copied to the shared intake folder, but its item still shows *Done* (E04).
- **Give each file a unique name.** If two referrals are both called `scan.pdf`, the second one overwrites the first in the shared intake folder (E08). Bad: `scan.pdf`. Good: `referral_2026-10-23_01.pdf`.
- Every attachment becomes an item, including signature images and logos. Those always end up *Quarantined*. That is expected (E03).
## 4. During a run
- Don't open or move emails in the referrals mailbox while the bot runs. Never mark referral emails as read by hand. The bot only fetches **unread** emails, so an email you mark as read is skipped for good.
- Don't rename, move or delete files in `data/tmp/`. The bot needs them until each item is finished.
- To stop the bot: ask the RPA dev team on #rpa-support to **disable the scheduled task**. Don't end the bot's process (python) halfway through a run. That leaves an item stuck in *Processing*, which blocks every later run (E02). It can also lose emails the bot had already marked as read (E07).
- One run normally takes well under a minute. If the mailbox is down, a run can take about 90 seconds (3 tries × 30 s).
- **A run looks stuck** when the log shows "Previous item still Processing - aborting" on two runs in a row, or gets no new lines for more than 20 minutes. **The bot never clears this by itself.** It has no watchdog, so don't wait. Escalate right away (E02, E06).
## 5. Where to look when something goes wrong
Ops can open `logs/intake.log` and the shared intake folder. The queue database holds the sender, subject and status of each item, but ops cannot open it. The queue viewer spreadsheet named in the bot's README is not delivered (section 13).
| Question | Where to look |
|---|---|
| Is the bot running at all? | `logs/intake.log`: there should be new lines at least every 10-20 minutes while mail comes in. No new lines → E06. |
| Did the last run finish? | `logs/intake.log`: the last lines are `item <number> -> ...` lines, with no "aborting" line after them. |
| Which items failed? | `logs/intake.log`: search for `-> Failed`, `-> Quarantined` and `encrypted`. |
| Why did one item fail? | `logs/intake.log`: search for `item <number> ->`. The text in brackets is the reason. |
| Did the bot already retry? | `logs/intake.log`: "IMAP connect failed (1/3)", "(2/3)" and "(3/3)" are the mailbox retries. Nothing else is ever retried. |
| Did the PDF arrive? | The shared intake folder: look for the file name. |
| Which email / sender was it? | Ops cannot see this. Ask dev on #rpa-support and quote the item number. |
| Which version is running? | There is no version shown anywhere yet (section 13). |
## 6. Statuses
The bot has no screen, so statuses have no colours. Each status appears in the log line `item <number> -> <status> (<note>)`. Log lines are marked INFO (normal), WARNING (a retry) or ERROR (a run stopped).
| Status (note) | Colour | Meaning | What ops does |
|---|---|---|---|
| New (no note) | none (never in the log) | The attachment is saved and waiting. The next run picks it up. | nothing |
| Processing | none (never in the log) | The bot is working on this item right now. If it is still Processing after the run has ended, the item is stuck and blocks the bot. | If "Previous item still Processing - aborting" repeats, see E02 |
| Done - "<n> pages saved" | INFO line | The PDF was copied to the shared intake folder. | nothing |
| Done - "encrypted - not readable" | INFO line | **Needs follow-up.** The PDF has a password. It was **not** copied to the shared intake folder, even though the status says Done. | See E04 |
| Quarantined - "not a PDF or over 20 MB" | INFO line | Not a PDF, or too big. It was not copied. One note covers both reasons. | See E03 |
| Failed - "share unreachable: <error text>" | INFO line | The copy to the shared intake folder failed. **The bot never retries it.** | See E05 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Log shows "IMAP connect failed (1/3)", then (2/3) and (3/3), and no `item` lines after them. New referrals stop arriving in the shared intake folder. | The bot cannot sign in to the mailbox: network down, mail server down, or a wrong or expired password. It tries 3 times, 30 s apart, then the run ends. It cannot tell a password problem from a network problem. | `logs/intake.log`, WARNING lines. The final "Mailbox unreachable" is **not** written to the log. | System | Bot, then Dev | Wait for the next run (10 min). If the text after the colon mentions login, authentication or credentials, the password has probably expired, so escalate at once. | The warnings appear on 3 runs in a row (about 30 min), or the text points to a password problem. Send: the time of the first and last warning and the text after the colon. |
| E02 | Log shows "ERROR Previous item still Processing - aborting" on every run. No new items, and nothing new in the shared intake folder. | At the start of each run the bot checks for an item still marked Processing. This happens when an earlier run crashed in the middle of an item, for example on a damaged PDF, or when someone ended the bot by hand. The bot then stops the run, every time. | `logs/intake.log`, ERROR line. The stuck item is visible only in the queue database. | System | **Dev** | One single line followed by normal runs can be two runs overlapping. Ignore it. If it repeats, the bot is fully blocked and **does not recover by itself**. Only dev can release the stuck item, because ops cannot open the database. | Two "aborting" lines in a row. Send: the time of the first "aborting" line and the last `item <number> ->` line before it. |
| E03 | `item <number> -> Quarantined (not a PDF or over 20 MB)` | The attachment's name does not end in .pdf, or the file is bigger than 20 MB. | `logs/intake.log`; status Quarantined | Business | Ops (dev looks up the sender) | Most of these are signature images or logos. Ignore them. If a referral is expected and missing, ask dev on #rpa-support for the sender and subject of that item number. Then ask the sender to resend it as a PDF of 20 MB or less. | You cannot find who sent it. Send: the item number and the time of the log line. |
| E04 | `item <number> -> Done (encrypted - not readable)`, and the file is **not** in the shared intake folder | The PDF has a password, so the bot stops before copying it. It still marks the item Done. | `logs/intake.log`; status Done with the note "encrypted" | Business | Ops (dev looks up the sender) | Treat it as **not delivered**. Ask dev for the sender and subject of that item number, then ask the sender for a copy with no password. | You cannot find who sent it. Send: the item number and the time of the log line. |
| E05 | `item <number> -> Failed (share unreachable: ...)` | The copy to the shared intake folder fails: the folder is offline, full, or the bot has no permission to write there. | `logs/intake.log`; status Failed | System | Ops, then Dev | 1. Check that you can open the shared intake folder. 2. Check whether a file with that name is already there. It may be half-copied, and if so, tell dev. 3. The bot **never retries a Failed item**. Only dev can set it back to New. | Always, once the folder works again. Send: the item numbers, the time, and the text after "share unreachable:". |
| E06 | `logs/intake.log` gets no new lines for more than 20 minutes, even though mail is arriving | The bot stops before it writes anything. Possible causes: the scheduled task is off; the settings file is missing; the database or a folder is missing; the mailbox gave up after its retries; or an attachment it could not save crashed the run. None of these is written to the log. | Nowhere ops can see. The reason is only in the Task Scheduler's last-run result. | System | Dev | Nothing ops can fix. | After 20 minutes with no new lines. Send: the time of the last line in `logs/intake.log`. |
| E07 | A sender says they emailed a referral, but it is not in the shared intake folder and no `item` line matches it | Three possible causes. The bot crashed while fetching mail, after the mailbox had already marked the email as read, so it is never fetched again. Or the email had no named attachment, which the bot skips without a word. Or someone marked the email as read by hand. | Nowhere | System | Dev | Don't resend from your side yet. Escalate so dev can check the mailbox and the working folder. | Always. Send: the approximate time the email was sent and the sender's organisation. Never paste patient details in chat. |
| E08 | The file in the shared intake folder is the wrong referral, or one of two referrals is missing | Two attachments with the same file name (for example `scan.pdf`). The second overwrites the first, both in the working folder and in the shared intake folder. Both items can still show Done. | Two `item` lines Done; only one file in the shared intake folder | System | Dev | Escalate so dev can find both emails. Ask senders to give files unique names (section 3). | Always. Send: the file name and the two item numbers. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Can't sign in to the mailbox | Tries again | 3 tries × 30 s (about 90 s) | The run ends with nothing logged. The next scheduled run tries again in 10 min (E01). |
| Items left New (e.g. a run ended early) | The next run picks them up | no limit | n/a |
| Item stuck in Processing | **Nothing.** Every run stops at once | no automatic recovery, no watchdog | Blocked until dev releases it (E02) |
| Copy to the shared intake folder failed | **Nothing.** Never retried | none | Item stays Failed until dev resets it (E05) |
| Damaged PDF, or attachment that can't be saved | **Nothing.** The run crashes | none | Damaged PDF → stuck Processing (E02). Save failure → emails lost (E07). |
## 9. Known incidents
Newest first. All three are dated 2026-10-23, the date they were recorded at import. **None of them is fixed in the code.** Each "fix" only added a note, so all three can happen again.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-10-23 / no version | Mailbox password expired; no mail was processed and nothing clear appeared in the log | A login failure is treated like a network blip: 3 retries, then the run ends without logging the final failure. If the bot is also blocked (E02), the mailbox is never even tried, so the password problem stays hidden. | Watch for "IMAP connect failed" lines (E01). If they mention login or authentication, escalate for a password reset. | Partly |
| 2026-10-23 / no version | Password-protected referrals never reached the shared intake folder | Protected PDFs are skipped before the copy step, but the item is marked Done | Search the log for "encrypted" and get an unprotected copy (E04) | Partly |
| 2026-10-23 / no version | Nothing was picked up for a whole day; the log repeated "Previous item still Processing - aborting" | A crash in the middle of an item, for example a damaged PDF, left it stuck in Processing. Every later run stopped at once. Someone released it by hand in the database. | Escalate after two "aborting" lines in a row (E02). Ops cannot release it. | Partly |
## 10. Releases and version check
- How an update arrives: dev replaces the bot files in the bot folder on the bot machine. There is no launcher or auto-update.
- How to confirm it applied: there is **no version number** in the bot or its log, so ops cannot check this yet. Ask dev on #rpa-support to confirm the change and its date (section 13).
- If the version did not change: report it on #rpa-support.
## 11. Escalating to dev
Post in Teams **#rpa-support**. Send:
- the exception ID from section 7 (e.g. "E02")
- the time of the first and last related log lines
- the item numbers
- the relevant lines copied from `logs/intake.log`
Never send: passwords, the settings file (`.env`), the PDFs themselves, or patient or sender details in chat.
## 12. Failure-class check
- F1 covered in section 3 (settings file, folders, database, shared intake folder) and E06; gaps in section 13 #16.
- F2 not applicable: the bot drives no windows. It only talks to the mailbox, files and its database.
- F3 covered in E01 (mailbox) and E05 (shared intake folder), with limits in section 8. No mail-server timeout: section 13 #19.
- F4 covered in section 3 input rules (PDF only, 20 MB, no password, unique names), E03 and E08; gaps in section 13 #8 and #9.
- F5 covered in E04; section 13 #3.
- F6 covered in E05 step 2 (check for a half-copied file before dev resets the item).
- F7 covered in section 6 ("Done - encrypted" needs follow-up) and E04; section 13 #3.
- F8 covered in E02 and section 8: the bot stops every 10 minutes, forever, with no limit. There are no re-run loops or watchdog.
- F9 covered in section 4 (disable the scheduled task; never end the process mid-run).
- F10 covered in E02 (one stuck item blocks every run); section 13 #1.
- F11 covered in section 5, E06 and E07; section 13 #5, #10 and #16.
- F12 covered in section 10; section 13 #11.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | A stuck Processing item stops every run forever. There is no try/except around `process()`, no automatic reset and no age limit. Commit b2ca175 added only a comment. | The bot is blocked for hours until dev edits the DB. Ops cannot release it. | `intake.py:74-76`, `intake.py:79-81`, `intake.py:89` |
| 2 | A damaged PDF makes `PdfReader` raise, which crashes the run and leaves the item in Processing. A missing temp file does the same in `p.stat()`. | Leads straight to #1 | `intake.py:59`, `intake.py:61`, `intake.py:68` |
| 3 | An encrypted PDF returns status `Done` without being copied | Ops thinks it was delivered (F5/F7) | `intake.py:62-63` |
| 4 | `Failed` items are never retried. Only `New` rows are selected. | Every share outage needs a DB fix by dev | `intake.py:66-67`, `intake.py:78` |
| 5 | The final `RuntimeError("Mailbox unreachable")` is not logged, and a login failure is retried just like a network error. Commit 610a536 added only a comment. | The run fails silently, and an expired password looks like a network blip | `intake.py:34-37`, `intake.py:91` |
| 6 | The RFC822 fetch marks mail as Seen, but rows are committed only after the whole loop. A crash mid-fetch loses those emails. | Referrals vanish with no trace (E07) | `intake.py:44`, `intake.py:54` |
| 7 | `write_bytes(None)` on an empty attachment payload crashes the fetch | Triggers #6 | `intake.py:51` |
| 8 | The attachment name is used as-is. Duplicate names overwrite each other in tmp and in the share, and a name containing a path can write outside `data/tmp`. | Wrong or missing referral (E08); security risk | `intake.py:50`, `intake.py:65` |
| 9 | Parts with no filename are skipped silently. Inline images and signatures create Quarantined noise. The Quarantined note mixes "not PDF" and "too big". | Ops can't tell real problems from noise | `intake.py:47-49`, `intake.py:59-60` |
| 10 | `queue_viewer.xlsx` is named in the README but is not in the repo. Ops cannot open the DB, and the log line has only the item id, with no sender, subject or file name. | Ops cannot act on E03/E04 without dev looking up every item | `README.md:6`, `intake.py:84` |
| 11 | No version string anywhere, in the code or the log | Ops cannot confirm an update applied | `intake.py:1-2` (add one and log it at the start of each run) |
| 12 | `.env` (mailbox credentials) is committed to git in the initial import | Credentials exposed in history: rotate them and remove the file from git | `.env` (commit 15fcd88) |
| 13 | `logs/intake.log` holds a line with a sender's email address, written in a format the current code does not produce. That suggests an older or different build. | Personal data in the log; unclear which build ran | `logs/intake.log` |
| 14 | `data/tmp/` is never cleaned | Referral PDFs pile up on the bot machine (privacy, disk space) | `intake.py:50-51` |
| 15 | The `BOT_WAS_RUN` marker file is written every run and nothing documents it or reads it | Unknown purpose. Best reading: a "ran at least once" flag, not a health check. | `intake.py:72` |
| 16 | Errors that happen before the first log line (missing `.env` key, missing DB, table or folders) and "Mailbox unreachable" go only to the Task Scheduler result | Ops sees just silence (E06) | `intake.py:16-18`, `intake.py:23`, `intake.py:31`, `intake.py:73-74` |
| 17 | No lock against overlapping runs. A slow run plus the next scheduled run gives a false "still Processing" line. | False alarm that looks like E02 | `intake.py:74` |
| 18 | Commits b2ca175, 78e06c9 and 610a536 are labelled "fix" but only add comments | Readers of the history believe the incidents are fixed | `intake.py:89-91` |
| 19 | No timeout on the IMAP connection | A hanging mail server could stall a run with no log line | `intake.py:31` |
tests/evals/results/handoff/h-mail-intake-full-r3.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-mail-intake/full",
"rep": 3,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"stuck Processing item blocks the queue": true,
"20 MB limit": true
},
"cost_usd": 1.5101666000000002,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-mail-intake-full-jnbdhnw2\\project",
"reply": "`HANDOVER.md` ready, repo root la: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-mail-intake-full-jnbdhnw2\\project\\HANDOVER.md`. Bot ah run pannala, code change um illa. Code, log, git history mattum padichen.\n\n| Item | Result |\n|---|---|\n| Sections | 13 (template order same) |\n| Exceptions | 8 (E01\u2013E08) |\n| Failure classes | 11 covered, 1 not applicable (F2: bot entha window um drive pannala) |\n| Fix before handover | 14 items |\n\n**Fix list la mukkiyamaanavai:**\n- **Password-protected PDF:** status \"Done\" nu kaattum, aana share folder ku copy aagala.\n- **Processing la stuck aana item:** adhukku apram ella run um block aagum, bot thaane clear pannaadhu. DB access illaama ops ah clear panna mudiyaadhu.\n- **Final mailbox failure and crashes:** log la varaadhu, ops ku onnum theriyaadhu.\n- **Failed item:** bot thirumba try pannaadhu.\n- **Mail lost:** crash aanaa read-mark aana mail poidum.\n- **Same file name:** rendu file overwrite aagum.\n- **`.env` (mailbox password) git la commit aagirukku** (`bc90067`). Password ah rotate pannunga.\n- **Missing viewer + no version:** `queue_viewer.xlsx` repo la illa, version number um illa.\n\n**Also noticed:**\n- Moonu \"fix\" commit um comment line mattum add pannirukku. Code la change illa.\n- `data/` folder ah code create pannaadhu. Doc la adhu prerequisite ah pottirukken.\n- Incident dates record aagala. Log la 2026-09-04 lockout mattum dhaan theriyudhu.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-mail-intake-full-r3.md
+168
View File
@@ -0,0 +1,168 @@
# Mail intake bot - ops handover
Version: none (the bot has no version number, see section 10) · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every 10 minutes the bot checks the referrals mailbox for **unread** mail. It saves every attachment, checks that it is a PDF under 20 MB that is not password-protected, and copies it to the **shared intake folder**. It records one queue entry per attachment, with a status and a short note, and writes one line per attachment to its log. It has no screen of its own. It does not drive any desktop app. It only talks to the mailbox and the shared folder.
## 2. How it works
```
Windows Task Scheduler (every 10 min)
└─ starts → Mail intake bot (no window)
├─ reads ──> Referrals mailbox (unread mail only)
├─ saves → bot's temp folder (data/tmp)
├─ copies → Shared intake folder <- the output ops can open
├─ writes → queue database (data/queue.db) <- ops cannot open
└─ writes → logs/intake.log <- ops can open
```
- **Task Scheduler** starts the bot every 10 minutes. There is nothing to click.
- **Referrals mailbox:** the bot only picks up mail that is still unread. Fetching a mail marks it as read.
- **Temp folder:** every attachment is saved here first, and it stays here even when it is rejected.
- **Shared intake folder:** good PDFs are copied here under their original file name.
- **Queue database:** one entry per attachment, with its status and note. Only the dev team can open it.
- **logs/intake.log:** one line per attachment, plus warnings and errors. This is ops' main window into the bot.
**One run, step by step:**
1. Check that no attachment is left half-done from an earlier run (status Processing). If one is, the run stops at once (E03).
2. Log in to the mailbox. If that fails it tries 3 times, 30 seconds apart (E01, E02).
3. Take every unread mail and save all its attachments, including signature logos and images, as new queue entries.
4. For each new entry, check it, copy it to the shared intake folder, and write the result to the queue and the log.
## 3. Before a run
- [ ] The settings file sits next to the bot. It holds the mailbox server, user and password, and the shared folder path. If anything is missing the bot stops without writing anything to the log (E07).
- [ ] The bot's `data` folder exists and holds the queue database and a `tmp` subfolder, and the `logs` folder exists. The bot does not create these itself (E07).
- [ ] The shared intake folder can be reached from the bot machine (E06).
- [ ] The mailbox password has not expired. Know its expiry date and tell the dev team before it expires (E02).
- [ ] Tell senders the input rules:
- Send referrals as **PDF**, with a file name ending in `.pdf`, under **20 MB** and **not password-protected**.
- Good: `referral_2026-10-23_REF123.pdf`, 3 MB, opens without a password.
- Bad: `referral.docx`, a scan of 25 MB, or a PDF that asks for a password.
- **Use a unique file name per referral.** Two attachments with the same name overwrite each other in the shared folder (E08).
- Good: `referral_<date>_<ref no>.pdf`.
- Bad: every sender attaching `referral.pdf`.
## 4. During a run
- **Do not open or read new mail in the referrals mailbox** (Outlook, webmail, phone). The bot only picks up **unread** mail. A mail someone has read is never picked up.
- If a mail was read by mistake, mark it **unread** again. The next run picks it up.
- **To stop the bot:** disable the scheduled task in Task Scheduler and let the current run finish. **Never end a run while it is running.** That leaves an attachment in Processing and blocks every later run (E03).
- **A run looks stuck when:**
- the log shows "Previous item still Processing - aborting" every 10 minutes (E03), or
- referrals are known to have been sent, yet neither the log nor the shared folder has changed for over 30 minutes (E07).
- **The bot has no watchdog and no time limit.** It never gives up or unblocks by itself. Escalate as soon as you see either sign.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | There is no "run started" or "run finished" line. A run with no new mail writes nothing. Look at the time of the newest line in `logs/intake.log`. |
| Is the bot blocked? | `logs/intake.log`: the line "Previous item still Processing - aborting" repeating every 10 minutes (E03). |
| Can it reach the mailbox? | `logs/intake.log`: lines "IMAP connect failed (1/3)", "(2/3)", "(3/3)" (E01, E02). |
| Which attachments failed? | `logs/intake.log`: lines `item <number> -> Failed`, `-> Quarantined` or `-> Done (encrypted - not readable)`. |
| Why did one attachment fail? | The text in brackets at the end of that same log line. |
| Did the file arrive? | The shared intake folder: look for the file name. |
| Did the bot already retry? | Mailbox login is retried 3 times per run (the "(n/3)" lines). Attachments are **never** retried. |
| Which version is running? | There is no version number (section 10). |
The log line shows only the item number, not the sender or the file name. To turn an item number into a sender or file, ask the dev team (#rpa-support).
## 6. Statuses
The bot has no screen, so no status has a colour. Ops see the final statuses as plain text in the log line `item <number> -> <status> (<note>)`.
| Status (note) | Colour | Meaning | What ops does |
|---|---|---|---|
| New ("") | none, not in the log | Saved from the mailbox and waiting to be checked and copied. | Nothing. It is handled later in the same run, or in the next run if the mailbox login works. |
| Processing ("") | none, not in the log | Being checked or copied right now. If it is still there on the next run, the bot crashed on it and **every later run is blocked**. | If the log shows "Previous item still Processing - aborting", follow E03. |
| Done ("<n> pages saved") | none (log text) | The PDF was copied to the shared intake folder. | Nothing. |
| **Done ("encrypted - not readable")** | none (log text) | **Needs follow-up.** The PDF is password-protected. It was **NOT copied** to the shared folder, even though the status says Done. | Follow E05: ask the sender for a copy without a password. |
| Failed ("share unreachable: ...") | none (log text) | The copy to the shared intake folder failed. **The bot never retries it.** | Follow E06. |
| Quarantined ("not a PDF or over 20 MB") | none (log text) | Rejected. The name does not end in `.pdf`, or the file is over 20 MB. Signature logos and images in mail land here too. | Follow E04. |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | One or two lines "IMAP connect failed (1/3)" or "(2/3)", then normal item lines again | The mailbox login or connection fails; the bot waits 30 s and tries again | `logs/intake.log` (warning line with the error text) | System | Bot | Nothing. It recovered within the same run. | It happens on several runs in a row, which means E02 |
| E02 | "IMAP connect failed" lines on every run, and no new item lines or new files | 3 connection tries, 30 s apart, all fail; the run then ends. The final failure is **not** written to the log. | `logs/intake.log` (warning lines only) | System | Dev | Read the error text. "getaddrinfo failed" means a network or name problem on the bot machine; check that its network is up. A login or authentication error usually means the mailbox password expired or changed. | Always, if it lasts more than 30 minutes. Send the time of the first failing line and the error text. **Never send the password in chat.** |
| E03 | "Previous item still Processing - aborting" every 10 minutes. No new files in the shared folder. | At start the bot finds an attachment left in Processing from an earlier run (a crash, a damaged PDF, or a run ended by hand), so it stops. **It never clears this by itself**, and the line does not say which item. | `logs/intake.log` (error line) | System | Dev | Do not end or restart anything. Mail is not lost while blocked: it stays unread and is picked up after the fix. | Always, at once. Send the time of the first "still Processing" line and the last `item <number> ->` line before it. The dev team checks whether the file reached the shared folder, then resets the item in the database. |
| E04 | `item <number> -> Quarantined (not a PDF or over 20 MB)` | The file name does not end in `.pdf`, or the file is over 20 MB. The note does not say which. | `logs/intake.log`; the file stays in the bot's temp folder | Business | Ops | Usually a signature logo or image: ignore it. If a real referral is missing from the shared folder, ask the sender to resend it as a PDF under 20 MB. | A real `.pdf` under 20 MB is quarantined |
| E05 | `item <number> -> Done (encrypted - not readable)` | The PDF is password-protected (or restricted), so it is not copied | `logs/intake.log`; the file is **not** in the shared folder | Business | Ops | Ask the sender for the referral without a password. The new mail is picked up by itself. Check the log for this note at least daily, because the status says Done. | The sender's file opens without a password, yet it is still rejected. Send the item number and time. |
| E06 | `item <number> -> Failed (share unreachable: ...)` | Copying to the shared intake folder fails | `logs/intake.log` | System | Ops, then Dev | Check that the shared intake folder opens from the bot machine. Even when it works again, **the bot does not retry this attachment.** | Always. Once the share works, send the item numbers and times of the Failed lines; the dev team re-queues them. |
| E07 | Referrals were sent, but nothing new appears in the log or the shared folder for over 30 minutes, and there are no warning lines | The bot crashed before writing to the log (settings file or folder missing, a damaged mail, or a hung mailbox connection: there is no time limit). It does not detect this. | Nowhere ops can see. Only Task Scheduler's run result shows it. | System | Dev | Make sure nobody has read the mails in the mailbox (section 4). If they did, mark them unread. | When the mails are still unread and nothing happens. Send the time the referral was sent, the time of the newest log line, and the subject line. Do not send patient details in chat. |
| E08 | A referral is missing or the wrong one sits in the shared folder; two senders used the same file name | Not detected. Same-name files overwrite each other, both in the temp folder and in the shared folder. | Two `item <number> -> Done` lines; only one file in the shared folder | Business | Ops, then Dev | Ask the senders to resend with unique file names. | You cannot tell which referral was lost. Send both item numbers and times. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Mailbox connection or login fails | Tries again after 30 s | 3 tries per run (about 1.5 minutes) | The run ends. The next run, 10 minutes later, tries again. This repeats forever with no alert (E02). |
| Attachments saved but not yet processed (New) | Carried over to the next run | None | They are processed only after a mailbox login succeeds |
| Attachment left in Processing | **Nothing.** Every run stops. | None, it never clears | Dev resets it (E03) |
| Copy to the shared folder fails | **Nothing.** The attachment is marked Failed. | No retry | Dev re-queues it (E06) |
| Run hangs on the mailbox or the shared folder | **Nothing.** There is no time limit. | None | Dev (E07) |
## 9. Known incidents
No dates or versions were recorded for the fixes below. The log does show one recorded lockout: on 2026-09-04 a mailbox network failure at 08:10 was followed by "still Processing" aborts at 08:20 and 08:30.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before handover, no version | Mailbox password expired and every run failed with nothing in the log | Login errors were not logged | The connection errors are now logged as warnings, but the final failure still is not. Follow E02. | Yes, with E02 |
| Before handover, no version | Password-protected PDFs arrived in the shared folder empty | Encrypted PDFs were copied without a check | They are no longer copied, but they show as Done. Follow E05. | Partly: ops can chase the senders, but only by watching the note |
| Before handover, no version (also seen 2026-09-04) | No new referrals for a whole day. The queue was blocked by one item stuck in Processing, and ops reset it by hand. | A crash during one attachment left it in Processing. Every later run stops. | The bot now logs "still Processing - aborting", but still never clears it. Escalate at once (E03). | Partly: ops can spot it, but only dev can clear it |
## 10. Releases and version check
- **How an update arrives:** there is no updater. The dev team copies the new `intake.py` onto the bot machine.
- **How to confirm it applied:** you can't. The bot shows no version anywhere (Fix item 10). Ask the dev team to confirm in #rpa-support.
- **If the version did not change:** ask the dev team to copy the files again.
- Only one machine runs this bot, and it rewrites nothing shared at start, so an old build cannot undo a newer one.
## 11. Escalating to dev
Post in Teams **#rpa-support**. Send:
- the exception ID from section 7,
- the time of the first bad log line,
- the item number(s) from the `item <number> ->` lines,
- the related lines copied from `logs/intake.log`.
Never send: the mailbox password, the settings file, referral PDFs, or patient or sender details in chat.
## 12. Failure-class check
- F1 covered in section 3 (settings file, folders, share, password expiry) and E07; Fix items 3 and 11.
- F2 not applicable: the bot drives no windows; it talks to the mailbox and the shared folder directly. The closest risk, a person reading mail in the mailbox, is a rule in section 4.
- F3 covered in E01, E02, E06 and E07; no time limit on the mailbox connection is Fix item 4.
- F4 covered in section 3 input rules (PDF name, size, unique file names), E04 and E08; Fix items 7 and 9.
- F5 covered in E05 (Done although nothing was copied); Fix item 1.
- F6 covered in E03 (dev checks whether the file reached the shared folder before resetting) and section 4 (read mail is never picked up); mail marked read before being saved is Fix item 6.
- F7 covered in section 6 ("encrypted" Done needs follow-up) and E05; Fix item 1.
- F8 covered in section 8 (mailbox retries repeat forever) and E03 (the abort repeats every 10 minutes forever).
- F9 covered in section 4: disable the scheduled task, never end a running run.
- F10 covered in E03; Fix item 2.
- F11 covered in section 5 (log only, item number only); Fix items 3, 8 and 9.
- F12 covered in section 10; no version number is Fix item 10.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Password-protected PDFs are marked Done but never copied. Restricted PDFs that open without a password are rejected too (`is_encrypted`). | Ops filtering on Failed or Quarantined never see them, and referrals go missing. Use Quarantined or Failed. | `intake.py:62-63` |
| 2 | Any error inside `process()` (damaged PDF, missing temp file, a killed run) leaves the item in Processing. The start check aborts every run forever, never names the item, and never resets it. | A whole day without referrals; only dev can clear it (DB) | `intake.py:59-61`, `intake.py:74-76`, `intake.py:79-83` |
| 3 | The final "Mailbox unreachable" error and every crash (missing `.env` key, missing table or folder, `TypeError` on an empty payload) go to stderr only, which Task Scheduler drops. Login errors are retried the same way as network errors (account lockout risk). | Ops see nothing, or only warnings | `intake.py:31-37`, `intake.py:51`, `intake.py:87-88` |
| 4 | No timeout on the IMAP connection or the share copy | A hung run blocks all later scheduled runs silently | `intake.py:31`, `intake.py:65` |
| 5 | Failed items are never retried; re-queueing needs a DB edit | Every share outage needs dev | `intake.py:66-67`, `intake.py:78` |
| 6 | Fetching with RFC822 marks mail as read before the rows are committed. A crash in `fetch_new` loses that mail for good. | Referrals silently lost | `intake.py:44-54` |
| 7 | Attachment file names are used as-is: same names overwrite each other in `data/tmp` and on the share, and an unsanitised name can write outside `data/tmp` (path traversal) | Wrong or missing referral; security risk | `intake.py:50`, `intake.py:65` |
| 8 | The log line has the item number only: no sender, subject, file name, or run start/end line. `queue_viewer.xlsx` is named in the README but is not in the repo, and it reads the DB, which ops cannot open. | Ops cannot tie a failure to a referral without dev | `intake.py:84`, `README.md:6` |
| 9 | The Quarantined note doesn't say which check failed (extension or size); only the extension is checked, not the content; rejected files stay in `data/tmp`, which ops cannot reach and which is never cleaned | Ops cannot tell senders what to fix | `intake.py:59-60` |
| 10 | No version string anywhere, and no update mechanism | Ops cannot confirm an update applied | `intake.py:1-2` |
| 11 | `data/`, `data/tmp/`, `logs/` and the `queue` table are never created by the code. The share path falls back to a hardcoded default. | A fresh install crashes silently | `intake.py:17-19`, `intake.py:23`, `intake.py:73` |
| 12 | `.env` (mailbox credentials) is committed to git in the initial import | Credential exposure: rotate the mailbox password and remove the file from history | `.env`, commit `bc90067` |
| 13 | `logs/intake.log` contains a line format the current code never writes, and it includes a sender email address | Personal data in the log | `logs/intake.log:6` |
| 14 | A `BOT_WAS_RUN` marker file is written on every run and never read | Unclear purpose; confirm or remove | `intake.py:72` |
tests/evals/results/handoff/h-mail-intake-full-r4.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-mail-intake/full",
"rep": 4,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": false,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"stuck Processing item blocks the queue": true,
"20 MB limit": true
},
"cost_usd": 0.7672178000000001,
"subagents": 0,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-mail-intake-full-m22iw_c5\\project",
"reply": "`HANDOVER.md` ready \u2014 repo root-la irukku (`HANDOVER.md`). Only that file create pannen; code edhuvum change pannala, bot-ah run pannala.\n\n| Check | Result |\n|---|---|\n| Sections | 13 (template order apdiye) |\n| Exceptions | 10 (E01\u2013E10) |\n| Failure classes | 11 covered, F2 not applicable (desktop window edhuvum drive pannala) |\n| Statuses | 5 status, 6 status+note combo, ellaam listed |\n| Fix before handover | 15 items |\n\n- **Mukkiyamaana vishayam:** 3 \"fix\" commits-um code-la oru comment mattum add pannudhu. Behaviour maaravae illa, so moonu incident-um innum nadakkalam.\n- Stuck \"Processing\" item vandhaa, adhukku appuram ella run-um stop aagidum. Ops-ku DB access illa, so idhu ippo Dev escalation (E02).\n- Encrypted PDF \"Done\"-nu kaattum, aana share folder-ku copy aagaadhu (E04).\n- Unga answers use pannen: escalation #rpa-support; ops-ku log folder + share folder mattum, DB illa; unclear statuses code-la irundhu purinjukitten.\n- Repo chinnadhu (oru ~90-line file), so scan-ku subagents anuppaama naanae padichen.\n- Verify check-la PHI / credential / IP edhuvum doc-la illa. Aana `logs/intake.log`-la oru sender email address irukku; adha doc-la copy pannala (Fix #14).\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions mean where the code doesn't make it clear?",
"stderr": ""
}
tests/evals/results/handoff/h-mail-intake-full-r4.md
+164
View File
@@ -0,0 +1,164 @@
# Mail intake bot - ops handover
Version: none shown by the bot (code as of commit 2e13216) · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every 10 minutes Windows Task Scheduler starts the bot. It reads the **unread** emails in the referrals mailbox, saves every attachment, checks that it is a PDF of 20 MB or less, and copies it to the **shared intake folder**. Each attachment becomes one item in the bot's queue with a status and a note. Input: emails with PDF attachments. Output: PDF files in the shared intake folder, plus one line per item in the log file `logs/intake.log`. It drives no desktop apps or websites. It only talks to the mailbox and the shared folder.
## 2. How it works
```
Windows Task Scheduler (every 10 min)
└─ starts → intake bot (no window) ──reads──> referrals mailbox (unread mail only)
│
├─ saves attachments → bot's temp folder (dev only)
├─ copies PDFs → shared intake folder (ops can open)
├─ records items → queue database (dev only)
└─ writes → logs/intake.log (ops can open)
```
- **Task Scheduler** starts the bot. The bot has no screen, so you never see it running.
- **Referrals mailbox:** the bot reads unread mail only. **Reading a mail marks it as read**, so the bot never sees it again.
- **Temp folder:** every attachment is saved here first. Quarantined files stay here. Ops cannot open it.
- **Shared intake folder:** good PDFs end up here. This is the bot's output.
- **Queue database:** one row per attachment (sender, subject, status, note). Only dev can open it.
- **logs/intake.log:** one line per item processed, plus errors. This is ops' main window into the bot.
**One run, step by step:**
1. Check that no item from an earlier run is still "Processing". If one is, **stop the whole run** (see E02).
2. Sign in to the mailbox: 3 tries, 30 seconds apart.
3. For every unread mail: save each attachment and add it to the queue as **New**. The mail is now marked read.
4. For each New item, oldest first: mark it **Processing**, check it, copy it to the shared folder, then set its final status and note.
5. Write one log line per item: `item <number> -> <status> (<note>)`.
## 3. Before a run
The bot runs by itself, so "before a run" means "keep these true all the time":
- [ ] The settings file is in the bot folder. It holds the mailbox address, user and password and the shared-folder path. If the mailbox password expires or changes, dev must update this file (E01).
- [ ] The bot folder has its `logs` folder, plus the `data` folder holding the queue database and a `tmp` folder. The bot does not create these itself. If one is missing, the run crashes before it writes anything (E10).
- [ ] The bot machine can reach the shared intake folder (E05).
- [ ] **Only one machine runs the bot.** Two copies would read the same mailbox and split or duplicate the work.
- Rules for senders of referrals (tell the referral sources):
- Send **PDF only, 20 MB or less**. Good: `referral.pdf` at 3 MB. Bad: `referral.docx`, a phone photo `.jpg`, or a 35 MB scan (all go to Quarantined, E03).
- **No password-protected PDFs** (E04).
- Give each attachment its **own file name**. Bad: two mails that both attach `scan.pdf` - the second one overwrites the first (E08).
- Logos and images in email signatures count as attachments too. They show up as Quarantined. That is noise, not a lost referral.
## 4. During a run
- **Do not end the bot's task in Task Scheduler and do not restart the machine while a run is going.** If the bot stops halfway through an item, that item stays "Processing" and **every later run stops at once** until dev clears it (E02).
- To stop the bot, ask dev to **disable** the scheduled task. Disabling stops future runs without cutting a run in half.
- Do not open, move or rename files in the shared intake folder while they are being copied.
- A run normally takes seconds, or about 1.5 minutes when the mailbox is slow to answer. The bot has **no watchdog**: if the mailbox or the shared folder stops responding, the run can hang with no time limit (E09).
## 5. Where to look when something goes wrong
Ops can open `logs/intake.log` and the shared intake folder. The queue database is dev only.
| Question | Where to look |
|---|---|
| Did the run finish? | `logs/intake.log`. The bot writes **no** "run started" or "run finished" line. A run with no new mail writes nothing at all, so a silent log does not tell you whether the bot is healthy (Fix #10). |
| Which items failed? | `logs/intake.log` → search for `-> Failed`, `-> Quarantined` and `encrypted` |
| Why did one item fail? | The same log line. The reason is in brackets at the end, e.g. `item 312 -> Quarantined (not a PDF or over 20 MB)` |
| Is the bot blocked? | `logs/intake.log` → `Previous item still Processing - aborting` repeated every 10 minutes (E02) |
| Can it reach the mailbox? | `logs/intake.log` → `IMAP connect failed (1/3)`, `(2/3)`, `(3/3)` (E01) |
| Did the bot already retry? | Only mailbox sign-in is retried (the `(n/3)` lines). **Items are never retried.** |
| Did the PDF reach the shared folder? | The shared intake folder. Look for the attachment's file name. The log shows the item number only, not the file name (Fix #11). |
| Who sent the item? | Not visible to ops. Sender and subject are only in the queue database. Ask dev with the item number. |
| Which version is running? | The bot shows no version. Ask dev for the commit it runs (section 10). |
## 6. Statuses
The bot has no screen, so statuses have no colour. You see them in `logs/intake.log` as `-> <status> (<note>)`.
| Status | Note written | Meaning | What ops does |
|---|---|---|---|
| New | (empty) | Saved from the mailbox, waiting for processing. You see it only if a run stopped early. | Nothing. The next run picks it up. |
| Processing | (unchanged) | The bot is working on it right now. **If it is still Processing after the run, it blocks every later run.** | Nothing during a run. If it stays, see E02. |
| Done | `<n> pages saved` | PDF checked and copied to the shared intake folder | Nothing |
| Done | `encrypted - not readable` | **Needs follow-up.** Password-protected PDF. It was **not** copied to the shared folder, even though the status says Done. | E04 |
| Quarantined | `not a PDF or over 20 MB` | **Needs follow-up** (unless it is a signature image). Not copied to the shared folder. | E03 |
| Failed | `share unreachable: <error text>` | **Needs follow-up.** The copy to the shared folder failed. **The bot never retries it.** | E05 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | New referrals are not reaching the shared folder. The log shows `IMAP connect failed (1/3)` up to `(3/3)` at every 10-minute run. | It tries to sign in to the mailbox 3 times, 30 seconds apart. After the 3rd failure the run ends. | Log: one warning line per try with the mailbox's error text. **The final "gave up" is not logged** (Fix #3). Nothing is queued, and unread mail stays unread (not lost). | System | Bot, then Ops / Dev | Read the error text. Words like "authentication", "login failed" or "password" mean the mailbox password expired or changed: go to dev. Network words like "getaddrinfo failed" or "timed out" mean the machine has no network: check the bot machine's network and wait for the next run. | Password-type error at any time, or network errors on 3 runs in a row (30 min). Send: times of the first and last `(3/3)` lines and the error text (never the password). |
| E02 | The log shows `Previous item still Processing - aborting` every 10 minutes, and no new referrals arrive. | At the start of every run the bot checks whether any item is still Processing. If one is, it stops at once, before it opens the mailbox. | Log: one error line per run. The blocking item is in the queue database only. | System | **Dev** | Ops cannot clear this (database only). New mail is safe: it stays unread in the mailbox and is picked up once dev clears the blocked item. | Always, at the **first** occurrence. Send: time of the first abort line, and the last `item <n> ->` line before it. The blocking item is usually the next number. |
| E03 | Log: `-> Quarantined (not a PDF or over 20 MB)` | The file name does not end in `.pdf`, or the file is bigger than 20 MB | Log line. The file stays in the bot's temp folder and is not copied to the shared folder. | Business | Ops | If it was a signature image or logo, do nothing. If it was a real referral, ask the sender to resend it as a PDF of 20 MB or less. Ops cannot find out the sender alone, so ask dev with the item number. | You need the original file or the sender's details: send the item number and log time. |
| E04 | Log: `-> Done (encrypted - not readable)`. The referral is missing from the shared folder although the status says Done. | The PDF is password-protected | Log line. The file is **not** copied to the shared folder. | Business | Ops | Ask the sender for an unprotected copy. Get the sender's details from dev with the item number. | You need the original file: send the item number and log time. |
| E05 | Log: `-> Failed (share unreachable: ...)` | Copying to the shared intake folder gives an error (folder offline, no permission, disk full) | Log line, with the error text in brackets | System | Ops, then Dev | Check that the shared folder opens from the bot machine and has free space. Check whether a half-copied file with that name is in the shared folder; if so, tell dev. **The bot will not retry this item.** | Always: only dev can put the item back in the queue. Send: item number(s), log time, the error text. |
| E06 | The run stops with no log line for an item, then E02 starts at the next run. | A damaged PDF, or a file that disappeared from the temp folder, makes the bot crash mid-item | **Nothing in the log** for the crash (Fix #3, #4). The item stays Processing. | System | Dev | Same as E02 | Always. Send: the time E02 started. |
| E07 | A sender says they sent a referral, but nothing appears in the log or the shared folder. | Mail with no attachment, or an attachment without a file name, is skipped | **Nowhere.** The mail is marked read and no item is created (Fix #8). | Business | Ops | Check the referrals mailbox for that mail. If it has no attachment, ask the sender to resend with the PDF attached. If it does have a PDF, go to dev. | The mail has a PDF attachment but no item. Send: the mail's received time (not its content). |
| E08 | Two items both show `Done`, but the shared folder has only one file, or the file has the wrong content. | No check. Two attachments with the same file name overwrite each other. | Log: two Done lines. Only one file survives. | System | Dev | Ask dev to recover the overwritten referral from the mailbox. Remind senders to use unique file names. | Always. Send: both item numbers and log times. |
| E09 | Referrals stop arriving with no error in the log, and no `Previous item still Processing` lines either. | No check. The mailbox or the shared folder stopped responding mid-run, and the bot waits with no time limit. | Nothing (Fix #9, #10) | System | Dev | Ops cannot tell this apart from "no new mail" in the log. If senders confirm mail was sent more than 30 minutes ago and nothing arrived, escalate. | 30 minutes with known mail and no log lines. Send: the time of the last log line. |
| E10 | No log lines at all from any run, even when mail is waiting. | The settings file, the `data` or `tmp` folder, the queue database or the `logs` folder is missing, or the temp folder is full | Nothing (the crash happens before or outside logging) | System | Dev | Check that `logs/intake.log` still exists in the bot folder, then escalate. Mail read during a crash in step 3 may be marked read but never queued (lost from the bot's view). | Always. Send: the time of the last log line, and any change to the machine since then (moves, cleanups, password changes). |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Mailbox sign-in fails | Waits 30 s and tries again | 3 tries (about 1.5 minutes) | Run ends with nothing logged. The next scheduled run 10 minutes later tries again (E01). |
| Items left New (run stopped before processing them) | Picked up by the next run | Every 10 minutes | - |
| Anything else | **Nothing.** Failed, Quarantined and encrypted items are never retried. A Processing item left by a crash blocks all runs until dev clears it. | - | E02, E05 |
How long to wait before acting: **one or two scheduled runs (10-20 minutes)** for mailbox errors. **Do not wait** for E02: escalate at the first abort line.
## 9. Known incidents
Newest first. The three "fix" changes in the history are notes in the code only. **The behaviour did not change**, so all three can still happen.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Logged as fixed 2026-10-23 | The mailbox password expired and every run failed silently | Mailbox password expiry. The per-try warnings were in the log, but the final failure was not. | Watch for `IMAP connect failed (3/3)` with a password-type error and escalate (E01). The final failure is still not logged. | Partly |
| Logged as fixed 2026-10-23 | Password-protected PDFs reached the shared folder empty | Encrypted PDFs | Today they are marked `Done (encrypted - not readable)` and **not** copied. Follow E04. | Partly |
| Logged as fixed 2026-10-23 (seen in the log 2026-09-04) | The bot was blocked for a whole day after a crash left one item in Processing. Ops reset it by hand. | A crash mid-item leaves it Processing, and every run then stops | Still happens. Ops no longer has database access, so escalate at the first `Previous item still Processing` line (E02). | Partly |
## 10. Releases and version check
- How an update arrives: dev replaces the bot's code on the bot machine. There is no launcher or auto-update.
- How to confirm it applied: the bot shows no version anywhere. Ask dev to confirm the commit running on the machine (Fix #13).
- If the version did not change: report it in #rpa-support. Only one machine may run the bot (section 3), so there is never a second copy to keep in step.
## 11. Escalating to dev
Post in the Teams channel **#rpa-support**. Send: the exception ID from section 7 (e.g. "E02"), the item number(s), the time of the log line(s), and the matching lines copied from `logs/intake.log`.
Never send: passwords, the settings file, sender addresses, email subjects, or attachment contents.
## 12. Failure-class check
- F1 covered in section 3 (settings file, folders, database, share access) and E10.
- F2 not applicable: the bot drives no desktop windows. It works with a mailbox and a file share only.
- F3 covered in E09 (no time limit on the mailbox or the share) and Fix #9. No screens to wait for.
- F4 covered in section 3 sender rules, E03 (signature images), E07 (no attachment) and E08 (same file name).
- F5 covered in E07 and E10 (mail marked read, nothing recorded) and Fix #8.
- F6 covered in E05 (a half-copied file may be in the share after Failed).
- F7 covered in section 6 and E04 (encrypted shown as Done) and Fix #5.
- F8 covered in E02 (every run aborts with no limit), E09 (hang, no watchdog) and section 8. There is no auto-rerun loop.
- F9 covered in section 4 (never end the task mid-run; disable it instead).
- F10 covered in E02 and E06.
- F11 covered in section 5 and Fix #3, #10, #11, #12 (crashes not logged, sender and file name only in the database).
- F12 covered in section 10 and Fix #13 (no version shown).
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The three "fix" commits (9e47ec2, 9f400df, 2e13216) only add comments. No behaviour changed. | Ops and dev believe these incidents are closed | `intake.py:89-91` |
| 2 | A stale Processing row blocks every run, the blocking id is not logged, and releasing it needs a database edit | Ops has no database access, so every crash becomes a dev call plus lost hours | `intake.py:74-76` |
| 3 | Uncaught errors are never written to the log: "Mailbox unreachable", PDF read errors, file errors. They go only to the console, which nobody sees. | Ops cannot see why a run died | `intake.py:37`, `intake.py:71-88` |
| 4 | A damaged PDF crashes the run (the PDF read is outside any error handling), leaving the item Processing | One bad attachment stops all intake (E06 → E02) | `intake.py:61` |
| 5 | An encrypted PDF is stored as `Done` and not copied | Reads as success but needs follow-up (F7) | `intake.py:62-63` |
| 6 | Failed items are never retried, because only New items are picked | A short share outage needs a dev database edit | `intake.py:78` |
| 7 | Attachment file names are used as-is: the same name overwrites the earlier file in temp and in the share, and names are not sanitised (path characters) | Silent loss of a referral; security risk | `intake.py:50`, `intake.py:65` |
| 8 | Fetching marks mail read before it is committed (one commit after all mails). Mails with no attachment leave no record. | A crash during fetch loses mail from the bot's view; "missing referral" questions cannot be answered | `intake.py:44-54` |
| 9 | No timeout on the mailbox connection or the share copy, and no watchdog | A run can hang without any sign | `intake.py:31`, `intake.py:65` |
| 10 | No "run started / finished / N new mails" log line | A silent log means either "healthy, no mail" or "dead": ops cannot tell | `intake.py:71-84` |
| 11 | The item log line has no file name, sender or subject reference, and the README's `queue_viewer.xlsx` is not in the repo and reads the database | Ops cannot match a log line to a mail or a file in the share | `intake.py:84`, `README.md:6` |
| 12 | The code does not create `data/`, `data/tmp/`, `logs/` or the `queue` table | A fresh or cleaned machine crashes silently (E10) | `intake.py:17-18`, `intake.py:23`, `intake.py:73` |
| 13 | No version string anywhere | Ops cannot confirm an update applied | repo-wide |
| 14 | `logs/intake.log` holds a sender email address, in a line format the current code does not write | Personal data in a file ops reads and may paste into tickets; suggests an older build ran at some point | `logs/intake.log:6` |
| 15 | Quarantined files stay in the temp folder, out of ops' reach, and signature images create Quarantined noise | Ops cannot inspect or forward a quarantined file | `intake.py:46-60` |
tests/evals/results/handoff/h-mail-intake-full-r5.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-mail-intake/full",
"rep": 5,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": false,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": false,
"stuck Processing item blocks the queue": true,
"20 MB limit": true
},
"cost_usd": 0.8563970000000002,
"subagents": 0,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-mail-intake-full-e8dj8tc6\\project",
"reply": "`HANDOVER.md` ready, repo root-la irukku. Bot-a run pannala, code-um change pannala. Repo-la oru script, 4 commits mattum dhaan irundhudhu, so subagents spawn pannaama naane direct-aa padichen.\n\n| Item | Result |\n|---|---|\n| Sections | 13 (template sections ellaam irukku) |\n| Exceptions | 11 (E01\u2013E11) |\n| Statuses | 6 (code ezhudhura ella status-um irukku) |\n| Failure classes | 11 covered, F2 not applicable (bot endha window-um drive pannala) |\n| Fix before handover | 15 items |\n\nOps-ku mukkiyamaana findings:\n- **Moonu \"fix\" commit-um code-la oru comment mattum dhaan add pannirukku.** Bot behaviour maaridala. Ella incidents-um innum nadakka mudiyum.\n- **Oru item \"Processing\"-la stuck aanaa, adhukku appuram ella run-um eppavume abort aagum** (E03). Ops-ku database access illa, so dev dhaan clear pannanum.\n- **Password potta PDF \"Done\"-nu kaattum, aanaa intake folder-ku copy aagaadhu** (E06).\n- **\"Failed\" items-a bot eppavume retry pannaadhu.**\n- **Log-la item number mattum dhaan varum.** Sender, file name rendum varaadhu, so ops edhuvum panna mudiyaadhu.\n- **Rendu attachment-ku ore file name irundhaa, onnu innonna overwrite pannidum** (E08).\n- **README sollra `queue_viewer.xlsx` repo-la illa**, so ops statuses-a log-la mattum dhaan paakka mudiyum.\n\nUnclear-aa irundha statuses-ku code padi en best reading use pannen. \"encrypted\" Done-a \"needs follow-up\"-nu mark pannirukken. Check pannadhu: doc-la PHI, emails, server path, credentials edhuvum illa. Log-la oru pazhaya line-la sender email address irukku; adha doc-ku copy pannala, Fix #15-la flag pannirukken.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions whose meaning the code doesn't make clear mean?",
"stderr": ""
}
tests/evals/results/handoff/h-mail-intake-full-r5.md
+167
View File
@@ -0,0 +1,167 @@
# Mail intake bot - ops handover
Version: none shown by the bot (current code = commit 7d6daa4) · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every 10 minutes (Windows Task Scheduler) the bot reads the **unread** emails in the referrals mailbox. It saves every attachment, checks that it is a PDF under 20 MB, and copies it to the **shared intake folder**, where the referrals team picks it up. Each attachment becomes one numbered item in the bot's queue with a status (Done, Failed, Quarantined...). The bot drives no windows and no desktop apps: it talks to the mailbox and the shared folder directly. Its only output is the PDFs in the intake folder and the log file `logs/intake.log`.
## 2. How it works
```
Task Scheduler (every 10 min)
└─ starts → intake bot (no window)
├─ reads unread mail ──> Referrals mailbox (marks the mail as read)
├─ saves attachments → bot folder data/tmp
├─ records each item → queue database (data/queue.db - dev only)
├─ copies good PDFs → Shared intake folder
└─ writes → logs/intake.log (one line per item)
```
- **Task Scheduler** starts the bot every 10 minutes. There is no app window to watch.
- **Intake bot** does one pass and exits. If the previous pass left an item half-done, it refuses to start (E03).
- **Referrals mailbox**: the bot only looks at unread mail. Once it has read an email, that email is never looked at again, even if something went wrong.
- **data/tmp**: working copy of every attachment. Ops cannot reach it.
- **Queue database**: one row per attachment with sender, subject, file, status and note. Ops cannot open it. Only the dev team can read or change it.
- **Shared intake folder**: the result. A referral that is not here was not delivered.
- **logs/intake.log**: the only place ops can see what happened to each item.
**One run, step by step:**
1. Stop at once if any item from an earlier run is still "Processing" (E03).
2. Sign in to the mailbox (3 tries, 30 s apart).
3. For every unread email: mark it read, save each attachment, add a "New" item per attachment.
4. For every "New" item, oldest first: set it to "Processing", check it, copy it to the intake folder, then set Done / Failed / Quarantined.
5. Write one line per item to the log: `item <number> -> <status> (<note>)`.
## 3. Before a run
- [ ] The bot machine can open the shared intake folder and reach the mailbox.
- [ ] The settings file (`.env`) is in the bot folder and holds the mailbox address, user, password and the intake folder location. Ops never opens or sends it. If it is missing or the mailbox password expired, every run fails (E01, E02).
- [ ] The `data` folder exists in the bot folder, with the queue database and an empty `tmp` folder inside. The bot does not create them; without them the run stops at once and writes nothing to the log (E09).
- [ ] When the mailbox password is changed or about to expire, tell the dev team first so they update the settings file.
- [ ] Input rules for senders (share these with the referral senders):
- Attach the referral as a **PDF**. Bad: a Word file or a phone photo. Good: `referral.pdf`.
- **Under 20 MB** per file.
- **No password** on the PDF. A protected PDF is not delivered, although it is marked Done (E06).
- Give each file a **distinct name**. Bad: every referral named `scan.pdf`. Good: `referral_2026-10-23_01.pdf`. Same-named files overwrite each other (E08).
- Put the referral in an attachment. Text in the email body is ignored (E10).
- [ ] Logos and pictures in email signatures are attachments too. They will show as Quarantined. That is expected.
- [ ] The bot has no version display. See section 10.
## 4. During a run
- Do not move, rename or open files in the bot folder while it runs.
- Do not mark emails in the referrals mailbox as read by hand: the bot only picks up unread mail. An email someone opened is skipped for good.
- To stop the bot: **disable its task in Task Scheduler** (ask the dev team if you have no access to the bot machine), then wait until the log has no new lines for 2 minutes. **Never end the bot's process while it is working on an item** - the item stays "Processing" and blocks every later run (E03).
- A run looks stuck when the log repeats "Previous item still Processing - aborting" every 10 minutes (E03), or when the log has no new line for more than 30 minutes while mail is arriving (E09). The bot has **no time limit and no watchdog**: it never gives up by itself. Escalate at that point.
## 5. Where to look when something goes wrong
Ops can open: the shared intake folder and `logs/intake.log`. Everything else needs the dev team.
| Question | Where to look |
|---|---|
| Did the bot run? | `logs/intake.log`: a new line about every 10 minutes when mail arrives. No line for a quiet mailbox is normal. |
| Did a run stop early? | `logs/intake.log`: "IMAP connect failed (3/3)" with no item lines after it (E01/E02), or "Previous item still Processing - aborting" (E03). |
| Which items failed? | `logs/intake.log`: search for `-> Failed`, `-> Quarantined` and `encrypted - not readable`. |
| Why did one item fail? | The note in brackets at the end of that log line. |
| Which email / sender was it? | **Not visible to ops.** The log shows only the item number. Ask the dev team in #rpa-support with the item number. |
| Was the referral delivered? | Look for the file in the shared intake folder. |
| Did the bot already retry? | Only the mailbox sign-in is retried ("IMAP connect failed (1/3)", "(2/3)", "(3/3)"). Nothing else is ever retried. |
| Which version is running? | Not shown. Ask the dev team (section 10). |
## 6. Statuses
The bot has no screen, so statuses have no colour. Ops sees them in the log line `item <number> -> <status> (<note>)`.
| Status (note) | Colour | Meaning | What ops does |
|---|---|---|---|
| New | none, not in the log | Attachment saved, waiting to be copied in this run | nothing |
| Processing | none, not in the log | Being checked and copied right now. **If it stays, it blocks every later run.** | If "Previous item still Processing - aborting" appears, escalate (E03) |
| Done ("<n> pages saved") | none | PDF copied to the intake folder | nothing |
| Done ("encrypted - not readable") | none | **Needs follow-up.** Password-protected PDF. It was **not** copied to the intake folder, although it says Done. | E06 |
| Failed ("share unreachable: ...") | none | Could not copy to the intake folder. **Never retried.** | E07 |
| Quarantined ("not a PDF or over 20 MB") | none | Not copied: wrong file type or too big | E05 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Log: "IMAP connect failed (1/3)" ... "(3/3)" with a network message (e.g. "getaddrinfo failed"), then nothing more for that run | It cannot reach the mailbox. It tries 3 times, 30 s apart, then gives up on this run | Log warning lines only. The final "gave up" is not logged | System | Bot, then Ops | Wait for the next run (10 min). Check whether the bot machine has network | "(3/3)" appears in 3 runs in a row (about 30 min). Send: times of the lines and `logs/intake.log` |
| E02 | Same "IMAP connect failed" lines, but the message mentions login, authentication or credentials | The mailbox rejects the user or password (for example the password expired) | Log warning lines only | System | Dev | Nothing ops can fix: the password is in the settings file | At once. Send: time of the first line and `logs/intake.log`. Never send the password in chat |
| E03 | Log: "Previous item still Processing - aborting" every 10 minutes; no new files arrive in the intake folder | At start it finds an item still "Processing" from an earlier run and stops before reading mail | Log error line, repeated each run | System | Dev | Nothing is lost: new mail stays unread in the mailbox and is picked up once the block is cleared. Note the last `item <number> ->` line before the first abort: the blocked item is usually the next number. Check whether that file already reached the intake folder and tell dev | At the first abort line. Send: time of the first abort, the last item number before it, whether the file is in the intake folder, `logs/intake.log` |
| E04 | A run stops in the middle: the next item number never gets a log line, and the next run shows E03 | Not detected. An attachment the bot cannot read (a damaged PDF, or a file named `.pdf` that is not really a PDF) makes the bot crash mid-item | Nothing in the log; leads to E03 | System | Dev | Same as E03 | Same as E03 |
| E05 | Log: `-> Quarantined (not a PDF or over 20 MB)` | The file name does not end in `.pdf`, or the file is over 20 MB | Log line; file not in the intake folder | Business | Ops (with Dev to identify the email) | If it is a signature logo or picture (most cases), nothing. Otherwise ask dev in #rpa-support for the sender and subject of the item, then ask the sender to resend it as a PDF under 20 MB | You cannot tell whether it was a real referral - send the item number and time |
| E06 | Log: `-> Done (encrypted - not readable)` | The PDF is password-protected | Log line; **file not in the intake folder** although it says Done | Business | Ops (with Dev to identify the email) | Ask dev for the sender and subject of the item, then ask the sender to resend without a password | Always, to get the sender. Send the item number and time |
| E07 | Log: `-> Failed (share unreachable: ...)`, often for several items in a row | Copying to the intake folder fails (folder offline, no permission, full disk) | Log line; file not in the intake folder | System | Ops, then Dev | Check that the intake folder opens from the bot machine; if not, raise it with IT. Failed items are **never retried** by the bot | Always, once the folder works again: dev must re-queue the failed items. Send the item numbers and times |
| E08 | A sender says they sent two referrals but only one arrived, or a file in the intake folder suddenly has different content | Not detected. Two attachments with the same file name overwrite each other, in the bot folder and in the intake folder | Nothing; both items show Done | System | Dev | Ask senders to use distinct file names (section 3) | When a referral is reported missing. Send the approximate send time and the file name if known |
| E09 | The log has no new line for over 30 minutes while referrals are arriving, and no E01/E03 lines | Not detected. The bot either hangs waiting for the mailbox or the intake folder (it has no time limit), or stops at start because the `data` folder or the settings file is missing | Nothing in the log | System | Dev | Check whether the intake folder still opens. Nothing else ops can do | After 30 minutes. Send the time of the last log line |
| E10 | A sender says they sent a referral but nothing arrived and no new item appears in the log | The email had no attachment (referral typed in the body, or a link). The bot marks it read and moves on | Nothing | Business | Ops | Ask the sender to resend with the referral attached as a PDF | Sender insists it had an attachment - send the approximate send time |
| E11 | Referrals sent around the time of a crash (E04 or a restart) never appear, and the log has no items for them | Not detected. The bot marks emails read while downloading and only records them at the end of the download; a crash in between loses them | Nothing | System | Dev | Ask the sender to resend | Always. Send the approximate time window |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Mailbox unreachable or sign-in rejected | Tries again after 30 s | 3 tries, about 1.5 minutes | This run stops. The next scheduled run (10 min later) starts over |
| New mail missed by a stopped run | The next run reads all mail still unread | every 10 minutes | - |
| Anything else (copy failed, item stuck in Processing, hang) | **Nothing.** No retry, no restart, no watchdog | - | Stays as it is until the dev team acts |
## 9. Known incidents
The three fixes in the history only added a note to the code; the bot's behaviour did not change with them. What is described below is how the bot behaves today.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Logged 2026-10-23 (commit 7d6daa4) | Every run failed and nobody noticed | The mailbox password expired | The sign-in error now shows in the "IMAP connect failed" lines. Watch for them and escalate (E02). Tell dev before the password changes | Partly |
| Logged 2026-10-23 (commit 0749014) | Password-protected PDFs reached the intake folder empty | The bot could not read them | They are no longer copied, but they still show **Done** (E06). Search the log for "encrypted" | Partly |
| Logged 2026-10-23 (commit ae3d441) | No referrals delivered for a whole day | A crash left one item in Processing; every later run stopped (E03). Ops reset it by hand in the database | Still possible. Ops now has no database access: spot the abort line and escalate at once (E03) | Partly |
| 2026-09-04 (log) | Mailbox sign-in failed with a network error, then two runs in a row stopped with "Previous item still Processing" | Network lookup failure, then a stuck item | E01, then E03 | Yes, with E01/E03 |
## 10. Releases and version check
- How an update arrives: the dev team copies the new bot files onto the bot machine by hand. There is no launcher and no automatic update.
- How to confirm it applied: the bot shows no version anywhere. Ask the dev team to confirm in #rpa-support which version is on the machine.
- If the version did not change: report it in #rpa-support. Ops does not copy bot files.
## 11. Escalating to dev
Post in the Teams channel **#rpa-support** to the RPA dev team.
Send: the exception ID from section 7, the item number(s) from the log, the time of the first bad log line, whether the file is in the intake folder, and `logs/intake.log`.
Never send: passwords, the settings file (`.env`), patient details, sender names or email subjects in chat.
## 12. Failure-class check
- F1 covered in section 3 (settings file, `data` folder) and E02, E09.
- F2 not applicable: the bot drives no windows; it talks to the mailbox and the shared folder directly.
- F3 covered in E01, E09 (no time limit on mailbox or folder) and Fix #6. Screen timing is not applicable.
- F4 covered in section 3 input rules and E05, E08, E10.
- F5 covered in E06 (Done without delivery) and E11 (emails marked read but never recorded); Fix #3, #5.
- F6 covered in E03 (check whether the file already reached the intake folder before dev re-queues) and Fix #1.
- F7 covered in section 6 and E06; Fix #3.
- F8 covered in section 8 and E03 (abort repeats every 10 min forever), E09 (hang, no watchdog); Fix #1, #6.
- F9 covered in section 4 (disable the scheduled task; never end the process mid-item).
- F10 covered in E03/E04; Fix #1, #2.
- F11 covered in section 5 (queue and sender not visible to ops) and Fix #4, #7, #8.
- F12 covered in section 10; Fix #11.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Any item left "Processing" makes every later run abort, forever. No timeout, no auto-reset, and the item is not named in the log | One crash stops all intake until dev edits the database (incident ae3d441, "fix" was a comment only) | `intake.py:74-76`, `intake.py:89` |
| 2 | Reading the PDF is not protected: a damaged or fake PDF crashes the run mid-item (also a missing file at `p.stat()`) | Causes E03; ops sees no reason in the log | `intake.py:59-61`, `intake.py:79-81` |
| 3 | Password-protected PDFs are stored as status Done | Ops thinks the referral was delivered | `intake.py:62-63` |
| 4 | The final "Mailbox unreachable" error and any crash traceback go to the console, not to `logs/intake.log` | Ops sees a run stop with no reason | `intake.py:37`, `intake.py:23-24`, `intake.py:87-88` |
| 5 | Mail is fetched (marked read) before its queue rows are committed; a crash in between loses the email. Emails with no attachment are marked read with no log line | Referrals silently lost (E10, E11) | `intake.py:43-54` |
| 6 | No timeout on the mailbox connection or on the copy to the share; no watchdog | A hang stops intake with no log line (E09) | `intake.py:31`, `intake.py:65` |
| 7 | Log lines show only the item number - no file name, no sender/subject reference | Ops cannot act on E05/E06 without dev | `intake.py:84` |
| 8 | README says ops view the queue in `queue_viewer.xlsx`, but that file is not in the repo and reads the database ops cannot open | Ops has no view of statuses except the log | `README.md:6` |
| 9 | Failed items are never retried; only "New" items are processed | A short share outage needs dev to re-queue every item (E07) | `intake.py:78` |
| 10 | Attachments are saved and copied under their original name: same names overwrite each other in `data/tmp` and in the share; the name is not cleaned (path characters) | Lost referrals (E08); a crafted file name could write outside the folder | `intake.py:50-51`, `intake.py:65` |
| 11 | No version string anywhere; deploy is a manual copy | Ops cannot check which build runs (F12) | `intake.py` (whole file) |
| 12 | The bot does not create `data/`, `data/tmp` or the queue table; the share defaults to a hard-coded server path when the setting is missing | A fresh or rebuilt machine fails at start with nothing in the log | `intake.py:17-19`, `intake.py:73` |
| 13 | A `BOT_WAS_RUN` file is written in the bot folder at every start; purpose not documented | Ops may wonder whether to delete it | `intake.py:72` |
| 14 | The mailbox sign-in waits 30 s after the last failed try as well | Each failed run takes 30 s longer than needed (minor) | `intake.py:36` |
| 15 | `logs/intake.log` has no rotation, and holds a sender email address in an older line format the current code no longer writes | Log grows forever; personal data sits in a file ops is asked to send to dev | `intake.py:23`, `logs/intake.log:6` |
tests/evals/results/handoff/h-mail-intake-screens-r2.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-mail-intake/screens",
"rep": 2,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": false,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"stuck Processing item blocks the queue": true,
"20 MB limit": true
},
"cost_usd": 0.7964053999999999,
"subagents": 0,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-mail-intake-screens-k8d75ic5\\project",
"reply": "`HANDOVER.md` ready \u2014 repo root-la irukku: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-mail-intake-screens-k8d75ic5\\project\\HANDOVER.md`. Bot-a run pannala, code-um maathala.\n\n| Item | Result |\n|---|---|\n| Sections | 13 (template order apdiye) |\n| Exceptions | 10 (E01\u2013E10) |\n| Failure classes | 11 covered, 1 not applicable (F2 \u2014 bot entha window-um drive pannala, mailbox + share mattum) |\n| Fix before handover | 12 items |\n\n**Mukkiyamana findings:**\n- Ops-ku intake folder mattum dhaan theriyum. Statuses, notes, log ellame dev-ku mattum. README sollura `queue_viewer.xlsx` repo-la illa. Adhanaala \"Where to look\" table-la perumbaalum \"ticket podunga\" nu dhaan irukku.\n- Moonu \"fix\" commits-um comment line mattum dhaan add pannirukku. \"Processing\" stuck block (E01) code-la innum fix aagala.\n- Password-protected PDF-a status **Done** nu podudhu, aana intake folder-ku copy aagadhu.\n- `.env` (credentials) and oru sender email address irukura log, rendum git-la commit aagirukku.\n- `data/` folder kaali \u2014 queue table illa, so first run-e crash aagum.\n\n**Process notes:**\n- Repo-la oru Python file dhaan, so 2 subagents anuppama naane neraa padichen.\n- Unga answers munnadiye irundhadhaala questions kekkala.\n- Statuses-ku entha screen-um illa, so colour column-la \"none (not shown)\" nu pottiruken.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-mail-intake-screens-r2.md
+166
View File
@@ -0,0 +1,166 @@
# Mail intake bot - ops handover
Version: none shown by the bot (code as of commit 07255fa) · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Every 10 minutes, Windows Task Scheduler starts the bot. It reads the **unread** emails in the referrals mailbox and saves every attachment. It checks that each attachment is a PDF under 20 MB, then copies it into the **shared intake folder**. That folder is the only output, and the only place ops can see. The bot keeps an internal queue with a status and a note for every attachment. Ops cannot open that queue or the bot's log (see section 5). The bot drives no desktop apps or websites, so no windows need to be left alone. It talks only to the mailbox and the shared folder.
## 2. How it works
```
Windows Task Scheduler (every 10 min)
└─ starts → Mail intake bot ──reads──> Referrals mailbox (unread mail only)
│
├─ saves attachments → the bot's own temp folder
├─ copies good PDFs → Shared intake folder ← ops can see this
├─ writes → internal queue (status + note per attachment) ← ops cannot see
└─ writes → bot log file ← ops cannot see
```
- **Task Scheduler** starts the bot every 10 minutes. The bot has no window and no screen.
- **Referrals mailbox:** the bot signs in and takes every unread email. When it reads an email, the mailbox marks it as read.
- **Temp folder:** every attachment lands here first. Rejected or failed files **stay here**, not in the intake folder.
- **Shared intake folder:** only PDFs that passed the checks and copied correctly end up here.
- **Internal queue and log:** they hold the status, the reason and the time of each item. Only the dev team can open them today.
**One run, step by step:**
1. If an item from an earlier run is still marked "Processing", the bot **stops straight away** and does nothing more in this run (see E01).
2. It signs in to the mailbox. If that fails, it tries up to 3 times, 30 seconds apart.
3. It takes all unread emails, saves every attachment to the temp folder and adds each one to the queue as "New".
4. For each New item: it marks the item Processing, checks the file, copies it to the intake folder, then sets the item's final status and note.
## 3. Before a run
Ops cannot check these items. The dev team sets them up. They are listed here so ops knows what a "nothing arrives" ticket can be about.
- [ ] (Dev) The bot's settings file has the mailbox address, user and password, and the intake folder location. If one is missing, every run crashes at start and writes **nothing** to the log.
- [ ] (Dev) The bot's data folder holds the queue and a temp folder. On the current machine the data folder is empty, so the first run would crash.
- [ ] (Dev) The mailbox password has not expired. An expired password stops all intake (E02).
- [ ] (Dev) The scheduled task is enabled and runs every 10 minutes.
- [ ] Rules for referral senders, which ops can pass on:
- Send each referral as a **PDF**, **under 20 MB**, **without a password**.
- Bad: `referral.docx`, a 35 MB scan, a password-protected PDF. Good: `referral.pdf`, 3 MB, opens without a password.
- Give every file a **unique name**. Two attachments with the same name overwrite each other (E06).
- Bad: two emails that both attach `scan.pdf`. Good: `scan_<date>_<number>.pdf`.
- Logos and pictures in an email signature also count as attachments. They are rejected as "not a PDF", which is harmless.
## 4. During a run
- There is nothing to watch. The bot has no window.
- **Do not mark emails in the referrals mailbox as read.** The bot only picks up unread mail. An email marked read by hand is never processed.
- **Do not move, rename or open-and-lock files in the intake folder while they are being copied.**
- To stop the bot: ops cannot. Raise an RPA Support ticket and ask the dev team to **disable the scheduled task between runs**. Never end the bot in the middle of a run. A run killed halfway leaves an item in "Processing", and that blocks every later run (E01).
- How a stuck bot looks: **no new files arrive in the intake folder for more than 30 minutes while referrals are being sent.** The bot does not restart itself and has no stuck-run limit. It stays blocked until the dev team acts, so raise a ticket straight away.
## 5. Where to look when something goes wrong
Ops can open the shared intake folder and nothing else. The bot has no screen or report of its own. Most answers are therefore "raise a ticket". Section 13 lists the gaps for the dev team.
| Question | Where to look |
|---|---|
| Did the run finish? | Ops cannot see this. The only sign is new files in the intake folder, sorted by date modified. |
| Is the bot working at all? | Intake folder: was the newest file added in the last 30 minutes (during hours when referrals come in)? If not, raise a ticket. |
| Did a particular referral arrive? | Intake folder: look for the attachment's file name. If it is missing, raise a ticket with the email's received time. |
| Why did one item fail? | Ops cannot see this. The status and note are only in the bot's queue and log. Raise a ticket. |
| Did the bot already retry? | It only retries signing in to the mailbox (3 tries, 30 seconds apart, every run). Nothing else is retried (section 8). |
| Which version is running? | Not shown anywhere (section 10). |
## 6. Statuses
The bot gives every attachment one status and one note. **No screen shows them to ops and they have no colour.** Only the dev team can read them. They are listed so ops and dev use the same words in a ticket.
| Status | Note written | Colour | Meaning | What ops does |
|---|---|---|---|---|
| New | (empty) | none (not shown) | Saved from the mailbox, waiting for this run's checks | nothing |
| Processing | (empty) | none (not shown) | Being checked and copied right now. **If it is still there after the run, it blocks every later run.** | Ticket if no new files arrive for 30 minutes (E01) |
| Done | "<n> pages saved" | none (not shown) | PDF copied to the intake folder | nothing |
| Done | "encrypted - not readable" | none (not shown) | **Needs follow-up.** A password-protected PDF. It was **not** copied to the intake folder, even though the status says Done. | Ask the sender to resend it without a password (E05) |
| Quarantined | "not a PDF or over 20 MB" | none (not shown) | Rejected. It stays in the bot's temp folder and is not copied. | Ask the sender to resend it as a PDF under 20 MB (E04) |
| Failed | "share unreachable: <error>" | none (not shown) | The intake folder could not be reached. **Never retried.** | Ticket. The dev team must re-queue it (E03). |
## 7. Exception catalogue
Ops sees most of these the same way: **a file is missing from the intake folder.** Quote the E-number in the ticket when you know it. When you don't, say "no new files since <time>".
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | No new files in the intake folder at all, for 30 minutes or longer | At start, the bot finds an item still marked Processing from an earlier run, which crashed or was stopped. It stops at once, every run, until someone clears that item. | Bot log: "Previous item still Processing - aborting", once every 10 minutes. Not visible to ops. | System | Dev | Raise a ticket. Send: when the last file arrived in the intake folder, and the time you noticed. Dev must check whether the blocked file **is already in the intake folder** before re-queuing it, otherwise it is copied twice. | Always. It never clears by itself. |
| E02 | No new files in the intake folder at all | Signing in to the mailbox fails 3 times, 30 seconds apart (wrong or expired password, or the mailbox server cannot be reached). The run then crashes. | Bot log: "IMAP connect failed (1/3)" … "(3/3)" every run. The final crash is not logged. Not visible to ops. | System | Bot, then Dev | Wait 10 minutes for the next run. If still nothing arrives, raise a ticket: "no new files since <time>, possibly mailbox sign-in". | No files for 30 minutes |
| E03 | One referral's PDF is missing from the intake folder, while others still arrive | Copying to the intake folder fails (the folder is unreachable, full, or the file is locked). | Status Failed, note "share unreachable: …". Not visible to ops. | System | Dev | Raise a ticket with the email's received time and the attachment's file name. The dev team copies it or re-queues it. **The bot never retries it.** | Always |
| E04 | One referral's attachment is missing from the intake folder | The attachment is not a .pdf file, or it is over 20 MB. | Status Quarantined, note "not a PDF or over 20 MB". It stays in the bot's temp folder. Not visible to ops. | Business | Ops (after dev confirms) | Ask the sender to resend it as a PDF under 20 MB, in a **new** email. | The file is a PDF under 20 MB and was still rejected |
| E05 | One referral's PDF is missing from the intake folder, but the bot reports it as Done | The PDF is password-protected. | Status **Done**, note "encrypted - not readable". Not visible to ops. | Business | Ops (after dev confirms) | Ask the sender to resend it without a password, in a new email. | Never. This is a known gap (section 13). |
| E06 | A file in the intake folder has the right name but the wrong content, or a referral seems to be missing | Not detected. Two attachments with the same file name overwrite each other in the temp folder and in the intake folder. | Nowhere. Both queue items show Done. | Business | Dev | Raise a ticket with the file name and the received times of both emails. Ask the senders to use unique file names. | Always |
| E07 | No new files at all (it then turns into E01) | Not detected. A damaged or fake PDF makes the bot crash while it reads the file. The item stays Processing. | Nowhere. The crash is not logged, and from the next run the log shows E01's line. | System | Dev | Same as E01. In the ticket, mention any referral that arrived just before files stopped. | Always |
| E08 | A referral email is never processed, and it shows as **read** in the mailbox | Not detected. The bot marks emails read as it fetches them. If it crashes before the email reaches the queue (an empty or unusual attachment, a missing temp folder), the email is lost. | Nowhere. Not in the queue, not in the log. | System | Dev | Raise a ticket with the email's received time. The dev team marks it unread so the next run picks it up. | Always |
| E09 | No new files at all, from the very first run or right after a change on the bot machine | Not detected. A setting is missing (mailbox details, intake folder location) or the bot's data folder or queue is missing, so the bot crashes at start. | Nowhere. The crash happens before anything is logged. | System | Dev | Raise a ticket: "no files since <time>, after <change, if known>". | Always |
| E10 | No new files at all, and the problem lasts for hours | Not detected. Sign-in or copying hangs with no time limit, so the run never ends. Task Scheduler does not start new runs while one is still running. | Nowhere | System | Dev | Raise a ticket. The dev team ends the hung run, then checks for a Processing item (E01). | Always |
Never send in a ticket or chat: patient names, referral contents, the PDF itself, mailbox passwords or the bot's settings file. Use the received time and the file name.
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Can't sign in to the mailbox | Tries again after 30 seconds | 3 tries per run (about 1.5 minutes) | The run crashes. The next run, 10 minutes later, tries again, and that repeats forever with no alert (E02). |
| Intake folder unreachable | Nothing. No retry. | - | Item Failed for good (E03) |
| Item left in Processing | Nothing. No retry. | - | Every run stops until the dev team clears it (E01) |
| Run hangs | Nothing. No watchdog. | - | Blocked until the dev team ends it (E10) |
Ops never needs to wait more than **one run (10 minutes)** before acting. Apart from mailbox sign-in, the bot does not fix anything by itself.
## 9. Known incidents
Dates come from the commit history (all recorded 2026-10-23) and the bot log (2026-09-04). The bot has no version numbers.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Recorded 2026-10-23 | Nothing arrived for days. Every run failed silently. | The mailbox password expired. Sign-in failures were not logged. | Sign-in failures are now logged (E02), but there is still no alert and ops cannot see the log. Raise a ticket after 30 minutes with no files. | Partly. Ops can spot "no files", not the cause. |
| Recorded 2026-10-23 | Password-protected referrals reached the intake folder as empty copies | The bot copied PDFs it could not read | They are now **not copied at all** and marked Done "encrypted - not readable" (E05). Still invisible to ops. | Partly. Ops can ask senders not to use passwords. |
| Recorded 2026-10-23 | No referrals arrived for a whole working day | A crash left one item in Processing, which blocked every run. Ops reset it by hand. | **Not fixed in the code.** It still happens (E01, E07). Raise a ticket after 30 minutes with no files. | Partly. Spotting it, yes. Ops can no longer clear it, because they have no access to the queue. |
| 2026-09-04 (log) | Files stopped at about 08:00 | The mailbox server could not be found, then an item was left in Processing and blocked the following runs | As E02 and E01 | Partly |
## 10. Releases and version check
- How an update arrives: not defined in the code. There is no launcher, no installer and no version number. The dev team changes the code on the bot machine.
- How to confirm it applied: ops cannot. The bot does not show its version anywhere.
- If you suspect an update did not apply (a fixed problem comes back), raise a ticket naming the incident from section 9.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue.
Send: the E-number from section 7 if you know it, when the last good file appeared in the intake folder, when you noticed, and for a single missing referral the email's received time and the attachment's file name.
Never send: passwords, the bot's settings file, the referral PDF, or patient or customer details in the ticket or chat.
## 12. Failure-class check
- F1 Missing prerequisite: covered in section 3 (Dev checklist) and E09. Section 13 items 4 and 5.
- F2 Wrong window / focus: not applicable. The bot drives no windows or apps, only the mailbox and a file share.
- F3 Target slower or different: covered in E02, E10 and section 8. Section 13 item 6 (no time limits).
- F4 Input read differently: covered by the sender rules in section 3 (PDF, 20 MB, no password, unique names) and E04, E06. Section 13 items 7 and 8.
- F5 Marked done when nothing was done: covered in E05 and E08. Section 13 items 2 and 9.
- F6 Action happened but not confirmed: covered in E01 ("check the file is already in the intake folder before re-queuing").
- F7 Business exception shown as success: covered in section 6 (Done "encrypted - not readable") and E05. Section 13 item 2.
- F8 Endless re-runs / watchdog: covered in section 8 (endless mailbox retries every 10 minutes, no watchdog) and E10. Section 13 item 6.
- F9 Stopping the wrong way: covered in section 4 (disable the scheduled task between runs; never kill a run midway).
- F10 One stuck item blocks the queue: covered in E01 and E07, section 4, section 8. Section 13 item 1.
- F11 Cause hidden from ops: covered in section 5. Section 13 item 3.
- F12 Update didn't arrive: covered in section 10. Section 13 item 10.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | One item left in "Processing" stops every later run, with no time limit and no auto-reset. The commit "fix: worker blocked…" (c4765ff) only added a comment and changed no code. Any crash in `process()` (a damaged PDF in `PdfReader`, a missing temp file in `stat()`) causes it. | Intake stops for good and ops can neither see it nor clear it | `intake.py:74-76`, `intake.py:61`, `intake.py:79-83`, `intake.py:89` |
| 2 | Encrypted PDFs get status **Done** but are not copied | A referral is silently missing while its status says Done | `intake.py:62-63` |
| 3 | Statuses, notes and the log are visible only to dev. The README's `queue_viewer.xlsx` does not exist in the repo and nothing creates it. Ops has no screen and no access to the database or logs. | Ops cannot tell which item failed or why, and can only report "no files" | `README.md:6`, `intake.py:23-24`, `intake.py:82` |
| 4 | Uncaught crashes are not logged (sign-in exhausted `RuntimeError`, missing `.env` key `KeyError`, missing `queue` table, `PdfReader` errors). There is no alert of any kind. | Silent outages, as in the expired-password incident | `intake.py:37`, `intake.py:31-32`, `intake.py:71-88` |
| 5 | `data/` is empty on this machine. Nothing creates `data/tmp` or the `queue` table. | The first run crashes (E09) | `intake.py:17-18`, `intake.py:50-52`, `intake.py:73` |
| 6 | No timeout on the IMAP connection or the share copy | A hung run blocks all later runs with no limit (E10) | `intake.py:31`, `intake.py:65` |
| 7 | The attachment file name from the email is used as-is. Same names overwrite each other in tmp and on the share, and names containing `..\` can write outside the temp folder (security). | Lost or overwritten referrals (E06), plus a path-traversal risk | `intake.py:47-51`, `intake.py:65` |
| 8 | Signature images and other inline attachments are queued and Quarantined | Noise in the queue that hides real rejections | `intake.py:46-49`, `intake.py:59-60` |
| 9 | Emails are marked read when fetched, but the queue is committed only after all of them. A crash in between (e.g. `get_payload` returning None) loses the email. | Referrals lost with no record (E08) | `intake.py:42-54` |
| 10 | No version string and no defined update process. The three "fix" commits changed only comments, so the history does not show what is actually fixed. | Nobody can confirm which build runs or that a fix arrived | `intake.py:89-91`, whole repo |
| 11 | `Failed` items are never retried. Only `New` is picked up. | One share outage loses those referrals until dev re-queues them | `intake.py:66-67`, `intake.py:78` |
| 12 | `.env` (mailbox credentials) and `logs/intake.log` are committed to git, and the committed log contains a sender email address | Credentials and personal data sit in the repo history | `.env`, `logs/intake.log:6` |
tests/evals/results/handoff/h-mail-intake-screens-r3.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-mail-intake/screens",
"rep": 3,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": true,
"stuck Processing item blocks the queue": true,
"20 MB limit": true
},
"cost_usd": 1.2259993999999999,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-mail-intake-screens-57a6wls8\\project",
"reply": "`HANDOVER.md` ready, repo root-la irukku. Bot-a run pannala; code-um change pannala.\n\n| Item | Result |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 7 (E01\u2013E07) |\n| Failure classes | 11 covered, 1 not applicable (F2: bot entha app window-um drive pannadhu) |\n| Statuses | 5 status-um, ovvoru note variant-um section 6-la irukku |\n| Fix before handover | 15 items |\n\n**Mukkiyamana Fix before handover items:**\n- Oru damaged PDF vandha, adhoda row \"Processing\"-la stuck aagidum. Adhukku apram ovvoru run-um start-laye stop aagum, full intake nikkum (`intake.py:81`, `:74-76`).\n- Mailbox login fail aana last error enga-yum record aagaradhu illa. Silent-a outage nadakkum (`:34-37`, `:77`).\n- Password-protected PDF share-ku copy aagaradhu illa, aana \"Done\"-nu mark aagudhu (`:62-63`).\n- Failed rows-a bot thirumba retry pannadhu (`:78`).\n- Crash aana, read-nu mark aana mails thirumba edukkapadaadhu, referrals lose aagum (`:44`, `:54`).\n- Same peru attachments onnu innonna overwrite pannum (`:50`, `:65`).\n- Mailbox credentials irukkura `.env` git-la commit aagirukku. Password rotate pannanum.\n\n**Also noticed:**\n- Moonu \"fix:\" commits-um comment line mattum dhaan add pannirukku. Section 9-la irukura moonu incidents-um code-la innum fix aagala.\n- Ops-oda ore screen `queue_viewer.xlsx`, adhu repo-la illa. Adhanala status colours confirm panna mudiyala; doc-la \"plain text\"-nu dhaan potturukken.\n- Bot-ku version number illa. Version line-la \"no version number (commit 6a62717)\"-nu potturukken.\n\nQUESTIONS: Escalation: who does ops call for a dev issue, and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-mail-intake-screens-r3.md
+162
View File
@@ -0,0 +1,162 @@
# Mail intake bot - ops handover
Version: no version number (repo commit 6a62717) · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Every 10 minutes the bot reads new (unread) emails in the referrals mailbox and saves each email attachment. It checks that each attachment is a PDF under 20 MB and not password-protected. It copies good PDFs to the shared intake folder (the output folder). Each attachment gets one row in the intake queue, with a status and a note. Ops see the queue in the **queue viewer Excel**, which shows id, sender, subject, status and note; click Data > Refresh to update it. The bot has no window of its own and does not drive any app on screen. It runs in the background on a schedule.
## 2. How it works
```
Windows Task Scheduler (every 10 min)
└─ starts → Mail intake bot (background, no window)
├─ reads → referrals mailbox (unread mail only)
├─ saves → temporary folder on the bot machine
├─ copies → shared intake folder (output folder - ops can open)
├─ writes → intake queue (one row per attachment)
│ └─ shown in → queue viewer Excel (ops can open)
└─ writes → bot log file (ops cannot open)
```
- **Task Scheduler** starts the bot every 10 minutes. Nobody clicks anything.
- **The mail intake bot** does all the work and closes when finished.
- **The referrals mailbox** is where referrals arrive. When the bot reads an email, the email is marked as read.
- **The temporary folder** holds every attachment the bot saved. It is never emptied.
- **The shared intake folder** is where finished PDFs go. A PDF is "delivered" only when it is here.
- **The intake queue** has one row per attachment with a status and a note. Ops see it through the queue viewer Excel.
- **The bot log** records each item's result and mailbox warnings. Only dev can open it.
**One run, step by step:**
1. The bot checks the queue. If any row is still **Processing** from an earlier run, it stops at once and does nothing else (see E04).
2. It signs in to the mailbox. If that fails it tries 3 times, waiting 30 s each time.
3. It reads every unread email, saves every attachment, and adds one **New** row per attachment.
4. For each New row, oldest first, it sets the row to **Processing**, checks the file, copies it to the shared intake folder and writes the final status and note.
5. The run ends. The next run starts 10 minutes later.
## 3. Before a run
The bot runs by itself, so these are things to keep true, not steps to do each time.
- [ ] The Task Scheduler task is enabled on the bot machine (dev/IT).
- [ ] The mailbox password in the bot's settings has not expired (dev/IT). An expired password stops all intake (E05).
- [ ] The shared intake folder is reachable from the bot machine. Check by opening it yourself.
- [ ] The bot's settings file, queue, temporary folder and log folder exist on the bot machine (dev). The bot does not create them, and if one is missing, nothing happens (E05).
- [ ] Rules for senders' attachments:
- **PDF only.** Bad: `referral.docx`, `scan.jpg`. Good: `referral.pdf` (`.PDF` in capitals is fine too).
- **Under 20 MB.** Bad: a 35 MB scan. Good: the same scan split into two files.
- **Not password-protected.** Bad: a PDF that asks for a password when opened. Good: the same PDF saved without a password.
- **A unique file name.** Bad: two referrals both called `scan.pdf`, since the second one overwrites the first (E07). Good: `scan_smith_1.pdf`, `scan_smith_2.pdf`.
- [ ] There is no version check. See section 10.
## 4. During a run
- Ops do not need to stay away from the machine. The bot has no window and clicks nothing.
- Do not rename, move or delete files in the shared intake folder while they are still New or Processing in the queue viewer.
- **To stop the bot:** ask dev/IT (RPA Support ticket) to disable the Task Scheduler task. Do not end the bot's process in the middle of a run. If it is stopped while an item is Processing, that item stays Processing and blocks every later run (E04).
- **A run looks stuck when** one row shows **Processing** on two refreshes of the queue viewer at least 15 minutes apart. Normally an item is in Processing for a few seconds only. **The bot has no watchdog and never frees a stuck item by itself.** Don't wait it out. Raise a ticket (E04).
## 5. Where to look when something goes wrong
Ops can open only the queue viewer Excel and the shared intake folder. Everything else (bot log, queue database, Task Scheduler history, temporary folder) is dev-only.
| Question | Where to look |
|---|---|
| Is the bot running at all? | Queue viewer → Data > Refresh. New rows should appear within about 10 minutes of a referral email. If none appear for 30 minutes, see E05. |
| Is the bot blocked? | Queue viewer → any row with status **Processing** that stays that way for 15+ minutes (E04). |
| Which items failed? | Queue viewer → filter status = **Failed** or **Quarantined**, and status = **Done** with note "encrypted - not readable". |
| Why did one item fail? | Queue viewer → the **note** column for that row. Then look up the exception in section 7. |
| Did the file really arrive? | Shared intake folder → look for the file name. **Done is not proof by itself.** An encrypted PDF is marked Done but is not copied (E02). |
| Did the bot already retry? | Only the mailbox sign-in is retried, and you can't see it (dev log only). Nothing else is ever retried. |
| Which version is running? | You can't tell (section 10). |
## 6. Statuses
The queue viewer shows statuses as plain text. The bot sets no colours. If the Excel adds colour, that is a formatting rule inside the Excel and not something this document can confirm.
| Status | Note | Meaning | What ops does |
|---|---|---|---|
| New | (empty) | Attachment saved, waiting to be checked in this or the next run | nothing |
| Processing | (empty) | Being checked right now. Normal for a few seconds. | If it stays for 15+ minutes, raise a ticket (E04) |
| Done | "<n> pages saved" | PDF copied to the shared intake folder | nothing |
| Done | "encrypted - not readable" | **Not delivered - needs follow-up.** Password-protected PDF, **not** copied to the shared intake folder. | Ask the sender for an unlocked copy (E02) |
| Quarantined | "not a PDF or over 20 MB" | **Not delivered.** Wrong file type or too big. The note does not say which. | Check the attachment and ask the sender to resend (E01) |
| Failed | "share unreachable: ..." | **Not delivered.** Could not copy to the shared intake folder. **Never retried.** | Check the shared intake folder, then raise a ticket (E03) |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Quarantined - "not a PDF or over 20 MB" | The file name does not end in .pdf, or the file is bigger than 20 MB | Queue viewer status and note; dev log | Business | Ops | Look at the file name in the row. If it is a logo or signature image from the email (for example `image001.png`), ignore the row. Otherwise ask the sender to resend as a PDF under 20 MB. The resent email gets a new row. | A real PDF under 20 MB is quarantined |
| E02 | Done - "encrypted - not readable", and the file is not in the shared intake folder | The PDF is password-protected. The bot skips the copy but still marks the row Done. | Queue viewer note; dev log | Business | Ops | Filter for this note every day; the Done status hides it. Ask the sender for a copy without a password. | Never, unless the sender says the file has no password |
| E03 | Failed - "share unreachable: ..." | Copying the PDF to the shared intake folder gave an error | Queue viewer status and note; dev log | System | Ops, then Dev | 1. Open the shared intake folder. If you can't, the folder or network is down; tell IT. 2. If a file with that name is there, it may be a half-copied file; don't trust it. 3. The bot **never** retries Failed rows. Raise a ticket so dev can put the row back to New once the folder works. | Always (only dev can re-queue the row). Send: item id, time of the row, "E03". |
| E04 | One row stuck on **Processing**, and no new rows appear at all. Every referral since then is waiting. | At the start of every run the bot checks for any Processing row. If it finds one, it stops at once. This happens when the bot crashed on that item (for example a damaged PDF) or was stopped mid-item. | Queue viewer (the stuck row); dev log "Previous item still Processing - aborting" every 10 minutes | System | Dev | Raise a ticket at once. The bot will **never** free itself. Intake is stopped until dev resets the row. No mail is lost: emails stay unread in the mailbox and are picked up after the reset. | Always, as soon as it is seen. Send: the stuck item's id, the time you first saw it, "E04". |
| E05 | No new rows appear for 30+ minutes although referrals were emailed, and no row is stuck in Processing | The bot cannot sign in to the mailbox (expired password, network or mailbox problem). It tries 3 times, 30 s apart, then gives up until the next run. The same symptom appears if the bot's settings, queue, temporary folder or log folder is missing. | Nowhere ops can see. The dev log shows "IMAP connect failed" warnings. The final failure is not in the log at all; it is only in Task Scheduler history. | System | Bot, then Dev | Wait 30 minutes (3 runs). Then raise a ticket. Mention whether the mailbox password is due to expire, if you know. | No new rows for 30 minutes during business hours. Send: time of the last new row, time the missing referral was sent, "E05". |
| E06 | A sender says they emailed a referral, the email is already marked read in the mailbox, but there is no row for it | The bot crashed while saving attachments, after the emails were marked read but before the rows were saved. The next run only looks at unread mail, so it skips them. | Nowhere. No row and no log line. | System | Dev | Raise a ticket. Until it is fixed, ask the sender to resend, or have someone with mailbox access mark the email unread again. | Always. Send: approximate time the email was sent, "E06". Don't paste the sender or subject into chat. |
| E07 | Two rows are Done with "<n> pages saved", but the shared intake folder holds only one file, or a file has the wrong content | Two attachments had the same file name. The second overwrote the first, both in the temporary folder and in the shared intake folder. The bot does not check for duplicate names. | Queue viewer shows two rows with the same file name; nothing flags it | Business | Ops, then Dev | Ask the senders to resend with unique file names. Raise a ticket so dev knows it happened. | Always. Send: both item ids, "E07". |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Mailbox sign-in fails | Tries again after 30 s | 3 tries per run (about 1.5 minutes) | The run ends with no new rows. The next run in 10 minutes tries again. |
| Rows left as New at the end of a run | Picked up by the next run | Every 10 minutes | Nothing more is needed |
| Anything else (Failed rows, stuck Processing, lost attachments) | **Nothing.** No retry, no watchdog, no restart. | - | Needs dev (E03, E04, E06) |
So the longest ops should ever wait before acting is **30 minutes** (E05). For a stuck Processing row (E04), don't wait.
## 9. Known incidents
Newest first. All three were logged as fixes, but **none of them changed how the bot behaves**, so each can happen again.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| commit 6a62717 | No new referrals came in, and nothing obvious showed anywhere | The mailbox password expired. Every run failed to sign in and the final failure was not recorded anywhere ops or dev would look. | E05: no new rows for 30 minutes → ticket. Dev renews the password. | Yes, with E05 |
| commit 72c7f55 | PDFs in the shared intake folder were empty or unreadable | Password-protected PDFs were copied as if fine | They are no longer copied, but they are marked **Done**. Filter for the note "encrypted - not readable" (E02). | Partly. The Done status hides them. |
| commit a45615d | No referrals processed for a whole day | One item crashed mid-processing and stayed Processing. Every later run stopped at the start. Someone reset the row by hand. | E04: raise a ticket at once. Still happens on any damaged PDF. | Partly. Ops can spot it, but only dev can reset the row. |
## 10. Releases and version check
- **How an update arrives:** there is no updater. Dev copies the new code onto the bot machine.
- **How to confirm it applied:** ops can't. The bot has no version number and no screen. Ask dev to confirm in the ticket that the new code is in place on the bot machine.
- **If the version did not change:** raise a ticket in RPA Support. There is evidence that the bot on the machine has at some point been different from the code in the repo (Fix before handover #13).
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue. Send: the exception ID from section 7, the item id(s) from the queue viewer, the time you first saw the problem, and the time of the last row that looked normal.
Never send: passwords, the bot's settings file, or the sender, subject or file content of a referral (patient data). Item ids are enough.
## 12. Failure-class check
- F1 (missing prerequisite) is covered in section 3 and E05, plus Fix #8 and #12.
- F2 (wrong window or focus) is not applicable: the bot drives no app on screen and uses no windows.
- F3 (target slower or different than expected) is covered in E05 (mailbox) and E03 (shared folder). Mailbox sign-in is the only wait the bot has.
- F4 (input read differently than written) is covered in section 3 attachment rules, E01 and E07, plus Fix #6 and #7.
- F5 (marked done when nothing was done) is covered in E02 and E07, plus Fix #3.
- F6 (action happened but not confirmed) is covered in E03 step 2: a failed copy may leave a partial file on the share.
- F7 (business exception shown as success) is covered in section 6 (Done / "encrypted"), E02, and Fix #3.
- F8 (endless re-runs or watchdog kill) is covered in E04 and section 8. The bot has no watchdog; a stuck row makes it stop at the start of every run, forever.
- F9 (stopping the bot the wrong way) is covered in section 4: disable the scheduled task, and never kill a run mid-item.
- F10 (one stuck item blocks the queue) is covered in E04 and Fix #1.
- F11 (cause hidden from ops) is covered in section 5, plus Fix #2, #7, #9 and #10.
- F12 (update didn't arrive or old build) is covered in section 10, plus Fix #11 and #13.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Any error inside one item (damaged PDF, missing temp file) is not caught, so the row stays Processing. Every later run then aborts, and there is no stale-row timeout. | One bad file stops all intake until dev resets it by hand (E04, incident a45615d) | `intake.py:59`, `:61`, `:68`, `:81`; abort at `:74-76` |
| 2 | The final "Mailbox unreachable" error is never logged. Login and network errors are not told apart. There is no alert. | Expired password = silent outage (E05, incident 6a62717) | `intake.py:34-37`, `:77` |
| 3 | An encrypted PDF is stored as **Done** although it was not delivered | Ops treats it as finished (F7) | `intake.py:62-63` |
| 4 | Failed rows are never retried, and ops can't re-queue them | Every share outage needs a dev ticket per item | `intake.py:66-67`, `:78` |
| 5 | Mail is marked read before the queue rows are saved, and rows are committed once after all messages | A crash while saving attachments loses referrals for good (E06) | `intake.py:44`, `:54` |
| 6 | Attachment file names are used as paths without being cleaned or checked for duplicates | Same name overwrites another referral (E07); a crafted name could write outside the temp folder | `intake.py:50`, `:65` |
| 7 | The quarantine note doesn't say whether the file was the wrong type or too big, and every named email part (logos, signatures) gets queued | Ops must guess, and their list fills with noise rows | `intake.py:48-49`, `:59-60` |
| 8 | The code never creates the temp folder, log folder or queue table | A fresh or rebuilt machine fails silently (E05) | `intake.py:18`, `:23`, `:73` |
| 9 | Crashes go only to Task Scheduler history, not the log. Failed items are logged at INFO level. Ops can't open the log anyway. | Causes are invisible to ops (F11) | `intake.py:23-24`, `:84` |
| 10 | The queue viewer Excel, the only screen ops has, is not in the repo | It can't be checked or rebuilt; its colours and columns are unknown | `README.md:6` |
| 11 | No version number anywhere and no update process | Nobody can confirm which code is running | `intake.py` (whole file) |
| 12 | The settings file with the mailbox credentials is committed to git, and the shared-folder path is hard-coded as a fallback | Credential leak; rotate the mailbox password and remove the file from history | `.env`, `intake.py:19` |
| 13 | The log contains a line format the current code does not write, and that line includes a sender's email address | The running bot may differ from the repo; personal data is in the log | `logs/intake.log:6` |
| 14 | The three "fix" commits add only comments | The incidents in section 9 are not fixed | `intake.py:89-91` |
| 15 | The temporary folder is never emptied | Patient PDFs pile up on the bot machine; disk fills | `intake.py:50-51` |
tests/evals/results/handoff/h-mail-intake-screens-r4.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-mail-intake/screens",
"rep": 4,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": false,
"misleading status flagged": true,
"stuck Processing item blocks the queue": false,
"20 MB limit": true
},
"cost_usd": 1.2779054,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-mail-intake-screens-p589__0w\\project",
"reply": "Na `HANDOVER.md` ezhudhi repo root la vechiten. Bot-a run pannala, code ethuvum maathala.\n\n| Item | Count |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 7 (E01\u2013E07) |\n| Failure classes | 11 covered, 1 not applicable (F2: bot entha window-um drive pannala) |\n| Fix before handover | 19 items |\n\n**Mukkiyamaana vishayangal:**\n- **3 \"fix\" commits-um oru comment line mattum add pannirukku.** Code maarave illa, so moonu incident-um innum fix aagala. Doc-la \"not fixed\" nu thaan pottiruken.\n- Password-protected PDF-a **Done**-nu mark pannudhu, aana share folder-ku copy aagala. Ithu Statuses-la \"needs follow-up\" nu irukku (E02), Fix list-la #3.\n- Oru row **Processing**-la stuck aanaa, ella intake-um nindrum. Ops-ku DB access illa, so idha Dev mattum thaan clear panna mudiyum (E04).\n- Mailbox login fail aanaa (E05), ops paakkura entha idathilayum theriyaadhu. Log-la mattum thaan irukku.\n\n**Assumption, nee check pannu:** Bot-ku sondha screen edhuvum illa. README-la ops use pannra `queue_viewer.xlsx`-a thaan \"bot's own screen\"-nu eduthukitten. Output folder-na shared intake folder. Log, database rendum ops-ku open panna mudiyaathu, so andha info ellam Dev-ku pogudhu, illa Fix list-la irukku.\n\n**Verification:**\n- Ella status/note string-um Statuses table-la irukku.\n- PHI, email, server path, `.env` value edhuvum doc-la illa. Sample log-la oru sender email irundhuchu, adha copy pannala. Adhu Fix #16-la irukku.\n- file:line refs Fix section-la mattum.\n\n**Also noticed:** `queue_viewer.xlsx` repo-la illa. So viewer-la status colours enna nu verify panna mudiyala. Doc-la \"bot sets no colours\"-nu thaan pottiruken.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel). | Ops access: which of the bot's places ops can open - the app's screens, the output folder, the log folder, the database. | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear.",
"stderr": ""
}
tests/evals/results/handoff/h-mail-intake-screens-r4.md
+165
View File
@@ -0,0 +1,165 @@
# Mail intake bot - ops handover
Version: none (the bot has no version number, see section 10) · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Every 10 minutes the bot checks the referrals mailbox for unread emails. It saves each attachment, checks that it is a PDF under 20 MB and not password-protected, copies it to the shared intake folder, and records one row per attachment in its queue. Ops follow the queue in the **queue viewer** workbook (`queue_viewer.xlsx`, Data > Refresh), which shows id, sender, subject, status and note for each row. The bot drives no desktop app or website. It has no window of its own and runs in the background on a Windows schedule.
## 2. How it works
```
Windows Task Scheduler (every 10 minutes)
└─ starts → Mail intake bot (no window)
├─ reads ──> Referrals mailbox (unread emails, marks them read)
├─ copies ─> Shared intake folder (the PDFs)
├─ writes ─> Queue (what you see in the queue viewer)
└─ writes ─> Bot log file (dev only)
```
- **Task Scheduler** starts the bot every 10 minutes. Each start is one run.
- **The bot** works through the mail and the queue, then stops until the next run.
- **Referrals mailbox:** the bot logs in and reads every unread email. When it reads an email, the mailbox marks it as read.
- **Shared intake folder:** this is the output folder. Every accepted PDF lands here under its original file name.
- **Queue:** one row per attachment, with a status and a note. You see it through the queue viewer.
- **Bot log:** one line per finished item, plus warnings. Ops cannot open it, so ask dev to check it.
**One run, step by step:**
1. The bot checks whether any queue row is still **Processing**. If one is, it stops the run straight away and nothing else happens (see E04).
2. It logs into the mailbox. If the login fails, it tries 3 times, 30 seconds apart, and then gives up the run.
3. It reads every unread email and saves every attachment. Each attachment becomes a **New** row.
4. It takes the New rows one by one, oldest first, and sets each one to **Processing**.
5. It checks the file. If it is not a PDF or is over 20 MB, the row becomes **Quarantined**. If it is password-protected, the row becomes **Done** with the note "encrypted - not readable" and the file is **not** copied.
6. It copies the PDF to the shared intake folder. The row becomes **Done** ("<n> pages saved"), or **Failed** if the folder could not be reached.
## 3. Before a run
- [ ] The bot is set up on the bot machine with its settings file, which holds the mailbox login and the shared folder location. Dev maintains this file and ops never edit it. A missing setting makes every run fail silently (E05).
- [ ] The bot's working folders and its queue already exist on the bot machine. On a new install, ask dev to confirm this. Without them, every run fails silently (E05).
- [ ] The mailbox password has not expired. When the mailbox password is changed, dev must update the settings file the **same day** (incident 3).
- [ ] The shared intake folder opens from the bot machine.
- [ ] Rules for senders. The bot can't fix these itself:
- Send each referral as a **PDF** under **20 MB**. Bad: `referral.docx`, or a 35 MB scan. Good: `referral.pdf`, 4 MB.
- **No password** on the PDF. A protected PDF is never delivered (E02).
- Give each attachment a **unique file name**. Bad: every referral called `scan.pdf`. Good: `scan_2026-10-23_0915.pdf`. Two files with the same name overwrite each other (E07).
- [ ] Signature logos and inline images in emails show up as Quarantined rows. This is normal and needs no action (E01).
## 4. During a run
- Nothing on the bot machine needs to stay hands-off. The bot drives no window, so using the machine doesn't disturb it.
- Don't rename, move or delete files in the shared intake folder while a row is Processing.
- **To stop the bot:** it is stopped by disabling its scheduled task in Windows Task Scheduler. Ops have no access to that, so raise a ticket. Don't end the bot's process by force: if it is killed mid-item, the row stays Processing and intake stops (E04).
- **A run looks stuck when** a row has been **Processing** for more than **10 minutes**, or no new rows have appeared for an hour while senders say they have sent referrals. The bot never recovers from a stuck Processing row by itself, so don't wait. Raise a ticket (E04, E05).
## 5. Where to look when something goes wrong
Ops can open two places: the **queue viewer** and the **shared intake folder**. Everything else needs dev.
| Question | Where to look |
|---|---|
| Did the run finish? | No place ops can open shows this directly. Refresh the queue viewer. If no row is **Processing** and new referrals keep appearing, runs are finishing. A Processing row older than 10 minutes means intake has stopped (E04). |
| Which items failed? | Queue viewer: filter status = **Failed** or **Quarantined**, and also status = **Done** with a note containing "encrypted". |
| Why did one item fail? | Queue viewer, **note** column of that row. For anything else, ticket dev with the row id. |
| Did the bot already retry? | For a single item: **no, never**. Failed and Quarantined rows stay that way. Only the mailbox login is retried, and you can't see that. |
| Did the PDF actually arrive? | Shared intake folder. Look for the file name shown in the queue viewer. |
| Which version is running? | The bot has no version number. Ask dev. |
## 6. Statuses
The bot sets no colours. Ops see these as plain text in the queue viewer's status and note columns.
| Status - note | Colour | Meaning | What ops does |
|---|---|---|---|
| New - (empty) | none | Attachment saved from the email, waiting for the next item step | Nothing. It is handled in this run or the next. |
| Processing - (note left from before) | none | The bot is working on this row right now. It normally lasts only seconds. | If it is still Processing after **10 minutes**, intake has stopped. See E04. |
| Done - "<n> pages saved" | none | PDF copied to the shared intake folder | Nothing |
| Done - "encrypted - not readable" | none | **Needs follow-up.** The PDF is password-protected and was **not** copied. It is labelled Done anyway. | See E02 |
| Quarantined - "not a PDF or over 20 MB" | none | Rejected. Either it is not a PDF, or it is over 20 MB. The note doesn't say which. | See E01 |
| Failed - "share unreachable: <reason>" | none | The PDF could not be copied to the shared intake folder. **The bot never retries it.** | See E03 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Row **Quarantined** - "not a PDF or over 20 MB" | The file name does not end in .pdf, or the file is bigger than 20 MB | Queue viewer, status and note | Business | Ops | Look at the file name in the viewer. If it is a logo or image from an email signature (.png, .jpg, .gif), ignore it. If it is a real referral, ask the sender to resend it as a PDF under 20 MB. | A file named .pdf that the sender confirms is under 20 MB is still Quarantined. Ticket with the row id. |
| E02 | Row **Done** - "encrypted - not readable", and the file is **not** in the shared intake folder | The PDF reports that it is password-protected | Queue viewer, note | Business | Ops | Treat it as **not delivered**. Ask the sender for a copy without a password. The new copy arrives as a new row. | Never, unless an unprotected resend still shows this note. |
| E03 | Row **Failed** - "share unreachable: <reason>" | Copying the PDF to the shared intake folder raised an error | Queue viewer, note | System | Ops, then Dev | 1. Check that the shared intake folder opens. 2. **Check whether the file is already in the folder.** A partial copy can be left there, so don't just assume it is missing. 3. Raise a ticket so dev can re-queue the row. The bot never retries Failed rows, and ops can't re-queue them. | Always, once the folder works again. Send the row ids, the time, and the note text. |
| E04 | One row stuck at **Processing**. No new rows appear at all. | At the start of every run the bot checks for a Processing row. If it finds one, it stops the run and doesn't even read the mailbox. This happens every 10 minutes until someone clears the row. | Queue viewer shows the Processing row. The cause is only in the bot log (dev). | System | Dev | Raise a ticket at once. The bot will **never** clear this itself, and all intake is stopped until dev does. Dev should mark the row Failed, not New, if the file itself caused the crash; otherwise it blocks again. | A row is Processing for more than 10 minutes. Send the row id, its file name, and the time you first saw it. |
| E05 | No new rows for a long time, **no** Processing row, and senders say they have sent referrals | Usually the mailbox login fails (expired password, network down) or a setting is missing. The bot tries the login 3 times, 30 s apart, then gives up the run. Each run 10 minutes later does the same. | Nothing ops can see. Warnings are in the bot log, and the final failure appears only in Task Scheduler's run history (dev). | System | Dev | Raise a ticket. If you know the mailbox password was changed recently, say so. | No new rows for 1 hour in working hours while mail is known to be arriving. Send the time of the last new row and of a referral that was sent but never appeared. |
| E06 | A sender says they emailed a referral, but no row exists for it | Emails with no attachment are marked read and skipped without any record. If a run crashes while reading the mailbox, the emails it already read are marked read but never queued, and are never picked up again. | Nowhere | Business / System | Ops, then Dev | 1. Ask the sender whether the file was attached, or only linked or pasted. 2. If it was attached, ask them to resend, and raise a ticket. | The email had an attachment. Send the time it was sent. Never send the email itself. |
| E07 | Two or more rows show the **same file name**. The file in the shared intake folder belongs to only one of them. | The bot saves each attachment under its original name. A later file with the same name overwrites the earlier one, both in its working folder and in the shared intake folder. | Queue viewer, file name repeated | Business | Ops, then Dev | Open the file in the shared intake folder and check which sender it belongs to. Ask the other senders to resend with a unique file name. | Always. Send the row ids. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Mailbox login fails | Tries again 30 s later | 3 tries (about 90 s) per run | Gives up that run. It tries again 10 minutes later, forever, with no alert (E05). |
| New rows left over from a run that stopped early | Picks them up in the next run | none | n/a |
| Anything else (Failed row, stuck Processing row, crash) | **Nothing** | n/a | Stays as it is until dev acts (E03, E04, E06) |
## 9. Known incidents
None of these three is fixed in code. Each "fix" recorded in the project history is a note only, and the bot behaves exactly as it did before. The project has no version numbers, so the incidents are listed newest first without one.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before handover | No new referrals arrived, and nobody saw an error. The mailbox password had expired. | The login failure was retried and then the run gave up. The final failure was not written anywhere ops could see. | Not fixed. Use E05: raise a ticket when no new rows appear for an hour. When the mailbox password changes, dev updates the settings file the same day. | Partly |
| Before handover | Referrals arrived empty or unreadable in the shared intake folder | Password-protected PDFs were copied without a check | The bot now skips them, but marks them **Done** with "encrypted - not readable". Use E02. | Yes |
| Before handover | No referrals arrived for a whole day, and one row sat at Processing | A crash left one row at Processing, which blocks every later run. Someone reset it by hand. | Not fixed. Use E04: raise a ticket after 10 minutes. | Partly (ops can spot it, but only dev can clear it) |
## 10. Releases and version check
- **How an update arrives:** dev replaces the bot's files on the bot machine by hand. There is no updater.
- **How to confirm it applied:** the bot has no version number, so ops can't check this. Ask dev to confirm in the ticket that the change is live.
- **If the version did not change:** n/a. Raise a ticket if behaviour described as fixed still happens.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue.
Send:
- the exception ID from section 7
- the queue row id(s) and file name(s)
- the time you first saw the problem
- the time of the last new row in the queue viewer
Never send: passwords, the settings file, the referral PDFs, patient details, or sender email addresses in the ticket or in chat.
## 12. Failure-class check
- F1 (missing prerequisite) is covered in section 3 and E05, and in Fix items 5 and 13.
- F2 not applicable: the bot drives no window. It talks to the mailbox and copies files directly.
- F3 (target slower or different) is covered in E03 and E05, and in Fix item 19 (no time limit on the copy to the shared folder).
- F4 (input read differently) is covered in the section 3 sender rules, E01 and E07, and in Fix items 8 and 10.
- F5 (done when nothing was done): the normal Done is set only after the copy succeeds. The encrypted Done is covered under F7.
- F6 (done but not confirmed) is covered in E03, step 2: check the folder before re-queueing, because a partial copy can exist.
- F7 (business exception shown as success) is covered in section 6, E02 and Fix item 3.
- F8 (endless re-runs) is covered in section 8, E04 and E05: the bot repeats the failed login and the Processing stop every 10 minutes forever. See Fix items 2 and 4.
- F9 (stopping the wrong way) is covered in section 4: stop it only through the scheduled task, via a ticket, and never kill it mid-item.
- F10 (one stuck item blocks the queue) is covered in E04 and Fix item 2.
- F11 (cause hidden from ops) is covered in section 5, and in Fix items 5, 9 and 12.
- F12 (update didn't arrive) is covered in section 10 and Fix item 11.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The three "fix" commits (30ad14c, c0af495, fab0a64) only append a comment, so none of the incidents is fixed in code | Ops and dev believe the incidents are fixed | `intake.py:89-91` |
| 2 | A row left in Processing makes every later run abort before it reads the mailbox. Nothing catches errors inside `process()`, and nothing resets a stale row. | One bad PDF or one killed run stops all intake until dev edits the DB. Ops can't clear it. | `intake.py:57-68`, `intake.py:74-81` |
| 3 | An encrypted PDF is stored as `Done` although it is not copied | Looks like success. The referral is silently missing from the share. | `intake.py:62-63` |
| 4 | `RuntimeError("Mailbox unreachable")` is never caught or logged. A login failure is retried like a network failure, 90 s per run, forever. | An expired password goes unnoticed. Nothing alerts anyone. | `intake.py:34-37`, `intake.py:77` |
| 5 | Unhandled crashes (a missing `.env` key, missing `data/` or table, a corrupt PDF, a fetch error) write no log line. The traceback goes to the console or Task Scheduler only. | Nobody can tell why intake stopped | `intake.py:31-32`, `intake.py:41-54`, `intake.py:61`, `intake.py:71-84` |
| 6 | RFC822 fetch marks mail read before the rows are committed | A crash mid-fetch loses referrals for good (E06) | `intake.py:43-54` |
| 7 | `Failed` rows are never retried. Only `New` is selected. | A short share outage needs a dev DB edit for every row | `intake.py:66-67`, `intake.py:78` |
| 8 | Attachments are saved under their original name in `data/tmp` and the share, with no unique prefix | Same-name referrals overwrite each other (E07) | `intake.py:50`, `intake.py:65` |
| 9 | One note covers both "not a PDF" and "over 20 MB" | Ops can't tell the sender what to fix | `intake.py:59-60` |
| 10 | Every named MIME part is queued, including inline images | Quarantined noise in the viewer | `intake.py:47-49` |
| 11 | There is no version string anywhere | Ops can't confirm an update was applied | `intake.py` (none) |
| 12 | Ops have no access to the log and there are no alerts. Run-level failures (E04, E05) are invisible in the queue viewer. | Ops only notice when senders complain | `intake.py:23-24`, `intake.py:75` |
| 13 | Nothing creates `data/`, `data/tmp/`, `logs/` or the `queue` table | A fresh install crashes on every run | `intake.py:17-18`, `intake.py:23`, `intake.py:73` |
| 14 | Nothing guards against two runs overlapping | The same `New` row can be processed twice if a run takes over 10 minutes | `intake.py:71-84` |
| 15 | `main()` writes a `BOT_WAS_RUN` file on every run, and its purpose is unknown | Unexplained file in the bot folder | `intake.py:72` |
| 16 | The sample log contains a line in an old format that includes a sender email address. The log never rotates. | Possible PHI/PII in a file that may be shared | `logs/intake.log:6`, `intake.py:23` |
| 17 | The share path falls back to a hardcoded server path when `INTAKE_SHARE` is missing | A missing setting silently copies to the wrong place | `intake.py:19` |
| 18 | `queue_viewer.xlsx` is not in the repo, so its columns and colours can't be verified | The handover relies on an unversioned file | `README.md:6` |
| 19 | The copy to the share has no timeout | A hung share leaves the row in Processing, which then becomes E04 | `intake.py:65` |
tests/evals/results/handoff/h-mail-intake-screens-r5.json
+31
View File
@@ -0,0 +1,31 @@
{
"case": "h-mail-intake/screens",
"rep": 5,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": false,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"misleading status flagged": false,
"stuck Processing item blocks the queue": true,
"20 MB limit": true
},
"cost_usd": 0.7572796,
"subagents": 0,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-mail-intake-screens-9fxhg12l\\project",
"reply": "`HANDOVER.md` ready \u2014 repo root la irukku. Bot-a run pannala; code, README, log, git history mattum padichen.\n\n| Item | Result |\n|---|---|\n| Sections | 13 (template order maaraama) |\n| Exceptions | 10 (E01\u2013E10) |\n| Failure classes | 11 covered, 1 not applicable (F2 \u2014 bot entha window-um drive pannala) |\n| Fix before handover | 13 items |\n\n**Mukkiyamana findings:**\n- Moonu \"fix\" commits-um (e59f92b, 5b6f24d, 958759d) oru comment line mattum add pannirukku \u2014 code maaravae illa. Stuck \"Processing\" item innum ella run-aiyum block pannum (E02).\n- Ops-ku intake folder mattum dhaan theriyum. Statuses, notes, log ellam avangalukku theriyaathu. README la sollra `queue_viewer.xlsx` repo la illa, code-um adha create pannala.\n- Password-protected PDF-a \"Done\"-nu mark pannudhu, aana intake folder-ku copy aagala.\n- `.env` (mailbox credentials) git-la commit aagirukku. `logs/intake.log`-la oru sender email address irukku. Rendu-um history-la irundhu remove pannanum, password-um rotate pannanum.\n\n**Verification:** file:line refs ellam section 13-la mattum dhaan irukku (grep pannitten). Doc-la credentials, IPs, server names, email address edhuvum illa.\n\n**Also noticed:** repo romba chinnadhu (oru 90-line script), adhanala skill sonna rendu subagents vechu scan pannala, naane padichen. Commit history-la `.env` irukkradhu oru security issue \u2014 doc mattum ezhudhinen, adha fix pannala.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel) | Ops access: which of the bot's places ops can open - the app's screens, the output folder, the log folder, the database | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear",
"stderr": ""
}
tests/evals/results/handoff/h-mail-intake-screens-r5.md
+158
View File
@@ -0,0 +1,158 @@
# Mail intake bot - ops handover
Version: none (the project has no version number; current code is commit 958759d) · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Every 10 minutes the bot reads new (unread) mail in the referrals mailbox, saves every attachment, checks that it is a PDF of 20 MB or less, and copies each good PDF into the **shared intake folder**. It records every attachment in its own queue with a status and a short note. It drives no desktop apps or websites: it talks to the mailbox and the shared folder directly, so there are no windows to keep clear while it runs.
**What ops can see:** only the shared intake folder (the PDFs the bot copied there). The queue, its statuses and notes, and the log are not open to ops today (see section 13). In practice the one sign ops has is: **new referral PDFs stop appearing in the intake folder.**
## 2. How it works
```
Windows Task Scheduler (every 10 minutes)
└─ starts → Mail intake bot ──reads──> Referrals mailbox (unread mail)
│
├─ saves → bot's temporary folder (every attachment)
├─ copies → Shared intake folder (good PDFs) <- ops can open
├─ writes → Queue (one row per attachment: <- ops cannot open
│ sender, subject, file, status, note)
└─ writes → Bot log <- ops cannot open
```
- **Task Scheduler** starts the bot every 10 minutes. Nobody needs to click anything.
- **Referrals mailbox:** the bot reads only unread mail. Reading a mail marks it as read, so the bot never looks at it again.
- **Temporary folder:** every attachment is first saved here under its original file name.
- **Shared intake folder:** the bot's output. Each good PDF lands here under its original file name.
- **Queue:** the bot's record of every attachment and what happened to it.
- **Bot log:** one line per item processed, plus mailbox connection warnings.
**One run, step by step:**
1. Check the queue. If any item is still marked Processing from an earlier run, **stop the whole run** (see E02).
2. Sign in to the mailbox (up to 3 tries, 30 s apart).
3. For each unread mail: mark it read, save each attachment to the temporary folder, add one queue row per attachment as New.
4. For each New item, oldest first: mark it Processing, check it (PDF? 20 MB or less? password-protected?), copy it to the intake folder, then set the final status and note.
## 3. Before a run
The bot runs on its own; ops has nothing to start. These are the rules that decide whether a referral makes it through. Pass them on to referrers when a referral goes missing.
- [ ] **The referral must be an attachment**, not text or a link in the mail body. A mail with no attachment is read and then dropped with no record (E07).
- Bad: "Please see the referral below" with the details typed in the mail. Good: the referral attached as a `.pdf` file.
- [ ] **PDF only, 20 MB or less.** Word files, images, zip files and PDFs over 20 MB are set aside, not copied (E03).
- Bad: `referral.docx`, `scan.jpg`, a 35 MB PDF. Good: `referral.pdf` of 3 MB.
- [ ] **Not password-protected.** A password-protected PDF is marked Done but is **not** copied (E04).
- [ ] **A unique file name.** If two referrals arrive with the same file name (for example `scan.pdf`), the later one replaces the earlier one in the intake folder (E08).
- Bad: every referral named `referral.pdf`. Good: a name that includes the date or a reference, such as `referral_2026-10-23_A.pdf`.
- [ ] Set-up (Dev confirms once, at install): the settings file with the mailbox sign-in is next to the bot, the bot's data folder, temporary folder and queue exist, and the shared intake folder setting points to the right folder. Without these the bot stops at the start of every run (E10).
- [ ] The bot has no version number to check (see section 10).
## 4. During a run
- There is nothing to keep clear on screen: the bot drives no windows.
- **Do not stop the bot mid-run** (for example by ending its process in Task Manager). An item caught half-way stays marked Processing, and then **every later run stops at step 1** until a developer clears it (E02).
- To stop the bot: ops cannot. Raise a ticket asking Dev to **disable the scheduled task**; Dev waits for the current run to finish first.
- A run looks stuck when no new PDFs appear in the intake folder although referrals are expected. A normal run takes seconds to a couple of minutes. The bot gives up on the mailbox after about 1.5 minutes (3 tries x 30 s) and tries again at the next 10-minute start. **If no new PDF has appeared for 30 minutes during working hours while referrals are expected, raise a ticket.** The bot has no watchdog of its own for a run that hangs.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | Ops cannot see this. The only sign is new PDFs in the shared intake folder (check the file times). If none for 30 min while referrals are expected, raise a ticket. |
| Which items failed? | Ops cannot see this (the queue is not open to ops). Raise a ticket and ask Dev for queue items not Done. |
| Why did one item fail? | Ops cannot see this (the status note and the log are not open to ops). Ask Dev in the ticket, giving the referral's sender, subject and roughly when it was sent. |
| Did the bot already retry? | It retries only the mailbox sign-in (section 8). Nothing else is retried. A Failed or Quarantined item stays that way until Dev resets it. |
| Did a referral arrive at all? | Shared intake folder: look for its file name. Not there means it was set aside (E03, E04), failed (E05), dropped (E07, E09), replaced (E08) or the bot is blocked (E01, E02, E06, E10). |
| Which version is running? | No version number (section 10). |
## 6. Statuses
The statuses live in the bot's queue. The bot has no screen, so there are no colours; ops sees them only when Dev reads them out (section 13, item 2).
| Status | Colour | Meaning | What ops does |
|---|---|---|---|
| New | none (no screen) | Attachment saved from the mailbox, waiting to be checked. Picked up later in the same run. | nothing |
| Processing | none (no screen) | The bot is working on this item right now. **If it stays Processing after the run, every later run stops** (E02). | If no new PDFs for 30 min, raise a ticket (E02) |
| Done - "<n> pages saved" | none (no screen) | PDF copied to the intake folder. | nothing |
| Done - "encrypted - not readable" | none (no screen) | **Needs follow-up. Not copied to the intake folder** although it says Done: the PDF is password-protected. | Ask the sender for a copy without a password (E04) |
| Quarantined - "not a PDF or over 20 MB" | none (no screen) | Not a PDF, or larger than 20 MB. Not copied. Never retried. | Ask the sender to resend as a PDF of 20 MB or less (E03) |
| Failed - "share unreachable: <reason>" | none (no screen) | The intake folder could not be reached when copying. Not copied. **Never retried by the bot.** | Raise a ticket so Dev re-queues it once the folder is back (E05) |
## 7. Exception catalogue
Almost every problem looks the same to ops: **a referral PDF does not appear in the intake folder.** Quote the ID in your ticket when you can tell which one it is; if you can't, say "missing referral" and give the sender, subject and time sent.
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | No new PDFs for every run since some point; referrers say they sent mail | It tries to sign in to the mailbox 3 times, 30 s apart. If all 3 fail (wrong or expired password, mailbox or network down) the run ends. It tries again 10 minutes later. | Bot log: "IMAP connect failed (1/3)…(3/3)" with the reason. Not visible to ops. | System | Bot, then Dev | Wait 30 minutes; the bot retries every 10 minutes by itself. | Still no new PDFs after 30 minutes. Send: time the PDFs stopped, and ask Dev to check the mailbox password (it has expired before). |
| E02 | No new PDFs at all, for hours, and it never recovers by itself | At the start of every run it checks for any item still marked Processing. If it finds one it stops the run before reading any mail. This happens after an earlier run was cut off half-way (crash, stopped by hand, machine restart). | Bot log: "Previous item still Processing - aborting", every 10 minutes. Not visible to ops. | System | Dev | Raise a ticket at once. The bot **never** clears this by itself. New mail waits unread in the mailbox meanwhile and is picked up once it is cleared. | Always. Send: time the PDFs stopped. Ask Dev to check the stuck item in the intake folder and in the target system before resetting it. |
| E03 | One referral missing from the intake folder; other referrals arrive normally | It checks the file name ends in ".pdf" and the size is 20 MB or less. | Queue: Quarantined, note "not a PDF or over 20 MB". Not visible to ops. | Business | Ops | Ask the sender to resend as a PDF of 20 MB or less (split large scans). The resent mail is picked up automatically. | The sender says it was already a PDF under 20 MB. Send: sender, subject, time sent. |
| E04 | One referral missing from the intake folder; the sender says it was a protected PDF | It opens the PDF and checks whether it is password-protected. If so it stops there and does **not** copy it, but marks it Done. | Queue: Done, note "encrypted - not readable". Not visible to ops. | Business | Ops | Ask the sender for a copy without a password. | The sender's copy has no password. Send: sender, subject, time sent. |
| E05 | Several referrals missing from about the same time; the intake folder itself was unavailable or full | The copy to the intake folder fails. | Queue: Failed, note "share unreachable: <reason>". Bot log line for that item. Not visible to ops. | System | Dev | Raise a ticket once the intake folder works again. The bot does **not** retry Failed items; Dev must re-queue them. | Always. Send: the time window when the folder was down. |
| E06 | No new PDFs at all, starting right after one particular referral arrived | A damaged PDF (or a file that vanished from the temporary folder) makes the bot crash while checking it. The item is left as Processing, which then triggers E02 on every later run. | Item stays Processing. The crash reason is **not** written to the log. Not visible to ops. | System | Dev | Raise a ticket as for E02. Then ask the sender for a fresh copy of that referral. | Always. Send: time the PDFs stopped, and the last referral you know arrived. |
| E07 | Referral missing; the sender put the details in the mail body or sent a link | Mails with no attachment are read and skipped. No queue row is made. | Nowhere. The mail is only marked read. | Business | Ops | Ask the sender to resend with the referral attached as a PDF. | The sender is sure there was an attachment. Send: sender, subject, time sent. |
| E08 | A PDF in the intake folder holds a different referral than its name suggests, or one of two referrals is missing | Files keep the sender's file name. A second file with the same name replaces the first, in the temporary folder and in the intake folder. | Nowhere as an error; both queue rows look normal. | Business | Ops, then Dev | Ask senders to use unique file names. Ask the sender of the lost referral to resend it with a different name. | Always tell Dev (they can check which referral was replaced). Send: the file name and both senders. |
| E09 | Referrals from a short window missing, though nothing else looks wrong | If the bot crashes while saving attachments, the mails it already read stay marked read but get no queue rows. They are never picked up again. | Nowhere. Not in the queue, not in the log. | System | Dev | Raise a ticket. Dev must find these mails in the mailbox and mark them unread. | Always. Send: the time window and any known senders. |
| E10 | No new PDFs ever, starting right after the bot was installed or moved | The settings file, the bot's data folder, temporary folder or queue is missing, so the bot stops at the start. | The reason is **not** written to the log. Not visible to ops. | System | Dev | Raise a ticket. | Always. Send: when the bot was installed or moved. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Mailbox sign-in fails | Tries again after 30 s | 3 tries (about 1.5 minutes) | Run ends. Task Scheduler starts a fresh run 10 minutes later, forever. Raise a ticket after 30 minutes without new PDFs. |
| Anything else (folder down, bad file, stuck item) | Nothing | - | Item stays Failed / Quarantined / Processing until Dev acts. A stuck Processing item stops every run (E02). |
## 9. Known incidents
Newest first. Note: the three fix records in the project history changed **only a comment line**, not the bot's behaviour (see section 13, item 1). Treat these problems as still possible.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| commit 958759d | Every run failed with no sign anywhere; no new PDFs | The mailbox password expired | The sign-in error goes to the bot log, which ops can't open. Treat "no new PDFs for 30 min" as E01 and raise a ticket. | Partly: ops can spot it and escalate; only Dev can fix the password |
| commit 5b6f24d | Password-protected referrals arrived in the intake folder as empty files | The bot copied PDFs it could not read | Now marked Done "encrypted - not readable" and **not copied**. Ask the sender for an unprotected copy (E04). | Partly: ops acts once Dev tells them which referral |
| commit e59f92b | Nothing came through for a whole day | A crash left one item Processing; every run then stopped (E02). Ops reset it by hand. | Still happens. Ops no longer has queue access, so raise a ticket at once (E02). | Partly: ops can spot it within 30 minutes; only Dev can clear it |
## 10. Releases and version check
- How an update arrives: there is no updater. A developer replaces the bot's files on the machine.
- How to confirm it applied: the bot has no version number and no screen, so ops cannot. Ask Dev to confirm in the ticket which commit is running.
- If the version did not change: ask Dev in the RPA Support queue.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue. Send: the exception ID from section 7 (or "missing referral"), the time new PDFs stopped or the time the referral was sent, the sender and subject of the affected mail, and the file name if you know it.
Never send: passwords, the settings file, the referral PDFs themselves, or patient details in the ticket text.
## 12. Failure-class check
- F1 covered in section 3 (set-up item) and E10; gaps in section 13, items 10 and 13.
- F2 not applicable: the bot drives no windows; it talks to the mailbox and the shared folder directly.
- F3 covered in E01 and section 8 (mailbox sign-in timing); no time limit on a hanging mailbox connection in section 13, item 9. No screens, so screen-loading checks do not apply.
- F4 covered in section 3 input rules and E03, E07, E08.
- F5 covered in E04 and E09; section 13, items 4 and 7.
- F6 covered in E09 (mail marked read before the bot recorded it); section 13, item 7.
- F7 covered in section 6 and E04 ("Done" for a protected PDF that was not copied); section 13, item 4.
- F8 covered in E02 (the same "aborting" stop repeats every 10 minutes forever) and section 8.
- F9 covered in section 4 (never stop it mid-run; ask Dev to disable the scheduled task).
- F10 covered in E02 and E06; section 13, items 1 and 5.
- F11 covered in section 5 (ops sees only the intake folder); section 13, items 2 and 3.
- F12 covered in section 10 and section 9 note (fix records did not change behaviour); section 13, items 1 and 11.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The three "fix" commits (e59f92b, 5b6f24d, 958759d) each add only a comment line; no code changed. The stuck-Processing abort is unchanged. | Ops and dev believe these incidents are fixed; E02 can still block a whole day. | `intake.py:89-91`; abort at `intake.py:74-76` |
| 2 | No ops view of the queue. README says ops use `queue_viewer.xlsx`, but it is not in the repo and nothing creates it; ops have no database access. | Ops can't see any status or note, so every exception becomes a Dev ticket. | `README.md:6`; queue at `intake.py:17` |
| 3 | Crashes are not logged: "Mailbox unreachable" and PDF read errors are uncaught and go to stderr only; and the log folder is not open to ops anyway. No alert of any kind. | E06/E10 causes are invisible to everyone; ops learns only from missing PDFs. | `intake.py:37`, `intake.py:61`, `intake.py:77`; logging `intake.py:23-24` |
| 4 | Password-protected PDF is stored as Done although it was not copied. | Reads as success; referral silently missing. | `intake.py:62-63` |
| 5 | A damaged PDF or missing temp file crashes the run and leaves the item Processing, which blocks every later run. No try/except per item, no stale-Processing reset. | One bad attachment stops all intake until Dev steps in. | `intake.py:59`, `intake.py:61`, `intake.py:79-82`, `intake.py:74` |
| 6 | Failed items are never retried (only New is picked up). | A short share outage loses referrals until Dev re-queues by hand. | `intake.py:66-67`, `intake.py:78` |
| 7 | Mails are marked read during fetch, but queue rows are committed only after all mails; a crash in between loses those referrals. | Silent referral loss (E09). | `intake.py:42-54` |
| 8 | Attachment file names are used as-is: same name overwrites in temp and in the share; names are not sanitised. | Referrals silently replaced (E08); unsafe file names reach the share. | `intake.py:50-51`, `intake.py:65` |
| 9 | No timeout on the mailbox connection. | A hung connection can stall the bot with no sign. | `intake.py:31` |
| 10 | Mail with no attachment is dropped with no record. | Ops can't tell "never sent" from "dropped". | `intake.py:47-49` |
| 11 | No version string anywhere. | Nobody can confirm which build is running. | `intake.py:1-2` |
| 12 | `.env` (mailbox credentials) and `logs/intake.log` are committed to git; the log contains a sender email address. | Credential and personal-data exposure; rotate the password and remove both from history. | `.env`, `logs/intake.log:6` |
| 13 | Data folder, temp folder and queue table are never created; missing settings stop the bot at start with no clear message. | A fresh install or move fails silently (E10). | `intake.py:17-18`, `intake.py:31-32`, `intake.py:73` |
tests/evals/results/handoff/h-packaged-desktop-full-r1.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-packaged-desktop/full",
"rep": 1,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"display scaling rule": true,
"version check": false,
"shortcut prerequisite": true,
"silent update failure flagged": true
},
"cost_usd": 1.3562872000000001,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-packaged-desktop-full-hy65g4yp\\project",
"reply": "I wrote `HANDOVER.md` in the repo root for **LedgerBot v3.2.0**, and changed no code. I only read code, history and logs; I never ran the bot.\n\n| Check | Result |\n|---|---|\n| Sections | 13, same order as the template |\n| Exceptions catalogued | 11 (E01\u2013E11); every Dev row says what to send in #rpa-support |\n| Failure classes | 10 covered, 2 not applicable (F8: no auto-rerun or watchdog; F10: no batches or locks) |\n| Statuses | All 6 Result + Note combinations listed; the file has no colours |\n| Sensitive data | No customer data, passwords, internal addresses or `.env` values; code references only in section 13 |\n\n**Fix before handover (14 items for dev):**\n1. The launcher still hides update errors, so the 30cb604 \"fix\" only added a comment.\n2. \"Posted\" is written without confirming the save worked.\n3. Crash causes are never logged.\n4. All results are lost if a run stops part-way.\n5. Screenshots overwrite each other, and the run summary is never saved.\n6. No check that LedgerPro has focus; a failed Save leaves a half-filled form.\n7. Fixed waits instead of real checks, and nothing checks that the LedgerPro export is fresh, which can post entries twice.\n8. Duplicate or blank references silently drop out of the results.\n9. \"Missing\" still shows on rows that were posted.\n10. The version is not visible anywhere, and the build spec is a commented-out stub.\n11. No display-scaling check at startup.\n12. `.env` with credentials is committed to git.\n13. An old log contains a customer name.\n14. pyautogui is not pinned.\n\n**Where I used my own reading of the code:**\n- I couldn't find who saves `ledger_export.xlsx` into `data/`. The doc tells ops to confirm it with dev.\n- The bot has no window, so the doc tells ops to check the version by the Date modified of `LedgerBot.exe`.\n- Ops can't open the `data` folder, so they can't check the export themselves before a run.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions mean where the code doesn't make it clear?",
"stderr": ""
}
tests/evals/results/handoff/h-packaged-desktop-full-r1.md
+187
View File
@@ -0,0 +1,187 @@
# LedgerBot - ops handover
Version 3.2.0 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
LedgerBot reconciles the daily bank statement Excel against the LedgerPro desktop accounting app. You give it the statement file. It opens LedgerPro, exports LedgerPro's entries to Excel, and compares the two by reference:
- When the reference and amount match, it does nothing.
- When the amount differs, it reports the difference and changes nothing.
- When the reference is not in LedgerPro, it **types the entry into LedgerPro and saves it**.
It writes one result file per run, `data/output/Reconcile_<date>_<time>.xlsx`. It drives one app, **LedgerPro**, by finding buttons on the screen from pictures and clicking them. Entries that are in LedgerPro but not in the statement are ignored and are not reported.
## 2. How it works
```
launcher.exe (what you start, with the statement file)
├─ checks the update server → downloads and unpacks a new build if there is one
└─ starts → LedgerBot.exe (no window of its own) ──drives──> LedgerPro
│ (do not touch while it runs)
├─ writes → data/output/Reconcile_<date>_<time>.xlsx (only at the very end)
├─ writes → logs/ledger_<date>.log (one line per entry)
└─ writes → logs/notfound_<button>.png (screenshot when a button is not found)
```
- **launcher.exe** checks for an update, then starts the bot and closes itself straight away.
- **LedgerBot.exe** does all the work. It has no window and no Stop button.
- **LedgerPro** is the app the bot clicks and types in. While the bot runs, the mouse and keyboard belong to the bot.
- **The Reconcile file** is the result of the run. It is saved only after the last entry is done.
- **The daily log** gets one line per entry: the reference, the result and the note.
- **The screenshot** is a picture of the whole screen at the moment a button could not be found.
**One run, step by step:**
1. Starts LedgerPro from the LedgerPro shortcut on the Desktop.
2. Waits 15 seconds, then clicks LedgerPro's Export to Excel button.
3. Waits 10 seconds, then reads the LedgerPro export.
4. Reads the bank statement.
5. For each statement row: OK, Mismatch, or a missing entry, which it posts with New Entry → reference → Tab → amount → Save.
6. Saves the Reconcile file.
## 3. Before a run
- [ ] **Windows display scaling is 100% and the resolution is 1920x1080** (Settings → System → Display). At 125% the bot cannot find the buttons (incident in section 9). LedgerPro must open on that screen.
- [ ] The **LedgerPro shortcut** (named exactly `LedgerPro`) is on your Desktop or the Public Desktop.
- [ ] LedgerPro **opens already signed in** and ready within 15 seconds. The bot does not sign in.
- [ ] LedgerPro's Export to Excel saves to the bot's `data` folder as `ledger_export.xlsx`, overwriting the old one. The bot only clicks Export. If an old export is still there, **the bot compares against yesterday's ledger and can post entries twice** (E07). Ask dev to confirm this is set up on your machine.
- [ ] The `logs`, `data` and `data/output` folders exist next to `LedgerBot.exe`. If `logs` is missing, nothing starts. If `data/output` is missing, the whole run's results are lost at the end.
- [ ] The statement Excel follows these rules. The bot reads the first sheet, starting from row 2:
| Rule | Good | Bad |
|---|---|---|
| Row 1 is a header | `Reference \| Amount` | The first entry in row 1. It is skipped. |
| Column A holds the reference, column B the amount | `TX-1001 \| 250.00` | Amount in column C |
| The amount is a plain number | `250.00` | `$250.00`, `250,00`, `n/a`. The whole run stops with no result file. |
| Each reference appears once | One row per reference | The same reference twice. Only the last one is checked, and the other disappears from the result file. |
| No blank reference in the middle | Every row has a reference | A blank reference. That row is skipped without any message. |
- [ ] Start the bot by **dropping the statement Excel onto `launcher.exe`**, or by starting `launcher.exe` with the file path. If no file is given, the bot stops at once and nothing happens.
- [ ] After a new release, check the version (section 10).
## 4. During a run
- **Don't use the mouse or keyboard on this machine.** The bot types into whatever window has focus. If you click elsewhere, the reference and amount are typed into that window.
- Don't open other windows over LedgerPro, and don't move LedgerPro to another monitor.
- **Don't move the mouse into a screen corner.** That is the emergency stop, see below.
- To stop the bot there is no Stop button. Use one of these:
- Move the mouse hard into a corner of the screen. The bot stops at its next click or keystroke.
- Or end **LedgerBot.exe** in Task Manager.
- Closing LedgerPro does **not** stop the bot. It keeps clicking and typing into whatever is on the screen.
- Closing launcher.exe does nothing, because it has already exited.
- **Either way, no Reconcile file is written.** Use the log (E05).
- A run looks stuck when LedgerPro does nothing for more than about 20 seconds. The bot waits 20 seconds for each button, then gives up on that entry. There is no other watchdog. The bot finishes or stops by itself, so wait for it.
- The run is finished when a new `Reconcile_<date>_<time>.xlsx` appears in `data/output`.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | A new `Reconcile_<date>_<time>.xlsx` in `data/output` with the time of this run. No new file means the run stopped early (E04 / E05). |
| Which entries failed? | The Reconcile file, Result column = `Error` or `Mismatch` |
| Why did one entry fail? | The Reconcile file, Note column. Then `logs/ledger_<date>.log`: search for the reference. |
| What did the screen look like when a button was not found? | `logs/notfound_new_entry.png`, `logs/notfound_save_btn.png`, `logs/notfound_export_excel.png`. Each one shows **only the most recent** failure for that button. Check its time. |
| The run stopped early. What was done? | `logs/ledger_<date>.log`. Each line written during this run is an entry the bot finished. A line saying `Missing Posted` means the bot posted that entry. |
| Did the bot already retry? | It never retries an entry. It only waits up to 20 s for each button (section 8). |
| Was the entry really saved in LedgerPro? | Only LedgerPro can tell you. The bot does not check after Save (E06). |
| Which version is running? | Nothing on screen shows it. See section 10. |
## 6. Statuses
The Reconcile file has no colours. Each row's status is the Result column plus the Note column.
| Result | Note | Colour | Meaning | What ops does |
|---|---|---|---|---|
| `OK` | (empty) | none | The reference and amount match LedgerPro | nothing |
| `Missing` | `Posted` | none | **The entry was not in LedgerPro and the bot posted it.** "Missing" here means "was missing, now posted". The bot did not confirm that the save worked. | Spot-check a few in LedgerPro (E06) |
| `Missing` | `Not in LedgerPro - posting` | none | Temporary note. It is always replaced by `Posted` or by an error before the file is saved. | You should never see it. If you do, contact Dev. |
| `Mismatch` | `LedgerPro has <amount>` | none | **Needs follow-up.** The reference exists, but the amount in LedgerPro differs by more than 0.01. The bot changed nothing. | Correct it by hand (E01) |
| `Error` | `Button new_entry.png not found on screen` | none | The entry was not posted. The bot could not find New Entry. | E02 |
| `Error` | `Button save_btn.png not found on screen` | none | The entry was typed but **not saved**. A half-filled form may still be open in LedgerPro. | E03 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Row `Mismatch`, note "LedgerPro has 240.00" | The reference is in both files, but the amounts differ by more than 0.01 | Reconcile file; log line | Business | Ops | Check the bank statement and LedgerPro, then correct the right one by hand. The bot never fixes amounts. | Many rows mismatch by the same pattern, which suggests a wrong export or a wrong file |
| E02 | Row `Error`, note "Button new_entry.png not found on screen" | It looks for the New Entry button picture for 20 s. When it is not found, it saves a screenshot and moves on to the next entry. | Reconcile file; log line; `logs/notfound_new_entry.png` | System | Ops, then Dev | 1) Check that display scaling is 100% and 1920x1080. 2) Open the screenshot: was a popup or another window covering LedgerPro? 3) Post the entry by hand in LedgerPro, or re-run (see E07 first). | Scaling is 100%, nothing covered LedgerPro, and it still happens. Send the time, the reference, the screenshot and the log file. |
| E03 | Row `Error`, note "Button save_btn.png not found on screen" | The bot clicked New Entry and typed the reference and amount. It then could not find Save for 20 s, saved a screenshot and moved on. | Reconcile file; log line; `logs/notfound_save_btn.png` | System | Ops | 1) In LedgerPro, look for an **unsaved, half-filled entry** and cancel it, or save it if it is correct. 2) **Check the next entry after this one too.** The bot may have typed it into the same open form. 3) Check scaling as in E02. 4) Post the entry by hand. | Same as E02 |
| E04 | **No new Reconcile file, and no new lines in today's log** | The run stopped before checking any entry. Possible causes: LedgerPro shortcut not found; LedgerPro not ready within 15 s; Export button not found (a new `logs/notfound_export_excel.png` appears); the export file missing; a statement amount that is not a number; no statement file given. **The cause is not written anywhere.** | Nothing, except possibly the export screenshot | System | Ops, then Dev | Work through section 3: shortcut, LedgerPro signed in, scaling, the statement amounts are plain numbers, you dropped the file onto launcher.exe. Then run again. Nothing was posted, so a re-run is safe. | It still fails after the checklist. Send the time, the statement file name (not its contents) and any `notfound_export_excel.png`. |
| E05 | **No new Reconcile file, but today's log has entry lines from this run** | The run stopped part-way, or at the end while saving: you used the corner stop or Task Manager, the `data/output` folder is missing, or an unexpected error happened. The cause is not written anywhere. | Log lines only | System | Ops, then Dev | 1) Use the log as the result. Every `Missing Posted` line **was posted**. 2) Entries with no line were not handled. 3) Re-run **only** with a fresh LedgerPro export (E07), or post the rest by hand. | Always report it unless you stopped the bot yourself. Send the time, the last log line and the log file. |
| E06 | An entry marked `Missing` / `Posted`, but it is not in LedgerPro | After clicking Save, the bot does not check that LedgerPro saved the entry. It writes "Posted" as soon as the Save click happens. | Reconcile file says Posted. LedgerPro is the only proof. | System | Ops | Search LedgerPro for the reference. If it is not there, post it by hand. | More than one in a run. Send the time, the references and the log file. |
| E07 | The same entry appears **twice** in LedgerPro | The bot posts every statement reference that is not in `ledger_export.xlsx`. If that export is old (the export didn't save over it, or a re-run happened after a partial run), entries already posted look "missing" again. | LedgerPro; two runs' Reconcile files or log lines both say Posted | System | Ops, then Dev | Delete the duplicate in LedgerPro. Before any re-run, make sure LedgerPro's export overwrites `ledger_export.xlsx`. | Always. The export setup needs fixing. Send both run times and the references. |
| E08 | Typing appears in the wrong window, or LedgerPro gets odd values | The bot types blind into the window that has focus. It does not check that the window is LedgerPro. | Nowhere | System | Ops | Find and undo the stray typing. In LedgerPro, check the entries posted around that time. Keep hands off during runs (section 4). | It happens with nobody touching the machine |
| E09 | launcher.exe shows an error or closes, and the bot never starts | Downloading or unpacking an update failed. The launcher has no error handling for this. | Nowhere, except the error box, if any | System | Dev | Take a screenshot of the error. Don't keep retrying. | Always. Send the time, the screenshot and the machine name. |
| E10 | A fix that dev announced isn't working on this machine | The launcher could not reach the update server. It may be offline, or the machine may not trust the server's certificate. The launcher then runs the old build **without any message**. | Nowhere | System | Dev | Check the bot files' date (section 10). | The date did not change after a release. Send the machine name. |
| E11 | A statement row is missing from the Reconcile file | The same reference appears twice in the statement, so only the last row is checked. Or the reference cell is blank. | Nowhere | Business | Ops | Fix the statement (section 3 rules) and check those rows by hand | - |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| A button (Export, New Entry, Save) is not on screen yet | Looks again every second | 20 s per button | New Entry / Save: row = Error, screenshot saved, next entry. Export: the run stops (E04). |
| LedgerPro slow to start | Nothing. It waits a fixed time. | 15 s | Carries on regardless. The usual result is E04. |
| Export slow to save | Nothing. It waits a fixed time. | 10 s | Reads whatever `ledger_export.xlsx` is there, possibly an old one (E07) |
| Failed entry | **No retry** | - | Ops posts it by hand or re-runs |
| Run stopped | **No restart, no resume** | - | Ops follows E04 / E05 |
The longest the bot ever waits on one thing is 20 seconds. If nothing has happened for a minute, the run has ended. Check for the Reconcile file.
## 9. Known incidents
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-09 (fix 3b3625f) | Rows `Error` "Button save_btn.png not found on screen" on some laptops | Windows display scaling at 125%. The button pictures are matched at 100% only. | Button pictures were re-captured at 100%. **Keep scaling at 100% and 1920x1080** (section 3). Follow E03 for affected rows. | Yes |
| (fix 30cb604) | Two laptops never received an update and kept running the old build | The update server's certificate isn't trusted on those laptops. The launcher hides the error and runs the old build. | **Not fixed in code:** the error is still hidden. After every release, check the version on each machine (section 10), then E10. | Partly |
| (fix 6694f19) | The packaged bot could not find any buttons | The button pictures were left out of the exe build | Build fix. Do a test run after each new release. | No |
## 10. Releases and version check
- **How an update arrives:** launcher.exe checks the update server on every start. When the server reports a different version, the launcher downloads it and unpacks it into the bot folder.
- **How to confirm it applied:** the bot has no window, so **the version is not shown anywhere**. The only sign is the *Date modified* of `LedgerBot.exe` in the bot folder. It should match the release date dev announced in #rpa-support.
- **If the date did not change:** the update was skipped without any message (E10). Post the machine name in #rpa-support. Dev will copy the build over by hand or fix the certificate on the machine.
- After any release, do one test run with a small statement. Confirm a Reconcile file appears and that the rows are not all `Error`.
## 11. Escalating to dev
Post in Teams **#rpa-support**. Send:
- the exception ID from section 7 (for example "E05")
- the date and time of the run, and the machine name
- the `logs/ledger_<date>.log` file
- the matching `logs/notfound_*.png` screenshot, if there is one
- the Reconcile file name (attach it only if your team allows sharing it)
Never send passwords, the bot's settings file, or bank statement or ledger contents pasted into chat. The screenshots and logs show financial data, so share them only in #rpa-support.
## 12. Failure-class check
- F1 covered in section 3 (shortcut, folders, LedgerPro signed in, how to start) and E04, E09.
- F2 covered in section 4 (hands off, nothing over LedgerPro) and E08; no window check is in Fix #6.
- F3 covered in section 3 (100% / 1920x1080), E02, E03 and E04, and section 8 (20 s, 15 s and 10 s limits); fixed waits are in Fix #7.
- F4 covered in the section 3 input table and E11; fix in Fix #8.
- F5 covered in E06; fix in Fix #2.
- F6 covered in E03 and E06 (check in LedgerPro before re-posting); fix in Fix #2.
- F7 covered in section 6 ("Missing / Posted" means posted; Mismatch needs follow-up); fix in Fix #9.
- F8 not applicable: there is no automatic re-run and no watchdog. The risk from a manual re-run (double posting) is covered in E07.
- F9 covered in section 4 (corner stop or Task Manager; closing LedgerPro does not stop it).
- F10 not applicable: there are no batches, queues or locks, and each run stands alone. The one case of an item spilling into the next one (a half-filled form) is in E03.
- F11 covered in Fix #3, #4 and #5 (crash cause not recorded, summary not saved, screenshots overwritten).
- F12 covered in section 10, E09 and E10; fixes in Fix #1 and #10.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The update check swallows every error (SSL, offline, bad JSON) and runs the old build without a message. The 30cb604 "fix" only added a comment. | Machines stay on old builds without anyone knowing (E10) | `launcher.py:15-20` |
| 2 | "Posted" is written right after the Save click. Nothing confirms that LedgerPro saved the entry. | Rows show Posted when nothing was posted (E06) | `app.py:89-90`, `app.py:65-70` |
| 3 | Crashes are not logged and not caught outside the posting step: missing shortcut, Export not found, missing export file, bad amount, missing argument, missing `data/output`. Only `RuntimeError` is caught. pyautogui's own exceptions (newer `ImageNotFoundException`, `FailSafeException`) also crash the run. | No result file and no cause anywhere (E04 / E05) | `app.py:34`, `app.py:77-81`, `app.py:91`, `app.py:96`, `app.py:101` |
| 4 | The Reconcile file is saved only at the end. A stop mid-run loses every result, including what was already posted. | Ops must rebuild the result from the log (E05) | `app.py:96` |
| 5 | Screenshots are named per button only, with no time or reference, so they overwrite each other. The run summary goes to stdout only, and nothing reads it. | Evidence is lost. There is no run start/end marker in the log. | `app.py:45`, `app.py:97`, `app.py:95` |
| 6 | No check that LedgerPro has focus before typing. A Save failure leaves a half-filled form, and the next entry types into it. | Stray or merged entries (E03, E08) | `app.py:65-70`, `app.py:91-92` |
| 7 | Fixed sleeps (15 s and 10 s) instead of waiting for LedgerPro or the export. The export file's freshness is never checked, and nothing in the code saves the export to `data/ledger_export.xlsx`, although the docstring says the bot does. | Stale export leads to double posting (E07) | `app.py:76-79`, `app.py:50` |
| 8 | Duplicate references in the statement or the export silently collapse into one. Blank references are skipped. A non-numeric amount crashes the run. | Rows vanish from the result (E11) | `app.py:52`, `app.py:81` |
| 9 | Posted entries keep Result = "Missing". Ops reads "Missing" as "not done". | Misleading status | `app.py:87-90` |
| 10 | No version is visible anywhere. The comment claims a window title, but the bot has no window. The spec is a commented-out stub, so the 6694f19 asset fix can't be verified from this repo. The launcher compares against the VERSION frozen into its own exe, and a running launcher.exe can't overwrite itself if the zip contains it (likely re-download on every start, or an unpack error). The download and unpack have no error handling. | Ops can't confirm a release; possible E09 | `app.py:15`, `ledgerbot.spec:1-3`, `launcher.py:9`, `launcher.py:25-29` |
| 11 | No display-scaling or resolution check at startup. Ops must remember the 100% rule. | E02 / E03 at any other scaling | `app.py:20` |
| 12 | `.env` with LedgerPro credentials is committed to git, and no code reads it. | Credential exposure in the repo | `.env` (tracked) |
| 13 | The existing log from 2026-09-05 has a counterparty name in a Mismatch line, which the current code doesn't write. It came from an older build. | Logs may hold customer data. Purge or restrict the old logs. | `logs/ledger_20260905.log:3` |
| 14 | `requirements.txt` doesn't pin pyautogui or pyscreeze. Newer versions raise an exception instead of returning "not found", which breaks the 20 s wait and the screenshot. | E02 / E03 could become run crashes after a rebuild | `requirements.txt`, `app.py:40` |
tests/evals/results/handoff/h-packaged-desktop-full-r2.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-packaged-desktop/full",
"rep": 2,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"display scaling rule": true,
"version check": false,
"shortcut prerequisite": true,
"silent update failure flagged": true
},
"cost_usd": 1.444084,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-packaged-desktop-full-tmclh49o\\project",
"reply": "`HANDOVER.md` ready: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-packaged-desktop-full-tmclh49o\\project\\HANDOVER.md`. Code edhuvum change pannala, bot-a run pannala.\n\n| Item | Count |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 16 (E01\u2013E16) |\n| Failure classes | 11 covered, 1 not applicable (F8: auto-rerun, watchdog rendum illa) |\n| Fix before handover | 15 items |\n\n**Fix before handover-la mukkiyamaanavai:**\n- **3 \"fix\" commits-um code-la apply aagala.** Launcher innum update error-a amaidhiya maraikkudhu. Spec-la `datas` line commented out-a irukku. 125% scaling fix-um button pictures-a thirumba capture pannadhu dhaan, code change illa.\n- **Duplicate posting risk:** export 10 s-kulla save aagalaina, bot pazhaya export file-oda compare panni already irukkura entries-a thirumba post pannidum (E06).\n- **Save button kidaikkalaina:** type panna entry LedgerPro-la save aagaama open-a nikkum. Adhukku apram vara lines adhe form-la type aagalaam (E08).\n- **\"Posted\":** Save click panna udane \"Posted\" nu ezhudhidum, LedgerPro-la save aachaa nu check pannaadhu. Adhoda, bot post panna lines-ku Result column \"Missing\" nu kaattum.\n- **Version theriyaadhu:** version endha screen-layum kaattala. Run crash aana reason-um enga-yum record aagala.\n- **Repo-la sensitive files:** `.env` git-la track aagudhu (naan adha open pannala). Commit aana log file-la oru payee name irukku.\n\n**Verification gaps:**\n- `data\\output` folder repo-la illa, code-um adha create pannaadhu. Adhanaala doc-la \"Before a run\" checklist-la vechirukken.\n- History scan-la `git show` deny aachu. Adhanaala incidents commit messages + current code vechu dhaan ezhudhinen, commit diffs paakkala.\n\nNeenga munnaadiye kudutha answers use pannen: escalation #rpa-support, ops access (bot screens, output folder, logs folder, DB illa), theliva illaadha statuses-ku code reading. Adhanaala neenga reply pannaama ellaa questions-um resolve aagiduchu.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions whose meaning the code doesn't make clear mean?",
"stderr": ""
}
tests/evals/results/handoff/h-packaged-desktop-full-r2.md
+185
View File
@@ -0,0 +1,185 @@
# LedgerBot - ops handover
Version 3.2.0 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
LedgerBot reconciles the bank statement Excel against the LedgerPro desktop accounting app and posts any entries LedgerPro is missing. It runs on demand, once per statement. You give it the statement Excel file. It opens LedgerPro, exports LedgerPro's entries to Excel, and compares the two files by reference number. It posts each missing entry into LedgerPro by clicking LedgerPro's buttons and typing the reference and amount. It reports amount differences but does not fix them. The output is one Excel file per run, `data/output/Reconcile_<date>_<time>.xlsx`. It drives one app: **LedgerPro**.
## 2. How it works
```
launcher.exe (what you start, with the statement Excel)
└─ checks for an update, then starts → LedgerBot.exe ──clicks & types──> LedgerPro
│ (do not touch while it runs)
├─ reads → the export LedgerPro saves (data/ledger_export.xlsx)
├─ writes → data/output/Reconcile_<date>_<time>.xlsx (at the very end)
└─ writes → logs/ledger_<date>.log, logs/notfound_<button>.png
```
- **launcher.exe** looks for a newer build on the update server and installs it if it finds one. It then starts LedgerBot and closes. It shows nothing on screen.
- **LedgerBot.exe** is the bot. It has no screen of its own. It works by finding LedgerPro's buttons on screen from saved pictures of them, then clicking and typing like a person would.
- **LedgerPro** is the app being driven. While the bot runs, the mouse and keyboard belong to the bot.
- **The export file** is the list of entries LedgerPro already has. The bot reads it to decide what is missing.
- **The Reconcile file** is the result. There is one row per statement line.
- **The log** has one line per statement line. The **notfound picture** is a screenshot taken when a button could not be found.
**One run, step by step:**
1. The launcher checks for an update and starts the bot.
2. The bot opens LedgerPro from its Desktop shortcut and waits 15 seconds.
3. It clicks LedgerPro's Export to Excel button and waits 10 seconds.
4. It reads LedgerPro's export and your statement.
5. It goes through every statement line:
- Same reference and same amount: **OK**.
- Same reference, different amount: **Mismatch**. Nothing is posted.
- Reference not in LedgerPro: the bot clicks New entry, types the reference and amount, and clicks Save.
6. It saves the Reconcile file. This happens only after the last line.
## 3. Before a run
- [ ] The display is at **100% scaling and 1920 x 1080** (Settings > System > Display). At any other setting, for example 125% or a docking station with another monitor, the bot cannot find LedgerPro's buttons (see E03, E07 and E08).
- [ ] A shortcut named exactly **LedgerPro** is on your Desktop or the Public Desktop. Otherwise the bot stops at once (E02).
- [ ] The folders `logs` and `data\output` exist next to LedgerBot.exe. Without `logs` the bot stops before doing anything (E01). Without `data\output` the bot posts everything and then stops without saving the results (E11).
- [ ] LedgerPro opens without a login prompt, update notice or popup within 15 seconds, and its buttons are visible on screen (window maximised).
- [ ] Start the bot **with the statement file**, for example by dragging the statement Excel onto launcher.exe. If you start it without a file, it stops at once (E01).
- [ ] Statement layout: **reference in column A, amount in column B, headers in row 1, data from row 2.** The bot reads the first sheet only.
- [ ] References must be written exactly as LedgerPro has them:
- No spaces before or after the reference. Bad: `INV1001 ` with a trailing space. Good: `INV1001`.
- The same format as in LedgerPro. Bad: `1001` stored as a number when LedgerPro has `001001`. Good: `001001` stored as text.
- A reference that doesn't match exactly is treated as missing and **posted again** (E12).
- [ ] Every row that has a reference must have a numeric amount. Bad: blank, `1,234.00 CR` or `n/a`. Good: `1234.00`. One bad amount stops the whole run before anything is posted (E05).
- [ ] Each reference appears only once in the statement. When a reference appears twice, the bot keeps only the last row and silently ignores the other one (E12).
- [ ] The launcher installed the current release (see section 10).
## 4. During a run
- Hands off the machine. Don't use the mouse or keyboard and don't open other windows. The bot types into **whatever window is in front**. It never checks that LedgerPro is the active window (E13).
- Don't lock the screen or let it sleep. The bot needs to see LedgerPro's buttons.
- Don't close or minimise LedgerPro. Closing LedgerPro **does not stop the bot**, which keeps looking for buttons and typing.
- **To stop the bot:**
- Move the mouse into the **top-left corner of the screen**. The bot stops at its next click or keystroke.
- Or end **LedgerBot.exe** in Task Manager (Details tab).
- A stopped run saves no Reconcile file. The log still lists the lines finished before the stop. Afterwards, check LedgerPro for a half-typed, unsaved entry and cancel it.
- **A run looks stuck** when the mouse hasn't moved for more than a minute. The bot never waits longer than 20 seconds for one button and has no watchdog. If nothing moves for a minute, it has stopped. Check Task Manager for LedgerBot.exe and look for a new Reconcile file.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `data\output\`: there is a new `Reconcile_<date>_<time>.xlsx` with the run's time. No new file means the run stopped early (E01-E05, E11). |
| Which lines failed? | The Reconcile file, filter the **Result** column for `Error` or `Mismatch` |
| Which lines did the bot post? | The Reconcile file: Result `Missing` and Note `Posted` |
| Why did one line fail? | The **Note** column. Then `logs\ledger_<date>.log`: search for the reference. Each line reads `<time> INFO <reference> <Result> <Note>`. Search for " Error ", not "ERROR". |
| What was on screen when a button wasn't found? | `logs\notfound_<button>.png`, for example `notfound_save_btn.png`. Only the **latest** failure of each button is kept. |
| What happened if the run stopped with no Reconcile file? | The log lists the lines finished before the stop. The reason the run stopped is **not recorded anywhere** (section 13). |
| Did the bot already retry? | It doesn't retry lines. It only keeps looking for each button for 20 seconds. Nothing in the log shows the search. |
| Which version is running? | No screen shows the version. Check the **Date modified** of LedgerBot.exe (section 10). |
## 6. Statuses
The Reconcile file has no colours. The statuses are values in the **Result** column, with details in **Note**.
| Result | Note | Meaning | What ops does |
|---|---|---|---|
| OK | (empty) | LedgerPro already has this reference with the same amount (difference of 0.01 or less) | nothing |
| Missing | Posted | **The bot posted this entry.** "Missing" means it *was* missing before this run. The bot clicked Save but never checked that LedgerPro saved it. | Spot-check a few in LedgerPro (E09) |
| Mismatch | LedgerPro has <amount> | The reference exists, but LedgerPro has a different amount. **Nothing was posted or changed.** Needs follow-up. | Review it in LedgerPro and correct it by hand (E10) |
| Error | Button new_entry.png not found on screen | The bot couldn't open a new entry. Nothing was typed or posted. | E07 |
| Error | Button save_btn.png not found on screen | The bot typed the entry but couldn't click Save. **The entry is left unsaved and open in LedgerPro.** | E08 |
| (no file) | - | The run stopped before the end | E01-E05, E11 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | The bot closes within a second. LedgerPro never opens. | It doesn't. It stops when there's no `logs` folder, or no statement file was given. | Nowhere | System | Ops | Check that the `logs` folder exists next to LedgerBot.exe. Start again by dragging the statement onto launcher.exe. | Both are correct and it still closes. Send the time and the folder listing. |
| E02 | LedgerPro never opens and the bot closes. A brief message says "LedgerPro shortcut not found on the Desktop". | It looks for a shortcut named LedgerPro on your Desktop, then on the Public Desktop | On screen only, for a moment | System | Ops | Put a shortcut named exactly `LedgerPro` on the Desktop and run again | The shortcut is there and it still fails |
| E03 | LedgerPro opens but no export happens. The bot closes and `logs\notfound_export_excel.png` appears. | It looks for the Export to Excel button for 20 seconds (after a 15-second wait for LedgerPro) | The notfound picture only. No Reconcile file. | System | Ops | Open the picture: (1) a login prompt or popup covers LedgerPro → clear it; (2) LedgerPro still loading → run again; (3) scaling is not 100% → fix the display. Run again. | The picture shows the button clearly at 100% and 1920 x 1080. Send the picture and the time. |
| E04 | LedgerPro exports, then the bot closes. No Reconcile file. | It doesn't. It stops when the export isn't saved at the place the bot reads it, or LedgerPro took longer than 10 seconds to save. | Nowhere | System | Ops, then Dev | Run again once, with nothing else open on the machine | It happens again. Send the time and say whether LedgerPro showed a save dialog. |
| E05 | The bot closes after the export and before posting anything. No Reconcile file and no new log lines. | It doesn't. It stops when the statement can't be read: an amount that isn't a number, or a reference with a blank amount. | Nowhere | Business | Ops | Fix the statement (section 3: numeric amounts, reference in A, amount in B, headers in row 1) and run again. Nothing was posted. | The statement is clean and it still stops |
| E06 | Entries appear **twice** in LedgerPro. Lines you know were already in LedgerPro show Missing / Posted. | It doesn't. If LedgerPro's export didn't finish within 10 seconds, the bot may compare against an **older export** and post entries again. | The Reconcile file: Missing / Posted on lines that shouldn't be there | System | Ops, then Dev | Don't run again yet. In LedgerPro, find the duplicate entries and remove them by your normal correction process. | Always: report it so dev can fix the cause. Send the Reconcile file, the log and the time. |
| E07 | Error - "Button new_entry.png not found on screen" | It looks for the New entry button for 20 seconds | The Reconcile file (Result, Note), the log, `logs\notfound_new_entry.png` | System | Ops | Check the picture: (1) popup → clear; (2) scaling → fix. Then run again with the same statement. Lines already posted come back OK; this one gets posted. | Same line fails on two runs in a row |
| E08 | Error - "Button save_btn.png not found on screen". **Often every line after it is Error too.** | It looks for the Save button for 20 seconds after typing the reference and amount | The Reconcile file, the log, `logs\notfound_save_btn.png` | System | **Ops** | **Check LedgerPro first.** An unsaved entry with that reference is probably still open. Cancel it. Make sure no entry with a garbled reference or amount was saved. The bot may have typed later lines into the open form. Then run again. | A wrong entry was saved, or it happens again at 100% scaling. Send the picture, the Reconcile file and the time. |
| E09 | Missing / Posted, but the entry isn't in LedgerPro | It doesn't. The bot treats clicking Save as success and never checks that LedgerPro saved it (for example, when LedgerPro showed a validation popup). | The Reconcile file says Posted, wrongly | System | Ops | Run again with the same statement. A fresh export will show the entry as missing and the bot posts it again. If LedgerPro shows a popup for that entry, enter it by hand. | It's not posted after two runs |
| E10 | Mismatch - "LedgerPro has <amount>" | The reference exists in both, but the amounts differ by more than 0.01 | The Reconcile file, the log | Business | Ops | Check the entry in LedgerPro against the bank statement and correct it by hand. The bot never changes amounts. | Never, unless the amounts actually match. Then it's a reading problem: send the line. |
| E11 | The bot posted entries in LedgerPro but closed at the end. No Reconcile file. | It doesn't. It stops when the `data\output` folder is missing at save time. | The log only (one line per statement line) | System | Ops | Create the `output` folder inside `data`. Use the log to see what was posted. Running again with the same statement is safe: posted entries come back OK. | The folder exists and it still happens |
| E12 | A statement line is missing from the Reconcile file, or a line already in LedgerPro is posted again | It doesn't. Rows with an empty reference are skipped. For a repeated reference, only the last row is kept. A reference written differently from LedgerPro (spaces, number vs text, leading zeros) counts as missing. | Nowhere. Compare row counts. | Business | Ops | Compare the statement row count with the Reconcile row count. Fix the references (section 3) and remove any duplicates posted in LedgerPro. | The references are exactly as in LedgerPro and it still happens |
| E13 | Text typed into another window, or clicks landing in another app | It doesn't. The bot types into whichever window is in front. | Nowhere | System | Ops | Stop the bot (section 4). Undo anything typed into the other app. Check LedgerPro for half-typed entries. Run again hands-off. | It happens with nobody touching the machine |
| E14 | An old build keeps running after a release was announced | It doesn't. The launcher ignores **any** update error (no network, untrusted certificate) and starts the old build without saying so. | Nowhere | System | Dev | Check the LedgerBot.exe date (section 10). If it's older than the release, report it. | Always. Send the machine name and the LedgerBot.exe date. |
| E15 | After a release, launcher.exe closes and the bot never starts | It doesn't. The update download or unzip failed. | Nowhere | System | Dev | Report it. Don't run the bot from this machine. | Always. Send the machine name and the time. |
| E16 | Right after a new build, **every** line is Error "not found", or E03 on every run, at 100% scaling | It doesn't. The new build is missing the button pictures. | The Reconcile file, notfound pictures | System | Dev | Stop using this machine for the bot until dev ships a fixed build | Always. Send one notfound picture and the LedgerBot.exe date. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| A button isn't on screen yet | Looks again every second | 20 seconds per button | Export button: the run stops (E03). New entry / Save: the line is marked Error and the bot moves to the next line (E07, E08). |
| LedgerPro is slow to open | Waits | A fixed 15 seconds, no check | Export button not found (E03) |
| The export is slow to save | Waits | A fixed 10 seconds, no check | The run stops (E04), or the bot reads an old export (E06) |
| Anything else | **Nothing.** There's no line retry, no LedgerPro restart, no automatic re-run, no watchdog and no resume. | - | Ops runs again. Running the same statement again is the recovery. Lines posted earlier come back OK, **provided LedgerPro's export is fresh** (E06). |
## 9. Known incidents
The history has no dates, so the version is the only marker. All three fixes are in the 3.2.0 code history.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 3.2.0 (log of 2026-09-05) | Lines Error "Button save_btn.png not found on screen". Missing entries not posted. | The laptops were at 125% display scaling. The button pictures only match at 100%. | **Still a rule:** keep 100% scaling and 1920 x 1080 on every bot machine (section 3). Handle as E08. | Yes |
| Before 3.2.0 | Two laptops stayed on an old build, with no error | The update server's certificate isn't trusted, and the launcher hides the error | **Still possible:** the code still hides update errors. Check the version after every release (section 10, E14). | Partly. Ops can spot it, but can't see why it failed. |
| Before 3.2.0 | A build that failed every click | The button pictures weren't packed into the exe | Escalate (E16) | No |
## 10. Releases and version check
- **How an update arrives:** every time launcher.exe starts, it checks the update server. If the version differs, it downloads and installs it before starting the bot. You see nothing either way.
- **How to confirm it applied:** no screen shows the version number. In the bot folder, right-click LedgerBot.exe → Properties → **Date modified**. An update sets this to the day it was installed. If the date is older than the release announcement in #rpa-support, the update didn't apply. A leftover `update.zip` in the same folder shows when the last download happened.
- **If the version did not change:** report it in #rpa-support (E14) with the machine name and the date. Don't keep running an old build if the release fixed something you rely on.
## 11. Escalating to dev
Post in **#rpa-support** (Teams). Send:
- the exception ID from section 7 and the date and time of the run;
- the machine name and the LedgerBot.exe Date modified;
- the files, **attached** rather than pasted:
- `logs\ledger_<date>.log`
- any `logs\notfound_<button>.png`
- the `data\output\Reconcile_<date>_<time>.xlsx` from that run
- a screenshot of LedgerPro if it shows a message.
Never send:
- passwords or the `.env` file;
- statement rows, references, amounts or payee names typed into the chat message.
## 12. Failure-class check
- **F1** (missing prerequisite): section 3 (shortcut, `logs` and `data\output` folders, display) and E01, E02, E11, E16. Silent crashes are item 5 in section 13.
- **F2** (wrong window or focus): section 4 hands-off rule and E13. The missing focus check is item 12 in section 13.
- **F3** (target slower or different): E03, E04, E07, E08, the section 8 waits, and the 100% / 1920 x 1080 rule in section 3. Fixed sleeps are item 6 and hard-coded scaling is item 13 in section 13.
- **F4** (input read differently): section 3 input rules and E05, E12. Parsing gaps are item 10 in section 13.
- **F5** (done when nothing was done): E09, because Posted is written without confirmation. That is item 7 in section 13.
- **F6** (action happened but not confirmed): E08 (unsaved form left open) and E06 (stale export reposts). These are items 6 and 8 in section 13.
- **F7** (business exception shown as success): Mismatch is shown as needing follow-up in section 6. Result "Missing" really means posted, which is item 9 in section 13. There are no colours.
- **F8** (endless re-runs or watchdog): not applicable. There is no auto-rerun and no watchdog, and every wait is capped at 20 seconds. Section 4 gives the "stuck" sign.
- **F9** (stopping the wrong way): section 4. Closing LedgerPro doesn't stop the bot. Use the mouse corner or Task Manager.
- **F10** (one stuck item blocks the queue): there are no batches or locks. The nearest case is one failed Save that cascades into the following lines (E08).
- **F11** (cause hidden from ops): section 5 points only at the output and log folders. Hidden causes are items 4, 5, 10 and 11 in section 13.
- **F12** (update didn't arrive, or old build undid new): section 10, E14 and E15. The missing version display is item 3 and silent update failure is item 1 in section 13. Nothing shared is rewritten on start, so mixed versions don't corrupt each other.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The launcher catches every exception and returns None, so an SSL or network failure is silent. The "fix" commit f3a0b30 only added a comment. | Machines stay on old builds without anyone knowing (E14) | `launcher.py:16-20`, `launcher.py:35` |
| 2 | The update uses `!=` for the version (it will downgrade), has no hash or signature check, has no rollback, and doesn't catch download or unzip errors | A bad or old zip breaks or downgrades the machine (E15) | `launcher.py:25-29` |
| 3 | The version isn't shown anywhere. The comment claims a window title "LedgerBot v3.2.0", but no window exists. | Ops can't confirm a release applied | `app.py:15` |
| 4 | The spec has `datas` commented out. The fix in commit 9f03d5a isn't in effect in this tree, and there's no spec for launcher.exe. | The next build may ship without button pictures (E16) | `ledgerbot.spec:2` |
| 5 | `main` isn't wrapped in a try/except, so run-level crashes are never logged and the console closes | E01-E05 and E11 leave no reason anywhere | `app.py:100-101` |
| 6 | Fixed sleeps of 15 s and 10 s. The bot never checks that the export file is new, and never deletes the old one before exporting. | A slow export makes the bot reconcile against old data and **post duplicates** (E06) | `app.py:76-79` |
| 7 | "Posted" is written as soon as Save is clicked, with no check in LedgerPro | Entries LedgerPro didn't save show as Posted (E09) | `app.py:70`, `app.py:89-90` |
| 8 | When Save isn't found, the typed entry is left open. There's no cancel or escape, and later lines type into the open form. | Error cascade and possibly garbled saved entries (E08) | `app.py:65-70`, `app.py:91-92` |
| 9 | A posted line is written as Result "Missing". The README says "Missing" is a status of its own. | Ops reads "Missing" as "not done" | `app.py:58`, `app.py:87-90`, `README.md:6` |
| 10 | Statement parsing: `str()` without trimming (`1001.0` vs `1001`), a duplicate reference overwrites silently, blank or text amounts crash the run, and blank references are skipped | Duplicate postings, silently lost lines, whole-run crashes (E05, E12) | `app.py:52`, `app.py:81` |
| 11 | The Reconcile file is written only at the end, and `data/output` and `logs` aren't created if missing | A crash loses every result even though entries were posted (E01, E11) | `app.py:24`, `app.py:96` |
| 12 | There's no check that LedgerPro is the active window before typing | Keystrokes go into other apps (E13) | `app.py:66-69` |
| 13 | The button pictures only match at 100% / 1920 x 1080. The 125% "fix" was recapturing the pictures, not a code change. | Any display change breaks every run | `app.py:20`, `app.py:102` |
| 14 | The notfound screenshot is named by button only and overwritten each time. Line errors are logged at INFO. The run summary is printed to a console nobody sees. | Ops can't match a screenshot to a line, and searching for "ERROR" finds nothing | `app.py:45`, `app.py:95`, `app.py:97` |
| 15 | `.env` is tracked in git although no code reads it. `logs/ledger_20260905.log` is committed and its last line holds a payee name (written by an older build). | Possible secrets and customer data in the repo and in shipped zips | `.env`, `logs/ledger_20260905.log:3` |
tests/evals/results/handoff/h-packaged-desktop-full-r3.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-packaged-desktop/full",
"rep": 3,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"display scaling rule": true,
"version check": true,
"shortcut prerequisite": true,
"silent update failure flagged": true
},
"cost_usd": 1.4724382,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-packaged-desktop-full-xzlont2s\\project",
"reply": "`HANDOVER.md` ready, repo root-la irukku: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-packaged-desktop-full-xzlont2s\\project\\HANDOVER.md`. Code change edhuvum pannala, bot-um run pannala.\n\n| Item | Result |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 10 (E01\u2013E10) |\n| Failure classes | 10 covered, 2 not applicable (F8: re-run loop or watchdog illa; F10: queue or lock illa) |\n| Fix before handover | 15 items |\n| Checks | ella status-um doc-la irukku; file:line section 13-la mattum; PHI, client names, IPs, `.env` values edhuvum illa |\n\nOps-ku main points:\n- **Run nadula stop aanaa result Excel varaadhu.** Andha nerathula post aana entries log-la mattum dhaan irukkum (E07).\n- **Correctly post aana line-kum Result \"Missing\"-nu dhaan varum.** Note column-la \"Posted\" irukkum. Save click-ku apram LedgerPro-la save aachaa-nu bot check pannaadhu (E05).\n- **Pazhaya LedgerPro export file-a bot silent-a padikkalaam.** Appo re-run panna same entries rendu dhadava post aagum (E08).\n- **Version screen-la enga-yum theriyaadhu.**\n\nFix before handover-la top items:\n- Moonu \"fix\" commits-la oru comment line mattum dhaan add aagirukku. Launcher innum ella update error-aiyum maraikkudhu, spec file muzhusum comment, `assets/` images 16-byte placeholders.\n- Run nadula stop aanaa result file illa.\n- Save confirmation illa.\n- Pazhaya export file check illa.\n- Version check `!=` use pannudhu, adhanaala server-la pazhaya build irundhaa ella machine-um downgrade aagum.\n- Version display illa.\n- `.env` git-la commit aagirukku.\n- 2026-09-05 log-la oru party name irukku, adhu current code ezhudhuradhu illa. So deploy aana build indha source-la irundhu vandhadhu illa-nu thonudhu.\n\nNeenga munnaadiye sonna answers-a use pannen: escalation #rpa-support-ku, ops-ku screens, output folder, logs folder access mattum (database illa), theliva illaadha statuses-ku code-a vachu enga best reading. Indha bot database-ae use pannala, adhanaala DB point-ae doc-la varala.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions mean whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-packaged-desktop-full-r3.md
+172
View File
@@ -0,0 +1,172 @@
# LedgerBot - ops handover
Version 3.2.0 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
LedgerBot checks the day's bank statement against the **LedgerPro** accounting app and adds any entry that is missing from LedgerPro. You give it one bank-statement Excel file. It opens LedgerPro, exports LedgerPro's entries to Excel, compares every statement line with them by reference number and amount, and types each missing entry into LedgerPro (New Entry → reference → amount → Save). At the end it writes one result file, `data/output/Reconcile_<date>_<time>.xlsx`, with one row per statement line. It drives one desktop app: **LedgerPro**. It does this by looking for pictures of LedgerPro's buttons on the screen and clicking them, so the screen must look exactly the way it did when those pictures were taken.
## 2. How it works
```
launcher.exe (what you start)
└─ checks for an update, then starts → LedgerBot.exe ──clicks & types──> LedgerPro
│ (do not touch while it runs)
├─ reads → your bank statement Excel
├─ reads → data/ledger_export.xlsx (LedgerPro's export)
├─ writes → data/output/Reconcile_<date>_<time>.xlsx
└─ writes → logs/ledger_<date>.log
logs/notfound_<button>.png (only on a problem)
```
- **launcher.exe** asks the update server whether a newer LedgerBot exists, installs it if so, then starts LedgerBot.exe. If it cannot reach the server it starts the current build without telling you.
- **LedgerBot.exe** does the actual work. It has no window of its own with buttons; it only moves the mouse and types.
- **LedgerPro** is the accounting app the bot drives. Hands off it while the bot runs.
- **Your bank statement Excel** is the input (see section 3 for its layout).
- **data/ledger_export.xlsx** is LedgerPro's own export; the bot reads it as "what LedgerPro already has".
- **data/output/Reconcile_….xlsx** is the result file: columns Reference, Amount, Result, Note.
- **logs/ledger_<date>.log** gets one line per statement line (reference, result, note). One log file per day; several runs on the same day go into the same file.
- **logs/notfound_<button>.png** is a screenshot taken when the bot could not find a button.
**One run, step by step:**
1. Finds the LedgerPro shortcut on the Desktop and starts LedgerPro.
2. Waits 15 seconds for LedgerPro to open.
3. Clicks LedgerPro's **Export to Excel** button, then waits 10 seconds.
4. Reads `data/ledger_export.xlsx` (LedgerPro's entries) and your statement Excel.
5. For each statement line: OK if LedgerPro has the same amount, Mismatch if the amount differs, Missing if LedgerPro does not have the reference at all.
6. For each Missing line: clicks **New Entry**, types the reference, Tab, types the amount, clicks **Save**.
7. After the last line, saves the result file. **The result file only exists if the bot reaches this step.**
## 3. Before a run
- [ ] Start the bot by **dragging the bank statement Excel onto `launcher.exe`** (or a shortcut that passes the statement file). Started without a file, the bot stops at once with no result file.
- [ ] LedgerPro's **Export to Excel** saves to `data\ledger_export.xlsx` next to LedgerBot.exe. Check the file's modified time changes on each run (E08).
- [ ] **Display scaling is 100%** (Settings → System → Display → Scale), screen resolution 1920 × 1080. At 125% the bot misses buttons (section 9).
- [ ] The **LedgerPro shortcut** is named exactly `LedgerPro` and sits on your Desktop or on the Public Desktop. Otherwise the run stops at once (E01).
- [ ] The folders `logs` and `data\output` exist next to LedgerBot.exe. If `logs` is missing the bot does not start; if `data\output` is missing the bot does all the posting and then **cannot save the result file** (E07).
- [ ] Bank statement Excel layout - the bot reads only the **first sheet**:
- Row 1 is a header; data starts at row 2.
- Column A = reference, column B = amount. Other columns are ignored.
- Amount must be a real number cell. Good: `1234.50`. Bad: `1,234.50 USD`, `$1234.50` typed as text, or an empty amount cell. One bad amount stops the whole run before anything is compared (E06).
- A row with an empty reference is skipped silently.
- **Each reference only once.** If the same reference appears twice, the bot keeps only the last one and the other line is not in the result file at all.
- The reference must be spelled exactly as it will appear in LedgerPro (`REF-1001` and `REF 1001` are different references).
- [ ] Amounts are compared to the cent: a difference of more than 0.01 is a Mismatch.
- [ ] Note the version: see section 10.
## 4. During a run
- **Hands off the mouse and keyboard.** The bot types into whatever window has focus. If you click elsewhere, the reference and amount get typed into that window.
- Keep LedgerPro **maximised and visible**, with nothing on top of it. No other window may show something that looks like LedgerPro's buttons (for example a screenshot or a second LedgerPro).
- Do not lock the screen and do not let the screen saver start - the bot can only click what is visible.
- **To stop the bot:** close the LedgerBot console window (or end `LedgerBot.exe` in Task Manager). Moving the mouse into the top-left corner of the screen also stops it. Closing LedgerPro does **not** stop it - the bot keeps looking for buttons and then stops with an error. **Whichever way you stop it, no result file is written**, so treat the run as in section 7, E07.
- How long it waits: about 25 seconds to open LedgerPro and export, then up to **20 seconds per button**. If nothing moves on screen for more than a minute, the bot has stopped or is stuck - check whether the LedgerBot window is still open.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | A new `Reconcile_<date>_<time>.xlsx` in `data\output` with the time of your run. No new file = the run did **not** finish (E07). |
| Which lines failed? | Result file → filter column **Result** = `Error` (and review `Mismatch`). |
| Why did one line fail? | Result file → column **Note**. Then `logs\ledger_<date>.log`, search for the reference. |
| What did the screen look like? | `logs\notfound_<button>.png` (only the latest one per button - older ones are overwritten). |
| Why did the whole run stop? | The LedgerBot console window, **if it is still open** - the reason is shown there only. It is not in the log file (see section 13). |
| Did the bot already retry? | It never retries. Every Error is final for that run. |
| Which version is running? | Not shown anywhere today (see section 10 and section 13). |
## 6. Statuses
The result file has no colours. These are all the values the bot writes in **Result** and **Note**:
| Result | Note | Meaning | What ops does |
|---|---|---|---|
| OK | (empty) | LedgerPro already has this reference with the same amount. | nothing |
| **Missing** | **Posted** | The reference was not in LedgerPro. The bot typed it in and clicked Save. **"Missing" here means "was missing, bot added it"** - it is the normal outcome of a post. The bot does **not** check that LedgerPro accepted the save. | Spot-check a few in LedgerPro. If one is not there, see E05. |
| **Mismatch** | `LedgerPro has <amount>` | LedgerPro has this reference but with a different amount. **The bot changes nothing.** | Needs follow-up: compare with the statement and correct LedgerPro by hand. |
| **Error** | `Button new_entry.png not found on screen` | The bot could not find LedgerPro's New Entry button. Nothing was typed for this line. | E03 |
| **Error** | `Button save_btn.png not found on screen` | The bot typed reference and amount but could not find Save. The entry is **not saved**, and the half-filled entry form may still be open. | E04 |
A value you will never see but which the code mentions: Note `Not in LedgerPro - posting` is always replaced by `Posted` or by an error text before the file is saved.
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Bot stops at once, LedgerPro never opens; console shows "LedgerPro shortcut not found on the Desktop" | It looks for a shortcut named `LedgerPro` on your Desktop and the Public Desktop | Console window only. No result file. | System | Ops | Put the LedgerPro shortcut on the Desktop with that exact name. Run again. | The shortcut is there and it still fails |
| E02 | LedgerPro opens but nothing is clicked; bot stops after ~20 s; console shows "Button export_excel.png not found on screen" | It searches the screen for 20 s for a picture of the Export to Excel button | Console window; `logs\notfound_export_excel.png`. No result file. | System | Ops, then Dev | Check display scaling is 100%, LedgerPro is maximised and fully loaded (on a slow day it can take longer than 15 s to open). Run again. | Fails twice with scaling at 100% - send the screenshot |
| E03 | Result = Error, Note `Button new_entry.png not found on screen` | Searches 20 s for the New Entry button | Result file; log line; `logs\notfound_new_entry.png` | System | Ops | Check LedgerPro: nothing was entered for this reference. Look at the screenshot: a popup or dialog in LedgerPro usually explains it - close it. Run again with the same statement; lines already posted now show OK. | More than one line per run, or it repeats on the re-run |
| E04 | Result = Error, Note `Button save_btn.png not found on screen` | Clicked New Entry and typed, then searched 20 s for Save | Result file; log line; `logs\notfound_save_btn.png` | System | Ops | In LedgerPro, check the entry for this reference: it is usually **not** saved and a half-filled entry form may still be open - cancel it. The **next** line may also have been typed into that open form, so check the next reference too. Then run again. | It happens on every line, or scaling is already 100% (this was the 125% scaling incident) |
| E05 | Result = Missing / Posted, but the entry is not in LedgerPro (or has wrong text in it) | The bot only clicks Save; it does not check for a "saved" message | Nothing - the row looks normal | System | Ops | Add the entry in LedgerPro by hand, or run again (the line then comes out Missing again and is re-posted). **Check LedgerPro before re-running** so you do not post twice. | It happens more than once a week |
| E06 | Bot stops right after the export; console shows an error about converting a value to a number (for example "could not convert string to float") | It turns every amount in column B (statement and LedgerPro export) into a number; a text or empty amount stops the run | Console window only. No result file. Nothing was posted yet. | Business | Ops | Fix the amount column in the statement (real numbers, no currency signs, no empty amounts). Run again. If the statement is fine, the bad value is in LedgerPro's export - escalate. | The statement is clean and it still stops |
| E07 | **No new result file** after the run (bot window closed, or stopped half-way) | Any stop in the middle (button not found before posting, mouse in the corner, window closed, `data\output` missing, crash) | Only the log lines for lines already handled. The console message is lost once the window closes. | System | Ops, then Dev | Lines in today's log **were** handled - Missing lines there were posted. Do **not** run again blindly: first check that `data\output` exists and that LedgerPro is closed, then run again. Posted lines will now show OK. | Twice in a row, or the console message is not one of E01-E06 |
| E08 | Result file shows Missing for references that are already in LedgerPro, or OK for ones that are not | The bot reads `data\ledger_export.xlsx`. If today's export did not save there, it reads an **older** export without noticing | Nothing - looks like a normal run | System | Ops, then Dev | Before a run, check the date of `data\ledger_export.xlsx` changes when the bot exports. If it did not, stop the bot and escalate - re-running would post duplicates. | Always - the export path needs a dev fix (section 13) |
| E09 | A statement line is missing from the result file | Duplicate reference in the statement: the bot keeps only the last one; or an empty reference cell: skipped | Nothing | Business | Ops | Make every reference unique in the statement; fill empty references. | Never - input fix |
| E10 | Version did not change after an update, or the bot starts but behaves like an older build | The launcher gives up silently if the update server is unreachable or its certificate is not trusted | Nothing | System | Dev | Report the machine name and what version you expected. | Always |
**What to send for every "Dev" escalation** (in #rpa-support): the exception ID (E02…), the date and time of the run, the statement file name (not its content), `logs\ledger_<date>.log`, any `logs\notfound_*.png` - **check the screenshot shows no client names or amounts first, or crop it** - and a photo or copy of the console message if the window is still open.
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Button not on screen yet | Looks again every second | 20 s per button | Export button: run stops (E02). New Entry / Save: line = Error, next line (E03/E04) |
| LedgerPro slow to open | Fixed wait, no check | 15 s, then the 20 s button search | E02 |
| Export slow to save | Fixed wait, no check | 10 s | Reads whatever export file is there (E08) |
| Anything else | **Nothing.** No retries, no restarts, no resume. | - | Run stops, no result file (E07) |
There is no watchdog and no automatic re-run. A run either finishes with a result file or stops; you never need to wait for the bot to "try again".
## 9. Known incidents
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-10-23 (fix recorded), 3.2.0 | Bot started but could not find any button in the exe build | Button pictures were not packed into the exe | Fix recorded in git; see section 13 item 3. If **every** button fails on a fresh install, escalate (E02). | No |
| 2026-10-23 (fix recorded) | Save clicked in the wrong place / missed on some laptops | Laptops at 125% display scaling; button pictures taken at 100% | Keep display scaling at **100%** (section 3) | Yes |
| 2026-10-23 (fix recorded) | Two laptops never got the new version | Launcher silently ignored the update server's untrusted (self-signed) certificate and kept starting the old build | Compare the version after every release (section 10); escalate E10 | Partly |
| 2026-09-05 (log) | One line Error "Button save_btn.png not found on screen" | Matches the 125% scaling incident | E04 | Yes |
## 10. Releases and version check
- How an update arrives: start the bot through **launcher.exe**, never LedgerBot.exe directly. The launcher downloads and unpacks the new build before starting the bot.
- How to confirm it applied: **today you cannot see the version on screen** - the bot has no window title or About box. Until the dev team adds one (section 13), ask in #rpa-support for the expected release date and compare the date of `LedgerBot.exe` in the program folder.
- If the version did not change: the launcher failed silently (no internet, or the update server's certificate is not trusted on this machine). Escalate E10 with the machine name. Do not copy files between laptops yourself.
## 11. Escalating to dev
Send in Teams **#rpa-support**: exception ID from section 7, date and time of the run, machine name, `logs\ledger_<date>.log`, the `logs\notfound_*.png` screenshot (cropped of client names and amounts), and the console message if still visible.
Never send: passwords, the `.env` settings file, the bank statement itself, or client names and amounts in chat.
## 12. Failure-class check
- F1 Missing prerequisite - covered in section 3 (shortcut, folders, scaling) and E01, E07; section 13 items 3, 7.
- F2 Wrong window / focus - covered in section 4 (hands off, LedgerPro maximised, nothing on top) and E04 (typing into the wrong form).
- F3 Target app slower or different - covered in E02-E04, section 8 (15 s / 10 s fixed waits, 20 s per button), section 3 (100%, 1920 × 1080); section 13 item 5.
- F4 Input read differently - covered in section 3 input rules and E06, E09.
- F5 Marked done when nothing was done - covered in E08 (stale export read as current); section 13 item 4.
- F6 Action happened but not confirmed - covered in E05 and E04; section 13 item 2.
- F7 Business exception shown as success - covered in section 6 (Missing = posted; Mismatch needs follow-up); section 13 item 6.
- F8 Endless re-runs / watchdog - not applicable: the bot has no re-run loop and no watchdog; every wait has a fixed limit (section 8).
- F9 Stopping the wrong way - covered in section 4 (close the console window; closing LedgerPro does not stop it) and E07.
- F10 One stuck item blocks the queue - not applicable: there is no batch, queue or lock; each run is independent and an Error line does not block anything.
- F11 Cause hidden from ops - covered in section 5 (console-only messages) and section 13 items 1, 8.
- F12 Update did not arrive - covered in section 10 and E10; section 13 items 9, 10.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | A run that stops writes **no result file**: the workbook is saved only after the last line, and nothing catches errors outside `post_entry`. Lines already posted are only in the log. | Ops cannot tell which entries were posted; risk of double posting on re-run | `app.py:86-96` |
| 2 | No confirmation after Save: `post_entry` returns as soon as Save is clicked; the note "Posted" is written without checking LedgerPro accepted it. After a Save-not-found error the half-filled form stays open and the next line types into it. | Silent missing or corrupted entries (E04, E05) | `app.py:65-70`, `app.py:88-92` |
| 3 | The "fix" commits only added comments. `ledgerbot.spec` is entirely commented out (the `datas` line is a comment), and the files in `assets/` are 16-byte placeholders, not real button pictures - the 100%-scaling recapture is not in the repo. | A fresh build may again have no button images | `ledgerbot.spec:2-3`, `assets/*.png`, commits `2ade1d3`, `158175d` |
| 4 | Stale export: bot clicks Export and reads `data/ledger_export.xlsx` after a fixed 10 s, with no check that the file is new (and nothing tells LedgerPro to save to that path). | An old export makes the bot re-post entries already in LedgerPro (E08) | `app.py:77-79` |
| 5 | Fixed waits (15 s start, 10 s export) instead of waiting for a screen. Also: newer pyautogui raises an exception instead of returning nothing when an image is not found, which would skip the 20 s retry loop entirely; `requirements.txt` does not pin versions. | Run stops on a slow day; behaviour depends on which library version was packed | `app.py:39-41`, `app.py:76-78`, `requirements.txt` |
| 6 | Result "Missing" is kept for lines that were successfully posted (Note "Posted"); summary counts them as Missing. | Ops reads a successful post as a problem | `app.py:86-93` |
| 7 | `logs/` and `data/output/` are not created by the code. Missing `logs` stops the bot on start; missing `data/output` fails **after** all posting. | Posted entries with no result file | `app.py:24`, `app.py:96` |
| 8 | Run-stopping errors and the summary counts go to the console only (`print`), not to the log file. | Cause lost when the window closes | `app.py:97`, `app.py:101` |
| 9 | Launcher still swallows every update error (`except Exception: return None`) - the SSL-incident "fix" is only a comment. A download or unzip failure is **not** caught and stops the launcher, so the bot never starts. Version check is `!=`, so an older version on the server downgrades every machine. | Machines silently stay on, or roll back to, old builds | `launcher.py:15-20`, `launcher.py:25-29`, commit `0085285` |
| 10 | Version is not shown anywhere (the comment says "window title" but there is no window); launcher compares against its own built-in copy of the version, so after an update it may download again on every start. | Ops cannot confirm an update applied | `app.py:15`, `launcher.py:9` |
| 11 | `notfound_<button>.png` is overwritten each time and is a full-screen screenshot that can contain client names and amounts. | Evidence lost; screenshots need cropping before sharing | `app.py:45` |
| 12 | `.env` is committed to git (the code does not read it). | Possible secrets in the repo history | `.env`, commit `cd9e04d` |
| 13 | Duplicate references in the statement collapse silently (dict keeps the last). | A real bank line disappears from the result | `app.py:81` |
| 14 | The 2026-09-05 log has a Mismatch line with a party name in brackets that the current code does not write. Either the deployed build differs from this source, or names reach the log. | Ops may be running code nobody can read; client names in a file ops share | `logs/ledger_20260905.log:3` vs `app.py:60` |
| 15 | No popup handling: any LedgerPro dialog (login prompt, Save As on export, warning) blocks every button search. | Run stops with "not found" and no hint that a dialog was the cause | `app.py:37-46`, `app.py:77` |
tests/evals/results/handoff/h-packaged-desktop-full-r4.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-packaged-desktop/full",
"rep": 4,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"display scaling rule": true,
"version check": true,
"shortcut prerequisite": true,
"silent update failure flagged": true
},
"cost_usd": 1.6434040000000003,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-packaged-desktop-full-x2g5j69o\\project",
"reply": "`HANDOVER.md` ready, repo root-la irukku: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-packaged-desktop-full-x2g5j69o\\project\\HANDOVER.md`. Bot-ah run pannala, code-um maathala, commit-um pannala.\n\n| Item | Result |\n|---|---|\n| Sections | 13 (template order apdiye) |\n| Exceptions catalogued | 12 (E01\u2013E12) |\n| Failure classes | 10 covered. F8 (auto-rerun / watchdog illa) and F10 (queue / lock illa) not applicable |\n| Fix before handover | 17 items |\n| Doc-la PHI / creds / IPs | Illa. `.env` padikkala. Log-la oru customer name irundhuchu, adha doc-la podala |\n\nNeenga sonna answers use pannen: escalation = #rpa-support, ops-ku DB access illa. Bot-ku DB-ae illa, adhunaala \"Where to look\" table output folder, logs folder, console window mattum point pannudhu.\n\n**Fix before handover-la mukkiyamaana items:**\n- Moonu \"fix:\" commits-um oru comment line mattum add pannirukku, code edhuvum maarala. SSL error swallow, 125% scaling, spec stub moonum innum appadiye irukku. Bot ippo run aagradhu 100% scaling rule-ah nambi dhaan.\n- \"Posted\" note Save click aana udane ezhudhudhu, LedgerPro-la save aachaannu check pannala. Posted row-um Result = \"Missing\" nu dhaan kaattudhu.\n- Crash aanaa result Excel kedaikkadhu (kadaisila dhaan save aagudhu). Error console window-la mattum dhaan theriyum.\n- Export file pazhasaa irundhaa check illa. Neraiya duplicate post aagalaam.\n- Version endha screen-layum theriyadhu. Code comment sollra window title exist-ae aagala.\n- `logs` and `data\\output` folders repo-la illa, code-um create pannala.\n\nVerification: status strings, exception IDs, leak search ellam doc-ku ethiraa check pannen. Paths ellam repo-la irukku, illa code create pannum. Exception: `data\\` and `data\\output\\` repo-la illa, adhunaala doc-la adhu oru pre-run check-aa irukku.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel) | Ops access: which of the bot's places ops can open - the app's screens, the output folder, the log folder, the database | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear",
"stderr": ""
}
tests/evals/results/handoff/h-packaged-desktop-full-r4.md
+172
View File
@@ -0,0 +1,172 @@
# LedgerBot - ops handover
Version 3.2.0 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
LedgerBot reconciles the daily bank statement Excel with **LedgerPro** (the desktop accounting app). It exports LedgerPro's entries to Excel, compares them with the statement line by line (by reference and amount), and **posts every statement line that LedgerPro doesn't have yet** by clicking LedgerPro's buttons and typing the entry. Lines where the amounts differ are flagged, never posted. It runs once per statement (normally daily). Input: the bank statement Excel. Output: one result Excel per run in the `data\output` folder, plus a log in the `logs` folder. It drives one app: **LedgerPro**.
## 2. How it works
```
launcher.exe (what you start)
└─ checks for an update, then starts → LedgerBot.exe (black console window, no screens)
│
├──drives──> LedgerPro (do not touch while it runs)
│ └─ export → data\ledger_export.xlsx
├─ reads → the bank statement Excel you gave it
├─ writes → data\output\Reconcile_<date>_<time>.xlsx (at the very end)
└─ writes → logs\ledger_<date>.log and logs\notfound_<button>.png
```
- **launcher.exe** looks for a newer build on the update server, installs it if there is one, starts LedgerBot and then closes. It does not watch the bot.
- **LedgerBot.exe** is the bot itself. It has no windows of its own, only a black console window. Error details appear **only in that console window**.
- **LedgerPro** is opened by the bot from its Desktop shortcut. The bot finds LedgerPro's buttons by **matching pictures of them on the screen**, then types into whatever window is in front.
- **The result Excel** has one row per statement line: Reference, Amount, Result, Note. It is written **only when the run finishes**.
- **The log** has one line per statement line, the same as the result Excel. A run that crashes leaves only the lines up to the crash.
**One run, step by step:**
1. Opens LedgerPro from the Desktop shortcut and waits 15 seconds.
2. Clicks LedgerPro's "Export to Excel" button and waits 10 seconds for `data\ledger_export.xlsx`.
3. Reads that export and your bank statement (column A = reference, column B = amount, from row 2).
4. For each statement line: same reference and same amount (within 0.01) → OK. Amount differs → Mismatch, not posted. Reference not in LedgerPro → the bot clicks "New entry", types the reference, Tab, the amount, and clicks "Save".
5. Saves the result Excel in `data\output` and ends.
## 3. Before a run
- [ ] **Windows display scaling is 100% and the screen resolution is 1920 x 1080.** At any other setting the bot can't find LedgerPro's buttons (E03, E04).
- [ ] The **LedgerPro shortcut** (named exactly `LedgerPro`) is on your Desktop or the Public Desktop (E01).
- [ ] **LedgerPro opens straight to its main screen**: already signed in, no login box, no "what's new" or licence popup. The bot can't sign in or close popups.
- [ ] LedgerPro's Excel export saves to the bot's `data` folder as `ledger_export.xlsx`. **Delete yesterday's `ledger_export.xlsx` before the run**, so an export that fails can't be mistaken for today's (E06).
- [ ] The bot folder has a `logs` folder and a `data\output` folder. Without `logs` the bot won't start; without `data\output` it posts everything and then loses the result file (E09).
- [ ] Start the bot **with the statement file**: drag the bank statement Excel onto `launcher.exe`. Starting it with no file closes it at once (E08).
- [ ] Bank statement rules:
- Column A = reference, column B = amount, row 1 = headers, data from row 2.
- **Every amount is a number.** A blank or text amount stops the whole run (E08). Bad: `1,250.00 CR` or empty. Good: `1250.00`.
- **Each reference appears once.** If a reference appears twice, the bot keeps only the last row and silently ignores the other one. Bad: two rows `INV-1001`. Good: combine or give them distinct references.
- **Store references as text, the same way LedgerPro shows them.** A reference typed as a plain number (for example `5512`) may be read as `5512.0`, not match LedgerPro, and get **posted twice**. Bad: `5512` as a number. Good: `5512` formatted as Text.
- [ ] Check the version: see section 10. Today there is **no place on screen that shows it**.
## 4. During a run
- **Hands off the machine.** Don't touch the mouse or keyboard, don't open other windows, don't lock the screen; turn off the screensaver and sleep. The bot types into whatever window is in front: if you click another window, the reference and amount get typed there.
- Keep LedgerPro maximised and in front. Don't move or resize it.
- **To stop the bot:** close the black LedgerBot console window. (Moving the mouse hard into a screen corner also stops it.) Closing LedgerPro does **not** stop it: the bot keeps looking for buttons and marks the remaining lines Error.
- After stopping it early: **no result Excel is written.** Check in LedgerPro which entries were already posted before you re-run (see E07).
- A run looks stuck when nothing happens on screen for more than about a minute. The bot never gives up by itself on the whole run: each button gets 20 seconds, then it moves on. There is no watchdog. A run with many Missing lines where LedgerPro has frozen takes about 40 seconds per line; stop it.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `data\output` → is there a `Reconcile_<date>_<time>.xlsx` from this run? **No file = the run crashed** (or was stopped). |
| Which lines failed? | The result Excel → filter **Result** = `Error` or `Mismatch` |
| Why did one line fail? | The result Excel → **Note** column. Then `logs\ledger_<date>.log` → search for the reference. |
| Why did the whole run stop? | The black console window, if still open: the last lines say why. The log only shows the last line processed before the crash. |
| What did the screen look like when a button wasn't found? | `logs\notfound_<button>.png` (only the most recent one per button; each failure overwrites the last) |
| Did the bot already retry? | It keeps looking for each button for 20 seconds; it never re-tries a line or a run. |
| Which version is running? | Nowhere on screen today. Ask dev on #rpa-support (see section 10). |
The log file is named by the date the run **started**. A run that crosses midnight keeps writing to the previous day's log.
## 6. Statuses
These are the values in the **Result** and **Note** columns of the result Excel (and the log). The Excel has no colours.
| Result | Note | Meaning | What ops does |
|---|---|---|---|
| OK | (empty) | LedgerPro already has this reference with the same amount | nothing |
| Missing | Posted | **The bot typed the entry and clicked Save. It did not check that LedgerPro saved it.** Result stays "Missing" even though it was posted. | Spot-check a few in LedgerPro. Any LedgerPro error popup during the run means check them all (E07). |
| Mismatch | LedgerPro has <amount> | Reference exists in LedgerPro with a different amount. **Not posted.** | Needs follow-up: compare the statement with LedgerPro and correct by hand (E02) |
| Error | Button new_entry.png not found on screen | Couldn't find LedgerPro's "New entry" button. **Not posted.** | E03 |
| Error | Button save_btn.png not found on screen | The entry was typed but "Save" wasn't found. **Probably not saved; a half-filled entry may be left open.** | E04 |
| (no result file) | - | The run crashed or was stopped | E01, E05, E06, E08, E09, E10, E11 |
The bot also writes a file called `BOT_WAS_RUN` in its folder on every start. It means nothing to ops; ignore it.
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | LedgerPro never opens; console shows "LedgerPro shortcut not found on the Desktop"; no result file | It looks for a shortcut called `LedgerPro` on your Desktop and the Public Desktop; if neither has it, it stops | Console window only | System | Ops | Put the LedgerPro shortcut on the Desktop with exactly that name, re-run | The shortcut is there and it still fails: send the console text and time |
| E02 | Result `Mismatch`, Note "LedgerPro has <amount>" | Reference found in LedgerPro but the amounts differ by more than 0.01 | Result Excel, log | Business | Ops | Compare the statement line with the LedgerPro entry, correct whichever is wrong by hand. The bot never fixes these. | Never (business decision) |
| E03 | Result `Error`, Note "Button new_entry.png not found on screen" | It looks for the "New entry" button picture for 20 s; if it never matches, it saves a screenshot and moves to the next line | Result Excel, log, `logs\notfound_new_entry.png` | System | Ops, then Dev | Check display scaling 100% and 1920 x 1080; open the screenshot (popup over LedgerPro? wrong window?). Close any popup, re-run. Lines already posted show as OK on the re-run. | **Every** Missing line is Error (likely a bad build, see the first incident in section 9), or it fails again with correct settings: send the log, the screenshot, run time |
| E04 | Result `Error`, Note "Button save_btn.png not found on screen" | Same picture search for "Save", after the reference and amount were already typed | Result Excel, log, `logs\notfound_save_btn.png` | System | Ops | **Check LedgerPro first.** A half-filled or saved entry may be there. Cancel or finish it by hand, then re-run. The following lines may also fail, because the half-filled form is still open. | Fails again with 100% scaling: send the log, screenshot, run time |
| E05 | No result file; console shows "Button export_excel.png not found on screen" | Picture search for the Export button fails after 20 s; the bot stops | Console; `logs\notfound_export_excel.png` | System | Ops, then Dev | Check scaling, that LedgerPro opened to its main screen (no login/popup), re-run | Fails twice with correct settings: send the screenshot, time |
| E06 | No result file, the console mentions `ledger_export.xlsx` not found; **or** lines that should be OK get posted again | The bot waits a fixed 10 s for the export and doesn't check it. If the export is slow it finds no file (stops) or **yesterday's** file (reconciles against old data) | Console only; or duplicates in LedgerPro | System | Ops | Delete the old `ledger_export.xlsx` before each run (section 3). If duplicates were posted, remove them in LedgerPro by hand. | It happens on most runs (the wait is too short for this laptop): send the time, log |
| E07 | An entry marked "Posted" is not in LedgerPro, or LedgerPro showed an error popup during the run | Nothing: the bot writes "Posted" as soon as it clicks Save | Result Excel shows "Posted" anyway | System | **Ops** | Check the line in LedgerPro. If not saved, post it by hand, or re-run (a fresh run posts what's still missing). **Never re-run before checking** if the bot was stopped mid-run: lines it already posted might be posted again if the export is stale. | LedgerPro rejects the bot's entries regularly: send the reference, time, log |
| E08 | Console flashes and closes at once, or shows an error about a number / "could not convert"; no result file | The bot stops on a statement amount that isn't a number, or when started without a statement file | Console only | Business | Ops | Start it by dragging the statement onto `launcher.exe`. Fix blank or text amounts in column B (section 3), re-run | Statement is clean and it still stops: send the statement's layout (not the data), time |
| E09 | Run seemed to finish but no result file in `data\output`; entries **are** in LedgerPro | The `data\output` folder is missing, so the final save fails **after** all posting | Console only; log has every line | System | Ops | Create the `data\output` folder. **Don't re-run blind**: use the log lines (reference + Posted) to see what was done; re-run only after deleting the old export, so posted lines show OK | Folder exists and it still happens: send console text, log |
| E10 | Nothing starts at all; or bot started from `launcher.exe` shows an error and closes | `logs` folder missing (bot and launcher fail on start), or the update download/unzip failed | Console only | System | Ops, then Dev | Create the `logs` folder in the bot folder, re-run | `logs` exists and it still fails: send console text, time. Possibly a broken update (section 10) |
| E11 | Bot stops in the middle with a "fail-safe" message in the console | The mouse was moved into a screen corner (this is also the manual stop) | Console only; no result file | System | Ops | Hands off the mouse (section 4). Check LedgerPro for lines already posted, delete the old export, re-run | Never |
| E12 | One laptop behaves differently from the others after a release (old behaviour, old errors) | The launcher hides any update failure (network or certificate) and runs the old build | Nowhere | System | Dev | Report it | Always: send laptop name, date of the release, time you started the bot |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| A LedgerPro button isn't on screen yet | Looks for it again every second | 20 s per button | "New entry" / "Save": line = Error, next line. "Export": the run stops |
| LedgerPro slow to open | Nothing: fixed 15 s wait | 15 s | Export button not found (E05) |
| Export slow | Nothing: fixed 10 s wait | 10 s | Missing or stale export (E06) |
| Update server unreachable | Runs the current build silently | 10 s | Old version keeps running (E12) |
Nothing else is automatic: no re-try of a line, no restart of LedgerPro, no resume. A re-run always starts from the top; lines posted earlier show as OK **only if the new export is fresh**.
## 9. Known incidents
No dates were recorded; newest first by commit order.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 3.2.0 (latest fix) | Probably every posting failed with "not found" in the packaged exe | The button pictures weren't packed into the exe build | Treat "every Missing line is Error" as a bad build: escalate (E03). The fix is not visible in the repo. | No |
| 3.2.0 | Lines Error "Button save_btn.png not found on screen", entries not posted (seen on 2026-09-05) | Laptops at 125% display scaling; buttons were captured at 100% | **Keep 100% scaling and 1920 x 1080** on every bot laptop (section 3). Not fixed in code: it's a rule. | Yes |
| 3.2.0 | Two laptops never got the update, with no error | Update server's certificate not trusted; the launcher hides the error | Report version mismatches (E12). Not fixed in code. | Partly |
## 10. Releases and version check
- How an update arrives: `launcher.exe` checks the update server on every start, and if the server's version differs, downloads it and unzips it over the bot folder, then starts the bot.
- How to confirm it applied: **there is no on-screen version today** (the bot has no window or title bar). Ask dev on #rpa-support to confirm the build after each release; until dev adds a version display, compare the date of `LedgerBot.exe` in the bot folder with the other laptops.
- If the version did not change: post in #rpa-support with the laptop name. Don't copy files from another laptop by hand unless dev tells you to.
- Every bot laptop should run the same version.
## 11. Escalating to dev
Post in **#rpa-support** (Teams). Send: the exception ID from section 7, the time the run started and failed, the laptop name, the log file `logs\ledger_<date>.log`, any `logs\notfound_<button>.png`, a photo or copy of the black console window's last lines, and the result Excel name if there is one.
Never send: passwords, the `.env` file, or bank statement or customer data in chat. Refer to lines by reference only if dev asks.
## 12. Failure-class check
- F1 covered in section 3 (shortcut, `logs` and `data\output` folders, statement file) and E01, E08, E09, E10; packaged-exe-only bug in section 9 (exe-assets incident) and Fix #5.
- F2 covered in section 4 (hands off, LedgerPro in front) and E11; no window check is Fix #3.
- F3 covered in E03, E04, E05, E06, section 3 (100% scaling, 1920 x 1080) and section 8; fixed waits are Fix #4.
- F4 covered in section 3 input rules (numeric amounts, unique references, references as text) and E08; Fix #8.
- F5 covered in E09 (posted but no result file); missing positive check is Fix #1.
- F6 covered in E04 and E07 (check LedgerPro before re-running); Fix #1.
- F7 covered in section 6: "Missing - Posted" stays "Missing"; Mismatch needs follow-up; Fix #2.
- F8 not applicable: the bot has no auto-rerun loop and no watchdog; a frozen LedgerPro just makes each line fail after 20-40 s (section 4).
- F9 covered in section 4 (close the console window; closing LedgerPro doesn't stop it) and E11.
- F10 not applicable: there are no batches, queues or locks; each run is independent. The nearest risk (stale export causing duplicates) is E06.
- F11 covered in section 5; causes that only reach the console are Fix #6, overwritten screenshots Fix #7.
- F12 covered in section 10 and E12; Fix #5 and #9.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | "Posted" is written right after clicking Save; nothing checks LedgerPro saved it | Unsaved entries look done; a crash mid-loop leaves no record of what was posted | `app.py:65-70`, `app.py:89-90` |
| 2 | Posted lines keep Result = "Missing"; the summary count "Missing" mixes posted and not posted | Ops can't tell done from not done at a glance | `app.py:86-93` |
| 3 | No check that LedgerPro is the active window before typing; no popup/login handling; failed Save leaves the form open and cascades into following lines | Keystrokes land in other windows; one failure spoils the rest of the run | `app.py:65-70`, `app.py:75-77` |
| 4 | Fixed 15 s / 10 s waits; export file's age is never checked | Slow laptop → crash, or reconcile against yesterday's export and post duplicates | `app.py:76-79` |
| 5 | The three "fix:" commits only added a comment; no code changed. `ledgerbot.spec` is three comment lines (assets `datas` commented out), so the real build spec isn't in the repo; no `assets/` change for the 100% recapture | The update and exe-assets incidents (section 9) can recur; builds aren't reproducible | `launcher.py:19-20,35`, `ledgerbot.spec:1-3`, `app.py:102` |
| 6 | Crashes (missing shortcut, Export not found, bad amount, missing folders, no argument) only go to the console; nothing logs errors or tracebacks; result Excel saved only at the end | After a crash ops has no file to look at; all row results lost | `app.py:24,34,79-81,96,101` |
| 7 | `notfound_<button>.png` has a fixed name, overwritten each time, not tied to a reference or time | Screenshot evidence is lost | `app.py:45` |
| 8 | Statement and export read into dicts: duplicate references silently dropped; numeric references become "5512.0" and may not match; non-numeric amount crashes the run | Lines skipped or posted twice without any message | `app.py:52`, `app.py:81` |
| 9 | Launcher swallows every update error (incl. self-signed certificate), installs on any version difference (can downgrade), no checksum, unzips over the live folder, doesn't log; launcher imports `app`, so a missing `logs` folder also breaks the launcher | Silent stale builds; a bad zip breaks the install | `launcher.py:9,15-29` |
| 10 | No version shown anywhere (the comment mentions a window title that doesn't exist) | Ops can't confirm an update applied | `app.py:15` |
| 11 | `logs` and `data\output` folders must pre-exist; neither is in the repo nor created by the code | Bot won't start, or loses the result file after posting | `app.py:24`, `app.py:96` |
| 12 | Exit code is 0 even when lines errored; summary printed to stdout only, and the launcher doesn't wait for or capture it | No way to alert on a failed run | `app.py:97`, `launcher.py:30` |
| 13 | Output name only to the minute | Two runs in the same minute overwrite each other | `app.py:96` |
| 14 | `.env` and `logs\` are committed to git (the log holds a customer name; `.env` is read by no code) | Data and secrets in repo history | `.env`, `logs/ledger_20260905.log` |
| 15 | Committed log line doesn't match the current code's Mismatch note, so it came from a different build | History in the repo can mislead | `logs/ledger_20260905.log:3` |
| 16 | pyautogui/pyscreeze versions are unpinned; newer versions raise an exception on "not found" instead of returning nothing, which would skip the 20 s retry and crash the run on the first miss (not tested) | Run could die on any slow screen | `requirements.txt`, `app.py:40` |
| 17 | Mouse-corner fail-safe stop isn't caught; no clean stop that saves partial results | Every stop loses the result file | `app.py:86-96` |
tests/evals/results/handoff/h-packaged-desktop-full-r5.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-packaged-desktop/full",
"rep": 5,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"display scaling rule": true,
"version check": false,
"shortcut prerequisite": true,
"silent update failure flagged": true
},
"cost_usd": 1.5265358,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-packaged-desktop-full-8nf_6t9w\\project",
"reply": "`HANDOVER.md` ready, repo root la iruku. Bot-a run pannala, code-um maathala; output idhu oru file mattum dhaan.\n\n| Check | Result |\n|---|---|\n| Sections | 13 (template order maathama) |\n| Exceptions | 17 (E01\u2013E17). Every Dev row-um enna send pannanum nu sollum. |\n| Failure classes | 12/12 map aagiruchu. 10 full-ah covered. F8 partly not applicable (auto-rerun, watchdog rendum illa). F10 queue-ah not applicable (batch illa; adhoda blocking case E06-la cover aagudhu). |\n| Statuses | 6 Result/Note combos + \"no Reconcile file\" case. Bot write panra ellaa string-um irukku. |\n| PHI / secrets | Log-la irundha transaction ID-um payee name-um copy pannala. `.env` open pannave illa. |\n| Fix before handover | 18 items |\n\n**Ops-ku mukkiyamaana gaps:**\n- \"Posted\" na bot Save click pannuchu nu mattum dhaan artham, save aachaa nu check pannala. Successful post aana row-um Result \"Missing\"-nu dhaan kaattum.\n- Save button kidaikkalana, half-filled form LedgerPro-la open-ave irukkum; next rows adhu mela type aagalaam.\n- Crash aana Reconcile file illa; record log-la mattum dhaan irukkum. Run-level errors (shortcut illa, Export fail, bad amount) enga-um log aagadhu.\n- Bot version enga-um theriyaadhu. Launcher update error-um silent-ah hide pannidum.\n- Build spec full-ah comment-out aagi iruku. Ennoda reading: \"assets\" fix repo-la illa.\n- Code-la illadha payee name ondru 2026-09-05 log-la iruku. Adhanaala running exe indha repo-voda match aagaama irukkalaam.\n\n**Path check:** `data\\output` folder repo-la illa, code-um create pannala. Adhanaala doc \"Before a run\"-la ops-a create panna sollirukku.\n\n**Assumptions:** Ops-ku bot-a illama LedgerPro access irukkaa nu theriyala, adhanaala E06, E07, E11 \"LedgerPro check pannunga, illana #rpa-support\". Bot-ku own screen illa, so \"Where to look\" output folder-um logs folder-um mattum point pannum.\n\nQUESTIONS: Who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Which of the bot's places can ops open: the app's screens, the output folder, the log folder, the database? | Are there any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-packaged-desktop-full-r5.md
+200
View File
@@ -0,0 +1,200 @@
# LedgerBot - ops handover
Version 3.2.0 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
LedgerBot compares the daily bank statement (an Excel file) with the entries in **LedgerPro**, the desktop accounting app. You start it with one statement file. First it opens LedgerPro and clicks Export to get LedgerPro's current entries. Then it checks every statement line. If a line is in LedgerPro with the same amount (within 1 cent), it is OK. If the amount differs, it is a Mismatch, which the bot only reports and never corrects. If the line is not in LedgerPro, the bot **types it into LedgerPro and clicks Save**. At the end it writes one result Excel to the output folder.
The bot has **no window of its own**. You only ever see LedgerPro being driven, the mouse moving by itself, and the result file when the run ends.
## 2. How it works
```
launcher.exe (what you start, with the statement file)
├─ checks the update server, installs a new build if there is one
└─ starts → LedgerBot.exe (no window) ──drives──> LedgerPro desktop app
│ (do not touch while it runs)
├─ reads → statement Excel you gave it
├─ reads → data\ledger_export.xlsx (LedgerPro's export)
├─ writes → logs\ledger_<date>.log (+ logs\notfound_<button>.png)
└─ writes → data\output\Reconcile_<date>_<time>.xlsx (at the very end)
```
- **launcher.exe** checks for updates, then starts the bot and closes. It writes nothing anywhere.
- **LedgerBot.exe** is the bot itself. It works LedgerPro by finding the buttons as pictures on the screen and clicking them.
- **LedgerPro** is the target app. The bot opens it from the LedgerPro shortcut on the Desktop.
- **ledger_export.xlsx** is LedgerPro's list of current entries, produced when the bot clicks Export.
- **The log** gets one line per statement line, plus a screenshot whenever a button could not be found.
- **The Reconcile file** is the result: one row per statement line, with the columns Reference, Amount, Result and Note.
**One run, step by step:**
1. The launcher checks for an update, then starts the bot with your statement file.
2. The bot opens LedgerPro and waits a fixed 15 seconds.
3. It clicks LedgerPro's **Export to Excel** button and waits a fixed 10 seconds.
4. It reads the export and your statement.
5. It goes through the statement line by line. A missing line is posted like this: click **New entry**, type the reference, press Tab, type the amount, click **Save**.
6. It writes the Reconcile file. **Nothing is saved to the Reconcile file until this last step.**
## 3. Before a run
- [ ] The **LedgerPro shortcut** (named `LedgerPro`) is on your Desktop or on the Public Desktop. If it is missing, the bot stops at once and leaves no message (E01).
- [ ] **Display scaling is 100% and the resolution is 1920 x 1080** (Windows Settings → System → Display). At any other setting the bot cannot find LedgerPro's buttons (E02, E05, E06). This also applies to a new laptop, an external monitor or a remote-desktop session.
- [ ] The **logs** folder and the **data\output** folder exist next to LedgerBot.exe. Without logs the bot does not start. Without data\output the bot does all the posting and then loses the result file (E12, E13).
- [ ] LedgerPro is closed, or at least has no half-filled entry form open.
- [ ] No Reconcile file from a run in the same minute is open in Excel.
- [ ] Statement file rules:
- Use the **first sheet**, with row 1 as the header. Column A is the reference and column B is the amount. Data starts on row 2.
- The **amount must be a real number**.
- Good: `1250.50`
- Bad: `1,250.50 USD`, `n/a`, or an empty amount cell next to a reference. Any of these stops the whole run with no message (E10).
- **Each reference only once.**
- Good: `TX-1001` on one row only.
- Bad: `TX-1001` on two rows. The bot keeps only the last one and says nothing about the first (E09).
- **Write each reference exactly as LedgerPro shows it.**
- Good: `TX-1001`.
- Bad: `TX 1001` or `1001`. These count as "not in LedgerPro", so the bot posts them a second time.
- [ ] To start the bot, **drag the statement file onto launcher.exe**, or use a shortcut that passes the file. If you start the launcher with no file, the bot stops at once and leaves no message (E14).
- [ ] You cannot check the version today, because the bot shows it nowhere (section 10).
## 4. During a run
- **Hands off the machine.** Don't touch the mouse or keyboard, and don't open other windows. The bot types into whichever window is in front. If another window comes forward, the reference and amount go into that window (E17).
- Don't click in LedgerPro and don't close it. Close any LedgerPro popups only after the run ends.
- **To stop the bot:** the bot has no Stop button and no window to close. Closing LedgerPro does **not** stop it. Use one of these:
1. Move the mouse quickly into the **top-left corner** of the screen. The bot stops at its next click or keystroke.
2. Or open Task Manager and end **LedgerBot.exe**.
Either way, **no Reconcile file is written** for that run. Use the log to see what was already posted (E11).
- **How long it takes:** an OK or Mismatch line takes no time. A posted line takes up to about 40 seconds. When a button can't be found, the bot gives up on that line after 20 seconds.
- **A run looks stuck** when the log gets no new line for more than 2 minutes after the first line. The bot has no watchdog and never restarts by itself. Stop it as described above, then post in #rpa-support.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | Output folder `data\output`: there is a new `Reconcile_<date>_<time>.xlsx` with the time the run ended. **No new file means the run did not finish.** |
| Which lines failed or need a look? | The Reconcile file. Filter the **Result** column for `Error` and `Mismatch`, and also check `Missing` rows (section 6). |
| Why did one line fail? | The **Note** column of the Reconcile file. Then `logs\ledger_<date>.log`: search for the reference. |
| A button "not found" error | `logs\notfound_<button>.png` is a screenshot of the screen at the moment the button wasn't found. It only shows the **latest** failure for that button, because each new one overwrites it. |
| The run died with no Reconcile file | `logs\ledger_<date>.log`. Each line written there was already handled, including posted lines. If the log has no line for today, the bot stopped before it reached the first line (E01, E02, E03, E10, E13, E14). |
| Did the bot already retry? | It does not retry lines or runs. Its only retry is looking for a button again for 20 seconds (section 8). |
| Which version is running? | You can't see it today. Ask in #rpa-support (section 10). |
The log file holds every run of the same day, one after another. Use the time at the start of each line to tell runs apart.
## 6. Statuses
The Reconcile file has no colours. It is a plain Excel file. The **Result** and **Note** columns together give the status.
| Result | Note | Meaning | What ops does |
|---|---|---|---|
| `OK` | (empty) | The line is in LedgerPro with the same amount, within 1 cent. | Nothing. |
| `Missing` | `Posted` | **The line was not in LedgerPro, so the bot typed it in and clicked Save.** "Missing" here means "was missing, now posted". The bot does **not** check that the save actually worked. | **Needs follow-up.** Spot-check these entries in LedgerPro (E07). |
| `Missing` | `Not in LedgerPro - posting` | Posting started but the run crashed. You should never see this in a saved file. | If you see it, post in #rpa-support. |
| `Mismatch` | `LedgerPro has <amount>` | The reference is in LedgerPro, but with a different amount. **The bot changed nothing.** | **Needs follow-up.** Investigate and correct by hand (E08). |
| `Error` | `Button new_entry.png not found on screen` | The bot could not open a new entry form. Nothing was typed. | E05. |
| `Error` | `Button save_btn.png not found on screen` | **The reference and amount were typed, but Save was never clicked.** The form may still be open in LedgerPro. | E06. Act before re-running. |
| (no Reconcile file) | - | The run stopped before the end. | Section 5, last rows. |
The log uses the same words on each line: `<reference> <Result> <Note>`.
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | LedgerPro never opens. There is no Reconcile file and no log line for the run. | It looks for the `LedgerPro` shortcut on your Desktop, then the Public Desktop. If neither has it, it stops. | **Nowhere.** The message goes to no file. | System | Ops | Put the LedgerPro shortcut, named exactly `LedgerPro`, on the Desktop. Re-run. | The shortcut is there and it still happens. |
| E02 | LedgerPro opens but nothing is clicked. There is no Reconcile file and no log line, but there is a `logs\notfound_export_excel.png`. | It looks for the Export button picture for 20 seconds, then gives up and stops the run. | Screenshot `logs\notfound_export_excel.png` only. | System | Ops, then Dev | Check that scaling is 100% and the resolution is 1920 x 1080, and that LedgerPro opened fully, maximised, with nothing on top of it. Re-run. | It happens again with the right display settings. Send the screenshot. |
| E03 | The run stops right after Export. There is no Reconcile file and no log line. | It **does not check**. It waits a fixed 10 seconds after Export and reads the export file. If the file isn't there yet, is open in Excel, or is in a different place, the run crashes. | **Nowhere.** | System | Ops, then Dev | Close any open copy of the export in Excel and re-run once. LedgerPro may be slow, for example on first start. | It fails twice. Say the time and that no log line was written. |
| E04 | Lines that were already in LedgerPro show `Missing / Posted`, and LedgerPro now has the same entry twice. | It **does not check**. If Export did not produce a new file in time, the bot reads the previous export. | Reconcile file (rows look normal). | System | Dev | Remove the duplicate entries in LedgerPro if you can. Post in #rpa-support with the Reconcile file name. | Always. |
| E05 | `Error - Button new_entry.png not found on screen` | It looks for the New entry button picture for 20 seconds, then marks the line Error and moves on. Nothing was typed. | Reconcile file, Note column. Log line. `logs\notfound_new_entry.png`. | System | Ops | Check the display settings and whether a LedgerPro popup was covering the screen. Re-run: a fresh run checks the line again and posts it. | Several lines in a row, or it happens again on the re-run. |
| E06 | `Error - Button save_btn.png not found on screen` | It typed the reference and amount, looked for Save for 20 seconds, then marked the line Error and moved on. **The half-filled form stays open in LedgerPro.** The next lines' clicks and typing may land on that form. | Reconcile file, Note column. Log line. `logs\notfound_save_btn.png`. | System | **Ops** | **Before re-running**, look in LedgerPro (if you have access; if not, ask in #rpa-support). Cancel any half-filled entry. Then check that every line *after* the Error that says `Posted` really exists and is correct. Re-run only after that. | You can't check LedgerPro, or entries after the Error look wrong. |
| E07 | `Missing - Posted`, but the entry is not in LedgerPro or is wrong. | It **does not check**. "Posted" only means Save was clicked. A LedgerPro popup or validation message is not noticed. | Reconcile file (looks successful). | System | Ops, then Dev | Spot-check posted lines in LedgerPro after each run. Missing entries are posted again on the next run. | It happens more than once. Send the reference, the time and the Reconcile file name. |
| E08 | `Mismatch - LedgerPro has <amount>` | The reference is in LedgerPro, but the amounts differ by more than 1 cent. | Reconcile file. Log line. | Business | Ops | Find out which amount is right and correct LedgerPro or the statement by hand. The bot never fixes it. | Never, unless the amounts are actually equal. |
| E09 | A statement line has no row in the Reconcile file. | It keeps one row per reference. A repeated reference keeps only its last row, silently. | **Nowhere.** | Business | Ops | Make each reference appear once. Handle the dropped line by hand. | - |
| E10 | The run stops with no Reconcile file and no log line for the run. | An amount in the statement or in the export is not a number, or is empty, so the run crashes while reading. | **Nowhere.** | Business | Ops, then Dev | Check column B of the statement: numbers only, nothing blank next to a reference. Re-run. | The statement is clean and it still happens (the export may be the cause). |
| E11 | The run stopped halfway (crash, mouse in the corner, Task Manager). There is no Reconcile file, but the log has lines for today. | Results are only saved at the end, so everything already done is in the log only. | Log only. | System | Ops | Lines in the log marked `Missing Posted` were already typed into LedgerPro. A re-run is safe **only because** it exports LedgerPro again first: those lines will then show OK. Re-run once and compare. | The re-run shows the same lines as Posted again (duplicates). |
| E12 | The bot posted everything, but no Reconcile file appeared. | The output folder `data\output` is missing, so the save at the end fails. | Log only (all lines are there). | System | Ops | Create the `data\output` folder next to LedgerBot.exe. **Don't re-run blind.** The postings are done; use the log as the record. | The folder exists and it still happens. |
| E13 | Nothing happens at all when you start the bot. | The `logs` folder is missing, so the bot can't even start. | **Nowhere.** | System | Ops | Create the `logs` folder next to LedgerBot.exe. Re-run. | The folder exists. |
| E14 | Nothing happens. LedgerPro does not open. | The launcher was started without a statement file. | **Nowhere.** | Business | Ops | Drag the statement file onto launcher.exe. | - |
| E15 | A new release was announced, but the bot behaves as before. | It **does not report it.** If the update server can't be reached, including when the laptop doesn't trust the server's certificate, the launcher quietly runs the old build. | **Nowhere.** | System | Dev | Post in #rpa-support with the machine name. | Always, after every release, until the version can be seen. |
| E16 | After a release, nothing happens when you start the bot. | The update download or unzip failed. The launcher crashes and never starts the bot. | **Nowhere.** | System | Dev | Post in #rpa-support with the machine name and the time. | Always. |
| E17 | References or amounts typed into another window, such as an email or chat. | It **does not check** which window is in front. It types into whatever has focus. | **Nowhere.** | System | Ops | Stop the bot (section 4). Delete the stray text. Check in LedgerPro which lines are really posted. Re-run hands-off. | It happens with nobody touching the machine. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| A LedgerPro button is not on screen yet | Looks again every second | 20 seconds per button | Export button: the run stops (E02). New entry or Save: that line is marked Error and the bot moves to the next line (E05, E06). |
| LedgerPro slow to open | Nothing. It waits a fixed 15 s. | 15 s | Usually E02. |
| Export slow | Nothing. It waits a fixed 10 s. | 10 s | E03 or E04. |
| Update server unreachable | Runs the build already installed | 10 s | Old version keeps running, with no message (E15). |
| Anything else | **No retries, restarts, watchdog or resume.** | - | Run stops. See E11. |
**So there is never a reason to wait for the bot to fix itself.** If no new log line appears for 2 minutes, it is not going to recover.
## 9. Known incidents
Newest first. The exact dates and versions are not recorded. The display-scaling incident matches a log from 2026-09-05.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 3.2.0 (latest fix) | The packaged bot could not find any button. | The button pictures were left out of the build. | Fixed by dev, but the build file in the repo still doesn't include them (section 13). If **every** run fails at Export right after a release, it is this. Post in #rpa-support. | No |
| Before 3.2.0, about 2026-09-05 | `Error - Button save_btn.png not found on screen` on some laptops. | The laptops ran at 125% display scaling. The button pictures only match at 100%. | Keep 100% scaling and 1920 x 1080 (section 3). After any Save error, clear the half-filled form first (E06). | Yes |
| Before 3.2.0 | Two laptops stayed on an old version after a release, with no error. | The launcher hid a certificate error from the self-signed update server and ran the old build. | It still hides it. After each release, ask dev to confirm every machine updated (E15). | Partly |
## 10. Releases and version check
- **How an update arrives:** every time launcher.exe starts, it asks the update server for the latest version. If that differs from its own, it downloads the new build and unpacks it over the bot's folder. Then it starts the bot.
- **How to confirm it applied:** **you can't today.** The bot has no window and shows its version nowhere. After each release, post in #rpa-support with your machine name and ask dev to confirm the version.
- **If the version did not change:** don't copy files by hand. Post in #rpa-support. Dev must check the certificate and the download on that machine (E15, E16).
- Every machine reads and writes only its own files, so a machine on an old build doesn't damage the others. It can still behave differently, though, so keep all machines on the same version.
## 11. Escalating to dev
Post in **#rpa-support** (Teams). Send:
- the exception ID from section 7 (for example "E06")
- the machine name and the time of the run
- the Reconcile file name, or "no Reconcile file"
- the log file `logs\ledger_<date>.log` (attach it, don't paste lines)
- `logs\notfound_<button>.png` if there is one
**Never send:** passwords, the `.env` or any settings file, or statement or ledger contents (references, amounts, payee names) pasted in chat. The log can contain payee names, so share it as a file in the channel only, never outside it.
## 12. Failure-class check
- **F1** (missing prerequisite) is covered in section 3 and E01, E12, E13, E14, and in section 13 items 8 and 12.
- **F2** (wrong window / focus) is covered in section 4 and E17. The bot never brings LedgerPro to the front itself (section 13 item 11).
- **F3** (target app slower or different) is covered in section 3 (scaling and resolution), E02, E03, E05, E06 and section 8. The fixed waits are section 13 item 6.
- **F4** (input read differently) is covered in section 3 (the statement rules), E09, E10 and E14.
- **F5** (marked done when nothing was done) is covered in E07 and section 13 item 2.
- **F6** (action happened but not confirmed) is covered in E06 and E07 and section 13 items 2 and 3.
- **F7** (needs follow-up shown as success) is covered in section 6 (`Missing - Posted`) and section 13 item 1.
- **F8** (endless re-runs / watchdog): partly not applicable, because the bot has no auto-rerun and no watchdog (section 8). One possible update loop is section 13 item 10.
- **F9** (stopping the wrong way) is covered in section 4 and E11.
- **F10** (one stuck item blocks the queue): not applicable as a queue, because there are no batches or locks and each run is independent. The one blocking case, a half-filled form left open, is E06.
- **F11** (cause hidden from ops) is covered in section 13 items 4, 5 and 7.
- **F12** (update didn't arrive) is covered in section 10, E15, E16 and section 13 items 9, 10 and 12.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Posted lines keep Result `Missing` (Note `Posted`), and the summary counts them as Missing. | Ops can't tell "posted" from "still missing" by Result alone. | `app.py:87-93` |
| 2 | Nothing confirms the save. "Posted" is written right after the Save click, and LedgerPro popups or validation are never detected. | An entry that was never saved shows as posted (F5/F6). | `app.py:65-70`, `app.py:90` |
| 3 | When Save is not found, the typed form is left open and the bot goes on to the next line. | Later lines can be typed into the leftover form, and the results after an Error can't be trusted. | `app.py:65-70`, `app.py:91-92` |
| 4 | Run-level failures write nothing to the log or the output: no shortcut, Export not found, export missing or bad, bad statement amount, no statement file given. | Ops sees "nothing happened" with no reason (E01-E03, E10, E13, E14). | `app.py:34`, `app.py:77-81`, `app.py:101` |
| 5 | Results are saved only at the very end. Any crash loses the whole Reconcile file, even though entries were posted. | The only record is the log (E11, E12). | `app.py:96` |
| 6 | Fixed waits of 15 s and 10 s, and no check that the export file is fresh. The docstring says the bot saves the export, but the code only clicks Export. | Slow LedgerPro means crashes or duplicate postings from yesterday's export (E03, E04). | `app.py:50`, `app.py:76-79` |
| 7 | The run summary goes only to the console, which the launcher discards. The not-found screenshot name has no time or reference, so each one overwrites the last. | Ops can't see the totals, and the evidence for earlier failures is lost. | `app.py:45`, `app.py:97`, `launcher.py:30` |
| 8 | `logs\` must already exist when the program starts. `data\` and `data\output\` are neither in the repo nor created by the code. | The bot won't start, or loses its output (E12, E13). | `app.py:24`, `app.py:96` |
| 9 | The launcher hides every update error, with no log. Download and unzip are unguarded. A version that is different but older is installed as an "update". There is no checksum. | Silent stale versions, a launcher crash after a bad release, or an accidental downgrade of every machine (E15, E16). | `launcher.py:19-20`, `launcher.py:25-29` |
| 10 | The launcher compares the server version against the version frozen into **its own** exe. If an update doesn't replace launcher.exe, every start may re-download. If the zip does contain launcher.exe, unpacking over the running exe can fail. Please confirm which happens. | Endless re-installs, or a crash on every start. | `launcher.py:9`, `launcher.py:25-29` |
| 11 | The version is shown nowhere. The comment claims a window title, but the bot has no window. There is also no Stop button, and nothing brings LedgerPro to the front before typing. | Ops can't verify releases, and stopping means Task Manager or the mouse-corner trick. | `app.py:15`, `app.py:65-70` |
| 12 | The build file in the repo is all comments. The "assets in the build" fix is not in it. Dependencies are not pinned. Newer versions of the screen-matching library raise an error instead of returning "not found", which would crash the run instead of marking the line Error. | The next build may ship without buttons, or behave differently from today's. | `ledgerbot.spec:2`, `requirements.txt:1-4`, `app.py:40` |
| 13 | Repeated references in the statement are dropped silently. | A line goes missing from the results with no trace (E09). | `app.py:52`, `app.py:81` |
| 14 | Display scaling and resolution are never checked at start. | A 125% laptop fails line by line instead of stopping with a clear message. | `app.py:20` |
| 15 | The 2026-09-05 log has a payee name on a Mismatch line. The current code doesn't write that, so the deployed exe may differ from this repo. | The repo may not be what is running, and logs may carry payee names. | `logs/ledger_20260905.log:3`, `app.py:60`, `app.py:95` |
| 16 | A `.env` file is committed, but no code reads it. It was not opened for this review. | Possible secrets in git. | `.env` |
| 17 | The `BOT_WAS_RUN` marker is written but never read or cleared. | Looks like run evidence, but it means nothing. | `app.py:74` |
| 18 | Ops access to LedgerPro is not confirmed. E06, E07 and E11 need someone to check entries in LedgerPro. | Without access, every Save error has to go to dev. | (process) |
tests/evals/results/handoff/h-packaged-desktop-screens-r1.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-packaged-desktop/screens",
"rep": 1,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"display scaling rule": true,
"version check": true,
"shortcut prerequisite": true,
"silent update failure flagged": true
},
"cost_usd": 1.4194375999999997,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-packaged-desktop-screens-2biod2mu\\project",
"reply": "I've written `HANDOVER.md` at the repo root. I only read the code and git history; I didn't run the bot, and no code was changed.\n\n| Item | Count |\n|---|---|\n| Sections | 13 (all from the template) |\n| Exceptions catalogued | 14 (E01\u2013E15; E06 isn't used) |\n| Failure classes | 10 covered, 2 not applicable (F8: no auto-rerun or watchdog; F10: no batches or locks) |\n| Fix before handover | 15 items |\n\n**Mukkiyamaana findings:**\n- **3 \"fix:\" commits um oru comment line mattum add pannudhu, code maaralai.** 125% scaling bug, launcher SSL bug, and the exe build missing the button pictures \u2014 moonum innum open. Ops rule-a \"100% scale, 1920x1080\" ah section 3-la pottiruken.\n- **Save button not found** aana, typed entry form open-aave irukkum; next line adhu mela type aagum. Double-posting risk irukku. So E08 la \"re-run pannradhukku munnaadi LedgerPro-la check pannunga\" nu pottiruken.\n- **Crash aana result file ae varaadhu**, kaaranam ops paakkura edathula edhuvum record aagadhu. Results ellam run mudiyumbodhu mattum dhaan save aagudhu.\n- **Version ops-ku engayum theriyaadhu.** Code comment \"window title\" nu sollum, aana bot-ku window ae illa.\n- `.env` (LedgerPro credentials) and oru log file git-la commit aagi irukku. Andha log-la business data irukku.\n\n**Also noticed:** Neenga \"no logs folder\" nu sonninga, aana code `logs` folder-la dhaan ezhudhudhu. Adhu illaatti rendu exe um start-laye crash aagum. \"Ops-ku logs access illa\" nu eduthukitten. `logs` folder dev/IT setup panna vendiya prerequisite-a pottiruken, *Where to look* table-la vekkala.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open (the app's screens, the output folder, the log folder, the database)? | Unknowns: are there any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-packaged-desktop-screens-r1.md
+178
View File
@@ -0,0 +1,178 @@
# LedgerBot - ops handover
Version 3.2.0 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
LedgerBot checks the daily bank statement against LedgerPro, the desktop accounting app. You give it the bank statement Excel when you start it. It opens LedgerPro, exports LedgerPro's entries to Excel and compares each statement line with them by reference and amount. Any statement line that is not in LedgerPro gets posted there by the bot: it clicks **New entry**, types the reference and amount, and clicks **Save**. At the end it writes one result file, `Reconcile_<date>_<time>.xlsx`, in the output folder. Each statement line gets a Result there: OK, Missing, Mismatch or Error. The only app it drives is **LedgerPro**. It runs once per statement, whenever ops starts it.
## 2. How it works
```
launcher.exe (what you start, with the statement file)
├─ checks for a newer LedgerBot build, installs it if found
└─ starts → LedgerBot.exe ──drives (mouse + keyboard)──> LedgerPro
│ (do not touch while it runs)
├─ reads → your bank statement Excel
├─ reads → LedgerPro's export (data\ledger_export.xlsx)
└─ writes → output folder: Reconcile_<date>_<time>.xlsx
```
- **launcher.exe** checks the update server, installs a new version if there is one, then starts LedgerBot. It shows nothing on screen.
- **LedgerBot.exe** does the work. It has no window of its own, no buttons and no progress display. It works by finding LedgerPro's buttons on the screen *by their picture* and clicking them.
- **LedgerPro** is the accounting app. While the bot runs, the mouse and keyboard belong to the bot.
- **Output folder** (`data\output`) gets one result file per run. It is the only record ops can open.
**One run, step by step:**
1. The bot opens LedgerPro from its Desktop shortcut and waits 15 seconds.
2. It clicks LedgerPro's **Export to Excel** button and waits 10 seconds.
3. It reads LedgerPro's exported entries and your statement (column A = reference, column B = amount, from row 2).
4. It compares each statement line with LedgerPro: OK, Missing or Mismatch.
5. For each Missing line it posts the entry: New entry → type reference → Tab → type amount → Save.
6. **Only at the very end** it saves the result file to the output folder. If the run stops earlier, there is no result file at all.
## 3. Before a run
- [ ] **Display: Scale 100% and resolution 1920 x 1080** (Windows Settings → System → Display). Check this again after docking the laptop, adding a monitor or a Windows update. At any other setting the bot cannot find LedgerPro's buttons (E03, E07, E08).
- [ ] The **LedgerPro** shortcut is on your Desktop or on the Public Desktop, and its name is exactly `LedgerPro`. Without it nothing happens (E02).
- [ ] LedgerPro opens straight to its main screen with **no sign-in prompt and no popup**. The bot does not sign in and does not close popups.
- [ ] LedgerPro is **closed** before you start; the bot opens it itself.
- [ ] The folders `logs`, `data` and `data\output` exist next to `LedgerBot.exe`. The bot does not create them. If one is missing, the run fails (E01, E05). Ask dev/IT to set them up once; ops does not need to open `logs`.
- [ ] Statement Excel rules:
- The data is on the **first sheet** (the one that opens by default). Row 1 is a header; data starts on row 2.
- Column A = reference, column B = amount. **Every row with a reference must have a number in column B**: no text, no blanks, no "1,234.00 CR". Otherwise the whole run fails with no output (E04).
- **Each reference only once.** If a reference appears twice, the bot silently keeps only the last line (E10).
- Type references **exactly as LedgerPro shows them**, and format column A as Text. Bad: `123` when LedgerPro has `00123` (Excel drops the zeros), or `INV-55 ` with a trailing space. Good: `00123`, `INV-55`. A reference that doesn't match exactly is treated as Missing and **posted again** (E15).
- [ ] Version: there is no place where ops can see the version today (see section 10).
## 4. During a run
- **Hands off the mouse and keyboard.** The bot types into whatever window is in front. If you click into another window (email, chat), the reference and amount get typed there and LedgerPro is not updated (E12).
- Don't cover, move or minimise LedgerPro, and don't open other windows on top of it.
- **Don't move the mouse into the top-left corner of the screen.** That is the bot's emergency stop: the run ends at once, with no result file.
- **To stop the bot:** move the mouse pointer into the top-left corner of the screen, or end `LedgerBot.exe` in Task Manager. **Closing LedgerPro does not stop it**; the bot keeps clicking and typing. After a stop, there is no result file. Check LedgerPro by hand for entries already posted before you re-run (E05).
- A run looks stuck when the mouse hasn't moved for more than a minute. The bot waits at most **20 seconds** for any button, so a real run never sits still that long. If nothing has happened for a few minutes, the run has ended, maybe without a result file. Check the output folder.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | Output folder `data\output`: a new `Reconcile_<date>_<time>.xlsx` with the time of your run. **No new file = the run did not finish** (E01-E05). |
| Which lines failed? | The Reconcile file → filter the **Result** column for `Error` and `Mismatch`. |
| Why did one line fail? | The **Note** column on the same row. |
| Was the entry really posted? | LedgerPro itself: search for the reference. The bot does not check after Save. |
| Did the bot already retry? | It never retries a line. For each button it keeps looking for 20 seconds; after that the line is Error. |
| Which version is running? | Ops cannot see it today (see section 10 and *Fix before handover*). |
| Why did a run end with no result file? | Nothing ops can open says why. Raise a ticket (section 11). |
## 6. Statuses
The Reconcile file is plain Excel, so there are no colours.
| Result | Note | Meaning | What ops does |
|---|---|---|---|
| OK | (empty) | Reference and amount match LedgerPro (within 0.01). | Nothing. |
| Missing | Posted | The line was not in LedgerPro and **the bot posted it**. "Missing" here means "was missing, now posted". The bot clicked Save but **did not check the save worked**. | Spot-check a few in LedgerPro. Nothing else if they are there. |
| Mismatch | LedgerPro has `<amount>` | The reference exists in LedgerPro with a different amount. **The bot changes nothing.** | Correct it by hand in LedgerPro or with Finance (E09). |
| Error | Button new_entry.png not found on screen | The line was Missing, but the bot could not find LedgerPro's **New entry** button. Probably not posted. | E07 |
| Error | Button save_btn.png not found on screen | The bot typed the entry but could not find **Save**. **It may or may not be saved.** | E08: check in LedgerPro before any re-run. |
| (no file) | - | The run stopped before the end. **Nothing is recorded for any line**, even lines already posted. | E01-E05 |
Not reported at all: entries that are in LedgerPro but **not** on the statement. The bot ignores them.
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | You start the bot and **nothing happens at all**: LedgerPro doesn't open, no result file | The bot stops before it begins: no statement file was given, or the `logs` folder is missing next to the exe | Nowhere ops can open | System | Ops, then Dev | Start it again with the statement file (for example, drag the statement Excel onto `launcher.exe`). | It still does nothing → ticket: laptop name, time, "E01". |
| E02 | Nothing opens; no result file | It looks for a shortcut named `LedgerPro` on your Desktop and the Public Desktop; if none, it stops | Nowhere ops can open | System | Ops | Put the LedgerPro shortcut back on the Desktop with the exact name `LedgerPro`, then re-run. | The shortcut is there and it still fails → ticket: laptop name, time. |
| E03 | LedgerPro opens, **nothing gets clicked**, the run ends with no result file | It looks for the **Export to Excel** button picture for 20 s; if it never matches, the run stops. Nothing was posted. | A screenshot in the dev log folder (ops can't open it) | System | Ops, then Dev | Check: Display scale 100% and 1920 x 1080; LedgerPro on its main screen with no sign-in or popup; nothing covering it. Fix that, then re-run once. | Fails again with the settings right, **or** it happens on every laptop right after an update (broken build) → ticket: laptop name, time, screenshot of the screen. |
| E04 | LedgerPro opens, Export is clicked, then **the run ends with no result file** and nothing is posted | It could not read the data: an amount that is not a number in your statement or LedgerPro's export, or the export file is not where it expects it | Nowhere ops can open | Business or System | Ops, then Dev | Check that every row of your statement has a plain number in column B, fix it, then re-run. | The statement is clean and it still fails → ticket: laptop name, time, the statement file (send it through the ticket, not chat). |
| E05 | Some entries were posted in LedgerPro but **there is no result file** | The run stopped part-way: someone stopped it, the mouse hit the top-left corner, a button error the bot can't handle, or the `data\output` folder is missing (then it fails at the very end, after posting everything) | Nowhere. The posted entries are in LedgerPro only. | System | Ops, then Dev | **Don't re-run straight away.** In LedgerPro, look up the statement references to see which were posted. Check `data\output` exists. Then re-run once: the bot re-exports, so posted lines should come back OK. | It happens twice → ticket: laptop name, time, which references were posted. |
| E07 | Result **Error**, note "Button new_entry.png not found on screen" | It looks for LedgerPro's **New entry** button for 20 s; if it never matches, it marks the line Error and goes to the next line | Reconcile file, Result + Note | System | Ops | Check display scale/resolution. Look the reference up in LedgerPro; if it isn't there, re-run (only Missing lines get posted). | Every Missing line errors with the settings right → ticket: Reconcile file, laptop name. |
| E08 | Result **Error**, note "Button save_btn.png not found on screen" | It typed the reference and amount, then looked for **Save** for 20 s and didn't find it. The half-filled form is **left open**, and the **next** line is typed on top of it. | Reconcile file, Result + Note | System | **Ops** | **Check in LedgerPro first**: is this reference saved, and is the amount right? Also check the **next** line's entry. Delete or fix any wrong or half-done entry by hand. Only then re-run. Re-running without checking can post it twice. | Several lines in one run → ticket: Reconcile file, laptop name, time. |
| E09 | Result **Mismatch**, note "LedgerPro has 123.45" | The reference is in LedgerPro but the amounts differ by more than 0.01 | Reconcile file | Business | Ops | Find out which amount is right (statement or LedgerPro) and correct it by hand. The bot never changes amounts. | Never; it is a business task. |
| E10 | A statement line has **no row** in the Reconcile file; fewer rows than the statement | The same reference appears twice in the statement (or in LedgerPro); the bot silently keeps only the last one | Nowhere; count the rows to notice | Business | Ops | Before the run, remove or merge duplicate references. After the run, compare row counts: statement vs Reconcile file. | - |
| E11 | **The same entry posted twice** in LedgerPro | Nothing detects it. Causes: re-running after E08 without checking, or LedgerPro's export not saving fresh, so the bot compares against an old export | Nowhere | System | Ops, then Dev | Delete the duplicate in LedgerPro by hand. | It happens with no E08 before it (stale export) → ticket: Reconcile file, time, references. |
| E12 | Reference and amount **typed into another window** (email, chat); entries missing in LedgerPro, maybe marked Missing/Posted | The bot types into whatever window is in front; it never checks it is LedgerPro | Nowhere; the Reconcile file may still say Posted | System | Ops | Check those references in LedgerPro and post them by hand, or re-run. Hands off during runs (section 4). | - |
| E13 | After a release, **the laptop still runs the old version** | The launcher couldn't reach the update server (offline, or the server's certificate not trusted) and starts the old version without saying so | Nowhere | System | Dev | Raise a ticket naming the laptop. | Always, once dev announces a release. |
| E14 | After a release, **double-clicking launcher.exe does nothing**; the bot never starts | The update was found but the download or unzip failed; the launcher stops | Nowhere | System | Dev | Raise a ticket: laptop name, time. | Always. |
| E15 | A line that **is** in LedgerPro shows **Missing / Posted**, so there is now a duplicate | The reference doesn't match LedgerPro character for character (lost leading zeros, extra space, different case) | Reconcile file shows Posted | Business | Ops | Delete the duplicate in LedgerPro. Fix the reference in the statement (column A as Text, exact spelling). | - |
(E06 is not used.)
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| A LedgerPro button is not on the screen yet | Looks again every second | 20 s per button | Export button: the run stops, no result file (E03). New entry / Save: the line is Error, next line (E07, E08). |
| LedgerPro slow to open | Fixed wait, no check | 15 s | Goes on regardless; usually ends as E03 |
| Export slow to save | Fixed wait, no check | 10 s | Goes on regardless; may read an old export (E11) |
| Update server unreachable | Starts the current version | 10 s | Old version keeps running (E13) |
There are **no** retries per line, no restarts and no resume. A failed line is retried only when ops re-runs the whole statement. Lines already posted then come back OK, as long as LedgerPro's export is fresh.
## 9. Known incidents
All three are recorded in the history as fixed on 2026-10-23. In the code, though, each "fix" is only a note; the bot's behaviour is unchanged (see section 13).
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-10, 3.2.0 | Lines Error "Button save_btn.png not found on screen" on some laptops | Laptops at 125% display scaling; the button pictures were taken at 100% | **Keep 100% scale and 1920 x 1080** (section 3). Check each Error line in LedgerPro (E08). | Partly |
| 2026-10, 3.2.0 | Two laptops stayed on the old version, with no message | Update server's certificate not trusted; the launcher hid the error | Raise a ticket after every release if a laptop didn't update (E13). Ops can't confirm the version yet. | Partly |
| 2026-10, 3.2.0 | Exe build couldn't find any button (every run fails at Export) | The button pictures were left out of the exe build | Dev rebuild. Ops: E03 on every laptop right after an update → ticket. | No |
## 10. Releases and version check
- **How an update arrives:** `launcher.exe` checks the update server every time you start it. If there is a different version, it downloads and installs it next to the exe, then starts the bot. It shows nothing on screen.
- **How to confirm it applied:** today **ops cannot see the version**. The bot has no window and no title bar, and neither the log nor the result file records the version. Ask dev to confirm per laptop.
- **If the version did not change:** raise a ticket with the laptop name (E13). Don't copy files between laptops yourself.
- Laptops on different versions don't break each other; the bot shares nothing between machines.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue. Send:
- the exception ID from section 7 (for example "E08"),
- the laptop name,
- the date and time you started the run,
- the statement file name,
- the Reconcile file from `data\output`, if there is one,
- a screenshot of the screen if the bot stopped on something.
Dev collects the log and the "button not found" screenshot from the laptop themselves.
Never send: passwords, the bot's settings file, or bank or customer data in chat. Attach files to the ticket only.
## 12. Failure-class check
- F1 covered in section 3 (shortcut, folders, no sign-in prompt), E01, E02, E03 (exe build missing pictures), and *Fix before handover* 3, 8.
- F2 covered in section 4 (hands off, don't cover LedgerPro), E12, and *Fix before handover* 6.
- F3 covered in section 3 (100% scale, 1920 x 1080), E03, E07, E08, section 8 (20 s / 15 s / 10 s limits), and *Fix before handover* 7.
- F4 covered in section 3 (statement rules with examples), E04, E10, E15, and *Fix before handover* 9.
- F5 covered in section 6 (Missing/Posted not confirmed) and *Fix before handover* 4.
- F6 covered in E08 (check LedgerPro before re-running), E11, and *Fix before handover* 4, 5.
- F7 covered in section 6: no business exception is shown as success, but "Missing" is the label for successfully posted lines, which is misleading. *Fix before handover* 10.
- F8 not applicable: the bot has no auto-rerun loop and no watchdog; each button wait ends after 20 s.
- F9 covered in section 4 (how to stop; closing LedgerPro doesn't stop it) and *Fix before handover* 6.
- F10 not applicable: there are no batches, locks or completion gates. Each run is independent and nothing blocks the next one. The "was run" marker file the bot writes is never read.
- F11 covered in section 5 (only the Reconcile file is visible to ops), E01-E05, and *Fix before handover* 1, 2.
- F12 covered in section 10, E13, E14, and *Fix before handover* 11, 12.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Crashes (no statement argument, missing shortcut, Export not found, bad amount, missing folder, failsafe) are not logged and leave no result file. Results are saved only at the end. | Ops sees "no file" with no reason; lines already posted aren't recorded anywhere | `app.py:73-97` (single `out.save` at `:96`) |
| 2 | The per-line log, the "not found" screenshots and the run summary (printed to stdout, which the launcher never captures) all sit where ops has no access. The screenshot has one fixed name per button and is overwritten. | Ops has only the Reconcile file; the cause stays hidden | `app.py:24`, `:45`, `:97` |
| 3 | `logs/`, `data/` and `data/output/` are never created; a missing `logs/` crashes both exes at import | Fresh install fails, or fails after all posting is done | `app.py:24`, `:45`, `:96`; `launcher.py:9` |
| 4 | "Posted" is written after clicking Save, with no check that LedgerPro saved | A Missing/Posted line may not be in LedgerPro | `app.py:65-70`, `:90` |
| 5 | A Save miss leaves the typed form open; the next line types on top of it | One Error can corrupt the next entry | `app.py:65-70`, `:91-92` |
| 6 | No window focus or "is LedgerPro in front" check before typing; no Stop button; closing LedgerPro doesn't stop the bot | Keystrokes land in other windows | `app.py:66-69` |
| 7 | Fixed sleeps (15 s / 10 s) instead of waiting for LedgerPro and the export; nothing checks the export file is fresh or where LedgerPro saves it | Old export means double posting (E11) | `app.py:76-79`, docstring `:50` |
| 8 | Scaling fix is a comment only: no DPI/resolution check at start. Exe build spec is all comments (`datas` commented out). Only `RuntimeError` is caught; newer pyautogui raises its own not-found error and `requirements.txt` is unpinned. | Any scale change or new build breaks every run | `app.py:20`, `:40`, `:91`, `:102`; `ledgerbot.spec:1-3`; `requirements.txt` |
| 9 | Duplicate references silently overwritten; references not trimmed or normalised; non-numeric amount crashes the run | Lines vanish, or are posted twice | `app.py:52`, `:81` |
| 10 | Result "Missing" is kept for lines that were successfully posted; a status like "Posted" would read correctly. The note "Not in LedgerPro - posting" is never shown. | Ops reads done lines as problems | `app.py:58`, `:87-92` |
| 11 | Launcher swallows every update error (incl. the self-signed certificate) with no log; download/unzip unprotected; no checksum; `!=` accepts downgrades; extracts over the running folder | Laptops silently stay old, or the launcher crashes | `launcher.py:15-30` (fix commit only added comment `:35`) |
| 12 | The version is not shown anywhere ops can see (the comment says window title, but the bot has no window); not in the log or the result file | Ops can't confirm an update | `app.py:15` |
| 13 | Two runs in the same minute overwrite each other's result file | A result file is lost | `app.py:96` |
| 14 | `.env` holding LedgerPro credentials and a log file containing business data are committed to git; `.env` is never read by the code | Credential and data exposure | `.env`, `logs/ledger_20260905.log` |
| 15 | The log note can carry counterparty names (seen in the committed log, from an older build) | Business data in logs | `logs/ledger_20260905.log` |
tests/evals/results/handoff/h-packaged-desktop-screens-r2.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-packaged-desktop/screens",
"rep": 2,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"display scaling rule": true,
"version check": true,
"shortcut prerequisite": true,
"silent update failure flagged": true
},
"cost_usd": 1.5397530000000001,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-packaged-desktop-screens-u9wl16io\\project",
"reply": "`HANDOVER.md` ready, repo root la irukku: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-packaged-desktop-screens-u9wl16io\\project\\HANDOVER.md`. Code edhuvum change pannala, bot-a run pannala.\n\n| Check | Result |\n|---|---|\n| Sections | 13 (template order maari) |\n| Exceptions catalogued | 13 (E01\u2013E13) |\n| Failure classes | 12/12 covered, \"not applicable\" onnum illa |\n| Fix before handover | 17 items |\n| Leak scan | PHI, URL, IP, key edhuvum illa; file:line section 13 la mattum |\n\n**Ops-ku mukkiyamaana vishayangal:**\n- Bot-ku sondha screen illa: window illa, Stop button illa, version kaatra edamum illa. Adhanaala ops paakka mudinjadhu output folder la irukkura `Reconcile_*.xlsx` file mattum dhaan.\n- Save click aana odane bot `Posted` nu ezhudhidum, save aachaannu check pannaadhu. Adhanaala re-run pannradhukku munnaadi LedgerPro la check pannanum (E04, E05, E07).\n- Run crash aana output file varaadhu. Adhukku munnaadi post aanadhu ellaam LedgerPro la appadiye irukkum.\n- History la 3 \"fix\" commits irukku. Adhula 2 code la illa, comment mattum dhaan add aagirukku: launcher innum update error-a maraikkudhu, build spec la assets line innum commented out. Section 9 and 13 la idha open-aa dhaan pottirukken.\n\n**Fix before handover (17), short-aa:**\n1. `Posted` save confirm pannaama ezhudhapadudhu\n2. Fixed 15 s / 10 s waits; export file pazhasaannu check illa\n3. Crash aana output-um illa, reason-um illa; watchdog-um illa\n4. Oru vagai error mattum dhaan catch aagudhu\n5. Save fail aana form open-aave irukku, adutha rows-um fail aagum\n6. Display scaling / resolution check illa\n7. Post aana rows-um `Missing` nu dhaan kaattudhu; colours illa\n8. Log, screenshots ops-ku edukka mudiyaadhu; screenshots overwrite aagudhu\n9. `logs\\`, `data\\output\\` folders munnaadiye irukkanum\n10. Statement input validation illa\n11. Build spec la assets bundle aagala\n12. Launcher update errors-a maraikkudhu\n13. Launcher version compare (`!=`), checksum illa\n14. Version kaatra edam illa\n15. Double-post guard illa\n16. `requirements.txt` la versions pin pannala\n17. `BOT_WAS_RUN` marker ezhudhapadudhu, aana yaarum padikkala\n\nUnga answers-a apdiye use pannen: escalation = RPA Support queue ticket, ops-ku output folder mattum. Logs ops-ku edukka mudiyaadhadhaala \"Where to look\" la vekkala, Fix list la pottirukken (#8).\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions mean where the code doesn't make it clear?",
"stderr": ""
}
tests/evals/results/handoff/h-packaged-desktop-screens-r2.md
+183
View File
@@ -0,0 +1,183 @@
# LedgerBot - ops handover
Version 3.2.0 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
LedgerBot reconciles the day's bank statement against LedgerPro, the desktop accounting app. You give it the bank statement Excel. It opens LedgerPro, exports LedgerPro's entries to Excel, and compares each reference and amount in the statement with that export. When a statement entry is not in LedgerPro, the bot creates it in LedgerPro (New Entry → reference → amount → Save). When the amounts differ, it only reports the difference. The output is one Excel file per run, `Reconcile_<date>_<time>.xlsx`, in the output folder. The bot runs once per statement, whenever ops starts it. It drives one app: **LedgerPro**. It finds LedgerPro's buttons by matching pictures of them on the screen.
## 2. How it works
```
launcher.exe (what you start)
├─ checks for an update → installs it into the bot folder
└─ starts → LedgerBot.exe ──drives (mouse + keyboard)──> LedgerPro
│ (do not touch while it runs)
├─ reads → your bank statement Excel
├─ reads → LedgerPro's export file
└─ writes → data\output\Reconcile_<date>_<time>.xlsx (at the very end only)
```
- **launcher.exe** checks the update server for a new version, installs it if there is one, then starts the bot. It does not wait for the bot or tell you whether the bot succeeded.
- **LedgerBot.exe** is the bot itself. It has **no window of its own**: no screen, no progress bar, no Stop button. All you see is LedgerPro being driven.
- **LedgerPro** is the accounting app. The bot clicks its buttons and types into its forms, so the mouse and keyboard belong to the bot during a run.
- **The output file** is written only when the bot has gone through every row. If no new file appears, the run did not finish.
**One run, step by step:**
1. The launcher checks for an update and starts the bot.
2. The bot opens LedgerPro from the `LedgerPro` shortcut on the Desktop, then waits 15 seconds.
3. It clicks LedgerPro's Export to Excel button, then waits 10 seconds for the export.
4. It reads the export and your bank statement.
5. It goes through the statement row by row: matching amount → OK; different amount → Mismatch; not in LedgerPro → it posts the entry.
6. It writes the output file to the output folder.
## 3. Before a run
- [ ] **Windows display scaling is 100%** and the screen resolution is **1920 x 1080** (Settings → System → Display). The button pictures were taken at that setting. At 125% the bot cannot find the buttons. Nothing checks this for you.
- [ ] LedgerPro's shortcut, named exactly `LedgerPro`, is on your Desktop or the Public Desktop. Without it, LedgerPro never opens (E01).
- [ ] LedgerPro opens fully within 15 seconds on this machine, with no login box, update prompt or popup in front. The bot does not handle popups.
- [ ] LedgerPro's export saves to the bot's usual export location (as set up by dev). The bot does not check that the export file is from today (E08).
- [ ] Only one monitor is in use, or LedgerPro is on the main monitor. The bot only looks at the main screen.
- [ ] Bank statement Excel, first sheet:
- Row 1 is a header. Data starts on row 2.
- Column A = reference, column B = amount. Nothing else is read.
- **Every row with a reference must have an amount, as a real number.** An empty amount or text like `1,200.00 CR` stops the whole run (E09).
Bad: `TX-1001 | (blank)` · Good: `TX-1001 | 1200.00`
- **Each reference once.** If a reference appears twice, only the last row is used and the other is silently dropped.
- Type references as text, exactly as LedgerPro shows them. A reference typed as a plain number can be read as `1001.0` and will not match LedgerPro's `1001`.
Bad: `1001` stored as a number · Good: `1001` stored as text (or `TX-1001`)
- Use only plain English letters, digits and punctuation in references. The bot cannot type accented or other special characters.
- [ ] Start the run by **dragging the statement Excel onto `launcher.exe`** (or the shortcut ops was given). Starting it without a file does nothing visible and produces no output.
- [ ] Version check: the bot has no screen that shows its version today (see section 10). Ask dev to confirm the version after each release.
## 4. During a run
- **Hands off the mouse and keyboard** for the whole run. The bot types into whatever window has focus. If you click elsewhere, it types references and amounts into that window.
- Do not open or move other windows over LedgerPro. A window covering a button makes the bot fail that row.
- **Do not move the mouse into a corner of the screen.** That is the emergency stop: the bot stops at once and **writes no output file**, but anything it already posted stays posted in LedgerPro.
- **To stop the bot:** there is no Stop button. Move the mouse into a corner of the screen, or end `LedgerBot.exe` in Task Manager. Closing LedgerPro does **not** stop the bot. After stopping, do the checks in E07 before running again.
- **How long a run takes:** about 30 seconds of start-up, then a few seconds per posted entry. Every row that fails costs an extra 20 seconds or more. If one Save fails, the following rows often fail too (E04), so a run with many failures can take a long time. There is no watchdog. If LedgerPro has shown no movement for more than 2 minutes and no output file has appeared, stop the bot as above and follow E07.
## 5. Where to look when something goes wrong
Ops can open the output folder and the output files only. The bot has no screens of its own, and the log folder is not open to ops (see *Fix before handover*).
| Question | Where to look |
|---|---|
| Did the run finish? | Output folder `data\output\`: is there a new `Reconcile_<date>_<time>.xlsx` with the time of this run? No new file = the run did not finish (E01, E02, E07, E09). |
| Which items failed? | The output file, **Result** column = `Error`. Also check `Mismatch` rows: they need a person. |
| Why did one item fail? | The output file, **Note** column of that row. Then look at LedgerPro itself. |
| Was an entry really posted? | Only LedgerPro can tell you. The bot writes `Posted` after clicking Save, without checking (E05). |
| Did the bot already retry? | It never retries an item or a run. It only searches for a button for 20 seconds before giving up on that row. |
| Which version is running? | Not visible to ops today. Ask dev (section 10). |
## 6. Statuses
The output file has no colours. Statuses are the text in the **Result** and **Note** columns.
| Result | Note | Meaning | What ops does |
|---|---|---|---|
| `OK` | (empty) | The reference is in LedgerPro with the same amount (within 1 cent). | Nothing. |
| `Missing` | `Posted` | The reference was not in LedgerPro, and the bot entered it and **clicked Save**. Despite the word "Missing", the bot **believes it posted the entry**. It did **not** check that the save worked. | Spot-check a few in LedgerPro. Check all of them if any Save-related errors appear in the same run (E04, E05). |
| `Mismatch` | `LedgerPro has <amount>` | The reference exists in LedgerPro with a different amount. **The bot changed nothing.** | **Needs follow-up.** Investigate and correct by hand in LedgerPro (E06). |
| `Error` | `Button new_entry.png not found on screen` | The bot could not find LedgerPro's New Entry button. Nothing was entered for this row. | E03. |
| `Error` | `Button save_btn.png not found on screen` | The bot typed the entry but could not find the Save button. The entry is **most likely not saved**, and the form may still be open. | E04. |
| (no output file) | - | The run stopped before the end. Nothing is recorded for any row, even rows already posted. | E07. |
`Not in LedgerPro - posting` is a note the bot sets internally before posting. It is always replaced, so you will never see it in the output.
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | LedgerPro never opens; no output file appears | It looks for a shortcut named `LedgerPro` on your Desktop and the Public Desktop. If neither has one, it stops at once. | Nowhere ops can see. No output file. | System | Ops | Put the `LedgerPro` shortcut on the Desktop (exact name), then run again. | The shortcut is there and LedgerPro still does not open. |
| E02 | LedgerPro opens, but nothing is clicked; no output file appears | After 15 s it looks for the Export to Excel button for 20 s. If it never matches the button picture, it stops the run. | Nowhere ops can see. No output file. | System | Ops, then Dev | Check scaling is 100% and resolution 1920 x 1080; close any popup or login box in LedgerPro; make sure LedgerPro is maximised on the main screen; run again. | It still happens with the right settings, or it happens on a fresh install or right after an update (the button pictures may be missing from the build, see incident #3). |
| E03 | Result `Error`, Note `Button new_entry.png not found on screen` | When posting, it looks for the New Entry button for 20 s and gives up on that row. | Output file, Result and Note | System | Ops | Check scaling and popups as in E02. Check in LedgerPro that the reference is not there, then run again with the statement. | Several runs in a row fail this way with correct settings. |
| E04 | Result `Error`, Note `Button save_btn.png not found on screen`, often followed by more Errors on later rows | It clicked New Entry and typed the reference and amount, then looked for Save for 20 s and gave up. It does not close the half-filled form, so later rows may type into the wrong place. | Output file, Result and Note | System | Ops | 1. In LedgerPro, close any open New Entry form without saving. 2. **Search LedgerPro for each Error reference and for the `Missing / Posted` rows after it**, to see what was really saved. 3. Fix or delete stray entries by hand. 4. Run again. | It happens again after a clean restart with correct settings. |
| E05 | Result `Missing`, Note `Posted`, but the entry is not in LedgerPro or has the wrong amount | The bot writes `Posted` as soon as it clicks Save. It does not check for a save confirmation, a validation message, or that the typed text landed in the right fields. | Output file says `Posted`; the truth is only in LedgerPro | System | Ops | **Check in LedgerPro before re-running.** Enter or correct the entry by hand, or run again with a statement holding only the missing rows. | More than one or two per run, or wrong amounts typed. Send the output file. |
| E06 | Result `Mismatch`, Note `LedgerPro has <amount>` | The reference is in LedgerPro, but the amounts differ by more than 1 cent. | Output file, Result and Note | Business | Ops | Find out which amount is right (statement or LedgerPro) and correct it by hand. The bot never changes existing entries. | Never for dev, unless the amounts are clearly read wrongly (e.g. every row mismatches). |
| E07 | The run stops midway: LedgerPro stops moving and no output file appears | Any unexpected problem stops the whole run: the mouse touched a screen corner, an amount could not be read, an unexpected error in LedgerPro. Nothing is recorded. | Nowhere. No output file, even for rows already posted. | System | Ops, then Dev | 1. **Do not re-run straight away.** 2. In LedgerPro, close any open form. 3. Check which statement references are already in LedgerPro. 4. Check the statement against the input rules in section 3. 5. Run again. | It stops again with a clean statement and nobody touching the machine. |
| E08 | Many rows `Missing / Posted` that were already in LedgerPro, or duplicate entries in LedgerPro after a run | The bot reads the last export file 10 s after clicking Export. If the export was slower, or saved elsewhere, it reads an older export without noticing. | Output file looks normal; duplicates only show in LedgerPro | System | Dev | Remove the duplicate entries in LedgerPro by hand. Do not run again until dev has checked the export. | Always. |
| E09 | No output file, right after LedgerPro exports | The bot could not read the statement: an empty or text amount, no file given, or the file is not an Excel workbook. | Nowhere. No output file. | Business | Ops | Fix the statement as per section 3 (every amount a number, one reference per row), then run again. | The statement is clean and it still fails. |
| E10 | A row you expected is missing from the output file | The statement had the same reference twice. The bot keeps only the last one. | Not recorded | Business | Ops | Remove duplicates from the statement, or post the dropped row by hand. | - |
| E11 | Text appears in another window; references or amounts typed in the wrong place | Someone used the mouse or keyboard, or another window came in front. The bot types into whatever has focus. | Not recorded | System | Ops | Undo the stray typing. Check LedgerPro for half-entered entries. Run again hands-off. | - |
| E12 | After a release, this machine behaves like the old version | The launcher could not reach the update server (offline, or the server's certificate is not trusted on this machine). It ignores the problem and starts the old version. | Not recorded | System | Dev | Raise a ticket so dev can check the version and update this machine. | Always, after every release where behaviour does not change. |
| E13 | Double-clicking the launcher does nothing; the bot never starts | The update download or install failed, and the launcher stopped before starting the bot. | Not recorded | System | Dev | Raise a ticket. Do not copy files into the bot folder yourself. | Always. |
## 8. What the bot recovers by itself
Very little. Ops should not expect it to fix anything on its own.
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| A button is not on screen yet | Looks for it once a second | 20 s per button | Export button: the whole run stops (E02). New Entry or Save: that row is `Error`, next row (E03, E04). |
| LedgerPro is slow to open | Nothing; it simply waits | Fixed 15 s | Same as above. |
| Export is slow | Nothing; it simply waits | Fixed 10 s | Reads whatever export file is there (E08). |
| Update server unreachable | Starts the old version without telling you | 10 s | Old version keeps running (E12). |
| Anything else | No retry, no restart, no resume | - | Run stops with no output file (E07). |
## 9. Known incidents
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Up to v3.2.0 (#3) | The packaged bot failed at its very first button on every machine | The button pictures were left out of the packaged build | Dev says this is fixed, but the current build file does not show it. If the bot fails at the first button on a fresh install or after an update, raise a ticket (E02). | No |
| Up to v3.2.0 (#2) | Rows `Error` "Button save_btn.png not found on screen" on some laptops | Laptops at 125% display scaling. The button pictures only match at 100%. | Keep scaling at 100% and resolution 1920 x 1080 (section 3). Check this first for any "not found" error. | Yes |
| Up to v3.2.0 (#1) | Two laptops stayed on the old version after a release; no error | The launcher could not trust the update server's certificate and quietly ran the old version | Still possible: the code still hides this. After each release, ask dev to confirm every machine's version (E12). | Partly |
## 10. Releases and version check
- **How an update arrives:** each time you start `launcher.exe`, it asks the update server for the latest version. If it differs, it downloads it and installs it into the bot folder, then starts the bot.
- **How to confirm it applied:** ops cannot see the version today. The bot has no window or title bar. Ask dev to confirm through the RPA Support ticket after each release.
- **If the version did not change** (behaviour after a release is unchanged): raise a ticket (E12). Do not copy files into the bot folder by hand.
- All machines should run the same version, so that results from different machines are comparable.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue. Send:
- the exception ID from section 7 (e.g. E04),
- the machine name and the date and time the run started,
- the file name of the bank statement used (not its contents in chat),
- the output file `Reconcile_<date>_<time>.xlsx`, if one was produced, or "no output file",
- the Result and Note of the failing rows,
- what LedgerPro showed at the time (a description, or a screenshot that shows no customer data).
Dev collects the bot's own log and "not found" screenshots from the machine; ops does not have access to them.
Never send: passwords, the bot's settings files, or customer or account data in chat.
## 12. Failure-class check
- **F1** missing prerequisite: covered in section 3 (shortcut, scaling, single screen) and E01, E02, E13; required folders and build contents in *Fix before handover* #9, #11.
- **F2** wrong window or focus: covered in section 4 (hands off, nothing over LedgerPro) and E11. The bot does not look for windows by title; it clicks wherever the button picture matches on the main screen.
- **F3** target app slower or different: covered in E02, E03, E04, section 8 (fixed 15 s and 10 s waits, 20 s per button) and section 3 (scaling, resolution, popups); *Fix before handover* #2, #6.
- **F4** input read differently: covered in section 3 input rules, E09, E10; *Fix before handover* #10.
- **F5** marked done when nothing was done: covered in E05 and E08; *Fix before handover* #1, #2.
- **F6** action happened but not confirmed: covered in E04, E05 and E07 (check in LedgerPro before re-running); *Fix before handover* #1, #5.
- **F7** business exception shown as success: covered in section 6. `Mismatch` needs follow-up but has no colour, and posted rows keep the result `Missing`; *Fix before handover* #7.
- **F8** endless re-runs or watchdog: covered in section 4. The bot has no re-runs and no watchdog, so a hung run has no time limit; *Fix before handover* #3.
- **F9** stopping the wrong way: covered in section 4 (corner or Task Manager; closing LedgerPro does not stop it) and E07.
- **F10** one stuck item blocks the queue: covered in E04. A failed Save leaves the form open and later rows fail too. There are no batches or locks between runs; *Fix before handover* #5.
- **F11** cause hidden from ops: covered in section 5. Crash reasons, the log and the screenshots are not reachable by ops; *Fix before handover* #3, #4, #8.
- **F12** update did not arrive: covered in section 10, E12, E13 and incident #1; *Fix before handover* #12, #13, #14.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | `Posted` is written as soon as Save is clicked; no save confirmation or check that the entry exists | Ops trusts entries that may not be in LedgerPro (E05) | `app.py:65-70`, `app.py:89-90` |
| 2 | Fixed sleeps for LedgerPro start (15 s) and export (10 s); the export file's date is never checked | A slow export means an old export is read, giving false `Missing` rows and duplicate posts (E08) | `app.py:75-79` |
| 3 | Output is written only at the end; a crash leaves no file and no reason anywhere; no watchdog | Ops cannot tell what was posted before a crash (E07) | `app.py:82-96`, `app.py:100-101` |
| 4 | Only one error type is caught per row; any other error (fail-safe corner, bad amount, image-library exception) ends the run | One bad row loses the whole run's results | `app.py:91`, `app.py:52`, `app.py:81` |
| 5 | After "Save not found" the half-filled form is left open; no reset before the next row | One failure spreads to later rows (E04) | `app.py:65-70`, `app.py:88-92` |
| 6 | No check of display scaling or resolution at start; images match only at 100% / 1920x1080 | Wrong setting shows up as many 20 s failures | `app.py:20` |
| 7 | Rows the bot posted keep Result `Missing`; nothing in the file is coloured; `Mismatch` is not flagged as needing follow-up | Ops reads `Missing` as "not done" and may post twice | `app.py:58`, `app.py:87-94` |
| 8 | The log and "not found" screenshots are in the log folder, which ops cannot open. Screenshots overwrite each other (no time or reference in the name) and may show customer data. All lines are INFO, with no start, end or error lines. The run summary goes only to the console. | Ops has no way to see the cause beyond the Note column | `app.py:24-25`, `app.py:45`, `app.py:95`, `app.py:97` |
| 9 | `logs\` and `data\output\` must already exist; missing `logs\` crashes at start, missing `data\output\` crashes after all posting | All posting done, no record of it | `app.py:24`, `app.py:96` |
| 10 | Statement input not validated: empty or text amount crashes the run; duplicate references silently collapse; numeric references become `1001.0`; no file argument crashes; non-ASCII cannot be typed | Silent skips and lost runs (E09, E10) | `app.py:81`, `app.py:67`, `app.py:101` |
| 11 | The build spec is only comments; the line that bundles the button pictures is commented out. The "assets missing" fix commit is not in the code. | A rebuild brings back incident #3 | `ledgerbot.spec:1-3` |
| 12 | The launcher still hides every update error (offline, SSL, bad reply). The "SSL error" fix commit only added a comment. | Machines stay on old versions unnoticed (E12) | `launcher.py:15-20`, `launcher.py:35` |
| 13 | The launcher compares versions with "not equal" and its own version is fixed at build time, so it can reinstall on every start, or install an older version. No checksum on the download; `update.zip` is left behind. | Slow starts, wrong versions, launcher crash (E13) | `launcher.py:9`, `launcher.py:25-29` |
| 14 | No visible version. The comment says the window title shows "LedgerBot v3.2.0", but there is no window. | Ops cannot do a version check (section 10) | `app.py:15` |
| 15 | No guard against posting twice when a run is repeated after a crash or with an old export | Duplicate entries in LedgerPro | `app.py:86-92` |
| 16 | Library versions are not pinned; newer versions of the image library raise an error instead of "not found", which would crash runs on any missing button | A rebuild can change behaviour silently | `requirements.txt` |
| 17 | The marker file `BOT_WAS_RUN` is written at start but never read | None today; misleading if someone relies on it | `app.py:74` |
tests/evals/results/handoff/h-packaged-desktop-screens-r3.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-packaged-desktop/screens",
"rep": 3,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": false,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"known incidents from history": true,
"display scaling rule": true,
"version check": true,
"shortcut prerequisite": true,
"silent update failure flagged": true
},
"cost_usd": 0.8733934000000001,
"subagents": 0,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-packaged-desktop-screens-qhg1yk_0\\project",
"reply": "I wrote `HANDOVER.md` at the repo root. Unga answers use pannen: escalation = RPA Support queue ticket, ops can open only the report in the output folder, logs folder illa, database illa. Bot-a run pannala and code-um maathala. Code moonu files dhaan (app.py, launcher.py, spec), so scan-ku subagents use pannama naane direct-a padichen.\n\n| Item | Count |\n|---|---|\n| Sections | 13 (template order) |\n| Exceptions | 14 (E01\u2013E14) |\n| Failure classes | 10 covered, 2 not applicable (F8: auto-rerun/watchdog illa; F10: batch/queue/lock illa) |\n| Fix before handover | 18 items |\n\n**Dev team-ku mukkiyamaana fixes:**\n- Bot Save click pannina udane, LedgerPro confirm pannadha-nu check pannama, row-la `Posted` nu ezhudhudhu.\n- LedgerPro-oda export file fresh-a-nu check panradhilla. Pazhaya file use aana, already irukka entries thirumba post aagum, entha warning-um illama.\n- Report run-oda kadaisila mattum dhaan save aagudhu. Naduvula crash aana report kidaikkadhu, aana appo varaikkum post aana entries LedgerPro-la irukkum.\n- `logs` and `data\\output` folders-a code create pannadhu. `data\\output` repo-laiyum illa. Install appo create pannanum-nu doc-la sollirukken.\n- Version enga-yum display aagala. Launcher update error-um innum silent-a dhaan irukku.\n- Rendu \"fix\" commits (b235445, ba4c1aa) oru comment line mattum dhaan add pannirukku. So doc-la ella fixes-um \"unconfirmed\" nu mark pannirukken.\n- `.env` file git-la track aagudhu. Adhai naan open pannala.\n- Posted rows-oda Result column `Missing` nu dhaan kaattudhu. Ops confuse aagalaam, so Statuses section-la clear-a explain pannirukken.\n\n**Checks done:**\n- Ella status strings-um Statuses section-la irukku.\n- file:line refs section 13-la mattum dhaan.\n- Log-la irundha customer name and transaction references doc-la illa.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel)? | Ops access: which of the bot's places ops can open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-packaged-desktop-screens-r3.md
+176
View File
@@ -0,0 +1,176 @@
# LedgerBot - ops handover
Version 3.2.0 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
LedgerBot compares the day's bank statement Excel with the LedgerPro desktop accounting app. Every reference in the statement that LedgerPro does not have is typed into LedgerPro and saved. References whose amount differs are only reported, never changed. It runs once per statement file, when an operator starts it. Input: one bank statement Excel file. Output: one Excel report, `data/output/Reconcile_<date>_<time>.xlsx`, with one row per statement reference. It drives one desktop app: **LedgerPro**, by finding its buttons on the screen from pictures and clicking them.
## 2. How it works
```
launcher.exe (what you start)
└─ checks for an update, then starts → LedgerBot.exe ──clicks and types into──> LedgerPro
│ (do not touch while it runs)
├─ reads → the bank statement Excel you gave it
├─ reads → LedgerPro's Excel export (data folder)
├─ writes → data/output/Reconcile_<date>_<time>.xlsx (only at the very end)
└─ writes → logs/ledger_<date>.log, and a screenshot when a button is not found
```
- **launcher.exe** looks for a newer LedgerBot version, installs it if found, starts LedgerBot.exe and closes itself.
- **LedgerBot.exe** is the bot. It has no window of its own; at most a black console window, which prints a one-line summary of counts at the end.
- **LedgerPro** is the accounting app. The bot opens it from the Desktop shortcut and moves the mouse and keyboard in it.
- **The report** in the output folder is the only record ops can open.
- **The log and screenshots** are in the bot's `logs` folder. Ops cannot open it; the dev team reads it.
**One run, step by step:**
1. The launcher checks for an update and starts the bot with the statement file.
2. The bot opens LedgerPro from the "LedgerPro" shortcut on the Desktop, then waits a fixed 15 seconds.
3. It clicks LedgerPro's "Export Excel" button, then waits a fixed 10 seconds for the export file.
4. It reads the export and the statement, and compares them reference by reference.
5. For each reference LedgerPro does not have, it clicks "New entry", types the reference, presses Tab, types the amount and clicks "Save".
6. After the **last** row it writes the report to the output folder. Nothing is written there before that.
## 3. Before a run
- [ ] The "LedgerPro" shortcut is on your Desktop or on the Public Desktop, named exactly **LedgerPro**. Otherwise the bot closes at once with no report (E01).
- [ ] Windows display scaling is **100%** and the screen is **1920 x 1080**. The button pictures were captured at that setting; at 125% the bot cannot find the buttons (E08, E09).
- [ ] Next to LedgerBot.exe there is a `logs` folder and a `data\output` folder. The bot does not create them. Without `logs` it closes at once (E02); without `data\output` it posts every entry and then fails to write the report (E12). Ask dev to confirm both at install.
- [ ] LedgerPro opens to the screen that shows the "Export Excel" button, without a login or a popup in front of it. The bot does not sign in and does not close popups.
- [ ] Statement Excel rules:
- The data is on the sheet that was **selected when the file was last saved**. The bot reads only that sheet.
- Row 1 is a header. From row 2: **column A = reference, column B = amount**. Other columns are ignored.
- Amount is a plain number. Bad: `₹ 410.00`, `410,00`, `n/a`. Good: `410.00`. One bad amount stops the whole run with no report (E06).
- Each reference appears **once**. If it appears twice, only the last row is used and the other is silently dropped (E07).
- Type references exactly as LedgerPro stores them, and as text. Bad: `1001` stored as a number when LedgerPro has `TX-1001`. A reference written differently counts as "not in LedgerPro" and is **posted again** (E07).
- A row with an empty column A is skipped.
- [ ] The update check has run and the version is the one dev announced (section 10).
## 4. During a run
- **Hands off the machine.** Don't move the mouse or type. The bot clicks wherever it sees a matching button on the screen and types into whatever has focus, so a moved window or a click from you sends its typing into the wrong place.
- Keep only LedgerPro open. Close other windows that could show a similar "New entry" or "Save" button, and don't let anything cover LedgerPro.
- Moving the mouse into a **corner of the screen** stops the bot immediately (a built-in safety switch). Don't do this by accident.
- **To stop the bot:** move the mouse into the top-left corner of the screen, or end **LedgerBot.exe** in Task Manager. Closing the launcher does not stop it; closing LedgerPro does not stop it either - it keeps clicking for each remaining row. Any way you stop it, **no report is written**, and entries already saved stay in LedgerPro (E13).
- A run looks stuck when the mouse has not moved for more than about 1 minute. There is no watchdog: the bot never gives up on its own except by failing a button search (20 seconds per button). If nothing moves for 2 minutes, stop it as above and raise a ticket.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | Output folder: a new `Reconcile_<date>_<time>.xlsx` with the run's start date and time. No new file = the run did not finish (E01-E06, E12, E13). The console window, if shown, prints a summary of counts at the very end. |
| Which items failed? | The report, **Result** column = `Error` (failed) or `Mismatch` (amounts differ). |
| Why did one item fail? | The report, **Note** column. For anything deeper, raise a ticket - the log and screenshots are in a folder ops cannot open. |
| Did the bot already retry? | It does not retry. Each button is searched for up to 20 seconds, once (section 8). |
| Which version is running? | Ops cannot see it today (section 10). Ask in the ticket. |
| What was actually posted? | **LedgerPro itself.** The report says `Posted` when the bot clicked Save, not when LedgerPro confirmed it (E10). |
## 6. Statuses
The report is plain Excel with no colours. Columns: Reference, Amount, Result, Note.
| Result - Note | Colour | Meaning | What ops does |
|---|---|---|---|
| `OK` - (empty) | none | LedgerPro has this reference with the same amount (within 0.01). | nothing |
| `Missing` - `Posted` | none | LedgerPro did not have it, **and the bot typed it in and clicked Save**. Despite the word "Missing", this is the row the bot posted. Save was not confirmed. | Spot-check a few in LedgerPro (E10). |
| `Mismatch` - `LedgerPro has <amount>` | none | LedgerPro has the reference with a different amount. **The bot changed nothing.** | Needs follow-up: correct it by hand in LedgerPro, or with the finance team (E11). |
| `Error` - `Button new_entry.png not found on screen` | none | The bot could not open a new entry. Not posted. | E08 |
| `Error` - `Button save_btn.png not found on screen` | none | The bot typed the entry but could not click Save. Probably not saved; the entry form may still be open. | **Check LedgerPro first** (E09). |
| `Missing` - `Not in LedgerPro - posting` | none | Never appears in a finished report: the bot always replaces it with `Posted` or turns the row into `Error`. | If you see it, raise a ticket. |
| (no report at all) | - | The run stopped before the end. | E01-E06, E12, E13 |
References that are in LedgerPro but not in the statement are **not listed anywhere**. The bot ignores them.
## 7. Exception catalogue
"Raise a ticket" always means a ticket in the RPA Support queue with the items in section 11.
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Bot closes at once, LedgerPro never opens, no report | It looks for a "LedgerPro" shortcut on your Desktop and the Public Desktop; if neither has one, it stops | Console window only (message "LedgerPro shortcut not found on the Desktop"); not in the log | System | Ops | Put a shortcut named exactly LedgerPro on the Desktop, run again | The shortcut is there and it still happens. Send: time, machine name. |
| E02 | Bot closes at once, nothing happens, no report | It cannot open its log file because the `logs` folder next to it is missing | Nowhere ops can see | System | Dev | Raise a ticket | Always. Send: time, machine name. |
| E03 | LedgerPro opens, nothing is clicked, bot ends, no report | It looks for the "Export Excel" button for 20 seconds; if it is not on screen, it saves a screenshot and stops | Screenshot in the logs folder; console message "Button export_excel.png not found on screen" | System | Ops, then Dev | Check: display scaling 100%, LedgerPro fully open with no popup or login in front, nothing covering it. Run again. | It happens again. Send: time, machine name, and say the screenshot is there. |
| E04 | LedgerPro export runs, then the bot ends, no report | It reads LedgerPro's export file from its data folder after a fixed 10 s; if the file is not there or not readable, it stops | Nowhere ops can see | System | Dev | Raise a ticket. Do not re-run repeatedly. | Always. Send: time, machine name. |
| E05 | Entries posted twice in LedgerPro; the report shows `Posted` for references LedgerPro already had | **Not identified.** If LedgerPro's export did not refresh the export file in 10 s, the bot compares against the previous run's file and treats today's existing entries as missing | Nowhere | System | Ops, then Dev | After every run, check that the number of `Posted` rows looks right for the day. If it is suspiciously high, check those references in LedgerPro for duplicates and remove them by hand. | Any duplicate found. Send: report file, time, machine name. |
| E06 | LedgerPro opens and exports, then the bot ends, no report, no entries posted | It cannot read the statement: wrong file, or an amount cell that is not a number | Console window only | Business | Ops | Check the statement against the rules in section 3 (plain numbers in column B, data on the selected sheet), fix, run again | The file follows the rules and it still happens. Send: time, machine name, statement attached to the ticket (not in chat). |
| E07 | A statement line missing from the report, or a reference posted although LedgerPro already had it under another spelling | **Not identified.** Duplicate references keep only the last row; references are compared as exact text | Nowhere | Business | Ops | Remove duplicates and write references exactly as in LedgerPro before the run. If a duplicate entry was posted, delete it by hand in LedgerPro. | - |
| E08 | Row `Error` - "Button new_entry.png not found on screen" | It looks for LedgerPro's "New entry" button for 20 seconds after each row; not found = Error, next row | Report, Result and Note columns; screenshot in logs folder | System | Ops | Not posted. Check display scaling (100%) and that LedgerPro is still open and in front. Run again - the next run posts it if it is still missing. | Several rows in one run, or again on the next run. Send: report file, time. |
| E09 | Row `Error` - "Button save_btn.png not found on screen" | It typed the reference and amount, then looked for "Save" for 20 seconds; not found = Error, next row | Report, Result and Note columns; screenshot in logs folder | System | **Ops** | **Check LedgerPro first.** The entry form may still be open with this row's data, and the following rows may have been typed into it. Close or cancel any open entry form, confirm whether this reference exists in LedgerPro, then run again. Expect the rows after it to also fail. | Rows after it show `Posted` but are not in LedgerPro. Send: report file, time. |
| E10 | Row `Missing` - `Posted`, but the entry is not in LedgerPro (or has a wrong amount) | **Not identified.** The bot writes `Posted` right after clicking Save; it does not wait for LedgerPro to confirm | Report says `Posted` | System | Ops | Spot-check posted rows in LedgerPro. A missing one is posted by the next run; a wrong amount must be fixed by hand. | More than one per run. Send: report file, references concerned (in the ticket, not chat). |
| E11 | Row `Mismatch` - "LedgerPro has <amount>" | The reference exists in LedgerPro but the amounts differ by more than 0.01 | Report, Result and Note columns | Business | Ops | Decide which amount is right with finance and correct it by hand. The bot never changes existing entries and will report it again until fixed. | - |
| E12 | The bot clicks through every row, then ends with **no report** | It writes the report to `data\output` only at the end; if the folder is missing, it fails there | Nowhere ops can see | System | Dev | Entries **were** posted. Do not re-run until dev creates the folder. Check today's entries in LedgerPro. | Always. Send: time, machine name. |
| E13 | Run stopped midway (you stopped it, the PC locked, or it crashed); no report | Any problem other than a row's button search stops the whole run | Nowhere ops can see | System | Ops | Entries saved before the stop stay in LedgerPro. Check the last entries in LedgerPro, close any open entry form, then run again with the same statement. | It stops again at the same point. Send: time, machine name. |
| E14 | A new version was announced but the bot behaves as before | The launcher hides every update error (no internet, untrusted certificate) and silently runs the old version | Nowhere | System | Dev | Raise a ticket; ops cannot check the version (section 10) | Always. Send: machine name, date of the announced update. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| A button is not on screen yet | Looks again every second | 20 seconds per button | "New entry" or "Save": row = Error, next row. "Export Excel": whole run stops, no report. |
| LedgerPro slow to open | Waits a fixed time, does not check | 15 seconds | Continues anyway; usually ends in E03 |
| Export slow | Waits a fixed time, does not check | 10 seconds | Continues anyway; ends in E04 or silently in E05 |
| Anything else | Nothing. No retries, no restarts, no resume. | - | Run stops, no report |
Ops never has to wait for a hidden retry: if the mouse has stopped for 2 minutes, nothing more will happen.
## 9. Known incidents
From the repository history. Each fix below is recorded in the history, but the change kept in the code is only a note, so **treat every fix as unconfirmed** until dev confirms it.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 3.2.0 | The bot never found its buttons (assets missing in the built exe) | The packaging file did not include the button pictures | Raise a ticket if every run ends at E03 straight after an update | No |
| Before 3.2.0 | Clicks missed "Save" on laptops set to 125% display scaling | Button pictures captured at 100% | Keep display scaling at **100%** (section 3) | Yes |
| Before 3.2.0 | Two laptops never received updates | The launcher hid the certificate error from the update server | Still possible today (E14). Raise a ticket when an update does not arrive. | Partly |
| 2026-09-05 | One row `Error` "Button save_btn.png not found on screen" in the same run as OK and Mismatch rows | Save button not found within 20 s | Follow E09 | Yes |
## 10. Releases and version check
- How an update arrives: every time you start **launcher.exe**, it asks the update server for the latest version and, if different, downloads and installs it over the bot's folder before starting the bot.
- How to confirm it applied: **ops cannot today.** The bot shows its version nowhere (it has no window title, and the report does not include it). Ask dev in the ticket.
- If an update did not arrive: raise a ticket (E14). Don't copy files over the bot's folder yourself.
- Start the bot only through **launcher.exe**, never LedgerBot.exe directly, so updates are applied.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue. Send: the exception ID from section 7, the date and time of the run, the machine name, the report file from the output folder if there is one, and a screenshot of LedgerPro as it was left. Say which statement file was used; attach it to the ticket only, never paste it into chat. Dev collects the log and screenshots from the bot's `logs` folder.
Never send: passwords, the settings file, bank or customer details in chat.
## 12. Failure-class check
- F1 covered in section 3 (shortcut, display scaling, `logs` and `data\output` folders, LedgerPro screen) and E01, E02, E12; packaging gap in section 13.
- F2 covered in section 4 (hands off, only LedgerPro open) and section 13 #9.
- F3 covered in E03, E04, E08, E09, section 8 (fixed 15 s / 10 s waits, 20 s button search), screen rule in section 3.
- F4 covered in section 3 input rules, E06, E07.
- F5 covered in E05 (stale export makes already-posted items look missing) and E10; section 13 #1, #2.
- F6 covered in E09, E10 (check LedgerPro before re-running).
- F7 covered in section 6 (`Missing` actually means posted; `Mismatch` needs follow-up; LedgerPro-only entries never reported) and section 13 #7, #16.
- F8 not applicable: the bot has no auto-rerun, no restart and no watchdog; it runs each row once and ends.
- F9 covered in section 4 (corner of the screen or Task Manager; closing the launcher or LedgerPro does not stop it).
- F10 not applicable: there is no batch, queue or lock; each run stands alone and one failed row never blocks another run. (A failed Save can make the following rows fail in the same run - covered in E09.)
- F11 covered in section 5 (report only) and section 13 #3, #5, #6.
- F12 covered in E14, section 10, section 13 #10-#13.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | `Posted` is written right after clicking Save, with no check that LedgerPro saved it | Report shows posted when it may not be | `app.py:65-70`, `app.py:90` |
| 2 | Bot reads `data/ledger_export.xlsx` without checking it was refreshed this run (fixed `sleep(10)`, no timestamp check, old file never deleted) | Stale export → existing entries posted twice, silently | `app.py:77-79` |
| 3 | Report is written only at the end; any unhandled error loses the whole report although entries were posted | Ops cannot tell what was posted | `app.py:82-96` |
| 4 | `logs/` and `data/output/` are not created by the code | Instant crash (logs) or crash after posting everything (output) | `app.py:24`, `app.py:96` |
| 5 | Crash errors go to the console only, not to the log (only `RuntimeError` inside posting is caught) | Cause lost once the console closes | `app.py:86-95`, `app.py:100-101` |
| 6 | Log and not-found screenshots are in `logs/`, which ops cannot open; report Note has only the picture file name | Ops cannot see causes themselves | `app.py:24`, `app.py:45`, `app.py:92` |
| 7 | Posted rows keep Result `Missing`; the "Not in LedgerPro - posting" note is dead text | Confusing status for ops | `app.py:58`, `app.py:87-90` |
| 8 | Fixed `sleep(15)` / `sleep(10)` instead of waiting for LedgerPro / the export file | Slow machine → E03/E04/E05 | `app.py:76`, `app.py:78` |
| 9 | No window focus or check that LedgerPro is in front; `pyautogui.write` types into whatever has focus | Typing into the wrong window | `app.py:37-46`, `app.py:65-70` |
| 10 | Launcher still swallows every update error (the "fix" commit only added a comment); version compare is `!=` so an older server build would be installed; download has no timeout or integrity check | Machines silently stay on, or roll back to, a different version | `launcher.py:16-20`, `launcher.py:25-29` |
| 11 | `VERSION` comment says it is in the window title, but nothing displays it; not in the report either | Ops cannot confirm an update | `app.py:15`, `app.py:96` |
| 12 | `ledgerbot.spec` is entirely commented out; asset inclusion cannot be verified from the repo | Repeat of the "assets missing in exe" incident | `ledgerbot.spec:2` |
| 13 | Fix commits b235445 and ba4c1aa changed only a comment line; recaptured 100% button images are not in that commit | Fixes listed in history may not be in the build | `app.py:102`, `launcher.py:35` |
| 14 | `.env` is tracked in git (and the code does not read it) | Possible secret exposure | `.env` |
| 15 | Statement reads the active sheet only; duplicate references collapse silently; a non-numeric amount crashes the run; references compared as raw `str()` (number vs text) | Silent skips or duplicate posts | `app.py:52`, `app.py:80-81` |
| 16 | LedgerPro entries not in the statement are never reported | Ops cannot see extras in the ledger | `app.py:55-62` |
| 17 | After a Save-not-found error the entry form is left open and the next row is typed into it | One failure cascades | `app.py:88-92` |
| 18 | Mouse in a screen corner (pyautogui fail-safe) aborts the run with no report | Accidental stop loses the report | `app.py:12` (default `FAILSAFE`) |
tests/evals/results/handoff/h-web-portal-full-r1.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-web-portal/full",
"rep": 1,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"no-history case says so": true,
"misleading status flagged": true,
"session re-login limit": true,
"screenshots folder": true,
"daily schedule": true
},
"cost_usd": 1.2729888,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-full-0_3xhfnx\\project",
"reply": "HANDOVER.md ready \u2014 `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-full-0_3xhfnx\\project\\HANDOVER.md`. Bot-ah run pannala, code-um maathala. Scan, doc writing, final check \u2014 ellaame code, config, log padichu thaan.\n\n| Item | Count |\n|---|---|\n| Sections | 13 (template order same) |\n| Exceptions catalogued | 11 (E01\u2013E11). Ella Dev rows-layum \"enna anuppanum\" irukku |\n| Failure classes | 11 covered, 1 not applicable (F10: batch/queue gate illa) |\n| Fix before handover | 18 items |\n\n**Fix before handover \u2014 mukkiyamaanavai:**\n- Portal maintenance vandha `Downloaded` nu green-la kaattudhu, aana file download aagala.\n- Folder-la endha pudhu CSV vandhaalum success-nu edukkudhu. Late-aa vandha file next provider-ku count aagidum.\n- Re-login fail aanaa run crash aagudhu. Meedhi providers report-la silent-aa missing.\n- Crash, run start/end log-la varadhu. Log-la remark-um illa.\n- `Login Failed` run-um Task Scheduler-la success-nu kaattum.\n- Stop button, watchdog, version number edhuvum illa.\n- `config.yaml`-la use aagaadha plaintext token irukku. Adha rotate panni remove pannanum.\n\n**Ops access:** \"Where to look\" table-la report file, `data/screens/`, `data/downloads/`, `logs/bot.log` mattum thaan point pannirukken. Task Scheduler-um database-um ops-ku access illa, adhanaala andha gaps \"Fix before handover\"-la irukku.\n\n**Also noticed:** `logs/bot.log`-la oru run (2026-09-02) mattum thaan irukku. Adhanaala \"Known incidents\"-la oru incident thaan. Dev team kitta pazhaya logs, reports, Task Scheduler history kekkanum.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions mean where the code doesn't make it clear?",
"stderr": ""
}
tests/evals/results/handoff/h-web-portal-full-r1.md
+192
View File
@@ -0,0 +1,192 @@
# Payer portal downloader - ops handover
Version: not versioned (the project has no version string) · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every day at 07:00 the bot signs in to the payer web portal and downloads each provider's remittance report as a CSV file. It reads the input workbook `data/input/providers.xlsx`. The **Login** sheet holds one portal username and password. The **Providers** sheet has one provider per row: NPI, name, and an active flag. For each active provider it opens that provider's remittance page and clicks the CSV export. The CSVs go to `data/downloads/`. At the end it writes one report, `data/output/Download_Report_<YYYYMMDD>.xlsx`, with one row per provider: Provider, NPI, Status, Remarks. Windows Task Scheduler starts it. It drives one website, the payer portal, in its own Chrome window.
## 2. How it works
```
Windows Task Scheduler (07:00 daily)
└─ starts → Downloader script (no window of its own)
│ ──drives──> Chrome window → payer web portal
│ (do not touch this window while it runs)
├─ reads → data/input/providers.xlsx
├─ saves → data/downloads/*.csv (Chrome saves the files)
├─ saves → data/screens/<NPI>.png (only when a page did not load)
├─ writes → data/output/Download_Report_<YYYYMMDD>.xlsx (at the end)
└─ writes → logs/bot.log (one line per provider)
```
- **Task Scheduler** starts the run. The bot has no screen or Stop button of its own.
- **Downloader script**: the bot. It works through the provider list in order, one at a time.
- **Chrome window**: opened and closed by the bot. Everything it does on the portal happens in this window.
- **data/input/providers.xlsx**: the input. Whoever maintains it controls which providers are downloaded.
- **data/downloads/**: the CSVs. The portal chooses the file names. The bot does **not** rename them, so a file name does not tell you which provider it belongs to.
- **data/screens/**: a screenshot of the portal page, saved only when a provider's report page did not load.
- **Download report**: the main place to check results. It is written only when the run ends, whether it finished or crashed.
- **logs/bot.log**: one line per provider with its status (not the remark), plus a warning line for each re-login.
**One run, step by step:**
1. Read the input workbook: the login from the Login sheet, the provider list from the Providers sheet.
2. Open Chrome and sign in to the portal. If the portal's home page doesn't appear within 30 s, every provider gets **Login Failed** and the run ends.
3. For each provider:
1. If it is marked inactive, write Skipped.
2. Otherwise open its remittance page.
3. If the portal has logged the bot out, sign in again and retry. It signs in again at most 2 times per provider.
4. Wait up to 30 s for the report grid.
5. If the portal says there are no remittances, write No Data.
6. Otherwise click Export CSV and wait up to 60 s for a new CSV in `data/downloads/`.
4. Close Chrome and save the download report.
## 3. Before a run
- [ ] Google Chrome is installed on the bot machine and can open the portal.
- [ ] These folders exist. The bot does **not** create them:
- `logs/`: without it, the bot stops before doing anything and writes no log.
- `data/output/`: without it, the report is not saved, even after a good run.
- `data/downloads/`
- `data/screens/`: without it, failure screenshots are silently not saved.
- [ ] `config.yaml` is next to the bot.
- [ ] `data/input/providers.xlsx` exists and has both sheets, spelled exactly **Login** and **Providers**.
- [ ] **Login** sheet: the username is in cell A2 and the password in B2. Check that this login works by signing in to the portal by hand, especially after a password change.
- [ ] **Providers** sheet, from row 2 down:
- **One provider per row.** Column A = NPI, column B = provider name, column C = active flag.
- Rows with an empty column A are skipped without any message. Don't leave the NPI blank.
- **To make a provider inactive, type a capital `N` in column C**, with nothing else in the cell.
- Good: `N`
- Bad: `n`, `No`, ` N` (leading space). The bot treats these as **active** and downloads that provider anyway.
- NPI as a whole number or text.
- Good: `1234567890`
- Bad: `1234567890.0`. Decimal formatting puts the wrong NPI into the portal address.
- [ ] If today's report already exists in `data/output/`, **copy it somewhere else before a second run today**. A second run on the same day overwrites it.
## 4. During a run
- **Do not click, type in, close or navigate the Chrome window the bot opened.** Any of these ends the run straight away, and the providers after that point are left out of the report (E08).
- Don't save or download anything into `data/downloads/` while the bot runs. **Any** new CSV in that folder counts as success for whichever provider the bot is on (E10).
- Other programs and other Chrome windows on the machine are fine. The bot does not need the mouse or keyboard.
- **To stop the bot:** there is no Stop button. The only clean way is to end the task in Windows Task Scheduler, and ops cannot open it. **Ask in #rpa-support.** Closing the bot's Chrome window also stops it, but it counts as a crash: the report keeps only the providers finished so far.
- **A run looks stuck when** `logs/bot.log` has had no new line for **10 minutes**. One provider normally takes under 2 minutes. The worst case, with all re-logins and waits, is about 4 minutes. **The bot has no stuck-detection or overall time limit of its own**, so it will not give up by itself. After 10 minutes with no new line, escalate (E09).
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `data/output/` should have `Download_Report_<today's date>.xlsx`, with **one row per provider in the input** (inactive ones included). No file today means the run never started or failed before saving (E09). Fewer rows than the input means it crashed partway (E08). |
| Which providers failed? | The download report, **Status** column. Filter for `Error` and `Login Failed`. **Also filter `Downloaded` and read Remarks**: a maintenance row is green but has no file (E05). |
| Why did one provider fail? | The download report, **Remarks** column on that row. If the remark is "Report grid did not load", open `data/screens/<NPI>.png` to see what the portal showed. |
| Did the bot already retry? | `logs/bot.log`: search for "session expired, re-login" just before that provider's line. That is the only retry the bot does. Grid, download and first-login timeouts are **never** retried. |
| When did it run, and how far did it get? | `logs/bot.log`: each line has a date and time. The last line of the day is the last provider finished. |
| Is the CSV really there? | `data/downloads/`, sorted by date modified. The number of new CSVs from today should match the number of `Downloaded - Report saved` rows. |
| Which version is running? | There is no version to check. See section 10. |
## 6. Statuses
The bot writes these values in the **Status** column of the download report. The colour is the fill of the Status cell.
| Status | Remarks | Colour | Meaning | What ops does |
|---|---|---|---|---|
| Downloaded | Report saved | green | A new CSV appeared in `data/downloads/` after the export click. The bot does not check that the file is really this provider's. | Nothing. If the CSV count doesn't match, see E10. |
| Downloaded | portal maintenance - no file | **green, but needs follow-up** | **No file was downloaded.** The portal showed a maintenance banner. | Download it again once maintenance is over (E05). |
| No Data | No remittances in the date range | green | The portal says this provider has no remittances. No file is expected. | Nothing. |
| Skipped - inactive provider | Marked N in the input | green | Column C of the input is `N`. Not attempted. | Nothing, unless the provider should be active. Then fix column C. |
| Login Failed | Check the portal username/password | **no colour (white)** | The first sign-in failed, so **no provider was attempted.** Every row in the report has this status. | See E01. |
| Error | Report grid did not load | red | The report page didn't load within 30 s. A screenshot was saved. | See E03. |
| Error | Download did not finish | red | Export was clicked, but no new CSV arrived within 60 s. | See E04. |
| Error | Session expired | red | The portal logged the bot out 3 times on this provider. | See E02. |
| *(row missing)* | - | - | The provider is in the input but not in the report. The run crashed before reaching it. | See E08. |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | **Every** row is `Login Failed` - "Check the portal username/password" (cells not coloured) | After signing in, it waits 30 s for the portal home page. If the page doesn't appear, it gives up. A wrong password and a slow portal look the same. | Download report, all rows. Nothing in `logs/bot.log` for that run. | System (or wrong login in the input) | Ops | 1. Sign in to the portal by hand with the Login sheet username and password. 2. If that fails, correct the login in cells A2/B2. 3. If it works by hand, the portal was probably slow. Ask in #rpa-support for a re-run. | Manual login works but the next run is all Login Failed again. Send: the report file, the run date and time. |
| E02 | One row: `Error` - "Session expired" | When it opens a provider's page and lands on the login page instead, it signs in again and retries, up to 3 attempts in total. | Download report. The log has "session expired, re-login 1" and "2" lines before that provider's line. | System | Bot, then Ops | The bot already retried twice. Download this provider's report by hand from the portal. | It happens on several providers in one run, or on two days running. Send: the report file, `logs/bot.log`, the run date. |
| E03 | One row: `Error` - "Report grid did not load" | It waits 30 s for the report grid on the provider's page. If the grid doesn't appear, it takes a screenshot and moves on. No retry. | Download report. Screenshot `data/screens/<NPI>.png` (replaced on the next failure for the same provider). | System | Ops | 1. Open the screenshot. 2. Wrong NPI or "provider not found" in the portal: check column A of the input. 3. Blank or slow page: download this provider by hand from the portal. | Many providers fail in one run, or the screenshot shows a new portal layout. Send: the report file, the screenshots, the run date and time. |
| E04 | One row: `Error` - "Download did not finish" | After clicking Export CSV, it checks `data/downloads/` every second for 60 s. If no new CSV appears, it moves on. No retry. | Download report only. No screenshot. | System | Ops | 1. Check `data/downloads/` for a CSV that arrived late, close to that time. **A late file may have been counted as the next provider's download** (see E10). 2. Download this provider by hand from the portal. | It happens to several providers in one run. Send: the report file, a listing of `data/downloads/` with times, the run date. |
| E05 | Green row: `Downloaded` - "**portal maintenance - no file**" | The portal page shows a maintenance banner. The bot doesn't wait or retry. | Download report, Remarks column. | System (portal side) | Ops | **Treat it as not downloaded.** Download it by hand once maintenance is over. | Every provider shows it on several days in a row. |
| E06 | Row: `No Data` - "No remittances in the date range" | The portal shows its "no results" message on the provider's page. | Download report. | Business | Ops (nothing to do) | Nothing. Confirm in the portal only if remittances were expected. | Never, unless the portal shows remittances the bot called No Data. |
| E07 | Row: `Skipped - inactive provider` - "Marked N in the input" / or an inactive provider was downloaded anyway | Column C is exactly `N`. Anything else (`n`, `No`, ` N`) counts as active. | Download report. | Business | Ops | To make a provider inactive, type a capital `N` alone in column C. To reactivate it, clear the cell. | - |
| E08 | **Report has fewer rows than the input.** The last providers are missing. No error row explains it. | Not detected. Something the bot doesn't handle stopped the run mid-list. Possible causes: a re-login failed, a portal button was missing, or the Chrome window was closed or touched. | Partial download report. The last line in `logs/bot.log` is the last provider finished. The error itself is **not** in the log. | System | Dev | Note the last provider in the log. Download the missing providers by hand if they are needed today. | Always. Send: the report file, `logs/bot.log`, the run date and time, the last provider in the log, and whether anyone touched the bot's Chrome window. |
| E09 | **No report for today** in `data/output/`, or the log has no lines today, or no new log line for 10+ minutes | Not detected. The run never started, failed before saving, or is hanging. Possible causes: a missing folder (section 3), a missing input file, a Chrome problem, the portal hanging. | Nothing ops can see. Only Task Scheduler (dev access) has the result. | System | Dev | 1. Check the section 3 list: folders, input file name, sheet names. 2. Post in #rpa-support. | Always. Send: the date, the time you checked, the last line of `logs/bot.log`, and which section 3 items you checked. |
| E10 | More `Downloaded - Report saved` rows than new CSVs, or a CSV that looks like the wrong provider's | The bot counts **any** new CSV in `data/downloads/` as this provider's file. A late file from the previous provider, or a file someone else saved there, counts. | Download report vs the contents of `data/downloads/`. | System | Ops, then Dev | 1. Count today's new CSVs. 2. Open any suspicious one and check its provider. 3. Download the doubtful providers by hand. | The counts don't match on more than one day. Send: the report file, a listing of `data/downloads/` with times. |
| E11 | This morning's report is gone, replaced by a later one | A second run on the same day writes a report with the same name. | `data/output/` - only one report per date. | Business (process) | Ops | Before any second run on a day, copy the existing report somewhere else. | - |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Portal logged the bot out (detected when opening a provider's page) | Signs in again and retries the same provider | 2 re-logins, so 3 attempts per provider | Row = `Error` - "Session expired", next provider. **If a re-login itself fails, the run stops** (E08). |
| Report grid slow | Waits | 30 s, **no retry** | Row = `Error` - "Report grid did not load", screenshot, next provider |
| CSV download slow | Waits, checking every second | 60 s, **no retry** | Row = `Error` - "Download did not finish", next provider |
| First sign-in slow or failed | Waits | 30 s, **no retry** | Every row `Login Failed`, run ends |
| Run hangs or stuck | **Nothing.** No watchdog, no restart. | None | Runs until someone ends it in Task Scheduler (dev). Escalate after 10 min with no new log line. |
| Run crashes | Closes Chrome and saves a partial report | - | Missing providers are not marked. No automatic re-run. A re-run starts from the first provider again and re-downloads everything. |
## 9. Known incidents
The log on the bot machine covers only one run (2026-09-02). Older history was not available.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-09-02 (unversioned) | One provider red `Error` in the day's report. In the log, a "session expired, re-login 1" warning, then that provider as Error about 80 s later. | The portal logged the bot out about a minute into the run. The re-login worked, but the retry then hit either the 30 s grid wait or the 60 s download wait. The log doesn't say which, because remarks are not logged. | Read that row's Remarks. If it is "Report grid did not load", check `data/screens/`. Download the provider by hand (E03 / E04). | Yes |
## 10. Releases and version check
- **How an update arrives:** there is no update mechanism and no version number. The dev team changes the bot's files on the bot machine directly.
- **How to confirm it applied:** ops cannot check this. Ask in #rpa-support for confirmation, with the date the change went live.
- **If behaviour did not change after a promised fix:** post in #rpa-support with the run date and the report file.
- There is only one bot machine. No shared resource is rewritten on start, so version drift between machines does not apply.
## 11. Escalating to dev
Post in Teams **#rpa-support**. Send:
- the exception ID from section 7 (E01-E11)
- the run date and the time you noticed
- the download report `data/output/Download_Report_<YYYYMMDD>.xlsx`
- `logs/bot.log`
- any screenshot from `data/screens/`
- the last provider line in the log, for a crash
**Never send:** the portal password, the input workbook (it holds the login), `config.yaml`, or provider or patient data pasted into chat.
## 12. Failure-class check
- F1 (missing prerequisite) covered in section 3 (Chrome, folders, config, input sheets) and E09.
- F2 (wrong window or focus) covered in section 4: don't touch the bot's Chrome window, don't save into `data/downloads/`. The bot uses its own browser session, not window titles or focus.
- F3 (target slower or different) covered in E01, E03 and E04, with limits in section 8. Portal layout changes lead to E08.
- F4 (input read differently) covered in section 3 input rules (the `N` flag, NPI format, blank rows, sheet names) and E07.
- F5 (marked done when nothing was done) covered in E05 and E10, and Fix before handover items 1 and 2.
- F6 (action happened but not confirmed) covered in E04: a late CSV can be credited to the next provider, so check `data/downloads/` before re-downloading. Fix item 2.
- F7 (business exception shown as success) covered in section 6 (maintenance row green, needs follow-up), E05 and Fix item 1.
- F8 (endless re-runs or watchdog): no auto-rerun loop exists. The re-login loop is capped at 2. The missing watchdog is covered in section 4, section 8 and Fix item 11.
- F9 (stopping the wrong way) covered in section 4 (no Stop button; ending the task needs dev) and Fix item 11.
- F10 not applicable: there is no batch or queue gate. Each daily run is independent, and a failed provider blocks nothing. The nearest risk is a crash dropping the remaining providers, covered in E08.
- F11 (cause hidden from ops) covered in Fix items 4, 5 and 6: crash errors and remarks are not in the log, and the run result is only in Task Scheduler. Section 5 points only at the report, `data/screens/`, `data/downloads/` and `logs/bot.log`.
- F12 (update didn't arrive) covered in section 10 and Fix item 12: no version string and no release process.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Portal maintenance returns status `Downloaded` (coloured green) with no file. | Ops reads green as done, and the report is never downloaded. | `bot.py:54-55`, colour map `bot.py:27` |
| 2 | Any new `*.csv` in the shared downloads folder counts as success. Files are never renamed to the provider. | A late file is credited to the wrong provider. A CSV can't be tied to a provider. | `bot.py:63-68` |
| 3 | A failed re-login inside the provider loop raises an exception that nothing catches. Missing `username`/`password`/`signin`/`export-csv` elements and any browser error do the same. | The run crashes. The remaining providers are silently absent from the report, with no "not attempted" row. | `bot.py:102`, `bot.py:41-43`, `bot.py:64`, loop `bot.py:92-107` |
| 4 | No traceback, run-start or run-end line is ever logged. | A crash or a run that never started looks like a quiet day. Ops can't tell a finished run from a dead one. | `bot.py:30-32`, `bot.py:85-110` |
| 5 | The log line has the status but not the remark. | The log alone can't say which of three `Error` reasons happened (seen on 2026-09-02). | `bot.py:105` |
| 6 | A `Login Failed` run still exits normally. Task Scheduler shows success, and no alert is sent. | A whole day of downloads can be lost without anyone noticing. | `bot.py:88-91` |
| 7 | `Login Failed` rows get no colour. | They don't stand out in the report. | `bot.py:90` (no `fill`), `bot.py:107` |
| 8 | The first login has no retry. A 30 s timeout means a slow portal and a wrong password give the same message. | Ops changes a correct password, or re-runs a wrong one. | `bot.py:44-47`, `bot.py:86-91` |
| 9 | Required folders are never created. A missing `logs/` crashes at import. A missing `data/output/` loses the report. A missing `data/screens/` silently drops screenshots. | Silent or total failure on a fresh machine. | `bot.py:20-21`, `bot.py:30`, `bot.py:59`, `bot.py:110` |
| 10 | The report name has the date only, so a same-day re-run overwrites it. There is no resume or single-provider re-run, so a re-run re-downloads every provider (duplicate CSVs). Screenshots are overwritten per NPI. | Evidence is lost. Ops can't re-do just the failed providers. | `bot.py:110`, `bot.py:92`, `bot.py:59` |
| 11 | No Stop mechanism, no overall run timeout, no stuck-detection. | Ops can't stop a hung run and has to wait on dev. | `bot.py:73-110` |
| 12 | No version string and no documented release process. | Ops can't confirm a fix is live. | `README.md`, `bot.py` (none) |
| 13 | Input not validated. The active flag is an exact `!= "N"` test. The NPI becomes `str()` of the cell, so a float cell gives `"123.0"`. Sheets and credentials are not checked. | A provider is downloaded when it should be skipped, or the wrong NPI is used, with no message. | `bot.py:76-78` |
| 14 | Session expiry is only detected when a page is opened. An expiry during the grid or download wait shows up as `Report grid did not load` or `Download did not finish` instead. | Misleading cause on the report. | `bot.py:52-53` |
| 15 | If `driver.quit()` raises in `finally`, the report is never saved. | The whole run's results are lost. | `bot.py:108-110` |
| 16 | Security: `config.yaml` holds a plaintext token that the code never reads, so **rotate and remove it**. The portal password is stored in plain cells of the input workbook. Provider names are written to every log line. | Exposure risk. Ops can't safely share the input or the log. | `config.yaml:2`, `bot.py:76`, `bot.py:105` |
| 17 | Undocumented marker file `BOT_WAS_RUN` is written to the project root on every run. | Confusing, with no stated purpose. | `bot.py:74` |
| 18 | `logs/bot.log` holds only one run (2026-09-02), although the bot appends daily. History was cleared or not provided. | Past incidents can't be reviewed. Please supply the older logs, past reports and Task Scheduler history. | `logs/bot.log`, `bot.py:30` |
tests/evals/results/handoff/h-web-portal-full-r2.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-web-portal/full",
"rep": 2,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"no-history case says so": true,
"misleading status flagged": true,
"session re-login limit": true,
"screenshots folder": true,
"daily schedule": true
},
"cost_usd": 1.090367,
"subagents": 1,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-full-6wiwxpom\\project",
"reply": "I wrote `HANDOVER.md` in the repo root. No code was changed and the bot was never run. It's based on reading the code, config and log only, so nothing in it has been tested against a live run.\n\n| Item | Count |\n|---|---|\n| Sections | 13 (all from the template) |\n| Exceptions catalogued | 10 (E01\u2013E10) |\n| Statuses documented | all 8 that the code writes |\n| Failure classes | all 12 covered, none marked not applicable |\n| Fix before handover | 19 items |\n\n**Fix before handover \u2014 the important ones:**\n- **Maintenance shown as success:** the row says \"Downloaded\" in green, but no file was downloaded (#1).\n- **Any new CSV counts as success:** the file isn't checked against the provider (#2).\n- **One crash drops the rest:** if a re-login fails or the Export button is missing, the run stops and every later provider is missing from the report (#4, #5).\n- **No reason in the log:** the log never says why a provider failed, and the cause of a crash is lost (#6, #16).\n- **No report at all on early failures:** if the input file is open or a folder is missing, today's report is never written (#7, #8).\n- **Plain-text secrets:** the portal password sits in the input Excel, and an unused `api_token` sits in `config.yaml`. Remove the token and get it rotated (#14, #15).\n- The rest: no watchdog, no retries, a same-day re-run overwrites the report, and no version number.\n\n**Gaps you should know about:**\n- **No history:** there's no git history or changelog, so I skipped the history scan. Known incidents has only one entry, from the single run in `logs/bot.log` (2026-09-02).\n- **Folders don't exist:** the `data/*` folders aren't in the repo and the code never creates them, so the doc lists them as a pre-run check.\n- **Task Scheduler:** your access list doesn't include it, so for stopping or re-running the bot the doc tells ops to ask in #rpa-support if they don't have rights.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-web-portal-full-r2.md
+204
View File
@@ -0,0 +1,204 @@
# Payer portal downloader - ops handover
Version: not set (the bot has no version number, see section 10) · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every day at 07:00 Windows Task Scheduler starts the bot. It signs in to the payer web portal and downloads one remittance report (a CSV file) for each active provider in the input workbook `data/input/providers.xlsx`. It takes two sheets from that workbook: **Login** (the portal username and password) and **Providers** (NPI, name, active flag). It produces:
- one CSV per provider in `data/downloads/`;
- a result workbook, `data/output/Download_Report_<YYYYMMDD>.xlsx`, with Provider, NPI, Status and Remarks for each provider;
- a screenshot in `data/screens/` when a provider's report page does not load.
It drives one app: a **Chrome** window that it opens itself.
## 2. How it works
```
Windows Task Scheduler (07:00 daily)
└─ starts → the bot (one background program, no window of its own)
│
├─ reads → data/input/providers.xlsx (Login + Providers sheets)
├─ drives → its own Chrome window ──> payer web portal
│ (shows "controlled by automated test software"; do not touch it)
├─ saves → data/downloads/*.csv (one per provider)
├─ saves → data/screens/<NPI>.png (only when a page fails to load)
├─ writes → logs/bot.log (one line per provider)
└─ writes → data/output/Download_Report_<YYYYMMDD>.xlsx (at the very end)
```
- **Task Scheduler** starts the bot every morning. Nothing else starts it.
- **The bot** is a background program. It has no screen, no Stop button and no progress display.
- **Chrome** is opened by the bot, and the bot controls it. Every portal page you see in it during a run is the bot's work.
- **The input workbook** is the only list of providers. A provider that is not in it is never downloaded.
- **The downloads folder** collects every CSV. Files keep the portal's own name and are never renamed or cleaned up.
- **The log** gets one line per provider (time, NPI, name, status). It does not give the reason.
- **The result workbook** is written only when the run ends. While the bot is running, today's file does not exist yet.
**One run, step by step:**
1. Open the input workbook and read the login and the provider list.
2. Open Chrome and sign in to the portal. The bot waits up to 30 s for the dashboard.
3. For each provider, in input order:
- If the active flag is `N`, record **Skipped** and move on.
- Open that provider's remittance page.
- If the portal has logged the bot out, sign in again and retry (up to 3 times).
- If a maintenance banner shows, record it (see E05) and move on.
- Wait up to 30 s for the report grid.
- If the portal says there are no results, record **No Data**.
- Otherwise click Export and wait up to 60 s for a new CSV in the downloads folder.
4. Close Chrome and save today's result workbook.
## 3. Before a run
- [ ] Google Chrome is installed and opens normally on this machine. The bot cannot start without it, and then no result workbook is written (E08).
- [ ] These folders exist: `data/input`, `data/output`, `data/downloads`, `data/screens` and `logs`. The bot does not create them.
- Without `logs`, the bot stops before it starts and records nothing.
- Without `data/output`, the result workbook is lost (E08).
- Without `data/screens`, screenshots are silently not saved.
- [ ] `data/input/providers.xlsx` is **closed** (nobody has it open in Excel) and still has the sheets named exactly **Login** and **Providers**.
- [ ] **Today's result workbook is not open in Excel** from an earlier run. If it is open, the bot cannot save, and the result is lost (E08).
- [ ] **Login sheet:** username in A2, password in B2. When the portal password changes, update B2 before the next 07:00 run. Otherwise every row shows "Login Failed" (E01).
- [ ] **Providers sheet:** one provider per row, starting at row 2.
- Column A is the NPI, column B the name, column C the active flag.
- A row with column A empty is ignored.
- **Only a capital `N` in column C skips a provider.**
- Good: `N`. The provider is skipped.
- Bad: `n`, `No`, `Inactive`. The provider is treated as **active** and downloaded.
- NPI: digits only, no spaces or dashes.
- Good: `1234567890`.
- Bad: `1234 567 890`. The portal finds no provider, and you get a No Data or grid error.
- [ ] The NPI must already be set up in the portal for this login. The bot does not check this; an unknown NPI shows up as E03 or E06.
## 4. During a run
- **Do not close, click in or type in the bot's Chrome window.** It shows a "Chrome is being controlled by automated test software" bar. Your own separate Chrome windows are fine.
- **Do not put, move or delete files in `data/downloads/` during a run.** The bot counts *any* new CSV that appears there as "the provider's file" (see E04 and section 6).
- **Do not open `data/input/providers.xlsx` in Excel around 07:00**, or today's result workbook while a run is going.
- **Do not log in to the portal with the same username during a run.** The portal may end the bot's session, which leads to E02.
- **To stop the bot:** in Windows Task Scheduler, right-click the bot's task and choose **End**. Then close any Chrome window still showing the "automated test software" bar.
- Stopping it this way means **no result workbook is saved** for that run.
- Closing the Chrome window alone does not stop it cleanly. The bot crashes on the next provider, and the result workbook ends up partial (E09).
- If you have no rights in Task Scheduler, ask #rpa-support.
- **A run looks stuck** when `logs/bot.log` has gained no new line for more than about 10 minutes. A normal provider takes under 2 minutes.
- The bot has **no watchdog**. A page that hangs is abandoned only after about 5 minutes, and then the run crashes (E09/E10).
- Wait 10 minutes, then escalate.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `data/output/` → does `Download_Report_<today's date YYYYMMDD>.xlsx` exist? If not, the run failed before the end (E08). Also compare its row count to the active rows in the input: fewer rows means the run stopped partway (E09). |
| Which providers failed? | Result workbook → Status column. Look for **red** (Error), **uncoloured** (Login Failed), **and** green rows whose Remarks say "portal maintenance - no file" (E05). |
| Why did one provider fail? | Result workbook → Remarks column. If it says "Report grid did not load", open `data/screens/<NPI>.png`. |
| Did the bot already retry? | `logs/bot.log` → lines with "session expired, re-login 1/2/3". That is the only retry the bot does. Every other failure is tried once (section 8). |
| Did a provider's file actually arrive? | `data/downloads/`, sorted by date modified. Look for a CSV from the time of that provider's log line. File names do not contain the NPI. |
| When did each provider run? | `logs/bot.log` → search for the NPI. Each line has the time and the status, but not the reason. |
| Which version is running? | No way to tell (section 10). |
## 6. Statuses
These are all the Status / Remarks pairs the bot writes in the result workbook.
| Status | Remarks | Colour | Meaning | What ops does |
|---|---|---|---|---|
| Downloaded | Report saved | green | A new CSV appeared in the downloads folder after Export | Nothing |
| Downloaded | portal maintenance - no file | green | **Needs follow-up. Nothing was downloaded.** The portal showed a maintenance banner | Re-run after the maintenance (E05) |
| No Data | No remittances in the date range | green | The portal shows no remittances for this provider in its default date range | Nothing, unless this provider is expected to have data (E06) |
| Skipped - inactive provider | Marked N in the input | green | Column C is `N` | Nothing, unless the provider should be active (E07) |
| Error | Report grid did not load | red | The provider's page did not show the report within 30 s | E03 |
| Error | Download did not finish | red | No new CSV within 60 s of Export | E04 |
| Error | Session expired | red | The portal logged the bot out 3 times in a row on this provider | E02 |
| Login Failed | Check the portal username/password | **no colour (white)** | The first sign-in failed. **Every** provider gets this row, including inactive ones. Nothing was downloaded | E01 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Every row says "Login Failed - Check the portal username/password", with no colour | After it clicks Sign in, the bot waits 30 s for the portal dashboard. If the dashboard doesn't appear, it gives up on the whole run. It cannot tell a wrong password from a slow or down portal | Result workbook only. **Nothing is written to the log** | Business (wrong password) or System (portal slow/down) | Ops | 1. In your own browser, sign in to the portal with the Login sheet's username and password. 2. If that fails, get the password fixed and update B2. 3. If it works, the portal was slow. Re-run the bot (section 11 has how to re-run). | Login works by hand but the bot fails again. Send the time of the run and the result workbook |
| E02 | Error - "Session expired" | The provider's page sent the bot back to the login page. The bot signed in again and retried, 3 times | Result workbook; log lines "session expired, re-login 1/2/3" | System | Bot, then Ops | Check that nobody else used the same portal login during the run (section 4). Re-run later | It happens on 2 runs in a row. Send the log lines and the run time |
| E03 | Error - "Report grid did not load" | The bot waits 30 s for the report grid on the provider's page. No retry | Result workbook; screenshot `data/screens/<NPI>.png` (overwritten on the next run, so copy it first) | System (or Business if the NPI is wrong) | Ops | 1. Open the screenshot. 2. If it shows an error such as "provider not found", check the NPI in the input. 3. Otherwise open that provider in the portal yourself. If it loads, re-run | The same provider fails on 2 runs while it loads fine by hand. Send the screenshot copy, NPI and run time |
| E04 | Error - "Download did not finish" | After Export, the bot checks the downloads folder every second for 60 s for a **new** CSV. No retry | Result workbook only. The log shows Error with no reason | System | Ops | **Check `data/downloads/` first.** A slow download may have finished after the bot gave up; look for a CSV from a minute or two after that provider's log time. If it is there, you have the file. If not, re-run (re-running only downloads again; it changes nothing in the portal) | It fails on 2 runs in a row. Send the run time and the log |
| E05 | **Green** "Downloaded" with Remarks "portal maintenance - no file" | The portal page shows a maintenance banner | Result workbook Remarks only. The log says "Downloaded" | System | Ops | Treat it as **not downloaded**. Re-run once the portal is out of maintenance. Filter Remarks, not just Status, every morning | Maintenance lasts more than a day |
| E06 | "No Data - No remittances in the date range" | The portal shows its "no results" message for that provider | Result workbook | Business | Ops | Usually nothing. If the provider should have remittances, open the provider in the portal yourself. The bot always uses the portal's **default** date range; it never sets dates | The portal shows remittances by hand but the bot keeps saying No Data |
| E07 | "Skipped - inactive provider - Marked N in the input" | Column C of the Providers sheet is `N` | Result workbook | Business | Ops | If the provider should run, clear column C and re-run | Never |
| E08 | **No result workbook** for today in `data/output/` | The bot stopped before it could save. Possible causes: input workbook missing, open or locked, or a sheet renamed; Chrome failed to start; a folder from section 3 is missing; today's result workbook was open in Excel when the bot tried to save | Nowhere ops can see. The log usually has nothing new for today | System | Ops, then Dev | 1. Go through the section 3 checklist (closed files, folders, sheet names, Chrome opens). 2. Fix what is wrong and re-run | Everything in section 3 is fine and it still fails. Send the date, the time it should have run, and `logs/bot.log` |
| E09 | Today's result workbook has **fewer rows** than the active providers in the input | The bot crashed mid-run. Possible causes: signing in again after a logout failed; the portal's Export button was missing; Chrome crashed or someone closed it. The rows done before the crash are saved; the provider it was on and all later ones are missing | Missing rows in the result workbook. The last line in `logs/bot.log` is the last provider that finished. No reason is recorded anywhere ops can see | System | Dev | **Copy today's result workbook to another name first**, because a re-run on the same day overwrites it. Then re-run. A re-run processes every provider again | Always. Send the copied workbook, the time of the last log line and `logs/bot.log` |
| E10 | The run takes far longer than usual; the log gets no new line for many minutes | No check exists. A portal page that never finishes loading holds the bot for about 5 minutes, then the run crashes as in E09 | Gap in `logs/bot.log` times | System | Ops, then Dev | Wait 10 minutes. If it has still not moved, stop it (section 4) and re-run later | It happens on 2 runs. Send the run time and `logs/bot.log` |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Portal logs the bot out on a provider's page | Signs in again and opens the same provider again | 3 re-logins per provider | Row = Error "Session expired", next provider. **If one of those sign-ins itself fails, the run stops (E09).** |
| First sign-in slow | Waits | 30 s, one try | All rows "Login Failed", run ends (E01) |
| Report grid slow | Waits | 30 s, one try | Row = Error, screenshot, next provider (E03) |
| Download slow | Waits | 60 s, one try | Row = Error, next provider (E04) |
| Run stuck or crashed | **Nothing.** No watchdog, no restart, no resume | - | Ops re-runs. A re-run starts from the first provider |
So for every failure except a logout, the bot has already given up by the time you see it. There is nothing to wait for. Re-running is safe: the bot only downloads, so a re-run never changes anything in the portal. It does overwrite today's result workbook and adds duplicate CSVs (Chrome appends "(1)" to their names).
## 9. Known incidents
There is no change history, bug log or release notes for this bot. The only record is the log, which covers a single run.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-09-02 / unversioned | About a minute into the run the portal logged the bot out. The bot signed in again, and the next provider ended as Error | Not recorded: the log does not keep the reason, so it was either a grid timeout (E03) or a download timeout (E04) | Follow E03/E04: check the screenshot and the downloads folder, then re-run | Partly |
## 10. Releases and version check
- **How an update arrives:** there is no update process. A developer copies new files into the bot's folder on this machine.
- **How to confirm it applied:** you can't. The bot shows no version anywhere. Ask the dev team to confirm the change in #rpa-support, and check the next morning's result workbook.
- **If the behaviour did not change:** report it in #rpa-support with the date of the expected update.
- The bot writes nothing shared on start-up, so another machine with an older copy does not undo a newer one. Only one machine should run the 07:00 task, though. Two machines would download everything twice into their own folders.
## 11. Escalating to dev
Post in the Teams channel **#rpa-support**. Send:
- the exception ID from section 7 (for example "E09");
- the run date and the time of the failure (from `logs/bot.log`);
- the result workbook `data/output/Download_Report_<YYYYMMDD>.xlsx` (a copy, if you are about to re-run);
- `logs/bot.log`;
- for E03, the screenshot `data/screens/<NPI>.png`.
**How to re-run:** in Task Scheduler, right-click the bot's task and choose **Run**. If you have no rights in Task Scheduler, ask in #rpa-support.
**Never send:** the input workbook (its Login sheet holds the portal password), `config.yaml`, passwords, or any patient data from the downloaded CSVs.
## 12. Failure-class check
- **F1** missing prerequisite: covered in section 3 (Chrome, folders, closed files, sheet names) and E08.
- **F2** wrong window / focus: covered in section 4 (don't touch the bot's Chrome window). The bot drives its own Chrome session, not windows found by title, so it cannot grab another app.
- **F3** target slower than expected: covered in E01, E03, E04 and E10, with the limits in section 8. Screen size does not apply; this is a browser, not a desktop-screen bot.
- **F4** input read differently: covered in section 3 (only capital `N` skips; NPI digits only; sheet names). Also in Fix #12.
- **F5** marked done when nothing was done: covered in E05 and Fix #1 and #2.
- **F6** action happened but couldn't be confirmed: covered in E04 (check the downloads folder before re-running). The action is a download only, so a re-run cannot double-post.
- **F7** business exception shown as success: covered in section 6 (maintenance row marked "needs follow-up"), E05 and Fix #1.
- **F8** endless re-runs / watchdog: covered in section 8 and E10. There is no auto re-run loop; the missing watchdog is Fix #9.
- **F9** stopping the wrong way: covered in section 4 (End the task in Task Scheduler, then close the leftover Chrome).
- **F10** one stuck item blocks the queue: covered in E09 (a crash on one provider drops all later providers). There are no batch gates or upload locks. Also in Fix #4 and #5.
- **F11** cause hidden from ops: covered in section 5, and Fix #6, #7 and #16.
- **F12** update didn't arrive: covered in section 10 and Fix #13.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Maintenance banner returns status "Downloaded" (green) with no file | Ops filtering on Status think the file arrived | `bot.py:54-55`, colour map `bot.py:27` |
| 2 | Any new `*.csv` in the downloads folder counts as success; the file is not renamed to the NPI or checked for completeness | False "Downloaded"; ops can't tell which CSV belongs to which provider | `bot.py:63-68` |
| 3 | "Login Failed" rows get no fill (the red in `COLOUR` is never applied on that path); inactive providers are also marked Login Failed | Failed rows look neutral; the inactive rows are misleading | `bot.py:89-91` |
| 4 | A failed re-login inside the retry loop raises `RuntimeError`, which `except SessionExpired` doesn't catch | Run stops; current and later providers missing from report (E09) | `bot.py:100-102` |
| 5 | No per-provider try/except: a missing `#export-csv`, a `NoSuchElementException` or a `WebDriverException` stops the whole run | One bad provider drops all later ones (E09) | `bot.py:64`, `bot.py:92-107` |
| 6 | The log has no remark text, errors are logged at INFO, a first-login failure is not logged, and tracebacks are never logged | The cause is only in the result workbook, or nowhere at all after a crash | `bot.py:105`, `bot.py:88-91`, `bot.py:113-114` |
| 7 | Failures before the `try` block (input missing or locked, wrong sheet, Chrome or chromedriver start, missing argument) write no report | Ops get no result workbook and no reason (E08) | `bot.py:74-81`, `bot.py:114` |
| 8 | `data/output`, `data/screens`, `data/downloads` and `logs` are never created; `save_screenshot` fails silently; the save in `finally` can raise and mask the original error | Lost report or screenshot with no message | `bot.py:20-21`, `bot.py:30`, `bot.py:59`, `bot.py:110` |
| 9 | No page-load timeout, no wait on the login fields, no watchdog or alert | A hung page holds the run about 5 min, then crashes; nobody is told | `bot.py:40-43`, `bot.py:51`, `bot.py:81` |
| 10 | No retry for grid or download timeouts | One-off portal slowness becomes a manual re-run | `bot.py:56-70` |
| 11 | A same-day re-run overwrites the result workbook; screenshots are overwritten on every run; the downloads folder is never cleaned (duplicates get "(1)") | Evidence lost on re-run; folder grows forever | `bot.py:59`, `bot.py:80`, `bot.py:110` |
| 12 | The active flag treats only exact `"N"` as inactive | `n` or `No` providers are downloaded anyway | `bot.py:77` |
| 13 | No version string, no release process | Ops can't confirm which code runs | `bot.py:1-2` |
| 14 | The portal password is in plain text in the input workbook (Login sheet) | Anyone with the input file has the portal login | `bot.py:76` |
| 15 | `config.yaml` holds a plain-text `api_token` that the code never uses | Secret exposure with no benefit. Remove it and rotate | `config.yaml:2` |
| 16 | Crash output only goes to the Task Scheduler console, and ops can't reach Task Scheduler history | Root cause of E08/E09 invisible to ops | `bot.py:113-114`, `README.md:6` |
| 17 | The bot never sets a date range, so "No Data" depends on the portal's default | No Data may hide real remittances if the portal default changes | `bot.py:51` |
| 18 | The `BOT_WAS_RUN` marker is written on every run and never read | Dead file; could be confused for a status flag | `bot.py:74` |
| 19 | Log lines include provider name and NPI | Log can't be shared freely in tickets | `bot.py:105` |
tests/evals/results/handoff/h-web-portal-full-r3.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-web-portal/full",
"rep": 3,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"no-history case says so": true,
"misleading status flagged": true,
"session re-login limit": true,
"screenshots folder": true,
"daily schedule": true
},
"cost_usd": 1.2585940000000002,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-full-mm0v52p2\\project",
"reply": "`HANDOVER.md` ready, repo root-la irukku: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-full-mm0v52p2\\project\\HANDOVER.md`. Bot-a naan run pannala; code, config, README and log mattum padichen.\n\n| Item | Result |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 10 (E01\u2013E10) |\n| Failure classes | 11 cover aachu, 1 not applicable (F10: batches illa, ovvoru run-um fresh-a start aagum) |\n| Fix before handover | 15 items |\n| Checks | Ellaa status string-um section 6-la irukku. file:line section 13-la mattum dhaan. URL, IP, token, password, PHI ethuvum illa. |\n\n**Fix before handover \u2014 mukkiyamaanavai:**\n1. Portal maintenance-la file download aagalainnaalum \"Downloaded\" (green)-nu kaattudhu.\n2. Edhaavadhu oru pudhu CSV vandhaale success-nu edukkudhu. Files provider vaariya rename aagaradhilla.\n3. Remark log-la varala. Run crash aana cause engeyum record aagaradhilla.\n4. Login fail log-la varala. Andha rows-ukku colour-um illa.\n5. Page-load timeout-um illa, watchdog-um illa. Run forever hang aagalaam.\n6. `logs` and `data/*` folders-a bot create pannaadhu.\n7. Version number illa.\n8. Same day re-run panna morning report overwrite aagidum.\n9. Inactive flag-a exact-a capital \"N\"-a irundha mattum dhaan skip pannum.\n10. `config.yaml`-la use aagaadha API token plaintext-la irukku; portal password input Excel-la irukku.\n11. Clean stop illa.\n12. README-la sonna maadhiri ellaa failure-kkum screenshot edukkaradhilla.\n13. \"No Data\" portal-oda default date range-a nambiyirukku.\n14. Log-la provider name-um NPI-yum ovvoru line-layum varudhu.\n\n**Note:**\n- Incident history: 2026-09-02 log-la oru run mattum dhaan irukku. Andha naal oru Error vandhuchu, adhu partly ops handle panna koodiyadhu.\n- Nee \"output folder\" mattum sonna. Naan `data/downloads` and `data/screens`-um ops open pannalaam-nu assume pannen.\n- Bot-ah stop panna doc enna solludhu-na: bot open panna Chrome window-a close pannunga. Task Scheduler-la task-a end panna report ezhudhaadhu. Idhu code-a padichu naan purinjukittadhu; test pannala.\n- Unga instruction padi, statuses-ku code-oda best reading-a dhaan use pannen.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do any statuses or exceptions mean whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-web-portal-full-r3.md
+193
View File
@@ -0,0 +1,193 @@
# Payer portal downloader - ops handover
Version: not versioned (see section 10) · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every day at 07:00 Windows Task Scheduler starts the bot. It signs in to the payer web portal and downloads the remittance report (a CSV file) for each provider listed in the input Excel file, `data/input/providers.xlsx`. It skips providers marked as inactive. When it finishes it writes a result report with one row per provider, showing a status and a remark. The only thing it drives is **a Google Chrome window that it opens itself**. There is no app window, no database and no email or alert.
## 2. How it works
```
Windows Task Scheduler (07:00 daily)
└─ starts → bot script ──drives──> its own Chrome window → payer web portal
│ (do not touch or close while it runs)
├─ reads → data/input/providers.xlsx (Login + Providers sheets)
├─ writes → data/downloads/*.csv (the downloaded reports)
├─ writes → data/screens/<NPI>.png (screenshot when a page didn't load)
├─ writes → data/output/Download_Report_<YYYYMMDD>.xlsx (result per provider)
└─ writes → logs/bot.log (one line per provider, all runs in one file)
```
- **Task Scheduler** starts the run. Nobody has to click anything.
- **The bot script** runs in the background and does all the work.
- **The Chrome window** is opened and controlled by the bot. Chrome shows a bar saying it is being controlled by automated software.
- **providers.xlsx** is the input. Ops maintains it.
- **data/downloads** receives the CSV files with Chrome's own file names. They are *not* renamed per provider.
- **data/screens** holds a screenshot named after the provider's NPI. One is taken only when the report page didn't load.
- **Download_Report_<date>.xlsx** is the main place to check results. It has the columns Provider, NPI, Status and Remarks.
- **logs/bot.log** gets one line per provider, plus a warning each time the bot had to sign in again.
**One run, step by step:**
1. Read the input file: the login details and the provider list.
2. Open Chrome and sign in to the portal. The bot waits up to 30 s for the portal dashboard.
3. For each provider, in input order:
- **inactive** → mark "Skipped" and move on;
- otherwise → open the provider's remittance page;
- wait up to 30 s for the report grid;
- click **Export CSV**;
- wait up to 60 s for a new CSV file to appear in `data/downloads`.
4. Write the provider's status to the result report and the log, then move to the next provider.
5. Close Chrome and save the result report. The report is saved even if the run stopped early, unless the output folder is missing.
## 3. Before a run
- [ ] **Google Chrome** is installed on the machine.
- [ ] **Python packages** are installed: selenium, openpyxl and pyyaml.
- [ ] **These folders exist** in the bot's folder. The bot does not create them:
- `logs` - without it, the bot fails at start and writes nothing at all;
- `data/output` - without it, the result report cannot be saved;
- `data/downloads` - the CSV files go here;
- `data/screens` - without it, screenshots are lost silently.
- [ ] **The settings file** `config.yaml` is in the bot's folder and holds the portal address.
- [ ] **Input file - sheet `Login`:**
- cell A2 holds the portal username;
- cell B2 holds the portal password;
- when the portal password changes, update B2 the same day.
- [ ] **Input file - sheet `Providers`:**
- data starts at row 2, one provider per row;
- column A is the NPI, column B the provider name, column C the active flag;
- a row with an empty column A is ignored silently, so never leave the NPI blank.
- [ ] **Active flag in column C:** only a capital **`N`** marks a provider inactive. Anything else, including a blank, means active.
- Bad: `n`, `No`, `N ` (with a space), `Inactive`. The bot will try to download these providers.
- Good: `N`.
- [ ] **Need a second run on the same day?** Copy that morning's `Download_Report_<today>.xlsx` somewhere safe first. A second run replaces it, downloads every provider's CSV again, and Chrome saves the duplicates as `name (1).csv`.
- [ ] **Version:** there is no version number yet (see section 10).
## 4. During a run
- **Don't touch the Chrome window the bot opened.** Don't click in it, close it or sign out of the portal in it. You can use the rest of the machine; the bot doesn't use the mouse or keyboard.
- **Don't sign in to the portal with the same username anywhere else during the run.** The portal may end the bot's session (see E05).
- **To stop the bot:** close the Chrome window the bot opened (the one with the "controlled by automated software" bar).
- The bot stops and normally saves the result report with the providers finished so far.
- Ending the task in Task Scheduler also stops it, but then **no report is written**, and Chrome may stay open. Close Chrome by hand afterwards.
- **A run looks stuck when** `logs/bot.log` gets no new line for more than 10 minutes. One provider takes at most about 5 minutes, even with every retry.
- **The bot has no watchdog**: a page that never finishes loading can hang it forever.
- If it is stuck, stop it as described above and raise it with dev (E04).
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `data/output/Download_Report_<today>.xlsx` exists and has one row per provider in the input (blank-NPI rows excluded). Fewer rows means the run stopped early (E04). No file at all: see E03. |
| Which providers failed? | The result report, Status column: red **Error** rows, uncoloured **Login Failed** rows. Also check green rows whose Remarks say **"portal maintenance - no file"** (E08). |
| Why did one provider fail? | The result report, Remarks column for that NPI. For "Report grid did not load", open `data/screens/<NPI>.png`. |
| Did the bot already retry? | `logs/bot.log`: lines saying "session expired, re-login 1/2/3" just before that provider's line. No such lines means no retry: only an expired session is retried. |
| Which CSV belongs to which provider? | You can't tell reliably today. The files keep the portal's names. Open the CSV and check its contents, or download it again by hand from the portal. |
| Which version is running? | Not shown anywhere (see section 10). |
## 6. Statuses
All statuses appear in the result report's Status column. Only the Status cell is coloured.
| Status - Remarks | Colour | Meaning | What ops does |
|---|---|---|---|
| Downloaded - "Report saved" | green | A new CSV appeared in `data/downloads` after Export was clicked | Nothing. Rarely, the file is a late download from the provider before (see section 13). |
| Downloaded - "portal maintenance - no file" | green | **Needs follow-up. Nothing was downloaded.** The portal showed its maintenance banner. | Download it again later (E08) |
| No Data - "No remittances in the date range" | green | The portal said there are no remittances for this provider in its default date range | Nothing, unless remittances were expected (E09) |
| Skipped - inactive provider - "Marked N in the input" | green | Column C is `N`, so the bot did not look at this provider | Nothing |
| Login Failed - "Check the portal username/password" | **no colour** | The bot could not sign in. Every provider in the run gets this row. | Fix the login (E01) |
| Error - "Report grid did not load" | red | The provider's report page did not load within 30 s | E06 |
| Error - "Download did not finish" | red | Export was clicked, but no new file appeared within 60 s | **Check the downloads folder first** (E07) |
| Error - "Session expired" | red | The portal signed the bot out three times in a row on this provider | E05 |
| *(provider has no row)* | - | The run stopped before reaching this provider | E04 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Every row says **Login Failed** - "Check the portal username/password" (no colour) | After signing in it waits 30 s for the portal dashboard; if it never appears, it gives up the whole run | Result report only. **Nothing in the log** for this run. | Business (usually wrong password) / System (portal slow) | Ops | Sign in to the portal by hand with the username and password in the Login sheet. If that fails, fix A2/B2 (or reset the password). If it works, the portal was slow: run again. | Manual sign-in works but the bot keeps failing. Send: date and time of the run, the result report, `logs/bot.log`. |
| E02 | Result report exists but has **only the header row**; nothing for today in the log | The sign-in page didn't show the username/password/sign-in fields (portal changed or down); the bot stops at once | Nowhere readable: the error only goes to the console, which isn't kept | System | Dev | Check whether the portal sign-in page opens normally in your own browser. If the portal is down, run again later. | The portal sign-in page works normally. Send: date and time, the result report, `logs/bot.log`, E02. |
| E03 | **No result report** for today | The bot stopped before the report could be saved: Chrome couldn't start, the input file was missing or unreadable, or the `logs` or `data/output` folder is missing | Nowhere readable (console only). A missing `logs` folder also means no log. | System | Ops, then Dev | Check the section 3 list: Chrome installed, folders exist, input file at `data/input/providers.xlsx` and not open in Excel, `config.yaml` present. Then run again. | Everything in section 3 is in place and there is still no report. Send: date and time, E03, screenshot of the folder. |
| E04 | Result report has **fewer rows than the input** (later providers missing), or the run hangs with no new log line for 10+ minutes | The run crashed or hung partway: signing in again failed after the portal ended the session, the Export button was missing, Chrome closed, or a page never finished loading | Result report (rows up to the crash). The log's last line is the last provider finished. The cause is not recorded anywhere. | System | Dev (Ops checks first) | Check the portal by hand: sign-in works, and the next provider's remittance page shows an Export CSV button. Download the missing providers by hand if they are needed today. | Always, if it happens two days running. Send: date, the NPI of the last row in the report, `logs/bot.log`, the result report, E04. |
| E05 | Error - "Session expired" | Opening a provider's page lands on the sign-in page. The bot signs in again and retries, 3 tries in total. | Result report; "session expired, re-login 1/2/3" lines in `logs/bot.log` | System | Bot, then Ops | Make sure nobody else uses the same portal login during the run. Download this provider by hand, or run again (see section 3, second run). | More than 2 providers per run, or it happens on several days. Send: date, NPI(s), `logs/bot.log`, E05. |
| E06 | Error - "Report grid did not load" | It waits 30 s for the remittance grid on the provider's page; if it doesn't appear, it takes a screenshot and moves on. No retry. | Result report; `data/screens/<NPI>.png` (missing if the `data/screens` folder is missing) | System | Ops | Look at the screenshot. If the portal shows an error for this provider (wrong NPI, no access), fix the input. Otherwise download it by hand or run again later. | Same provider fails 3 days in a row and the page works by hand. Send: NPI, date, the screenshot, E06. |
| E07 | Error - "Download did not finish" | It clicks Export CSV and waits 60 s for a *new* CSV file name in `data/downloads`. No retry. | Result report | System | **Ops** | **Check `data/downloads` first.** The file may have arrived after 60 s, or under a name that was already there. Open the newest CSV and check it is this provider's. If there is no file, download it by hand from the portal. | It happens for many providers in one run. Send: date, NPIs, `logs/bot.log`, E07. |
| E08 | **Green** Downloaded with Remarks "portal maintenance - no file" | The provider's page shows the portal's maintenance banner. The bot records this as Downloaded without downloading anything. No retry. | Result report (Remarks column only) | System (portal down) | Ops | Filter the Remarks column for "maintenance" after every run. When the portal is back, download those providers by hand or run again. | Never needed for the maintenance itself. The green status is a known gap (section 13). |
| E09 | No Data - "No remittances in the date range" (green) | The portal page shows its "no results" message for the provider | Result report | Business | Ops (only if unexpected) | Usually nothing. If remittances were expected, open the provider in the portal by hand. The bot uses the portal's **default** date range and cannot set its own. | Remittances are visible by hand but the bot says No Data. Send: NPI, date, E09. |
| E10 | A provider you meant to skip **was** processed, or one you expected is missing from the report | Only a capital `N` in column C counts as inactive. A row with an empty NPI is left out of the run completely. | Result report (row present or missing) | Business | Ops | Fix the input: put `N` exactly in column C; make sure column A has the NPI on every row. | Never |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Portal ended the session (provider page shows sign-in) | Signs in again and retries the same provider | 3 tries per provider. Each sign-in waits up to 30 s. | Provider = Error "Session expired", bot moves to the next provider. If signing in again itself fails, the run stops (E04). |
| Report grid doesn't load | Nothing: takes a screenshot and moves on | 30 s, no retry | Error "Report grid did not load" |
| Download doesn't appear | Nothing: moves on | 60 s, no retry | Error "Download did not finish" |
| First sign-in fails | Nothing | 30 s, no retry | Every provider = Login Failed, run ends |
| Maintenance banner | Nothing | no retry | Green Downloaded "portal maintenance - no file" (E08) |
| Page never finishes loading / bot hangs | **Nothing - no watchdog** | none | Hangs until someone stops it (E04) |
So: apart from an expired session, **the bot never retries**. Ops never needs to wait for an automatic recovery once the run has ended.
## 9. Known incidents
Only one run is on record: one day of log, no change history, no bug log.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-09-02 / unversioned | One provider red **Error**, about 80 s after a "session expired, re-login 1" line in the log | The portal ended the session. The bot signed in again, but the retry then failed with a grid or download timeout. The log doesn't say which, because remarks aren't logged. | Read the Remarks column, check `data/screens/<NPI>.png`, and download that provider by hand (E05-E07) | Partly |
## 10. Releases and version check
- **How an update arrives:** by hand. A developer replaces the bot's files on the machine. There is no launcher or automatic update.
- **How to confirm it applied:** you can't yet. The bot has no version number on screen, in the log or in the report (section 13). Ask dev to confirm in #rpa-support which files changed and when.
- **If an update seems not to have taken effect:** report it in #rpa-support with the date and the behaviour you still see.
- The bot writes to no shared database, so a machine running an older copy cannot undo a newer one. Each machine only affects its own folders.
## 11. Escalating to dev
Post in **#rpa-support** (Teams). Send:
- the exception ID from section 7 (for example E04);
- the date and approximate time of the run;
- the NPI(s) affected - the NPI only, not provider or patient details;
- the result report `data/output/Download_Report_<date>.xlsx`;
- `logs/bot.log`;
- for E06, the screenshot `data/screens/<NPI>.png`.
**Never send:** the portal password, the `Login` sheet, `config.yaml`, the downloaded CSV files (they contain remittance data), or patient data in chat.
## 12. Failure-class check
- **F1 Missing prerequisite** - covered in section 3 (Chrome, packages, folders, settings file) and in E03. The missing folders are also in section 13 #7.
- **F2 Wrong window / focus** - covered in section 4: don't touch the bot's Chrome window. Otherwise the bot controls its own Chrome directly, with no window matching or focus.
- **F3 Target slower or different** - covered in E02, E04, E06 and E07, with the limits in section 8. There is no page-load timeout (section 13 #6).
- **F4 Input read differently** - covered in section 3 (the exact `N` and blank-NPI rules, with examples) and E10. Also in section 13 #10.
- **F5 Marked done when nothing was done** - covered in E08 (maintenance = Downloaded). Also in section 13 #1 and #2 (any new CSV counts as a success).
- **F6 Action happened but not confirmed** - covered in E07: check the downloads folder before re-running.
- **F7 Business exception shown as success** - covered in section 6 (maintenance row marked "needs follow-up") and E08. Also in section 13 #1.
- **F8 Endless re-runs / watchdog** - covered in section 8 (the only retry is bounded at 3) and in section 4 (the stuck sign). There is no watchdog (section 13 #6).
- **F9 Stopping the wrong way** - covered in section 4: close the bot's Chrome window, not the task. Also in section 13 #12.
- **F10 One stuck item blocks the queue** - not applicable: there are no batches or completion gates, and each run starts fresh. The nearest risk, a same-day re-run replacing the report, is covered in section 3 and section 13 #9.
- **F11 Cause hidden from ops** - covered in section 5. Also in section 13 #3, #4 and #5 (remarks, crashes and login failures not logged).
- **F12 Update didn't arrive** - covered in section 10. Also in section 13 #8 (no version number).
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Maintenance banner is recorded as status "Downloaded" (green) with no file | Ops treats the provider as done; the missing file is found later, if at all | `bot.py:54-55`, colour map `bot.py:27-28` |
| 2 | Download check accepts *any* new CSV in the folder, and files are not renamed per provider | A late file from the previous provider can be counted as this one's. Nobody can tell which CSV belongs to which NPI. | `bot.py:63-68`, `bot.py:80` |
| 3 | The remark is not written to the log; only the status is | The log can't tell the three "Error" reasons apart (see the 2026-09-02 incident) | `bot.py:105` |
| 4 | Unexpected errors are not caught or logged. A failed re-login, a missing page element or a Chrome crash ends the whole run. The traceback goes only to the console, which Task Scheduler doesn't keep. | Run stops partway with no recorded cause (E02, E03, E04) | `bot.py:41-43`, `:64`, `:81`, `:102`, `:108-110` |
| 5 | A first-login failure writes nothing to the log, and "Login Failed" rows get no colour | It is easy to miss in the report, and invisible in the log | `bot.py:88-91` (returns before `:107`) |
| 6 | No page-load timeout and no watchdog | A hung page hangs the run forever; nobody is told | `bot.py:40`, `:51`, `:79-81` |
| 7 | `logs`, `data/output`, `data/downloads` and `data/screens` are never created | Missing `logs` = bot dies at start with no trace. Missing `data/output` = the report save fails and hides the original error. Missing `data/screens` = screenshots lost silently. | `bot.py:20-21`, `:30`, `:59`, `:110` |
| 8 | No version string anywhere; dependencies unpinned | Ops can't confirm an update applied, and a package upgrade can change behaviour silently | `bot.py:1-2`, `README.md`, `requirements.txt:1-3` |
| 9 | A re-run downloads everything again and overwrites the same-day report | Duplicate CSVs, and the morning's results are lost | `bot.py:63-68`, `:110` |
| 10 | Inactive flag must be exactly "N", and a blank NPI drops the row silently | Lower-case or "No" providers are processed; blank-NPI rows vanish from the report | `bot.py:77-78` |
| 11 | Plaintext secrets: an unused API token in `config.yaml`, and the portal password in the input workbook | Anyone with the input file or the folder has the credentials. Rotate or remove the token. | `config.yaml:2`, `bot.py:76` |
| 12 | No clean stop: ending the task loses the report, and closing Chrome is the only partial-save path | Ops must use a workaround to stop safely | `bot.py:85-110` |
| 13 | README promises "screenshots of failed pages", but only a grid timeout takes one | Ops look for screenshots that don't exist | `README.md:5`, `bot.py:59` |
| 14 | "No Data" depends on the portal's default date range; the bot sets no range | A change in the portal default silently changes what "No Data" means | `bot.py:61-62` |
| 15 | Provider name and NPI written to the log on every line | The log file needs the same handling as the input file | `bot.py:105` |
tests/evals/results/handoff/h-web-portal-full-r4.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-web-portal/full",
"rep": 4,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"no-history case says so": true,
"misleading status flagged": true,
"session re-login limit": true,
"screenshots folder": true,
"daily schedule": true
},
"cost_usd": 1.0619698000000002,
"subagents": 1,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-full-_jnxboxu\\project",
"reply": "`HANDOVER.md` ready in the repo root: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-full-_jnxboxu\\project\\HANDOVER.md`. Bot-ah run pannala, code-um change pannala, only intha oru file dhaan ezhudhinen.\n\n| Item | Result |\n|---|---|\n| Sections | 13 |\n| Statuses | 8 (bot ezhudhra 7 status + \"Login Failed\" + \"no row\" case), ellam vandhiruku |\n| Exceptions | 10 (E01\u2013E10). Dev rows-la enna anuppanum-nu sollirukku |\n| Failure classes | 12/12 covered. F2, F8, F10, F12 la oru part mattum not applicable (reason doc-la irukku) |\n| Fix before handover | 15 items |\n\n**Mukkiyamana fixes:**\n1. Portal maintenance page vandha \"Downloaded\" (green)-nu podudhu, aana file download aagala.\n2. Edhavadhu unexpected error vandha run full-ah stop aagudhu. Log-la edhuvum record aagala.\n3. Late-ah vandha CSV adutha provider-ku count aagidum. File name-la NPI illa.\n4. Folders bot create pannadhu. `data/output` illana andha naal report full-ah poyidum. Repo-la `data/` folders eppovum illa, adhanaala doc ops-ah create panna solludhu.\n5. Portal password input Excel-la plain text-la irukku. Unused `api_token` `config.yaml`-la irukku (value-ah doc-la copy pannala).\n\n**Gaps:**\n- Git history illa, oru log run (2026-09-02) mattum dhaan irukku. Adhanaala Known incidents-la oru row dhaan.\n- Ops access-ku database illa, aana indha bot-ku DB-ve illa.\n- Unclear statuses-ku code padichu best reading use pannen.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-web-portal-full-r4.md
+189
View File
@@ -0,0 +1,189 @@
# Payer portal downloader - ops handover
Version: not versioned (no version number exists yet, see section 10) · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every day at 07:00 Windows Task Scheduler starts the bot. It signs in to the payer web portal in Chrome and downloads each provider's remittance report as a CSV file. Its input is the Excel workbook `data/input/providers.xlsx`, which has two sheets: **Login** (the portal username and password) and **Providers** (one provider per row). It produces three things: one CSV per provider in `data/downloads/`, a daily result workbook `data/output/Download_Report_<YYYYMMDD>.xlsx` with one row per provider, and screenshots of pages that failed to load in `data/screens/`. The only thing it drives is the payer portal, in a Chrome window it opens itself.
## 2. How it works
```
Windows Task Scheduler (07:00 daily)
└─ starts → the bot (no window of its own)
│
├─ opens and drives → Chrome ("controlled by automated test software")
│ └─> payer web portal: sign-in page, then each provider's remittance page
│ (do not touch this Chrome window while it runs)
├─ reads → data/input/providers.xlsx (sheets Login, Providers)
├─ writes → data/downloads/*.csv (one file per provider, portal's own file name)
├─ writes → data/screens/<NPI>.png (only when a page did not load)
├─ writes → data/output/Download_Report_<YYYYMMDD>.xlsx (saved at the END of the run)
└─ writes → logs/bot.log (one line per provider)
```
- **Task Scheduler** is the only thing that starts the bot. The bot has no screen and no Start/Stop buttons.
- **The bot** runs in the background, works through the providers one at a time from top to bottom, and closes Chrome when it finishes.
- **Chrome** is opened by the bot. A bar at the top says it is controlled by automated software. That window belongs to the bot.
- **The payer portal** is the website the bot signs in to and downloads from.
- **providers.xlsx** is the input that ops maintains.
- **data/downloads** holds the downloaded CSV files. They keep the portal's own file names, so a file name does not tell you which provider it belongs to.
- **data/screens** holds a picture of the page for each provider whose report table did not load.
- **Download_Report_<date>.xlsx** is the result of the run. It is written only once, when the run ends.
- **logs/bot.log** gets one line per provider giving the time, the NPI, the name and the status. It also gets a warning line every time the bot has to sign in again.
**One run, step by step:**
1. Read the Login and Providers sheets.
2. Open Chrome and sign in to the portal. The bot waits up to 30 s for the portal's dashboard.
3. For each provider row:
- If column C is `N`, skip it.
- Otherwise open that provider's remittance page.
- If the portal has signed the bot out, sign in again and retry, up to 3 tries in total.
- Wait up to 30 s for the report table.
- If the portal says there are no results, record "No Data".
- Otherwise click the export button and wait up to 60 s for a new CSV file to appear.
4. Write one line to the log and one row in the report.
5. Close Chrome and save the report workbook.
## 3. Before a run
- [ ] The machine has Google Chrome installed, and Python with the bot's packages installed. If either is missing the bot stops at start and no report is created (E09).
- [ ] These folders exist in the bot folder: `data/input`, `data/output`, `data/downloads`, `data/screens` and `logs`. The bot does not create them.
- If `data/output` is missing, the whole day's report is lost at the end of the run.
- If `data/screens` is missing, no screenshots are saved, and nothing tells you so.
- If `logs` is missing, the bot does not start.
- [ ] `config.yaml` is in the bot folder. Do not edit it.
- [ ] `data/input/providers.xlsx` exists and has sheets named exactly **Login** and **Providers**.
- [ ] **Login** sheet: cell A2 holds the portal username and B2 the password. When the portal password changes, update B2 before the next 07:00 run, or every row comes out "Login Failed" (E01).
- [ ] **Providers** sheet: row 1 is the header and data starts at row 2. Column A is the NPI, column B the provider name and column C the active flag. Rules for these columns:
- **Column A (NPI) must not be blank.** A row with a blank column A is ignored without any message and gets no row in the report (E10).
- Bad: blank A, name in B.
- Good: `1234567890` in A.
- **Column C: only a capital `N` means inactive.** `n`, `No` or `no` are treated as active, and the bot will try to download those providers.
- Bad: `No`.
- Good: `N`. Leave C blank for active providers.
- [ ] A report for today already in `data/output` will be **overwritten** if the bot runs again today. Copy it somewhere first if you need to keep it.
- [ ] There is no version number to check. See section 10.
## 4. During a run
- Do not click, type in or close the Chrome window marked "controlled by automated test software". Do not sign in to the portal with the same username in another browser during the run, because the portal may sign the bot out.
- You can use other programs on the machine. The bot does not need the mouse or keyboard.
- **To stop the bot:** close the bot's Chrome window. The bot then stops, and the report is still saved for the providers it had finished; the rest get no row. Do **not** use "End" in Task Scheduler. That stops the bot without saving any report, and can leave Chrome open.
- A run looks stuck when `logs/bot.log` gets no new line for 10 minutes. No single wait in the bot is longer than 60 s, but the bot has **no watchdog of its own**: it never restarts or gives up by itself. If it is stuck, stop it as described above and escalate (E03).
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `data/output/Download_Report_<today>.xlsx` exists, and it has one row for every provider in the input that has an NPI. Fewer rows means the run stopped partway (E03). No file at all means it never got going or lost the report (E09). |
| Which providers failed? | The report workbook, **Status** column: red **Error**, or **Login Failed** (no colour). Also check every **Downloaded** row whose Remarks say "portal maintenance - no file". Those failed too, even though they are green (E04). |
| Why did one provider fail? | The report workbook, **Remarks** column. For "Report grid did not load", open `data/screens/<NPI>.png`. The log `logs/bot.log` shows only the time and the status. It does not record the reason. |
| Did the bot already retry? | `logs/bot.log`: lines with "session expired, re-login 1" (or 2, 3) just before that provider's line. Sign-in after the portal signs the bot out is the only thing the bot retries. |
| Which CSV belongs to which provider? | There is no reliable way. The files in `data/downloads` keep the portal's names. They can only be matched by file time against the log line times. |
| Which version is running? | No version is shown anywhere (section 10). |
## 6. Statuses
All statuses appear in the report workbook's **Status** column, with the reason in **Remarks**.
| Status - Remarks | Colour | Meaning | What ops does |
|---|---|---|---|
| Downloaded - "Report saved" | green | A new CSV appeared in `data/downloads` | Nothing. See E07 for one rare case where the file belongs to the previous provider. |
| Downloaded - "portal maintenance - no file" | green | **Not downloaded.** The portal showed a maintenance notice, and the bot recorded it as Downloaded. **Needs follow-up.** | Re-run after the maintenance window (E04) |
| No Data - "No remittances in the date range" | green | The portal shows no remittances for this provider | Nothing, unless you expected remittances (E06) |
| Skipped - inactive provider - "Marked N in the input" | green | Column C was `N`, so the bot did not try | Nothing, unless the provider is active again. Then clear column C. |
| Error - "Report grid did not load" | red | The report table did not appear within 30 s. A screenshot was saved. | E05 |
| Error - "Download did not finish" | red | No new CSV appeared within 60 s of clicking export | E07 |
| Error - "Session expired" | red | The portal signed the bot out 3 times in a row on this provider | E02 |
| Login Failed - "Check the portal username/password" | **no colour** (white) | The first sign-in failed. Every provider gets this row and nothing is downloaded. | E01 |
| *(no row for a provider)* | - | Either its NPI cell is blank (E10), or the run stopped before reaching it (E03) | See E10 / E03 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Every row is "Login Failed - Check the portal username/password" | After signing in, it waits 30 s for the portal dashboard. If the dashboard never appears, it gives up and nothing is downloaded. | Report workbook only. **Nothing is written to the log.** | Business (wrong password) or System (portal slow or down) | Ops | Sign in to the portal by hand with the Login sheet's A2/B2 values. If that fails, fix the password in B2. If it works, the portal was slow: run the bot again. | Manual sign-in works but the bot fails twice in a row. Send: date, report file, "E01". |
| E02 | Error - "Session expired" | The provider's page sends the bot back to the sign-in page. It signs in again and retries, up to 3 tries in total. | Report workbook; "session expired, re-login N" lines in `logs/bot.log` | System | Bot, then Ops | Re-run the bot later. Check that nobody else was using the same portal login at that time. | Several providers in the same run, or on 2 days in a row. Send: date, log time of the warnings, "E02". |
| E03 | The report has fewer rows than the input, and the rows stop at some provider | Not detected. An unexpected problem stopped the bot. Possible causes: the portal's sign-in failed while signing back in; a button or field it needs was missing on the page; Chrome was closed or crashed; the run was stopped by hand. | Nothing. The report is saved with the rows done so far. **The log has no error line.** | System | Dev | Note the first provider that is missing, then run the bot again. A same-day re-run overwrites the report, so copy the partial report first. | Always, unless you stopped it yourself. Send: date, the partial report, the first missing provider's row number in the input, the last time in `logs/bot.log`, "E03". |
| E04 | **Green** "Downloaded" with Remarks "portal maintenance - no file" | The provider's page shows the portal's maintenance notice. The bot records it as Downloaded and moves on. | Report workbook (green); the log says "Downloaded" | System | Ops | Treat it as **not downloaded**. Check the portal's maintenance window and run the bot again after it. | The notice is still there after the portal says maintenance is over. Send: date, "E04". |
| E05 | Error - "Report grid did not load" | It waits 30 s for the provider's report table. If it never appears, it saves a screenshot and moves on. | Report workbook; `data/screens/<NPI>.png` | System | Ops | Open the screenshot. If it shows a slow or blank page, run the bot again later. If it shows an error message from the portal, follow that message. | The same provider fails on 2 runs, or the screenshot shows the page looking different from usual. Send: date, the screenshot, "E05". |
| E06 | No Data - "No remittances in the date range" | The portal shows its "no results" message for that provider | Report workbook (green) | Business | Ops | Usually nothing. If you expected remittances, check the provider in the portal by hand, and check the NPI in column A. | You see remittances in the portal by hand but the bot says No Data. Send: date, "E06". |
| E07 | Error - "Download did not finish" | After clicking export, it checks `data/downloads` every second for 60 s for a new CSV file. If none appears, it moves on. | Report workbook | System | Ops | **Check `data/downloads` first.** The file may have arrived late. A late file can also be counted as the *next* provider's download, so that provider's "Downloaded" may be wrong. If no late file is there, run the bot again. | It happens for several providers in one run, or on 2 days in a row. Send: date, log times, "E07". |
| E08 | "Skipped - inactive provider" for a provider you expected | Column C of its row is exactly `N` | Report workbook (green) | Business | Ops | Clear column C if the provider is active. | Never. This is an input fix. |
| E09 | No report file for today in `data/output` | Not detected. Possible causes: the bot never started (missing `logs` folder, missing `config.yaml`, missing input file, Chrome or Python problem, often after a Chrome update); or the `data/output` folder is missing so the report could not be saved. | Nothing. Check whether `logs/bot.log` has lines for today. | System | Ops, then Dev | Check the folders and files in section 3. If they are all there, escalate. | The folders and input are fine and there is still no report. Send: date, whether `logs/bot.log` has lines for today, "E09". |
| E10 | A provider from the input has no row in the report, but the providers after it do | Rows with a blank NPI (column A) are ignored without a message | Nothing | Business | Ops | Fill in the NPI in column A. | Never. This is an input fix. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| The portal signs the bot out | Signs in again and opens the provider's page again | 3 tries per provider | Row = Error "Session expired", next provider. If signing in again itself fails, the run stops (E03). |
| First sign-in slow | Waits for the dashboard | 30 s, **no retry** | Every row "Login Failed", run ends |
| Report table slow | Waits | 30 s, **no retry** | Row = Error "Report grid did not load", screenshot, next provider |
| CSV download slow | Checks every second | 60 s, **no retry** | Row = Error "Download did not finish", next provider |
| Anything else (a missing button, Chrome crash, stuck page) | Nothing. No restart, no watchdog, no resume. | - | The run stops. The report is saved with the rows done so far (E03). |
A re-run always starts again from the first provider. It downloads every provider again (duplicate CSVs in `data/downloads`) and overwrites today's report and screenshots.
## 9. Known incidents
There is no recorded history: no version control, changelog or bug log was found. The only evidence is `logs/bot.log`, which covers one run on 2026-09-02.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-09-02 / not versioned | One provider re-logged in once ("session expired, re-login 1"); one provider ended in Error | The log records no reason for the Error, so the cause is unknown | Read the Remarks column in that day's report, then use E02/E05/E07 | Partly. The log alone cannot tell you which Error it was. |
## 10. Releases and version check
- How an update arrives: there is no updater. A developer replaces the bot's files on the machine by hand.
- How to confirm it applied: there is no version number in the bot. Ask the dev team in #rpa-support to confirm what changed.
- If something stops working after an update: escalate with the date of the change. Chrome updates itself. If the bot suddenly stops starting the day after a Chrome update (E09), say so in the ticket.
- Only one machine runs this bot, through Task Scheduler. Do not set it up on a second machine. Two runs on the same day overwrite each other's report.
## 11. Escalating to dev
Post in the Teams channel **#rpa-support**. Send:
- the date of the run
- the exception ID from section 7 (E01-E10)
- the report workbook from `data/output`
- the relevant time from `logs/bot.log`
- the screenshot from `data/screens` (for E05)
- for one provider, its **row number** in the input
Never send in chat: the portal password, the input workbook (it contains the password), `config.yaml`, or provider names and NPIs.
## 12. Failure-class check
- **F1** (missing prerequisite) is covered in section 3 (Chrome, Python, folders, `config.yaml`, input sheets) and E09.
- **F2** (wrong window or focus) is covered in section 4: do not touch the bot's Chrome window. Matching windows by title and taking focus are not applicable, because the bot controls its own Chrome directly, not through screen clicks.
- **F3** (target app slower or different than expected) is covered in E01, E05 and E07, and section 8 (30 s and 60 s waits, no retry). The fixed waits with no retry are in Fix before handover #9.
- **F4** (input read differently than written) is covered in section 3 (blank NPI, capital `N` only, sheet names) and E08 and E10. Fix before handover #8.
- **F5** (marked done when nothing was done) is covered in E04: maintenance is recorded as Downloaded. Fix before handover #1 and #4.
- **F6** (action happened but could not be confirmed) is covered in E07: check `data/downloads` before re-running, because a late file can arrive and be counted as the next provider's. Fix before handover #4.
- **F7** (business exception shown as success) is covered in section 6 and E04: the maintenance row is green. "Login Failed" has no colour. Fix before handover #1 and #6.
- **F8** (endless re-runs or watchdog kills): an endless loop is not applicable, because the bot has no auto-rerun and no watchdog, and the sign-in retry stops at 3. The missing watchdog is covered in section 4 (stuck sign) and section 8.
- **F9** (stopping the bot the wrong way) is covered in section 4: close the bot's Chrome window, not End in Task Scheduler. Fix before handover #11.
- **F10** (one stuck item blocks the queue): a queue lock is not applicable, because there are no batches or locks. One crash does stop all remaining providers, which is covered in E03 and Fix before handover #2.
- **F11** (cause hidden from ops) appears in Fix before handover #3, #4 and #5: crash details are not logged, remarks are not logged, and CSV names do not identify the provider.
- **F12** (update did not arrive, or an old build undid a new one) is covered in section 10. Fix before handover #10. Rewriting shared resources on start is not applicable, because the bot writes nothing shared on start.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The maintenance banner returns status "Downloaded" (green) although no file was downloaded | Ops thinks the provider is done. The report is missing and nobody follows up. | `bot.py:54-55` |
| 2 | Unexpected errors are not caught: a failed re-login (`login()` raises `RuntimeError`), `NoSuchElementException` from `find_element`, a WebDriver crash or a missing sheet `KeyError`. The run stops, and the remaining providers get no row. There is also a pointless extra re-login after the 3rd `SessionExpired`. | One problem stops the whole day's downloads, with no message saying why | `bot.py:41-43`, `:64`, `:77`, `:96-104`, `:108-110` |
| 3 | No exception or traceback is logged. The first-login failure is not logged. Under Task Scheduler, stderr is lost. | E01, E03 and E09 leave no trace for ops or dev | `bot.py:30-31`, `:86-91`, `:113-114` |
| 4 | Download detection counts *any* new `*.csv`, so a late file from the previous provider is credited to the current one. The files are not renamed to the NPI. | A wrong "Downloaded" is possible, and ops cannot tell which CSV belongs to which provider | `bot.py:63-68` |
| 5 | The log line has no remark (status only) | The log cannot explain an Error | `bot.py:105` |
| 6 | "Login Failed" rows get no colour: `ws.append` at the first-login path skips the fill | The failure is not red like other failures | `bot.py:88-90` |
| 7 | Folders are never created. A missing `data/output` fails the final save and loses the whole report. A missing `data/screens` makes `save_screenshot` return False silently. A missing `logs` fails at import. | Silent loss of the report or screenshots on a fresh machine | `bot.py:20-21`, `:30`, `:59`, `:110` |
| 8 | Only an exact uppercase `"N"` counts as inactive. Rows with a blank NPI are dropped without a message. | Input mistakes are silent | `bot.py:77-78` |
| 9 | Login, grid and download waits are fixed (30 s / 30 s / 60 s) with no retry. The login wait reuses `GRID_WAIT_S`. | A slow portal morning becomes a day of Errors, or "Login Failed" on every row | `bot.py:22-23`, `:44-47`, `:56-60`, `:65-70` |
| 10 | No version string. `requirements.txt` has no pinned versions. | Ops cannot confirm an update, and a package upgrade can break the bot without warning | `bot.py` (none), `requirements.txt:1-3` |
| 11 | No safe stop. Ending the task in Task Scheduler skips the `finally` block, so there is no report and Chrome may be left open. | The only safe stop is closing Chrome, which is not obvious | `bot.py:108-110` |
| 12 | A same-day re-run overwrites `Download_Report_<date>.xlsx` and `data/screens/<NPI>.png` | The first run's evidence is lost on re-run | `bot.py:59`, `:110` |
| 13 | The portal password is stored in plain text in the input workbook (Login!B2) | Credential exposure every time the input is shared | `bot.py:76` |
| 14 | `config.yaml` holds an `api_token` value that the code never uses | An unused secret on disk. Remove it, or rotate it if it is live. | `config.yaml` (key `api_token`) |
| 15 | `BOT_WAS_RUN` marker file is written on every start, and nothing reads it | Ops may wonder what it is. Remove it, or document it. | `bot.py:74` |
tests/evals/results/handoff/h-web-portal-full-r5.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-web-portal/full",
"rep": 5,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"no-history case says so": true,
"misleading status flagged": true,
"session re-login limit": true,
"screenshots folder": true,
"daily schedule": true
},
"cost_usd": 1.1317614,
"subagents": 1,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-full-w76kxbeu\\project",
"reply": "Bot ku `HANDOVER.md` ready: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-full-w76kxbeu\\project\\HANDOVER.md`. Bot-a run pannala, code-um change pannala. Git history, bug log, changelog edhuvum illa, adhanaala history subagent skip pannen. `logs/bot.log`-la oru run (2026-09-02) mattum dhaan irundhuchu.\n\n| Item | Count |\n|---|---|\n| Sections | 13 (template order apdiye) |\n| Exceptions catalogued | 10 (E01\u2013E10) |\n| Failure classes | 11 covered, 1 not applicable (F10: batch gate-o lock-o illa) |\n| Fix before handover | 14 items |\n\n**Fix before handover \u2014 mukkiyamaanadhu:**\n- Portal maintenance-la irundha, file download aagaama irundhaalum row green \"Downloaded\"-nu kaatudhu.\n- Oru error vandhaale full run stop aagidudhu. Andha crash reason enga-yum save aagala.\n- \"Login Failed\" log-la varala, colour-um illa.\n- `data/output`, `data/downloads`, `data/screens` folders-a bot create pannala. Folder illana report-e save aagadhu.\n- Pudhusa endha CSV vandhaalum adha current provider-oda download-aa eduthukkudhu. File-ah provider per rename-um pannala.\n- Grid load aagala-na, download mudiyala-na retry illa.\n- Version number engayum illa.\n- Password input Excel-la plain text-la irukku. `config.yaml`-la use aagaadha oru token irukku, adha remove panni rotate pannanum.\n- Log-la NPI-um provider name-um irukku, log rotate-um aagala.\n- Same day re-run panna andha naal report overwrite aagudhu.\n- Active flag-la capital \"N\" mattum dhaan inactive.\n- Moonaavadhu session expiry-kku apram oru extra login nadakkudhu. Re-login fail aana full run crash aagudhu.\n- `requirements.txt` unpinned.\n\n**Verify panninadhu:** code ezhudhura ellaa status-um Section 6-la irukku. Every Dev row-um enna anuppanum-nu sollidhu. PHI, password, IP edhuvum doc-la illa. file:line Section 13-la mattum dhaan.\n\n**Note:** Unga answers-a apdiye use pannen: escalation #rpa-support Teams channel, ops-ku database access illa, clear illaadha statuses-ku en best reading. Ops paakka mudiyaadha crash reason Section 13-la poyirukku. Doc-la sollirukkura `data/` folders repo-la illa, bot-um create pannala. Adhanaala avai Section 3-la \"run-ku munnaadi irukkanum\" checklist-a vandhirukku.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel) | Ops access: which of the bot's places ops can open - the app's screens, the output folder, the log folder, the database | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear",
"stderr": ""
}
tests/evals/results/handoff/h-web-portal-full-r5.md
+184
View File
@@ -0,0 +1,184 @@
# Payer portal downloader - ops handover
Version: not set (the bot has no version number, see section 10) · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every morning at 07:00 Windows Task Scheduler starts the bot. It signs in to the payer web portal once. Then, for each provider in the input Excel `data/input/providers.xlsx`, it opens that provider's remittance page and downloads the remittance report as a CSV file. The input file has two sheets: **Login** (portal username and password) and **Providers** (NPI, name, active flag). The bot produces three things:
- a report Excel, `data/output/Download_Report_<YYYYMMDD>.xlsx`, with one row per provider and its Status and Remarks
- the CSV files, in `data/downloads/`
- a screenshot in `data/screens/` when a provider's report page does not load
The bot drives one app: a **Google Chrome window** that it opens itself.
## 2. How it works
```
Windows Task Scheduler (07:00 daily)
└─ starts → bot (no window of its own) ──drives──> its own Chrome window → payer web portal
│ (do not touch while it runs)
├─ reads → data/input/providers.xlsx
├─ writes → data/downloads/*.csv (one CSV per downloaded provider)
├─ writes → data/screens/<NPI>.png (only when a page does not load)
├─ writes → data/output/Download_Report_<YYYYMMDD>.xlsx (saved at the very end)
└─ writes → logs/bot.log (one line per provider)
```
- **Task Scheduler** starts the bot. The bot has no screen or window of its own.
- **The bot** works in the background and controls a Chrome window that it opens. It closes that window when it finishes.
- **The payer web portal** is the website that Chrome shows.
- **The input Excel** tells the bot who to sign in as and which providers to download.
- **`data/downloads/`** gets one CSV per downloaded provider. The file keeps the name the portal gives it; it is **not** renamed per provider.
- **`data/screens/`** gets one picture per provider whose report page did not load. It is named by NPI and replaced every day.
- **The report Excel** is written only when the run ends. A run that is still going, or one that was killed, has no report.
- **`logs/bot.log`** gets one line per provider (`<NPI> <name> -> <status>`) and one warning line each time the bot has to sign in again.
**One run, step by step:**
1. Read the input Excel.
2. Open Chrome.
3. Sign in to the portal (wait up to 30 s for the dashboard).
4. For each provider:
- inactive → skip
- otherwise open its remittance page → wait up to 30 s for the report grid → click Export CSV → wait up to 60 s for a new CSV file
5. Close Chrome and save the report Excel.
## 3. Before a run
- [ ] Google Chrome is installed and up to date on the machine. Python and the bot's packages are installed. If Chrome cannot start, there is no report at all (E02).
- [ ] These folders exist inside the bot folder: `data/input`, `data/output`, `data/downloads`, `data/screens`, `logs`. **The bot does not create them.** Without `data/output` there is no report (E02). Without `data/screens` the screenshot is silently not saved.
- [ ] The settings file `config.yaml` is in the bot folder.
- [ ] Input file `data/input/providers.xlsx` has sheets named exactly **Login** and **Providers**.
- [ ] **Login** sheet: the username is in cell A2 and the password in B2. Both must be filled. If either is blank, the run stops before any provider (E02).
- [ ] **Providers** sheet: one provider per row, starting at row 2, with:
- Column A = NPI. A row with a blank NPI is ignored and does **not** appear in the report.
- Column B = provider name.
- Column C = active flag. Only a capital **N** means inactive. Good: `N`. Bad: `n`, `No`, `inactive`. Those, and a blank cell, all count as *active*, so the provider is downloaded.
- [ ] Move or archive yesterday's CSVs from `data/downloads/` if you need to tell today's files apart. The bot never cleans that folder.
## 4. During a run
- Don't click, type in, close or resize the Chrome window the bot opened. It belongs to the bot. Closing it stops the whole run (E03).
- Don't put any CSV file into `data/downloads/` while the bot runs. The bot counts **any** new CSV there as the current provider's download (E10).
- Don't run the bot twice on the same day without a reason. A second run downloads every provider again and **overwrites** that day's report.
- **To stop the bot:**
1. Open Task Scheduler, right-click the bot's task, and choose **End**.
2. Then close any Chrome window the bot left open.
3. A run stopped this way writes **no report**.
- **A run looks stuck** when the Chrome page has not changed for more than 5 minutes.
- One provider normally takes under 2 minutes.
- The bot has **no watchdog**: it never restarts itself.
- If one portal page keeps loading, the run stops on its own with a crash after about 5 minutes (E03). Wait that long, then stop it as above.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `data/output/` has `Download_Report_<today>.xlsx`, and its row count matches the providers in the input (minus rows with a blank NPI). No file means E02. Fewer rows means E03. |
| Which providers failed? | The report Excel, Status column: red cells, and white cells ("Login Failed"). **Also** check the Remarks column for "portal maintenance - no file". Those rows are green but nothing was downloaded (E07). |
| Why did one provider fail? | The Remarks column on that provider's row. For "Report grid did not load", open `data/screens/<NPI>.png`. |
| Did the bot already retry? | `logs/bot.log`: search for the NPI, then look just above it for lines `session expired, re-login 1/2/3`. Only an expired session is ever retried. Nothing else is. |
| Was a CSV really downloaded? | `data/downloads/`: compare the number of today's CSVs with the number of rows whose remark is "Report saved". The files are not named per provider. |
| Which version is running? | Not possible today (see section 10). |
| Why did the run crash? | Not recorded anywhere ops can open. Escalate (E02 / E03). |
## 6. Statuses
All statuses are in the report Excel, in the Status column with the Remarks column next to it.
| Status - Remarks | Colour | Meaning | What ops does |
|---|---|---|---|
| Downloaded - "Report saved" | green | A new CSV appeared in `data/downloads/` | nothing (optionally check the CSV count, E10) |
| Downloaded - "portal maintenance - no file" | green | **Needs follow-up.** The portal showed a maintenance banner. **Nothing was downloaded.** | Download it later (E07) |
| No Data - "No remittances in the date range" | green | The portal says there are no remittances for this provider | nothing, unless the provider is expected to have payments (E08) |
| Skipped - inactive provider - "Marked N in the input" | green | Column C was a capital N | nothing |
| Error - "Report grid did not load" | red | The provider's page did not load in 30 s. No retry. | E04 |
| Error - "Download did not finish" | red | Export was clicked but no CSV appeared in 60 s. No retry. | E05 |
| Error - "Session expired" | red | The portal logged the bot out 3 times on this provider | E06 |
| Login Failed - "Check the portal username/password" | **no colour (white)** | Sign-in failed. Every provider gets this row and nothing was downloaded. | E01 |
| *(provider missing from the report)* | - | The NPI was blank in the input, or the run crashed before reaching that provider | E03 / section 3 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Every row says "Login Failed - Check the portal username/password" (white) | After signing in, it waits 30 s for the portal dashboard. If the dashboard never appears, it gives up. | Report Excel only. Nothing in the log. | Business (wrong password) or System (portal slow/down) | Ops | Sign in to the portal by hand with the Login sheet's username and password. If that fails, fix the password in the Login sheet (B2) and re-run. If the portal itself is down, re-run later. | The manual sign-in works but the bot still fails. Send the date and the report file. |
| E02 | No `Download_Report_<today>.xlsx` at all, or a report with only the header row | The bot crashed before or during sign-in. Causes: Chrome could not start, a folder is missing, the input file or a sheet is missing, a blank username or password, or the login page looks different. | Nowhere ops can open. The crash message is lost. | System | Ops, then Dev | Work through the section 3 checklist: Chrome installed, all folders exist, input sheets named right, A2/B2 filled. Re-run by hand. | It still fails after the checklist. Send the date, the time, the input file's sheet names (not the password), and the last lines of `logs/bot.log`. |
| E03 | The report stops partway: later providers are missing (no row at all) | Something the bot does not handle stopped the whole run. Causes: the Export CSV button was missing, a page kept loading for 5 minutes, the bot could not sign in again after being logged out, or someone closed the Chrome window. | The report has rows up to the crash. `logs/bot.log`: the last provider line shows where it stopped. The cause is not recorded. | System | Ops, then Dev | Note the last provider in the report. Re-run once (it downloads everyone again). Or download the missing providers by hand. | It stops at the same provider twice. Send the date, the time, the last NPI in `logs/bot.log`, and the report file. |
| E04 | Error - "Report grid did not load" (red) | It opens the provider's remittance page and waits 30 s for the report grid. If the grid never appears, it saves a screenshot and moves on. | Report Excel. Screenshot `data/screens/<NPI>.png`. Log line `<NPI> ... -> Error`. | System | Ops | Look at the screenshot. Open that provider in the portal by hand. If the report is there, download it by hand or re-run later. | More than a few providers in one run, or the same provider every day. Send the date, the NPIs, and the screenshots. |
| E05 | Error - "Download did not finish" (red) | It clicks Export CSV and checks `data/downloads/` every second for 60 s for a new CSV. None appeared. | Report Excel. Log line `<NPI> ... -> Error`. | System | Ops | **Look in `data/downloads/` first.** The file may have arrived late. If it is not there, download it from the portal by hand. A late file can also be counted for the next provider (E10). | It happens on most providers in a run. Send the date, the time, and the NPIs. |
| E06 | Error - "Session expired" (red) | The portal sent the bot back to its login page. The bot signs in again and retries, 3 attempts in all. | Report Excel. Log lines `session expired, re-login 1/2/3` just before that provider. | System | Bot, then Ops | Download that provider by hand or re-run later. | Several providers in the same run, or every day. Send the date and the log lines. |
| E07 | Downloaded - "portal maintenance - no file" (**green**) | The portal page shows a maintenance banner. The bot records Downloaded and moves on without trying. | Report Excel, Remarks column only. The log says just "Downloaded". | System | Ops | **Filter the Remarks column for "maintenance" after every run.** Re-run after the maintenance window, or download by hand. | Never needed. The fix is in section 13. |
| E08 | No Data - "No remittances in the date range" (green) | The portal shows its "no results" message for that provider | Report Excel | Business | Ops (only if it is unexpected) | If the provider should have payments, check the portal by hand. | The portal shows remittances but the bot says No Data. Send the NPI and the date. |
| E09 | A provider you expected is missing, or "Skipped - inactive provider" for an active provider | A row with a blank NPI is ignored. Only a capital N in column C counts as inactive. | Report Excel (missing row, or the Skipped status) | Business | Ops | Fix the input: fill in the NPI, and use a capital N only for inactive providers. | - |
| E10 | The number of CSVs in `data/downloads/` doesn't match the number of "Report saved" rows, or a CSV holds the wrong provider's data | The bot counts **any** new CSV in the folder as the current provider's download. A late file from the previous provider, or a file someone else dropped there, makes a provider look Downloaded. | Not recorded. Only visible by counting the files and opening them. | System | Ops, then Dev | Open the CSVs and check which provider each belongs to. Download the missing providers by hand. | It happens again. Send the date and the file names. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| The portal logs the bot out (session expired) | Signs in again and retries the same provider | 3 attempts per provider (each sign-in waits up to 30 s) | Row = Error "Session expired", next provider. If the new sign-in itself fails, the whole run stops (E03). |
| Report grid does not load | **No retry.** Screenshot, then the next provider. | 30 s, once | Row = Error |
| CSV does not arrive | **No retry** | 60 s, once | Row = Error |
| Sign-in fails at the start | **No retry** | 30 s, once | Every row = Login Failed |
| Run is stuck or crashes | **Nothing.** No watchdog, no restart, no resume. | - | Ops re-runs by hand. A re-run repeats **every** provider. |
The worst case for one provider is about 3 minutes (three sign-ins plus the waits). Anything longer is not normal.
## 9. Known incidents
There is no change history or bug log for this bot. The only record is `logs/bot.log`, which covers one run.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-09-02 / no version | One provider showed Error, right after a "session expired, re-login" warning | Not recorded: the log doesn't keep the remark. Most likely a slow page right after signing in again. | Follow E04 / E05 / E06 by the remark | Partly |
## 10. Releases and version check
- **How an update arrives:** there is no updater. The developer replaces the bot files in its folder. The next 07:00 run uses the new files.
- **How to confirm it applied:** not possible today. The bot has no version number in its files, its log or its report (see section 13).
- **If you suspect an old build:** ask in #rpa-support for the expected file date of the bot, and compare it with the date modified of the bot's main file in the bot folder.
## 11. Escalating to dev
Post in the **#rpa-support** Teams channel and send:
- the run date and the time it failed
- the exception ID from section 7
- the report file `data/output/Download_Report_<date>.xlsx`
- the relevant lines of `logs/bot.log`
- for E04, the screenshot `data/screens/<NPI>.png`
Never send: the portal password, the Login sheet, `config.yaml`, or provider or patient data in chat. Attach files rather than pasting their rows.
## 12. Failure-class check
- **F1** (missing prerequisite) covered in section 3 (Chrome, folders, input sheets, Login cells) and E02.
- **F2** (wrong window or focus) covered in section 4: don't touch the bot's Chrome window. The bot does not match windows by title or take focus, because it opens its own Chrome.
- **F3** (target app slower or different) covered in E03, E04, E05 and section 8. Grid and download waits are not retried (section 13).
- **F4** (input read differently) covered in section 3 (capital N only, blank NPI ignored, exact sheet names) and E09.
- **F5** (marked done when nothing was done) covered in E07 and E10, and section 13 items 1 and 6.
- **F6** (action done but not confirmed) covered in E05: check `data/downloads/` before downloading again. A late file can be credited to the next provider (E10).
- **F7** (business exception shown as success) covered in section 6 ("portal maintenance" shows green), E07, and section 13 item 1.
- **F8** (endless re-runs or watchdog kill): there is no rerun loop, so it cannot loop. There is also no watchdog: covered in sections 4 and 8, and section 13 item 2.
- **F9** (stopping the wrong way) covered in section 4 (End the task in Task Scheduler, then close Chrome).
- **F10** not applicable: there is no batch, completion gate or upload lock; each provider is independent. The nearest risk, a crash stopping all later providers, is E03.
- **F11** (cause hidden from ops) covered in section 5 ("why did the run crash" is not recorded) and section 13 items 2, 3 and 4.
- **F12** (update didn't arrive) covered in section 10 (no version shown) and section 13 item 8. The bot rewrites no shared resources on start.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The maintenance banner returns "Downloaded" (green) with no file | Ops thinks the report was downloaded | `bot.py:54-55` |
| 2 | No per-provider exception handling, and no watchdog. A missing Export button, a page-load timeout (driver default about 5 min) or a failed re-login ends the whole run. | Later providers are silently missing (E03) | `bot.py:51`, `bot.py:64`, `bot.py:92-107`, `bot.py:102` |
| 3 | Crash messages go to stderr only, which is lost under Task Scheduler. No traceback reaches `bot.log`. | The cause of E02/E03 is unrecoverable | `bot.py:30`, `bot.py:113-114` |
| 4 | "Login Failed" is never logged and its row is never coloured (the colour map has red, but it isn't applied). The log line never includes the remark. | Ops can't spot it by colour. The log can't tell the Error types apart (section 9). | `bot.py:88-91`, `bot.py:105` |
| 5 | The bot never creates `data/output`, `data/downloads` or `data/screens`. If `data/output` is missing, the report save in `finally` fails and hides the original error. A screenshot to a missing folder fails silently. | No report at all, and the cause is masked | `bot.py:20-21`, `bot.py:59`, `bot.py:108-110` |
| 6 | The download check accepts any new `*.csv`. Files are not renamed per provider. | A wrong or late file is credited to the wrong provider (E10). CSVs can't be tied to providers. | `bot.py:63-68` |
| 7 | Grid-load and download failures are never retried | Avoidable red rows that ops has to handle by hand | `bot.py:56-60`, `bot.py:65-70` |
| 8 | No version string anywhere (report, log, file) | Ops can't confirm which build ran | `bot.py:1-2` |
| 9 | The portal password is stored in plain text in the input Excel. `config.yaml` holds an access token that the code never uses: remove it and rotate it. | Credential exposure | `bot.py:76`, `config.yaml` |
| 10 | NPI and provider name are written to `bot.log` on every line, and the log is never rotated | Sensitive data piles up in an ops-visible file | `bot.py:30`, `bot.py:105` |
| 11 | A same-day re-run overwrites the report. Screenshots are overwritten daily. Downloads are never cleaned. There is no resume. | A re-run after E03 loses the first report and re-downloads everyone | `bot.py:59`, `bot.py:110` |
| 12 | The active flag is case-sensitive: only "N" counts | Inactive providers marked "n" are downloaded | `bot.py:77` |
| 13 | After the 3rd session expiry the bot signs in once more before giving up. A failed re-login is not caught. | Extra 30 s wait, or a whole-run crash | `bot.py:96-104` |
| 14 | `requirements.txt` is unpinned. A Selenium or openpyxl upgrade can change behaviour between machines. | "Works on one machine only" | `requirements.txt` |
tests/evals/results/handoff/h-web-portal-screens-r1.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-web-portal/screens",
"rep": 1,
"passed": true,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"no-history case says so": true,
"misleading status flagged": true,
"session re-login limit": true,
"screenshots folder": true,
"daily schedule": true
},
"cost_usd": 1.1106014,
"subagents": 1,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-screens-_682nhjg\\project",
"reply": "`HANDOVER.md` ready, bot repo root-la irukku. Bot-a run pannala, code-um change pannala. Bot code, README, config and log mattum read panniten.\n\n| Item | Count |\n|---|---|\n| Sections | 13, template order-la |\n| Exceptions catalogued | 11 (E01\u2013E11) |\n| Statuses | 8. Code ezhudhura ella status + remark combination-um irukku |\n| Failure classes | 11 covered, 1 not applicable (F10: batch, queue, lock edhuvum illa) |\n| Fix before handover | 16 items |\n\n**Top Fix before handover items:**\n- Portal maintenance-la file download aagala, aana row green \"Downloaded\"-nu kaatudhu.\n- Folder-la pudhusa vara endha CSV-yum current provider-oda download-nu count aagudhu. Late-aa vandha file adutha provider-ku credit aagidum.\n- \"Login Failed\" rows-ku colour illa, log-layum varadhu.\n- Bot crash aana, meedhi providers report-la missing. Reason report-layum log-layum illa.\n- Bot-ku version number illa, watchdog illa, Stop button illa.\n- Folders-a bot create pannadhu. `data/output` illana report save aagadhu, `logs` illana bot start-e aagadhu.\n\n**Unga answers-a naan eppadi eduthukitten:**\n- \"No logs folder\"-a, ops-ku logs folder open panna access illa-nu eduthukitten. Repo-la `logs/bot.log` irukku, so log-a \"Where to look\"-la podala, Fix item-aa pottirukken.\n- Bot-ku sondha screen illa, Chrome mattum dhaan drive pannudhu. So \"bot's own screens\"-a `data/screens/` screenshots-nu eduthukitten.\n\n**Code mattum padichu ezhudhinadhu, test pannala:**\n- Bot-a stop panna, adhu open panna Chrome window-a close pannanum. Appo idhuvarai mudincha providers-oda partial report save aagum.\n- History illa: git, changelog edhuvum illa. Known incidents-la log-la irukkura 2026-09-02 run mattum dhaan irukku.\n\nQUESTIONS: Who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Are there any statuses or exceptions whose meaning the code doesn't make clear?",
"stderr": ""
}
tests/evals/results/handoff/h-web-portal-screens-r1.md
+184
View File
@@ -0,0 +1,184 @@
# Payer portal downloader - ops handover
Version: none (the bot has no version number, see section 10) · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Every day at 07:00 (Windows Task Scheduler) the bot signs in to the payer web portal and downloads the remittance report of each provider as a CSV file. It reads the input Excel `data/input/providers.xlsx`: the portal username and password are on the **Login** sheet, and the providers are on the **Providers** sheet with their NPI, name and active flag. It produces one CSV per provider in `data/downloads/`, and one result workbook, `data/output/Download_Report_<YYYYMMDD>.xlsx`, with a Status and a Remarks column for each provider. When the report page fails to load it saves a screenshot to `data/screens/<NPI>.png`. The only app it drives is **Google Chrome**: it opens its own Chrome window.
## 2. How it works
```
Windows Task Scheduler (07:00 daily)
└─ starts → bot (no window of its own) ──drives──> its own Chrome window → payer web portal
│ (do not touch this window while it runs)
├─ reads → data/input/providers.xlsx
├─ writes → data/downloads/*.csv (one CSV per provider)
├─ writes → data/screens/<NPI>.png (only when a page did not load)
├─ writes → data/output/Download_Report_<YYYYMMDD>.xlsx
└─ writes → logs/bot.log (ops cannot open this)
```
- **Task Scheduler** starts the bot every morning. You will not see a bot window, only Chrome.
- **The bot** works through the Providers sheet from top to bottom, one provider at a time.
- **The Chrome window** is opened and closed by the bot. Any click in it can break the run.
- **The output workbook** is the place where ops checks the result. It is written once, at the end of the run.
- **The log** records one line per provider and the re-logins. Only the dev team can read it.
**One run, step by step:**
1. Read the input Excel: the login details and the list of providers.
2. Open Chrome and sign in to the portal. If the sign-in fails, every provider is marked Login Failed and the run ends.
3. For each provider: open the provider's remittance page.
4. If the portal sent the bot back to the sign-in page, sign in again and retry the provider.
5. Wait for the report grid. If the page says there are no results, the provider is marked No Data.
6. Click the CSV export and wait for a new CSV file to appear in `data/downloads/`.
7. Write the provider's row to the output workbook and go to the next provider.
8. Close Chrome and save `Download_Report_<YYYYMMDD>.xlsx`.
## 3. Before a run
- [ ] Google Chrome is installed on the machine.
- [ ] These folders exist next to the bot: `logs`, `data/input`, `data/downloads`, `data/screens`, `data/output`. The bot does not create any of them.
- Without `logs`, the bot stops before doing anything.
- Without `data/output`, no report is saved (E02).
- Without `data/screens`, screenshots are lost without any message.
- [ ] The input file is at `data/input/providers.xlsx` and still has the sheet names **Login** and **Providers**.
- [ ] **Login sheet:** username in A2, password in B2. When the portal password changes, update B2, or the next run gives Login Failed for everyone (E01).
- [ ] **Providers sheet**, one provider per row, starting at row 2: column A = NPI, B = provider name, C = active flag.
- Every row needs an NPI in column A. A row with column A empty is dropped without a message (E10).
- Good: `1234567890 | Example Clinic | Y`. Bad: `(empty) | Example Clinic | Y`.
- [ ] **Active flag:** to skip a provider, put a capital **N** in column C, and nothing else.
- Good: `N`.
- Bad: `n`, `No` or `N ` (with a trailing space). The bot treats these as active and downloads the provider (E09).
- [ ] Don't run the bot twice on the same day unless you mean to. A second run replaces that day's report and downloads every CSV again (E11).
- [ ] You can't check which version is running (see section 10).
## 4. During a run
- Don't click, type in, or close the Chrome window the bot opened.
- Don't copy, move or open files in `data/downloads/` while the bot runs. The bot counts **any** new CSV appearing there as the current provider's download (E08).
- **To stop the bot:** close the Chrome window it opened.
- The run then ends after at most about a minute, and the report is saved with the providers done so far. The rest are missing (E03).
- This is our best reading of the code. It has not been tested.
- There is no Stop button.
- **A run looks stuck** when the Chrome window stays on the same page for more than 10 minutes.
- Each step normally gives up by itself within 60 seconds.
- There is no watchdog, so a page that never finishes loading can hold the run.
- In that case, stop the bot as described above and raise a ticket.
## 5. Where to look when something goes wrong
Ops can open the output files and folders below. The log file and Task Scheduler history are for the dev team only.
| Question | Where to look |
|---|---|
| Did the run finish? | `data/output/` has `Download_Report_<today's date>.xlsx`, and it has one row per provider in the input. No file means E02. Fewer rows means E03. |
| Which providers failed? | Report workbook, **Status** column: red cells (Error), and **Login Failed** (no colour). Also check every **Downloaded** row whose Remarks say "portal maintenance - no file" (E04). |
| Why did one provider fail? | The **Remarks** column on the same row. For "Report grid did not load", open `data/screens/<NPI>.png`. |
| Is the CSV there? | `data/downloads/`. Files keep the portal's own names, not the provider's, so you can't tell from the file name which provider a file belongs to. |
| Did the bot already retry? | It retries only after the portal signs it out, and you can only see this in the log. If the Remarks say "Session expired", it already tried 3 times. |
| Which version is running? | You can't check this (section 10). |
## 6. Statuses
The colour is the fill of the Status cell in the report workbook.
| Status | Remarks | Colour | Meaning | What ops does |
|---|---|---|---|---|
| Downloaded | Report saved | green | A new CSV appeared in `data/downloads/` after the export click | Nothing. If the CSV count looks wrong, see E08. |
| Downloaded | portal maintenance - no file | green | **Nothing was downloaded.** The portal showed a maintenance banner. | **Needs follow-up.** Treat it as not downloaded and get it re-run after the maintenance window (E04). |
| No Data | No remittances in the date range | green | The portal shows no remittances for this provider in its default date range | Nothing |
| Skipped - inactive provider | Marked N in the input | green | Column C had `N`, so the bot did not open this provider | Nothing, unless the provider should be active |
| Error | Report grid did not load | red | The report page did not appear within 30 s | E05 |
| Error | Download did not finish | red | No new CSV appeared within 60 s after the export click | E06 |
| Error | Session expired | red | The portal kept signing the bot out, 3 times in a row for this provider | E07 |
| Login Failed | Check the portal username/password | **no colour** (white) | The first sign-in failed. Every provider gets this row and nothing is downloaded. | E01 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Every row says **Login Failed** - "Check the portal username/password" (no colour) | After clicking Sign in, it waits 30 s for the portal dashboard. If the dashboard never appears, it stops. A wrong password and a slow portal look the same. | Report workbook, every row. Nothing is in the log. | Business or System | Ops | Sign in to the portal by hand with the Login sheet details. If they fail, fix the password in the Login sheet (B2). If they work, the portal was slow, so request a re-run. | The details work by hand but the next run fails again. Send: report file name, run date. |
| E02 | No `Download_Report_<today>.xlsx` in `data/output/` | Nothing. The bot stopped before it could save: a missing folder, missing input file, Chrome would not start, or the output folder is missing. | Nowhere ops can see. Only the dev team can see it, in Task Scheduler history. | System | Dev | Check the folders listed in section 3 exist, and that the input file is in place. | Always. Send: run date and time, and which folders exist under `data/`. |
| E03 | The report has **fewer rows** than the input. The last providers are missing, with no Error row. | Nothing. An unexpected problem stopped the whole run: the sign-in page or the export button changed, a re-login after a sign-out failed, or the Chrome window was closed. The report is saved with the rows done so far. | Missing rows in the report. The cause is not in the report or the log. | System | Dev | Note the first missing provider. A re-run redoes **all** providers and replaces today's report (E11). | Always. Send: report file name, the first missing provider's NPI, run time, and whether anyone touched Chrome. |
| E04 | **Downloaded** (green) with Remarks "portal maintenance - no file" | The portal page showed a maintenance banner. The bot writes Downloaded and moves on, without retrying. | Report workbook | System | Ops | **No file was downloaded**, although the row is green. Re-run after the portal maintenance ends, then check the row again. | Every provider shows it for 2 days in a row. Send: report file names. |
| E05 | **Error** - "Report grid did not load" | It waits 30 s for the report grid on the provider's page. It does not retry. | Report workbook, plus a screenshot `data/screens/<NPI>.png` | System | Ops | Open the screenshot. If it shows a portal error or a slow page, request a re-run. If it shows a page you don't recognise (a changed layout, or a new prompt), escalate. | The same provider fails on 2 runs in a row. Send: report file name, NPI, screenshot. |
| E06 | **Error** - "Download did not finish" | After clicking Export it checks `data/downloads/` every second for 60 s. No new CSV appeared. No screenshot is taken. | Report workbook | System | Ops | The file may have arrived late. A late file can be counted as the **next** provider's download (E08). Check the next provider's row, and request a re-run. | It happens on several providers in one run, or on the same provider twice. Send: report file name, NPIs, run time. |
| E07 | **Error** - "Session expired" | The portal sent the bot back to the sign-in page. It signed in again and retried, 3 times in total for this provider. | Report workbook. The re-logins appear only in the log. | System | Bot, then Ops | The bot already retried. Request a re-run. | It happens on 2 runs in a row. Send: report file name, NPI, run time. |
| E08 | A provider shows **Downloaded - Report saved** but its CSV is missing, or the CSV holds another provider's data. Or the number of CSVs doesn't match the number of Downloaded rows. | The bot counts **any** new CSV in `data/downloads/` as the current provider's download. Files are not named per provider. | Not recorded. The row looks green and normal. | System | Dev | Don't trust the CSV-to-provider match for that run. Note which providers look wrong. | Always. Send: report file name, run date, the list of CSV files with their times. |
| E09 | A provider marked inactive was downloaded anyway | Only a capital `N` in column C counts as inactive. `n`, `No` and `N ` are treated as active. | Report workbook: the row is not "Skipped - inactive provider" | Business | Ops | Change column C to exactly `N`. | Never - this is an input fix. |
| E10 | A provider from the input is missing from the report, while the rows after it are present | A row with an empty NPI (column A) is dropped without a message. | Not recorded | Business | Ops | Fill in the NPI in column A and re-run. | The NPI is filled in and the row is still missing. |
| E11 | Duplicate CSVs such as `name (1).csv`, or this morning's report changed | A second run on the same day downloads every CSV again (Chrome adds "(1)", "(2)") and **replaces** that day's report file. | `data/downloads/` and `data/output/` | System | Ops | Copy the day's report somewhere else before a same-day re-run. Use the newest copy of each CSV. | Never, unless files are missing. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Portal signs the bot out | Signs in again immediately and retries the provider | 3 tries per provider, no pause between them | Row = Error "Session expired", next provider. If one of these re-logins fails, the whole run stops (E03). |
| Sign-in at the start | Waits for the dashboard | 30 s, no retry | Every row = Login Failed, run ends (E01) |
| Report page slow | Waits for the grid | 30 s, no retry | Row = Error "Report grid did not load", next provider |
| CSV slow to download | Checks the downloads folder every second | 60 s, no retry | Row = Error "Download did not finish", next provider |
| Portal maintenance | Nothing | - | Row = Downloaded "portal maintenance - no file" (E04) |
| Run stuck, or crashed | Nothing: no watchdog, no restart, no resume | - | Ops stops it (section 4) and escalates. A re-run starts again from the first provider. |
## 9. Known incidents
There is no change history, bug log or release notes for this bot. The only trace of past runs is the log, which shows one run.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-09-02 / no version | One provider in Error, right after the portal signed the bot out and it signed in again | The log does not record why. The timing fits "Report grid did not load" or "Download did not finish". | Follow E05 / E06 | Partly |
## 10. Releases and version check
- How an update arrives: there is no updater. A developer replaces the bot's files on this machine.
- How to confirm it applied: you can't. The bot has no version number in its files or its output.
- If you suspect an old build: raise a ticket, and ask the dev team to confirm which files are on the machine.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue. Send:
- the run date and time
- the exception ID from section 7
- the report file name (`Download_Report_<YYYYMMDD>.xlsx`)
- the NPIs of the affected providers
- the screenshot from `data/screens/`, if there is one
- whether anyone touched the machine or the Chrome window during the run
Never send: the portal password, the Login sheet, `config.yaml`, the CSV contents, or patient data in chat or in the ticket text.
## 12. Failure-class check
- **F1** Missing prerequisite on the ops machine: covered in section 3 (Chrome, folders, input file and sheet names) and E02.
- **F2** Wrong window or focus: covered in section 4. The bot drives its own Chrome window by handle and does not match windows by title, so the only risk is someone touching that window or its downloads folder (E08).
- **F3** Target slower or different than expected: covered in E01, E05, E06 and E03 (a changed page layout), with the limits in section 8.
- **F4** Input read differently than ops writes it: covered in section 3 input rules, E09 and E10.
- **F5** Marked done when nothing was done: covered in E04 and E08, and Fix before handover #1 and #2.
- **F6** Action happened but not confirmed: covered in E06. A late CSV is not recorded and can be credited to the next provider. The bot only reads from the portal, so a re-run cannot double-post anything, but it does duplicate files (E11).
- **F7** Business exception shown as success: covered in section 6 ("portal maintenance" shown as green Downloaded), E04, and Fix before handover #1.
- **F8** Endless re-runs or watchdog: covered in section 8. There is no re-run loop. There is also no watchdog, so a hang is covered in section 4 and Fix before handover #13.
- **F9** Stopping the bot the wrong way: covered in section 4 (close the bot's Chrome window) and Fix before handover #13.
- **F10** One stuck item blocks the queue: not applicable. There are no batches, queues or locks. Each run is independent, and a failed provider never blocks another one or the next day's run.
- **F11** Cause hidden from ops: covered in section 5, and Fix before handover #3, #4, #5 and #6. The log and the crash messages are outside ops access.
- **F12** Update didn't arrive or old build: covered in section 10 and Fix before handover #14. There is no version number and no updater. The bot writes nothing shared on start.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | A maintenance banner is recorded as `Downloaded` (green) with no file | Ops believes the report was fetched | `bot.py:54-55` |
| 2 | Any new `*.csv` in the downloads folder counts as success for the current provider, and files are never renamed or linked to the NPI | A late download is credited to the wrong provider, and CSVs can't be matched to providers | `bot.py:63-68`, `bot.py:80` |
| 3 | `Login Failed` rows get no fill (the RED colour is never applied on that path) and are not logged | They don't stand out in the report, and there is no trace in the log | `bot.py:88-91`, `bot.py:27-28` |
| 4 | Unhandled errors end the run with no row for the current or remaining providers. The traceback goes to stderr only, not to `bot.log`. | Ops sees missing rows with no reason (E03) | `bot.py:41-43`, `bot.py:64`, `bot.py:102`, `bot.py:85-110` |
| 5 | Chrome start-up and input loading happen before the `try`/`finally`. A missing `data/output` folder makes the save in `finally` fail and hides the original error. | No report at all, and no reason (E02) | `bot.py:75-81`, `bot.py:110` |
| 6 | Remarks are not written to the log, and the log is outside ops access | The reason for an Error is only in the report, and the retries can't be seen | `bot.py:105`, `bot.py:101` |
| 7 | Folders are not created (`logs`, `data/downloads`, `data/screens`, `data/output`). A screenshot save into a missing folder fails silently. | The bot stops at start-up, or loses evidence | `bot.py:30`, `bot.py:59`, `bot.py:110` |
| 8 | Session-expiry retry: `RELOGIN_TRIES = 2` gives 3 tries and 3 re-logins (the last one is wasted), with no pause between tries | The retry limit is misleading, and it hammers the portal | `bot.py:24`, `bot.py:96-104` |
| 9 | The sign-in failure can't tell a wrong password from a slow portal | Ops changes a password that was correct | `bot.py:44-47` |
| 10 | No retry on a grid timeout or a download timeout. A screenshot is taken only for a grid timeout. | Short portal hiccups become Errors, and download failures leave no evidence | `bot.py:56-70` |
| 11 | The active flag matches only an exact `"N"` (no trimming, case-sensitive) | Providers ops meant to skip are downloaded | `bot.py:77` |
| 12 | Rows with an empty NPI are dropped silently. A same-day re-run overwrites the report and duplicates CSVs. | Missing providers and lost reports go unnoticed | `bot.py:78`, `bot.py:110` |
| 13 | There is no watchdog, no page-load timeout and no Stop button | A hung page holds the run indefinitely, and the only way to stop it is to close Chrome | `bot.py:51`, `bot.py:81` |
| 14 | No version string anywhere, and no update mechanism | Ops can't confirm which build runs | whole repo |
| 15 | Portal credentials are stored in plain text in the input Excel. An unused `api_token` sits in plain text in `config.yaml`. Provider names are written to the log. | Secrets and identifiers are exposed to anyone with folder access | `bot.py:76`, `config.yaml:2`, `bot.py:105` |
| 16 | Every run writes an unexplained `BOT_WAS_RUN` marker file in the bot folder | Unclear purpose, and leftover clutter | `bot.py:74` |
tests/evals/results/handoff/h-web-portal-screens-r2.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-web-portal/screens",
"rep": 2,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": false,
"every status listed": true,
"catalogue: 4+ complete rows": true,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"no-history case says so": true,
"misleading status flagged": true,
"session re-login limit": true,
"screenshots folder": true,
"daily schedule": true
},
"cost_usd": 0.7157622,
"subagents": 0,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-screens-j_6fk6py\\project",
"reply": "**UNVERIFIED** (doc mattum ezhuthinen. Rule padi bot-a run pannala; ellaame code padichu ezhuthinathu.)\n\n`HANDOVER.md` repo root-la ready: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-screens-j_6fk6py\\project\\HANDOVER.md`\n\n| Check | Result |\n|---|---|\n| Sections | 13, template padi |\n| Exceptions | 9 (E01\u2013E09) |\n| Failure classes | 12/12 covered. F10 partly not applicable (queue illa; mid-run stop-a E07-la cover pannirukken) |\n| Statuses | 8 status/remark combos + \"row missing\" + \"file missing\", ellaam section 6-la iruku |\n| Secrets scan | Token, provider names, log-la irundha NPIs, IPs: doc-la onnum illa. file:line refs section 13-la mattum |\n\n**Fix before handover (14 items), mukkiyamaanavai:**\n1. Portal maintenance-ah irundha \"Downloaded\" green-nu kaattum, aana file download aagaathu (`bot.py:54`)\n2. Late-aa vara CSV adutha provider-oda download-aa count aagidum (`bot.py:63-70`)\n3. Re-login fail aanaa run silent-aa stop aagum; meedhi providers report-la varave varaadhu (`bot.py:96-102`)\n4. `config.yaml`-la commit aana API token iruku (value-a copy pannala)\n5. Version string, watchdog rendum illa. Same day re-run pannaa morning report overwrite aagidum\n\nNotes:\n- Repo-la `bot.py` oru file mattum thaan, git history-um illa. Adhanaala subagents launch pannala, naane direct-aa padichen. Incidents section-la `logs/bot.log`-la irundha oru event mattum thaan iruku.\n- `data/screens` and `data/downloads` folders-ai ops open panna mudiyura \"output files\"-aa eduthukitten. Ops-ku access illana, section 5-a maathanum.\n\nQUESTIONS: Escalation: who ops calls for a dev issue (team or role, not a personal phone number), and how (ticket, chat channel) | Ops access: which of the bot's places ops can open - the app's screens, the output folder, the log folder, the database | Unknowns: any statuses or exceptions whose meaning the code doesn't make clear",
"stderr": ""
}
tests/evals/results/handoff/h-web-portal-screens-r2.md
+165
View File
@@ -0,0 +1,165 @@
# Payer portal downloader - ops handover
Version: not versioned (no version string in the project, see section 10) · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Every day at 07:00 (Windows Task Scheduler) the bot opens Chrome, signs in to the payer web portal, and downloads each provider's remittance report as a CSV file. It reads the input Excel `data/input/providers.xlsx`: the **Login** sheet (portal username and password) and the **Providers** sheet (one provider per row: NPI, name, Active). It produces one result file per day, `data/output/Download_Report_<YYYYMMDD>.xlsx`, with a Status and Remarks for every provider, puts the CSVs in `data/downloads/`, and saves a screenshot to `data/screens/` when a report page fails to load. It drives one app: **Google Chrome** (a Chrome window it opens itself).
## 2. How it works
```
Windows Task Scheduler (07:00 daily)
└─ starts → bot (no window of its own) ──drives──> Chrome window → payer portal
│ (do not touch while it runs)
├─ reads → data/input/providers.xlsx (Login + Providers sheets)
├─ writes → data/output/Download_Report_<date>.xlsx (at the end)
├─ writes → data/downloads/*.csv (one per provider downloaded)
└─ writes → data/screens/<NPI>.png (only when a report page fails)
```
- **Task Scheduler** starts the bot at 07:00; nobody needs to click anything.
- **The bot** has no window. You only see the Chrome window it opens; the bot closes it at the end.
- **Chrome** is the browser the bot controls. Clicking or typing in it during a run breaks the run.
- **The result file** is written once, when the run ends (also when the run ends early with a problem). Until then it does not exist.
- **The downloads folder** collects every CSV. The bot decides "downloaded" by noticing a new CSV appear there.
- **The screens folder** gets a picture of the portal page when a report did not load.
**One run, step by step:**
1. Read the input Excel (Login and Providers sheets).
2. Open Chrome and sign in to the portal. If sign-in fails, every provider is marked Login Failed and the run ends.
3. For each provider row, top to bottom:
- Active = `N` → marked "Skipped - inactive provider", nothing is opened.
- Open the provider's remittance page. If the portal has logged the bot out, sign in again and retry.
- Portal shows a maintenance banner → marked Downloaded with "portal maintenance - no file" (**no file was downloaded**, see E05).
- Wait up to 30 s for the report table. Not there → screenshot, Error.
- Portal says no results → No Data.
- Click Export CSV and wait up to 60 s for a new CSV in the downloads folder.
4. Close Chrome and save the result file.
## 3. Before a run
- [ ] Google Chrome is installed and up to date on the bot machine. The bot cannot start without it, and no result file is written (E08).
- [ ] These folders exist: `data/input`, `data/output`, `data/downloads`, `data/screens`. If `data/output` is missing, the result file cannot be saved and the whole day's results are lost (E08).
- [ ] The input file has a sheet named exactly **Login** (username in A2, password in B2) and a sheet named exactly **Providers** (row 1 = headers; from row 2: NPI in column A, name in column B, Active in column C).
- [ ] Providers sheet rules:
- One provider per row. A row with an empty column A is skipped silently and does **not** appear in the result file.
- To skip a provider, put a capital `N` in column C. Bad: `n`, `No`, ` N`, `Inactive` - these all count as **active** and the bot will try to download. Good: `N`. Leave column C empty (or `Y`) for active providers.
- NPI as the plain 10 digits, e.g. `1000000000`. Bad: `NPI 1000000000`, `1000-000-000`.
- [ ] When the portal password changes, update Login!B2 **before 07:00**, otherwise every provider shows Login Failed (E01).
- [ ] Running the bot by hand on the same day **overwrites** that day's result file. Copy or rename the 07:00 result file first.
- [ ] Don't put your own CSV files into `data/downloads/` - the bot counts any new CSV there as "downloaded".
## 4. During a run
- Don't click, type or close tabs in the bot's Chrome window. Don't use that Chrome window for your own browsing.
- Don't open, save or move files in `data/downloads/` while the bot runs.
- Don't keep `providers.xlsx` or today's result file open in Excel during the run. An open result file can block the save at the end (E08).
- To stop the bot: in **Task Scheduler**, right-click the task → **End**. Then close any Chrome window the bot left open. Stopping this way writes **no result file** for that run. Closing Chrome alone also stops it, but the result file may then be missing or cut short.
- There is no stuck detection. Each provider normally takes under 2 minutes (30 s for the page + 60 s for the download, longer if it had to sign in again). If the Chrome window shows the same page for more than 10 minutes, the run is stuck: stop it as above and raise a ticket (E09).
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `data/output/` → is there a `Download_Report_<today>.xlsx`? If not, the run crashed before saving (E08). If it exists, compare its row count with the Providers sheet: fewer rows = the run stopped early (E07). |
| Which providers failed? | Result file → Status column. Red = Error. Login Failed rows have no colour. |
| Why did one provider fail? | Result file → Remarks column (see section 6 and 7). |
| What did the portal show? | `data/screens/<NPI>.png` (only for "Report grid did not load"). |
| Was the file really downloaded? | `data/downloads/` → a CSV with a time just after that provider was processed. The CSV name comes from the portal and does not say which provider it belongs to. |
| Did the bot already retry? | Only for "Session expired": it signed in again up to 3 times before giving up. Nothing else is retried. The retry record is in the bot log, which ops cannot open - ask Dev. |
| Which version is running? | Not shown anywhere (see section 13). |
## 6. Statuses
| Status - Remarks | Colour | Meaning | What ops does |
|---|---|---|---|
| Downloaded - "Report saved" | green | A new CSV appeared in the downloads folder within 60 s | nothing |
| Downloaded - "portal maintenance - no file" | green | **Nothing was downloaded.** The portal showed a maintenance banner. | **Needs follow-up**: re-run this provider later (E05) |
| No Data - "No remittances in the date range" | green | Portal said there is nothing to download | nothing |
| Skipped - inactive provider - "Marked N in the input" | green | Column C was `N`; the bot did not open the portal for it | nothing |
| Login Failed - "Check the portal username/password" | no colour | Sign-in failed at the start; **no provider was processed** (every row gets this, even inactive ones) | E01 |
| Error - "Report grid did not load" | red | Report table didn't appear in 30 s | E02 |
| Error - "Download did not finish" | red | No new CSV within 60 s after Export | E03 - check the downloads folder first |
| Error - "Session expired" | red | Portal kept logging the bot out, even after 3 sign-ins | E04 |
| *(provider missing from the result file)* | - | Run stopped before reaching it | E07 |
| *(no result file for today)* | - | Run crashed before or while saving | E08 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Every row: Login Failed - "Check the portal username/password" | After clicking Sign in, it waits 30 s for the portal dashboard; if it never appears, sign-in counts as failed | Result file, Status/Remarks on every row | Business (wrong password) or System (portal down/slow) | Ops | Sign in to the portal by hand with the Login sheet details. If that fails, fix the password in Login!B2. If it works by hand, wait 30 min and re-run (copy the morning result file first). | Manual sign-in works but the bot still gets Login Failed twice in a row |
| E02 | Error - "Report grid did not load" | Waits 30 s for the remittance table on the provider's page | Result file; picture in `data/screens/<NPI>.png` | System | Ops | Open the screenshot. If it shows a portal error or an empty page, re-run later. If it shows "provider not found" or similar, check the NPI in the input. | Same provider fails on 2 runs and the screenshot shows a normal-looking page |
| E03 | Error - "Download did not finish" | After clicking Export CSV, waits 60 s for a new CSV in the downloads folder | Result file | System | Ops | **Check `data/downloads/` first** - the file may have arrived late. A late file can also be miscounted as the *next* provider's download (see section 13), so check that provider too. If no file: re-run. | Fails again on re-run |
| E04 | Error - "Session expired" | The portal sent the bot back to the sign-in page; it signed in again and retried, 3 times | Result file | System | Bot, then Ops | Re-run later. If many rows show this, the portal is unstable that day. | More than 3 providers with this in one run, or on 2 days in a row |
| E05 | Downloaded (green) - "portal maintenance - no file" | Portal page shows a maintenance banner | Result file | System | **Ops** | Treat as **not downloaded**. Re-run this provider after the maintenance window. Do not trust the green colour. | Banner still there after 24 h |
| E06 | A provider you expected to be skipped was processed, or one you expected is missing | Only a capital `N` in column C means inactive; rows with empty column A are ignored | Result file (row status, or row absent) | Business | Ops | Fix the input per section 3 and re-run if needed | - |
| E07 | Result file has fewer rows than the Providers sheet; the last row looks normal | The portal logged the bot out and the **re-sign-in** failed, or a portal button/field was missing. The bot stops at once; providers after that point are not processed | Result file (rows missing); the cause is only in the console output, which ops cannot see | System | Ops, then Dev | Note the first missing provider. Re-run (copy the morning result file first). | Happens again on re-run - send the result file and the time of the run |
| E08 | No `Download_Report_<today>.xlsx` in `data/output/` after the run should have finished | Chrome couldn't start, the input file couldn't be read, `data/output` is missing, or the result file was open in Excel | Nowhere ops can see | System | Ops, then Dev | Check section 3 (Chrome installed, folders exist, input file present, result file not open). Fix and re-run. | Everything in section 3 is fine and still no file - send the date/time and say whether a Chrome window appeared |
| E09 | Chrome window stuck on the same page for >10 min | No detection - the bot has no watchdog | Nowhere | System | Ops, then Dev | Stop the bot (section 4) and re-run | Stuck again on re-run - send the time and a screenshot of the Chrome window |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Portal logged the bot out | Signs in again and reopens the provider's page | 3 sign-ins per provider (each waits up to 30 s) | Row = Error "Session expired", next provider |
| Sign-in during that retry fails | Nothing - the run stops | 0 retries | Remaining providers missing from the result file (E07) |
| Report table slow | Waits | 30 s, no retry | Row = Error "Report grid did not load", next provider |
| Download slow | Waits | 60 s, no retry | Row = Error "Download did not finish", next provider |
| First sign-in fails | Nothing | 0 retries | All rows Login Failed, run ends |
Nothing else is retried. There is no automatic re-run of failed providers; ops re-runs by hand.
## 9. Known incidents
There is no git history, bug log or release notes for this bot. One event is visible in the bot log:
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-09-02 07:01 (unversioned) | One provider Error right after the portal logged the bot out once | Unknown - the bot log does not record the Remarks text, only the status | Treat per E02-E04 using the Remarks column | Partly |
## 10. Releases and version check
- How an update arrives: there is no launcher or update process. A developer replaces the files in the bot folder.
- How to confirm it applied: not possible today - the bot shows no version anywhere (section 13, item 7).
- If you suspect an old build: raise a ticket and ask Dev which version is in the folder.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue. Send: the date and start time of the run, the exception ID from section 7, the result file `Download_Report_<date>.xlsx`, the NPI of the failed provider, and the screenshot from `data/screens/` if there is one.
Never send: the portal password, the Login sheet, `config.yaml`, or downloaded remittance CSVs (they contain patient payment data).
## 12. Failure-class check
- F1 covered in section 3 (Chrome, folders, sheet names) and E08.
- F2 covered in section 4 (hands off the bot's Chrome window; no files into the downloads folder). The bot does not match windows by title.
- F3 covered in E02, E03, E09 and section 8 (30 s / 60 s waits, no retry).
- F4 covered in section 3 (exact `N`, plain NPI, no empty column A) and E06.
- F5 covered in E05 and section 13 item 1 (maintenance marked Downloaded with no file).
- F6 covered in E03 and section 13 item 2 (late CSV can be credited to the next provider).
- F7 covered in section 6 (maintenance row green, "needs follow-up"), E05 and section 13 item 1.
- F8 covered in section 8 (re-sign-in limited to 3; no auto-rerun loop) and E09 / section 13 item 5 (no watchdog).
- F9 covered in section 4 (stop via Task Scheduler → End, then close Chrome) and section 13 item 6.
- F10 not applicable for queues: each provider is independent and nothing stays Pending across runs. The one blocker - a failed re-sign-in stops all later providers - is E07.
- F11 covered in section 13 items 3, 4 and 8 (crash cause only on console, remarks not logged, CSV not tied to provider).
- F12 covered in section 10 and section 13 item 7 (no version string). No shared resources are rewritten on start.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Maintenance banner returns status "Downloaded" (green) with no file | Ops thinks the report was downloaded | `bot.py:54-55` |
| 2 | Download check counts any new `*.csv` in the shared downloads folder; a late file from one provider is credited to the next. CSV is never renamed to the NPI | Wrong provider marked Downloaded; ops can't tell which CSV is whose | `bot.py:63-70` |
| 3 | Re-sign-in failure inside the retry loop (`RuntimeError` from `login`) and any `NoSuchElementException` are not caught; run stops, remaining providers vanish from the report, traceback goes to console only | Silent partial runs (E07); cause not visible anywhere ops can open | `bot.py:96-102`, `bot.py:41-43`, `bot.py:64` |
| 4 | Crashes before the `try` (input load, Chrome start) produce no report at all, and nothing is logged | No result file, no reason (E08) | `bot.py:75-81` |
| 5 | No watchdog / overall run timeout | A hung page holds the run until someone notices (E09) | `bot.py:85-109` |
| 6 | Ending the task kills Python without closing Chrome or saving the report; `driver.quit()` raising in `finally` would also skip the save | Stop = lost results, orphan Chrome | `bot.py:108-110` |
| 7 | No version string anywhere | Ops can't confirm which build is running | project-wide |
| 8 | Log line records status but not the remark; log file `logs/bot.log` is not reachable by ops | Incident cause can't be reconstructed (section 9) | `bot.py:105` |
| 9 | Result file is named by date only; a same-day re-run overwrites the 07:00 report | Morning evidence lost | `bot.py:110` |
| 10 | Login Failed rows get no colour fill (and inactive providers are also marked Login Failed) | Inconsistent with the colour legend | `bot.py:89-91` |
| 11 | Active check is `!= "N"`: `n`, `No`, ` N` count as active | Providers meant to be skipped are processed | `bot.py:77` |
| 12 | `data/output` (and other data folders) are not created by the bot | Missing folder = whole day's report lost | `bot.py:20-21`, `bot.py:110` |
| 13 | A live-looking API token is committed in `config.yaml` (unused by the code); portal password is stored in plain text in the input Excel | Credential exposure | `config.yaml:2`, `bot.py:76` |
| 14 | `requirements.txt` has no pinned versions | A Selenium update can break the bot without any code change | `requirements.txt:1-3` |
tests/evals/results/handoff/h-web-portal-screens-r3.json
+32
View File
@@ -0,0 +1,32 @@
{
"case": "h-web-portal/screens",
"rep": 3,
"passed": false,
"checks": {
"HANDOVER.md written": true,
"no other file changed": true,
"bot never run": true,
"no placeholders left": true,
"13 sections in order": true,
"used subagents": true,
"every status listed": true,
"catalogue: 4+ complete rows": false,
"F1-F12 all accounted for": true,
"no code refs in ops sections": true,
"fix-before-handover has file refs": true,
"no secrets or personal data": true,
"where-to-look respects ops access": true,
"asks escalation": true,
"asks ops access": true,
"no-history case says so": true,
"misleading status flagged": true,
"session re-login limit": true,
"screenshots folder": true,
"daily schedule": true
},
"cost_usd": 1.2818285999999999,
"subagents": 2,
"work_dir": "C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-screens-w1v4yfhg\\project",
"reply": "`HANDOVER.md` ready \u2014 repo root la irukku: `C:\\Users\\ec10866\\AppData\\Local\\Temp\\handoff-h-web-portal-screens-w1v4yfhg\\project\\HANDOVER.md`. Code mattum padichen, bot-a run pannala, code edhuvum maathala.\n\n**Scan la kandupidichadhu:**\n- Idhu single Python script. Task Scheduler daily 07:00 ku start pannum, Chrome open panni payer portal la irundhu ovvoru provider-oda remittance report-um CSV-a download pannum.\n- Result file: `data/output/Download_Report_<date>.xlsx` (Status + Remarks). Database illa, screen illa, watchdog illa.\n- History romba kammi: log la 2026-09-02 oru run mattum dhaan irukku (session expire aagi oru provider Error).\n- Repo la `data/` folders illa, code-um adha create pannadhu. `data/output` illana andha naal result file full-a pogum.\n\n**Doc summary:**\n\n| Item | Count |\n|---|---|\n| Sections | 13 |\n| Exceptions catalogued | 11 (E01\u2013E11) |\n| Failure classes | 11 cover aachu, 1 N/A (F10: batch gate illa) |\n| Fix before handover | 13 items |\n\n**Fix before handover \u2014 mukkiyamaana items:**\n- Maintenance banner vandha kooda status green \"Downloaded\" nu varudhu, aana file illa.\n- Edhaavadhu pudhu CSV vandhaale success nu edukkudhu, so oru provider file next provider-ku credit aagalaam.\n- Run crash aana meedhi providers result file la silent-a missing. Crash reason endha file-layum record aagaradhilla.\n- Login Failed rows ku colour-um illa, log line-um illa.\n- Folders auto-create aagaradhilla.\n- Retry info log la mattum dhaan irukku, ops adha open panna mudiyaadhu.\n- Version string illa. Watchdog-um illa.\n- Run-a correct-a stop panna vazhi illa (stop panna andha run-oda result file save aagaadhu).\n- Security: password input Excel la irukku, use aagaadha API token config la plain text-a irukku, provider names log la poguthu.\n\n**Neenga confirm pannanum:**\n- \"No logs folder\" nu sonnadha \"ops ku logs folder access illa\" nu eduthukitten, yen-na repo la `logs/bot.log` irukku. Log la mattum irukkura info ellam dev kitta ticket la ketka sollirukken.\n- `data/screens` screenshots-um, `data/downloads` CSVs-um ops paarkalaam nu assume pannen.\n- Ops ku Task Scheduler access irukkaa nu theriyala, so stop/re-run-ku \"illana RPA Support ticket podunga\" nu ezhudhirukken.\n\nQUESTIONS: Escalation: who does ops call for a dev issue (team or role), and how (ticket, chat channel)? | Ops access: which of the bot's places can ops open - the app's screens, the output folder, the log folder, the database? | Unknowns: what do the statuses or exceptions mean whose meaning the code doesn't make clear (e.g. \"Downloaded - portal maintenance - no file\", uncoloured \"Login Failed\")?",
"stderr": ""
}
tests/evals/results/handoff/h-web-portal-screens-r3.md
+181
View File
@@ -0,0 +1,181 @@
# Payer portal downloader - ops handover
Version: not versioned (see section 10) · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue
## 1. What the bot does
Every day at 07:00, Windows Task Scheduler starts the bot. It signs in to the payer web portal and downloads each provider's remittance report as a CSV file. It reads the input Excel file `data/input/providers.xlsx`. The **Login** sheet holds the portal username and password, and the **Providers** sheet lists one provider per row (NPI, name, active flag). The bot produces:
- one CSV per provider in `data/downloads/`
- a daily result file, `data/output/Download_Report_<YYYYMMDD>.xlsx`, with one row per provider (Provider, NPI, Status, Remarks)
- a screenshot in `data/screens/<NPI>.png` when a provider's report page fails to load
It drives one app: a **Google Chrome** window that it opens itself.
## 2. How it works
```
Windows Task Scheduler (07:00 daily)
└─ starts → Downloader (no window of its own) ──drives──> Chrome → payer portal
│ (do not touch while it runs)
├─ saves → data/downloads/*.csv (the downloaded reports)
├─ saves → data/screens/<NPI>.png (only when a page fails to load)
└─ saves → data/output/Download_Report_<YYYYMMDD>.xlsx (at the very end)
```
- **Task Scheduler** starts the bot at 07:00. Nothing appears on screen except Chrome.
- **The downloader** runs in the background. It has no window, no Stop button and no progress display.
- **Chrome** is opened and closed by the bot. Chrome shows a bar saying it is controlled by automated software. That is the bot's window.
- **The result file** is written only once, when the run ends. It does not exist while the run is still going.
**One run, step by step:**
1. Read the input Excel (Login and Providers sheets).
2. Open Chrome and sign in to the portal.
3. For each provider row:
- If the provider is marked inactive, skip it.
- Otherwise open its remittance page, wait for the report table, click "export CSV" and wait for the file to arrive.
4. Close Chrome and save the result file.
## 3. Before a run
- [ ] Google Chrome is installed and opens normally on the bot machine. If Chrome can't start, no result file is produced for the day (E02).
- [ ] These folders exist inside the bot folder: `data/input`, `data/downloads`, `data/screens`, `data/output` and `logs`. **The bot does not create them.** What happens when one is missing:
- `data/output` missing: the whole day's result file is lost (E02).
- `data/screens` missing: screenshots are silently not saved.
- `logs` missing: the bot does not start at all (E02).
- [ ] The input file is at `data/input/providers.xlsx`. The scheduled task points at this path.
- [ ] **Login sheet:** the username is in cell A2 and the password is in cell B2. When the portal password changes, update B2 before 07:00, or every row will say Login Failed (E01).
- [ ] **Providers sheet, from row 2:**
- column A = NPI, column B = provider name, column C = active flag
- one provider per row
- a row with an empty NPI cell is ignored, with no row in the result file
- [ ] **Active flag, column C:** only a capital **N** marks a provider inactive.
- Good: `N`
- Bad: `n`, `No`, or ` N` with a space. These are treated as **active**, and the bot will try to download that provider.
- [ ] Don't keep files open or move files inside `data/downloads` around 07:00. The bot decides a download worked when any new CSV appears in that folder (E10).
## 4. During a run
- Don't use, close or click in the Chrome window the bot opened. Closing it stops the run part-way (E03).
- Don't open the result file for today until the run has ended. It is written only at the end.
- **To stop the bot:** end the task "Payer portal downloader" in Task Scheduler. If you don't have Task Scheduler access, raise an RPA Support ticket.
- Stopping this way skips the final save, so **no result file is written for that run**.
- Close any Chrome window the bot left behind.
- **When a run looks stuck:** Chrome stays on the same portal page for more than 10 minutes, or today's result file has still not appeared well after the run should have finished (allow about 2 minutes per provider).
- The bot has **no watchdog**. It does not restart itself.
- If Chrome is frozen on one page for more than 10 minutes, raise a ticket.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `data/output/Download_Report_<today's date>.xlsx` exists, **and** its row count (not counting the header) equals the number of providers in the input with an NPI. Fewer rows means the run stopped part-way (E03). No file means the run failed or never started (E02). |
| Which providers failed? | Result file, **Status** column: red cells (Error). Rows with **Login Failed** have **no colour**, so check that text too. Green "Downloaded" rows can also be failures; read the Remarks (E07). |
| Why did one provider fail? | Result file, **Remarks** column, same row. For "Report grid did not load", open `data/screens/<NPI>.png`. |
| Is the downloaded file there? | `data/downloads/`. The files keep the portal's own names, so match them by the time they were saved, not by name. |
| Did the bot already retry? | Only session expiries are retried (up to 3 attempts per provider). The retry is recorded only in the bot's log, which ops can't open. If you need to know, ask in the ticket. |
| Which version is running? | The bot has no version number (see section 10). |
## 6. Statuses
These are the values in the result file's Status column, each with its Remarks text.
| Status (Remarks) | Colour | Meaning | What ops does |
|---|---|---|---|
| Downloaded ("Report saved") | green | A new CSV appeared in `data/downloads` after the export click | Nothing. If CSV counts don't match, see E10. |
| **Downloaded ("portal maintenance - no file")** | **green** | **Not downloaded.** The portal showed a maintenance banner, so no file was saved. | **Needs follow-up:** re-run or download by hand after maintenance (E07) |
| No Data ("No remittances in the date range") | green | The portal had no remittances for this provider | Nothing |
| Skipped - inactive provider ("Marked N in the input") | green | Column C was a capital N, so the bot did not open this provider | Nothing, unless the provider should be active: fix column C |
| Error ("Report grid did not load") | red | The provider's report table didn't appear within 30 seconds. A screenshot was taken. | E04 |
| Error ("Download did not finish") | red | The export was clicked but no new CSV arrived within 60 seconds | E05 |
| Error ("Session expired") | red | The portal kept logging the bot out, 3 attempts in a row | E06 |
| Login Failed ("Check the portal username/password") | **no colour** (should be red) | The first sign-in failed. Every row gets this status and nothing is downloaded. | E01 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Every row says **Login Failed**, "Check the portal username/password". No colour. Nothing downloaded. | After sign-in it waits 30 s for the portal dashboard. If the dashboard never appears, it marks every provider Login Failed and stops. | Result file, every row | Business (wrong or expired credentials) or System (portal slow or down) | Ops | Sign in to the portal by hand with the username and password from the Login sheet (A2/B2). If that fails, fix the credentials in the input. Then ask for a re-run. | Manual sign-in works but the bot still gets Login Failed. Send the date and the result file. |
| E02 | **No result file for today** in `data/output` | The run stopped before it could save. Causes: Chrome couldn't start; the input file is missing or unreadable; the `data/output` or `logs` folder is missing; the task was ended in Task Scheduler. | Nowhere ops can see. The error is not written to any file. | System | Ops, then Dev | Check that the folders in section 3 exist and that `data/input/providers.xlsx` is there and not open in Excel. Then ask for a re-run. | The folders and input are fine and there is still no file. Send the date, the expected run time (07:00), and whether Chrome opened. |
| E03 | The result file has **fewer rows than the input**. The last providers are simply missing. | The run stopped part-way. Causes: a re-login failed; a page element was missing (sign-in fields or the export button); the Chrome window was closed; a page hung. The bot saves what it had and stops. | Result file (missing rows). The cause is not recorded anywhere ops can see. | System | Dev | Note which provider is the first missing one: that is where it stopped. Ask for a re-run. **A re-run processes every provider again** and replaces today's result file. | Always. Send the date, the result file, and the first missing provider's NPI. |
| E04 | Error, "Report grid did not load" | It opens the provider's page and waits 30 s for the report table. If the table never appears, it saves a screenshot and moves on. | Result file; screenshot `data/screens/<NPI>.png` | System (portal slow) or Business (NPI not on this portal account) | Ops | Open the screenshot. If it shows "not found" or "no access", check the NPI in column A. If the page was just slow, download that provider by hand or ask for a re-run. | The same provider fails on 2 days in a row with a correct NPI. Send the date, the NPI and the screenshot. |
| E05 | Error, "Download did not finish" | It clicks export and watches `data/downloads` for 60 s for any new CSV. If none appears, it moves on. No screenshot. | Result file | System | Ops | **Check `data/downloads` first.** The file may have arrived late, and it may then have been counted as the *next* provider's download (E10). If it is not there, download it by hand. Re-running is safe: downloads change nothing in the portal. | It happens for several providers in one run, or every day. Send the date and the NPIs. |
| E06 | Error, "Session expired" | Opening the provider's page sent it back to the sign-in page. It signs in again and retries, up to 3 attempts. | Result file. The retries are only in the bot's log. | System | Bot, then Ops | Download that provider by hand or ask for a re-run. | It appears on more than one provider in a run, or on 2 days in a row. Send the date and the NPIs. |
| E07 | **Green** Downloaded, but Remarks say "portal maintenance - no file" | The portal page showed a maintenance banner. The bot records it as Downloaded and moves on. | Result file, Remarks column only | System (portal side) | Ops | **No file exists.** After the maintenance ends, download by hand or ask for a re-run. | Maintenance shows for more than one day |
| E08 | No Data, "No remittances in the date range" | The portal's report page said there were no results | Result file | Business | Nobody | Nothing | A provider you know has payments shows No Data |
| E09 | Skipped - inactive provider, or an inactive provider was **not** skipped | Only a capital N in column C counts as inactive | Result file | Business | Ops | Fix column C: capital N for inactive, anything else for active | Never (input fix) |
| E10 | The number of new CSVs in `data/downloads` doesn't match the number of "Report saved" rows, or a CSV holds the wrong provider's data | The bot counts *any* new CSV as success. It does not check which provider the file belongs to. A late file from one provider can be credited to the next. | Not recorded. Visible only by comparing files with the result file. | System | Ops, then Dev | Open the CSVs saved around that time and check which provider each belongs to. Download any missing provider by hand. | It happens again. Send the date and the provider rows involved. |
| E11 | Chrome sits on the same portal page for more than 10 minutes, and no result file yet | There is no watchdog. A hung page is not detected. | Nowhere | System | Dev | End the task (section 4), close Chrome, raise a ticket | Always. Send the date, the time and a photo or screenshot of the Chrome window. |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Portal logged the bot out (session expired) | Signs in again and retries the same provider | 3 attempts per provider (30 s wait for each sign-in) | Row = Error "Session expired", next provider. If one of those sign-ins fails outright, the run stops (E03). |
| Report table didn't load | Nothing. No retry. | 30 s, once | Row = Error, screenshot, next provider |
| Download didn't arrive | Nothing. No retry. | 60 s, once | Row = Error, next provider |
| Anything else (Chrome closed, page element missing, page hung) | Nothing. No restart, no resume. | none | Run stops; result file saved with the rows done so far (E03) or not at all (E02, E11) |
Timing: one provider normally takes seconds. A failing one takes up to about 2 minutes before the bot gives up. No retry happens later in the day. The next automatic attempt is tomorrow's 07:00 run.
## 9. Known incidents
There is little history: one log of one run, and no change log.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-09-02, unversioned | 1 of 3 providers red "Error"; the other 2 Downloaded | The portal session expired during the run. The bot signed in again once and the retry failed. Whether it was a slow table (E04) or a slow download (E05) is only in the Remarks column. | Read Remarks, follow E04 or E05 | Partly |
## 10. Releases and version check
- How an update arrives: there is no updater. A developer replaces `bot.py` in the bot folder by hand.
- How to confirm it applied: not possible today. The bot shows no version number anywhere (see section 13).
- If you think an update did not arrive: raise an RPA Support ticket.
## 11. Escalating to dev
Raise a ticket in the **RPA Support** queue.
Send:
- the run date and time (07:00 unless re-run)
- the exception ID from section 7 (e.g. E04)
- the NPI(s) affected
- today's result file `data/output/Download_Report_<YYYYMMDD>.xlsx`
- any `data/screens/<NPI>.png` for those NPIs
Ask dev to check the bot's log for that date, because ops can't open it.
Never send: the portal password, the input file's Login sheet, the settings file `config.yaml`, or downloaded CSV contents in chat.
## 12. Failure-class check
- F1 covered in section 3 (Chrome, folders, input path) and E02.
- F2 covered in section 4 (don't touch the bot's Chrome window). The bot opens its own Chrome, so it can't grab another window.
- F3 covered in E01, E04, E05 and E11, with limits in section 8.
- F4 covered in section 3 (active flag, empty NPI, Login cells) and E09; fix item 8.
- F5 covered in E07 and E10; fix items 1 and 2.
- F6 covered in E05: the file may arrive after the bot gave up. Re-running is safe because downloads don't change the portal.
- F7 covered in section 6 ("portal maintenance" is green) and E07; fix item 1.
- F8 covered in section 8 and E11. There is no auto-rerun loop. There is no watchdog: fix item 10.
- F9 covered in section 4 (end the task in Task Scheduler; no result file is saved then); fix item 10.
- F10 not applicable: the bot has no batch or completion gate. Each provider row is independent, and every run starts from scratch.
- F11 covered in section 5 (the log is not open to ops); fix items 3, 4, 5 and 7.
- F12 covered in section 10; fix item 9. There is no updater and nothing shared is rewritten on start.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Maintenance banner returns status "Downloaded" (green) with no file | Ops thinks the report was downloaded | `bot.py:54-55` |
| 2 | Download success = "any new `*.csv` in the folder". Files are not renamed per provider. | A late file is credited to the wrong provider; CSVs can't be tied to an NPI | `bot.py:63-68` |
| 3 | Uncaught errors (failed re-login, missing sign-in fields or export button, Chrome failure, page-load timeout) stop the run. Remaining providers get no row, and no reason is recorded. | Result file looks complete but is short (E03) | `bot.py:41-43`, `bot.py:64`, `bot.py:81`, `bot.py:102`, `bot.py:113-114` |
| 4 | Crashes and tracebacks are not written to `logs/bot.log`, only to the console, which Task Scheduler doesn't keep | No record anywhere of why a run died (E02, E03) | `bot.py:30-31`, `bot.py:113-114` |
| 5 | Login Failed rows get no colour and no log line | Easy to miss; the log shows nothing for that day | `bot.py:88-91` (colour map `bot.py:27-28` is never applied there) |
| 6 | Folders `data/downloads`, `data/screens`, `data/output` and `logs` are never created. A missing `data/output` loses the report in the final save; a missing `data/screens` loses screenshots silently. | Whole day's result lost (E02) | `bot.py:20-21`, `bot.py:30`, `bot.py:59`, `bot.py:110` |
| 7 | Retries and the Remarks reason are only in the log (or not logged at all). The session-expiry warning has no NPI. | Ops can't open the log, so "did it retry?" can't be answered. Suggest an Attempts column in the result file. | `bot.py:101`, `bot.py:105` |
| 8 | Active flag: only exact capital "N" is inactive | `n` / `No` / blank-padded values get processed | `bot.py:77` |
| 9 | No version string anywhere | Ops can't confirm which build runs | `bot.py:1-2` |
| 10 | No watchdog, no page-load timeout, no Stop mechanism that still saves the result file | Hung runs go unnoticed; stopping loses the day's result file | `bot.py:79-81`, `bot.py:108-110` |
| 11 | A re-run reprocesses every provider and overwrites the day's result file; no resume | A re-run for one failed provider redoes all and erases the earlier result | `bot.py:92`, `bot.py:110` |
| 12 | Security: portal password kept in the input Excel; an unused API token in plain text in config; provider names written to the log; screenshots may show patient data | Credential and PHI exposure | `bot.py:76`, `config.yaml:2`, `bot.py:105`, `bot.py:59` |
| 13 | After the third failed attempt the bot signs in once more for nothing | Wasted 30 s per such provider; one more chance to crash (E03) | `bot.py:96-104` |
Loaded 100 of 143 files, more files were not shown because too many files have changed in this diff. Show more