Add rpa-bot-handoff skill: ops guide for bot handover

The skill scans a bot with subagents (never runs it) and writes
HANDOVER.md, an 11-section ops guide in the operator's order: where is
what, how to run, how long to wait, after the run, how to see an
exception, exception list, how you will know, how to tell us, don'ts,
issues seen before. Dev material (check against 12 failure classes from
Tebra Auto Post, gaps with file:line, incidents) goes to HANDOVER-DEV.md.

Also adds ADR-0004, glossary terms, five fixture bots and a handoff eval
runner (--resume, --regrade) with results from 47 runs of the earlier
13-section version. The grader still expects that structure and must be
updated before the next eval run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tests/evals/results/handoff/h-api-sync-full-r2.md

@@ -0,0 +1,170 @@
+# Invoice sync - ops handover
+
+Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
+
+## 1. What the bot does
+
+Every hour, Windows Task Scheduler starts the bot. It reads the finance export file `data/input/invoices.csv` (one approved invoice per row), checks each row, and sends each valid invoice to the billing system over its web API. For each run it writes one report file, `data/output/sync_report_<date>_<time>.csv`, with a Status and a Message for every invoice, and adds a few lines to `logs/sync.log`. It has no screen and drives no desktop app. The only system it talks to is the billing system.
+
+## 2. How it works
+
+```
+Windows Task Scheduler (every hour)
+   └─ starts → Invoice sync script ──reads──> data/input/invoices.csv
+                    │
+                    ├─ sends each invoice ──> Billing system (web API)
+                    ├─ writes → data/output/sync_report_<date>_<time>.csv  (one row per invoice)
+                    └─ writes → logs/sync.log  (one line per invoice + "run done")
+```
+
+- **Task Scheduler** starts the bot on the hour. The bot has no window of its own.
+- **Invoice sync script** does all the work, one invoice at a time, top to bottom.
+- **Input file** is the finance export. The bot reads the whole file on every run.
+- **Billing system** is where invoices are created. The bot never deletes or changes an invoice there.
+- **Report file** is the main place to see what happened to each invoice in that run. It contains customer names, so treat it as confidential.
+- **Log file** records only the invoice number and its status, plus a "run done" line at the end of each complete run.
+
+**One run, step by step:**
+1. Open the input file and create a new report file for this run.
+2. For each row: check that the invoice number, customer ID, amount and currency are filled and that the amount is a positive number.
+3. If the row passes, send it to billing. If billing doesn't give a clear yes or no, ask billing whether the invoice now exists. If it doesn't, wait and send again, up to 5 tries.
+4. Write the result to the report and the log, then go to the next row.
+5. After the last row, write `run done -> <report name>` in the log.
+
+Each run handles the whole input file again. Invoices that were already sent come back as **Duplicate**, which is normal.
+
+## 3. Before a run
+
+- [ ] The folder `data/output` exists. **The bot does not create it.** If it is missing, every run stops before the first invoice and writes no report (E11).
+- [ ] The folder `logs` exists.
+- [ ] The settings file (`.env`) is in the bot folder with the billing address and billing key. The dev team maintains it. Ops never opens or copies it. If the key is missing or expired, every invoice ends up **Failed** (E09).
+- [ ] The input file is at `data/input/invoices.csv`, with the header row `invoice_no, customer_id, customer_name, amount, currency`.
+- [ ] One invoice per row. Invoice number, customer ID, amount and currency must be filled.
+- [ ] Amount is a plain number with a dot for decimals. Good: `1250.00`. Bad: `1,250.00`, `$1250`, `1250 USD` (each becomes **Invalid** E03).
+- [ ] Amount is above zero. Credit notes or zero-amount rows become **Invalid** (E02).
+- [ ] Version: ops cannot see which version is running (see section 10). Ask the dev team to confirm 0.9.1.
+
+## 4. During a run
+
+- Don't save over or edit the input file while a run is in progress. The bot reads it row by row as it goes.
+- Don't delete or move the newest report file until the log shows `run done` for it.
+- **To stop the bot:** whoever has access to Task Scheduler on the bot machine selects the invoice sync task and chooses **End**. To stop future hourly runs as well, choose **Disable**. If ops don't have Task Scheduler access, ask in #rpa-support. A stopped run leaves a short report and no `run done` line. Running it again is safe: invoices already created come back as **Duplicate**.
+- **A run looks stuck when** the newest report file stops growing for several minutes. When billing is down, **each invoice can take up to about 6 minutes** before it is marked Failed (5 tries with waits). There is no automatic stop or restart. Wait for that. Escalate when a run is still going when the next hourly run is due, or when the report hasn't grown for more than 10 minutes.
+
+## 5. Where to look when something goes wrong
+
+| Question | Where to look |
+|---|---|
+| Did the run finish? | `logs/sync.log`: look for `run done -> sync_report_<date>_<time>.csv` for that hour. No such line means the run stopped partway (E10) or never started (E11). |
+| Which invoices failed? | The report for that run in `data/output`: Status column is anything other than **Synced**. Also check **Synced** rows whose Message says "some lines rejected" (E06). |
+| Why did one invoice fail? | The same report, Message column. For **Failed** rows the real reason is **not recorded anywhere ops can open** (fix item 3). Escalate with the invoice number. |
+| When was an invoice last tried? | `logs/sync.log`: search for the invoice number. Each line has the date, time and status. |
+| Did the bot already retry? | **Failed** always means 5 tries were made. "Created (confirmed after retry)" means a retry happened and worked. Nothing else records retries. |
+| Why did a whole run stop? | Not visible to ops. The reason is only shown on a console nobody sees (fix item 1). Escalate (E10). |
+| Which version is running? | Not visible to ops (fix item 15). Ask the dev team. |
+
+## 6. Statuses
+
+The report is a plain CSV file, so there are no colours. The Status column holds one of four values. The Message column explains it.
+
+| Status - Message | Colour | Meaning | What ops does |
+|---|---|---|---|
+| Synced - "Created" | none (plain text) | Invoice created in billing | Nothing |
+| Synced - "Created (confirmed after retry)" | none | Billing didn't answer clearly the first time, but the bot checked and the invoice is there | Nothing |
+| Synced - "Created (some lines rejected)" | none | **Needs follow-up.** The invoice was created, but billing refused some of its lines. Shown as Synced anyway. | Check the invoice in billing and add the missing lines by hand (E06) |
+| Duplicate - "Invoice already exists in billing" | none | Billing already has this invoice number. Normal for invoices sent in an earlier hour. | Nothing, unless billing shows it twice (E13) |
+| Invalid - "missing <field names>" | none | A required field in the input row is empty | Fix the row in the input (E01) |
+| Invalid - "amount must be positive" | none | Amount is zero or negative | Check the row with finance (E02) |
+| Invalid - "amount '<value>' is not a number" | none | Amount isn't a plain number | Fix the amount format (E03) |
+| Invalid - billing's own text, or "Rejected by billing" | none | Billing looked at the invoice and refused it | Read the message and fix the data with finance (E04) |
+| Failed - "No success after 5 tries" | none | Billing couldn't be reached or kept failing. Not created. | See E08 / E09 |
+
+## 7. Exception catalogue
+
+| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
+|---|---|---|---|---|---|---|---|
+| E01 | Invalid - "missing amount, currency" (lists the empty fields) | Before sending, it checks that invoice number, customer ID, amount and currency are filled | Report: Status, Message. Log: invoice number + Invalid | Business | Ops | Ask finance to fill the fields in the export. The next hourly run picks the row up. | Never, unless the fields are filled and it still says missing (check the header row spelling) |
+| E02 | Invalid - "amount must be positive" | Amount is zero or below | Report; log | Business | Ops | Check with finance. Zero and credit invoices can't go through this bot. | Finance needs credits to sync (change request) |
+| E03 | Invalid - "amount '1,250.00' is not a number" | The amount can't be read as a plain number | Report; log | Business | Ops | Remove thousands separators, currency signs and text from the amount | The amount is already plain digits |
+| E04 | Invalid - with billing's own explanation, or "Rejected by billing" | Billing replied that the invoice data is wrong | Report; log | Business | Ops | Read the message; fix the customer, currency or amount with finance | The message doesn't make sense, or a correct invoice is rejected again |
+| E05 | Duplicate - "Invoice already exists in billing" every hour for the same invoices | Billing replied that the invoice number already exists | Report; log | Business | Ops | Normal for invoices already sent: the export keeps old rows. Only check billing when you don't expect the invoice to exist yet. | The invoice is **not** in billing although the bot says Duplicate |
+| E06 | Synced - "Created (some lines rejected)" | Billing created the invoice but said some lines were refused | Report Message column only. The log just says Synced. | Business | Ops | Open the invoice in billing, compare with the export, add the missing lines by hand | Every invoice of a run shows this |
+| E07 | Synced - "Created (confirmed after retry)" | Billing timed out or errored, so the bot asked billing whether the invoice now exists, and it did | Report; log says Synced | System | Bot | Nothing. The bot avoided sending it twice. | Never |
+| E08 | Failed - "No success after 5 tries" on some invoices | Five tries with no clear answer from billing, and billing doesn't have the invoice. Causes: billing slow/down, too many requests, network trouble. | Report; log. **The real cause isn't recorded.** | System | Bot, then Dev | Nothing at first: if the invoice is still in the input file, the next hourly run tries it again | The same invoice is Failed in 2 runs in a row. Send: invoice number, report file name, the log lines for it. |
+| E09 | Failed on **every** invoice of a run | Same as E08, but for all rows. Usually the billing key/address in the settings file is wrong or expired, or billing is down. | Report; log | System | Dev | Don't edit anything. Escalate straight away. | Always. Send: report file name, run time, the `logs/sync.log` lines for that hour. |
+| E10 | Report ends early; no `run done` line in the log for that hour; later invoices never appear | An unexpected error ended the whole run, e.g. billing replied in a form the bot can't read, or the network dropped while it was checking whether an invoice exists. The bot does not skip the row; it stops. | Report is cut short; log has no `run done`. Reason only on a console nobody sees. | System | Dev | Note the last invoice in the report; the stop happened on the **next** row of the input. If the same row stops every hourly run, nothing after it is ever synced. | Always. Send: report file name, last invoice number in the report, run time. |
+| E11 | No new report and no log lines for an hour | The run stopped before the first invoice: `data/output` folder missing, input file missing or renamed, or the scheduled task didn't start | Nowhere ops can see | System | Ops, then Dev | Check that `data/output` exists (create it if not) and that `data/input/invoices.csv` is there | Both are in place and still no report next hour. Send: the hours with no report. |
+| E12 | Two reports a few minutes apart, or one hour's report missing | A slow run (billing down, large file) was still going when the next hourly run started. Nothing stops two runs overlapping. Reports from two runs started in the same minute overwrite each other. | Report file names and times in `data/output`; log lines from two runs mixed | System | Dev | Don't delete anything | Always. Send: the report file names and times. |
+| E13 | Billing (or finance) reports the same invoice twice | Billing created it but the reply was lost. The bot checks before re-sending, but if that check also fails it sends again. Usually billing then refuses it as Duplicate, so the report shows **Duplicate** for an invoice this run actually created. | Report shows Synced or Duplicate; nothing marks it | System | Dev | Don't delete in billing yourself unless that is your normal billing process | Always. Send: invoice number, both report files around it. |
+
+## 8. What the bot recovers by itself
+
+| Problem | What the bot does | Limit | After the limit |
+|---|---|---|---|
+| Billing times out, is busy (too many requests), errors, or the network fails | Asks billing whether the invoice now exists; if not, waits and sends again. Waits are 2, 4, 8, 16, 32 s. Each request can take up to 30 s. | 5 tries, about 1 to 6 minutes per invoice | Failed - "No success after 5 tries", next invoice |
+| Billing created the invoice but its reply was lost | Finds it with the "does it exist" check | Each try | Synced - "Created (confirmed after retry)" |
+| An invoice not synced this hour | Nothing during the run. The next hourly run reads the whole input file again. | Every hour, as long as the row stays in the input file | - |
+| Run stopped partway (E10, E11) | **Nothing.** No restart until the next hour. | - | If the cause is the same row, every hourly run stops at that row |
+
+There is no watchdog, no stuck detection and no automatic restart.
+
+## 9. Known incidents
+
+Dates aren't recorded. Both were fixed before version 0.9.1.
+
+| When / version | What ops saw | Cause | What to do now | Ops could handle? |
+|---|---|---|---|---|
+| Before 0.9.1 | The same invoice twice in billing | Billing created it but timed out; the bot sent it again | Fixed: the bot checks before re-sending. Not fully: if the check fails too, it can still re-send (E13). | Partly |
+| Before 0.9.1 | The hourly run died partway; later invoices weren't sent | Billing said "too many requests" and the bot stopped | Fixed: now retried 5 times. Not fully: other errors can still stop the run (E10). Compare the report row count with the input; escalate. | Partly |
+
+## 10. Releases and version check
+
+- How an update arrives: the dev team copies the new code into the bot folder on the bot machine. There is no launcher or automatic update.
+- How to confirm it applied: **ops can't.** The version is written only in the project's README, not in the log, the report or anywhere ops can open. Ask the dev team in #rpa-support to confirm the version after each update (fix item 15).
+- If the version did not change: escalate in #rpa-support.
+
+## 11. Escalating to dev
+
+Post in **#rpa-support** (Teams). Send: the exception ID from section 7, the report file name (it carries the run date and time), the invoice number(s), and the matching lines from `logs/sync.log`.
+Never send: the settings file (`.env`), billing keys or passwords, or the report file itself. It contains customer names, so give the file name and the dev team opens it on the machine.
+
+## 12. Failure-class check
+
+- F1 covered in section 3 (output folder, settings file, input path) and E09, E11.
+- F2 not applicable: the bot drives no windows; it only talks to the billing web API.
+- F3 covered in section 8 (30 s per request, 5 tries) and E07, E08, E12.
+- F4 covered in section 3 (amount format, good/bad examples) and E01-E03.
+- F5 covered in fix items 9 and 10 (a run where every invoice failed still looks successful to Task Scheduler) and E09.
+- F6 covered in E07, E13 and fix item 14.
+- F7 covered in section 6 ("some lines rejected" shown as Synced), E06 and fix item 9.
+- F8 covered in section 8 (no watchdog, hourly re-read of the whole file), E05, E12 and fix item 11.
+- F9 covered in section 4 (End the scheduled task) and fix item 21.
+- F10 covered in E10: one bad row stops that run and every later run at the same row. Fix item 7.
+- F11 covered in section 5 and fix items 1, 2, 3.
+- F12 covered in section 10 and fix item 15.
+
+## 13. Fix before handover (dev team)
+
+| # | Gap | Why it hurts ops | Where (file:line) |
+|---|---|---|---|
+| 1 | Network errors and the crash reason go only to the console (`print`), never to `sync.log` | Under Task Scheduler the reason for a stopped run is lost | `sync.py:52`, `sync.py:84` |
+| 2 | Log line has only invoice + status, always INFO; no message, no WARNING/ERROR | Ops can't search the log for problems or reasons | `sync.py:75` |
+| 3 | Failed rows don't record the last HTTP status or error | Ops can't tell billing-down from bad key from bad data | `sync.py:60-63` |
+| 4 | Every non-handled status is retried, including 400/401/403/404 (docstring says only 429/5xx) | A bad key costs ~1 min per invoice and shows only as "Failed" | `sync.py:47`, `sync.py:60-62` |
+| 5 | The `exists()` check has no error handling | A timeout during the check kills the whole run (same symptom as the 429 incident) | `sync.py:41-43`, `sync.py:60` |
+| 6 | 422 handling calls `r.json()` without a guard | A non-JSON 422 body kills the run | `sync.py:59` |
+| 7 | No per-row error handling in the main loop | One bad row stops the run and, if it stays in the input, every later hourly run | `sync.py:72-76` |
+| 8 | `data/output` isn't created and isn't in the repo | A fresh deploy fails on every run with no report | `sync.py:68-69` |
+| 9 | HTTP 207 (some lines rejected) is reported as Synced | Partly created invoices look done | `sync.py:54-55` |
+| 10 | Exit code is 0 even when every invoice Failed | Task Scheduler shows success for a fully failed run | `sync.py:80-85` |
+| 11 | No lock against overlapping runs; report name only to the minute | Overlapping runs; reports overwrite each other | `sync.py:68` |
+| 12 | Sleeps 32 s after the final attempt too | Wastes time per Failed invoice, more overrun risk | `sync.py:62` |
+| 13 | `Retry-After` header from billing is ignored on 429 | Retries may hit the rate limit again | `sync.py:62` |
+| 14 | If the existence check returns non-200 for any reason, the bot re-posts; a 409 then reports **Duplicate** for an invoice this run created. Relies on billing returning 409 (unconfirmed). | Misleading status; double-post risk if billing doesn't reject duplicates | `sync.py:43`, `sync.py:56-57` |
+| 15 | Version only in README (plus a stray commit-message line); not in log or report | Ops can't confirm which version runs | `README.md:5-6` |
+| 16 | The two "fix" commits (6be93ce, 28da8f3) only add comment lines; the retry/check code was in the initial import | Can't tell when the fixes went live, or that the deployed copy has them | `sync.py:86-87` |
+| 17 | The report includes `customer_name` | Personal data sits in the output folder ops share | `sync.py:76` |
+| 18 | `BOT_WAS_RUN` marker written to the project root on every run, read by nothing | Confusing file; remove or document | `sync.py:67` |
+| 19 | `nan` / `inf` amounts pass validation | Garbage amounts sent to billing | `sync.py:33-37` |
+| 20 | Missing billing key or URL isn't checked at start; falls back to a placeholder URL / empty key | Shows up only as every row Failed (E09) | `sync.py:18-19` |
+| 21 | The scheduled task isn't in the repo; console output not redirected; ops have no listed Task Scheduler access | Ops can't stop the bot or see crash output themselves | `sync.py:3` |