Rpa_Skills/tests/evals/results/handoff/h-api-sync-full-r5.md

# Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**

## 1. What the bot does

Every hour, the bot reads the finance export of approved invoices (`data/input/invoices.csv`) and sends each invoice to the billing system over its web API. It writes one report file per run in `data/output/` with a Status and a Message for every invoice, and adds one line per invoice to `logs/sync.log`. It has no window or screen of its own and does not drive any desktop app or website. It is a background job that Windows Task Scheduler starts.

## 2. How it works

```
Windows Task Scheduler (every hour)
   └─ starts → Invoice sync (background job, no window)
                    │  reads  ← data/input/invoices.csv  (finance export)
                    ├─ sends  → Billing system (web API, over the internet)
                    ├─ writes → data/output/sync_report_<YYYYMMDD_HHMM>.csv
                    ├─ writes → logs/sync.log
                    └─ writes → BOT_WAS_RUN (marker file in the bot folder, "a run started")
```

- **Task Scheduler** starts the job once an hour. The sample log shows runs at the top of the hour.
- **Invoice sync** is the bot. It runs and exits by itself, with nothing on screen.
- **The input file** is the finance export. The bot only reads it and never moves, renames or marks it.
- **The billing system** is where invoices get created. The bot never deletes or changes an invoice there.
- **The report file** is the only place that says *why* an invoice did not sync.
- **`sync.log`** shows when each invoice was handled and its status, plus a "run done" line at the end.
- **`BOT_WAS_RUN`** only shows that a run started. Ignore it.

**One run, step by step:**
1. Write the `BOT_WAS_RUN` marker and create a new report file named after the current date and minute.
2. Read the input file one row (one invoice) at a time.
3. Check the row: invoice number, customer ID, amount and currency must be filled in, and the amount must be a number above 0. If the row fails a check, it is marked **Invalid** and the bot goes to the next row.
4. Send the invoice to billing. The bot reads billing's answer and marks the row Synced, Duplicate or Invalid. If billing does not answer or answers with an error, the bot retries (section 8).
5. Write the row's result to the report and `sync.log`.
6. After the last row, write `run done -> sync_report_<time>.csv` to `sync.log`.

**Every hour the bot sends the whole input file again.** Invoices that already went through come back from billing as **Duplicate**. This is normal and does not create a second invoice.

## 3. Before a run

- [ ] The folder `data/output` exists inside the bot folder. **If it is missing, every run crashes before doing anything** (E10). The bot does not create it.
- [ ] The folder `logs` exists inside the bot folder.
- [ ] The settings file (`.env`) in the bot folder contains the billing address and API key. Dev maintains this file. If the key is missing or wrong, every invoice ends **Failed** (E09).
- [ ] The input file has exactly this header row: `invoice_no,customer_id,customer_name,amount,currency`.
- [ ] Input rules for whoever prepares the export:
  - One invoice per row, with each invoice number used once.
  - **Amount** is a plain number with no currency sign and no thousands separator.
    - Bad: `$1,200.00`, `1.200,00`, `-50`
    - Good: `1200.00`
  - **Currency** is filled in, e.g. `USD`.
  - **Customer ID** is filled in. The customer name is not checked.
  - Save it as a plain CSV. If every row comes back "missing invoice_no" although the column is filled, the file was saved in a format the bot can't read (E12).
- [ ] The bot has no version display. See section 10.

## 4. During a run

- There is nothing to watch and nothing to click. Don't open, edit or move the input file while a run is going on.
- Don't open the newest report file in Excel while the run is still writing it. Wait for the "run done" line in `sync.log`.
- **To stop the bot:** it has no Stop button. End the running "Invoice sync" task in Windows Task Scheduler. If you don't have access to Task Scheduler, ask in #rpa-support. A stopped run leaves a partial report and no "run done" line. That is safe: the next run sends everything again, and invoices that already went through come back Duplicate.
- **A run looks stuck** when `sync.log` gets no new line for several minutes. One invoice that billing keeps rejecting can take **up to about 6 minutes** before the bot gives up on it (section 8). Wait 10 minutes before escalating. A run that is still going when the next hour starts is E13.

## 5. Where to look when something goes wrong

| Question | Where to look |
|---|---|
| Did the run finish? | `logs/sync.log`: the run should end with a line `run done -> sync_report_<time>.csv`. If there is no such line for an hour, the run crashed or is still going (E10, E13). |
| Which invoices failed? | Newest file in `data/output/` (`sync_report_<YYYYMMDD_HHMM>.csv`). Filter the **Status** column for anything other than Synced and Duplicate. Also filter **Message** for "some lines rejected" and "confirmed after retry" (E06, E07). |
| Why did one invoice fail? | The same report file, **Message** column on that invoice's row. `sync.log` shows only the time and status, never the reason. |
| Did the bot already retry? | Only partly visible. "No success after 5 tries" means it retried 5 times. "Created (confirmed after retry)" means at least one retry happened. A gap of 30+ seconds between two lines in `sync.log` also means retries. Each retry is not logged. |
| When was one invoice last handled? | Search `logs/sync.log` for the invoice number, then open the report file whose time matches that line. |
| Which version is running? | Not visible to ops. Ask in #rpa-support (section 10). |

The report files contain customer names. Keep them inside the output folder, and don't paste their contents into chat or tickets.

## 6. Statuses

The report file is plain text, so no status has a colour.

| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | Billing created the invoice. | Nothing. |
| Synced | Created (some lines rejected) | **Needs follow-up.** Billing created the invoice but rejected some of its lines, so the invoice is incomplete. | Check the invoice in billing and follow up (E06). |
| Synced | Created (confirmed after retry) | Sending failed, but billing then showed an invoice with this number. Usually the bot's own earlier attempt went through. It can also be an older invoice with the same number. | Spot-check the invoice in billing (E07). |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number. It is usually an invoice an earlier run already sent. | Nothing, unless the number was reused for a different invoice (E05). |
| Invalid | missing `<field names>` | A required field is empty on that row. | Fix the input (E01). |
| Invalid | amount must be positive | The amount is 0 or negative. | Fix the input (E02). |
| Invalid | amount '`<value>`' is not a number | The amount has a symbol, separator or text in it. | Fix the input (E03). |
| Invalid | billing's own reason text, or "Rejected by billing" | Billing refused the invoice's content. | Fix the data as billing says (E04). |
| Failed | No success after 5 tries | Billing never accepted it after 5 tries over about a minute or more. | See E08. If every row says this, see E09. |

## 7. Exception catalogue

| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing customer_id, amount" (or other field names) | Before sending, it checks that invoice number, customer ID, amount and currency are filled in. | Report: Status, Message. Log: invoice number + Invalid. | Business | Ops | Get the export corrected. The next hourly run picks the row up again. | Never, unless the fields are filled in and it still says missing (then see E12). |
| E02 | Invalid - "amount must be positive" | The amount is 0 or below. | Report: Status, Message | Business | Ops | Credit notes and zero invoices are not supported. Get the row removed or corrected in the export. | Finance needs credit notes synced. |
| E03 | Invalid - "amount '…' is not a number" | The amount cannot be read as a plain number. | Report: Status, Message | Business | Ops | Fix the amount format, e.g. `1200.00` instead of `$1,200.00`. | Never. |
| E04 | Invalid - billing's own reason, or "Rejected by billing" | Billing answers that the invoice content is not acceptable. | Report: Status, Message (billing's text) | Business | Ops | Read the message and fix that data in the export, e.g. an unknown customer or currency. | The message isn't understandable, or the data looks right. |
| E05 | Duplicate - "Invoice already exists in billing" | Billing answers that the invoice number already exists. | Report: Status, Message | Business | Ops (usually nothing) | Normal for invoices sent in an earlier hour. Act only if this invoice number was **never** synced before (search older reports and `sync.log`). Then two different invoices share a number, so check in billing. | Billing holds a different invoice under that number. |
| E06 | Synced - "Created (some lines rejected)" | Billing answers "created, but some lines refused". | Report: Message only. The log shows plain "Synced". | Business | Ops | Open the invoice in billing, see which lines are missing and add them or follow up with finance. **The bot will not resend it.** Every later run shows it as Duplicate. | Billing gives no way to see the rejected lines. |
| E07 | Synced - "Created (confirmed after retry)" | Sending failed, so before retrying the bot asked billing whether this invoice number exists, and billing said yes. | Report: Message | System | Ops | Spot-check in billing that the invoice under this number has the right customer and amount. Usually it is the bot's own earlier attempt. If it is an older, different invoice, this one was **not** created. | The invoice in billing doesn't match the export. |
| E08 | Failed - "No success after 5 tries" | Billing didn't answer, timed out, said "too many requests", had a server error, or answered with an error the bot doesn't recognise, 5 times in a row. | Report: Status, Message. The log shows "Failed" with no reason. | System | Bot, then Ops | **Check billing first** for this invoice number before entering anything by hand. The next hourly run tries again if the invoice is still in the export. It then shows Synced, Duplicate (already there) or Failed again. | The same invoice is Failed in 3 runs in a row. |
| E09 | **Every** invoice in a run is Failed, and the run is much slower than usual | Every send is refused: wrong or missing API key, wrong billing address, or billing down. The bot can't tell these apart and retries each one. | Report: every row Failed. Log: every row Failed. | System | Dev | Nothing to fix on the ops side. Post in #rpa-support right away. | Always. |
| E10 | No new report file, or a report that stops part-way, and **no "run done" line** in `sync.log` for that hour | The bot crashed. Known causes: the output folder is missing; the network dropped while the bot was checking billing after a failed send; billing sent an answer the bot couldn't read; the input file is missing or unreadable. | Only the missing "run done" line and the partial report. **The crash reason is not written to any file ops can open.** | System | Dev | Check that `data/output` exists, and create it if missing. Otherwise escalate. Invoices after the crash point were not sent. | The output folder exists and the next hourly run also has no "run done". |
| E11 | Every hour the report stops at **the same invoice**, and invoices after it never get a row | One invoice makes billing send an answer the bot can't read. The bot crashes there each hour (as in E10), so every row after it is blocked. | Partial report, ending just before the same invoice each hour. No "run done". | System | Ops, then Dev | Note the first invoice that is missing from the report. Ask whoever prepares the export to take that row out for now, so the rest can sync. | Always: send the invoice number. |
| E12 | Every row Invalid - "missing invoice_no" although the column is filled | The file was saved with a hidden marker at the start (Excel's "CSV UTF-8" option), so the bot doesn't recognise the first column name. | Report: every row Invalid | Business | Ops, then Dev | Re-save the export as plain "CSV (Comma delimited)" and wait for the next run. | Re-saving doesn't help, or names with accents then make the run crash (E10). |
| E13 | A run is still going when the next hour starts, or two report files with close times overlap | Each invoice that keeps failing can take up to about 6 minutes, so a file with many such invoices runs past the hour. Nothing stops a second run from starting. | `sync.log`: lines from two runs mixed together. A report file may be overwritten if two runs start in the same minute. | System | Bot, then Dev | Wait for both to finish. Results are still safe because Duplicate protects against double invoices. | It happens 2 hours in a row, or E09 is the cause. |

## 8. What the bot recovers by itself

| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing doesn't answer within 30 seconds, the network drops, billing says "too many requests", or billing gives any error the bot doesn't recognise | Asks billing whether the invoice already exists. If yes: Synced (E07). If not: waits and sends again. | 5 tries. Waits 2, 4, 8, 16 and 32 s between them (about 1 minute), plus up to 30 s per call. **Worst case about 6 minutes per invoice.** | Row = Failed (E08). The bot goes to the next invoice. |
| Billing timed out but had actually created the invoice | The existence check above finds it, so no second invoice is created. | Same as above | Synced - "Created (confirmed after retry)" |
| An invoice failed this hour | Nothing is resent within this run. The **next hourly run** sends the whole input file again. | Only while the invoice is still in the export | Shows Duplicate once it has gone through |
| The bot crashed | **Nothing.** There is no restart, resume or alert. The next run starts on the hour as usual. | - | See E10 / E11 |

## 9. Known incidents

Both incidents are recorded only in the version history. Their dates were not recorded, but both were fixed before 0.9.1.

| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | The same invoice appeared twice in billing. | Billing created the invoice but answered "timed out", so the bot sent it again. | Fixed: the bot now checks whether the invoice exists before every retry (E07). Still spot-check "confirmed after retry" rows. Removing a duplicate in billing needs billing or dev access. | Partly |
| Before 0.9.1 | The hourly run died part-way. Invoices after that point got no report row, and there was no "run done" line. | Billing answered "too many requests" and the bot had no retry. | Fixed: up to 5 tries with waits (section 8). A network drop during the existence check can still crash the run the same way (E10). | Partly: ops could see the missing "run done" and wait for the next hour. |

## 10. Releases and version check

- **How an update arrives:** the bot has no updater. The RPA dev team replaces the files in the bot folder on the machine.
- **How to confirm it applied:** the bot shows no version anywhere ops can see. Ask dev in #rpa-support to confirm the version (currently 0.9.1) after every update.
- **If the version did not change:** report it in #rpa-support. Ops can't update the bot themselves.

## 11. Escalating to dev

Post in **#rpa-support** (Teams) with:
- the exception ID from section 7 (e.g. E10);
- the date and hour of the run;
- the report file name (e.g. `sync_report_20260903_0900.csv`), the name only;
- the invoice numbers involved;
- the matching lines from `logs/sync.log`, or say "no run done line".

**Never send:** the `.env` settings file, API keys or passwords. Don't paste or attach report files either, because they contain customer names. Give the file name and dev will open it.

## 12. Failure-class check

- **F1** Missing prerequisite: covered in section 3 (output folder, logs folder, settings file), E09 and E10, and Fix #2 and #9.
- **F2** not applicable: the bot drives no windows. It calls the billing web API with no screen, mouse or keyboard.
- **F3** Target slower than expected: covered in E08 and E13, section 8 (30 s wait, 5 tries) and the section 4 "stuck" rule.
- **F4** Input read differently: covered in the section 3 input rules (amount format, header, CSV type) and E01-E03 and E12, and Fix #10.
- **F5** Marked done when nothing was done: covered in E07 ("confirmed after retry" can be an older invoice) and Fix #7.
- **F6** Action happened but not confirmed: covered in E07 and E08 ("check billing first"), and incident 1.
- **F7** Business exception shown as success: covered in Statuses and E06 (partly rejected invoice shown as Synced), and Fix #6.
- **F8** Endless re-runs / watchdog: there is no watchdog or auto-rerun. The hourly re-send of the whole file is covered in sections 2 and 8, and the overlap risk in E13 and Fix #11.
- **F9** Stopping the wrong way: covered in section 4 (end the Task Scheduler task; a partial report is safe).
- **F10** One item blocks the queue: covered in E11 (one unreadable billing answer crashes the run at the same invoice every hour), and Fix #3 and #4.
- **F11** Cause hidden from ops: covered in sections 5 and E10. Crash and network messages are not saved to any file, and the log has no reason. See Fix #5 and #12.
- **F12** Update didn't arrive: covered in section 10 (no version visible), and Fix #13.

## 13. Fix before handover (dev team)

| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | `.env` (API key) is committed to git | The key is exposed to anyone with repo access. Rotate the key and untrack the file. | `.env` (tracked); loaded at `sync.py:17-19` |
| 2 | `data/output/` is not in the repo and not created by the code | A fresh install crashes every run before processing anything (E10) | `sync.py:68-69` |
| 3 | `exists()` has no error handling and is called right after network errors | A network drop during the duplicate check crashes the whole run (E10). This is the 429 incident again by another route. | `sync.py:41-43`, `sync.py:60` |
| 4 | `r.json()` on a 422 with a non-JSON body is unguarded | One bad billing answer crashes the run, at the same row every hour (E11) | `sync.py:59` |
| 5 | Network errors and `CRASH:` go to `print()` only | The cause of a crash is lost under Task Scheduler. Ops only sees a missing "run done". | `sync.py:52`, `sync.py:84-85` |
| 6 | HTTP 207 (some lines rejected) is stored as Synced | Ops treats an incomplete invoice as done (E06) | `sync.py:54-55` |
| 7 | "Created (confirmed after retry)" only checks that the invoice number exists, and also fires after non-retryable errors such as 400 | A pre-existing or different invoice is reported as Synced, and this one was never created (E07) | `sync.py:60-61` |
| 8 | 400/401/403/404 are retried 5 times like temporary errors | A bad key makes every row wait about 1 min and then Fail. This drives runs past the hour (E09, E13). | `sync.py:60-62` |
| 9 | A missing API key silently sends an empty token. A missing URL silently falls back to a built-in default address. | Misconfiguration looks like E09 instead of a clear startup error | `sync.py:18-19` |
| 10 | Input is read as `utf-8`, not `utf-8-sig` | An Excel "CSV UTF-8" export makes every row "missing invoice_no" (E12). A cp1252 export with accents crashes. | `sync.py:69` |
| 11 | No guard against overlapping runs, and the report name only goes down to the minute | Two runs can overlap and overwrite each other's report (E13) | `sync.py:68` |
| 12 | `sync.log` holds only invoice number + status, with no message, HTTP code or attempt count | Ops can't answer "did it retry / why" from the log. "Synced" hides 207 rows. | `sync.py:75` |
| 13 | No version anywhere ops can see. A stray commit message in the README and two in the code. | Ops can't confirm an update (section 10) | `README.md:5-6`, `sync.py:86-87` |
| 14 | The Task Scheduler task is not in the repo (schedule, "do not start a new instance", stdout capture) | No one can check how the bot is actually started or stopped | `sync.py:3` |
| 15 | No alert on Failed rows or a crash | Problems are found only if someone opens the report or log | `sync.py:63`, `sync.py:80-85` |
| 16 | Sleeps 32 s after the 5th try before giving up | Adds 32 s to every Failed invoice for nothing | `sync.py:62-63` |
| 17 | `BOT_WAS_RUN` marker is undocumented | Unclear purpose. Ops may delete it or worry about it. | `sync.py:67` |
| 18 | No regression tests for the 429 and 504 fixes. Both fix commits only add a comment. | The incident fixes can regress unnoticed, and history doesn't show what changed | `sync.py:46-63` |