Rpa_Skills/tests/evals/results/handoff/h-api-sync-full-r1.md

# Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**

## 1. What the bot does

Every hour, Windows Task Scheduler starts the bot. It reads the finance export file `data/input/invoices.csv`, which has one approved invoice per row. It checks each row, then sends the invoice to the billing system over the internet.

For every run it writes:
- a report `data/output/sync_report_<YYYYMMDD_HHMM>.csv`, with one line per invoice and a Status and Message for each;
- a short line per invoice in `logs/sync.log`.

The bot has no window and drives no desktop app. It talks to the billing system directly, so you can use the machine while it runs.

## 2. How it works

```
Windows Task Scheduler (every hour)
   └─ starts → Invoice sync (no window) ──sends invoices──> Billing system (online)
                    │
                    ├─ reads  → data/input/invoices.csv
                    ├─ writes → data/output/sync_report_<YYYYMMDD_HHMM>.csv
                    └─ writes → logs/sync.log
```

- **Task Scheduler** starts the bot once an hour. Nothing else starts it.
- **Invoice sync** is a background program with no screen. It works through the input file from top to bottom, one invoice at a time.
- **Billing system** receives each invoice. It can accept it, reject it, say it already has it, or not answer.
- **invoices.csv** is the finance export. The bot never moves, renames or empties it, so every hour it sends the whole file again.
- **sync_report_….csv** is the only place where the reason for each result is written.
- **sync.log** keeps the invoice number and status for every run, plus a "run done" line when a run finishes.

**One run, step by step:**
1. Create a new report file named after the current date and time.
2. For each row of the input file:
   - Check the row: required fields filled in, and the amount is a number above 0. If the check fails, the row is **Invalid** and the bot moves to the next row.
   - Send the invoice to billing.
   - If billing does not answer, or answers with a temporary error, check whether billing already has the invoice. If it does, the row is **Synced**. If not, wait and try again, up to 5 tries in total.
   - Write the result to the report and the log.
3. Write `run done -> sync_report_<time>.csv` to the log.

## 3. Before a run

- [ ] The bot folder contains these three items. Without them the bot stops at the very start and writes no report.
  - The folder `data/output/`. The bot does not create it.
  - The folder `logs/`.
  - The settings file `.env`.
- [ ] The settings file has both the billing address and the billing key filled in. If the key is missing or expired, every invoice takes about 1 minute and then ends as **Failed** (E08). Nothing on screen says the key is the problem.
- [ ] The input file is at `data/input/invoices.csv`.
- [ ] The first line of the input file is exactly: `invoice_no,customer_id,customer_name,amount,currency`. If a heading is renamed, every row comes back **Invalid - missing …** (E01, E09).
- [ ] One invoice per row. Each row has an invoice number, a customer ID, an amount and a currency.
- [ ] Amounts are plain numbers above 0.
  - Good: `1200.50`
  - Bad: `1,200.50` or `$1200` (Invalid - "is not a number"), `0` or `-50` (Invalid - "must be positive").
- [ ] Only **one** machine runs this bot. Two machines would send the same invoices twice every hour.
- [ ] The bot does not show its version anywhere. Confirm with dev that 0.9.1 is installed (section 10).

## 4. During a run

- You can use the machine. The bot has no window and clicks nothing.
- Don't replace or edit `invoices.csv` while a run is in progress. Wait until the log shows the `run done` line for that hour.
- You can open a report file to read it. Don't save changes to the report of a run that is still in progress.
- To stop the bot: in Windows Task Scheduler, right-click the invoice sync task and choose **End**. To stop the hourly runs, choose **Disable**. Ops access to Task Scheduler is not confirmed, so ask #rpa-support if you can't do it.
  - If you stop it mid-run, the report for that run is incomplete, and the invoice it was sending at that moment may or may not have reached billing. The next run reports that invoice as **Duplicate** if it did.
- A run looks stuck when `logs/sync.log` gets no new line for more than **6 minutes**. One invoice can take up to about 6 minutes when billing does not answer at all.
  - There is no watchdog: the bot never gives up on a run by itself.
  - If many invoices fail, one run can take longer than an hour and overlap the next hourly run (E11).

## 5. Where to look when something goes wrong

| Question | Where to look |
|---|---|
| Did the run finish? | `logs/sync.log`. Look for `run done -> sync_report_<time>.csv` for that hour. If the line is missing, the run stopped partway (E10). |
| Did the run start at all? | `data/output/`. There should be one `sync_report_<YYYYMMDD_HHMM>.csv` per hour. A missing hour means the bot did not start or stopped at the very start (E10). |
| Which invoices failed? | Open that hour's report. Look at every row where Status is **Invalid** or **Failed**, and every **Synced** row whose Message is "Created (some lines rejected)". |
| Why did one invoice fail? | The **Message** column of the report. The log only shows the status, never the reason. |
| What happened to one invoice over several runs? | Search `logs/sync.log` for the invoice number. Each hit gives the run time, which points you to the matching report file. |
| Did the bot already retry? | Not recorded directly. "No success after 5 tries" means it tried 5 times. "Created (confirmed after retry)" means the first try failed and billing turned out to have the invoice. Other results came from the first try. |
| Why did the whole run stop? | Not visible to ops. The reason goes only to a console window nobody sees. Send it to dev (E10). |
| Which version is running? | Not shown anywhere ops can see. Ask dev (section 10). |

## 6. Statuses

The report is a plain CSV file, so statuses have no colour. Read the **Status** and **Message** columns together.

| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | Billing accepted the invoice | nothing |
| Synced | Created (some lines rejected) | **Needs follow-up.** Billing created the invoice but dropped some of its lines. | Check the invoice in billing and fix the missing lines (E06) |
| Synced | Created (confirmed after retry) | The first send failed or got no answer. Billing turned out to have the invoice, so the bot did not send it again. | Usually nothing. Spot-check the amount in billing (E07). |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number | Normally nothing: the same file is re-sent every hour (E05) |
| Invalid | missing `<field names>` | A required field is empty in the input | Fix the input row (E01) |
| Invalid | amount must be positive | Amount is 0 or below | Fix the input row (E02) |
| Invalid | amount '`<value>`' is not a number | The amount has commas, a currency sign, text, etc. | Fix the input row (E03) |
| Invalid | billing's own explanation, or "Rejected by billing" | Billing refused the invoice and gave a reason | Act on the reason (E04) |
| Failed | No success after 5 tries | Billing never accepted it in 5 tries, and the bot could not find the invoice in billing | Check billing first, then see E08 |

## 7. Exception catalogue

| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing invoice_no" (or customer_id, amount, currency) | Before sending, it checks that these four fields are not empty | Report: Status and Message. Log: status only. | Business | Ops | Fill in the field in the finance export. The next hourly run picks the row up. | The field is filled in but still reported missing (see E09) |
| E02 | Invalid - "amount must be positive" | The amount is 0 or below | Report | Business | Ops | Correct the amount, or remove the row if it is a credit note. The bot does not send zero or negative invoices. | Credit notes need to go through. That is a change request to dev. |
| E03 | Invalid - "amount '1,200.50' is not a number" | It cannot read the amount as a number | Report | Business | Ops | Write the amount as a plain number with a dot for decimals, e.g. `1200.50`. No thousands separators, currency signs or spaces. | The amount already looks plain |
| E04 | Invalid - billing's message (e.g. unknown customer), or "Rejected by billing" | Billing refuses the invoice and explains why | Report: Message holds billing's reason | Business | Ops | Fix the data that billing's message names, usually in the finance export or in billing's customer records. | The message is "Rejected by billing" with no reason, or you can't tell what to fix. Send the invoice number, report file name and message. |
| E05 | Duplicate - "Invoice already exists in billing" | Billing says it already has this invoice number | Report | Business | Ops | Check earlier reports (search the log for the invoice number). If an earlier run shows Synced, this is normal: the bot re-sends the whole file every hour. If no run ever synced it, check in billing that the existing invoice is the right one, because the number may be used twice. | Two different invoices share a number |
| E06 | Synced - "Created (some lines rejected)" | Billing creates the invoice but rejects part of it | Report: Message. The Status still says Synced. | Business | Ops | Open the invoice in billing, find the missing lines and add or correct them by hand. The bot will not resend them: re-runs only get Duplicate. | Lines are rejected for many invoices in the same run |
| E07 | Synced - "Created (confirmed after retry)" | The first send failed or got no answer. Before retrying, it asked billing whether the invoice exists and got yes. | Report | System | Bot | Nothing usually. The bot cannot tell whether it created the invoice this run or the invoice already existed, so spot-check the amount in billing now and then. | The invoice in billing has the wrong amount or customer |
| E08 | Failed - "No success after 5 tries" | 5 sends, with waits of 2, 4, 8, 16 and 32 s between them. After each failed send it checks whether billing already has the invoice. | Report. The real cause (billing down, busy, key expired, wrong address) is not recorded anywhere. | System | Ops, then Dev | 1. **Search billing for the invoice number first.** It may have been created even though no confirmation came back. 2. If it isn't there, do nothing: the next hourly run tries again while the row is still in the input file. 3. If it is there, nothing more is needed. The next run will show Duplicate. | **Every** row in a run is Failed (likely the billing key or address). The same invoice is Failed in 2 runs in a row. Send the report file name, invoice numbers and run time. |
| E09 | **Every** row Invalid - "missing invoice_no", although the file looks fine | The first heading isn't read as `invoice_no`. Some Excel "CSV UTF-8" saves add an invisible mark before the first heading. | Report | Business | Ops, then Dev | Check the first line spelling exactly (section 3). If it is exact, re-export the file once. | It still happens. Send the file name and time, not the file itself. |
| E10 | No report for an hour, a report with fewer rows than the input, or no `run done` line in the log | The bot stops the whole run. Known causes: `data/output/` or `logs/` is missing, the input file is missing, billing sends back an unreadable rejection, or billing can't be reached while the bot checks for a duplicate. | Nothing ops can see. The reason goes only to a console window. The report stops at the last finished invoice. | System | Ops, then Dev | 1. Check that `data/output/`, `logs/` and `data/input/invoices.csv` exist. 2. Wait for the next hourly run. It starts again from the top, and already-sent invoices come back Duplicate. | The next run stops too, especially if it stops right after the same invoice. Send the run times, the last invoice number in the partial report, and `logs/sync.log`. |
| E11 | A run takes over an hour, two runs' lines are mixed in the log, or a report is overwritten | Many Failed invoices, each taking 1-6 minutes. The bot has no lock against two runs at once. | `logs/sync.log` (timestamps); `data/output/` | System | Dev | Treat it as E08 for the invoices. Tell dev runs are overlapping. | Always. Send the log and the times. |

## 8. What the bot recovers by itself

| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing doesn't answer, is busy, or returns a temporary error | Checks whether billing already has the invoice. If not, waits and sends again. | 5 tries, waits of 2+4+8+16+32 s. About 1 minute per invoice; up to about 6 minutes if billing doesn't answer at all (30 s wait per call). | Status Failed, moves to the next invoice (E08) |
| Billing created the invoice but the confirmation got lost | The existence check above finds it and does not send it again | Same as above | Synced - "Created (confirmed after retry)" (E07) |
| An invoice was not synced this hour | The next hourly run sends it again, because the input file is never emptied | As long as the row stays in the finance export | If finance removes the row, nobody retries it |
| The whole run stopped | No recovery inside the run | - | The next hourly run starts again from row 1 (E10) |
| Invalid input row | No retry | - | Stays Invalid every hour until the input is fixed |

## 9. Known incidents

The history records no dates or versions for these incidents. Both were fixed before 0.9.1.

| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| before 0.9.1 | The same invoice appeared twice in billing | Billing timed out but had created the invoice; the bot sent it again | Fixed: the bot now checks billing before every retry. Still **search billing before re-processing any Failed invoice** (E08). Duplicates are less likely but still possible when billing is slow to show a new invoice. | Partly |
| before 0.9.1 | The hourly run stopped partway, report incomplete | Billing's "too many requests" limit was not handled | Fixed: up to 5 tries with waits. If a run still stops, follow E10. | Partly |

## 10. Releases and version check

- How an update arrives: there is no launcher or auto-update. A developer replaces the bot files in the bot folder by hand.
- How to confirm it applied: the bot does not show or log its version. Ask in #rpa-support for the version installed after each update. The current version is 0.9.1.
- If the version did not change: report it in #rpa-support. Don't copy files yourself.
- Only one machine may run the bot (section 3).

## 11. Escalating to dev

Post in **#rpa-support** with:
- the exception ID from section 7;
- the invoice number(s);
- the report file name, e.g. `sync_report_20261023_1400.csv`, and the run time;
- `logs/sync.log`, which holds invoice numbers and statuses only.

Never send:
- the `.env` settings file or the billing key;
- rows copied out of the report or the input file. They contain customer names, so give the file name and invoice numbers instead.

## 12. Failure-class check

- F1 (missing prerequisite) covered in section 3 and E10. The missing output folder is also Fix #1.
- F2 not applicable: the bot has no window and drives no desktop app. It only sends data to billing online.
- F3 (target slower than expected) covered in E08 and section 8: 30 s wait per call, 5 tries. Fix #9 covers errors that are retried pointlessly.
- F4 (input read differently) covered in section 3 input rules and E01-E03, E09. Also Fix #12 and #15.
- F5 (marked done when nothing was done) covered in E07. "Confirmed after retry" can't tell whether this run created the invoice. Fix #6.
- F6 (action happened but not confirmed) covered in E08: search billing before re-processing. Incident 1. Fix #7.
- F7 (business exception shown as success) covered in section 6 and E06: partially rejected invoices show Synced. Fix #5.
- F8 (endless re-runs / watchdog) covered in section 4 (no watchdog), section 8, E05 (hourly Duplicates) and E11 (overlapping runs). Fix #10.
- F9 (stopping the wrong way) covered in section 4: End the task in Task Scheduler. Fix #13 covers the unconfirmed ops access.
- F10 (one stuck item blocks the queue) covered in E10. One bad invoice can stop the whole run and, if it stays in the file, every hourly run after it. Fix #4 and #7.
- F11 (cause hidden from ops) covered in section 5 and E08, E10. Fix #2 and #3.
- F12 (update didn't arrive) covered in section 10. Fix #11.

## 13. Fix before handover (dev team)

| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | `data/output/` is not in the repo and is never created. `open(report, "w")` raises FileNotFoundError. | On a fresh machine every run stops before row 1, and no report or log line is written | `sync.py:68-69` |
| 2 | Network errors and crashes go to `print()` (stdout) only, never to `sync.log` | Ops can't see why a run stopped (E10). Task Scheduler discards stdout. | `sync.py:52`, `sync.py:84` |
| 3 | Failed message is fixed text, and neither the last HTTP code nor the error is kept. The log line is status only. | An expired key, a billing outage and a rate limit all look the same (E08) | `sync.py:63`, `sync.py:75` |
| 4 | A non-JSON 422 body makes `r.json()` raise, which crashes the run | One bad invoice stops the whole run every hour while it stays in the file | `sync.py:59` |
| 5 | HTTP 207 (partial) is stored as Status `Synced` | Ops filtering on Status miss rejected lines (F7) | `sync.py:54-55` |
| 6 | `exists()` returning 200 is reported as "Created (confirmed after retry)" even if the invoice existed before this run, e.g. after a 400 | Report claims a creation that may not have happened (F5) | `sync.py:60-61` |
| 7 | `exists()` has no try/except, and any non-200 counts as "not found" | A network outage crashes the run instead of retrying. A 429/5xx on the check leads to a re-POST, so duplicates are still possible (F6). | `sync.py:41-43`, `sync.py:60` |
| 8 | Missing `BILLING_API_KEY` sends an empty Bearer token silently. Missing `BILLING_API_URL` silently falls back to `billing.example.test`. | Bad settings look like ordinary Failed rows, about 1 minute each | `sync.py:18-19` |
| 9 | 400/401/403/404 are retried 5 times, `Retry-After` is ignored, and the bot sleeps 32 s after the last attempt | Slow runs, wasted rate-limit budget, overlapping hourly runs (E11) | `sync.py:60-62` |
| 10 | Input is never archived. There is no lock against overlapping runs. Report name has minute precision and is opened with `"w"`. | Duplicate noise every hour hides real duplicates. Overlapping runs can overwrite a report (E11). | `sync.py:68`, `sync.py:72` |
| 11 | Version exists only in README. It is not logged or shown, and there is a stray line "chore: bump to 0.9.1" in the README. | Ops can't confirm an update applied (F12) | `README.md:5-6` |
| 12 | Input opened as `utf-8`, not `utf-8-sig` | An Excel BOM makes every row "missing invoice_no" (E09) | `sync.py:69` |
| 13 | The Task Scheduler task definition is not in the repo, and ops access to Task Scheduler is not confirmed | Ops can't stop the bot or see run history (F9) | n/a: add the task export and agree access |
| 14 | Report copies `customer_name` into the output folder ops open | Customer personal data is spread to every report file | `sync.py:71`, `sync.py:76` |
| 15 | `nan` / `inf` amounts pass validation | Bad amounts reach billing | `sync.py:33-37` |
| 16 | Retries are not logged | Ops can't tell whether, or how often, the bot retried | `sync.py:48-62` |
| 17 | The two "fix" commits only add comment lines. The retry and check-before-retry code came with the initial import, and there are no regression tests. | Fixes can't be traced or protected from regressing | `sync.py:86-87` |
| 18 | A `BOT_WAS_RUN` marker is written to the project root every run and never read | Clutter. It shows whether the bot ran, not whether it finished. | `sync.py:67` |