Rpa_Skills/tests/evals/results/handoff/h-api-sync-full-r4.md

# Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**

## 1. What the bot does

Every hour it takes the approved invoices from the finance export file and sends each one to the billing system. Input: the CSV file `data/input/invoices.csv`, one invoice per row, with an invoice number, a customer ID, a customer name, an amount and a currency. Output: one report file per run in `data/output/` with a status and a message for each invoice, and a line per invoice in `logs/sync.log`. It has no screens and drives no desktop apps. It talks to the billing system over the network in the background. It starts by itself from Windows Task Scheduler.

## 2. How it works

```
Windows Task Scheduler (every hour)
   └─ starts → Invoice sync (background, no window)
                    │  reads → data/input/invoices.csv
                    ├─ sends each invoice ──> Billing system (online)
                    ├─ writes → data/output/sync_report_<date>_<time>.csv
                    └─ writes → logs/sync.log
```

- **Task Scheduler** starts the bot once an hour. Nobody has to click anything.
- **Invoice sync** is the bot. It runs without a window and stops by itself when it reaches the end of the file.
- **The input file** is read from top to bottom, every row on every run.
- **The billing system** receives each invoice and answers "created", "already exists", "rejected" or nothing at all.
- **The report file** is the only place that says *why* an invoice did not go through (the Message column).
- **sync.log** gets one line per invoice with its status only, and a final "run done" line when the run finished cleanly.

**One run, step by step:**
1. Open a new report file named after the date and minute the run started.
2. For each row: check the row (invoice number, customer ID, amount and currency present; amount a positive number).
3. If the row is fine, send it to billing. If billing doesn't answer properly, ask billing whether the invoice is already there, wait, and try again. Up to 5 tries.
4. Write the result to the report and the log.
5. After the last row, write "run done -> <report name>" in the log.

## 3. Before a run

These are set up once by the dev team. Ops can't check most of them, but the message each one gives is in section 7.
- [ ] The folders `data/output/` and `logs/` exist. If either is missing, the run stops before it does anything (E11).
- [ ] The settings file next to the bot has the billing address and the billing key. If the key is missing or has expired, every invoice ends **Failed** and the run is very slow (E09).
- [ ] The finance export is saved as `data/input/invoices.csv` with the header row `invoice_no, customer_id, customer_name, amount, currency`.
- [ ] Input rules:
  - **Amount is a plain number.** Good: `1200.50`. Bad: `1,200.50`, `$1200`, `1200.50 USD`. These all end **Invalid** "is not a number" (E03).
  - **Amount above zero.** Credit notes or zero amounts end **Invalid** (E02).
  - **Invoice number, customer ID, amount and currency are never blank.** Otherwise the row ends **Invalid** "missing …" (E01). A cell that holds only spaces is *not* caught, so check those by eye.
  - **Each invoice number once per file.** The bot doesn't spot repeats in the same file.
- [ ] Remove invoices from the file once they are synced, if the finance process allows it. The bot sends **every row on every hourly run**, so old rows come back as **Duplicate** each hour (E05).

## 4. During a run

- Don't edit, move or save over `data/input/invoices.csv` while a run is going. Wait until the hour's "run done" line appears in `logs/sync.log`.
- Don't open the current report file in Excel while the run is still writing it. Open a copy, or wait for "run done".
- To stop the bot: end the "Invoice sync" task in Windows Task Scheduler. If ops can't open Task Scheduler on the bot machine, ask in #rpa-support. Invoices already sent stay in billing. Stopping it does not undo them.
- A run is normally finished within a few minutes. One stubborn invoice can hold the run for **up to about 6 minutes** while the bot retries it. A run looks stuck when `logs/sync.log` gets no new line for more than 10 minutes and there is no "run done" line. There is no automatic watchdog. Escalate (E13).

## 5. Where to look when something goes wrong

| Question | Where to look |
|---|---|
| Did the run finish? | `logs/sync.log`: the run ends with a line "run done -> sync_report_<date>_<time>.csv". No such line after that hour's invoice lines means the run crashed (E10). |
| Which invoices failed? | The run's report in `data/output/` (named `sync_report_YYYYMMDD_HHMM.csv`): filter the **Status** column for anything except Synced. |
| Why did one invoice fail? | The same report, **Message** column. The log has no reasons, only the status. |
| Which run handled this invoice? | Search `logs/sync.log` for the invoice number. The time on that line matches the report's file name. |
| Did the bot already retry? | Not recorded anywhere ops can see. **Failed** always means 5 tries were made. **Synced - "Created (confirmed after retry)"** means at least one retry happened. |
| Which version is running? | Not shown by the bot. Ask in #rpa-support (see section 10). |

## 6. Statuses

The report is a plain CSV file, so there are no colours. Read the **Status** column first, then the **Message** column.

| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | The invoice is in billing. | Nothing |
| Synced | Created (confirmed after retry) | Billing didn't answer at first. The bot then found the invoice in billing. It may also have been there from an earlier run. | Nothing. If billing shows it twice, see E12. |
| Synced | Created (some lines rejected) | **Needs follow-up.** The invoice was created, but billing refused some of its lines. | Check the invoice in billing and fix the missing lines (E06) |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number. On an hourly re-run this usually just means it was synced in an earlier run. | First time you see it: check in billing (E05). Otherwise nothing. |
| Invalid | missing <field names> | Required fields were blank. The row was not sent. | Fix the input row (E01) |
| Invalid | amount must be positive | The amount is zero or negative. Not sent. | Fix the input row (E02) |
| Invalid | amount '<value>' is not a number | The amount has commas, a currency sign or text. Not sent. | Fix the input row (E03) |
| Invalid | any other text, or "Rejected by billing" | Billing refused the invoice and gave this reason. Not created. | Fix the data as the message says (E04) |
| Failed | No success after 5 tries | Billing never gave a clear answer in 5 tries. **The invoice may or may not be in billing.** | Check in billing first (E08) |

## 7. Exception catalogue

| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing customer_id, amount" (or other fields) | Before sending, it checks that invoice number, customer ID, amount and currency are filled in | Report: Status, Message. Log: status only | Business | Ops | Fill the blank fields in the finance export. The next hourly run picks the row up again. | Never, unless the fields are filled and it still says missing |
| E02 | Invalid - "amount must be positive" | Amount is zero or below | Report: Status, Message | Business | Ops | Check with finance. Credit notes and zero invoices can't go through this bot. | Finance needs credit notes synced |
| E03 | Invalid - "amount '1,200.50' is not a number" | The amount can't be read as a number | Report: Status, Message | Business | Ops | Write the amount as a plain number, e.g. `1200.50`: no thousands commas, no currency sign | Never |
| E04 | Invalid - with a message from billing, or "Rejected by billing" | Billing answers that the invoice data is not acceptable | Report: Status, Message | Business | Ops | Fix the data as billing's message says (e.g. unknown customer, bad currency) | The message is unclear, or the data is already correct |
| E05 | Duplicate - "Invoice already exists in billing" | Billing answers that this invoice number already exists | Report: Status, Message | Business | Ops | **On re-runs this is normal.** The invoice was synced in an earlier hour. Search older reports for the same invoice number with Synced. If there's no earlier Synced, open it in billing: if it's a different invoice with the same number, give it to finance. | The invoice was never synced before and billing doesn't show it |
| E06 | Synced - "Created (some lines rejected)" | Billing answers "created, but part of it was refused" | Report: Message only. **Status says Synced.** | Business | Ops | Open the invoice in billing, compare its lines with the export, and add or fix the missing lines there | You can't see which lines were refused |
| E07 | Synced - "Created (confirmed after retry)" | Billing didn't answer clearly. The bot asked billing whether the invoice exists, and it does. | Report: Message | System | Bot | Nothing. The bot recovered. | Billing shows the invoice twice (E12) |
| E08 | Failed - "No success after 5 tries" | 5 tries with waits of 2, 4, 8, 16 and 32 s, and billing never confirmed the invoice | Report: Status, Message. The real reason is not saved. | System | Ops, then Dev | **1. Check in billing first.** The invoice may have been created even though the bot didn't get an answer. 2. If it's there, nothing to do: the next run will show it as Duplicate. 3. If it isn't, wait for the next hourly run, which tries again by itself. | The same invoice is Failed in 2 runs in a row. Send the invoice number and both report file names. |
| E09 | Most or all invoices in one run are **Failed**, and the run takes very long (about a minute or more per invoice) | Same as E08, for every invoice. Usually billing is down, or the billing key in the settings file is missing or expired. | Report: every row Failed | System | Dev | Don't edit the input. Check billing yourself if you can log in. | Always. Send the report file name and the time of the run. |
| E10 | The hour's invoice lines in `logs/sync.log` stop and there is **no "run done" line**. The report has fewer rows than the input. | The bot hit an unexpected error and stopped. Known triggers: billing unreachable while the bot checks whether an invoice exists, or an odd answer from billing. | Only the missing "run done" line. The error itself is not saved anywhere ops can see. | System | Dev | Invoices before the stop were handled as the report shows. The rest are tried again next hour. If it happens two hours in a row, escalate. | 2 runs in a row. Send the report file name, the last invoice number in the log, and the time. |
| E11 | No new report in `data/output/` for an hour and no new lines in `logs/sync.log` | The run stopped before the first invoice: the input file, the output folder or the logs folder is missing, or Task Scheduler didn't start it | Nothing is recorded | System | Ops, then Dev | Check that `data/input/invoices.csv` exists and has the right name | The input file is there. Send the hours with no report. |
| E12 | The same invoice appears **twice in billing** | Not detected by the bot. Billing finished creating the invoice after the bot gave up waiting, and the bot sent it again. Or two runs overlapped. | Not recorded. Report shows Synced, Failed or Duplicate. | System | Dev (and finance) | Don't delete anything in billing yourself. Collect the invoice number and the report rows for it. | Always |
| E13 | A run is still going after 10 minutes with no new log line, or two runs' lines are mixed in the log | Not detected. There is no watchdog and nothing stops two runs at once. | Log timestamps only | System | Dev | Don't start another run by hand. | Always. Send the time the run started and the last log line. |

## 8. What the bot recovers by itself

| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing doesn't answer, is busy, or answers with an error | Asks billing whether the invoice already exists. If yes: Synced. If not: waits, then sends it again. | 5 tries. Waits of 2, 4, 8, 16 and 32 s (about 1 minute), plus up to 30 s per try waiting for billing. Up to about 6 minutes for one invoice. | Failed - "No success after 5 tries", next invoice (E08) |
| An invoice ended Invalid or Failed | Nothing in that run. **The next hourly run reads the whole file again and retries it.** | Every hour, as long as the row is in the file | Stays Invalid until the input is fixed |
| Run crashed | Nothing. There is no resume. The next hourly run starts from the top of the file again. | - | E10 |

Ops rule of thumb: for one Failed invoice, wait one hour for the next run before acting. For a missing "run done" line or a whole run of Failed, escalate straight away.

## 9. Known incidents

Dates aren't reliable in the history. Both fixes are in version 0.9.1.

| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| before 0.9.1 | The same invoice twice in billing | Billing created the invoice but timed out before answering, and the bot sent it again | Partly fixed: the bot now asks billing whether the invoice exists before retrying. It can still happen when billing is slow (E12). For **Failed** rows, check billing before anything else (E08). | Partly |
| before 0.9.1 | Hourly run stopped partway, with no "run done" line | Billing said "too many requests" and the bot stopped | Partly fixed: the bot now waits and retries. It can still stop if billing is unreachable during the "does it exist" check (E10). | Partly |

## 10. Releases and version check

- How an update arrives: the dev team replaces the bot's files on the bot machine. There is no automatic updater.
- How to confirm it applied: the bot doesn't show or log its version. Ask in #rpa-support for the version on the machine. The version is only written in the bot's README file.
- If the version did not change: report it in #rpa-support. Ops has nothing to re-install.

## 11. Escalating to dev

Post in **#rpa-support** (Teams) and send: the exception ID from section 7, the run time, the report file name (`sync_report_YYYYMMDD_HHMM.csv`), the invoice number(s), and the matching lines from `logs/sync.log`.
Never send: the billing key, the settings file, or customer names and amounts in chat. Refer to invoices by invoice number only.

## 12. Failure-class check

- F1 covered in section 3 (folders, settings file, input file name) and E09 / E11; gaps in section 13 #7, #8.
- F2 not applicable: the bot drives no windows. It only talks to billing over the network.
- F3 covered in E08 / E09 and section 8 (30 s wait per try, 5 tries). No screens, so screen-size rules don't apply.
- F4 covered in section 3 input rules (amount format, blanks, repeated invoice numbers) and E01-E04; gaps in section 13 #9.
- F5 covered in E06 and E10. A crashed run leaves a short report with no "run done" line. Gap in section 13 #3.
- F6 covered in E08 and E12 (check billing before acting on Failed); gap in section 13 #2.
- F7 covered in section 6 and E06 ("some lines rejected" shows Synced); gap in section 13 #4.
- F8 covered in E05 / section 3 (every row resent every hour) and E13 (overlapping runs); gaps in section 13 #5, #6.
- F9 covered in section 4 (end the task in Task Scheduler); gap in section 13 #11.
- F10 not applicable as a queue: one bad invoice never blocks the others, and nothing locks the next run. The nearest case is a slow invoice holding a run for about 6 minutes (section 4, E13).
- F11 covered in section 5; the hidden causes are section 13 #1, #3.
- F12 covered in section 10; gap in section 13 #10.

## 13. Fix before handover (dev team)

| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The reason for Failed (HTTP code, network error) is printed to stdout only. The log has the status but no message. | Ops can't tell an expired key from billing being down. Everything reads "No success after 5 tries". | `sync.py:52`, `sync.py:63`, `sync.py:75` |
| 2 | The exists-check runs straight after the timeout, with no idempotency key on the POST. A slow create is retried, giving a duplicate or a misleading 409 "Duplicate". | Duplicate invoices in billing (E12). Ops chases false duplicates. | `sync.py:50`, `sync.py:60` |
| 3 | `exists()` and `r.json()` are unprotected, there is no per-row try/except, and the crash goes to stdout only. Exit code 0 when every row Failed. | A whole run dies with no saved cause (E10). A run where everything failed looks successful to Task Scheduler. | `sync.py:42`, `sync.py:59`, `sync.py:72-76`, `sync.py:84` |
| 4 | HTTP 207 (some lines rejected) is mapped to Synced | Rows needing follow-up are hidden when the report is filtered on Status (F7) | `sync.py:54-55` |
| 5 | Every row is resent every hour. Nothing records what was already synced. | Hourly Duplicate noise. If billing ever stops returning 409, it creates duplicates. | `sync.py:72-74` |
| 6 | No single-run lock. Report name is to the minute, opened with "w". | Overlapping runs (E13) can double-send and overwrite each other's report | `sync.py:68-69` |
| 7 | `data/output/` and `logs/` are not created by the code. `logs/` missing fails at import. | A fresh machine produces nothing and no error ops can see (E11) | `sync.py:24`, `sync.py:68` |
| 8 | A missing key becomes an empty Bearer token. 400/401/403/404 are retried 5 times like 429/5xx, and Retry-After is ignored. | An expired key turns one run into ~1 min+ per invoice, all Failed (E09) | `sync.py:19`, `sync.py:60-62` |
| 9 | Spaces-only fields and `nan`/`inf` amounts pass validation. Currency isn't checked. Duplicate invoice numbers within a file aren't checked. | Bad rows go to billing instead of being flagged as Invalid | `sync.py:30`, `sync.py:34` |
| 10 | Version isn't logged or shown, and README has a stray "chore: bump to 0.9.1" line | Ops can't confirm an update applied | `README.md:5-6`, `sync.py:77` |
| 11 | The Task Scheduler job is not in the repo or documented (name, account, output redirection), and ops access to it is not confirmed | Ops can't stop a stuck run or see the last result themselves | n/a (scheduler config) |
| 12 | The report's "customer" column holds the customer name, and the whole row (including the name) is sent to billing | Customer names sit in plain CSV files in the output folder that ops shares around | `sync.py:50`, `sync.py:76` |
| 13 | Writes an unused `BOT_WAS_RUN` marker file in the bot folder | Confusing leftover. Ops may think it's a lock. | `sync.py:67` |
| 14 | The "network error" print and the "CRASH" print are the only signal for network problems, and the sleep after the 5th try is wasted (32 s) | Slower runs, no record | `sync.py:52`, `sync.py:62` |