Rpa_Skills/tests/evals/results/handoff/h-api-sync-full-r3.md

# Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel #rpa-support

## 1. What the bot does

Every hour it takes the approved invoices in the finance export file (`data/input/invoices.csv`) and creates them in the billing system through billing's web API. It checks each invoice first: required fields filled in and a valid, positive amount. It then sends the invoice to billing. If billing is busy or slow, it retries. For each run it writes one report file in `data/output/`, listing every invoice with a Status and a Message, and adds a few lines to `logs/sync.log`. It has no window or screen of its own and does not drive any desktop app or browser. It talks to billing directly over the network.

## 2. How it works

```
Windows Task Scheduler (every hour)
   └─ starts → Invoice sync script ──sends invoices──> Billing system (web API)
                    │
                    ├─ reads  → data/input/invoices.csv
                    ├─ writes → data/output/sync_report_<YYYYMMDD_HHMM>.csv
                    └─ writes → logs/sync.log
```

- **Task Scheduler** starts the bot every hour on the bot machine. Nobody needs to click anything.
- **The Invoice sync script** is the bot. It runs with no window, works through the input file from top to bottom, then stops.
- **The billing system** is where invoices are created. The bot only creates invoices and checks whether they exist. It never edits or deletes them.
- **The input file** is the finance export. The bot reads it and never changes it. If the file is not replaced, the next run sends every row again (see Duplicate in section 6).
- **The report file** is one new file per run with columns `invoice_no`, `customer`, `Status` and `Message`. This is the main place to look.
- **The log file** gets one line per invoice (invoice number and status) and a closing `run done` line when the run finishes.

**One run, step by step:**
1. Open the input file and create a new report file for this run.
2. For each row, check that invoice number, customer ID, amount and currency are filled in, and that the amount is a positive number. If not, the row is marked **Invalid** and nothing is sent to billing.
3. Send the invoice to billing.
4. If billing is busy, slow or unreachable, check whether the invoice was created anyway, wait, and try again (up to 5 tries; section 8).
5. Write the outcome to the report and the log, then move to the next row.
6. Write `run done -> sync_report_<time>.csv` to the log.

## 3. Before a run

- [ ] The folder `data/output/` exists next to the bot. **The bot does not create it.** If it is missing, the run stops before processing any invoice (E09).
- [ ] The settings file `.env` is in the bot folder and contains the billing address and the billing access key. Ops never opens or edits this file. If it is missing or the key has expired, every invoice ends up **Failed** (E08).
- [ ] The input file `data/input/invoices.csv` is present and has the columns `invoice_no, customer_id, customer_name, amount, currency`.
- [ ] Input rules, one row per invoice:
  - Invoice number, customer ID, amount and currency must all be filled in. A blank in any of them gives **Invalid - missing ...**.
  - Amount must be a plain positive number. Use a dot for decimals and no thousands separator or currency symbol.
    - Bad: `1,250.00`, `$120`, `-50`, `0`
    - Good: `1250.00`, `120`
  - The bot cannot send credit notes or zero-value invoices, because the amount must be above 0.
- [ ] Any version change is confirmed by the dev team (section 10). The bot does not show its version anywhere ops can see.

## 4. During a run

- There is nothing to watch. A normal run takes seconds to a few minutes.
- Don't edit, rename or move the input file while a run is in progress. Replace it between runs.
- Don't open the newest report file in Excel while the run is still writing it. Wait for the `run done` line in the log.
- **To stop the bot:** in Task Scheduler on the bot machine, right-click the task, choose **End**, and then **Disable** it if it must not start again next hour. If you don't have access to Task Scheduler, ask in #rpa-support. Stopping halfway is safe: the next run re-sends the file, and invoices already in billing come back as **Duplicate**.
- **A run looks stuck** when the newest report file stops getting new rows and there is no `run done` line in the log. The bot has no stuck-detection of its own. Each invoice can take up to about 6 minutes when billing is not responding (section 8), so wait 10 minutes. If a run is still going when the next hourly run starts, escalate (E10).

## 5. Where to look when something goes wrong

| Question | Where to look |
|---|---|
| Did the run finish? | `logs/sync.log`: a line `run done -> sync_report_<time>.csv` for that hour. No such line means the run stopped partway (E09). **"run done" only means the run reached the end, not that the invoices were created.** |
| Which invoices failed? | The newest `data/output/sync_report_<YYYYMMDD_HHMM>.csv`. Filter Status for **Failed** and **Invalid**, and filter Message for **some lines rejected**. |
| Why did one invoice fail? | The **Message** column of the same report. The log shows only the status, never the reason. |
| Did the bot already retry? | **Failed** always means it tried 5 times. **Synced** with "Created (confirmed after retry)" means a retry found the invoice already in billing. Nothing else records the retries ops can see. |
| When did it fail? | `logs/sync.log`: search for the invoice number. Each line starts with date and time. |
| Why did the whole run stop? | Ops cannot see this. The reason is not written to any file ops can open. Escalate (E09). |
| Which version is running? | Ops cannot see this. Ask in #rpa-support. |

## 6. Statuses

The report is a plain CSV file, so there are no colours. The Status column holds one of four values. Read it together with the Message column.

| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | Invoice created in billing | nothing |
| Synced | Created (some lines rejected) | **Invoice created, but billing refused some of its lines. Needs follow-up.** | See E05 |
| Synced | Created (confirmed after retry) | Billing was slow or busy, but on a retry the bot found the invoice in billing | nothing |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number. Normal for rows that were sent in an earlier run. | See E06 |
| Invalid | missing *field names* | A required field is empty in the input. Not sent to billing. | See E01 |
| Invalid | amount must be positive | Amount is 0 or negative. Not sent to billing. | See E02 |
| Invalid | amount '*value*' is not a number | Amount is not a plain number. Not sent to billing. | See E03 |
| Invalid | Rejected by billing, or billing's own explanation text | Billing looked at the invoice and refused it | See E04 |
| Failed | No success after 5 tries | Billing never confirmed it after 5 tries. **It may still have been created.** | See E07 and E08 |

## 7. Exception catalogue

| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing customer_id" (or other field names) | Before sending, it checks that invoice number, customer ID, amount and currency are not blank | Report: Status, Message. Log: invoice number + Invalid | Business | Ops | Ask finance to fill the field in the export. The next hourly run picks the row up again. | The field is filled in the file and it still says missing. Send the invoice number and report file name. |
| E02 | Invalid - "amount must be positive" | The amount is 0 or less | Report: Status, Message | Business | Ops | Check with finance. The bot cannot send zero or negative amounts (credit notes); these must be entered in billing by hand. | Finance says this kind of invoice must be synced automatically. |
| E03 | Invalid - "amount '1,250.00' is not a number" | The amount cannot be read as a number (comma, currency symbol, text) | Report: Status, Message | Business | Ops | Ask finance to export the amount as a plain number, e.g. `1250.00`. | The amount looks plain and is still rejected. |
| E04 | Invalid - "Rejected by billing" or another explanation text | Billing replied that the invoice data is not acceptable | Report: Status, Message (billing's own text) | Business | Ops | Read the message (e.g. unknown customer, wrong currency) and get the data fixed at the source. Retrying unchanged data gives the same result. | The message is unclear, or the data looks right. Send the invoice number, run time and the exact message. |
| E05 | Synced - "Created (some lines rejected)" | Billing replied that it created the invoice but refused some lines | Report: Status, Message. The log shows only "Synced". | Business | Ops | **This is not done.** Open the invoice in billing (or ask the billing team), find the missing lines, and get them added by hand or fixed by finance. The bot will not resend them, because the invoice now exists. | You cannot tell which lines are missing. Send the invoice number and run time. |
| E06 | Duplicate - "Invoice already exists in billing" | Billing replied that this invoice number already exists | Report: Status, Message | Business | Ops (usually nothing) | If the invoice was sent in an earlier run, do nothing. This is normal when the input file is not replaced. If this is the **first** time you see this invoice number, check in billing that the existing invoice is the same one (same customer and amount). | The existing invoice in billing is a different one. Send the invoice number and run time. |
| E07 | Failed - "No success after 5 tries" on one or a few rows | Billing was busy, slow (no reply within 30 s), down, or gave an unexpected reply on all 5 tries. Before each retry it checked whether the invoice existed and did not find it. | Report: Status, Message. Log: invoice number + Failed | System | Bot, then Ops | **First check in billing whether the invoice was created anyway.** If it is there, nothing more is needed; the next run will show Duplicate. If it is not there and the row is still in the input file, the next hourly run tries again. | The same invoice is Failed in 3 runs in a row. Send the invoice number, run times and the report file names. |
| E08 | Failed on **all or almost all** rows, and the run takes much longer than usual | Same as E07, but for every row. This usually means the billing access key is missing or expired, the billing address is wrong, or billing is down. | Report: every Status Failed. Log: long gaps between lines. | System | Dev | Nothing ops can fix. Don't change the input file. | Always. Send the run time, the report file name and the `logs/sync.log` file. |
| E09 | No `run done` line for that hour; the report is missing, empty, or stops partway | The run stopped early. Known causes: `data/output/` folder missing, input file missing, network failure while the bot was checking whether an invoice exists, or an unexpected reply from billing. | `logs/sync.log` (the closing line is missing). The cause is not in any file ops can open. | System | Dev | Check that `data/output/` and the input file exist (section 3). Rows written to the report before the stop are real results. Rows after it were not processed; the next hourly run re-sends them. | It happens in 2 runs in a row, or the folders are present. Send the run time, the last report file name and `logs/sync.log`. |
| E10 | Two report files close together for the same hour, or a run still going when the next one starts | A run took more than an hour (many slow or failed rows). Nothing stops two runs at once. | Report file names and log times | System | Dev | Check first in billing for any invoice that shows Failed, as in E07. | Always. Send both report file names and the log. |

## 8. What the bot recovers by itself

| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing busy (too many requests), down, slow (no reply in 30 s), or an unexpected reply for one invoice | Checks whether the invoice was created anyway. If not, waits and tries again. Waits grow: 2, 4, 8, 16, 32 s. | 5 tries per invoice. About 1 minute of waiting, and up to about 6 minutes per invoice if every call times out. | Invoice marked Failed (E07), next invoice |
| The invoice was created but billing's reply got lost | The check before each retry finds it | Same 5 tries | Synced - "Created (confirmed after retry)" |
| Invoice sent again in a later run | Billing answers "already exists" | none | Duplicate (E06). Nothing is created twice. |
| Run stopped partway, or Failed / not-yet-sent rows | No restart within the hour. Task Scheduler's next hourly run starts fresh and re-sends the whole input file. | Once per hour | If the same problem repeats over 2-3 runs, escalate (E07, E09) |

Before acting on a Failed row, wait for at least one more hourly run. The bot will usually have retried it already.

## 9. Known incidents

Exact dates are not recorded. Both incidents happened before version 0.9.1.

| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | The same invoice appeared twice in billing. Finance or billing noticed it; the bot reported nothing. | Billing created the invoice but timed out before replying, and the bot's retry sent it again | Fixed in 0.9.1: the bot checks billing before each retry. The check can still miss when billing is down, so **for any Failed row, check billing before re-sending it by hand** (E07). | Partly |
| Before 0.9.1 | Hourly run died partway through when billing said "too many requests". Invoices after that row were not sent. | The bot treated "too many requests" as fatal | Fixed in 0.9.1: up to 5 tries with waits (section 8). A run can still stop for other reasons (E09). | Partly |

## 10. Releases and version check

- **How an update arrives:** the dev team copies the new version into the bot folder on the bot machine. There is no launcher or auto-update. The next hourly run uses the new code.
- **How to confirm it applied:** ops cannot see the version. The bot writes it to no file ops can open. Ask the dev team to confirm in #rpa-support after every update.
- **If the version did not change:** report it in #rpa-support. Don't copy files yourself.

## 11. Escalating to dev

Post in the Teams channel **#rpa-support** (RPA dev team). Send:
- the exception ID from section 7 (e.g. E08)
- the invoice number(s)
- the run time (from `logs/sync.log`)
- the report file name (`sync_report_<YYYYMMDD_HHMM>.csv`)
- the `logs/sync.log` file

Never send: the `.env` settings file or anything from it, the billing access key, or customer names and amounts pasted into chat. Refer to rows by invoice number.

## 12. Failure-class check

- F1 covered in section 3 (output folder, settings file, input file) and E08, E09.
- F2 not applicable: the bot has no windows. It talks to billing over its web API and drives no desktop app.
- F3 covered in E07 and section 8 (30 s reply limit, 5 tries).
- F4 covered in section 3 input rules (bad and good amount examples) and E01-E04.
- F5 covered in section 5 ("run done" is not success) and E08. Fix #4 and #5 in section 13.
- F6 covered in E07 (check billing before re-sending) and section 9. Fix #5 in section 13.
- F7 covered in section 6 and E05 ("some lines rejected" shows as Synced). Fix #1 in section 13.
- F8 covered in E10 and section 8. There is no automatic re-run loop; the only repeat is the hourly schedule. Fix #8 in section 13.
- F9 covered in section 4 (End the task in Task Scheduler).
- F10 not applicable: every invoice is handled on its own. No batch waits for all rows, and nothing locks the next run. A slow row only delays the run (see F8).
- F11 covered by fixes #2, #3 and #10 in section 13. The reason a run stopped and network errors are not written to any file ops can open, and the log has no reason text.
- F12 covered in section 10 and fix #9 in section 13. The version is not visible to ops.

## 13. Fix before handover (dev team)

| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | HTTP 207 (partial) is written as Status `Synced` | Ops filtering on Status thinks the invoice is complete when lines are missing | `sync.py:54-55` |
| 2 | Crash reason (`CRASH: ...`) and network errors are only `print`ed to stdout, never logged | Ops cannot find why a run stopped (E09). Lost unless Task Scheduler captures stdout. | `sync.py:52`, `sync.py:84` |
| 3 | Log line has status only, no Message | Ops must open the report to learn why | `sync.py:75` |
| 4 | `data/output/` is not in the repo and never created | Fresh machine: the first run crashes before any invoice | `sync.py:68-69` |
| 5 | `exists()` is not in a try block, and any non-200 counts as "not there" | A network error during the check crashes the run. During an outage it can re-POST an invoice that was already created (possible duplicate). | `sync.py:41-43`, `sync.py:60` |
| 6 | A 422 with a non-JSON body makes `r.json()` raise | Whole run crashes on one bad reply | `sync.py:59` |
| 7 | Every non-success code is retried (400/401/403/404 too, not only 429/5xx as the docstring says), and `Retry-After` is ignored | A bad or expired key shows as generic `Failed` on every row and takes about 1-6 min per row (E08) | `sync.py:47`, `sync.py:54-62` |
| 8 | No lock against overlapping runs. The report name has minute resolution, so a same-minute run overwrites the report. It also sleeps 32 s after the last try for nothing. | Slow runs overlap the next hour (E10), and reports can be lost | `sync.py:62`, `sync.py:68` |
| 9 | Version only in `README.md` (plus a stray "chore: bump to 0.9.1" line). Not logged or shown anywhere. | Ops cannot confirm an update applied | `README.md:5-6` |
| 10 | `Invalid` is used both for local checks and for billing's 422 rejection | The status alone doesn't say whether to fix the file or ask billing | `sync.py:59`, `sync.py:74` |
| 11 | The report stores customer names. No retention policy, and `logs/sync.log` is never rotated. | Sensitive data piles up in `data/output/`. The log grows forever. | `sync.py:24`, `sync.py:71`, `sync.py:76` |
| 12 | A `BOT_WAS_RUN` marker file is written every run and never read | Unclear purpose. Confirm or remove. | `sync.py:67` |
| 13 | Task Scheduler task definition (and whether stdout is captured) is not in the repo | Can't rebuild the schedule on a new machine | `sync.py:3` |
| 14 | No tests cover the two incident fixes (429 retry, check-before-retry) | Regressions would come back unnoticed | whole repo |