# Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue

## 1. What the bot does

Every hour, the bot takes the approved invoices from the finance export file (`data/input/invoices.csv`) and creates each one in the billing system through the billing system's web API. It reads the whole file on every run. For each invoice it writes one line to a report file in `data/output/`, named `sync_report_<date>_<time>.csv`, with the result (Status) and the reason (Message). It has no screen and drives no desktop app or website. Windows Task Scheduler starts it.

## 2. How it works

```
Windows Task Scheduler (every hour)
   └─ starts → Invoice sync (no window)
                 ├─ reads  → data/input/invoices.csv   (finance export)
                 ├─ sends  → Billing system API         (over the network)
                 ├─ writes → data/output/sync_report_<date>_<time>.csv   ← ops reads this
                 └─ writes → logs/sync.log              (dev only)
```

- **Task Scheduler** starts the bot once an hour. The task's settings are not stored with the bot.
- **Invoice sync** runs in the background. There is nothing on screen to watch or click.
- **The finance export** is the input. Every row is one invoice.
- **The billing system** is where invoices are created. The bot only talks to it over the network.
- **The report file** is the only place ops can see results. A new one is written every run.
- **The log file** holds only the invoice number and status. Ops cannot open it.

**One run, step by step:**
1. Open the finance export and start a new report file.
2. For each invoice, check the row: invoice number, customer ID, amount and currency must be filled in, and the amount must be a number above zero. If the check fails, the row is marked **Invalid** and the bot does not send it.
3. Send the invoice to billing.
4. If billing says "created", the row is **Synced**. If billing says it already has the invoice, the row is **Duplicate**. If billing rejects the data, the row is **Invalid**.
5. If billing gives any other answer, or does not answer within 30 seconds, the bot asks billing whether the invoice exists now. If it does, the row is **Synced**. If not, it waits and tries again, up to 5 times. After 5 tries the row is **Failed**.
6. Write the row to the report and go to the next invoice.
7. After the last invoice, the bot writes "run done" to its log. Ops cannot see this line.

## 3. Before a run

- [ ] The folder `data/output/` exists next to the bot. The bot does not create it. Without it, the run stops before the first invoice and no report appears.
- [ ] The settings file (`.env`) is next to the bot and holds the billing address and key. Without the key, every invoice ends as **Failed** after about a minute each (see E08).
- [ ] The finance export is at `data/input/invoices.csv`, and its first line has the column names `invoice_no, customer_id, customer_name, amount, currency`, spelled exactly like that.
- [ ] Every row has an invoice number, customer ID, amount and currency.
- [ ] The amount is a plain number with a dot for decimals:
  - Bad: `1,250.00`, `$1250`, `1250,00`
  - Good: `1250.00`
- [ ] Do not open the export in Excel and save it again. Excel can change the file so that every row fails as "missing invoice_no", or so that the run stops.
- [ ] Only **one** machine runs the hourly task. Two machines on the same file can create the same invoice twice.
- [ ] The bot is version 0.9.1 (see section 10).

## 4. During a run

- There is nothing on screen. Do not start the bot by hand while the hourly task may be running. The bot has no protection against two runs at once.
- To stop the bot: in **Task Scheduler**, right-click the task → **End**. To stop the hourly runs, **Disable** the task. If ops has no access to Task Scheduler, raise a ticket.
- A run can be slow. Each invoice that billing does not accept can take **up to about 6 minutes** (5 tries, 30 s wait for each answer, pauses of 2, 4, 8, 16 and 32 s). Many such invoices can make one run last longer than an hour, so the next hourly run may start on top of it.
- The bot has **no** stuck detection and no automatic restart. If no new report has appeared for more than 2 hours, raise a ticket.
- Expect every run to list **every** invoice in the export again. An invoice synced in an earlier hour shows as **Duplicate** in every later report. That is normal.

## 5. Where to look when something goes wrong

Ops can open only the output folder `data/output/` and its report files.

| Question | Where to look |
|---|---|
| Did the run happen? | `data/output/`: a new `sync_report_<date>_<time>.csv` for that hour. No new file means the run did not start or stopped at once (E09). |
| Did the run finish? | **Ops cannot tell for certain.** A report that has fewer rows than the previous hour's report, or that ends earlier than usual, means the run stopped partway (E09). The "run done" line is in the log only (Fix item 9). |
| Which invoices failed? | The report file, **Status** column: `Failed`, `Invalid`, and `Synced` with Message "Created (some lines rejected)" |
| Why did one invoice fail? | The report file, **Message** column, on that invoice's row |
| Did the bot already retry? | Yes, always: every `Failed` row was tried 5 times. Every invoice is also tried again in the next hourly run. |
| Did an earlier run already sync it? | Open the earlier report files in `data/output/` and look for the invoice number with Status `Synced` |
| Which version is running? | Ops cannot see it (Fix item 15). Ask in a ticket. |

## 6. Statuses

The report is a plain CSV file, so it has no colours. The Status column holds one of these values.

| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | Billing created the invoice | Nothing |
| Synced | Created (confirmed after retry) | Billing did not confirm the first send, but the bot then found the invoice in billing | Nothing |
| Synced | Created (some lines rejected) | **Needs follow-up.** Billing created the invoice but dropped some of its lines. It is shown as a success, but it is not complete. | Raise a ticket (E06) |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number. This is usually the bot's own earlier run. | Nothing, if an earlier report shows it as Synced. Otherwise see E05. |
| Invalid | missing `<column names>` | A required column is empty in the export | Fix the export (E01) |
| Invalid | amount must be positive | Amount is zero or negative | Fix the export (E02) |
| Invalid | amount '`<value>`' is not a number | The amount could not be read as a number | Fix the export (E03) |
| Invalid | billing's own reason, or "Rejected by billing" | Billing refused the invoice data | Read the reason, then fix the export (E04) |
| Failed | No success after 5 tries | Billing did not accept it after 5 tries | Wait for the next hour (E08) |

## 7. Exception catalogue

| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing invoice_no" (or customer_id, amount, currency) | Before sending, it checks that these four columns are filled in | Report, Status + Message | Business | Ops | Ask finance to fill in the column in the export. The next hourly run picks it up. | **Every** row says "missing invoice_no": the export's format changed (often after it was saved in Excel). Send the report file name. |
| E02 | Invalid - "amount must be positive" | Before sending, it checks that the amount is above zero | Report, Status + Message | Business | Ops | Ask finance to correct the amount, or remove credit notes from the export | Never; this is a data issue |
| E03 | Invalid - "amount '…' is not a number" | Before sending, it tries to read the amount as a number | Report, Status + Message | Business | Ops | Ask finance to write the amount as a plain number, e.g. `1250.00` | Many rows at once after a change to the export |
| E04 | Invalid - billing's reason, or "Rejected by billing" | Billing answers that the invoice data is wrong | Report, Status + Message | Business | Ops | Read the Message, fix the row with finance, and wait for the next hour | The Message is unclear, or the same row is rejected after the fix. Send the invoice number, report file name and Message. |
| E05 | Duplicate - "Invoice already exists in billing" | Billing answers that the invoice number already exists | Report, Status + Message | Business | Ops | Search earlier reports for the invoice. If one shows it as Synced, do nothing. | No earlier report shows it as Synced. Someone may have created it in billing by hand, or the number is reused. Send the invoice number and report file names. |
| E06 | Synced - "Created (some lines rejected)" | Billing answers "partly created" | Report, Status + Message only | Business | Dev | Raise a ticket. Ops cannot see which lines were dropped. | Always. Send the invoice number and report file name. |
| E07 | Synced - "Created (confirmed after retry)" | The send failed or timed out, then the bot asked billing and found the invoice | Report, Status + Message | System | Bot | Nothing. The bot did not send it again. | Finance reports the invoice twice in billing (E11) |
| E08 | Failed - "No success after 5 tries" | Billing gave no usable answer (busy, down, too many requests, wrong key) or no answer in 30 s, 5 times in a row | Report, Status + Message. The real reason is not recorded. | System | Bot, then Ops | Wait for the next hourly run; it tries again. Check the next report for this invoice. | The same invoice is Failed in **2 reports in a row**, or **most rows** in one report are Failed (billing is down or the key is wrong). Send the report file names and the times. |
| E09 | No new report for an hour, or a report that stops partway | The run stopped. Known causes: billing did not answer while the bot was checking for a duplicate; billing sent an unreadable rejection; the output folder or export file is missing. | **Nowhere ops can see.** The reason is lost (Fix items 2-4). | System | Dev | Raise a ticket. Invoices after the stop point were not handled in that run. | Always. Send the hour, the report file name (if any) and the **last invoice number** in it. If the same report stops at the same invoice every hour, say so: that invoice is blocking every run. |
| E10 | Report has rows for the same hour twice, or a report is overwritten | Two runs ran at the same time (a slow run overlapped the next one, or the bot was started by hand) | Report files in `data/output/` | System | Dev | Raise a ticket. Two runs at once can create duplicates in billing. | Always. Send the report file names and times. |
| E11 | Finance or billing reports the same invoice created twice | The bot sent it again because billing timed out after creating it, and the check before the retry did not see it | Nowhere in the bot. The report shows Synced or Failed. | System | Dev | Raise a ticket. Billing must void the extra copy. | Always. Send the invoice number and the report file names that list it. |

## 8. What the bot recovers by itself

| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing does not answer | Waits 30 s for each answer | 30 s per request | Counts as one failed try |
| Billing busy, down, "too many requests", or any unexpected answer | Asks billing whether the invoice already exists. If not, pauses and sends again. | 5 tries; pauses of 2, 4, 8, 16, 32 s; up to about 6 min per invoice | Row = **Failed**, then the next invoice |
| Invoice created but the bot did not get the confirmation | The check before each retry finds it, and the bot does not send it again | Once per failed try | Row = **Synced** "confirmed after retry" |
| Failed or Invalid invoice | The next hourly run sends every invoice in the export again | Every hour, for as long as the invoice is in the export | Never gives up; it shows in every report |
| Run stopped (E09) | Nothing. There is no restart within the hour. | - | The next hourly run starts again from the top |

## 9. Known incidents

Both were fixed before version 0.9.1. Their dates are not recorded.

| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | Invoices created **twice** in billing | Billing timed out after it had already created the invoice, so the bot sent it again | Partly fixed: the bot now checks billing before each retry. If that check itself fails to get an answer, the run stops (E09), and a duplicate is still possible (E11). Raise a ticket so billing can void the extra copy. | Partly (clean-up only) |
| Before 0.9.1 | Hourly run stopped when billing answered "too many requests" | Billing's rate limit was not retried | Fixed: up to 5 tries with pauses. If many rows are Failed, see E08. | Partly |

## 10. Releases and version check

- How an update arrives: there is no automatic update. A developer copies the new files to the bot machine.
- How to confirm it applied: the version is written only in the bot's README file, which ops cannot open. Ask the developer to state the version in the ticket when they finish the update (Fix item 15).
- If the version did not change: raise a ticket. Only one machine runs the hourly task, so there is no risk of an old machine undoing a new one, unless a second machine is set up (section 3).

## 11. Escalating to dev

Raise a ticket in the **RPA Support** queue. Send:
- the exception ID from section 7 (e.g. E09)
- the hour of the run
- the report file name from `data/output/`
- the invoice numbers involved

Never send: the API key, the `.env` settings file, customer names, or whole report files in chat. The report contains customer names; attach it to the ticket only.

## 12. Failure-class check

- F1 covered in section 3 (output folder, settings file, export location), E08 and E09.
- F2 not applicable: the bot drives no windows. It only talks to billing over the network.
- F3 covered in E08, E09 and section 8: the 30 s answer limit and 5 tries.
- F4 covered in section 3 input rules, E01-E03, and Fix item 12 (Excel-saved file).
- F5 covered: the bot writes Synced only when billing confirms the invoice. A stopped run writes no success for the rows it did not reach (E09). The partial-success case is under F7.
- F6 covered in E07, E09 and E11, and Fix items 2 and 14.
- F7 covered in section 6 (Synced "some lines rejected"), E06 and Fix item 6.
- F8 covered in section 4 (the whole file is re-sent every hour, no stuck detection), E10, and Fix items 10 and 11.
- F9 covered in section 4: stop it in Task Scheduler.
- F10 covered in E09: an invoice that makes the run stop blocks every invoice after it, every hour. Fix items 2 and 3.
- F11 covered in section 5 and Fix items 4, 5 and 9: the reasons are in the log or lost, not in the report.
- F12 covered in section 10 and Fix item 15.

## 13. Fix before handover (dev team)

| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The billing API key is committed to git in `.env` | The key is exposed to anyone with the repo. Rotate the key and remove the file from history. | `.env` (commit 76b9600) |
| 2 | The duplicate check has no error handling. A timeout or network error on it stops the whole run. | Run stops (E09) exactly when billing is slow, which is the case the 504 fix was meant for | `sync.py:41-43`, `sync.py:60` |
| 3 | A 422 reply without JSON stops the whole run, and the same row then stops every hourly run | One invoice blocks all later invoices, every hour (E09) | `sync.py:59` |
| 4 | The crash reason and network errors go to stdout only, which Task Scheduler discards. Nothing reaches the log or the report. | The cause of E09 is lost | `sync.py:52`, `sync.py:81-85` |
| 5 | The log holds only invoice number and status: no HTTP code, attempt count or error. "Failed" never says why. Ops cannot open the log anyway. | Ops cannot tell a wrong key from an outage from a rate limit (E08) | `sync.py:63`, `sync.py:75` |
| 6 | HTTP 207 (partial) is written as Status "Synced" | A partly created invoice looks like a success (F7) | `sync.py:54-55` |
| 7 | 400, 401, 403 and 404 are retried 5 times like temporary errors, with no abort for the whole run | A wrong or expired key makes every invoice take about 1-6 min and end as Failed; the run can overlap the next | `sync.py:60-62` |
| 8 | `data/output/` is not created by the code | The first run on a new machine stops with no report | `sync.py:68-69` |
| 9 | No end-of-run marker that ops can see ("run done" is in the log only) | Ops cannot tell a finished run from a stopped one | `sync.py:77` |
| 10 | No lock against overlapping runs. The report name has minute precision and is opened for overwrite. The Task Scheduler task settings are not in the repo. | Overlapping runs can create duplicates and overwrite reports (E10) | `sync.py:68-69` |
| 11 | No resume and no memory of synced invoices: every row is posted again every hour | Report fills with "Duplicate" rows, it uses up the rate limit, and the risk of duplicates grows | `sync.py:72-74` |
| 12 | The export is read as `utf-8`, not `utf-8-sig`. A file saved by Excel with a BOM would make every row "missing invoice_no" (possible, not seen yet). | Whole run Invalid after one re-save | `sync.py:69` |
| 13 | `Retry-After` is ignored, a duplicate-check GET is added to every failed POST, and the bot sleeps 32 s after the final attempt | Slower runs and more rate-limit hits | `sync.py:60-62` |
| 14 | The duplicate check treats any non-200 (429, 5xx) as "not there" and posts again | Duplicates are still possible (E11). Needs an idempotency key or a billing-side unique constraint. | `sync.py:43` |
| 15 | The version exists only in the README (with a stray "chore: bump to 0.9.1" line), not in the report or log | Ops cannot confirm which version is running | `README.md:5-6` |
| 16 | The report copies customer names | The output folder holds customer data | `sync.py:76` |
| 17 | `nan` and `inf` pass the amount check, and currency is not checked against a list | Bad rows reach billing and come back as Invalid or Failed | `sync.py:33-37` |
| 18 | Both "fix" commits contain only a comment line, and there are no regression tests | The incident fixes were never reviewed as code changes | `sync.py:86-87`, commits 244fb15 and 8656aa3 |
| 19 | The `BOT_WAS_RUN` marker file is written and never read | Clutter, and it can be mistaken for a lock | `sync.py:67` |