# Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue

## 1. What the bot does

Every hour, the bot takes the approved invoices in the finance export file (`data/input/invoices.csv`) and creates each one in the billing system through the billing system's web API. It has no window and drives no desktop app or website. It only talks to the billing API in the background. For each run it writes one report file to `data/output/`, with one line per invoice saying what happened to it.

## 2. How it works

```
Windows Task Scheduler (every hour)
   └─ starts → Invoice sync (no window) ──sends each invoice──> Billing API
                    │
                    ├─ reads  → data/input/invoices.csv
                    ├─ writes → data/output/sync_report_<YYYYMMDD_HHMM>.csv   (ops can open)
                    └─ writes → logs/sync.log                                  (dev only)
```

- **Windows Task Scheduler** starts the bot at the top of every hour. Nobody clicks anything.
- **Invoice sync** is a background program. It has no screen.
- **Billing API** is the billing system. Invoices the bot creates appear there.
- **The input file** is the finance export. The bot never moves or empties it, so every run reads the whole file again.
- **The report file** is the only record ops can open. Each run makes a new file, named after the date and time the run started, for example `sync_report_20261023_1400.csv` for the 14:00 run.
- **The log** holds one line per invoice (number and status only) plus an end-of-run line. Ops has no access to it.

**One run, step by step:**
1. Task Scheduler starts the bot.
2. The bot creates a new, empty report file in `data/output/`.
3. For each line of the input file, in order:
   1. It checks the line: invoice number, customer ID, amount and currency must be filled in, and the amount must be a number above 0. If the line fails the check, the bot marks it **Invalid** and moves on without sending it.
   2. It sends the invoice to billing.
   3. If billing gives no clear answer, it asks billing "does this invoice exist now?". If yes, it marks the invoice **Synced**. If not, it waits and tries again, up to 5 times (section 8).
   4. It writes the result as one line in the report.
4. When every line is done, it writes "run done" to the log. Ops cannot see this. To check a run finished, see section 5.

## 3. Before a run

- [ ] The folder `data/output/` exists next to the bot. **The bot does not create it.** If it is missing, every run stops at once and no report appears.
- [ ] The folder `logs/` exists next to the bot. If it is missing, the bot stops at once.
- [ ] The settings file (`.env`) with the billing address and key is next to the bot. If it is missing or the key has expired, every invoice ends **Failed** after about a minute of retries each (E09).
- [ ] The input file is at `data/input/invoices.csv` and keeps its header line exactly: `invoice_no, customer_id, customer_name, amount, currency`.
- [ ] Input rules, one line per invoice:

| Field | Rule | Bad | Good |
|---|---|---|---|
| invoice_no, customer_id, currency | Must not be empty | (blank) | `INV-1001` |
| amount | A plain number above 0. No currency sign, no thousands separator. | `$1,200.50`, `0`, `-5` | `1200.50` |
| File format | Save as **CSV UTF-8** without the "BOM" marker. Some Excel "CSV UTF-8" exports add an invisible marker at the start, and then **every** line comes back **Invalid - missing invoice_no** (E04). | Excel "CSV UTF-8" with marker | Plain CSV / UTF-8 without marker |

- [ ] Only one copy of the bot is scheduled. Nothing stops two runs from overlapping (section 4).

## 4. During a run

- There is nothing to watch. The bot has no window. Leave the input file alone while a run is going. Don't open it in Excel and save over it at the top of the hour.
- **To stop the bot:** whoever has access to the bot machine ends the "Invoice sync" task in Task Scheduler, and disables it to stop future runs. Ops has no access to that machine, so raise a ticket. Stopping mid-run is safe: invoices already created come back as **Duplicate** on the next run, not as second copies.
- **A run looks stuck or unfinished when:**
  - the newest report has fewer lines than the input file (not counting the header) more than an hour after it started, or
  - no new report appears for an hour.
- The bot has no watchdog. Each invoice can take up to about **6 minutes** when billing is slow or down: 5 tries, each waiting up to 30 s to send and 30 s to check, plus 62 s of pauses. A run that is behind can still be going when the next hourly run starts. Both then write their own report.

## 5. Where to look when something goes wrong

Ops can open only the `data/output/` folder and its report files. The bot has no screens.

| Question | Where to look |
|---|---|
| Did the run start? | `data/output/` has a `sync_report_<date>_<time>.csv` for that hour |
| Did the run finish? | That report has one line per input line (not counting the header). Fewer lines means the run stopped part way (E10-E13). |
| Which invoices failed? | Report → **Status** column: anything other than `Synced` or `Duplicate`, plus `Synced` lines whose **Message** is "Created (some lines rejected)" (E07) |
| Why did one invoice fail? | Report → **Message** column on that invoice's line |
| Did the bot already retry? | Status `Failed` with Message "No success after 5 tries" means it retried 5 times. Message "Created (confirmed after retry)" means at least one try had no clear answer. Otherwise it did not retry. |
| Why did the whole run stop? | Not visible to ops. Raise a ticket (section 11). |
| Which version is running? | Not visible to ops (section 10). |

## 6. Statuses

The report is a plain CSV file, so nothing has a colour. These are every Status and Message the bot writes.

| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | The invoice was created in billing | nothing |
| Synced | Created (some lines rejected) | **Needs follow-up.** Billing created the invoice but rejected some of its lines. The invoice in billing is incomplete. | Open the invoice in billing and fix the missing lines (E07) |
| Synced | Created (confirmed after retry) | Billing did not answer clearly, but the invoice exists in billing now. It may have existed before this run (E08). | nothing, unless the invoice in billing looks wrong |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number, usually from an earlier hourly run. This is normal for every invoice that stays in the input file. | nothing. Look into it only if the invoice is new and was never sent before (E06). |
| Invalid | missing `<field names>` | A required field is empty on that line | Fix the input line (E01). If every line says "missing invoice_no", see E04. |
| Invalid | amount must be positive | The amount is 0 or negative | Fix the amount (E02) |
| Invalid | amount '`<value>`' is not a number | The amount has a currency sign, comma or text | Write it as a plain number (E03) |
| Invalid | (billing's own explanation), or "Rejected by billing" | Billing refused the invoice's content | Read the message, fix the input (E05) |
| Failed | No success after 5 tries | Billing failed 5 times and the invoice is not in billing | See E09 |

## 7. Exception catalogue

| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing customer_id" (or other field names) | Before sending, it checks that invoice number, customer ID, amount and currency are filled in | Report, Status and Message | Business | Ops | Fill in the field in the finance export. The next hourly run picks it up. | Never |
| E02 | Invalid - "amount must be positive" | The amount is 0 or below | Report, Status and Message | Business | Ops | Correct the amount, or remove credit notes from the export | Never |
| E03 | Invalid - "amount '1,200.50' is not a number" | The amount cannot be read as a number | Report, Status and Message | Business | Ops | Write amounts as plain numbers: `1200.50` | Never |
| E04 | **Every** line Invalid - "missing invoice_no" (and maybe other fields) | It cannot find the invoice-number column because the file starts with an invisible marker, or the header was renamed | Report, Status and Message | Business | Ops | Re-save the input as plain CSV/UTF-8 without the marker. Check the header line spelling. | It keeps happening with a clean file |
| E05 | Invalid - billing's own explanation, or "Rejected by billing" | Billing answered "content not acceptable" | Report, Status and Message | Business | Ops | Read the message, fix the data in the finance export | The message makes no sense, or billing rejects data that is correct |
| E06 | Duplicate - "Invoice already exists in billing" | Billing answered "this invoice number already exists" | Report, Status and Message | Business | Ops (usually nothing) | Normal for invoices left in the input from earlier runs. For a **new** invoice, check in billing whether its number is already used by a different invoice. | A new invoice is blocked by someone else's invoice with the same number |
| E07 | **Synced** - "Created (some lines rejected)" | Billing answered "partly accepted" | Report, Status and Message. **Status still says Synced.** | Business | Ops | Open the invoice in billing and complete or correct the rejected lines. Filter the report on Message, not only Status. | The cause of the rejection is not clear |
| E08 | Synced - "Created (confirmed after retry)" | Sending got no clear answer or an error, so the bot asked billing whether the invoice exists, and it does | Report, Status and Message | System | Bot | Nothing. If the invoice already existed before this run (for example, someone typed it in by hand), the bot still writes this message. | The invoice in billing is not the one in the input |
| E09 | Failed - "No success after 5 tries" | 5 tries without success. Each time, billing was unreachable, timed out (30 s), was busy, or refused the key, and the invoice still was not in billing. | Report, Status and Message. The real reason (busy, down, wrong key) is **not** shown. | System | Bot, then Ops, then Dev | 1. If only a few invoices failed, wait. The next hourly run tries them again. 2. Before re-entering anything by hand, **search billing for the invoice number**: a timed-out send may have created it anyway. | **Every** invoice in a run is Failed (likely an expired key or billing down), or the same invoice fails 2 runs in a row |
| E10 | No new report for an hour | The bot stops before it opens the report: the `data/output/` or `logs/` folder is missing, the input file is missing, or the computer did not run the task | Nothing ops can see | System | Dev | Raise a ticket with the hour that has no report | Always |
| E11 | The report stops part way and later lines never appear. The **same line** is the last one hour after hour. | While checking "does this invoice exist?" billing was unreachable, and the bot stopped the whole run | Report is cut short. The cause is visible to dev only. | System | Bot, then Dev | Wait one run. If billing was only briefly down, the next run gets past it. | The report stops at the same place 2 runs in a row |
| E12 | The report stops part way, at an invoice billing rejected | Billing rejected the invoice with an answer the bot could not read, and the bot stopped the whole run | Report is cut short. The cause is visible to dev only. | System | Dev | Note the invoice number on the line **after** the last line in the report. That is the one that stopped the run. Every invoice after it is blocked every hour until it is fixed or removed from the input. | Always |
| E13 | Report cut short for any other reason | Any unexpected error stops the run | Report is cut short. Details are visible to dev only. | System | Dev | Raise a ticket | Always |
| E14 | The same invoice appears **twice** in billing | A send timed out after billing had created it, and the check-then-retry did not catch it | Not recorded by the bot | System | Ops, then Dev | Void the extra copy in billing following the finance process | Always. It means the duplicate protection did not work (incident 2). |

## 8. What the bot recovers by itself

| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing slow, busy, down, unreachable, or refusing the key | Checks whether the invoice was created anyway, then waits and tries again. The waits are 2, 4, 8, 16 and 32 s. | 5 tries per invoice, up to about 6 minutes per invoice | Invoice marked **Failed**, the bot moves to the next invoice (E09) |
| Send timed out after billing actually created the invoice | Before each retry, it asks billing whether the invoice exists | Same 5 tries | Marked **Synced - Created (confirmed after retry)** (E08) |
| Invoice not processed in this run (Failed, Invalid, or the run stopped) | Nothing within the run. The next hourly run reads the whole input file again. | Every hour, without end, while the line stays in the input | Fixed invoices go through. Invoices already created show **Duplicate**. |
| The whole run stops | Nothing. No restart, no resume. | - | The next hourly run starts again from the first line |

## 9. Known incidents

Newest first. The history has no dates. Both were fixed before or at 0.9.1.

| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | The same invoice appeared twice in billing | Billing timed out after it had already created the invoice. The bot thought it failed and sent it again. | Fixed: the bot now checks whether the invoice exists before each retry. That check can still stop the run if billing is down (E11). If doubles appear again: void the extra copy and raise a ticket (E14). | Partly. The clean-up, yes. Prevention needed a code change. |
| Before 0.9.1 | The hourly run stopped part way | Billing said "too many requests" and the bot had no retry | Fixed: 5 tries with waits. Re-running is safe because invoices already created come back as Duplicate. | Partly |

## 10. Releases and version check

- **How an update arrives:** a developer replaces the bot's files on the bot machine. There is no auto-update.
- **How to confirm it applied:** ops cannot see the version. The bot does not show it anywhere ops can reach. Ask dev to confirm the version in the ticket that announced the update.
- **If the version did not change:** reply on the same ticket in the RPA Support queue.
- Only one machine runs this bot. It writes nothing shared at start-up, so an old copy cannot overwrite a new one. But **two copies scheduled on two machines would both send the same invoices**. Keep it to one.

## 11. Escalating to dev

Raise a ticket in the **RPA Support** queue with:
- the exception ID from section 7 (for example, E11)
- the report file name, for example `sync_report_20261023_1400.csv`. It gives the run date and time.
- the invoice number(s) affected, and the last invoice number in a cut-short report
- whether the previous hour's run showed the same problem

Never send: the settings file (`.env`), the billing key, or the report file itself in chat. The report contains customer names, so reference it by name.

## 12. Failure-class check

- F1 (missing prerequisite) covered in section 3 (output folder, logs folder, settings file, input file), E09, E10, and Fix before handover #1.
- F2 not applicable: the bot drives no windows. It only calls a web API in the background.
- F3 (target slower than expected) covered in E09, E11, section 8 (30 s timeout, 5 tries) and section 4 (6 minutes per invoice).
- F4 (input read differently) covered in section 3 input rules, E01-E04, and Fix before handover #7.
- F5 (marked done when nothing was done) covered in E08 and Fix before handover #5.
- F6 (action happened, not confirmed) covered in E08, E09 (search billing before re-entering), E14, and Fix before handover #4.
- F7 (business exception shown as success) covered in E07, section 6, and Fix before handover #3.
- F8 (endless re-runs / watchdog) covered in section 4 (no watchdog, overlapping runs), section 8 (whole file every hour), and Fix before handover #6 and #8.
- F9 (stopping the wrong way) covered in section 4 (end the task in Task Scheduler, via ticket).
- F10 (one stuck item blocks the queue) covered in E11 and E12 (one invoice can stop every later invoice, every hour) and Fix before handover #4.
- F11 (cause hidden from ops) covered in section 5 and Fix before handover #2.
- F12 (update did not arrive) covered in section 10 and Fix before handover #9.

## 13. Fix before handover (dev team)

| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | `data/output/` is not in the repo and the code never creates it. `logs/` must also exist before `logging.basicConfig` opens the file. | A fresh install fails every run with no report and nothing ops can see | `sync.py:24`, `sync.py:68-69` |
| 2 | Run crashes and network errors go only to `print()` (console), which Task Scheduler throws away. `sync.log` holds the status but never the Message. Ops cannot open `logs/`. | Ops cannot tell why a run stopped (E10-E13). Even dev has no traceback. | `sync.py:52`, `sync.py:75`, `sync.py:84` |
| 3 | HTTP 207 (partial) is stored as Status `Synced` | Incomplete invoices look done when filtering by Status (E07) | `sync.py:54-55` |
| 4 | `exists()` has no try/except, and the 422 `r.json()` is not guarded. Neither is caught per row. | One bad invoice or a short outage stops the whole run, and every later invoice is blocked every hour (E11, E12) | `sync.py:42`, `sync.py:59`, `sync.py:72-76` |
| 5 | `exists()` returning 200 is reported as "Created (confirmed after retry)", even on the first attempt and even if the invoice existed before this run | Synced does not prove this run created it (E08) | `sync.py:60-61` |
| 6 | Every status other than 200/201/207/409/422 is retried, including 400/401/403/404. The docstring says only 429/5xx. `Retry-After` is ignored, and the bot sleeps after the last try too. The Failed message has no HTTP code. | An expired key costs about a minute per invoice and shows only "No success after 5 tries" (E09) | `sync.py:47`, `sync.py:60-63` |
| 7 | Input read as `utf-8`, not `utf-8-sig`. The amount check accepts `nan`/`inf`. | An Excel export with a BOM turns every line Invalid (E04) | `sync.py:34`, `sync.py:69` |
| 8 | Input is never archived and there is no run lock | The whole file is re-sent every hour (noisy Duplicates), and slow runs can overlap the next hourly run | `sync.py:66-77` |
| 9 | Version exists only in README text, with a stray `chore: bump to 0.9.1` line. Nothing ops can see shows the version. | Ops cannot confirm an update applied (F12) | `README.md:5-6` |
| 10 | The report includes `customer_name` | Personal data in a folder ops can open. Confirm this is allowed. | `sync.py:76` |
| 11 | `.env` with the billing key is committed to git | Credential exposure. Rotate the key and untrack the file. | `.env` (commit `88b2057`) |
| 12 | Duplicate protection relies on billing answering 409, yet incident 2 produced doubles | Confirm billing enforces unique invoice numbers. Otherwise E14 can recur. | `sync.py:56-57` |
| 13 | `BOT_WAS_RUN` marker file is written into the project root every run | Leftover file with no purpose for ops | `sync.py:67` |