Rpa_Skills/tests/evals/results/handoff/h-api-sync-screens-r1.md

# Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: raise a ticket in the **RPA Support** queue

## 1. What the bot does

Every hour it takes the approved invoices from the finance export file (`data/input/invoices.csv`) and creates each one in the billing system through the billing system's web service. For every invoice in the file it writes one line to a report file in `data/output/`, saying whether the invoice was created, was already there, was rejected, or failed. It has **no window and no screens**: it runs in the background from Windows Task Scheduler, and it drives no desktop apps or websites. Ops works from the report files in the output folder.

## 2. How it works

```
Windows Task Scheduler (every hour)
   └─ starts → Invoice sync (background script, no window)
                  │
                  ├─ reads  ← data/input/invoices.csv        (finance export)
                  ├─ sends ──> Billing system web service    (creates invoices)
                  ├─ writes → data/output/sync_report_<date>_<time>.csv   (ops can open)
                  └─ writes → logs/sync.log                                (ops cannot open)
```

- **Task Scheduler** starts the bot once an hour. Ops has no access to it.
- **Invoice sync** reads the whole export from top to bottom, one invoice at a time.
- **The finance export** is the input. Every invoice still in it is sent again every hour.
- **The billing web service** creates the invoice, or answers "already exists", "rejected" or with an error.
- **The report file** has one line per invoice: invoice number, customer, Status, Message. A new file is made every run, named by date and time (for example `sync_report_20261023_1400.csv` = 23 Oct 2026, 14:00).
- **The log file** holds only the invoice number and status, plus a "run done" line. Ops cannot open it, and it doesn't hold the reasons anyway.

**One run, step by step:**
1. Task Scheduler starts the bot on the hour.
2. The bot creates a new, empty report file in `data/output/`.
3. For each invoice in the export, it checks that the invoice number, customer ID, amount and currency are filled in and that the amount is a positive number. If not, the line is **Invalid** and it moves on.
4. It sends the invoice to billing. If billing doesn't answer clearly, it checks whether billing has the invoice anyway, waits and tries again, up to 5 tries (section 8).
5. It writes the result line to the report and goes to the next invoice.
6. After the last invoice the run ends. Nothing in the report marks the end (see E11).

## 3. Before a run

Ops cannot change the bot machine. These items are for whoever sets it up; ops only needs to know them so the symptom makes sense.

- [ ] The bot folder has a `.env` settings file with the billing address and access key. If it is missing or the key is wrong or expired, every invoice ends **Failed** (E10).
- [ ] The folders `data/output/` and `logs/` exist in the bot folder. The bot does not create them. If one is missing, every run crashes and no report appears (E11).
- [ ] Python is installed on the bot machine with the two packages listed in `requirements.txt`.
- [ ] The finance export follows these rules (if it doesn't, ask whoever produces the export to fix it):

| Rule | Bad | Good |
|---|---|---|
| Column headers exactly, lower case: `invoice_no, customer_id, customer_name, amount, currency` | `Invoice No` | `invoice_no` |
| Invoice number, customer ID, amount and currency are always filled in (customer name may be empty) | amount empty | `1200.50` |
| Amount is a plain number: dot for decimals, no thousands separator, no currency sign | `1,200.50` or `$1200.50` | `1200.50` |
| Amount is above zero. Zero and negative amounts (credit notes) are never synced | `-50.00` | `50.00` |
| The file comes straight from the finance system. **Don't re-save it in Excel**: Excel's "CSV UTF-8" option makes every line Invalid "missing invoice_no", and other Excel CSV options can crash the run when names have accents | re-saved in Excel | original export |
| No extra columns. The bot sends every column to billing, so an extra column can be rejected or leak data into billing | an added "notes" column | the five columns only |

Sample of the layout: `data/input/invoices.csv` (headers only - don't copy its rows).

## 4. During a run

- Don't move, rename or delete files in `data/output/` while a run is going. The newest report is still being written.
- Don't open the newest report until its "modified" time has stopped changing for a few minutes.
- **To stop the bot:** there is no Stop button. It can only be stopped in Task Scheduler, which ops cannot reach. Raise a ticket. Stopping it mid-invoice can leave that one invoice created in billing but missing from the report.
- **A run looks stuck** when the newest report's modified time hasn't changed for more than 6 minutes. One invoice can legitimately take up to about 6 minutes while billing is not answering (section 8). The bot has no watchdog: it never gives up on the run as a whole, it only gives up on one invoice at a time. A run with many failing invoices can last longer than an hour (E12).

## 5. Where to look when something goes wrong

Ops can open only the output folder `data/output/`. Everything below uses it.

| Question | Where to look |
|---|---|
| Did the run start? | A report file exists for that hour: `sync_report_<date>_<hour><minute>.csv`. No file for the hour = the run did not start or crashed at once (E11). |
| Did the run finish? | **Ops cannot tell for sure.** The report has no end line. If it has fewer invoices than the export, or stops at the same invoice every hour, treat it as a crash (E11). |
| Which invoices failed? | Open the report, filter the **Status** column for `Failed`, `Invalid`, and `Synced` lines whose Message says "some lines rejected" or "confirmed after retry". |
| Why did one invoice fail? | The **Message** column on that invoice's line. That is the only reason the bot gives ops. For `Failed` there is no further detail anywhere ops can see. |
| Did the bot already retry? | Yes, always, for every `Failed` line (5 tries). A `Synced` line with "confirmed after retry" also went through a retry. The next hourly run tries again any invoice still in the export. |
| Which version is running? | Ops cannot see it. Ask in the ticket (section 10). |

## 6. Statuses

The report is a plain CSV file: it has **no colours**. The Status is the third column, the Message the fourth.

| Status | Message | Meaning | What ops does |
|---|---|---|---|
| Synced | Created | Invoice created in billing | nothing |
| Synced | Created (confirmed after retry) | The first send had no clear answer; the bot then found the invoice in billing | **Check billing for a second copy** (E07) |
| Synced | Created (some lines rejected) | **Needs follow-up.** Invoice created, but billing refused some of its lines | Fix the missing lines in billing (E08) |
| Duplicate | Invoice already exists in billing | Billing already has this invoice number. Normal for every invoice synced in an earlier hour | nothing, unless the invoice is new (E06) |
| Invalid | missing <field names> | A required field is empty in the export | Fix the export (E01, E02) |
| Invalid | amount must be positive | Amount is zero or negative | Business decision (E03) |
| Invalid | amount '<value>' is not a number | Amount is written in a way the bot can't read | Fix the export (E04) |
| Invalid | billing's own reason text, or "Rejected by billing" | Billing checked the invoice and refused it | Read the reason, fix the data (E05) |
| Failed | No success after 5 tries | Billing never confirmed it after 5 tries. **It may still have been created** | Check billing first (E09, E10) |

## 7. Exception catalogue

| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Invalid - "missing customer_id" (or another field name) on some lines | Before sending, it checks that invoice number, customer ID, amount and currency are not empty | Report, Status and Message | Business | Ops | Ask the export owner to fill the field. The next hourly run picks the invoice up. | The field is filled in the export but still reported missing |
| E02 | Invalid - "missing invoice_no" (often with other fields) on **every** line | Same check, but the bot can't find the invoice number column at all: the file was re-saved in Excel as "CSV UTF-8", or the headers were renamed | Report, Status and Message | Business | Ops, then Dev | Ask for the file straight from the finance system with the exact headers (section 3). | The file is an untouched export and every line is still Invalid. Send: report file name, E02. |
| E03 | Invalid - "amount must be positive" | Amount is zero or below | Report, Status and Message | Business | Ops | Zero and negative invoices are never synced by this bot. Handle them in billing by hand if the business needs them. | The amount in the export is above zero |
| E04 | Invalid - "amount '1,200.50' is not a number" (shows the value) | The amount can't be read as a number: comma separator, currency sign, text | Report, Status and Message | Business | Ops | Ask for the amount as a plain number, e.g. `1200.50`. | The value shown looks like a plain number |
| E05 | Invalid - billing's own reason (for example about the customer), or "Rejected by billing" | Billing answered that the invoice data is not acceptable. The bot does not retry this | Report, Status and Message | Business | Ops | Read the reason. Fix the data in the export, or in billing (for example a customer ID billing doesn't know). | The reason is unclear, or every line has it. Send: report file name, invoice numbers, E05. |
| E06 | Duplicate - "Invoice already exists in billing" | Billing answered that the invoice number already exists. No retry | Report, Status and Message | Business | Ops | Normally nothing: every invoice synced in an earlier hour shows this every hour after. If the invoice is **new**, look it up in billing: another invoice may already use that number. | Billing shows no invoice with that number. Send: invoice number, report file name, E06. |
| E07 | Synced - "Created (confirmed after retry)" | The first send timed out or billing gave an error. Before trying again, the bot looked the invoice number up in billing and found it | Report, Status and Message | System | Bot, then Ops | Look the invoice up in billing and make sure there is **only one** copy (see the 504 incident, section 9). Remove an extra copy using the billing team's process. | Two copies found. Send: invoice number, report file name, E07. |
| E08 | Synced - "Created (some lines rejected)" | Billing created the invoice but said part of it was refused. The bot records this as Synced anyway | Report, Status and Message | Business | Ops | **Not finished despite "Synced".** Open the invoice in billing, compare its lines with finance, add the missing lines with the finance team. The report doesn't say which lines were refused. | You can't work out which lines are missing. Send: invoice number, report file name, E08. |
| E09 | Failed - "No success after 5 tries" on a **few** lines | Billing didn't answer, timed out, or gave an error 5 times in a row (section 8), and the invoice number was never found in billing | Report, Status and Message | System | Bot, then Ops | **Check billing first.** The invoice may have been created even though the bot couldn't confirm it. Found: nothing to do, the next run will show Duplicate. Not found: wait for the next hourly run, which tries again if the invoice is still in the export. | The same invoice is Failed in 2 runs in a row. Send: invoice number, both report file names, E09. |
| E10 | Failed - "No success after 5 tries" on **all or most** lines | Same as E09, for every invoice. Usually billing is down, or the bot's billing address or access key is wrong or expired. The bot can't tell these apart and gives no reason | Report, Status and Message | System | Dev | Raise a ticket at once. Expect the run to be slow: each invoice takes over a minute (E12). | Always. Send: report file name, number of Failed lines, time noticed, E10. |
| E11 | No report for the hour, or a report that stops part-way (fewer invoices than the export, often stopping at the same invoice every hour) | The run crashed. Known causes: the billing look-up after a timeout failed too; billing refused an invoice with an answer the bot couldn't read; the `data/output` or `logs` folder is missing; the export file is missing or saved in an unexpected encoding. The cause is not recorded anywhere ops can see | Nothing ops can see. The report simply ends | System | Dev | Raise a ticket. Don't re-run anything: the next hourly run starts again from the top and will probably stop in the same place. Check billing for the invoice **after** the last line in the report: it may have been created. | Always. Send: report file name (or "no report for <hour>"), the last invoice number in the report, E11. |
| E12 | The newest report is still growing after the next hour has started, or an hour's report is missing while the previous one is still being written | The run is longer than an hour, usually because many invoices are failing slowly (E10). Nothing stops two runs from overlapping | Report file times in `data/output/` | System | Dev | Raise a ticket. Two runs at once can send the same invoice twice. | Always. Send: the report file names and their times, E12. |

## 8. What the bot recovers by itself

| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Billing doesn't answer, times out, is busy, or gives an error (this includes a wrong access key) | Waits up to 30 s for an answer. Then checks whether billing has the invoice anyway; if it does, records Synced "confirmed after retry". If not, waits and sends again | 5 tries. Waits of 2, 4, 8, 16 and 32 s between tries. Worst case about **6 minutes for one invoice** | Line = Failed "No success after 5 tries", next invoice |
| Billing says the invoice already exists | No retry | - | Line = Duplicate, next invoice |
| Billing refuses the invoice data | No retry | - | Line = Invalid with billing's reason, next invoice |
| Invoice not synced this hour (Failed, Invalid, or after a crash) | The next hourly run sends every invoice in the export again, from the top | Every hour, for as long as the invoice is in the export | Already-created invoices come back as Duplicate |
| Run crash | **No recovery inside the run.** The rest of the export is not processed until the next hour | - | Next hourly run starts from the top (E11) |

There is no watchdog and no automatic restart. A Failed line is final for that run; ops waits for the next hourly report.

## 9. Known incidents

| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| Before 0.9.1 | The same invoice appeared twice in billing | Billing timed out after it had already created the invoice; the bot sent it again and billing accepted the second copy | Fixed: the bot now looks the invoice up before every retry. Still check every "confirmed after retry" and every Failed line in billing (E07, E09). | Partly |
| Before 0.9.1 | The hourly run died part-way; invoices after that point got no report line | Billing limited how fast the bot could send (rate limit) and the bot didn't retry | Fixed for the send: it now retries with waits. The look-up after a timeout can still crash the run the same way (E11, section 13 item 1). | Partly |

## 10. Releases and version check

- How an update arrives: not handled by the bot. There is no launcher or auto-update; a developer replaces the bot's files on the bot machine.
- How to confirm it applied: ops can't. The version (0.9.1) is written only in the project's `README.md`, which ops can't open. Ask the developer to confirm the version in the ticket.
- If the version did not change: raise a ticket in RPA Support.

## 11. Escalating to dev

Raise a ticket in the **RPA Support** queue.

Send: the exception ID from section 7, the report file name (it carries the date and time of the run), the invoice numbers affected, and the time you noticed. For a crash (E11), the last invoice number in the report.

Never send: the `.env` settings file, the billing access key, or customer names and amounts pasted into chat. The report files contain customer names: give the file name and let the developer open it.

## 12. Failure-class check

- F1 covered in section 3 (settings file, folders, packages) and E10 / E11.
- F2 not applicable: the bot has no windows and drives no desktop app; it only talks to a web service.
- F3 covered in E07, E09, E10 and section 8 (30 s wait, 5 tries).
- F4 covered in section 3 input rules and E01-E04; also section 13 items 8 and 9.
- F5 covered in E07 (Synced only when billing confirms or the invoice is found by number). Remaining gap: section 13 item 16.
- F6 covered in E09 and E11 (check billing before assuming not created); section 13 items 1 and 16.
- F7 covered in section 6 and E08 ("some lines rejected" shown as Synced); section 13 item 5.
- F8 covered in section 8 and E12: no loop limit beyond 5 tries per invoice, no watchdog, runs can overlap; section 13 items 3 and 10.
- F9 covered in section 4: no Stop button, Task Scheduler only; section 13 item 17.
- F10 covered in E11: one invoice that crashes the run blocks every invoice after it, every hour; section 13 items 1 and 2.
- F11 covered in section 5 and section 13 items 4 and 6: reasons for Failed and for crashes are not shown anywhere ops can open.
- F12 covered in section 10: no auto-update and no version ops can see; section 13 item 12.

## 13. Fix before handover (dev team)

| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | The existence check has no error handling. A timeout or connection error there crashes the whole run: the same symptom as the 429 incident, on the second call | Run stops mid-file, every hour, at the same invoice (E11) | `sync.py:41-43`, called at `sync.py:60` |
| 2 | A 422 with a non-JSON body (e.g. HTML error page) raises in `r.json()` and crashes the run | Same as above (E11) | `sync.py:59` |
| 3 | Every non-success status is retried, including 400/401/403/404; the docstring says only 429/5xx. A bad key makes every invoice take ~62 s+ and end Failed; the run doesn't stop early | Slow runs that overlap the next hour, no clue it's the key (E10, E12) | `sync.py:47`, `sync.py:60-62` |
| 4 | Failed message has no HTTP status or reason; the log records only invoice + status; network errors and the CRASH line go to stdout, lost under Task Scheduler | Ops (and dev) can't tell a rate limit from an auth failure from an outage | `sync.py:52`, `sync.py:63`, `sync.py:75`, `sync.py:84` |
| 5 | HTTP 207 (partial) is stored as Status `Synced` | Ops filtering on Status misses invoices that need follow-up (E08) | `sync.py:54-55` |
| 6 | The "run done" line goes only to `logs/sync.log` (no ops access); the `BOT_WAS_RUN` marker is in the bot folder and holds only "1". Nothing in the output folder says a run finished or crashed | Ops can't tell a finished run from a crashed one. Suggest a final summary line or a `.done` / `.failed` file in `data/output/` | `sync.py:67`, `sync.py:77`, `sync.py:80-85` |
| 7 | `data/output/` and `logs/` are not created by the code and `data/output/` isn't in the repo. A fresh install crashes every run | No report at all (E11) | `sync.py:24`, `sync.py:68-69` |
| 8 | Input opened as `utf-8`, not `utf-8-sig`. A BOM (Excel "CSV UTF-8") makes every line Invalid "missing invoice_no"; a non-UTF-8 file crashes the run | E02, E11 | `sync.py:69` |
| 9 | The whole CSV row, including any extra columns and `customer_name`, is posted as JSON | Extra columns can be rejected or leak into billing | `sync.py:50` |
| 10 | No lock against overlapping runs; report name is to the minute, so two runs in one minute overwrite one file. Task Scheduler's "if already running" setting isn't documented | Same invoice sent twice at once; lost report (E12) | `sync.py:68`; Task Scheduler task |
| 11 | `Retry-After` is ignored on 429; the 32 s sleep after the 5th try is wasted | Longer runs | `sync.py:62` |
| 12 | Version shown only in README; nothing ops can see shows it. README also has a stray commit-message line | Can't confirm an update applied | `README.md:5-6` |
| 13 | `.env` with the live billing key is committed (initial import) | Key exposed in git history: rotate it, remove `.env` from the repo, add to `.gitignore` | `.env`, commit 88b2057 |
| 14 | Customer names are copied into every report; no retention or access rule for `data/output/` | Customer data piles up in a folder ops opens daily | `sync.py:76` |
| 15 | Both fix commits only add comments; there are no tests | Nothing proves the 429 / 504 fixes work or keeps them from regressing | `sync.py:86-87` |
| 16 | Every hourly run re-posts every invoice in the export and relies on billing answering 409. The 504 incident shows billing accepted a second copy at least once; the existence check only matches by invoice number | Duplicate invoices remain possible (E07, E09) | `sync.py:72-74`, `sync.py:41-43` |
| 17 | No stop mechanism ops can use; killing mid-POST leaves the invoice unrecorded | Ops must open a ticket to stop a runaway run | `sync.py:80-85`; Task Scheduler task |