Rpa_Skills/tests/evals/results/handoff/h-api-sync-screens-r4.md

Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: raise a ticket in the RPA Support queue

1. What the bot does

Every hour, the bot takes the approved invoices from the finance export file (data/input/invoices.csv) and creates each one in the billing system through the billing system's web API. It reads the whole file on every run. For each invoice it writes one line to a report file in data/output/, named sync_report_<date>_<time>.csv, with the result (Status) and the reason (Message). It has no screen and drives no desktop app or website. Windows Task Scheduler starts it.

2. How it works

Windows Task Scheduler (every hour)
   └─ starts → Invoice sync (no window)
                 ├─ reads  → data/input/invoices.csv   (finance export)
                 ├─ sends  → Billing system API         (over the network)
                 ├─ writes → data/output/sync_report_<date>_<time>.csv   ← ops reads this
                 └─ writes → logs/sync.log              (dev only)

• Task Scheduler starts the bot once an hour. The task's settings are not stored with the bot.
• Invoice sync runs in the background. There is nothing on screen to watch or click.
• The finance export is the input. Every row is one invoice.
• The billing system is where invoices are created. The bot only talks to it over the network.
• The report file is the only place ops can see results. A new one is written every run.
• The log file holds only the invoice number and status. Ops cannot open it.

One run, step by step:

1. Open the finance export and start a new report file.
2. For each invoice, check the row: invoice number, customer ID, amount and currency must be filled in, and the amount must be a number above zero. If the check fails, the row is marked Invalid and the bot does not send it.
3. Send the invoice to billing.
4. If billing says "created", the row is Synced. If billing says it already has the invoice, the row is Duplicate. If billing rejects the data, the row is Invalid.
5. If billing gives any other answer, or does not answer within 30 seconds, the bot asks billing whether the invoice exists now. If it does, the row is Synced. If not, it waits and tries again, up to 5 times. After 5 tries the row is Failed.
6. Write the row to the report and go to the next invoice.
7. After the last invoice, the bot writes "run done" to its log. Ops cannot see this line.

3. Before a run

• The folder data/output/ exists next to the bot. The bot does not create it. Without it, the run stops before the first invoice and no report appears.
• The settings file (.env) is next to the bot and holds the billing address and key. Without the key, every invoice ends as Failed after about a minute each (see E08).
• The finance export is at data/input/invoices.csv, and its first line has the column names invoice_no, customer_id, customer_name, amount, currency, spelled exactly like that.
• Every row has an invoice number, customer ID, amount and currency.
• The amount is a plain number with a dot for decimals:
  • Bad: 1,250.00, $1250, 1250,00
  • Good: 1250.00
• Do not open the export in Excel and save it again. Excel can change the file so that every row fails as "missing invoice_no", or so that the run stops.
• Only one machine runs the hourly task. Two machines on the same file can create the same invoice twice.
• The bot is version 0.9.1 (see section 10).

4. During a run

• There is nothing on screen. Do not start the bot by hand while the hourly task may be running. The bot has no protection against two runs at once.
• To stop the bot: in Task Scheduler, right-click the task → End. To stop the hourly runs, Disable the task. If ops has no access to Task Scheduler, raise a ticket.
• A run can be slow. Each invoice that billing does not accept can take up to about 6 minutes (5 tries, 30 s wait for each answer, pauses of 2, 4, 8, 16 and 32 s). Many such invoices can make one run last longer than an hour, so the next hourly run may start on top of it.
• The bot has no stuck detection and no automatic restart. If no new report has appeared for more than 2 hours, raise a ticket.
• Expect every run to list every invoice in the export again. An invoice synced in an earlier hour shows as Duplicate in every later report. That is normal.

5. Where to look when something goes wrong

Ops can open only the output folder data/output/ and its report files.

Question	Where to look
Did the run happen?	data/output/: a new sync_report_<date>_<time>.csv for that hour. No new file means the run did not start or stopped at once (E09).
Did the run finish?	Ops cannot tell for certain. A report that has fewer rows than the previous hour's report, or that ends earlier than usual, means the run stopped partway (E09). The "run done" line is in the log only (Fix item 9).
Which invoices failed?	The report file, Status column: Failed, Invalid, and Synced with Message "Created (some lines rejected)"
Why did one invoice fail?	The report file, Message column, on that invoice's row
Did the bot already retry?	Yes, always: every Failed row was tried 5 times. Every invoice is also tried again in the next hourly run.
Did an earlier run already sync it?	Open the earlier report files in data/output/ and look for the invoice number with Status Synced
Which version is running?	Ops cannot see it (Fix item 15). Ask in a ticket.

6. Statuses

The report is a plain CSV file, so it has no colours. The Status column holds one of these values.

Status	Message	Meaning	What ops does
Synced	Created	Billing created the invoice	Nothing
Synced	Created (confirmed after retry)	Billing did not confirm the first send, but the bot then found the invoice in billing	Nothing
Synced	Created (some lines rejected)	Needs follow-up. Billing created the invoice but dropped some of its lines. It is shown as a success, but it is not complete.	Raise a ticket (E06)
Duplicate	Invoice already exists in billing	Billing already has this invoice number. This is usually the bot's own earlier run.	Nothing, if an earlier report shows it as Synced. Otherwise see E05.
Invalid	missing <column names>	A required column is empty in the export	Fix the export (E01)
Invalid	amount must be positive	Amount is zero or negative	Fix the export (E02)
Invalid	amount '<value>' is not a number	The amount could not be read as a number	Fix the export (E03)
Invalid	billing's own reason, or "Rejected by billing"	Billing refused the invoice data	Read the reason, then fix the export (E04)
Failed	No success after 5 tries	Billing did not accept it after 5 tries	Wait for the next hour (E08)

7. Exception catalogue

ID	What you see	How the bot identifies it	Where it is recorded	Type	Who acts	What to do	Escalate when
E01	Invalid - "missing invoice_no" (or customer_id, amount, currency)	Before sending, it checks that these four columns are filled in	Report, Status + Message	Business	Ops	Ask finance to fill in the column in the export. The next hourly run picks it up.	Every row says "missing invoice_no": the export's format changed (often after it was saved in Excel). Send the report file name.
E02	Invalid - "amount must be positive"	Before sending, it checks that the amount is above zero	Report, Status + Message	Business	Ops	Ask finance to correct the amount, or remove credit notes from the export	Never; this is a data issue
E03	Invalid - "amount '…' is not a number"	Before sending, it tries to read the amount as a number	Report, Status + Message	Business	Ops	Ask finance to write the amount as a plain number, e.g. 1250.00	Many rows at once after a change to the export
E04	Invalid - billing's reason, or "Rejected by billing"	Billing answers that the invoice data is wrong	Report, Status + Message	Business	Ops	Read the Message, fix the row with finance, and wait for the next hour	The Message is unclear, or the same row is rejected after the fix. Send the invoice number, report file name and Message.
E05	Duplicate - "Invoice already exists in billing"	Billing answers that the invoice number already exists	Report, Status + Message	Business	Ops	Search earlier reports for the invoice. If one shows it as Synced, do nothing.	No earlier report shows it as Synced. Someone may have created it in billing by hand, or the number is reused. Send the invoice number and report file names.
E06	Synced - "Created (some lines rejected)"	Billing answers "partly created"	Report, Status + Message only	Business	Dev	Raise a ticket. Ops cannot see which lines were dropped.	Always. Send the invoice number and report file name.
E07	Synced - "Created (confirmed after retry)"	The send failed or timed out, then the bot asked billing and found the invoice	Report, Status + Message	System	Bot	Nothing. The bot did not send it again.	Finance reports the invoice twice in billing (E11)
E08	Failed - "No success after 5 tries"	Billing gave no usable answer (busy, down, too many requests, wrong key) or no answer in 30 s, 5 times in a row	Report, Status + Message. The real reason is not recorded.	System	Bot, then Ops	Wait for the next hourly run; it tries again. Check the next report for this invoice.	The same invoice is Failed in 2 reports in a row, or most rows in one report are Failed (billing is down or the key is wrong). Send the report file names and the times.
E09	No new report for an hour, or a report that stops partway	The run stopped. Known causes: billing did not answer while the bot was checking for a duplicate; billing sent an unreadable rejection; the output folder or export file is missing.	Nowhere ops can see. The reason is lost (Fix items 2-4).	System	Dev	Raise a ticket. Invoices after the stop point were not handled in that run.	Always. Send the hour, the report file name (if any) and the last invoice number in it. If the same report stops at the same invoice every hour, say so: that invoice is blocking every run.
E10	Report has rows for the same hour twice, or a report is overwritten	Two runs ran at the same time (a slow run overlapped the next one, or the bot was started by hand)	Report files in data/output/	System	Dev	Raise a ticket. Two runs at once can create duplicates in billing.	Always. Send the report file names and times.
E11	Finance or billing reports the same invoice created twice	The bot sent it again because billing timed out after creating it, and the check before the retry did not see it	Nowhere in the bot. The report shows Synced or Failed.	System	Dev	Raise a ticket. Billing must void the extra copy.	Always. Send the invoice number and the report file names that list it.

8. What the bot recovers by itself

Problem	What the bot does	Limit	After the limit
Billing does not answer	Waits 30 s for each answer	30 s per request	Counts as one failed try
Billing busy, down, "too many requests", or any unexpected answer	Asks billing whether the invoice already exists. If not, pauses and sends again.	5 tries; pauses of 2, 4, 8, 16, 32 s; up to about 6 min per invoice	Row = Failed, then the next invoice
Invoice created but the bot did not get the confirmation	The check before each retry finds it, and the bot does not send it again	Once per failed try	Row = Synced "confirmed after retry"
Failed or Invalid invoice	The next hourly run sends every invoice in the export again	Every hour, for as long as the invoice is in the export	Never gives up; it shows in every report
Run stopped (E09)	Nothing. There is no restart within the hour.	-	The next hourly run starts again from the top

9. Known incidents

Both were fixed before version 0.9.1. Their dates are not recorded.

When / version	What ops saw	Cause	What to do now	Ops could handle?
Before 0.9.1	Invoices created twice in billing	Billing timed out after it had already created the invoice, so the bot sent it again	Partly fixed: the bot now checks billing before each retry. If that check itself fails to get an answer, the run stops (E09), and a duplicate is still possible (E11). Raise a ticket so billing can void the extra copy.	Partly (clean-up only)
Before 0.9.1	Hourly run stopped when billing answered "too many requests"	Billing's rate limit was not retried	Fixed: up to 5 tries with pauses. If many rows are Failed, see E08.	Partly

10. Releases and version check

• How an update arrives: there is no automatic update. A developer copies the new files to the bot machine.
• How to confirm it applied: the version is written only in the bot's README file, which ops cannot open. Ask the developer to state the version in the ticket when they finish the update (Fix item 15).
• If the version did not change: raise a ticket. Only one machine runs the hourly task, so there is no risk of an old machine undoing a new one, unless a second machine is set up (section 3).

11. Escalating to dev

Raise a ticket in the RPA Support queue. Send:

• the exception ID from section 7 (e.g. E09)
• the hour of the run
• the report file name from data/output/
• the invoice numbers involved

Never send: the API key, the .env settings file, customer names, or whole report files in chat. The report contains customer names; attach it to the ticket only.

12. Failure-class check

• F1 covered in section 3 (output folder, settings file, export location), E08 and E09.
• F2 not applicable: the bot drives no windows. It only talks to billing over the network.
• F3 covered in E08, E09 and section 8: the 30 s answer limit and 5 tries.
• F4 covered in section 3 input rules, E01-E03, and Fix item 12 (Excel-saved file).
• F5 covered: the bot writes Synced only when billing confirms the invoice. A stopped run writes no success for the rows it did not reach (E09). The partial-success case is under F7.
• F6 covered in E07, E09 and E11, and Fix items 2 and 14.
• F7 covered in section 6 (Synced "some lines rejected"), E06 and Fix item 6.
• F8 covered in section 4 (the whole file is re-sent every hour, no stuck detection), E10, and Fix items 10 and 11.
• F9 covered in section 4: stop it in Task Scheduler.
• F10 covered in E09: an invoice that makes the run stop blocks every invoice after it, every hour. Fix items 2 and 3.
• F11 covered in section 5 and Fix items 4, 5 and 9: the reasons are in the log or lost, not in the report.
• F12 covered in section 10 and Fix item 15.

13. Fix before handover (dev team)

#	Gap	Why it hurts ops	Where (file:line)
1	The billing API key is committed to git in .env	The key is exposed to anyone with the repo. Rotate the key and remove the file from history.	.env (commit 76b9600)
2	The duplicate check has no error handling. A timeout or network error on it stops the whole run.	Run stops (E09) exactly when billing is slow, which is the case the 504 fix was meant for	sync.py:41-43, sync.py:60
3	A 422 reply without JSON stops the whole run, and the same row then stops every hourly run	One invoice blocks all later invoices, every hour (E09)	sync.py:59
4	The crash reason and network errors go to stdout only, which Task Scheduler discards. Nothing reaches the log or the report.	The cause of E09 is lost	sync.py:52, sync.py:81-85
5	The log holds only invoice number and status: no HTTP code, attempt count or error. "Failed" never says why. Ops cannot open the log anyway.	Ops cannot tell a wrong key from an outage from a rate limit (E08)	sync.py:63, sync.py:75
6	HTTP 207 (partial) is written as Status "Synced"	A partly created invoice looks like a success (F7)	sync.py:54-55
7	400, 401, 403 and 404 are retried 5 times like temporary errors, with no abort for the whole run	A wrong or expired key makes every invoice take about 1-6 min and end as Failed; the run can overlap the next	sync.py:60-62
8	data/output/ is not created by the code	The first run on a new machine stops with no report	sync.py:68-69
9	No end-of-run marker that ops can see ("run done" is in the log only)	Ops cannot tell a finished run from a stopped one	sync.py:77
10	No lock against overlapping runs. The report name has minute precision and is opened for overwrite. The Task Scheduler task settings are not in the repo.	Overlapping runs can create duplicates and overwrite reports (E10)	sync.py:68-69
11	No resume and no memory of synced invoices: every row is posted again every hour	Report fills with "Duplicate" rows, it uses up the rate limit, and the risk of duplicates grows	sync.py:72-74
12	The export is read as utf-8, not utf-8-sig. A file saved by Excel with a BOM would make every row "missing invoice_no" (possible, not seen yet).	Whole run Invalid after one re-save	sync.py:69
13	Retry-After is ignored, a duplicate-check GET is added to every failed POST, and the bot sleeps 32 s after the final attempt	Slower runs and more rate-limit hits	sync.py:60-62
14	The duplicate check treats any non-200 (429, 5xx) as "not there" and posts again	Duplicates are still possible (E11). Needs an idempotency key or a billing-side unique constraint.	sync.py:43
15	The version exists only in the README (with a stray "chore: bump to 0.9.1" line), not in the report or log	Ops cannot confirm which version is running	README.md:5-6
16	The report copies customer names	The output folder holds customer data	sync.py:76
17	nan and inf pass the amount check, and currency is not checked against a list	Bad rows reach billing and come back as Invalid or Failed	sync.py:33-37
18	Both "fix" commits contain only a comment line, and there are no regression tests	The incident fixes were never reviewed as code changes	sync.py:86-87, commits 244fb15 and 8656aa3
19	The BOT_WAS_RUN marker file is written and never read	Clutter, and it can be mistaken for a lock	sync.py:67