Rpa_Skills/tests/evals/results/handoff/h-api-sync-screens-r3.md

Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: raise a ticket in the RPA Support queue

1. What the bot does

Every hour, the bot takes the approved invoices in the finance export file (data/input/invoices.csv) and creates each one in the billing system through the billing system's web API. It has no window and drives no desktop app or website. It only talks to the billing API in the background. For each run it writes one report file to data/output/, with one line per invoice saying what happened to it.

2. How it works

Windows Task Scheduler (every hour)
   └─ starts → Invoice sync (no window) ──sends each invoice──> Billing API
                    │
                    ├─ reads  → data/input/invoices.csv
                    ├─ writes → data/output/sync_report_<YYYYMMDD_HHMM>.csv   (ops can open)
                    └─ writes → logs/sync.log                                  (dev only)

• Windows Task Scheduler starts the bot at the top of every hour. Nobody clicks anything.
• Invoice sync is a background program. It has no screen.
• Billing API is the billing system. Invoices the bot creates appear there.
• The input file is the finance export. The bot never moves or empties it, so every run reads the whole file again.
• The report file is the only record ops can open. Each run makes a new file, named after the date and time the run started, for example sync_report_20261023_1400.csv for the 14:00 run.
• The log holds one line per invoice (number and status only) plus an end-of-run line. Ops has no access to it.

One run, step by step:

1. Task Scheduler starts the bot.
2. The bot creates a new, empty report file in data/output/.
3. For each line of the input file, in order:
   1. It checks the line: invoice number, customer ID, amount and currency must be filled in, and the amount must be a number above 0. If the line fails the check, the bot marks it Invalid and moves on without sending it.
   2. It sends the invoice to billing.
   3. If billing gives no clear answer, it asks billing "does this invoice exist now?". If yes, it marks the invoice Synced. If not, it waits and tries again, up to 5 times (section 8).
   4. It writes the result as one line in the report.
4. When every line is done, it writes "run done" to the log. Ops cannot see this. To check a run finished, see section 5.

3. Before a run

• The folder data/output/ exists next to the bot. The bot does not create it. If it is missing, every run stops at once and no report appears.
• The folder logs/ exists next to the bot. If it is missing, the bot stops at once.
• The settings file (.env) with the billing address and key is next to the bot. If it is missing or the key has expired, every invoice ends Failed after about a minute of retries each (E09).
• The input file is at data/input/invoices.csv and keeps its header line exactly: invoice_no, customer_id, customer_name, amount, currency.
• Input rules, one line per invoice:

Field	Rule	Bad	Good
invoice_no, customer_id, currency	Must not be empty	(blank)	INV-1001
amount	A plain number above 0. No currency sign, no thousands separator.	$1,200.50, 0, -5	1200.50
File format	Save as CSV UTF-8 without the "BOM" marker. Some Excel "CSV UTF-8" exports add an invisible marker at the start, and then every line comes back Invalid - missing invoice_no (E04).	Excel "CSV UTF-8" with marker	Plain CSV / UTF-8 without marker

• Only one copy of the bot is scheduled. Nothing stops two runs from overlapping (section 4).

4. During a run

• There is nothing to watch. The bot has no window. Leave the input file alone while a run is going. Don't open it in Excel and save over it at the top of the hour.
• To stop the bot: whoever has access to the bot machine ends the "Invoice sync" task in Task Scheduler, and disables it to stop future runs. Ops has no access to that machine, so raise a ticket. Stopping mid-run is safe: invoices already created come back as Duplicate on the next run, not as second copies.
• A run looks stuck or unfinished when:
  • the newest report has fewer lines than the input file (not counting the header) more than an hour after it started, or
  • no new report appears for an hour.
• The bot has no watchdog. Each invoice can take up to about 6 minutes when billing is slow or down: 5 tries, each waiting up to 30 s to send and 30 s to check, plus 62 s of pauses. A run that is behind can still be going when the next hourly run starts. Both then write their own report.

5. Where to look when something goes wrong

Ops can open only the data/output/ folder and its report files. The bot has no screens.

Question	Where to look
Did the run start?	data/output/ has a sync_report_<date>_<time>.csv for that hour
Did the run finish?	That report has one line per input line (not counting the header). Fewer lines means the run stopped part way (E10-E13).
Which invoices failed?	Report → Status column: anything other than Synced or Duplicate, plus Synced lines whose Message is "Created (some lines rejected)" (E07)
Why did one invoice fail?	Report → Message column on that invoice's line
Did the bot already retry?	Status Failed with Message "No success after 5 tries" means it retried 5 times. Message "Created (confirmed after retry)" means at least one try had no clear answer. Otherwise it did not retry.
Why did the whole run stop?	Not visible to ops. Raise a ticket (section 11).
Which version is running?	Not visible to ops (section 10).

6. Statuses

The report is a plain CSV file, so nothing has a colour. These are every Status and Message the bot writes.

Status	Message	Meaning	What ops does
Synced	Created	The invoice was created in billing	nothing
Synced	Created (some lines rejected)	Needs follow-up. Billing created the invoice but rejected some of its lines. The invoice in billing is incomplete.	Open the invoice in billing and fix the missing lines (E07)
Synced	Created (confirmed after retry)	Billing did not answer clearly, but the invoice exists in billing now. It may have existed before this run (E08).	nothing, unless the invoice in billing looks wrong
Duplicate	Invoice already exists in billing	Billing already has this invoice number, usually from an earlier hourly run. This is normal for every invoice that stays in the input file.	nothing. Look into it only if the invoice is new and was never sent before (E06).
Invalid	missing <field names>	A required field is empty on that line	Fix the input line (E01). If every line says "missing invoice_no", see E04.
Invalid	amount must be positive	The amount is 0 or negative	Fix the amount (E02)
Invalid	amount '<value>' is not a number	The amount has a currency sign, comma or text	Write it as a plain number (E03)
Invalid	(billing's own explanation), or "Rejected by billing"	Billing refused the invoice's content	Read the message, fix the input (E05)
Failed	No success after 5 tries	Billing failed 5 times and the invoice is not in billing	See E09

7. Exception catalogue

ID	What you see	How the bot identifies it	Where it is recorded	Type	Who acts	What to do	Escalate when
E01	Invalid - "missing customer_id" (or other field names)	Before sending, it checks that invoice number, customer ID, amount and currency are filled in	Report, Status and Message	Business	Ops	Fill in the field in the finance export. The next hourly run picks it up.	Never
E02	Invalid - "amount must be positive"	The amount is 0 or below	Report, Status and Message	Business	Ops	Correct the amount, or remove credit notes from the export	Never
E03	Invalid - "amount '1,200.50' is not a number"	The amount cannot be read as a number	Report, Status and Message	Business	Ops	Write amounts as plain numbers: 1200.50	Never
E04	Every line Invalid - "missing invoice_no" (and maybe other fields)	It cannot find the invoice-number column because the file starts with an invisible marker, or the header was renamed	Report, Status and Message	Business	Ops	Re-save the input as plain CSV/UTF-8 without the marker. Check the header line spelling.	It keeps happening with a clean file
E05	Invalid - billing's own explanation, or "Rejected by billing"	Billing answered "content not acceptable"	Report, Status and Message	Business	Ops	Read the message, fix the data in the finance export	The message makes no sense, or billing rejects data that is correct
E06	Duplicate - "Invoice already exists in billing"	Billing answered "this invoice number already exists"	Report, Status and Message	Business	Ops (usually nothing)	Normal for invoices left in the input from earlier runs. For a new invoice, check in billing whether its number is already used by a different invoice.	A new invoice is blocked by someone else's invoice with the same number
E07	Synced - "Created (some lines rejected)"	Billing answered "partly accepted"	Report, Status and Message. Status still says Synced.	Business	Ops	Open the invoice in billing and complete or correct the rejected lines. Filter the report on Message, not only Status.	The cause of the rejection is not clear
E08	Synced - "Created (confirmed after retry)"	Sending got no clear answer or an error, so the bot asked billing whether the invoice exists, and it does	Report, Status and Message	System	Bot	Nothing. If the invoice already existed before this run (for example, someone typed it in by hand), the bot still writes this message.	The invoice in billing is not the one in the input
E09	Failed - "No success after 5 tries"	5 tries without success. Each time, billing was unreachable, timed out (30 s), was busy, or refused the key, and the invoice still was not in billing.	Report, Status and Message. The real reason (busy, down, wrong key) is not shown.	System	Bot, then Ops, then Dev	1. If only a few invoices failed, wait. The next hourly run tries them again. 2. Before re-entering anything by hand, search billing for the invoice number: a timed-out send may have created it anyway.	Every invoice in a run is Failed (likely an expired key or billing down), or the same invoice fails 2 runs in a row
E10	No new report for an hour	The bot stops before it opens the report: the data/output/ or logs/ folder is missing, the input file is missing, or the computer did not run the task	Nothing ops can see	System	Dev	Raise a ticket with the hour that has no report	Always
E11	The report stops part way and later lines never appear. The same line is the last one hour after hour.	While checking "does this invoice exist?" billing was unreachable, and the bot stopped the whole run	Report is cut short. The cause is visible to dev only.	System	Bot, then Dev	Wait one run. If billing was only briefly down, the next run gets past it.	The report stops at the same place 2 runs in a row
E12	The report stops part way, at an invoice billing rejected	Billing rejected the invoice with an answer the bot could not read, and the bot stopped the whole run	Report is cut short. The cause is visible to dev only.	System	Dev	Note the invoice number on the line after the last line in the report. That is the one that stopped the run. Every invoice after it is blocked every hour until it is fixed or removed from the input.	Always
E13	Report cut short for any other reason	Any unexpected error stops the run	Report is cut short. Details are visible to dev only.	System	Dev	Raise a ticket	Always
E14	The same invoice appears twice in billing	A send timed out after billing had created it, and the check-then-retry did not catch it	Not recorded by the bot	System	Ops, then Dev	Void the extra copy in billing following the finance process	Always. It means the duplicate protection did not work (incident 2).

8. What the bot recovers by itself

Problem	What the bot does	Limit	After the limit
Billing slow, busy, down, unreachable, or refusing the key	Checks whether the invoice was created anyway, then waits and tries again. The waits are 2, 4, 8, 16 and 32 s.	5 tries per invoice, up to about 6 minutes per invoice	Invoice marked Failed, the bot moves to the next invoice (E09)
Send timed out after billing actually created the invoice	Before each retry, it asks billing whether the invoice exists	Same 5 tries	Marked Synced - Created (confirmed after retry) (E08)
Invoice not processed in this run (Failed, Invalid, or the run stopped)	Nothing within the run. The next hourly run reads the whole input file again.	Every hour, without end, while the line stays in the input	Fixed invoices go through. Invoices already created show Duplicate.
The whole run stops	Nothing. No restart, no resume.	-	The next hourly run starts again from the first line

9. Known incidents

Newest first. The history has no dates. Both were fixed before or at 0.9.1.

When / version	What ops saw	Cause	What to do now	Ops could handle?
Before 0.9.1	The same invoice appeared twice in billing	Billing timed out after it had already created the invoice. The bot thought it failed and sent it again.	Fixed: the bot now checks whether the invoice exists before each retry. That check can still stop the run if billing is down (E11). If doubles appear again: void the extra copy and raise a ticket (E14).	Partly. The clean-up, yes. Prevention needed a code change.
Before 0.9.1	The hourly run stopped part way	Billing said "too many requests" and the bot had no retry	Fixed: 5 tries with waits. Re-running is safe because invoices already created come back as Duplicate.	Partly

10. Releases and version check

• How an update arrives: a developer replaces the bot's files on the bot machine. There is no auto-update.
• How to confirm it applied: ops cannot see the version. The bot does not show it anywhere ops can reach. Ask dev to confirm the version in the ticket that announced the update.
• If the version did not change: reply on the same ticket in the RPA Support queue.
• Only one machine runs this bot. It writes nothing shared at start-up, so an old copy cannot overwrite a new one. But two copies scheduled on two machines would both send the same invoices. Keep it to one.

11. Escalating to dev

Raise a ticket in the RPA Support queue with:

• the exception ID from section 7 (for example, E11)
• the report file name, for example sync_report_20261023_1400.csv. It gives the run date and time.
• the invoice number(s) affected, and the last invoice number in a cut-short report
• whether the previous hour's run showed the same problem

Never send: the settings file (.env), the billing key, or the report file itself in chat. The report contains customer names, so reference it by name.

12. Failure-class check

• F1 (missing prerequisite) covered in section 3 (output folder, logs folder, settings file, input file), E09, E10, and Fix before handover #1.
• F2 not applicable: the bot drives no windows. It only calls a web API in the background.
• F3 (target slower than expected) covered in E09, E11, section 8 (30 s timeout, 5 tries) and section 4 (6 minutes per invoice).
• F4 (input read differently) covered in section 3 input rules, E01-E04, and Fix before handover #7.
• F5 (marked done when nothing was done) covered in E08 and Fix before handover #5.
• F6 (action happened, not confirmed) covered in E08, E09 (search billing before re-entering), E14, and Fix before handover #4.
• F7 (business exception shown as success) covered in E07, section 6, and Fix before handover #3.
• F8 (endless re-runs / watchdog) covered in section 4 (no watchdog, overlapping runs), section 8 (whole file every hour), and Fix before handover #6 and #8.
• F9 (stopping the wrong way) covered in section 4 (end the task in Task Scheduler, via ticket).
• F10 (one stuck item blocks the queue) covered in E11 and E12 (one invoice can stop every later invoice, every hour) and Fix before handover #4.
• F11 (cause hidden from ops) covered in section 5 and Fix before handover #2.
• F12 (update did not arrive) covered in section 10 and Fix before handover #9.

13. Fix before handover (dev team)

#	Gap	Why it hurts ops	Where (file:line)
1	data/output/ is not in the repo and the code never creates it. logs/ must also exist before logging.basicConfig opens the file.	A fresh install fails every run with no report and nothing ops can see	sync.py:24, sync.py:68-69
2	Run crashes and network errors go only to print() (console), which Task Scheduler throws away. sync.log holds the status but never the Message. Ops cannot open logs/.	Ops cannot tell why a run stopped (E10-E13). Even dev has no traceback.	sync.py:52, sync.py:75, sync.py:84
3	HTTP 207 (partial) is stored as Status Synced	Incomplete invoices look done when filtering by Status (E07)	sync.py:54-55
4	exists() has no try/except, and the 422 r.json() is not guarded. Neither is caught per row.	One bad invoice or a short outage stops the whole run, and every later invoice is blocked every hour (E11, E12)	sync.py:42, sync.py:59, sync.py:72-76
5	exists() returning 200 is reported as "Created (confirmed after retry)", even on the first attempt and even if the invoice existed before this run	Synced does not prove this run created it (E08)	sync.py:60-61
6	Every status other than 200/201/207/409/422 is retried, including 400/401/403/404. The docstring says only 429/5xx. Retry-After is ignored, and the bot sleeps after the last try too. The Failed message has no HTTP code.	An expired key costs about a minute per invoice and shows only "No success after 5 tries" (E09)	sync.py:47, sync.py:60-63
7	Input read as utf-8, not utf-8-sig. The amount check accepts nan/inf.	An Excel export with a BOM turns every line Invalid (E04)	sync.py:34, sync.py:69
8	Input is never archived and there is no run lock	The whole file is re-sent every hour (noisy Duplicates), and slow runs can overlap the next hourly run	sync.py:66-77
9	Version exists only in README text, with a stray chore: bump to 0.9.1 line. Nothing ops can see shows the version.	Ops cannot confirm an update applied (F12)	README.md:5-6
10	The report includes customer_name	Personal data in a folder ops can open. Confirm this is allowed.	sync.py:76
11	.env with the billing key is committed to git	Credential exposure. Rotate the key and untrack the file.	.env (commit 88b2057)
12	Duplicate protection relies on billing answering 409, yet incident 2 produced doubles	Confirm billing enforces unique invoice numbers. Otherwise E14 can recur.	sync.py:56-57
13	BOT_WAS_RUN marker file is written into the project root every run	Leftover file with no purpose for ops	sync.py:67