Rpa_Skills/tests/evals/results/handoff/h-api-sync-full-r2.md

Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel #rpa-support

1. What the bot does

Every hour, Windows Task Scheduler starts the bot. It reads the finance export file data/input/invoices.csv (one approved invoice per row), checks each row, and sends each valid invoice to the billing system over its web API. For each run it writes one report file, data/output/sync_report_<date>_<time>.csv, with a Status and a Message for every invoice, and adds a few lines to logs/sync.log. It has no screen and drives no desktop app. The only system it talks to is the billing system.

2. How it works

Windows Task Scheduler (every hour)
   └─ starts → Invoice sync script ──reads──> data/input/invoices.csv
                    │
                    ├─ sends each invoice ──> Billing system (web API)
                    ├─ writes → data/output/sync_report_<date>_<time>.csv  (one row per invoice)
                    └─ writes → logs/sync.log  (one line per invoice + "run done")

• Task Scheduler starts the bot on the hour. The bot has no window of its own.
• Invoice sync script does all the work, one invoice at a time, top to bottom.
• Input file is the finance export. The bot reads the whole file on every run.
• Billing system is where invoices are created. The bot never deletes or changes an invoice there.
• Report file is the main place to see what happened to each invoice in that run. It contains customer names, so treat it as confidential.
• Log file records only the invoice number and its status, plus a "run done" line at the end of each complete run.

One run, step by step:

1. Open the input file and create a new report file for this run.
2. For each row: check that the invoice number, customer ID, amount and currency are filled and that the amount is a positive number.
3. If the row passes, send it to billing. If billing doesn't give a clear yes or no, ask billing whether the invoice now exists. If it doesn't, wait and send again, up to 5 tries.
4. Write the result to the report and the log, then go to the next row.
5. After the last row, write run done -> <report name> in the log.

Each run handles the whole input file again. Invoices that were already sent come back as Duplicate, which is normal.

3. Before a run

• The folder data/output exists. The bot does not create it. If it is missing, every run stops before the first invoice and writes no report (E11).
• The folder logs exists.
• The settings file (.env) is in the bot folder with the billing address and billing key. The dev team maintains it. Ops never opens or copies it. If the key is missing or expired, every invoice ends up Failed (E09).
• The input file is at data/input/invoices.csv, with the header row invoice_no, customer_id, customer_name, amount, currency.
• One invoice per row. Invoice number, customer ID, amount and currency must be filled.
• Amount is a plain number with a dot for decimals. Good: 1250.00. Bad: 1,250.00, $1250, 1250 USD (each becomes Invalid E03).
• Amount is above zero. Credit notes or zero-amount rows become Invalid (E02).
• Version: ops cannot see which version is running (see section 10). Ask the dev team to confirm 0.9.1.

4. During a run

• Don't save over or edit the input file while a run is in progress. The bot reads it row by row as it goes.
• Don't delete or move the newest report file until the log shows run done for it.
• To stop the bot: whoever has access to Task Scheduler on the bot machine selects the invoice sync task and chooses End. To stop future hourly runs as well, choose Disable. If ops don't have Task Scheduler access, ask in #rpa-support. A stopped run leaves a short report and no run done line. Running it again is safe: invoices already created come back as Duplicate.
• A run looks stuck when the newest report file stops growing for several minutes. When billing is down, each invoice can take up to about 6 minutes before it is marked Failed (5 tries with waits). There is no automatic stop or restart. Wait for that. Escalate when a run is still going when the next hourly run is due, or when the report hasn't grown for more than 10 minutes.

5. Where to look when something goes wrong

Question	Where to look
Did the run finish?	logs/sync.log: look for run done -> sync_report_<date>_<time>.csv for that hour. No such line means the run stopped partway (E10) or never started (E11).
Which invoices failed?	The report for that run in data/output: Status column is anything other than Synced. Also check Synced rows whose Message says "some lines rejected" (E06).
Why did one invoice fail?	The same report, Message column. For Failed rows the real reason is not recorded anywhere ops can open (fix item 3). Escalate with the invoice number.
When was an invoice last tried?	logs/sync.log: search for the invoice number. Each line has the date, time and status.
Did the bot already retry?	Failed always means 5 tries were made. "Created (confirmed after retry)" means a retry happened and worked. Nothing else records retries.
Why did a whole run stop?	Not visible to ops. The reason is only shown on a console nobody sees (fix item 1). Escalate (E10).
Which version is running?	Not visible to ops (fix item 15). Ask the dev team.

6. Statuses

The report is a plain CSV file, so there are no colours. The Status column holds one of four values. The Message column explains it.

Status - Message	Colour	Meaning	What ops does
Synced - "Created"	none (plain text)	Invoice created in billing	Nothing
Synced - "Created (confirmed after retry)"	none	Billing didn't answer clearly the first time, but the bot checked and the invoice is there	Nothing
Synced - "Created (some lines rejected)"	none	Needs follow-up. The invoice was created, but billing refused some of its lines. Shown as Synced anyway.	Check the invoice in billing and add the missing lines by hand (E06)
Duplicate - "Invoice already exists in billing"	none	Billing already has this invoice number. Normal for invoices sent in an earlier hour.	Nothing, unless billing shows it twice (E13)
Invalid - "missing "	none	A required field in the input row is empty	Fix the row in the input (E01)
Invalid - "amount must be positive"	none	Amount is zero or negative	Check the row with finance (E02)
Invalid - "amount '' is not a number"	none	Amount isn't a plain number	Fix the amount format (E03)
Invalid - billing's own text, or "Rejected by billing"	none	Billing looked at the invoice and refused it	Read the message and fix the data with finance (E04)
Failed - "No success after 5 tries"	none	Billing couldn't be reached or kept failing. Not created.	See E08 / E09

7. Exception catalogue

ID	What you see	How the bot identifies it	Where it is recorded	Type	Who acts	What to do	Escalate when
E01	Invalid - "missing amount, currency" (lists the empty fields)	Before sending, it checks that invoice number, customer ID, amount and currency are filled	Report: Status, Message. Log: invoice number + Invalid	Business	Ops	Ask finance to fill the fields in the export. The next hourly run picks the row up.	Never, unless the fields are filled and it still says missing (check the header row spelling)
E02	Invalid - "amount must be positive"	Amount is zero or below	Report; log	Business	Ops	Check with finance. Zero and credit invoices can't go through this bot.	Finance needs credits to sync (change request)
E03	Invalid - "amount '1,250.00' is not a number"	The amount can't be read as a plain number	Report; log	Business	Ops	Remove thousands separators, currency signs and text from the amount	The amount is already plain digits
E04	Invalid - with billing's own explanation, or "Rejected by billing"	Billing replied that the invoice data is wrong	Report; log	Business	Ops	Read the message; fix the customer, currency or amount with finance	The message doesn't make sense, or a correct invoice is rejected again
E05	Duplicate - "Invoice already exists in billing" every hour for the same invoices	Billing replied that the invoice number already exists	Report; log	Business	Ops	Normal for invoices already sent: the export keeps old rows. Only check billing when you don't expect the invoice to exist yet.	The invoice is not in billing although the bot says Duplicate
E06	Synced - "Created (some lines rejected)"	Billing created the invoice but said some lines were refused	Report Message column only. The log just says Synced.	Business	Ops	Open the invoice in billing, compare with the export, add the missing lines by hand	Every invoice of a run shows this
E07	Synced - "Created (confirmed after retry)"	Billing timed out or errored, so the bot asked billing whether the invoice now exists, and it did	Report; log says Synced	System	Bot	Nothing. The bot avoided sending it twice.	Never
E08	Failed - "No success after 5 tries" on some invoices	Five tries with no clear answer from billing, and billing doesn't have the invoice. Causes: billing slow/down, too many requests, network trouble.	Report; log. The real cause isn't recorded.	System	Bot, then Dev	Nothing at first: if the invoice is still in the input file, the next hourly run tries it again	The same invoice is Failed in 2 runs in a row. Send: invoice number, report file name, the log lines for it.
E09	Failed on every invoice of a run	Same as E08, but for all rows. Usually the billing key/address in the settings file is wrong or expired, or billing is down.	Report; log	System	Dev	Don't edit anything. Escalate straight away.	Always. Send: report file name, run time, the logs/sync.log lines for that hour.
E10	Report ends early; no run done line in the log for that hour; later invoices never appear	An unexpected error ended the whole run, e.g. billing replied in a form the bot can't read, or the network dropped while it was checking whether an invoice exists. The bot does not skip the row; it stops.	Report is cut short; log has no run done. Reason only on a console nobody sees.	System	Dev	Note the last invoice in the report; the stop happened on the next row of the input. If the same row stops every hourly run, nothing after it is ever synced.	Always. Send: report file name, last invoice number in the report, run time.
E11	No new report and no log lines for an hour	The run stopped before the first invoice: data/output folder missing, input file missing or renamed, or the scheduled task didn't start	Nowhere ops can see	System	Ops, then Dev	Check that data/output exists (create it if not) and that data/input/invoices.csv is there	Both are in place and still no report next hour. Send: the hours with no report.
E12	Two reports a few minutes apart, or one hour's report missing	A slow run (billing down, large file) was still going when the next hourly run started. Nothing stops two runs overlapping. Reports from two runs started in the same minute overwrite each other.	Report file names and times in data/output; log lines from two runs mixed	System	Dev	Don't delete anything	Always. Send: the report file names and times.
E13	Billing (or finance) reports the same invoice twice	Billing created it but the reply was lost. The bot checks before re-sending, but if that check also fails it sends again. Usually billing then refuses it as Duplicate, so the report shows Duplicate for an invoice this run actually created.	Report shows Synced or Duplicate; nothing marks it	System	Dev	Don't delete in billing yourself unless that is your normal billing process	Always. Send: invoice number, both report files around it.

8. What the bot recovers by itself

Problem	What the bot does	Limit	After the limit
Billing times out, is busy (too many requests), errors, or the network fails	Asks billing whether the invoice now exists; if not, waits and sends again. Waits are 2, 4, 8, 16, 32 s. Each request can take up to 30 s.	5 tries, about 1 to 6 minutes per invoice	Failed - "No success after 5 tries", next invoice
Billing created the invoice but its reply was lost	Finds it with the "does it exist" check	Each try	Synced - "Created (confirmed after retry)"
An invoice not synced this hour	Nothing during the run. The next hourly run reads the whole input file again.	Every hour, as long as the row stays in the input file	-
Run stopped partway (E10, E11)	Nothing. No restart until the next hour.	-	If the cause is the same row, every hourly run stops at that row

There is no watchdog, no stuck detection and no automatic restart.

9. Known incidents

Dates aren't recorded. Both were fixed before version 0.9.1.

When / version	What ops saw	Cause	What to do now	Ops could handle?
Before 0.9.1	The same invoice twice in billing	Billing created it but timed out; the bot sent it again	Fixed: the bot checks before re-sending. Not fully: if the check fails too, it can still re-send (E13).	Partly
Before 0.9.1	The hourly run died partway; later invoices weren't sent	Billing said "too many requests" and the bot stopped	Fixed: now retried 5 times. Not fully: other errors can still stop the run (E10). Compare the report row count with the input; escalate.	Partly

10. Releases and version check

• How an update arrives: the dev team copies the new code into the bot folder on the bot machine. There is no launcher or automatic update.
• How to confirm it applied: ops can't. The version is written only in the project's README, not in the log, the report or anywhere ops can open. Ask the dev team in #rpa-support to confirm the version after each update (fix item 15).
• If the version did not change: escalate in #rpa-support.

11. Escalating to dev

Post in #rpa-support (Teams). Send: the exception ID from section 7, the report file name (it carries the run date and time), the invoice number(s), and the matching lines from logs/sync.log. Never send: the settings file (.env), billing keys or passwords, or the report file itself. It contains customer names, so give the file name and the dev team opens it on the machine.

12. Failure-class check

• F1 covered in section 3 (output folder, settings file, input path) and E09, E11.
• F2 not applicable: the bot drives no windows; it only talks to the billing web API.
• F3 covered in section 8 (30 s per request, 5 tries) and E07, E08, E12.
• F4 covered in section 3 (amount format, good/bad examples) and E01-E03.
• F5 covered in fix items 9 and 10 (a run where every invoice failed still looks successful to Task Scheduler) and E09.
• F6 covered in E07, E13 and fix item 14.
• F7 covered in section 6 ("some lines rejected" shown as Synced), E06 and fix item 9.
• F8 covered in section 8 (no watchdog, hourly re-read of the whole file), E05, E12 and fix item 11.
• F9 covered in section 4 (End the scheduled task) and fix item 21.
• F10 covered in E10: one bad row stops that run and every later run at the same row. Fix item 7.
• F11 covered in section 5 and fix items 1, 2, 3.
• F12 covered in section 10 and fix item 15.

13. Fix before handover (dev team)

#	Gap	Why it hurts ops	Where (file:line)
1	Network errors and the crash reason go only to the console (print), never to sync.log	Under Task Scheduler the reason for a stopped run is lost	sync.py:52, sync.py:84
2	Log line has only invoice + status, always INFO; no message, no WARNING/ERROR	Ops can't search the log for problems or reasons	sync.py:75
3	Failed rows don't record the last HTTP status or error	Ops can't tell billing-down from bad key from bad data	sync.py:60-63
4	Every non-handled status is retried, including 400/401/403/404 (docstring says only 429/5xx)	A bad key costs ~1 min per invoice and shows only as "Failed"	sync.py:47, sync.py:60-62
5	The exists() check has no error handling	A timeout during the check kills the whole run (same symptom as the 429 incident)	sync.py:41-43, sync.py:60
6	422 handling calls r.json() without a guard	A non-JSON 422 body kills the run	sync.py:59
7	No per-row error handling in the main loop	One bad row stops the run and, if it stays in the input, every later hourly run	sync.py:72-76
8	data/output isn't created and isn't in the repo	A fresh deploy fails on every run with no report	sync.py:68-69
9	HTTP 207 (some lines rejected) is reported as Synced	Partly created invoices look done	sync.py:54-55
10	Exit code is 0 even when every invoice Failed	Task Scheduler shows success for a fully failed run	sync.py:80-85
11	No lock against overlapping runs; report name only to the minute	Overlapping runs; reports overwrite each other	sync.py:68
12	Sleeps 32 s after the final attempt too	Wastes time per Failed invoice, more overrun risk	sync.py:62
13	Retry-After header from billing is ignored on 429	Retries may hit the rate limit again	sync.py:62
14	If the existence check returns non-200 for any reason, the bot re-posts; a 409 then reports Duplicate for an invoice this run created. Relies on billing returning 409 (unconfirmed).	Misleading status; double-post risk if billing doesn't reject duplicates	sync.py:43, sync.py:56-57
15	Version only in README (plus a stray commit-message line); not in log or report	Ops can't confirm which version runs	README.md:5-6
16	The two "fix" commits (6be93ce, 28da8f3) only add comment lines; the retry/check code was in the initial import	Can't tell when the fixes went live, or that the deployed copy has them	sync.py:86-87
17	The report includes customer_name	Personal data sits in the output folder ops share	sync.py:76
18	BOT_WAS_RUN marker written to the project root on every run, read by nothing	Confusing file; remove or document	sync.py:67
19	nan / inf amounts pass validation	Garbage amounts sent to billing	sync.py:33-37
20	Missing billing key or URL isn't checked at start; falls back to a placeholder URL / empty key	Shows up only as every row Failed (E09)	sync.py:18-19
21	The scheduled task isn't in the repo; console output not redirected; ops have no listed Task Scheduler access	Ops can't stop the bot or see crash output themselves	sync.py:3