Rpa_Skills/tests/evals/results/handoff/h-api-sync-screens-r1.md

Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: raise a ticket in the RPA Support queue

1. What the bot does

Every hour it takes the approved invoices from the finance export file (data/input/invoices.csv) and creates each one in the billing system through the billing system's web service. For every invoice in the file it writes one line to a report file in data/output/, saying whether the invoice was created, was already there, was rejected, or failed. It has no window and no screens: it runs in the background from Windows Task Scheduler, and it drives no desktop apps or websites. Ops works from the report files in the output folder.

2. How it works

Windows Task Scheduler (every hour)
   └─ starts → Invoice sync (background script, no window)
                  │
                  ├─ reads  ← data/input/invoices.csv        (finance export)
                  ├─ sends ──> Billing system web service    (creates invoices)
                  ├─ writes → data/output/sync_report_<date>_<time>.csv   (ops can open)
                  └─ writes → logs/sync.log                                (ops cannot open)

• Task Scheduler starts the bot once an hour. Ops has no access to it.
• Invoice sync reads the whole export from top to bottom, one invoice at a time.
• The finance export is the input. Every invoice still in it is sent again every hour.
• The billing web service creates the invoice, or answers "already exists", "rejected" or with an error.
• The report file has one line per invoice: invoice number, customer, Status, Message. A new file is made every run, named by date and time (for example sync_report_20261023_1400.csv = 23 Oct 2026, 14:00).
• The log file holds only the invoice number and status, plus a "run done" line. Ops cannot open it, and it doesn't hold the reasons anyway.

One run, step by step:

1. Task Scheduler starts the bot on the hour.
2. The bot creates a new, empty report file in data/output/.
3. For each invoice in the export, it checks that the invoice number, customer ID, amount and currency are filled in and that the amount is a positive number. If not, the line is Invalid and it moves on.
4. It sends the invoice to billing. If billing doesn't answer clearly, it checks whether billing has the invoice anyway, waits and tries again, up to 5 tries (section 8).
5. It writes the result line to the report and goes to the next invoice.
6. After the last invoice the run ends. Nothing in the report marks the end (see E11).

3. Before a run

Ops cannot change the bot machine. These items are for whoever sets it up; ops only needs to know them so the symptom makes sense.

• The bot folder has a .env settings file with the billing address and access key. If it is missing or the key is wrong or expired, every invoice ends Failed (E10).
• The folders data/output/ and logs/ exist in the bot folder. The bot does not create them. If one is missing, every run crashes and no report appears (E11).
• Python is installed on the bot machine with the two packages listed in requirements.txt.
• The finance export follows these rules (if it doesn't, ask whoever produces the export to fix it):

Rule	Bad	Good
Column headers exactly, lower case: invoice_no, customer_id, customer_name, amount, currency	Invoice No	invoice_no
Invoice number, customer ID, amount and currency are always filled in (customer name may be empty)	amount empty	1200.50
Amount is a plain number: dot for decimals, no thousands separator, no currency sign	1,200.50 or $1200.50	1200.50
Amount is above zero. Zero and negative amounts (credit notes) are never synced	-50.00	50.00
The file comes straight from the finance system. Don't re-save it in Excel: Excel's "CSV UTF-8" option makes every line Invalid "missing invoice_no", and other Excel CSV options can crash the run when names have accents	re-saved in Excel	original export
No extra columns. The bot sends every column to billing, so an extra column can be rejected or leak data into billing	an added "notes" column	the five columns only

Sample of the layout: data/input/invoices.csv (headers only - don't copy its rows).

4. During a run

• Don't move, rename or delete files in data/output/ while a run is going. The newest report is still being written.
• Don't open the newest report until its "modified" time has stopped changing for a few minutes.
• To stop the bot: there is no Stop button. It can only be stopped in Task Scheduler, which ops cannot reach. Raise a ticket. Stopping it mid-invoice can leave that one invoice created in billing but missing from the report.
• A run looks stuck when the newest report's modified time hasn't changed for more than 6 minutes. One invoice can legitimately take up to about 6 minutes while billing is not answering (section 8). The bot has no watchdog: it never gives up on the run as a whole, it only gives up on one invoice at a time. A run with many failing invoices can last longer than an hour (E12).

5. Where to look when something goes wrong

Ops can open only the output folder data/output/. Everything below uses it.

Question	Where to look
Did the run start?	A report file exists for that hour: sync_report_<date>_<hour><minute>.csv. No file for the hour = the run did not start or crashed at once (E11).
Did the run finish?	Ops cannot tell for sure. The report has no end line. If it has fewer invoices than the export, or stops at the same invoice every hour, treat it as a crash (E11).
Which invoices failed?	Open the report, filter the Status column for Failed, Invalid, and Synced lines whose Message says "some lines rejected" or "confirmed after retry".
Why did one invoice fail?	The Message column on that invoice's line. That is the only reason the bot gives ops. For Failed there is no further detail anywhere ops can see.
Did the bot already retry?	Yes, always, for every Failed line (5 tries). A Synced line with "confirmed after retry" also went through a retry. The next hourly run tries again any invoice still in the export.
Which version is running?	Ops cannot see it. Ask in the ticket (section 10).

6. Statuses

The report is a plain CSV file: it has no colours. The Status is the third column, the Message the fourth.

Status	Message	Meaning	What ops does
Synced	Created	Invoice created in billing	nothing
Synced	Created (confirmed after retry)	The first send had no clear answer; the bot then found the invoice in billing	Check billing for a second copy (E07)
Synced	Created (some lines rejected)	Needs follow-up. Invoice created, but billing refused some of its lines	Fix the missing lines in billing (E08)
Duplicate	Invoice already exists in billing	Billing already has this invoice number. Normal for every invoice synced in an earlier hour	nothing, unless the invoice is new (E06)
Invalid	missing	A required field is empty in the export	Fix the export (E01, E02)
Invalid	amount must be positive	Amount is zero or negative	Business decision (E03)
Invalid	amount '' is not a number	Amount is written in a way the bot can't read	Fix the export (E04)
Invalid	billing's own reason text, or "Rejected by billing"	Billing checked the invoice and refused it	Read the reason, fix the data (E05)
Failed	No success after 5 tries	Billing never confirmed it after 5 tries. It may still have been created	Check billing first (E09, E10)

7. Exception catalogue

ID	What you see	How the bot identifies it	Where it is recorded	Type	Who acts	What to do	Escalate when
E01	Invalid - "missing customer_id" (or another field name) on some lines	Before sending, it checks that invoice number, customer ID, amount and currency are not empty	Report, Status and Message	Business	Ops	Ask the export owner to fill the field. The next hourly run picks the invoice up.	The field is filled in the export but still reported missing
E02	Invalid - "missing invoice_no" (often with other fields) on every line	Same check, but the bot can't find the invoice number column at all: the file was re-saved in Excel as "CSV UTF-8", or the headers were renamed	Report, Status and Message	Business	Ops, then Dev	Ask for the file straight from the finance system with the exact headers (section 3).	The file is an untouched export and every line is still Invalid. Send: report file name, E02.
E03	Invalid - "amount must be positive"	Amount is zero or below	Report, Status and Message	Business	Ops	Zero and negative invoices are never synced by this bot. Handle them in billing by hand if the business needs them.	The amount in the export is above zero
E04	Invalid - "amount '1,200.50' is not a number" (shows the value)	The amount can't be read as a number: comma separator, currency sign, text	Report, Status and Message	Business	Ops	Ask for the amount as a plain number, e.g. 1200.50.	The value shown looks like a plain number
E05	Invalid - billing's own reason (for example about the customer), or "Rejected by billing"	Billing answered that the invoice data is not acceptable. The bot does not retry this	Report, Status and Message	Business	Ops	Read the reason. Fix the data in the export, or in billing (for example a customer ID billing doesn't know).	The reason is unclear, or every line has it. Send: report file name, invoice numbers, E05.
E06	Duplicate - "Invoice already exists in billing"	Billing answered that the invoice number already exists. No retry	Report, Status and Message	Business	Ops	Normally nothing: every invoice synced in an earlier hour shows this every hour after. If the invoice is new, look it up in billing: another invoice may already use that number.	Billing shows no invoice with that number. Send: invoice number, report file name, E06.
E07	Synced - "Created (confirmed after retry)"	The first send timed out or billing gave an error. Before trying again, the bot looked the invoice number up in billing and found it	Report, Status and Message	System	Bot, then Ops	Look the invoice up in billing and make sure there is only one copy (see the 504 incident, section 9). Remove an extra copy using the billing team's process.	Two copies found. Send: invoice number, report file name, E07.
E08	Synced - "Created (some lines rejected)"	Billing created the invoice but said part of it was refused. The bot records this as Synced anyway	Report, Status and Message	Business	Ops	Not finished despite "Synced". Open the invoice in billing, compare its lines with finance, add the missing lines with the finance team. The report doesn't say which lines were refused.	You can't work out which lines are missing. Send: invoice number, report file name, E08.
E09	Failed - "No success after 5 tries" on a few lines	Billing didn't answer, timed out, or gave an error 5 times in a row (section 8), and the invoice number was never found in billing	Report, Status and Message	System	Bot, then Ops	Check billing first. The invoice may have been created even though the bot couldn't confirm it. Found: nothing to do, the next run will show Duplicate. Not found: wait for the next hourly run, which tries again if the invoice is still in the export.	The same invoice is Failed in 2 runs in a row. Send: invoice number, both report file names, E09.
E10	Failed - "No success after 5 tries" on all or most lines	Same as E09, for every invoice. Usually billing is down, or the bot's billing address or access key is wrong or expired. The bot can't tell these apart and gives no reason	Report, Status and Message	System	Dev	Raise a ticket at once. Expect the run to be slow: each invoice takes over a minute (E12).	Always. Send: report file name, number of Failed lines, time noticed, E10.
E11	No report for the hour, or a report that stops part-way (fewer invoices than the export, often stopping at the same invoice every hour)	The run crashed. Known causes: the billing look-up after a timeout failed too; billing refused an invoice with an answer the bot couldn't read; the data/output or logs folder is missing; the export file is missing or saved in an unexpected encoding. The cause is not recorded anywhere ops can see	Nothing ops can see. The report simply ends	System	Dev	Raise a ticket. Don't re-run anything: the next hourly run starts again from the top and will probably stop in the same place. Check billing for the invoice after the last line in the report: it may have been created.	Always. Send: report file name (or "no report for "), the last invoice number in the report, E11.
E12	The newest report is still growing after the next hour has started, or an hour's report is missing while the previous one is still being written	The run is longer than an hour, usually because many invoices are failing slowly (E10). Nothing stops two runs from overlapping	Report file times in data/output/	System	Dev	Raise a ticket. Two runs at once can send the same invoice twice.	Always. Send: the report file names and their times, E12.

8. What the bot recovers by itself

Problem	What the bot does	Limit	After the limit
Billing doesn't answer, times out, is busy, or gives an error (this includes a wrong access key)	Waits up to 30 s for an answer. Then checks whether billing has the invoice anyway; if it does, records Synced "confirmed after retry". If not, waits and sends again	5 tries. Waits of 2, 4, 8, 16 and 32 s between tries. Worst case about 6 minutes for one invoice	Line = Failed "No success after 5 tries", next invoice
Billing says the invoice already exists	No retry	-	Line = Duplicate, next invoice
Billing refuses the invoice data	No retry	-	Line = Invalid with billing's reason, next invoice
Invoice not synced this hour (Failed, Invalid, or after a crash)	The next hourly run sends every invoice in the export again, from the top	Every hour, for as long as the invoice is in the export	Already-created invoices come back as Duplicate
Run crash	No recovery inside the run. The rest of the export is not processed until the next hour	-	Next hourly run starts from the top (E11)

There is no watchdog and no automatic restart. A Failed line is final for that run; ops waits for the next hourly report.

9. Known incidents

When / version	What ops saw	Cause	What to do now	Ops could handle?
Before 0.9.1	The same invoice appeared twice in billing	Billing timed out after it had already created the invoice; the bot sent it again and billing accepted the second copy	Fixed: the bot now looks the invoice up before every retry. Still check every "confirmed after retry" and every Failed line in billing (E07, E09).	Partly
Before 0.9.1	The hourly run died part-way; invoices after that point got no report line	Billing limited how fast the bot could send (rate limit) and the bot didn't retry	Fixed for the send: it now retries with waits. The look-up after a timeout can still crash the run the same way (E11, section 13 item 1).	Partly

10. Releases and version check

• How an update arrives: not handled by the bot. There is no launcher or auto-update; a developer replaces the bot's files on the bot machine.
• How to confirm it applied: ops can't. The version (0.9.1) is written only in the project's README.md, which ops can't open. Ask the developer to confirm the version in the ticket.
• If the version did not change: raise a ticket in RPA Support.

11. Escalating to dev

Raise a ticket in the RPA Support queue.

Send: the exception ID from section 7, the report file name (it carries the date and time of the run), the invoice numbers affected, and the time you noticed. For a crash (E11), the last invoice number in the report.

Never send: the .env settings file, the billing access key, or customer names and amounts pasted into chat. The report files contain customer names: give the file name and let the developer open it.

12. Failure-class check

• F1 covered in section 3 (settings file, folders, packages) and E10 / E11.
• F2 not applicable: the bot has no windows and drives no desktop app; it only talks to a web service.
• F3 covered in E07, E09, E10 and section 8 (30 s wait, 5 tries).
• F4 covered in section 3 input rules and E01-E04; also section 13 items 8 and 9.
• F5 covered in E07 (Synced only when billing confirms or the invoice is found by number). Remaining gap: section 13 item 16.
• F6 covered in E09 and E11 (check billing before assuming not created); section 13 items 1 and 16.
• F7 covered in section 6 and E08 ("some lines rejected" shown as Synced); section 13 item 5.
• F8 covered in section 8 and E12: no loop limit beyond 5 tries per invoice, no watchdog, runs can overlap; section 13 items 3 and 10.
• F9 covered in section 4: no Stop button, Task Scheduler only; section 13 item 17.
• F10 covered in E11: one invoice that crashes the run blocks every invoice after it, every hour; section 13 items 1 and 2.
• F11 covered in section 5 and section 13 items 4 and 6: reasons for Failed and for crashes are not shown anywhere ops can open.
• F12 covered in section 10: no auto-update and no version ops can see; section 13 item 12.

13. Fix before handover (dev team)

#	Gap	Why it hurts ops	Where (file:line)
1	The existence check has no error handling. A timeout or connection error there crashes the whole run: the same symptom as the 429 incident, on the second call	Run stops mid-file, every hour, at the same invoice (E11)	sync.py:41-43, called at sync.py:60
2	A 422 with a non-JSON body (e.g. HTML error page) raises in r.json() and crashes the run	Same as above (E11)	sync.py:59
3	Every non-success status is retried, including 400/401/403/404; the docstring says only 429/5xx. A bad key makes every invoice take ~62 s+ and end Failed; the run doesn't stop early	Slow runs that overlap the next hour, no clue it's the key (E10, E12)	sync.py:47, sync.py:60-62
4	Failed message has no HTTP status or reason; the log records only invoice + status; network errors and the CRASH line go to stdout, lost under Task Scheduler	Ops (and dev) can't tell a rate limit from an auth failure from an outage	sync.py:52, sync.py:63, sync.py:75, sync.py:84
5	HTTP 207 (partial) is stored as Status Synced	Ops filtering on Status misses invoices that need follow-up (E08)	sync.py:54-55
6	The "run done" line goes only to logs/sync.log (no ops access); the BOT_WAS_RUN marker is in the bot folder and holds only "1". Nothing in the output folder says a run finished or crashed	Ops can't tell a finished run from a crashed one. Suggest a final summary line or a .done / .failed file in data/output/	sync.py:67, sync.py:77, sync.py:80-85
7	data/output/ and logs/ are not created by the code and data/output/ isn't in the repo. A fresh install crashes every run	No report at all (E11)	sync.py:24, sync.py:68-69
8	Input opened as utf-8, not utf-8-sig. A BOM (Excel "CSV UTF-8") makes every line Invalid "missing invoice_no"; a non-UTF-8 file crashes the run	E02, E11	sync.py:69
9	The whole CSV row, including any extra columns and customer_name, is posted as JSON	Extra columns can be rejected or leak into billing	sync.py:50
10	No lock against overlapping runs; report name is to the minute, so two runs in one minute overwrite one file. Task Scheduler's "if already running" setting isn't documented	Same invoice sent twice at once; lost report (E12)	sync.py:68; Task Scheduler task
11	Retry-After is ignored on 429; the 32 s sleep after the 5th try is wasted	Longer runs	sync.py:62
12	Version shown only in README; nothing ops can see shows it. README also has a stray commit-message line	Can't confirm an update applied	README.md:5-6
13	.env with the live billing key is committed (initial import)	Key exposed in git history: rotate it, remove .env from the repo, add to .gitignore	.env, commit 88b2057
14	Customer names are copied into every report; no retention or access rule for data/output/	Customer data piles up in a folder ops opens daily	sync.py:76
15	Both fix commits only add comments; there are no tests	Nothing proves the 429 / 504 fixes work or keeps them from regressing	sync.py:86-87
16	Every hourly run re-posts every invoice in the export and relies on billing answering 409. The 504 incident shows billing accepted a second copy at least once; the existence check only matches by invoice number	Duplicate invoices remain possible (E07, E09)	sync.py:72-74, sync.py:41-43
17	No stop mechanism ops can use; killing mid-POST leaves the invoice unrecorded	Ops must open a ticket to stop a runaway run	sync.py:80-85; Task Scheduler task