Rpa_Skills/tests/evals/results/handoff/h-api-sync-full-r5.md

Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel #rpa-support

1. What the bot does

Every hour, the bot reads the finance export of approved invoices (data/input/invoices.csv) and sends each invoice to the billing system over its web API. It writes one report file per run in data/output/ with a Status and a Message for every invoice, and adds one line per invoice to logs/sync.log. It has no window or screen of its own and does not drive any desktop app or website. It is a background job that Windows Task Scheduler starts.

2. How it works

Windows Task Scheduler (every hour)
   └─ starts → Invoice sync (background job, no window)
                    │  reads  ← data/input/invoices.csv  (finance export)
                    ├─ sends  → Billing system (web API, over the internet)
                    ├─ writes → data/output/sync_report_<YYYYMMDD_HHMM>.csv
                    ├─ writes → logs/sync.log
                    └─ writes → BOT_WAS_RUN (marker file in the bot folder, "a run started")

• Task Scheduler starts the job once an hour. The sample log shows runs at the top of the hour.
• Invoice sync is the bot. It runs and exits by itself, with nothing on screen.
• The input file is the finance export. The bot only reads it and never moves, renames or marks it.
• The billing system is where invoices get created. The bot never deletes or changes an invoice there.
• The report file is the only place that says why an invoice did not sync.
• sync.log shows when each invoice was handled and its status, plus a "run done" line at the end.
• BOT_WAS_RUN only shows that a run started. Ignore it.

One run, step by step:

1. Write the BOT_WAS_RUN marker and create a new report file named after the current date and minute.
2. Read the input file one row (one invoice) at a time.
3. Check the row: invoice number, customer ID, amount and currency must be filled in, and the amount must be a number above 0. If the row fails a check, it is marked Invalid and the bot goes to the next row.
4. Send the invoice to billing. The bot reads billing's answer and marks the row Synced, Duplicate or Invalid. If billing does not answer or answers with an error, the bot retries (section 8).
5. Write the row's result to the report and sync.log.
6. After the last row, write run done -> sync_report_<time>.csv to sync.log.

Every hour the bot sends the whole input file again. Invoices that already went through come back from billing as Duplicate. This is normal and does not create a second invoice.

3. Before a run

• The folder data/output exists inside the bot folder. If it is missing, every run crashes before doing anything (E10). The bot does not create it.
• The folder logs exists inside the bot folder.
• The settings file (.env) in the bot folder contains the billing address and API key. Dev maintains this file. If the key is missing or wrong, every invoice ends Failed (E09).
• The input file has exactly this header row: invoice_no,customer_id,customer_name,amount,currency.
• Input rules for whoever prepares the export:
  • One invoice per row, with each invoice number used once.
  • Amount is a plain number with no currency sign and no thousands separator.
    • Bad: $1,200.00, 1.200,00, -50
    • Good: 1200.00
  • Currency is filled in, e.g. USD.
  • Customer ID is filled in. The customer name is not checked.
  • Save it as a plain CSV. If every row comes back "missing invoice_no" although the column is filled, the file was saved in a format the bot can't read (E12).
• The bot has no version display. See section 10.

4. During a run

• There is nothing to watch and nothing to click. Don't open, edit or move the input file while a run is going on.
• Don't open the newest report file in Excel while the run is still writing it. Wait for the "run done" line in sync.log.
• To stop the bot: it has no Stop button. End the running "Invoice sync" task in Windows Task Scheduler. If you don't have access to Task Scheduler, ask in #rpa-support. A stopped run leaves a partial report and no "run done" line. That is safe: the next run sends everything again, and invoices that already went through come back Duplicate.
• A run looks stuck when sync.log gets no new line for several minutes. One invoice that billing keeps rejecting can take up to about 6 minutes before the bot gives up on it (section 8). Wait 10 minutes before escalating. A run that is still going when the next hour starts is E13.

5. Where to look when something goes wrong

Question	Where to look
Did the run finish?	logs/sync.log: the run should end with a line run done -> sync_report_<time>.csv. If there is no such line for an hour, the run crashed or is still going (E10, E13).
Which invoices failed?	Newest file in data/output/ (sync_report_<YYYYMMDD_HHMM>.csv). Filter the Status column for anything other than Synced and Duplicate. Also filter Message for "some lines rejected" and "confirmed after retry" (E06, E07).
Why did one invoice fail?	The same report file, Message column on that invoice's row. sync.log shows only the time and status, never the reason.
Did the bot already retry?	Only partly visible. "No success after 5 tries" means it retried 5 times. "Created (confirmed after retry)" means at least one retry happened. A gap of 30+ seconds between two lines in sync.log also means retries. Each retry is not logged.
When was one invoice last handled?	Search logs/sync.log for the invoice number, then open the report file whose time matches that line.
Which version is running?	Not visible to ops. Ask in #rpa-support (section 10).

The report files contain customer names. Keep them inside the output folder, and don't paste their contents into chat or tickets.

6. Statuses

The report file is plain text, so no status has a colour.

Status	Message	Meaning	What ops does
Synced	Created	Billing created the invoice.	Nothing.
Synced	Created (some lines rejected)	Needs follow-up. Billing created the invoice but rejected some of its lines, so the invoice is incomplete.	Check the invoice in billing and follow up (E06).
Synced	Created (confirmed after retry)	Sending failed, but billing then showed an invoice with this number. Usually the bot's own earlier attempt went through. It can also be an older invoice with the same number.	Spot-check the invoice in billing (E07).
Duplicate	Invoice already exists in billing	Billing already has this invoice number. It is usually an invoice an earlier run already sent.	Nothing, unless the number was reused for a different invoice (E05).
Invalid	missing <field names>	A required field is empty on that row.	Fix the input (E01).
Invalid	amount must be positive	The amount is 0 or negative.	Fix the input (E02).
Invalid	amount '<value>' is not a number	The amount has a symbol, separator or text in it.	Fix the input (E03).
Invalid	billing's own reason text, or "Rejected by billing"	Billing refused the invoice's content.	Fix the data as billing says (E04).
Failed	No success after 5 tries	Billing never accepted it after 5 tries over about a minute or more.	See E08. If every row says this, see E09.

7. Exception catalogue

ID	What you see	How the bot identifies it	Where it is recorded	Type	Who acts	What to do	Escalate when
E01	Invalid - "missing customer_id, amount" (or other field names)	Before sending, it checks that invoice number, customer ID, amount and currency are filled in.	Report: Status, Message. Log: invoice number + Invalid.	Business	Ops	Get the export corrected. The next hourly run picks the row up again.	Never, unless the fields are filled in and it still says missing (then see E12).
E02	Invalid - "amount must be positive"	The amount is 0 or below.	Report: Status, Message	Business	Ops	Credit notes and zero invoices are not supported. Get the row removed or corrected in the export.	Finance needs credit notes synced.
E03	Invalid - "amount '…' is not a number"	The amount cannot be read as a plain number.	Report: Status, Message	Business	Ops	Fix the amount format, e.g. 1200.00 instead of $1,200.00.	Never.
E04	Invalid - billing's own reason, or "Rejected by billing"	Billing answers that the invoice content is not acceptable.	Report: Status, Message (billing's text)	Business	Ops	Read the message and fix that data in the export, e.g. an unknown customer or currency.	The message isn't understandable, or the data looks right.
E05	Duplicate - "Invoice already exists in billing"	Billing answers that the invoice number already exists.	Report: Status, Message	Business	Ops (usually nothing)	Normal for invoices sent in an earlier hour. Act only if this invoice number was never synced before (search older reports and sync.log). Then two different invoices share a number, so check in billing.	Billing holds a different invoice under that number.
E06	Synced - "Created (some lines rejected)"	Billing answers "created, but some lines refused".	Report: Message only. The log shows plain "Synced".	Business	Ops	Open the invoice in billing, see which lines are missing and add them or follow up with finance. The bot will not resend it. Every later run shows it as Duplicate.	Billing gives no way to see the rejected lines.
E07	Synced - "Created (confirmed after retry)"	Sending failed, so before retrying the bot asked billing whether this invoice number exists, and billing said yes.	Report: Message	System	Ops	Spot-check in billing that the invoice under this number has the right customer and amount. Usually it is the bot's own earlier attempt. If it is an older, different invoice, this one was not created.	The invoice in billing doesn't match the export.
E08	Failed - "No success after 5 tries"	Billing didn't answer, timed out, said "too many requests", had a server error, or answered with an error the bot doesn't recognise, 5 times in a row.	Report: Status, Message. The log shows "Failed" with no reason.	System	Bot, then Ops	Check billing first for this invoice number before entering anything by hand. The next hourly run tries again if the invoice is still in the export. It then shows Synced, Duplicate (already there) or Failed again.	The same invoice is Failed in 3 runs in a row.
E09	Every invoice in a run is Failed, and the run is much slower than usual	Every send is refused: wrong or missing API key, wrong billing address, or billing down. The bot can't tell these apart and retries each one.	Report: every row Failed. Log: every row Failed.	System	Dev	Nothing to fix on the ops side. Post in #rpa-support right away.	Always.
E10	No new report file, or a report that stops part-way, and no "run done" line in sync.log for that hour	The bot crashed. Known causes: the output folder is missing; the network dropped while the bot was checking billing after a failed send; billing sent an answer the bot couldn't read; the input file is missing or unreadable.	Only the missing "run done" line and the partial report. The crash reason is not written to any file ops can open.	System	Dev	Check that data/output exists, and create it if missing. Otherwise escalate. Invoices after the crash point were not sent.	The output folder exists and the next hourly run also has no "run done".
E11	Every hour the report stops at the same invoice, and invoices after it never get a row	One invoice makes billing send an answer the bot can't read. The bot crashes there each hour (as in E10), so every row after it is blocked.	Partial report, ending just before the same invoice each hour. No "run done".	System	Ops, then Dev	Note the first invoice that is missing from the report. Ask whoever prepares the export to take that row out for now, so the rest can sync.	Always: send the invoice number.
E12	Every row Invalid - "missing invoice_no" although the column is filled	The file was saved with a hidden marker at the start (Excel's "CSV UTF-8" option), so the bot doesn't recognise the first column name.	Report: every row Invalid	Business	Ops, then Dev	Re-save the export as plain "CSV (Comma delimited)" and wait for the next run.	Re-saving doesn't help, or names with accents then make the run crash (E10).
E13	A run is still going when the next hour starts, or two report files with close times overlap	Each invoice that keeps failing can take up to about 6 minutes, so a file with many such invoices runs past the hour. Nothing stops a second run from starting.	sync.log: lines from two runs mixed together. A report file may be overwritten if two runs start in the same minute.	System	Bot, then Dev	Wait for both to finish. Results are still safe because Duplicate protects against double invoices.	It happens 2 hours in a row, or E09 is the cause.

8. What the bot recovers by itself

Problem	What the bot does	Limit	After the limit
Billing doesn't answer within 30 seconds, the network drops, billing says "too many requests", or billing gives any error the bot doesn't recognise	Asks billing whether the invoice already exists. If yes: Synced (E07). If not: waits and sends again.	5 tries. Waits 2, 4, 8, 16 and 32 s between them (about 1 minute), plus up to 30 s per call. Worst case about 6 minutes per invoice.	Row = Failed (E08). The bot goes to the next invoice.
Billing timed out but had actually created the invoice	The existence check above finds it, so no second invoice is created.	Same as above	Synced - "Created (confirmed after retry)"
An invoice failed this hour	Nothing is resent within this run. The next hourly run sends the whole input file again.	Only while the invoice is still in the export	Shows Duplicate once it has gone through
The bot crashed	Nothing. There is no restart, resume or alert. The next run starts on the hour as usual.	-	See E10 / E11

9. Known incidents

Both incidents are recorded only in the version history. Their dates were not recorded, but both were fixed before 0.9.1.

When / version	What ops saw	Cause	What to do now	Ops could handle?
Before 0.9.1	The same invoice appeared twice in billing.	Billing created the invoice but answered "timed out", so the bot sent it again.	Fixed: the bot now checks whether the invoice exists before every retry (E07). Still spot-check "confirmed after retry" rows. Removing a duplicate in billing needs billing or dev access.	Partly
Before 0.9.1	The hourly run died part-way. Invoices after that point got no report row, and there was no "run done" line.	Billing answered "too many requests" and the bot had no retry.	Fixed: up to 5 tries with waits (section 8). A network drop during the existence check can still crash the run the same way (E10).	Partly: ops could see the missing "run done" and wait for the next hour.

10. Releases and version check

• How an update arrives: the bot has no updater. The RPA dev team replaces the files in the bot folder on the machine.
• How to confirm it applied: the bot shows no version anywhere ops can see. Ask dev in #rpa-support to confirm the version (currently 0.9.1) after every update.
• If the version did not change: report it in #rpa-support. Ops can't update the bot themselves.

11. Escalating to dev

Post in #rpa-support (Teams) with:

• the exception ID from section 7 (e.g. E10);
• the date and hour of the run;
• the report file name (e.g. sync_report_20260903_0900.csv), the name only;
• the invoice numbers involved;
• the matching lines from logs/sync.log, or say "no run done line".

Never send: the .env settings file, API keys or passwords. Don't paste or attach report files either, because they contain customer names. Give the file name and dev will open it.

12. Failure-class check

• F1 Missing prerequisite: covered in section 3 (output folder, logs folder, settings file), E09 and E10, and Fix #2 and #9.
• F2 not applicable: the bot drives no windows. It calls the billing web API with no screen, mouse or keyboard.
• F3 Target slower than expected: covered in E08 and E13, section 8 (30 s wait, 5 tries) and the section 4 "stuck" rule.
• F4 Input read differently: covered in the section 3 input rules (amount format, header, CSV type) and E01-E03 and E12, and Fix #10.
• F5 Marked done when nothing was done: covered in E07 ("confirmed after retry" can be an older invoice) and Fix #7.
• F6 Action happened but not confirmed: covered in E07 and E08 ("check billing first"), and incident 1.
• F7 Business exception shown as success: covered in Statuses and E06 (partly rejected invoice shown as Synced), and Fix #6.
• F8 Endless re-runs / watchdog: there is no watchdog or auto-rerun. The hourly re-send of the whole file is covered in sections 2 and 8, and the overlap risk in E13 and Fix #11.
• F9 Stopping the wrong way: covered in section 4 (end the Task Scheduler task; a partial report is safe).
• F10 One item blocks the queue: covered in E11 (one unreadable billing answer crashes the run at the same invoice every hour), and Fix #3 and #4.
• F11 Cause hidden from ops: covered in sections 5 and E10. Crash and network messages are not saved to any file, and the log has no reason. See Fix #5 and #12.
• F12 Update didn't arrive: covered in section 10 (no version visible), and Fix #13.

13. Fix before handover (dev team)

#	Gap	Why it hurts ops	Where (file:line)
1	.env (API key) is committed to git	The key is exposed to anyone with repo access. Rotate the key and untrack the file.	.env (tracked); loaded at sync.py:17-19
2	data/output/ is not in the repo and not created by the code	A fresh install crashes every run before processing anything (E10)	sync.py:68-69
3	exists() has no error handling and is called right after network errors	A network drop during the duplicate check crashes the whole run (E10). This is the 429 incident again by another route.	sync.py:41-43, sync.py:60
4	r.json() on a 422 with a non-JSON body is unguarded	One bad billing answer crashes the run, at the same row every hour (E11)	sync.py:59
5	Network errors and CRASH: go to print() only	The cause of a crash is lost under Task Scheduler. Ops only sees a missing "run done".	sync.py:52, sync.py:84-85
6	HTTP 207 (some lines rejected) is stored as Synced	Ops treats an incomplete invoice as done (E06)	sync.py:54-55
7	"Created (confirmed after retry)" only checks that the invoice number exists, and also fires after non-retryable errors such as 400	A pre-existing or different invoice is reported as Synced, and this one was never created (E07)	sync.py:60-61
8	400/401/403/404 are retried 5 times like temporary errors	A bad key makes every row wait about 1 min and then Fail. This drives runs past the hour (E09, E13).	sync.py:60-62
9	A missing API key silently sends an empty token. A missing URL silently falls back to a built-in default address.	Misconfiguration looks like E09 instead of a clear startup error	sync.py:18-19
10	Input is read as utf-8, not utf-8-sig	An Excel "CSV UTF-8" export makes every row "missing invoice_no" (E12). A cp1252 export with accents crashes.	sync.py:69
11	No guard against overlapping runs, and the report name only goes down to the minute	Two runs can overlap and overwrite each other's report (E13)	sync.py:68
12	sync.log holds only invoice number + status, with no message, HTTP code or attempt count	Ops can't answer "did it retry / why" from the log. "Synced" hides 207 rows.	sync.py:75
13	No version anywhere ops can see. A stray commit message in the README and two in the code.	Ops can't confirm an update (section 10)	README.md:5-6, sync.py:86-87
14	The Task Scheduler task is not in the repo (schedule, "do not start a new instance", stdout capture)	No one can check how the bot is actually started or stopped	sync.py:3
15	No alert on Failed rows or a crash	Problems are found only if someone opens the report or log	sync.py:63, sync.py:80-85
16	Sleeps 32 s after the 5th try before giving up	Adds 32 s to every Failed invoice for nothing	sync.py:62-63
17	BOT_WAS_RUN marker is undocumented	Unclear purpose. Ops may delete it or worry about it.	sync.py:67
18	No regression tests for the 429 and 504 fixes. Both fix commits only add a comment.	The incident fixes can regress unnoticed, and history doesn't show what changed	sync.py:46-63