Rpa_Skills/tests/evals/results/handoff/h-api-sync-full-r1.md

Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel #rpa-support

1. What the bot does

Every hour, Windows Task Scheduler starts the bot. It reads the finance export file data/input/invoices.csv, which has one approved invoice per row. It checks each row, then sends the invoice to the billing system over the internet.

For every run it writes:

• a report data/output/sync_report_<YYYYMMDD_HHMM>.csv, with one line per invoice and a Status and Message for each;
• a short line per invoice in logs/sync.log.

The bot has no window and drives no desktop app. It talks to the billing system directly, so you can use the machine while it runs.

2. How it works

Windows Task Scheduler (every hour)
   └─ starts → Invoice sync (no window) ──sends invoices──> Billing system (online)
                    │
                    ├─ reads  → data/input/invoices.csv
                    ├─ writes → data/output/sync_report_<YYYYMMDD_HHMM>.csv
                    └─ writes → logs/sync.log

• Task Scheduler starts the bot once an hour. Nothing else starts it.
• Invoice sync is a background program with no screen. It works through the input file from top to bottom, one invoice at a time.
• Billing system receives each invoice. It can accept it, reject it, say it already has it, or not answer.
• invoices.csv is the finance export. The bot never moves, renames or empties it, so every hour it sends the whole file again.
• sync_report_….csv is the only place where the reason for each result is written.
• sync.log keeps the invoice number and status for every run, plus a "run done" line when a run finishes.

One run, step by step:

1. Create a new report file named after the current date and time.
2. For each row of the input file:
   • Check the row: required fields filled in, and the amount is a number above 0. If the check fails, the row is Invalid and the bot moves to the next row.
   • Send the invoice to billing.
   • If billing does not answer, or answers with a temporary error, check whether billing already has the invoice. If it does, the row is Synced. If not, wait and try again, up to 5 tries in total.
   • Write the result to the report and the log.
3. Write run done -> sync_report_<time>.csv to the log.

3. Before a run

• The bot folder contains these three items. Without them the bot stops at the very start and writes no report.
  • The folder data/output/. The bot does not create it.
  • The folder logs/.
  • The settings file .env.
• The settings file has both the billing address and the billing key filled in. If the key is missing or expired, every invoice takes about 1 minute and then ends as Failed (E08). Nothing on screen says the key is the problem.
• The input file is at data/input/invoices.csv.
• The first line of the input file is exactly: invoice_no,customer_id,customer_name,amount,currency. If a heading is renamed, every row comes back Invalid - missing … (E01, E09).
• One invoice per row. Each row has an invoice number, a customer ID, an amount and a currency.
• Amounts are plain numbers above 0.
  • Good: 1200.50
  • Bad: 1,200.50 or $1200 (Invalid - "is not a number"), 0 or -50 (Invalid - "must be positive").
• Only one machine runs this bot. Two machines would send the same invoices twice every hour.
• The bot does not show its version anywhere. Confirm with dev that 0.9.1 is installed (section 10).

4. During a run

• You can use the machine. The bot has no window and clicks nothing.
• Don't replace or edit invoices.csv while a run is in progress. Wait until the log shows the run done line for that hour.
• You can open a report file to read it. Don't save changes to the report of a run that is still in progress.
• To stop the bot: in Windows Task Scheduler, right-click the invoice sync task and choose End. To stop the hourly runs, choose Disable. Ops access to Task Scheduler is not confirmed, so ask #rpa-support if you can't do it.
  • If you stop it mid-run, the report for that run is incomplete, and the invoice it was sending at that moment may or may not have reached billing. The next run reports that invoice as Duplicate if it did.
• A run looks stuck when logs/sync.log gets no new line for more than 6 minutes. One invoice can take up to about 6 minutes when billing does not answer at all.
  • There is no watchdog: the bot never gives up on a run by itself.
  • If many invoices fail, one run can take longer than an hour and overlap the next hourly run (E11).

5. Where to look when something goes wrong

Question	Where to look
Did the run finish?	logs/sync.log. Look for run done -> sync_report_<time>.csv for that hour. If the line is missing, the run stopped partway (E10).
Did the run start at all?	data/output/. There should be one sync_report_<YYYYMMDD_HHMM>.csv per hour. A missing hour means the bot did not start or stopped at the very start (E10).
Which invoices failed?	Open that hour's report. Look at every row where Status is Invalid or Failed, and every Synced row whose Message is "Created (some lines rejected)".
Why did one invoice fail?	The Message column of the report. The log only shows the status, never the reason.
What happened to one invoice over several runs?	Search logs/sync.log for the invoice number. Each hit gives the run time, which points you to the matching report file.
Did the bot already retry?	Not recorded directly. "No success after 5 tries" means it tried 5 times. "Created (confirmed after retry)" means the first try failed and billing turned out to have the invoice. Other results came from the first try.
Why did the whole run stop?	Not visible to ops. The reason goes only to a console window nobody sees. Send it to dev (E10).
Which version is running?	Not shown anywhere ops can see. Ask dev (section 10).

6. Statuses

The report is a plain CSV file, so statuses have no colour. Read the Status and Message columns together.

Status	Message	Meaning	What ops does
Synced	Created	Billing accepted the invoice	nothing
Synced	Created (some lines rejected)	Needs follow-up. Billing created the invoice but dropped some of its lines.	Check the invoice in billing and fix the missing lines (E06)
Synced	Created (confirmed after retry)	The first send failed or got no answer. Billing turned out to have the invoice, so the bot did not send it again.	Usually nothing. Spot-check the amount in billing (E07).
Duplicate	Invoice already exists in billing	Billing already has this invoice number	Normally nothing: the same file is re-sent every hour (E05)
Invalid	missing <field names>	A required field is empty in the input	Fix the input row (E01)
Invalid	amount must be positive	Amount is 0 or below	Fix the input row (E02)
Invalid	amount '<value>' is not a number	The amount has commas, a currency sign, text, etc.	Fix the input row (E03)
Invalid	billing's own explanation, or "Rejected by billing"	Billing refused the invoice and gave a reason	Act on the reason (E04)
Failed	No success after 5 tries	Billing never accepted it in 5 tries, and the bot could not find the invoice in billing	Check billing first, then see E08

7. Exception catalogue

ID	What you see	How the bot identifies it	Where it is recorded	Type	Who acts	What to do	Escalate when
E01	Invalid - "missing invoice_no" (or customer_id, amount, currency)	Before sending, it checks that these four fields are not empty	Report: Status and Message. Log: status only.	Business	Ops	Fill in the field in the finance export. The next hourly run picks the row up.	The field is filled in but still reported missing (see E09)
E02	Invalid - "amount must be positive"	The amount is 0 or below	Report	Business	Ops	Correct the amount, or remove the row if it is a credit note. The bot does not send zero or negative invoices.	Credit notes need to go through. That is a change request to dev.
E03	Invalid - "amount '1,200.50' is not a number"	It cannot read the amount as a number	Report	Business	Ops	Write the amount as a plain number with a dot for decimals, e.g. 1200.50. No thousands separators, currency signs or spaces.	The amount already looks plain
E04	Invalid - billing's message (e.g. unknown customer), or "Rejected by billing"	Billing refuses the invoice and explains why	Report: Message holds billing's reason	Business	Ops	Fix the data that billing's message names, usually in the finance export or in billing's customer records.	The message is "Rejected by billing" with no reason, or you can't tell what to fix. Send the invoice number, report file name and message.
E05	Duplicate - "Invoice already exists in billing"	Billing says it already has this invoice number	Report	Business	Ops	Check earlier reports (search the log for the invoice number). If an earlier run shows Synced, this is normal: the bot re-sends the whole file every hour. If no run ever synced it, check in billing that the existing invoice is the right one, because the number may be used twice.	Two different invoices share a number
E06	Synced - "Created (some lines rejected)"	Billing creates the invoice but rejects part of it	Report: Message. The Status still says Synced.	Business	Ops	Open the invoice in billing, find the missing lines and add or correct them by hand. The bot will not resend them: re-runs only get Duplicate.	Lines are rejected for many invoices in the same run
E07	Synced - "Created (confirmed after retry)"	The first send failed or got no answer. Before retrying, it asked billing whether the invoice exists and got yes.	Report	System	Bot	Nothing usually. The bot cannot tell whether it created the invoice this run or the invoice already existed, so spot-check the amount in billing now and then.	The invoice in billing has the wrong amount or customer
E08	Failed - "No success after 5 tries"	5 sends, with waits of 2, 4, 8, 16 and 32 s between them. After each failed send it checks whether billing already has the invoice.	Report. The real cause (billing down, busy, key expired, wrong address) is not recorded anywhere.	System	Ops, then Dev	1. Search billing for the invoice number first. It may have been created even though no confirmation came back. 2. If it isn't there, do nothing: the next hourly run tries again while the row is still in the input file. 3. If it is there, nothing more is needed. The next run will show Duplicate.	Every row in a run is Failed (likely the billing key or address). The same invoice is Failed in 2 runs in a row. Send the report file name, invoice numbers and run time.
E09	Every row Invalid - "missing invoice_no", although the file looks fine	The first heading isn't read as invoice_no. Some Excel "CSV UTF-8" saves add an invisible mark before the first heading.	Report	Business	Ops, then Dev	Check the first line spelling exactly (section 3). If it is exact, re-export the file once.	It still happens. Send the file name and time, not the file itself.
E10	No report for an hour, a report with fewer rows than the input, or no run done line in the log	The bot stops the whole run. Known causes: data/output/ or logs/ is missing, the input file is missing, billing sends back an unreadable rejection, or billing can't be reached while the bot checks for a duplicate.	Nothing ops can see. The reason goes only to a console window. The report stops at the last finished invoice.	System	Ops, then Dev	1. Check that data/output/, logs/ and data/input/invoices.csv exist. 2. Wait for the next hourly run. It starts again from the top, and already-sent invoices come back Duplicate.	The next run stops too, especially if it stops right after the same invoice. Send the run times, the last invoice number in the partial report, and logs/sync.log.
E11	A run takes over an hour, two runs' lines are mixed in the log, or a report is overwritten	Many Failed invoices, each taking 1-6 minutes. The bot has no lock against two runs at once.	logs/sync.log (timestamps); data/output/	System	Dev	Treat it as E08 for the invoices. Tell dev runs are overlapping.	Always. Send the log and the times.

8. What the bot recovers by itself

Problem	What the bot does	Limit	After the limit
Billing doesn't answer, is busy, or returns a temporary error	Checks whether billing already has the invoice. If not, waits and sends again.	5 tries, waits of 2+4+8+16+32 s. About 1 minute per invoice; up to about 6 minutes if billing doesn't answer at all (30 s wait per call).	Status Failed, moves to the next invoice (E08)
Billing created the invoice but the confirmation got lost	The existence check above finds it and does not send it again	Same as above	Synced - "Created (confirmed after retry)" (E07)
An invoice was not synced this hour	The next hourly run sends it again, because the input file is never emptied	As long as the row stays in the finance export	If finance removes the row, nobody retries it
The whole run stopped	No recovery inside the run	-	The next hourly run starts again from row 1 (E10)
Invalid input row	No retry	-	Stays Invalid every hour until the input is fixed

9. Known incidents

The history records no dates or versions for these incidents. Both were fixed before 0.9.1.

When / version	What ops saw	Cause	What to do now	Ops could handle?
before 0.9.1	The same invoice appeared twice in billing	Billing timed out but had created the invoice; the bot sent it again	Fixed: the bot now checks billing before every retry. Still search billing before re-processing any Failed invoice (E08). Duplicates are less likely but still possible when billing is slow to show a new invoice.	Partly
before 0.9.1	The hourly run stopped partway, report incomplete	Billing's "too many requests" limit was not handled	Fixed: up to 5 tries with waits. If a run still stops, follow E10.	Partly

10. Releases and version check

• How an update arrives: there is no launcher or auto-update. A developer replaces the bot files in the bot folder by hand.
• How to confirm it applied: the bot does not show or log its version. Ask in #rpa-support for the version installed after each update. The current version is 0.9.1.
• If the version did not change: report it in #rpa-support. Don't copy files yourself.
• Only one machine may run the bot (section 3).

11. Escalating to dev

Post in #rpa-support with:

• the exception ID from section 7;
• the invoice number(s);
• the report file name, e.g. sync_report_20261023_1400.csv, and the run time;
• logs/sync.log, which holds invoice numbers and statuses only.

Never send:

• the .env settings file or the billing key;
• rows copied out of the report or the input file. They contain customer names, so give the file name and invoice numbers instead.

12. Failure-class check

• F1 (missing prerequisite) covered in section 3 and E10. The missing output folder is also Fix #1.
• F2 not applicable: the bot has no window and drives no desktop app. It only sends data to billing online.
• F3 (target slower than expected) covered in E08 and section 8: 30 s wait per call, 5 tries. Fix #9 covers errors that are retried pointlessly.
• F4 (input read differently) covered in section 3 input rules and E01-E03, E09. Also Fix #12 and #15.
• F5 (marked done when nothing was done) covered in E07. "Confirmed after retry" can't tell whether this run created the invoice. Fix #6.
• F6 (action happened but not confirmed) covered in E08: search billing before re-processing. Incident 1. Fix #7.
• F7 (business exception shown as success) covered in section 6 and E06: partially rejected invoices show Synced. Fix #5.
• F8 (endless re-runs / watchdog) covered in section 4 (no watchdog), section 8, E05 (hourly Duplicates) and E11 (overlapping runs). Fix #10.
• F9 (stopping the wrong way) covered in section 4: End the task in Task Scheduler. Fix #13 covers the unconfirmed ops access.
• F10 (one stuck item blocks the queue) covered in E10. One bad invoice can stop the whole run and, if it stays in the file, every hourly run after it. Fix #4 and #7.
• F11 (cause hidden from ops) covered in section 5 and E08, E10. Fix #2 and #3.
• F12 (update didn't arrive) covered in section 10. Fix #11.

13. Fix before handover (dev team)

#	Gap	Why it hurts ops	Where (file:line)
1	data/output/ is not in the repo and is never created. open(report, "w") raises FileNotFoundError.	On a fresh machine every run stops before row 1, and no report or log line is written	sync.py:68-69
2	Network errors and crashes go to print() (stdout) only, never to sync.log	Ops can't see why a run stopped (E10). Task Scheduler discards stdout.	sync.py:52, sync.py:84
3	Failed message is fixed text, and neither the last HTTP code nor the error is kept. The log line is status only.	An expired key, a billing outage and a rate limit all look the same (E08)	sync.py:63, sync.py:75
4	A non-JSON 422 body makes r.json() raise, which crashes the run	One bad invoice stops the whole run every hour while it stays in the file	sync.py:59
5	HTTP 207 (partial) is stored as Status Synced	Ops filtering on Status miss rejected lines (F7)	sync.py:54-55
6	exists() returning 200 is reported as "Created (confirmed after retry)" even if the invoice existed before this run, e.g. after a 400	Report claims a creation that may not have happened (F5)	sync.py:60-61
7	exists() has no try/except, and any non-200 counts as "not found"	A network outage crashes the run instead of retrying. A 429/5xx on the check leads to a re-POST, so duplicates are still possible (F6).	sync.py:41-43, sync.py:60
8	Missing BILLING_API_KEY sends an empty Bearer token silently. Missing BILLING_API_URL silently falls back to billing.example.test.	Bad settings look like ordinary Failed rows, about 1 minute each	sync.py:18-19
9	400/401/403/404 are retried 5 times, Retry-After is ignored, and the bot sleeps 32 s after the last attempt	Slow runs, wasted rate-limit budget, overlapping hourly runs (E11)	sync.py:60-62
10	Input is never archived. There is no lock against overlapping runs. Report name has minute precision and is opened with "w".	Duplicate noise every hour hides real duplicates. Overlapping runs can overwrite a report (E11).	sync.py:68, sync.py:72
11	Version exists only in README. It is not logged or shown, and there is a stray line "chore: bump to 0.9.1" in the README.	Ops can't confirm an update applied (F12)	README.md:5-6
12	Input opened as utf-8, not utf-8-sig	An Excel BOM makes every row "missing invoice_no" (E09)	sync.py:69
13	The Task Scheduler task definition is not in the repo, and ops access to Task Scheduler is not confirmed	Ops can't stop the bot or see run history (F9)	n/a: add the task export and agree access
14	Report copies customer_name into the output folder ops open	Customer personal data is spread to every report file	sync.py:71, sync.py:76
15	nan / inf amounts pass validation	Bad amounts reach billing	sync.py:33-37
16	Retries are not logged	Ops can't tell whether, or how often, the bot retried	sync.py:48-62
17	The two "fix" commits only add comment lines. The retry and check-before-retry code came with the initial import, and there are no regression tests.	Fixes can't be traced or protected from regressing	sync.py:86-87
18	A BOT_WAS_RUN marker is written to the project root every run and never read	Clutter. It shows whether the bot ran, not whether it finished.	sync.py:67