Rpa_Skills/tests/evals/results/handoff/h-api-sync-full-r3.md

Invoice sync - ops handover

Version 0.9.1 · Written 2026-10-23 · Escalation: RPA dev team, Teams channel #rpa-support

1. What the bot does

Every hour it takes the approved invoices in the finance export file (data/input/invoices.csv) and creates them in the billing system through billing's web API. It checks each invoice first: required fields filled in and a valid, positive amount. It then sends the invoice to billing. If billing is busy or slow, it retries. For each run it writes one report file in data/output/, listing every invoice with a Status and a Message, and adds a few lines to logs/sync.log. It has no window or screen of its own and does not drive any desktop app or browser. It talks to billing directly over the network.

2. How it works

Windows Task Scheduler (every hour)
   └─ starts → Invoice sync script ──sends invoices──> Billing system (web API)
                    │
                    ├─ reads  → data/input/invoices.csv
                    ├─ writes → data/output/sync_report_<YYYYMMDD_HHMM>.csv
                    └─ writes → logs/sync.log

• Task Scheduler starts the bot every hour on the bot machine. Nobody needs to click anything.
• The Invoice sync script is the bot. It runs with no window, works through the input file from top to bottom, then stops.
• The billing system is where invoices are created. The bot only creates invoices and checks whether they exist. It never edits or deletes them.
• The input file is the finance export. The bot reads it and never changes it. If the file is not replaced, the next run sends every row again (see Duplicate in section 6).
• The report file is one new file per run with columns invoice_no, customer, Status and Message. This is the main place to look.
• The log file gets one line per invoice (invoice number and status) and a closing run done line when the run finishes.

One run, step by step:

1. Open the input file and create a new report file for this run.
2. For each row, check that invoice number, customer ID, amount and currency are filled in, and that the amount is a positive number. If not, the row is marked Invalid and nothing is sent to billing.
3. Send the invoice to billing.
4. If billing is busy, slow or unreachable, check whether the invoice was created anyway, wait, and try again (up to 5 tries; section 8).
5. Write the outcome to the report and the log, then move to the next row.
6. Write run done -> sync_report_<time>.csv to the log.

3. Before a run

• The folder data/output/ exists next to the bot. The bot does not create it. If it is missing, the run stops before processing any invoice (E09).
• The settings file .env is in the bot folder and contains the billing address and the billing access key. Ops never opens or edits this file. If it is missing or the key has expired, every invoice ends up Failed (E08).
• The input file data/input/invoices.csv is present and has the columns invoice_no, customer_id, customer_name, amount, currency.
• Input rules, one row per invoice:
  • Invoice number, customer ID, amount and currency must all be filled in. A blank in any of them gives Invalid - missing ....
  • Amount must be a plain positive number. Use a dot for decimals and no thousands separator or currency symbol.
    • Bad: 1,250.00, $120, -50, 0
    • Good: 1250.00, 120
  • The bot cannot send credit notes or zero-value invoices, because the amount must be above 0.
• Any version change is confirmed by the dev team (section 10). The bot does not show its version anywhere ops can see.

4. During a run

• There is nothing to watch. A normal run takes seconds to a few minutes.
• Don't edit, rename or move the input file while a run is in progress. Replace it between runs.
• Don't open the newest report file in Excel while the run is still writing it. Wait for the run done line in the log.
• To stop the bot: in Task Scheduler on the bot machine, right-click the task, choose End, and then Disable it if it must not start again next hour. If you don't have access to Task Scheduler, ask in #rpa-support. Stopping halfway is safe: the next run re-sends the file, and invoices already in billing come back as Duplicate.
• A run looks stuck when the newest report file stops getting new rows and there is no run done line in the log. The bot has no stuck-detection of its own. Each invoice can take up to about 6 minutes when billing is not responding (section 8), so wait 10 minutes. If a run is still going when the next hourly run starts, escalate (E10).

5. Where to look when something goes wrong

Question	Where to look
Did the run finish?	logs/sync.log: a line run done -> sync_report_<time>.csv for that hour. No such line means the run stopped partway (E09). "run done" only means the run reached the end, not that the invoices were created.
Which invoices failed?	The newest data/output/sync_report_<YYYYMMDD_HHMM>.csv. Filter Status for Failed and Invalid, and filter Message for some lines rejected.
Why did one invoice fail?	The Message column of the same report. The log shows only the status, never the reason.
Did the bot already retry?	Failed always means it tried 5 times. Synced with "Created (confirmed after retry)" means a retry found the invoice already in billing. Nothing else records the retries ops can see.
When did it fail?	logs/sync.log: search for the invoice number. Each line starts with date and time.
Why did the whole run stop?	Ops cannot see this. The reason is not written to any file ops can open. Escalate (E09).
Which version is running?	Ops cannot see this. Ask in #rpa-support.

6. Statuses

The report is a plain CSV file, so there are no colours. The Status column holds one of four values. Read it together with the Message column.

Status	Message	Meaning	What ops does
Synced	Created	Invoice created in billing	nothing
Synced	Created (some lines rejected)	Invoice created, but billing refused some of its lines. Needs follow-up.	See E05
Synced	Created (confirmed after retry)	Billing was slow or busy, but on a retry the bot found the invoice in billing	nothing
Duplicate	Invoice already exists in billing	Billing already has this invoice number. Normal for rows that were sent in an earlier run.	See E06
Invalid	missing field names	A required field is empty in the input. Not sent to billing.	See E01
Invalid	amount must be positive	Amount is 0 or negative. Not sent to billing.	See E02
Invalid	amount 'value' is not a number	Amount is not a plain number. Not sent to billing.	See E03
Invalid	Rejected by billing, or billing's own explanation text	Billing looked at the invoice and refused it	See E04
Failed	No success after 5 tries	Billing never confirmed it after 5 tries. It may still have been created.	See E07 and E08

7. Exception catalogue

ID	What you see	How the bot identifies it	Where it is recorded	Type	Who acts	What to do	Escalate when
E01	Invalid - "missing customer_id" (or other field names)	Before sending, it checks that invoice number, customer ID, amount and currency are not blank	Report: Status, Message. Log: invoice number + Invalid	Business	Ops	Ask finance to fill the field in the export. The next hourly run picks the row up again.	The field is filled in the file and it still says missing. Send the invoice number and report file name.
E02	Invalid - "amount must be positive"	The amount is 0 or less	Report: Status, Message	Business	Ops	Check with finance. The bot cannot send zero or negative amounts (credit notes); these must be entered in billing by hand.	Finance says this kind of invoice must be synced automatically.
E03	Invalid - "amount '1,250.00' is not a number"	The amount cannot be read as a number (comma, currency symbol, text)	Report: Status, Message	Business	Ops	Ask finance to export the amount as a plain number, e.g. 1250.00.	The amount looks plain and is still rejected.
E04	Invalid - "Rejected by billing" or another explanation text	Billing replied that the invoice data is not acceptable	Report: Status, Message (billing's own text)	Business	Ops	Read the message (e.g. unknown customer, wrong currency) and get the data fixed at the source. Retrying unchanged data gives the same result.	The message is unclear, or the data looks right. Send the invoice number, run time and the exact message.
E05	Synced - "Created (some lines rejected)"	Billing replied that it created the invoice but refused some lines	Report: Status, Message. The log shows only "Synced".	Business	Ops	This is not done. Open the invoice in billing (or ask the billing team), find the missing lines, and get them added by hand or fixed by finance. The bot will not resend them, because the invoice now exists.	You cannot tell which lines are missing. Send the invoice number and run time.
E06	Duplicate - "Invoice already exists in billing"	Billing replied that this invoice number already exists	Report: Status, Message	Business	Ops (usually nothing)	If the invoice was sent in an earlier run, do nothing. This is normal when the input file is not replaced. If this is the first time you see this invoice number, check in billing that the existing invoice is the same one (same customer and amount).	The existing invoice in billing is a different one. Send the invoice number and run time.
E07	Failed - "No success after 5 tries" on one or a few rows	Billing was busy, slow (no reply within 30 s), down, or gave an unexpected reply on all 5 tries. Before each retry it checked whether the invoice existed and did not find it.	Report: Status, Message. Log: invoice number + Failed	System	Bot, then Ops	First check in billing whether the invoice was created anyway. If it is there, nothing more is needed; the next run will show Duplicate. If it is not there and the row is still in the input file, the next hourly run tries again.	The same invoice is Failed in 3 runs in a row. Send the invoice number, run times and the report file names.
E08	Failed on all or almost all rows, and the run takes much longer than usual	Same as E07, but for every row. This usually means the billing access key is missing or expired, the billing address is wrong, or billing is down.	Report: every Status Failed. Log: long gaps between lines.	System	Dev	Nothing ops can fix. Don't change the input file.	Always. Send the run time, the report file name and the logs/sync.log file.
E09	No run done line for that hour; the report is missing, empty, or stops partway	The run stopped early. Known causes: data/output/ folder missing, input file missing, network failure while the bot was checking whether an invoice exists, or an unexpected reply from billing.	logs/sync.log (the closing line is missing). The cause is not in any file ops can open.	System	Dev	Check that data/output/ and the input file exist (section 3). Rows written to the report before the stop are real results. Rows after it were not processed; the next hourly run re-sends them.	It happens in 2 runs in a row, or the folders are present. Send the run time, the last report file name and logs/sync.log.
E10	Two report files close together for the same hour, or a run still going when the next one starts	A run took more than an hour (many slow or failed rows). Nothing stops two runs at once.	Report file names and log times	System	Dev	Check first in billing for any invoice that shows Failed, as in E07.	Always. Send both report file names and the log.

8. What the bot recovers by itself

Problem	What the bot does	Limit	After the limit
Billing busy (too many requests), down, slow (no reply in 30 s), or an unexpected reply for one invoice	Checks whether the invoice was created anyway. If not, waits and tries again. Waits grow: 2, 4, 8, 16, 32 s.	5 tries per invoice. About 1 minute of waiting, and up to about 6 minutes per invoice if every call times out.	Invoice marked Failed (E07), next invoice
The invoice was created but billing's reply got lost	The check before each retry finds it	Same 5 tries	Synced - "Created (confirmed after retry)"
Invoice sent again in a later run	Billing answers "already exists"	none	Duplicate (E06). Nothing is created twice.
Run stopped partway, or Failed / not-yet-sent rows	No restart within the hour. Task Scheduler's next hourly run starts fresh and re-sends the whole input file.	Once per hour	If the same problem repeats over 2-3 runs, escalate (E07, E09)

Before acting on a Failed row, wait for at least one more hourly run. The bot will usually have retried it already.

9. Known incidents

Exact dates are not recorded. Both incidents happened before version 0.9.1.

When / version	What ops saw	Cause	What to do now	Ops could handle?
Before 0.9.1	The same invoice appeared twice in billing. Finance or billing noticed it; the bot reported nothing.	Billing created the invoice but timed out before replying, and the bot's retry sent it again	Fixed in 0.9.1: the bot checks billing before each retry. The check can still miss when billing is down, so for any Failed row, check billing before re-sending it by hand (E07).	Partly
Before 0.9.1	Hourly run died partway through when billing said "too many requests". Invoices after that row were not sent.	The bot treated "too many requests" as fatal	Fixed in 0.9.1: up to 5 tries with waits (section 8). A run can still stop for other reasons (E09).	Partly

10. Releases and version check

• How an update arrives: the dev team copies the new version into the bot folder on the bot machine. There is no launcher or auto-update. The next hourly run uses the new code.
• How to confirm it applied: ops cannot see the version. The bot writes it to no file ops can open. Ask the dev team to confirm in #rpa-support after every update.
• If the version did not change: report it in #rpa-support. Don't copy files yourself.

11. Escalating to dev

Post in the Teams channel #rpa-support (RPA dev team). Send:

• the exception ID from section 7 (e.g. E08)
• the invoice number(s)
• the run time (from logs/sync.log)
• the report file name (sync_report_<YYYYMMDD_HHMM>.csv)
• the logs/sync.log file

Never send: the .env settings file or anything from it, the billing access key, or customer names and amounts pasted into chat. Refer to rows by invoice number.

12. Failure-class check

• F1 covered in section 3 (output folder, settings file, input file) and E08, E09.
• F2 not applicable: the bot has no windows. It talks to billing over its web API and drives no desktop app.
• F3 covered in E07 and section 8 (30 s reply limit, 5 tries).
• F4 covered in section 3 input rules (bad and good amount examples) and E01-E04.
• F5 covered in section 5 ("run done" is not success) and E08. Fix #4 and #5 in section 13.
• F6 covered in E07 (check billing before re-sending) and section 9. Fix #5 in section 13.
• F7 covered in section 6 and E05 ("some lines rejected" shows as Synced). Fix #1 in section 13.
• F8 covered in E10 and section 8. There is no automatic re-run loop; the only repeat is the hourly schedule. Fix #8 in section 13.
• F9 covered in section 4 (End the task in Task Scheduler).
• F10 not applicable: every invoice is handled on its own. No batch waits for all rows, and nothing locks the next run. A slow row only delays the run (see F8).
• F11 covered by fixes #2, #3 and #10 in section 13. The reason a run stopped and network errors are not written to any file ops can open, and the log has no reason text.
• F12 covered in section 10 and fix #9 in section 13. The version is not visible to ops.

13. Fix before handover (dev team)

#	Gap	Why it hurts ops	Where (file:line)
1	HTTP 207 (partial) is written as Status Synced	Ops filtering on Status thinks the invoice is complete when lines are missing	sync.py:54-55
2	Crash reason (CRASH: ...) and network errors are only printed to stdout, never logged	Ops cannot find why a run stopped (E09). Lost unless Task Scheduler captures stdout.	sync.py:52, sync.py:84
3	Log line has status only, no Message	Ops must open the report to learn why	sync.py:75
4	data/output/ is not in the repo and never created	Fresh machine: the first run crashes before any invoice	sync.py:68-69
5	exists() is not in a try block, and any non-200 counts as "not there"	A network error during the check crashes the run. During an outage it can re-POST an invoice that was already created (possible duplicate).	sync.py:41-43, sync.py:60
6	A 422 with a non-JSON body makes r.json() raise	Whole run crashes on one bad reply	sync.py:59
7	Every non-success code is retried (400/401/403/404 too, not only 429/5xx as the docstring says), and Retry-After is ignored	A bad or expired key shows as generic Failed on every row and takes about 1-6 min per row (E08)	sync.py:47, sync.py:54-62
8	No lock against overlapping runs. The report name has minute resolution, so a same-minute run overwrites the report. It also sleeps 32 s after the last try for nothing.	Slow runs overlap the next hour (E10), and reports can be lost	sync.py:62, sync.py:68
9	Version only in README.md (plus a stray "chore: bump to 0.9.1" line). Not logged or shown anywhere.	Ops cannot confirm an update applied	README.md:5-6
10	Invalid is used both for local checks and for billing's 422 rejection	The status alone doesn't say whether to fix the file or ask billing	sync.py:59, sync.py:74
11	The report stores customer names. No retention policy, and logs/sync.log is never rotated.	Sensitive data piles up in data/output/. The log grows forever.	sync.py:24, sync.py:71, sync.py:76
12	A BOT_WAS_RUN marker file is written every run and never read	Unclear purpose. Confirm or remove.	sync.py:67
13	Task Scheduler task definition (and whether stdout is captured) is not in the repo	Can't rebuild the schedule on a new machine	sync.py:3
14	No tests cover the two incident fixes (429 retry, check-before-retry)	Regressions would come back unnoticed	whole repo