Skills/Rpa_Skills
8
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add rpa-bot-handoff skill: ops guide for bot handover

Browse Source 
The skill scans a bot with subagents (never runs it) and writes
HANDOVER.md, an 11-section ops guide in the operator's order: where is
what, how to run, how long to wait, after the run, how to see an
exception, exception list, how you will know, how to tell us, don'ts,
issues seen before. Dev material (check against 12 failure classes from
Tebra Auto Post, gaps with file:line, incidents) goes to HANDOVER-DEV.md.

Also adds ADR-0004, glossary terms, five fixture bots and a handoff eval
runner (--resume, --regrade) with results from 47 runs of the earlier
13-section version. The grader still expects that structure and must be
updated before the next eval run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
VasanthandClaude Opus 5.5 committed 2026-10-28 15:01:42 +05:30
1 parent 3d2db6aca6
commit 3942fad6cd
143 files changed
+11815 -2
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/evals/results/handoff/h-web-portal-full-r2.md
+204
View File
@@ -0,0 +1,204 @@
# Payer portal downloader - ops handover
Version: not set (the bot has no version number, see section 10) · Written 2026-10-23 · Escalation: RPA dev team, Teams channel **#rpa-support**
## 1. What the bot does
Every day at 07:00 Windows Task Scheduler starts the bot. It signs in to the payer web portal and downloads one remittance report (a CSV file) for each active provider in the input workbook `data/input/providers.xlsx`. It takes two sheets from that workbook: **Login** (the portal username and password) and **Providers** (NPI, name, active flag). It produces:
- one CSV per provider in `data/downloads/`;
- a result workbook, `data/output/Download_Report_<YYYYMMDD>.xlsx`, with Provider, NPI, Status and Remarks for each provider;
- a screenshot in `data/screens/` when a provider's report page does not load.
It drives one app: a **Chrome** window that it opens itself.
## 2. How it works
```
Windows Task Scheduler (07:00 daily)
└─ starts → the bot (one background program, no window of its own)
│
├─ reads → data/input/providers.xlsx (Login + Providers sheets)
├─ drives → its own Chrome window ──> payer web portal
│ (shows "controlled by automated test software"; do not touch it)
├─ saves → data/downloads/*.csv (one per provider)
├─ saves → data/screens/<NPI>.png (only when a page fails to load)
├─ writes → logs/bot.log (one line per provider)
└─ writes → data/output/Download_Report_<YYYYMMDD>.xlsx (at the very end)
```
- **Task Scheduler** starts the bot every morning. Nothing else starts it.
- **The bot** is a background program. It has no screen, no Stop button and no progress display.
- **Chrome** is opened by the bot, and the bot controls it. Every portal page you see in it during a run is the bot's work.
- **The input workbook** is the only list of providers. A provider that is not in it is never downloaded.
- **The downloads folder** collects every CSV. Files keep the portal's own name and are never renamed or cleaned up.
- **The log** gets one line per provider (time, NPI, name, status). It does not give the reason.
- **The result workbook** is written only when the run ends. While the bot is running, today's file does not exist yet.
**One run, step by step:**
1. Open the input workbook and read the login and the provider list.
2. Open Chrome and sign in to the portal. The bot waits up to 30 s for the dashboard.
3. For each provider, in input order:
- If the active flag is `N`, record **Skipped** and move on.
- Open that provider's remittance page.
- If the portal has logged the bot out, sign in again and retry (up to 3 times).
- If a maintenance banner shows, record it (see E05) and move on.
- Wait up to 30 s for the report grid.
- If the portal says there are no results, record **No Data**.
- Otherwise click Export and wait up to 60 s for a new CSV in the downloads folder.
4. Close Chrome and save today's result workbook.
## 3. Before a run
- [ ] Google Chrome is installed and opens normally on this machine. The bot cannot start without it, and then no result workbook is written (E08).
- [ ] These folders exist: `data/input`, `data/output`, `data/downloads`, `data/screens` and `logs`. The bot does not create them.
- Without `logs`, the bot stops before it starts and records nothing.
- Without `data/output`, the result workbook is lost (E08).
- Without `data/screens`, screenshots are silently not saved.
- [ ] `data/input/providers.xlsx` is **closed** (nobody has it open in Excel) and still has the sheets named exactly **Login** and **Providers**.
- [ ] **Today's result workbook is not open in Excel** from an earlier run. If it is open, the bot cannot save, and the result is lost (E08).
- [ ] **Login sheet:** username in A2, password in B2. When the portal password changes, update B2 before the next 07:00 run. Otherwise every row shows "Login Failed" (E01).
- [ ] **Providers sheet:** one provider per row, starting at row 2.
- Column A is the NPI, column B the name, column C the active flag.
- A row with column A empty is ignored.
- **Only a capital `N` in column C skips a provider.**
- Good: `N`. The provider is skipped.
- Bad: `n`, `No`, `Inactive`. The provider is treated as **active** and downloaded.
- NPI: digits only, no spaces or dashes.
- Good: `1234567890`.
- Bad: `1234 567 890`. The portal finds no provider, and you get a No Data or grid error.
- [ ] The NPI must already be set up in the portal for this login. The bot does not check this; an unknown NPI shows up as E03 or E06.
## 4. During a run
- **Do not close, click in or type in the bot's Chrome window.** It shows a "Chrome is being controlled by automated test software" bar. Your own separate Chrome windows are fine.
- **Do not put, move or delete files in `data/downloads/` during a run.** The bot counts *any* new CSV that appears there as "the provider's file" (see E04 and section 6).
- **Do not open `data/input/providers.xlsx` in Excel around 07:00**, or today's result workbook while a run is going.
- **Do not log in to the portal with the same username during a run.** The portal may end the bot's session, which leads to E02.
- **To stop the bot:** in Windows Task Scheduler, right-click the bot's task and choose **End**. Then close any Chrome window still showing the "automated test software" bar.
- Stopping it this way means **no result workbook is saved** for that run.
- Closing the Chrome window alone does not stop it cleanly. The bot crashes on the next provider, and the result workbook ends up partial (E09).
- If you have no rights in Task Scheduler, ask #rpa-support.
- **A run looks stuck** when `logs/bot.log` has gained no new line for more than about 10 minutes. A normal provider takes under 2 minutes.
- The bot has **no watchdog**. A page that hangs is abandoned only after about 5 minutes, and then the run crashes (E09/E10).
- Wait 10 minutes, then escalate.
## 5. Where to look when something goes wrong
| Question | Where to look |
|---|---|
| Did the run finish? | `data/output/` → does `Download_Report_<today's date YYYYMMDD>.xlsx` exist? If not, the run failed before the end (E08). Also compare its row count to the active rows in the input: fewer rows means the run stopped partway (E09). |
| Which providers failed? | Result workbook → Status column. Look for **red** (Error), **uncoloured** (Login Failed), **and** green rows whose Remarks say "portal maintenance - no file" (E05). |
| Why did one provider fail? | Result workbook → Remarks column. If it says "Report grid did not load", open `data/screens/<NPI>.png`. |
| Did the bot already retry? | `logs/bot.log` → lines with "session expired, re-login 1/2/3". That is the only retry the bot does. Every other failure is tried once (section 8). |
| Did a provider's file actually arrive? | `data/downloads/`, sorted by date modified. Look for a CSV from the time of that provider's log line. File names do not contain the NPI. |
| When did each provider run? | `logs/bot.log` → search for the NPI. Each line has the time and the status, but not the reason. |
| Which version is running? | No way to tell (section 10). |
## 6. Statuses
These are all the Status / Remarks pairs the bot writes in the result workbook.
| Status | Remarks | Colour | Meaning | What ops does |
|---|---|---|---|---|
| Downloaded | Report saved | green | A new CSV appeared in the downloads folder after Export | Nothing |
| Downloaded | portal maintenance - no file | green | **Needs follow-up. Nothing was downloaded.** The portal showed a maintenance banner | Re-run after the maintenance (E05) |
| No Data | No remittances in the date range | green | The portal shows no remittances for this provider in its default date range | Nothing, unless this provider is expected to have data (E06) |
| Skipped - inactive provider | Marked N in the input | green | Column C is `N` | Nothing, unless the provider should be active (E07) |
| Error | Report grid did not load | red | The provider's page did not show the report within 30 s | E03 |
| Error | Download did not finish | red | No new CSV within 60 s of Export | E04 |
| Error | Session expired | red | The portal logged the bot out 3 times in a row on this provider | E02 |
| Login Failed | Check the portal username/password | **no colour (white)** | The first sign-in failed. **Every** provider gets this row, including inactive ones. Nothing was downloaded | E01 |
## 7. Exception catalogue
| ID | What you see | How the bot identifies it | Where it is recorded | Type | Who acts | What to do | Escalate when |
|---|---|---|---|---|---|---|---|
| E01 | Every row says "Login Failed - Check the portal username/password", with no colour | After it clicks Sign in, the bot waits 30 s for the portal dashboard. If the dashboard doesn't appear, it gives up on the whole run. It cannot tell a wrong password from a slow or down portal | Result workbook only. **Nothing is written to the log** | Business (wrong password) or System (portal slow/down) | Ops | 1. In your own browser, sign in to the portal with the Login sheet's username and password. 2. If that fails, get the password fixed and update B2. 3. If it works, the portal was slow. Re-run the bot (section 11 has how to re-run). | Login works by hand but the bot fails again. Send the time of the run and the result workbook |
| E02 | Error - "Session expired" | The provider's page sent the bot back to the login page. The bot signed in again and retried, 3 times | Result workbook; log lines "session expired, re-login 1/2/3" | System | Bot, then Ops | Check that nobody else used the same portal login during the run (section 4). Re-run later | It happens on 2 runs in a row. Send the log lines and the run time |
| E03 | Error - "Report grid did not load" | The bot waits 30 s for the report grid on the provider's page. No retry | Result workbook; screenshot `data/screens/<NPI>.png` (overwritten on the next run, so copy it first) | System (or Business if the NPI is wrong) | Ops | 1. Open the screenshot. 2. If it shows an error such as "provider not found", check the NPI in the input. 3. Otherwise open that provider in the portal yourself. If it loads, re-run | The same provider fails on 2 runs while it loads fine by hand. Send the screenshot copy, NPI and run time |
| E04 | Error - "Download did not finish" | After Export, the bot checks the downloads folder every second for 60 s for a **new** CSV. No retry | Result workbook only. The log shows Error with no reason | System | Ops | **Check `data/downloads/` first.** A slow download may have finished after the bot gave up; look for a CSV from a minute or two after that provider's log time. If it is there, you have the file. If not, re-run (re-running only downloads again; it changes nothing in the portal) | It fails on 2 runs in a row. Send the run time and the log |
| E05 | **Green** "Downloaded" with Remarks "portal maintenance - no file" | The portal page shows a maintenance banner | Result workbook Remarks only. The log says "Downloaded" | System | Ops | Treat it as **not downloaded**. Re-run once the portal is out of maintenance. Filter Remarks, not just Status, every morning | Maintenance lasts more than a day |
| E06 | "No Data - No remittances in the date range" | The portal shows its "no results" message for that provider | Result workbook | Business | Ops | Usually nothing. If the provider should have remittances, open the provider in the portal yourself. The bot always uses the portal's **default** date range; it never sets dates | The portal shows remittances by hand but the bot keeps saying No Data |
| E07 | "Skipped - inactive provider - Marked N in the input" | Column C of the Providers sheet is `N` | Result workbook | Business | Ops | If the provider should run, clear column C and re-run | Never |
| E08 | **No result workbook** for today in `data/output/` | The bot stopped before it could save. Possible causes: input workbook missing, open or locked, or a sheet renamed; Chrome failed to start; a folder from section 3 is missing; today's result workbook was open in Excel when the bot tried to save | Nowhere ops can see. The log usually has nothing new for today | System | Ops, then Dev | 1. Go through the section 3 checklist (closed files, folders, sheet names, Chrome opens). 2. Fix what is wrong and re-run | Everything in section 3 is fine and it still fails. Send the date, the time it should have run, and `logs/bot.log` |
| E09 | Today's result workbook has **fewer rows** than the active providers in the input | The bot crashed mid-run. Possible causes: signing in again after a logout failed; the portal's Export button was missing; Chrome crashed or someone closed it. The rows done before the crash are saved; the provider it was on and all later ones are missing | Missing rows in the result workbook. The last line in `logs/bot.log` is the last provider that finished. No reason is recorded anywhere ops can see | System | Dev | **Copy today's result workbook to another name first**, because a re-run on the same day overwrites it. Then re-run. A re-run processes every provider again | Always. Send the copied workbook, the time of the last log line and `logs/bot.log` |
| E10 | The run takes far longer than usual; the log gets no new line for many minutes | No check exists. A portal page that never finishes loading holds the bot for about 5 minutes, then the run crashes as in E09 | Gap in `logs/bot.log` times | System | Ops, then Dev | Wait 10 minutes. If it has still not moved, stop it (section 4) and re-run later | It happens on 2 runs. Send the run time and `logs/bot.log` |
## 8. What the bot recovers by itself
| Problem | What the bot does | Limit | After the limit |
|---|---|---|---|
| Portal logs the bot out on a provider's page | Signs in again and opens the same provider again | 3 re-logins per provider | Row = Error "Session expired", next provider. **If one of those sign-ins itself fails, the run stops (E09).** |
| First sign-in slow | Waits | 30 s, one try | All rows "Login Failed", run ends (E01) |
| Report grid slow | Waits | 30 s, one try | Row = Error, screenshot, next provider (E03) |
| Download slow | Waits | 60 s, one try | Row = Error, next provider (E04) |
| Run stuck or crashed | **Nothing.** No watchdog, no restart, no resume | - | Ops re-runs. A re-run starts from the first provider |
So for every failure except a logout, the bot has already given up by the time you see it. There is nothing to wait for. Re-running is safe: the bot only downloads, so a re-run never changes anything in the portal. It does overwrite today's result workbook and adds duplicate CSVs (Chrome appends "(1)" to their names).
## 9. Known incidents
There is no change history, bug log or release notes for this bot. The only record is the log, which covers a single run.
| When / version | What ops saw | Cause | What to do now | Ops could handle? |
|---|---|---|---|---|
| 2026-09-02 / unversioned | About a minute into the run the portal logged the bot out. The bot signed in again, and the next provider ended as Error | Not recorded: the log does not keep the reason, so it was either a grid timeout (E03) or a download timeout (E04) | Follow E03/E04: check the screenshot and the downloads folder, then re-run | Partly |
## 10. Releases and version check
- **How an update arrives:** there is no update process. A developer copies new files into the bot's folder on this machine.
- **How to confirm it applied:** you can't. The bot shows no version anywhere. Ask the dev team to confirm the change in #rpa-support, and check the next morning's result workbook.
- **If the behaviour did not change:** report it in #rpa-support with the date of the expected update.
- The bot writes nothing shared on start-up, so another machine with an older copy does not undo a newer one. Only one machine should run the 07:00 task, though. Two machines would download everything twice into their own folders.
## 11. Escalating to dev
Post in the Teams channel **#rpa-support**. Send:
- the exception ID from section 7 (for example "E09");
- the run date and the time of the failure (from `logs/bot.log`);
- the result workbook `data/output/Download_Report_<YYYYMMDD>.xlsx` (a copy, if you are about to re-run);
- `logs/bot.log`;
- for E03, the screenshot `data/screens/<NPI>.png`.
**How to re-run:** in Task Scheduler, right-click the bot's task and choose **Run**. If you have no rights in Task Scheduler, ask in #rpa-support.
**Never send:** the input workbook (its Login sheet holds the portal password), `config.yaml`, passwords, or any patient data from the downloaded CSVs.
## 12. Failure-class check
- **F1** missing prerequisite: covered in section 3 (Chrome, folders, closed files, sheet names) and E08.
- **F2** wrong window / focus: covered in section 4 (don't touch the bot's Chrome window). The bot drives its own Chrome session, not windows found by title, so it cannot grab another app.
- **F3** target slower than expected: covered in E01, E03, E04 and E10, with the limits in section 8. Screen size does not apply; this is a browser, not a desktop-screen bot.
- **F4** input read differently: covered in section 3 (only capital `N` skips; NPI digits only; sheet names). Also in Fix #12.
- **F5** marked done when nothing was done: covered in E05 and Fix #1 and #2.
- **F6** action happened but couldn't be confirmed: covered in E04 (check the downloads folder before re-running). The action is a download only, so a re-run cannot double-post.
- **F7** business exception shown as success: covered in section 6 (maintenance row marked "needs follow-up"), E05 and Fix #1.
- **F8** endless re-runs / watchdog: covered in section 8 and E10. There is no auto re-run loop; the missing watchdog is Fix #9.
- **F9** stopping the wrong way: covered in section 4 (End the task in Task Scheduler, then close the leftover Chrome).
- **F10** one stuck item blocks the queue: covered in E09 (a crash on one provider drops all later providers). There are no batch gates or upload locks. Also in Fix #4 and #5.
- **F11** cause hidden from ops: covered in section 5, and Fix #6, #7 and #16.
- **F12** update didn't arrive: covered in section 10 and Fix #13.
## 13. Fix before handover (dev team)
| # | Gap | Why it hurts ops | Where (file:line) |
|---|---|---|---|
| 1 | Maintenance banner returns status "Downloaded" (green) with no file | Ops filtering on Status think the file arrived | `bot.py:54-55`, colour map `bot.py:27` |
| 2 | Any new `*.csv` in the downloads folder counts as success; the file is not renamed to the NPI or checked for completeness | False "Downloaded"; ops can't tell which CSV belongs to which provider | `bot.py:63-68` |
| 3 | "Login Failed" rows get no fill (the red in `COLOUR` is never applied on that path); inactive providers are also marked Login Failed | Failed rows look neutral; the inactive rows are misleading | `bot.py:89-91` |
| 4 | A failed re-login inside the retry loop raises `RuntimeError`, which `except SessionExpired` doesn't catch | Run stops; current and later providers missing from report (E09) | `bot.py:100-102` |
| 5 | No per-provider try/except: a missing `#export-csv`, a `NoSuchElementException` or a `WebDriverException` stops the whole run | One bad provider drops all later ones (E09) | `bot.py:64`, `bot.py:92-107` |
| 6 | The log has no remark text, errors are logged at INFO, a first-login failure is not logged, and tracebacks are never logged | The cause is only in the result workbook, or nowhere at all after a crash | `bot.py:105`, `bot.py:88-91`, `bot.py:113-114` |
| 7 | Failures before the `try` block (input missing or locked, wrong sheet, Chrome or chromedriver start, missing argument) write no report | Ops get no result workbook and no reason (E08) | `bot.py:74-81`, `bot.py:114` |
| 8 | `data/output`, `data/screens`, `data/downloads` and `logs` are never created; `save_screenshot` fails silently; the save in `finally` can raise and mask the original error | Lost report or screenshot with no message | `bot.py:20-21`, `bot.py:30`, `bot.py:59`, `bot.py:110` |
| 9 | No page-load timeout, no wait on the login fields, no watchdog or alert | A hung page holds the run about 5 min, then crashes; nobody is told | `bot.py:40-43`, `bot.py:51`, `bot.py:81` |
| 10 | No retry for grid or download timeouts | One-off portal slowness becomes a manual re-run | `bot.py:56-70` |
| 11 | A same-day re-run overwrites the result workbook; screenshots are overwritten on every run; the downloads folder is never cleaned (duplicates get "(1)") | Evidence lost on re-run; folder grows forever | `bot.py:59`, `bot.py:80`, `bot.py:110` |
| 12 | The active flag treats only exact `"N"` as inactive | `n` or `No` providers are downloaded anyway | `bot.py:77` |
| 13 | No version string, no release process | Ops can't confirm which code runs | `bot.py:1-2` |
| 14 | The portal password is in plain text in the input workbook (Login sheet) | Anyone with the input file has the portal login | `bot.py:76` |
| 15 | `config.yaml` holds a plain-text `api_token` that the code never uses | Secret exposure with no benefit. Remove it and rotate | `config.yaml:2` |
| 16 | Crash output only goes to the Task Scheduler console, and ops can't reach Task Scheduler history | Root cause of E08/E09 invisible to ops | `bot.py:113-114`, `README.md:6` |
| 17 | The bot never sets a date range, so "No Data" depends on the portal's default | No Data may hide real remittances if the portal default changes | `bot.py:51` |
| 18 | The `BOT_WAS_RUN` marker is written on every run and never read | Dead file; could be confused for a status flag | `bot.py:74` |
| 19 | Log lines include provider name and NPI | Log can't be shared freely in tickets | `bot.py:105` |