automation-tools/ecw-charge-validation
8
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial import: eCW Charge Validation bot and results Portal

Browse Source 
Snapshot of the working project as of 2026-10-23 (local 2d4d4fc), one
commit with no earlier history: the tracked files were de-identified
first (synthetic claim ids and patient data only; see Context.md
section 7). Credentials, run logs, screenshots, claim reports and client
workbooks are gitignored and not included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
ManikandanandClaude Opus 5.5 committed 2026-10-23 18:38:17 +05:30
commit ba15a574ab
205 files changed
+56772
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/2026-10-01-demo-production-review.md
+324
View File
@@ -0,0 +1,324 @@
# eCW Charge Validation bot: plan before the client demo and live production
I merged 60 verified findings into 38 items: 11 demo blockers, 16 production items, 5 logic items, 6 structure changes and 7 performance items (2 of the performance items are new; the rest point back to items above). Every severity below is the verifier's. When findings were merged, the item takes the highest verifier severity. Effort: **S** = under half a day, **M** = 1–3 days, **L** = more than 3 days. All paths are under `src/ecw_bot/` unless shown otherwise.
---
## 1. Demo blockers
**D1 · Test mode is on, so the demo Charge Summary is mostly blank and replaces the live file's name.** Severity: high. Effort: S
`flows/export_medcor_ready.py:552, 572-574, 600-613, 684-698` · `core/charge_summary.py:138-157, 181-191`
- **Problem:** `TEST_CLAIM_IDS_FILE` is set to a 2-column hand sheet. The summary is built from that sheet, so 16 of the 24 columns come out blank, Group Name included. The file is saved as `Charge Summary_<date>.xlsx`, which pushes a live same-day file aside to `__HHMMSS`. An unreadable sheet still exits 0, and with `SHOW_CLAIM_IDS=0` the operator gets no sign that this is a test run.
- **Fix:** For the demo, either clear `TEST_CLAIM_IDS_FILE` or point it at `Medcor Ready Claim Sheet.xlsx` (it has all 15 report columns). In code:
- write test runs to `Charge Summary_TEST_<date>.xlsx`;
- print a "TEST MODE" banner that always shows;
- return a non-zero exit code when the test sheet cannot be read;
- on demo morning, check that every test claim is still in the 108ME queue.
**D2 · The Charge Summary marks claims "Clean / Pass / Ready to Submit" even when they were never actually checked.** Severity: high. Effort: M (a demo-only version is S)
`core/charge_summary.py:85-116` · `conditions/base.py:69-90` · `flows/export_medcor_ready.py:426-442, 498-507` · `conditions/telehealth.py:112-136` · `conditions/vaccine_admin_codes.py:336-360, 635-639` · `pages/dental_notes.py:47-52`
- **Problem:** `claim_outcome` knows only two states: something fired, or the claim is clean. All of the following fall through to CLEAN/Pass:
- a condition's `detect()` or `apply()` raised, because `run_all` drops it;
- a condition detected a problem but declined to act (unmapped payer, CPT tab not confirmed, admin code could not be added);
- `APPLY_CORRECTIONS=0`;
- the progress note was not read (`opened` is nothing, absent, disabled, or an error);
- Dental `high_risk` is None, or the sliding fee verdict is "unknown".
A form-read error leaves a blank row that looks exactly like "not found". The per-claim box also says "none matched" right after the box that said "about to apply X".
- **Fix:**
- Give `ConditionResult` a `status` (changed / no_change / declined / failed) and a `reason`, and have `run_all` return every condition that was detected or failed.
- Store `ran`, `fired`, `problems` and `note_read` in `data["_conditions"]`.
- Map any problem, an unread note, `APPLY=0`, or an incomplete Dental check to Category "Review" / "Needs manual review" with the reason in Comments.
- Write an explicit "Bot could not read claim" row for read errors.
- Demo-only version: change just `claim_outcome` so a missing `_conditions`, an unread note, and Dental None/unknown become Review.
- Add tests for each of these paths.
**D3 · The telehealth conditions report a change on claims that are already correct (POS '2'→'2'), so correct claims are marked Fail.** Severity: high. Effort: S
`pages/claim_form.py:799-806` · `conditions/telehealth.py:131-144` · `conditions/medicare_telehealth.py:73-96` · `tests/test_telehealth_fix.py:127-128`
- **Problem:** `set_cpt_pos_modifier` always fills POS and M1 and always records a change, so every correctly coded telehealth claim gets both review boxes and goes to the summary as "Fail / Corrections pending".
- **Fix:** Skip rows that already match. Normalize POS by stripping leading zeros so that "02" equals "2", and compare M1 in upper case. Record a change only when the value after differs from the value before, return `none` when nothing moved, and add an apply-twice test.
**D4 · Audio-only detection searches the whole note for "audio", so "audio and video" visits get modifier 93.** Severity: high. Effort: M
`pages/claim_form.py:719-727, 754` · `pages/progress_note.py:106, 129` · `conditions/telehealth.py:67, 120-123` · `conditions/medicare_telehealth.py:43, 69-71`
- **Problem:** The bare substring tests on `document.body` cause two errors:
- "real-time audio and video", "Audiology" and "audiometry" all classify the visit as audio-only, giving Commercial and Medicare visits 93 instead of 95;
- a "telehealth" or "telephone" mention in the Plan or header makes an in-person visit count as telehealth.
- **Fix:** Classify only from the structured fields: the Session Setting (PoS), Mode of Treatment, the TELEHEALTH section, and the procedure-code (TH) tags and modifiers.
- Explicit "audio only" or "no video" means audio. "audio and video" or "video" means video. Anything else means unknown, and the claim is left for review.
- Use word boundaries, and show the matched phrase in `describe()`.
**D5 · telehealth writes modifier 93 for unmapped, blank, Program and Dentaquest payers on audio visits.** Severity: high. Effort: S
`conditions/telehealth.py:77-101 (99-100)` · `tests/test_conditions_telehealth.py:69`
- **Problem:** The `mode == "audio"` check runs before the group lookup. As a result:
- an unclassified Medicare payer gets rule #21 (POS 2 + 93) where rule #22 applies;
- an unmapped Medicaid MCO gets 93 instead of GT.
This contradicts the module's own "leave as read" rule.
- **Fix:** Return None unless the group is Commercial before the audio branch runs. Update the test at line 69, and report "skipped: unmapped payer X" as a declined result (see D2).
**D6 · medicare_telehealth hides a failed G2025 add; the box and the summary both read as success.** Severity: high. Effort: S
`conditions/medicare_telehealth.py:80-121` · `core/charge_summary.py:56-58` · `pages/claim_form.py:1543`
- **Problem:** When G2025 is not confirmed, the condition still returns changed=True and `describe()` never mentions G2025. The fixed summary text then says "G2025 added … other CPTs unchecked", which is false on the UB screen, where the Add flow has not been verified.
- **Fix:**
- Add a `{"code": "G2025", "failed": True}` record and have `describe()` print "G2025 NOT added — add by hand; other rows NOT unchecked".
- Build the summary comment from the actual changes, and mark the claim Review so Save is not offered.
- Read M1 back after `_set_row_modifier`.
**D7 · HCFA and Dental claims are read before Angular has filled the form.** Severity: high. Effort: S
`pages/claim_form.py:384-397, 618-625` · `pages/dental_notes.py:34-60`
- **Problem:** `open_claim` returns as soon as the HCFA or Dental Cancel button is visible. `extract()` then reads 0 CPT rows, Prog. Notes reads as "disabled", and blank Dental notes read as "not high risk". This was seen live on a claim-1 read (run_20260930_163006). Because no condition fires, the claim goes to the summary as Pass.
- **Fix:**
- HCFA: wait until `[ng-bind="ClaimData.InvId"]` equals the claim id (this also gives HCFA the identity check UB already has), then until the count of CPT rows with a code is at least 1 and stable for about 500 ms.
- Dental: wait on an equivalent Dental marker.
- Re-check the Prog. Notes enabled state for up to about 5 s.
- If any wait times out, mark the claim "not evaluated" (see D2).
**D8 · Progress Notes uses a fixed 3 s sleep and one probe, so slow notes become "nothing" and telehealth is never checked.** Severity: high. Effort: S
`pages/claim_form.py:629-639` · `pages/progress_note.py:148-161`
- **Problem:** Logs show 3 misses in about 115 note reads (2 of them on claim 1). In all 3 the viewer had opened but the document had not loaded yet. The claim is then reported Clean and Ready to Submit.
- **Fix:**
- Poll `find()` every 250 ms for up to about 15–25 s, accepting a frame only when it is visible, has `[prisma-section]`, and its section count is the same on two polls in a row. Retry the click once if nothing appears.
- Mark the claim `note_unread` for D2.
- Optional: check that the note's DOS matches the claim's CPT SDOS.
**D9 · The per-claim box shows a page of '?' for Dental claims, untranslated Tamil words, and up to three boxes per corrected claim.** Severity: medium. Effort: S
`flows/export_medcor_ready.py:357-381, 496-508, 244, 283` · `pages/claim_form.py:1622-1666 (1654)` · `pages/progress_note.py:221`
- **Problem:** For Dental data, `claim_form.summary()` prints "Patient : ? … Charges : ?" right after the HIGH RISK box. Other boxes show "… innum N" and "Chart locked: Aamaa/Illa". A corrected claim needs three OK clicks, each with a 180 s timeout.
- **Fix:**
- Fold the Dental verdict into the SHOW_EACH_CLAIM box instead of calling `claim_form.summary`. This also fixes the ungated Dental box (P6).
- Change the strings to "… and N more" and "Yes"/"No", and skip fields that came back empty or '?'.
- For the demo, merge the before-apply note into the review box and use a shorter `MSGBOX_TIMEOUT_SECONDS`.
**D10 · A second run on the same day loses its whole summary if the earlier file is open in Excel.** Severity: high. Effort: S
`core/charge_summary.py:181-192` · `flows/export_medcor_ready.py:172-186, 551-558`
- **Problem:** `out_path.rename(backup)` fails with WinError 32 when the file is open. The error is caught and a box is shown, but that run's outcomes are discarded. The Medcor Ready categorize step uses the same rename-aside pattern.
- **Fix:** If the rename or the write fails, write to `…__HHMMSS.xlsx` instead (write to a temp file, then move it into place) and say so in the log and the box. Apply the same fix to `_categorize_by_payer`, and add a test that holds the file open.
**D11 · (known; new data) Each failed claim click waits the full 30 s timeout, and the fallback check does not wait at all.** Severity: high. Effort: S
`pages/claim_form.py:349-376` · `core/page_utils.py:43-47`
- **Problem:** The "already open" fallback only runs after Playwright's full 30 s click timeout. It hit 43 of 92 opens (and 15/15 in the 16-Sep run), and the form sits open for about 30 s, which looks hung. The `visible()` check at line 372 is instant because Playwright 1.62 ignores its timeout, so a form that opens slowly gets Escaped by attempt 2. On HCFA that bypasses `closeClaimScreen()`.
- **Fix:** Use timeouts `(5_000, 15_000)`, replace line 372 with `locator.filter(visible=True).first.wait_for(timeout=5_000)`, and log `str(exc).splitlines()[0]` plus the elapsed time for the separate root-cause task.
---
## 2. Before live production
**P1 · Saving to eCW is on by default, and the .env reader can silently turn it on.** Severity: high. Effort: S
`settings.py:46-80, 367, 380, 442-456` · `.env.example:144, 152` · `flows/export_medcor_ready.py:670-671` · `core/msgbox.py:61-62`
- **Problem:** `SAVE_CORRECTIONS` and `APPLY_CORRECTIONS` both default to True, and the template ships them as 1. Values like `SAVE_CORRECTIONS=0 # note`, a blank value, or `O` all read as True; `CLAIM_PROCESS_LIMIT=5 # demo` reads as 1. No banner shows which mode is running, and the save box's default button is OK, so pressing Enter saves.
- **Fix:**
- Default both settings to False and set them to 0 in `.env.example`.
- Strip unquoted `\s+#…` comments when reading `.env`.
- Make unrecognised values a `validate()` error (exit 2).
- Print a `[MODE] VERIFY / LIVE-WRITE | limit | test file` banner, and make write mode require OK/Cancel at start.
- Add `MB_DEFBUTTON2` to the save confirmation so Enter means Cancel.
**P2 · A condition that fails partway through `apply()` leaves its edits on the form, and the next confirmed Save writes them unreviewed.** Severity: high. Effort: S (on top of the D2 `run_all` change)
`conditions/base.py:82-87` · `flows/export_medcor_ready.py:315-331` · `pages/claim_form.py:801-806, 868-869, 1074-1076, 1443-1456` · `docs/superpowers/specs/2026-09-16-condition-framework-design.md:174`
- **Problem:** The write helpers edit fields one at a time with no rollback. When one raises, the condition is dropped from `results`, the save box lists only the other conditions, and `saveAllData()` saves the whole form. The spec's statement that "only what's in results" gets saved is wrong. (Reproduced with fakes.)
- **Fix:** Mark the claim tainted whenever any `apply()` raised. `_confirm_and_save` then never offers Save: it shows "X failed mid-edit, not saved" and closes with Cancel. Correct the spec row, and add a gate test.
**P3 · One stuck form or an expired session ruins the rest of the run: no recovery, no stop, and errors are reported as "Not found".** Severity: high. Effort: M
`flows/export_medcor_ready.py:419-420, 444-446, 478-484, 510-526` · `pages/claim_form.py:372-376, 412-417, 494, 536-539` · `pages/claim_filters.py:343` · `pages/claim_results.py:246-247`
- **Problem:** Several gaps combine:
- `close_claim()`'s False return is ignored.
- If `open_claim` raises after the form appears, close is skipped entirely.
- A leftover form blocks the Lookup click (30 s per claim), and a lost session costs up to 300 s per claim.
- All of these become "Not found" with blank rows in the summary, and nothing counts consecutive failures.
- **Fix:**
- Close whenever `any_claim_form_open()`, inside try/except.
- Run a `_recover()` before each search: close the form, then re-run `open_claims` and re-apply the four filters.
- Record error separately from not_found everywhere.
- Stop with its own exit code after 5 consecutive errors, or when the login page is detected.
**P4 · Run results live only in memory: a crash or Ctrl+C loses the whole Charge Summary, and there is no resume.** Severity: high. Effort: M
`flows/export_medcor_ready.py:468-469, 475, 535, 540, 687-689, 334` · `pages/claim_form.py:1613-1618`
- **Problem:**
- The summary is written only after the last claim. A dead page in the `finally` close, or a KeyboardInterrupt, skips it and also skips `clear_claim_no`.
- A re-run starts again at claim 1.
- `<id>.json` is overwritten on every read.
- With saving on, only condition names are logged, never the changed fields.
- **Fix:**
- Append one flushed JSONL line per claim to `Output\progress_<date>.jsonl`.
- Wrap the loop in try/finally so the summary and `clear_claim_no` always run, with unreached claims marked "Not processed".
- Add `RESUME_TODAY=1`.
- Keep snapshots per run as `claims\<date>\<id>_<HHMMSS>.json`, and log the full save body.
**P5 · `save_claim` and `close_claim` count success as "#UBClaimDiv hidden", which is always true on HCFA and Dental.** Severity: high. Effort: S
`pages/claim_form.py:532-533, 1574-1582` · `selectors/claim_form.py:40, 229` · `flows/export_medcor_ready.py:444` · `core/charge_summary.py:113-116`
- **Problem:** On HCFA both functions return True immediately, even while the form is still up; this was reproduced. A save blocked by eCW validation is then logged as "Saved", skips Cancel, writes "Ready to Submit", and leaves the form blocking the next claims. `SEL_OK_HCFA` also matches the UB OK button, so a stuck UB save gets OK clicked a second time.
- **Fix:** Add `_wait_form_closed()` that polls `any_claim_form_open()`, and use it in both functions. Pick the OK button per screen type (Dental refused), click once, never use a cross-screen fallback, and return False if the form is still up. Add fixture tests with a no-op OK and a no-op Cancel.
**P6 · There is no unattended mode: the Dental box cannot be turned off, the end-of-run Enter prompt can hang or crash, and exit codes hide failures.** Severity: high for the Dental box, medium for the rest. Effort: M
`flows/export_medcor_ready.py:61-62, 375, 556, 578-581, 604-609, 693-697, 716-718` · `settings.py:311-439` · `core/msgbox.py:83-101`
- **Problem:**
- The Dental box costs 180 s per Dental claim even with every review flag off.
- Seven prompt flags must each be set to 0, and `ECW_KEEP_OPEN` is not in `.env.example`.
- With no stdin, `input()` raises RuntimeError, so a successful run exits 1, which means "login failed".
- No export, an unreadable sheet, or every claim failing all exit 0.
- **Fix:** Add one `RUN_MODE=attended|unattended` setting that switches all boxes and KEEP_OPEN together. Gate the Dental box (see D9). Catch `(EOFError, RuntimeError, OSError)` and skip the prompt when there is no TTY. Add distinct exit codes for "nothing processed" and "some claims errored", and update the README exit-code table.
**P7 · Write-adjacent buttons are clicked by numbered ids with no check of what they do.** Severity: medium. Effort: S
`selectors/claim_form.py:47-49, 55, 275` · `pages/claim_form.py:505-508, 1574` · `pages/sliding_fee.py:174-186`
- **Problem:** The UB Cancel/OK/X (`#UBClaimBtn17/18/3`) and Don't Save (`#savePrompt-tplBtn4`) are found only by their numbered id. If an eCW template renumbering moved those ids, Cancel would become a silent `saveAllData(true)`, and Don't Save would become Save, on a read-only check.
- **Fix:** Anchor on the handler instead:
- Cancel: `[ng-click*="closeUBClaimScreen"]`.
- Don't Save: `button[ng-click="saveData(2)"]` plus a check of its text. Do not scope it to `#savePrompt`, which is not in the real DOM.
- Before clicking, refuse any close target whose ng-click contains `saveAllData`.
- Add fixture ng-clicks and a swapped-id negative test.
**P8 · `visible(timeout=…)` never waits, so every caller that expects a grace period gets an instant check.** Severity: medium. Effort: S
`core/page_utils.py:43-47` · callers: `pages/claim_form.py:372, 436, 468, 589, 618` · `pages/progress_note.py:189` · `pages/sliding_fee.py:133-135` (already has a working `_appears`)
- **Problem:** Playwright 1.62 ignores `is_visible(timeout)`. As a result:
- a slow CPT tab gets read short;
- a slow Prog. Notes button reads as "absent";
- `.first` reports False when a hidden leftover element comes first.
- **Fix:**
- Change `visible()` to `filter(visible=True).count() > 0`.
- Add `appears(page, sel, timeout)` using `wait_for` and move only the callers that really mean "wait" onto it.
- Keep `is_ub_claim_form` and `is_dental_claim_form` as instant checks; a wait there would add 5–10 s to every claim.
**P9 · A bad `payer_groups.json` silently switches off 3 conditions for the whole run.** Severity: medium. Effort: S to validate at startup, M to centralize
`conditions/telehealth.py:88-91` · `conditions/medicare_telehealth.py:48-51` · `conditions/vaccine_admin_codes.py:312-316` · `core/payer_groups.py:27` · `core/charge_summary.py:179` · `settings.py:442-456`
- **Problem:** With 301 payers waiting to be added by hand, one trailing comma means telehealth, medicare_telehealth and vaccine_admin_codes return "not covered" on every claim, the log blames "unclassified" payers, and the summary write fails. Test mode skips the only warning box.
- **Fix:** Load the mapping in `validate()` and exit 2 if it is invalid, before login. Make the log messages say "mapping unavailable" rather than "unclassified". Centralizing the lookup is St6.
**P10 · TLS certificate checks are off for the session that types the password, opens the 2FA link and loads PHI.** Severity: medium. Effort: S
`core/browser.py:53-57` · `auth/mailbox.py:102-103`
- **Problem:** `ignore_https_errors=True` on the only browser context means a man-in-the-middle at or after the HTTP proxy is accepted silently. `tools/check_net.py` already passes with the default Windows certificate store, so the flag is not needed.
- **Fix:** Remove the flag, or put it behind `IGNORE_HTTPS_ERRORS` (default 0) with a loud warning. Accept the 2FA link only over https and only on an allow-listed host.
**P11 · Full clinical notes, sliding-fee screens and patient screenshots pile up with no minimisation and no deletion.** Severity: medium. Effort: M
`pages/progress_note.py:106-130` · `pages/claim_form.py:659, 664, 1601-1619` · `pages/sliding_fee.py:206` · `auth/login.py:197` · `core/page_utils.py:33-40` · `auth/mailbox.py:276` · `pages/dental_notes.py:56-59`
- **Problem:**
- Each claim's JSON stores the whole note: `text`, `sections` and `patient_header`.
- A success screenshot is taken on every claim, about 230–350 MB per full run, with no claim id in the name.
- Nothing is ever purged, the unit tests write into the same production folder, and the OTP code is printed.
- **Fix:**
- Save a redacted copy of the JSON (keep the in-memory `text`, which the conditions need).
- Add `DEBUG_SCREENSHOTS` (default 0) and put the claim id in failure-shot tags.
- Purge old Screenshots and Logs at start.
- Point `cfg.SCREENSHOTS` at a temp folder in tests.
- Stop logging the OTP value and the Billing Notes text.
- Ops: BitLocker plus a restricted folder ACL.
**P12 · Files likely holding PHI in the repo root are not gitignored.** Severity: medium. Effort: S
`.gitignore:1-35`
- **Problem:** `Dom.txt` (a live claim DOM capture), the two client workbooks and `~$…xlsx` are untracked but not ignored, sitting next to the uncommitted charge-summary work. One `git add -A` puts them in history.
- **Fix:** Move them to `Reference/` and update the comments that point to them. Add `Reference/`, `Dom*.txt`, `*.xlsx` and `~$*` to `.gitignore`. Stage files by name, and ship releases from `git archive`, not from a zip of the working folder.
**P13 · Every native `confirm()` from eCW is accepted automatically.** Severity: low. Effort: S
`core/page_utils.py:144-149`
- **Problem:** An eCW "save anyway?" or "save changes?" prompt on OK, Cancel or X would be answered Yes with no operator involved. None have been seen in 138 logs.
- **Fix:** Accept only `alert`. Dismiss `confirm` and `prompt` unless the message matches an allow-list, record what was seen per claim, and flag that claim Review.
**P14 · Claim No is cleared only at the end of a successful run, never before the bulk Lookup.** Severity: low. Effort: S
`flows/export_medcor_ready.py:540, 563-581` · `pages/claim_results.py:30-57`
- **Problem:** If eCW keeps the last Claim No (unproven), an aborted run would shrink the next export to 1 row. The confirm box does not show Claim No.
- **Fix:** Call `clear_claim_no` before `run_lookup` and raise if it fails. Add Claim No to `_filter_summary`, and warn when Total Counts is 0 or 1.
**P15 · The README, CHANGELOG and docstrings describe an older, read-only bot.** Severity: medium. Effort: S
`README.md:3, 60-89, 97-140, 384-386` · `CHANGELOG.md:299-304` · `pages/claim_form.py:2-7` · `conditions/vaccine_admin_codes.py:342-346` · `settings.py:360-361, 385` · `flows/export_medcor_ready.py:13, 351-352, 384` · `pages/dental_notes.py:9-11` · `pages/sliding_fee.py:10-11`
- **Problem:** The README says the form "is always closed with Cancel", while the code ships with writes on. The docs omit the 8 conditions, Dental, HCFA and the Charge Summary, and the CHANGELOG still lists telehealth as POS 11 on every row.
- **Fix:** Rewrite the Overview, Layout, Tests and claim-screen sections, including the write gates. Add one CHANGELOG entry per feature, and correct the stale comments.
**P16 · Nobody has proved that verification mode is really read-only.** Severity: low (unverified). Effort: S
`selectors/claim_form.py:204` (`addClaimFSCPTCode('save');addCptClaimEdit()`) · checkbox `updateBillToIns()`
- **Problem:** If these handlers save on the eCW server before OK is pressed, closing with Cancel does not undo an added G2025 row or a checkbox change, so `SAVE_CORRECTIONS=0` would not protect live claims, the demo included.
- **Fix:** On one test claim, add a row and toggle a checkbox, Cancel, reopen, and confirm that nothing persisted. Do this before the demo.
---
## 3. Logic improvements
**L1 · depression_dx_pointer can strip a billed line's only diagnosis pointer.** Severity: high. Effort: S
`pages/claim_form.py:860-864` · `conditions/depression_dx_pointer.py:108-130` · `tests/test_telehealth_fix.py:186-189`
- **Problem:** A checked 96127, 99213 or 90791 whose only pointer is Z13.31 ends up with ICD1–4 blank. That breaks the 837P SV107 rule and gets the claim rejected; this was reproduced on the HCFA fixture.
- **Fix:** If removing the pointer would leave nothing, leave the row alone and flag "only diagnosis is Z13.31 — needs review". Ask the client whether the primary pointer should be substituted instead, and update the pinned test.
**L2 · telehealth's "first real row" ignores whether the line is billed or excluded.** Severity: medium. Effort: S
`conditions/telehealth.py:131-132` · `conditions/medicare_telehealth.py:73-74` · `pages/claim_form.py:786-808`
- **Problem:** With BHT3 in row 1 (reproduced), POS 2/GT goes onto BHT3. dummy_codes then excludes that row, and the real billed service stays POS 11 with no modifier.
- **Fix:** Choose the first row that is checked, priced, and not in one shared exclusion set (BHT3/4, J2001/J2930, G8510/G8431, admin codes). If no row qualifies, decline with the reason. Add fixture tests.
**L3 · Exclusion conditions say "excluded" when no Assign-To-Patient checkbox was found (UB screen).** Severity: medium (plausible). Effort: S
`pages/claim_form.py:1443-1456` · `conditions/dummy_codes.py:54` · `conditions/j2930_j2001_update.py:62` · `selectors/claim_form.py:164-172`
- **Problem:** If the UB grid lacks `[ng-click*="updateBillToIns("]`, the fee is zeroed but the line stays checked and billed at $0. On those claims `billed=None`, so zero_fee_uncheck and priced_line_check never run.
- **Fix:** Spend two minutes in DevTools on one live UB claim to settle whether the selector exists. Record `checkbox_found`, and have `describe()` print "fee zeroed, NOT unchecked" when it is False. Flag the claim Review when every row has `billed=None`.
**L4 · A Sliding Fee screen that never fills in is reported as "NONE: no sliding fee dates assigned".** Severity: medium. Effort: S
`pages/sliding_fee.py:104-116, 143-153, 198-201`
- **Problem:** `_wait_filled` returns `{}` on a timeout or null, and the button's enabled state is checked once after 1 s. A slow modal therefore becomes a stated fact ("NONE") and a Pass (reproduced, 15.3 s).
- **Fix:** Return `{"error": …}` on a timeout or null, which makes the verdict unknown and therefore Review under D2. Poll the button's enabled state for about 5 s, and add a fixture test with `values=None`.
**L5 · The high-risk regex flags negations ("not high risk", "High Risk: No") and matches inside words.** Severity: low (plausible). Effort: S
`pages/dental_notes.py:31, 42-60` · `core/charge_summary.py:95-98`
- **Problem:** Any substring match routes the claim to the DentaQuest portal as Fail. The summary drops the matched text, so no reviewer can check it.
- **Fix:**
- Confirm with billing the exact marker they type.
- Add word boundaries and treat a negated match as "CHECK".
- Put the matched text in Comments, and add negation tests.
---
## 4. Structure changes
| # | Change | Where | Cost | Benefit |
|---|---|---|---|---|
| St1 | **One outcome model per claim.** `ConditionResult.status` + `reason`; `run_all` returns a RunReport (changed / declined / failed / tainted) that drives the box, the save gate, the summary and the log. | `conditions/base.py`, `flows/export_medcor_ready.py:426-442`, `core/charge_summary.py:71-116` | M | Implements D2, P2 and D6 together. Removes "none matched" lies and fixed-string comments. |
| St2 | **Run journal.** Per-claim JSONL checkpoint; summary built from the journal inside try/finally; resume; per-run snapshots. | `flows/export_medcor_ready.py:468-540` | M | Implements P3 and P4. A crash costs about 1 claim instead of the whole run; gives an audit trail of saves. |
| St3 | **Config profile.** `RUN_MODE` plus strict bool/int parsing plus a startup mode banner. | `settings.py:46-80, 311-439` | S–M | Implements P1 and P6. One switch replaces 7 flags; typos cannot silently enable writes. |
| St4 | **Production approval model** (decide before go-live): (a) `AUTO_SAVE_CONDITIONS` allow-list of client-signed conditions, or (b) two passes: read-only pass → client approves the summary → save pass on the approved ids, with no prompts. | `flows/export_medcor_ready.py:295-338` | M + client sign-off | Without it, about 270–570 corrected claims each need a human click, and every timeout means "not saved". |
| St5 | **Split claim_form.py (1,666 lines).** One `real_cpt_rows()` generator replaces 9 copied row loops; fee and telehealth rules move to a DOM-free module; rewrite the "read-only" docstring. | `pages/claim_form.py:2-7, 787-795 … 1430-1437` | L (docstring S) | One place to fix if eCW changes the grid. Do it after the demo; fix the docstring and do P16 now. |
| St6 | **Resolve payer and group once per claim** (report PAYER first, form name as fallback, log mismatches); conditions read `data["payer_group"]`. | 3 copies in P9 + `core/charge_summary.py:154` | M | The summary's Group Name and the condition decisions agree; removes 3 per-claim disk reads. |
| St7 | **Test runner.** `tests/run_all.py` (subprocess per file, totals, non-zero exit); AST-based layer check (catches the `core/charge_summary.py:174` violation and `from .. import X`); `check_root` skips when there is no .env. | `tests/test_settings.py:33, 53-56, 129-136`, `README.md:60-89` | S | One pre-demo/pre-deploy gate (397 cases, about 4.5 min) instead of 7 of the 28 files. |
---
## 5. Performance
**Expected time per claim (UB/HCFA, boxes off).** About 13–15 s of real work: search 3–8 s, open 1–2 s, CPT tab about 1.5 s, extract under 0.5 s, Prog. Notes about 4.5 s, close about 1 s.
- **Today:** about 43–45 s whenever the click fallback hits (43 of 92 opens, up to 15/15 in one run).
- **Dental:** the sliding-fee check takes 10–16 s; claim-to-claim cycles were seen at 59–61 s.
- **Not found:** 180 s or more each.
- **Boxes:** each unanswered box adds up to 180 s.
**Full run** (about 540 unique ids, with some queues of 1,145 or more):
- after the cuts below: about 2.3–4.8 h;
- today: about 4.5–14 h, before box time.
| # | Cut | Where | Saving | Effort |
|---|---|---|---|---|
| Pf1 | Click timeout 30 s → 5 s with a real wait (= D11, known) | `pages/claim_form.py:349-376` | about 15–30 s per claim | S |
| Pf2 | A not-found claim exits early once Total Counts reads a stable "0", and the single-claim search gets its own `CLAIM_SEARCH_TIMEOUT_MS` | `pages/claim_results.py:249-272`, `settings.py:263` | about 165 s per missed claim; removes the silent 3-min "hang" | S |
| Pf3 | Progress-note poll instead of a 3 s sleep (= D8); drop the success screenshot (= P11) | `pages/claim_form.py:631, 664` | about 2–3 s per claim plus fewer false Cleans | S |
| Pf4 | Single-claim search polls every 500 ms (`SEARCH_POLL_MS`), accepting only after the loader was seen and cleared; the bulk lookup keeps 3 s | `pages/claim_results.py:254, 262`, `settings.py:264` | about 1.5–3 s per claim; stops 0 s stale-grid matches | S |
| Pf5 | Gate the Dental box and add unattended mode (= P6, D9) | `flows/export_medcor_ready.py:375` | up to 180 s per Dental claim unattended | S |
| Pf6 | Circuit breaker (= P3) | `flows/export_medcor_ready.py:478-484` | caps a session-loss cascade (about 300 s × remaining claims) | M |
| Pf7 | Close leaked note tabs in a `finally`; soak test of 150+ claims while logging `context.pages`, DOM size and heap; reset the page every N claims only if the soak test shows growth | `pages/claim_form.py:629-662` | Unknown. No run has gone past 17 claims, so long-run behaviour is untested. | M |
---
## 6. Known items still open
- **SAVE_CORRECTIONS=0 verification mode** (known). New: the code default and the template still enable writes (see P1).
- **G2025 / admin code priced at 0.00 after being added** (known). Still waiting on an operator ruling.
- **medicare_telehealth leaves other rows when G2025 is unconfirmed; priced_line_check may then check them** (known). New: the failure is also hidden from the box and the summary (see D6).
- **"Click reported failure, but the form is already open"** (known). New: it costs about 30 s per hit, and its fallback check does not wait (see D11).
- **301 unmapped payers; AARP Medicare Advantage grouping disputed** (known). New: telehealth writes 93 for these payers (D5), declines are reported Clean (D2), and a JSON typo disables 3 conditions (P9).
- **Condition #9 (Medicaid D codes) removed** (known). Nothing new.
---
## Recommended order
1. **Today, before any commit:** gitignore and move the PHI files (P12). Default writes to off, strip inline comments, add the mode banner, and make Cancel the default button (P1). Run the Cancel-really-discards check on one test claim (P16).
2. **Demo setup:** clear or replace `TEST_CLAIM_IDS_FILE`, add the `_TEST` filename, check the demo claims are still in the queue (D1), and add the unique-name fallback for the summary write (D10).
3. **What the client sees on screen:** Dental summary box, "and N more", Yes/No, one box per claim (D9). Do the 2-minute UB checkbox selector check (L3).
4. **Telehealth correctness:** skip rows that are already correct (D3), decline unmapped payers (D5), classify audio/video from structured fields (D4), and report a failed G2025 (D6).
5. **Read completeness:** HCFA/Dental data wait (D7), progress-note polling (D8), fixed `visible()`/`appears()` (P8), click timeout cut (D11).
6. **Truthful summary:** minimal `claim_outcome` "Review" mapping before the demo (D2). Then the full St1 outcome model with the taint-blocks-save rule (P2).
7. **Write safety for production:** close/save success by `any_claim_form_open()` with no cross-screen OK (P5), handler-anchored selectors (P7), dismiss confirms by default (P13), depression pointer guard (L1).
8. **Run resilience:** JSONL journal plus try/finally summary plus resume (P4/St2), recovery plus error≠not-found plus circuit breaker (P3), early not-found exit (Pf2).
9. **Production operating model:** `RUN_MODE` unattended plus exit codes (P6/St3), approval-model decision with the client (St4), validate `payer_groups.json` at startup (P9), 150-claim soak test (Pf7).
10. **Hardening and hygiene:** remove the TLS flag (P10), PHI minimisation and retention (P11), L2/L4/L5 logic fixes, docs (P15), `tests/run_all.py` (St7). Split claim_form.py (St5) after the demo.
docs/Sarva_RCM_API_Assessment.docx
BIN
View File
Binary file not shown.
docs/adr/0001-run-events-in-files.md
+25
View File
@@ -0,0 +1,25 @@
# Run events go to a file per Run, not to SQL
Every Run writes its events (eCW Login Attempt steps, Run Errors, Claim
Errors, Warnings, each with its time) to `Logs\events_<run>.jsonl`, and the
Portal's Runs page reads those files — while the Charge Summary rows, Portal
Users and Sign-in history live in SQL Server (`Charge_Validation`). We chose
files because the events matter most exactly when things go wrong: during
the eCW login (before the database is touched at all), when SQL Server is
down, and when a Run is killed or the browser crashes — a line flushed to a
file survives all of those, the same way `Logs\run_<ts>.log` already does.
## Considered Options
- **SQL table (`CV_RunEvent`)**: easier filtering across months, but a Run
with no database would lose its own failure record, and the bot's results
write is deliberately never fatal — events would inherit that "maybe".
- **Parse the existing run logs**: no bot change, but the log lines carry
no time, so login durations and error times could not be shown.
## Consequences
- The Runs page reads one small file per Run; if that ever gets slow, load
the files into SQL as a copy, keeping the files as the source.
- Portal Users and Sign-ins are in SQL because the Portal already needs SQL
for its results; a Portal with no database shows nothing either way.
docs/agents/domain.md
+42
View File
@@ -0,0 +1,42 @@
# Domain Docs
How the engineering skills read this repo's domain documentation before working on it. The repo
is single-context: one glossary and one ADR folder, both at the repo root.
## Before exploring, read these
- **`GLOSSARY.md`** at the repo root: the project's terms (VERIFY / LIVE-WRITE, Charge Summary,
condition, ...).
- **`docs/adr/`**: read the ADRs that touch the area you're about to work in.
If either is missing, **proceed silently**: don't flag its absence and don't suggest creating it
upfront. The `domain-modeling` skill creates them lazily, when a term or a decision actually gets
settled.
## File structure
```
/
├── GLOSSARY.md
├── docs/adr/
│ ├── 0001-verify-sets-boxes-on-screen-only.md
│ └── 0002-save-corrections-off-until-client-approves.md
└── src/ecw_bot/
```
## Use the glossary's vocabulary
When your output names a domain concept (in a Jira summary, a refactor proposal, a hypothesis, a
test name), use the term as `GLOSSARY.md` defines it, e.g. **VERIFY** / **LIVE-WRITE** for the run
modes, **Charge Summary** for the results sheet, **condition** for one rule module. Don't drift to
synonyms the glossary explicitly avoids.
If the concept you need isn't in the glossary yet, that's a signal: either you're inventing
language the project doesn't use (reconsider) or there's a real gap (note it for the
`domain-modeling` skill).
## Flag ADR conflicts
If your output contradicts an existing ADR, surface it explicitly rather than silently overriding:
> _Contradicts ADR-0001 (VERIFY sets boxes on screen only), but worth reopening because…_
docs/agents/issue-tracker.md
+37
View File
@@ -0,0 +1,37 @@
# Issue tracker: Jira (EEWA)
Issues and specs for this repo live in Jira, project **EEWA** on ecareindia.atlassian.net,
reached through the Atlassian MCP tools (cloudId `dbc481dd-f4cd-4204-9f15-02f26c6914a1`).
## Conventions
- Every ticket is a **Subtask** of **EEWA-5 "Charge Validation"** (EEWA-5 is itself a Task, not an
epic: a Task with parent EEWA-5 is refused as "Please select valid parent issue").
- Assignee: **Manikandan** (accountId `712020:a3b618bb-f8b4-4105-9f6f-ba5f88b15804`), set at creation.
- Statuses: To Do → In Progress → Done. Work that starts moves to In Progress; a ticket is closed
only when the operator says so.
- **No claim ids, no patient data, no credentials** in titles, descriptions or comments.
- Summary style: `<module or condition> (<doc row #>: <one line>)`,
e.g. `injection_admin_code (#20: add 96372 for injection codes)`.
- A spec is one Subtask whose description holds the spec. Its implementation tickets are separate
Subtasks of EEWA-5 that mention the spec ticket's key.
- Labels: when a skill asks for `ready-for-agent`, add the Jira label `ready-for-agent`.
## When a skill says "publish to the issue tracker"
Create it with `createJiraIssue` (project EEWA, issueTypeName `Subtask`, parent `EEWA-5`,
assignee as above, label `ready-for-agent`), in dependency order (blockers first).
## When a skill says "fetch the relevant ticket"
`getJiraIssue` with the key (e.g. `EEWA-21`), including its comments.
## Blocking edges
Use Jira's native "Blocks" link: `createIssueLink` with type `Blocks`, `inwardIssue` = the
blocker, `outwardIssue` = the ticket it blocks.
## Comments and updates
`addCommentToJiraIssue` for notes; `transitionJiraIssue` for status. Never close or edit the
parent EEWA-5.
docs/superpowers/plans/2026-09-15-payer-group-categorization.md
+910
View File
@@ -0,0 +1,910 @@
# Payer Group Categorization Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** After the bot exports the Medcor Ready claim report, automatically add a `Group Name` column (Commercial / Medicaid / Medicare / Dentaquest / Program / …) based on each row's `PAYER`, writing the result to a new file so the existing pipeline's file layout is untouched.
**Architecture:** A new `core/payer_groups.py` does an exact-string lookup against a hand-editable `payer_groups.json`. A new `core/spreadsheet.write_xlsx()` (stdlib only, mirroring the existing reader) writes the categorized rows to `Output\Medcor Ready_<date>.xlsx`. `flows/export_medcor_ready.py` wires the two together as a new, non-fatal step right after the existing claim-id read-back.
**Tech Stack:** Python 3.13, stdlib only (`zipfile`, `xml.etree`, `json`, `xml.sax.saxutils`) — no new third-party dependencies, matching the project's existing "no openpyxl" constraint.
**Spec:** [docs/superpowers/specs/2026-09-15-payer-group-categorization-design.md](../specs/2026-09-15-payer-group-categorization-design.md)
## Global Constraints
- No new third-party dependencies (project has none beyond `playwright`; `spreadsheet.py`'s whole existence is to avoid `openpyxl`).
- Tests are hand-rolled `case_*` functions collected via `globals()` and run by a `main()` — no pytest, matching every existing file under `tests\`.
- `Input\Claim Report_<date>.xlsx` must never be modified by this feature — `claim_file.py` and the charge-entry phase depend on its exact column layout.
- Payer matching is **exact string, case-sensitive, no fuzzy/substring logic** — an unmapped payer must be flagged, never guessed (see spec's "Unknown payer handling").
- Every new step must be non-fatal to the run — matches the existing philosophy in `flows/export_medcor_ready.py` (`_read_back_claim_ids`, `_read_claim_form`).
---
## Task 1: Initialize git
This project currently has no git history at all (verified: `git status` outside a repo). The plan's commit-per-task workflow needs a repo to commit into, so this task creates one and takes a single baseline commit of the existing code before any feature work starts. This is additive only — no existing file is changed.
**Files:**
- Create: `D:\Ecw Process\.git\` (via `git init`)
**Interfaces:** none — infrastructure only.
- [ ] **Step 1: Confirm there is no existing git repo**
Run: `git -C "D:\Ecw Process" status`
Expected: `fatal: not a git repository (or any of the parent directories): .git`
- [ ] **Step 2: Initialize the repository**
Run: `git -C "D:\Ecw Process" init`
Expected: `Initialized empty Git repository in D:/Ecw Process/.git/`
- [ ] **Step 3: Check `.gitignore` already covers the right things**
Run: `cat "D:\Ecw Process\.gitignore"`
Expected: it already excludes `.env`, `Logs\`, `Screenshots\`, `Output\`, `Input\` (per `README.md`'s Layout section, all four are described as gitignored). If any of those four is missing, add it before continuing — this baseline commit must not include patient data or credentials.
- [ ] **Step 4: Stage and commit the current codebase as a baseline**
```bash
cd "D:\Ecw Process"
git add -A
git status
```
Read the `git status` output before committing: confirm nothing under `Input\`, `Output\`, `Logs\`, `Screenshots\`, or `.env` is staged. If any of those appear, fix `.gitignore` and re-run `git add -A`.
```bash
git commit -m "chore: initialize git repository (baseline)"
```
---
## Task 2: `write_xlsx()` in `core/spreadsheet.py`
**Files:**
- Modify: `src/ecw_bot/core/spreadsheet.py`
- Test: `tests/test_spreadsheet_write.py` (new)
**Interfaces:**
- Produces: `spreadsheet.write_xlsx(path, rows: list[list]) -> None` — `rows` is a header row plus data rows, each a list of values (any type, written as `str()`). Raises on I/O failure. Creates parent directories if needed.
- Consumes: nothing new — sits next to the existing `read_rows(path) -> list`.
- [ ] **Step 1: Write the failing round-trip test**
Create `tests/test_spreadsheet_write.py`:
```python
"""
write_xlsx(): the write side of core/spreadsheet.py.
python tests\\test_spreadsheet_write.py
Round-trips through the existing `read_rows()` reader rather than asserting
on raw XML — what matters is that a file this writes is a file this project's
own reader (and real Excel) can open, not the exact bytes.
"""
import sys
import pathlib
import tempfile
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent / "src"))
from ecw_bot.core import spreadsheet # noqa: E402
TMP = pathlib.Path(tempfile.mkdtemp())
def case_write_then_read_round_trips():
rows = [
["CLAIMS#", "PAYER", "Group Name"],
["9107000", "Medicaid", "Medicaid"],
["9106000", "DentaQuest", "Dentaquest"],
]
path = TMP / "roundtrip.xlsx"
spreadsheet.write_xlsx(path, rows)
back = spreadsheet.read_rows(path)
assert back == rows, back
def case_special_characters_are_escaped():
rows = [["PATIENT", "NOTE"],
["Smith & Sons", "A < B > C"],
["O'Brien", '"quoted"']]
path = TMP / "escaped.xlsx"
spreadsheet.write_xlsx(path, rows)
back = spreadsheet.read_rows(path)
assert back == rows, back
def case_wide_row_uses_correct_column_letters():
"""27 columns exercises the single-letter -> AA rollover."""
header = [f"col{i}" for i in range(27)]
data = [str(i) for i in range(27)]
path = TMP / "wide.xlsx"
spreadsheet.write_xlsx(path, [header, data])
back = spreadsheet.read_rows(path)
assert back[1][26] == "26", back[1] # column AA (index 26)
assert back[1] == data, back[1]
def case_creates_parent_directory():
path = TMP / "nested" / "new_dir" / "file.xlsx"
assert not path.parent.exists()
spreadsheet.write_xlsx(path, [["A"], ["1"]])
assert path.exists()
def case_non_string_values_are_stringified():
rows = [["N"], [5], [3.5]]
path = TMP / "numbers.xlsx"
spreadsheet.write_xlsx(path, rows)
back = spreadsheet.read_rows(path)
assert back == [["N"], ["5"], ["3.5"]], back
CASES = [v for k, v in sorted(globals().items()) if k.startswith("case_")]
def main() -> int:
failures = 0
for fn in CASES:
name = fn.__name__[5:].replace("_", " ")
try:
fn()
print(f" ok {name}")
except AssertionError as exc:
print(f" FAIL {name}: {exc}")
failures += 1
except Exception as exc:
print(f" ERROR {name}: {type(exc).__name__}: {exc}")
failures += 1
print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(main())
```
- [ ] **Step 2: Run it to verify it fails**
Run: `python tests\test_spreadsheet_write.py`
Expected: `ERROR` on every case with `AttributeError: module 'ecw_bot.core.spreadsheet' has no attribute 'write_xlsx'`
- [ ] **Step 3: Implement `write_xlsx()` in `core/spreadsheet.py`**
Add these imports at the top of `src/ecw_bot/core/spreadsheet.py` (alongside the existing `import csv / re / zipfile` block):
```python
from xml.sax.saxutils import escape
```
Append this to the end of `src/ecw_bot/core/spreadsheet.py`:
```python
# =============================================================================
# Writing (the categorization step's output)
# =============================================================================
# The read side above has no dependency on openpyxl; the write side keeps that
# constraint too. What follows is the minimal set of OOXML parts real Excel
# actually requires to open a file without a "repair" prompt — more than the
# reader above needs (it only ever looks at xl/worksheets/sheet*.xml), because
# this file is meant to be opened by a person, not just this codebase.
_CONTENT_TYPES = (
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
'<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
'<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>'
'<Default Extension="xml" ContentType="application/xml"/>'
'<Override PartName="/xl/workbook.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml"/>'
'<Override PartName="/xl/worksheets/sheet1.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml"/>'
'</Types>'
)
_ROOT_RELS = (
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
'<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
'<Relationship Id="rId1" '
'Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" '
'Target="xl/workbook.xml"/>'
'</Relationships>'
)
_WORKBOOK = (
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
'<workbook xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main" '
'xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">'
'<sheets><sheet name="Sheet1" sheetId="1" r:id="rId1"/></sheets>'
'</workbook>'
)
_WORKBOOK_RELS = (
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
'<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
'<Relationship Id="rId1" '
'Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" '
'Target="worksheets/sheet1.xml"/>'
'</Relationships>'
)
def _col_letter(idx: int) -> str:
"""0 -> 'A', 25 -> 'Z', 26 -> 'AA'. Inverse of `_col_index` above."""
letters = ""
idx += 1
while idx > 0:
idx, rem = divmod(idx - 1, 26)
letters = chr(65 + rem) + letters
return letters
def write_xlsx(path, rows: list) -> None:
"""
Write `rows` (a header row plus data rows, each a list of values) as a
minimal single-sheet xlsx.
Inline `t="str"` cells, no sharedStrings — the same shape `_read_xlsx`
above already parses, and what eCW's own export already sends (see the
module docstring). Every value is written through `str()`, so callers can
pass numbers or None without pre-formatting.
"""
path = Path(path)
body = []
for r, row in enumerate(rows, start=1):
cells = []
for c, value in enumerate(row):
text = escape(str(value))
cells.append(f'<c r="{_col_letter(c)}{r}" t="str"><v>{text}</v></c>')
body.append(f'<row r="{r}">{"".join(cells)}</row>')
sheet = (
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
'<worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main">'
f'<sheetData>{"".join(body)}</sheetData></worksheet>'
)
path.parent.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as zf:
zf.writestr("[Content_Types].xml", _CONTENT_TYPES)
zf.writestr("_rels/.rels", _ROOT_RELS)
zf.writestr("xl/workbook.xml", _WORKBOOK)
zf.writestr("xl/_rels/workbook.xml.rels", _WORKBOOK_RELS)
zf.writestr("xl/worksheets/sheet1.xml", sheet)
```
- [ ] **Step 4: Run the test to verify it passes**
Run: `python tests\test_spreadsheet_write.py`
Expected: `5/5 passed`
- [ ] **Step 5: Sanity-check the file actually opens in Excel/WPS**
This machine has Excel/WPS installed (seen in the video). Manually:
```bash
python -c "import sys; sys.path.insert(0,'src'); from ecw_bot.core import spreadsheet as s; s.write_xlsx('Output/_manual_check.xlsx', [['A','B'],['1','2']])"
```
Open `Output\_manual_check.xlsx` by hand and confirm it opens with no "repair" prompt and shows the two cells. Delete the file afterward — it's a throwaway check, not a fixture.
- [ ] **Step 6: Commit**
```bash
cd "D:\Ecw Process"
git add src/ecw_bot/core/spreadsheet.py tests/test_spreadsheet_write.py
git commit -m "feat: add write_xlsx() to core/spreadsheet.py"
```
---
## Task 3: `core/payer_groups.py` + seed `payer_groups.json`
**Files:**
- Create: `src/ecw_bot/core/payer_groups.py`
- Create: `src/ecw_bot/payer_groups.json`
- Test: `tests/test_payer_groups.py` (new)
**Interfaces:**
- Produces: `payer_groups.load_mapping(path=None) -> dict[str, str]`, `payer_groups.classify(payer: str, mapping: dict) -> str | None`.
- Consumes: nothing from Task 2.
- [ ] **Step 1: Write the failing tests**
Create `tests/test_payer_groups.py`:
```python
"""
Payer -> Group Name lookup.
python tests\\test_payer_groups.py
Exact-string match only, on purpose: eCW's own PAYER values already differ by
case and spacing for what is really the same carrier (`United Healthcare` /
`United Health Care` / `United healthcare` all appear in one real export), so
"close enough" matching would have to guess which of several real, distinct
strings a new one is closest to. Billing data does not get a guess — an
unrecognized string must come back None and get a human's attention.
"""
import sys
import json
import pathlib
import tempfile
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent / "src"))
from ecw_bot.core import payer_groups # noqa: E402
TMP = pathlib.Path(tempfile.mkdtemp())
MAPPING = {
"Medicaid": "Medicaid",
"DentaQuest": "Dentaquest",
"Anthem Blue Cross Blue Shield": "Commercial",
}
def case_exact_match_returns_group():
assert payer_groups.classify("Medicaid", MAPPING) == "Medicaid"
assert payer_groups.classify("DentaQuest", MAPPING) == "Dentaquest"
def case_unmapped_payer_returns_none():
assert payer_groups.classify("Some New Payer LLC", MAPPING) is None
def case_case_sensitive_no_fuzzy_matching():
"""Different case is a different string, on purpose — see module docstring."""
assert payer_groups.classify("medicaid", MAPPING) is None
assert payer_groups.classify("MEDICAID", MAPPING) is None
def case_blank_or_none_payer_is_unmapped():
assert payer_groups.classify("", MAPPING) is None
assert payer_groups.classify(None, MAPPING) is None
def case_load_mapping_reads_json_file():
path = TMP / "mapping.json"
path.write_text(json.dumps({"Medicaid": "Medicaid"}), encoding="utf-8")
loaded = payer_groups.load_mapping(path)
assert loaded == {"Medicaid": "Medicaid"}, loaded
def case_load_mapping_missing_file_raises():
try:
payer_groups.load_mapping(TMP / "does_not_exist.json")
except FileNotFoundError as exc:
assert "does_not_exist.json" in str(exc), exc
return
raise AssertionError("expected FileNotFoundError")
def case_seed_config_loads_and_covers_medicaid_and_dentaquest():
"""The real config file this ships with, not a fixture."""
mapping = payer_groups.load_mapping()
assert mapping.get("Medicaid") == "Medicaid", mapping.get("Medicaid")
assert mapping.get("DentaQuest") == "Dentaquest", mapping.get("DentaQuest")
CASES = [v for k, v in sorted(globals().items()) if k.startswith("case_")]
def main() -> int:
failures = 0
for fn in CASES:
name = fn.__name__[5:].replace("_", " ")
try:
fn()
print(f" ok {name}")
except AssertionError as exc:
print(f" FAIL {name}: {exc}")
failures += 1
except Exception as exc:
print(f" ERROR {name}: {type(exc).__name__}: {exc}")
failures += 1
print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(main())
```
- [ ] **Step 2: Run it to verify it fails**
Run: `python tests\test_payer_groups.py`
Expected: `ERROR` on every case — `ModuleNotFoundError: No module named 'ecw_bot.core.payer_groups'`
- [ ] **Step 3: Create the seed config, `src/ecw_bot/payer_groups.json`**
```json
{
"Medicaid": "Medicaid",
"DentaQuest": "Dentaquest",
"CHP+ DENTAQUEST": "Dentaquest",
"AARP Medicare Advantage": "Medicare",
"Aetna Medicare ADV": "Medicare",
"OUTREACH PROGRAM": "Program",
"Anthem Blue Cross Blue Shield": "Commercial",
"Cigna HealthCare Of Colorado": "Commercial",
"Cigna Healthcare of CO": "Commercial",
"Cigna Salud Employee": "Commercial",
"Cigna": "Commercial",
"Cypress Benefit Admin / Cigna PPO": "Commercial",
"United Healthcare": "Commercial",
"United Health Care": "Commercial",
"United healthcare": "Commercial",
"Unitedhealth One": "Commercial",
"Oxford Health Insurance UHC": "Commercial",
"UMR": "Commercial",
"Aetna": "Commercial",
"Aetna Alaska Care": "Commercial",
"AETNA HEALTH AND LIFE INSURANCE": "Commercial",
"BIND BENEFITS INC": "Commercial",
"Select Health": "Commercial"
}
```
This is the "Mapped (seeded)" table from the spec, byte-for-byte. The 17 payers in the spec's "Left unmapped" list are deliberately absent — `classify()` returning `None` for them, and the flow logging/flagging that, is the intended behavior until a human adds them.
- [ ] **Step 4: Implement `core/payer_groups.py`**
```python
"""
Payer -> Group Name lookup for the categorization step.
Exact-string match against a small hand-edited table (`payer_groups.json`,
next to this package) — no fuzzy or substring matching. eCW's own PAYER
values already differ by case and spacing for what is really the same
carrier (see `tests/test_payer_groups.py`'s docstring), so "close enough"
matching has no safe way to pick which known string a new one is closest to.
An unrecognized payer must come back as None and get a human's attention —
never a guess, on billing data.
Adding a payer is a one-line edit to `payer_groups.json`, no code change.
"""
import json
from pathlib import Path
_CONFIG_PATH = Path(__file__).resolve().parent.parent / "payer_groups.json"
def load_mapping(path=None) -> dict:
"""The payer -> group table. `path` defaults to the shipped config."""
p = Path(path) if path else _CONFIG_PATH
if not p.exists():
raise FileNotFoundError(
f"{p} not found — the payer group table is missing")
return json.loads(p.read_text(encoding="utf-8"))
def classify(payer: str, mapping: dict) -> str:
"""The Group Name for this exact PAYER string, or None if unmapped."""
return mapping.get((payer or "").strip())
```
- [ ] **Step 5: Run the tests to verify they pass**
Run: `python tests\test_payer_groups.py`
Expected: `7/7 passed`
- [ ] **Step 6: Commit**
```bash
cd "D:\Ecw Process"
git add src/ecw_bot/core/payer_groups.py src/ecw_bot/payer_groups.json tests/test_payer_groups.py
git commit -m "feat: add payer_groups lookup module and seed config"
```
---
## Task 4: Wire the categorization step into the export flow
**Files:**
- Modify: `src/ecw_bot/settings.py`
- Modify: `.env.example`
- Modify: `src/ecw_bot/flows/export_medcor_ready.py`
- Test: `tests/test_categorize_by_payer.py` (new)
**Interfaces:**
- Consumes: `spreadsheet.read_rows`, `spreadsheet.write_xlsx` (Task 2); `payer_groups.load_mapping`, `payer_groups.classify` (Task 3).
- Produces: `export_medcor_ready._categorize_by_payer(saved: Path | None) -> Path | None` — writes `Output\Medcor Ready_<date>.xlsx` and returns its path, or `None` if the step could not run (never raises).
- [ ] **Step 1: Add the new setting**
In `src/ecw_bot/settings.py`, immediately after the `SHOW_CLAIM_IDS` setting (currently ending around line 311, just before the `# ── Per-claim processing ──` section header), add:
```python
# After categorizing by payer, tell the operator about any PAYER value with
# no entry in payer_groups.json — those rows are saved with a blank Group
# Name rather than a guess. Informational only, same reasoning as
# NOTIFY_AFTER_EXPORT above. Set 0 for unattended runs; the full list still
# goes to the log either way.
NOTIFY_UNMAPPED_PAYERS = _env_bool("NOTIFY_UNMAPPED_PAYERS", True)
```
- [ ] **Step 2: Add the matching line to `.env.example`**
Open `.env.example`, find the block documenting `SHOW_CLAIM_IDS` (search for that key), and add directly after it:
```
# After categorizing the export by payer, show a box listing any PAYER value
# with no entry in src/ecw_bot/payer_groups.json (those rows get a blank
# Group Name, not a guess). 1 = show it, 0 = log only.
NOTIFY_UNMAPPED_PAYERS=1
```
Match whatever `1`/`0` convention the surrounding boolean settings in that file already use — check `SHOW_CLAIM_IDS`'s line right above it and mirror its exact style so the two files can't drift the way the earlier `OTP_WAIT_SECONDS` mismatch did.
- [ ] **Step 3: Write the failing test for `_categorize_by_payer`**
Create `tests/test_categorize_by_payer.py`:
```python
"""
export_medcor_ready._categorize_by_payer(): the payer -> Group Name step.
python tests\\test_categorize_by_payer.py
Builds a real xlsx (same helper shape as test_claim_file.py's `xlsx()`) so
this exercises the actual read -> classify -> write path, not a mock of it.
"""
import sys
import zipfile
import pathlib
import tempfile
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent / "src"))
from ecw_bot.flows import export_medcor_ready as flow # noqa: E402
from ecw_bot.core import spreadsheet # noqa: E402
from ecw_bot import settings as cfg # noqa: E402
TMP = pathlib.Path(tempfile.mkdtemp())
NS = 'xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"'
def xlsx(header, data_rows, name="t.xlsx") -> pathlib.Path:
"""A minimal real xlsx: header + data rows, both plain string lists."""
rows = [header] + data_rows
body = []
for r, row in enumerate(rows, start=1):
cells = "".join(
f'<c r="{chr(65 + c)}{r}" t="str"><v>{v}</v></c>'
for c, v in enumerate(row))
body.append(f'<row r="{r}">{cells}</row>')
sheet = (f'<?xml version="1.0"?><worksheet {NS}><sheetData>'
f'{"".join(body)}</sheetData></worksheet>')
path = TMP / name
with zipfile.ZipFile(path, "w") as z:
z.writestr("xl/worksheets/sheet1.xml", sheet)
return path
HEADER = ["CLAIMS#", "PAYER", "CHARGES"]
def case_categorizes_known_payers():
saved = xlsx(HEADER, [
["9107000", "Medicaid", "275.00"],
["9106000", "DentaQuest", "117.00"],
], name="known.xlsx")
out = flow._categorize_by_payer(saved)
assert out is not None and out.exists(), out
rows = spreadsheet.read_rows(out)
assert rows[0] == HEADER + ["Group Name"], rows[0]
assert rows[1] == ["9107000", "Medicaid", "275.00", "Medicaid"], rows[1]
assert rows[2] == ["9106000", "DentaQuest", "117.00", "Dentaquest"], rows[2]
def case_unmapped_payer_gets_blank_group_not_a_guess():
saved = xlsx(HEADER, [
["9108000", "Some Brand New Payer", "50.00"],
], name="unmapped.xlsx")
out = flow._categorize_by_payer(saved)
rows = spreadsheet.read_rows(out)
assert rows[1][-1] == "", rows[1] # blank, not a guessed group
def case_none_input_is_a_no_op():
assert flow._categorize_by_payer(None) is None
def case_missing_payer_column_does_not_raise():
saved = xlsx(["CLAIMS#", "CHARGES"], [["9108000", "50.00"]],
name="nopayer.xlsx")
assert flow._categorize_by_payer(saved) is None # skipped, not a crash
CASES = [v for k, v in sorted(globals().items()) if k.startswith("case_")]
def main() -> int:
failures = 0
for fn in CASES:
name = fn.__name__[5:].replace("_", " ")
try:
fn()
print(f" ok {name}")
except AssertionError as exc:
print(f" FAIL {name}: {exc}")
failures += 1
except Exception as exc:
print(f" ERROR {name}: {type(exc).__name__}: {exc}")
failures += 1
print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(main())
```
- [ ] **Step 4: Run it to verify it fails**
Run: `python tests\test_categorize_by_payer.py`
Expected: `ERROR` on every case — `AttributeError: module 'ecw_bot.flows.export_medcor_ready' has no attribute '_categorize_by_payer'`
- [ ] **Step 5: Implement `_categorize_by_payer` in `flows/export_medcor_ready.py`**
Change the import line near the top (currently `from ..core import guard, runlog, msgbox`) to:
```python
from ..core import guard, runlog, msgbox, spreadsheet, payer_groups
```
Add this new function directly after `_read_back_claim_ids` (which ends at the `return info` around line 85), before `_read_claim_form`:
```python
def _categorize_by_payer(saved):
"""
Add a Group Name column (Commercial / Medicaid / Medicare / Dentaquest /
Program / ...) next to the saved report, based on each row's PAYER, and
write it to Output\\Medcor Ready_<date>.xlsx.
Never touches `saved` itself: claim_file.py and everything after it reads
that file's columns by position (see claim_file.py's module docstring),
so this always writes a SEPARATE file rather than editing it in place.
Never fatal. An unmapped payer, a missing PAYER column, or a write
failure is worth reporting loudly, not worth losing the already-saved
report over — same reasoning as `_read_back_claim_ids` above.
"""
if saved is None:
return None
try:
rows = spreadsheet.read_rows(saved)
except Exception as exc:
print(f"[CATEGORIZE] Could not read {saved.name}: {exc}")
return None
if not rows:
print(f"[CATEGORIZE] {saved.name} has no rows — skipping")
return None
header, data = rows[0], rows[1:]
if "PAYER" not in header:
print(f"[CATEGORIZE] No PAYER column in {saved.name} — skipping. "
f"Header reads: {header}")
return None
payer_col = header.index("PAYER")
try:
mapping = payer_groups.load_mapping()
except Exception as exc:
print(f"[CATEGORIZE] Could not load payer_groups.json: {exc}")
return None
out_rows = [header + ["Group Name"]]
unmapped = {}
mapped_count = 0
for row in data:
payer = row[payer_col] if payer_col < len(row) else ""
group = payer_groups.classify(payer, mapping)
if group:
mapped_count += 1
else:
unmapped[payer] = unmapped.get(payer, 0) + 1
out_rows.append(list(row) + [group or ""])
out_path = cfg.ROOT / "Output" / f"Medcor Ready_{cfg.TODAY}.xlsx"
try:
spreadsheet.write_xlsx(out_path, out_rows)
except Exception as exc:
print(f"[CATEGORIZE] Could not write {out_path.name}: {exc}")
return None
print(f"[CATEGORIZE] {mapped_count} of {len(data)} row(s) categorized "
f"-> {out_path.name}")
if unmapped:
listed = sorted(unmapped.items(), key=lambda kv: -kv[1])
print(f"[CATEGORIZE] {sum(unmapped.values())} row(s) with no group "
f"in payer_groups.json:")
for payer, count in listed:
print(f"[CATEGORIZE] {count:4d} {payer!r}")
if cfg.NOTIFY_UNMAPPED_PAYERS:
body = (f"{sum(unmapped.values())} row(s)-ku Group Name "
f"kidaikkala.\n\n"
f"{out_path.name} file-la '' (blank) Group Name-oda "
f"irukkum. Idha payer_groups.json-la add pannunga:\n\n"
+ "\n".join(f" {p} ({n} row)" for p, n in listed[:15]))
if len(listed) > 15:
body += f"\n ... innum {len(listed) - 15}"
msgbox.info_box("eCW Bot — Group Name podala", body)
return out_path
```
Add the call into `_claims_phase` — change:
```python
saved = claim_report.export_claims_report(page)
info = _read_back_claim_ids(saved)
if info:
_process_claims(page, info)
```
to:
```python
saved = claim_report.export_claims_report(page)
info = _read_back_claim_ids(saved)
_categorize_by_payer(saved)
if info:
_process_claims(page, info)
```
Finally, update the module's own flow docstring at the top of the file — change:
```
→ Input\Claim Report_<MM-DD-YYYY>.xlsx
→ read the file back, pull the CLAIMS# column
→ search each claim id on its own (CLAIM_PROCESS_LIMIT)
```
to:
```
→ Input\Claim Report_<MM-DD-YYYY>.xlsx
→ read the file back, pull the CLAIMS# column
→ categorize by payer → Output\Medcor Ready_<MM-DD-YYYY>.xlsx
→ search each claim id on its own (CLAIM_PROCESS_LIMIT)
```
- [ ] **Step 6: Run the test to verify it passes**
Run: `python tests\test_categorize_by_payer.py`
Expected: `4/4 passed`
- [ ] **Step 7: Run the full existing test suite to confirm nothing else broke**
```bash
python tests\test_settings.py
python tests\test_guard.py
python tests\test_dead_end.py
python tests\test_claim_file.py
python tests\test_claim_form.py
python tests\test_progress_note.py
python tests\test_input_archive.py
python tests\test_spreadsheet_write.py
python tests\test_payer_groups.py
python tests\test_categorize_by_payer.py
```
Expected: every one reports all cases passed. `test_settings.py` in particular asserts the module graph stays acyclic and that no selector leaks into `settings` — confirm it still passes since this task touched `settings.py`.
- [ ] **Step 8: Commit**
```bash
cd "D:\Ecw Process"
git add src/ecw_bot/settings.py .env.example src/ecw_bot/flows/export_medcor_ready.py tests/test_categorize_by_payer.py
git commit -m "feat: categorize Medcor Ready export by payer after each run"
```
---
## Task 5: Document it
**Files:**
- Modify: `README.md`
- Modify: `CHANGELOG.md`
**Interfaces:** none — documentation only.
- [ ] **Step 1: Add a short section to `README.md`**
Find the "### Reading the claim ids back" section (the last subsection under "## Claim lookup", ending just before "## Known gaps"). Add a new subsection directly after it:
```markdown
### Categorizing by payer
Right after the claim ids are read back, every row of the just-saved report
gets a `Group Name` (`Commercial`, `Medicaid`, `Medicare`, `Dentaquest`,
`Program`, ...) based on its `PAYER`, and the result is saved to
`Output\Medcor Ready_<MM-DD-YYYY>.xlsx`. The original `Input\` file is never
touched — `claim_file.py` and the charge-entry phase depend on its exact
column layout.
The mapping is `src\ecw_bot\payer_groups.json` — a flat, hand-editable
`{"exact PAYER string": "Group Name"}` table. **Exact match only, no fuzzy
matching**: eCW's own PAYER values already differ by case and spacing for
what is really the same carrier (`United Healthcare` / `United Health Care`
/ `United healthcare` are three separate real values), so a PAYER string
with no entry gets a **blank** Group Name rather than a guess. Unmapped rows
are logged and, unless `NOTIFY_UNMAPPED_PAYERS=0`, shown in a message box —
adding one is a one-line edit to the JSON file, no code change.
Never fatal: a missing PAYER column, an unreadable report, or a write
failure is logged and the run continues — the export already succeeded by
the time this step runs.
```
- [ ] **Step 2: Add a CHANGELOG entry**
Append to the end of `CHANGELOG.md`:
```markdown
---
## Day 4 — 2026-09-15
**Automated the payer categorization step.** Reviewed against
`Salud-Charge Entry_Medcor Ready.mp4` (2:45-10:00): after the Medcor Ready
export, an operator was filtering the report by `PAYER` and hand-filling a
`Group Name` column (`Commercial`, `Medicaid`, `Medicare`, `Dentaquest`,
`Program`, ...) — the same manual, repetitive, rule-based step every other
phase of this bot exists to remove.
`core/payer_groups.py` does the lookup: exact string match against
`payer_groups.json`, on purpose — eCW's own PAYER values already differ by
case and spacing for what is really the same carrier (`United Healthcare` /
`United Health Care` / `United healthcare` are three separate values in one
real export), so anything looser has no safe way to pick which known string
a new one is closest to. An unrecognized payer gets a **blank** Group Name,
not a guess, and is logged and message-boxed — this is billing data, and
wrong is worse than blank.
`core/spreadsheet.py` gained a write side (`write_xlsx()`) to go with its
read side, built the same way — stdlib only, no openpyxl — but with the full
minimal OOXML part set (`[Content_Types].xml`, both `.rels` files,
`workbook.xml`) the reader doesn't need but real Excel does, since this file
is meant to be opened by a person.
The categorized output is a **new** file, `Output\Medcor Ready_<date>.xlsx`
— `Input\Claim Report_<date>.xlsx` is never modified, because `claim_file.py`
and the charge-entry phase read its columns by position.
**Seeded, not complete.** The mapping table ships with 23 payers covering
roughly 95% of a real export's row volume (`Medicaid`, `DentaQuest`, and the
clearly-commercial carriers), derived from the actual current export rather
than guessed from the video. 17 payers (`BH COLORADO ACCESS`, the Delta
Dental variants, `Tricare`, `Devoted`, several TPA names, ...) are
deliberately left out — first run will flag them, and filling them in is a
one-line JSON edit, no redeploy.
**Out of scope for this round:** splitting rows into the `Patient` /
`Pending with Error` / `Program` sibling workbooks the video also shows, and
writing anything to the network `Charge\Allocation\...` folder — both stay
manual for now.
```
- [ ] **Step 3: Commit**
```bash
cd "D:\Ecw Process"
git add README.md CHANGELOG.md
git commit -m "docs: document the payer categorization step"
```
---
## Self-review notes
- **Spec coverage:** architecture (Task 2-4), components (`payer_groups.py` Task 3, `write_xlsx` Task 2, flow wiring Task 4), data flow (Task 4 Step 5), error handling (unmapped payer / missing file / write failure, all in Task 4 Step 5 and covered by tests), testing (every task has its own `test_*.py`), seed mapping table (Task 3 Step 3, copied verbatim from the spec). All covered.
- **Placeholder scan:** no TBD/TODO; every step has real, complete code.
- **Type consistency:** `classify(payer, mapping)` signature is identical everywhere it's called (Task 3 tests, Task 4's `_categorize_by_payer`); `load_mapping(path=None)` likewise.
- **Scope:** single subsystem (categorization only), matches the spec's explicit out-of-scope list — no split-into-3-files logic anywhere in this plan.
docs/superpowers/plans/2026-10-08-results-db-and-web-page.md
+108
View File
@@ -0,0 +1,108 @@
# Results DB + Web Page Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Store every run's Charge Summary rows in SQL Server (`Charge_Validation.dbo.ChargeSummary`) and show them on a local FastAPI page.
**Architecture:** A pure module `core/results_db.py` turns Charge Summary rows (the same rows the Excel file gets) into typed DB records and inserts them with one parameterized statement; the flow calls it after the Excel write, behind `RESULTS_DB`, and any DB error is a log line only. A separate `web/` FastAPI app reads the table through a small repository class and renders one Jinja2 page plus two JSON endpoints, bound to 127.0.0.1.
**Tech Stack:** Python 3.13, pyodbc + ODBC Driver 18 for SQL Server, FastAPI 0.141 + Jinja2 + uvicorn, Starlette TestClient (httpx). All already installed.
**Spec:** `docs/superpowers/specs/2026-10-08-results-db-and-web-page-design.md`
## Global Constraints
- Server `MECARE331\MECARE331`, database `Charge_Validation`, table `dbo.ChargeSummary`.
- Default connection: `DRIVER={ODBC Driver 18 for SQL Server};SERVER=MECARE331\MECARE331;DATABASE=Charge_Validation;Trusted_Connection=yes;TrustServerCertificate=yes;`
- `RESULTS_DB` default **off**. The bot never creates/alters tables; the operator runs `tools/sql/create_tables.sql` in SSMS.
- A DB failure never stops the run and never touches the Excel summary.
- Only parameterized SQL; web is read-only (`SELECT` only).
- Web bound to `127.0.0.1` only. No external scripts/CDNs.
- No new entries in `requirements.txt`. Tests are standalone scripts (`python tests\test_x.py`, prints `N/M passed`), no pytest, no real DB.
- Never commit unless the operator asks; never commit `run.py`, `Input/`, `Output/`, `Logs/`, `Screenshots/`, `Dom.txt`, client `.xlsx`.
## Rulings (plan vs spec)
- Ruling: the flow builds the sheet once more for the DB (`charge_summary.build`) instead of threading one sheet through `write()` — keeps `write()` and its tests unchanged; both builds read the same report and outcomes on the same day, so the rows are identical — cost if wrong: a row differing between Excel and DB, visible on the page.
- Ruling: only rows the run filled in (non-blank Category) are saved — unprocessed / not-found rows are blank in the Excel too and would only clutter the page — cost if wrong: the page lacks blank rows the Excel has.
## File map
| File | Responsibility |
|---|---|
| `tools/sql/create_tables.sql` | Table + indexes, run by the operator |
| `src/ecw_bot/settings.py` | `RESULTS_DB`, `RESULTS_DB_CONN` |
| `src/ecw_bot/core/results_db.py` | Column map, value conversion, `details_for`, `insert_rows`, `save_summary` |
| `src/ecw_bot/core/charge_summary.py` | `build()` split out of `write()` |
| `src/ecw_bot/flows/export_medcor_ready.py` | `changes` in `_conditions`, `details` dict, `_run_mode()`, `_save_results_db()` |
| `web/__init__.py`, `web/repo.py`, `web/app.py`, `web/templates/index.html` | Read-only page + JSON API |
| `run_web.bat` | Start the page on 127.0.0.1:8000 |
| `tests/test_results_db.py`, `tests/test_web_app.py` | Tests |
---
### Task 1: SQL script, settings, `results_db` module
**Files:**
- Create: `tools/sql/create_tables.sql`, `src/ecw_bot/core/results_db.py`, `tests/test_results_db.py`
- Modify: `src/ecw_bot/settings.py` (after `SUMMARY_ASSIGNED_TO`)
**Interfaces:**
- Produces: `results_db.COLUMNS: list[tuple[str, str]]` (header, db column), `DB_COLUMNS: list[str]`, `TABLE = "dbo.ChargeSummary"`, `to_money(v) -> Decimal|None`, `to_date(v) -> date|None`, `to_record(row: dict, *, run_at, run_mode, details) -> dict`, `details_for(data: dict) -> dict`, `insert_rows(conn, records) -> int`, `save_summary(sheet, details, *, run_at, run_mode, conn_str, connect=None) -> int|None`; settings `RESULTS_DB: bool`, `RESULTS_DB_CONN: str`.
- [ ] **Step 1: Write the failing tests** — `tests/test_results_db.py` with cases: every HEADER mapped (24) and `DB_COLUMNS` = run_at, run_mode, 24 columns, details_json; `to_money("$1,234.50") == Decimal("1234.50")`, `"(12.00)"` → -12.00, blank/"abc"/"NaN"/"1e20" → None, `365` → 365.00; `to_date("7/31/2026") == date(2026,7,31)`, ISO and datetime accepted, blank/junk → None; `to_record` converts types, blank text → None, `details_json` is JSON or None; `details_for` copies `_conditions` keys, Dental results, error keys; `insert_rows` sends one `INSERT INTO dbo.ChargeSummary (...) VALUES (?, ...)` with values in `DB_COLUMNS` order and commits once; `save_summary` skips blank-Category rows, closes the connection, returns the count; a raising `connect` returns None and prints `[RESULTS-DB]`. (Full code: see the test file written in Step 1 of execution.)
- [ ] **Step 2: Run** `python tests\test_results_db.py` — Expected: ImportError (no `results_db`).
- [ ] **Step 3: Implement** `results_db.py` per the Interfaces above (money: strip `$ ,`, `(x)` negative, finite and `< 10**10`, quantize 0.01; dates: `mm/dd/yyyy`, `yyyy-mm-dd`, `date`/`datetime`; `INSERT` built only from `DB_COLUMNS` names and `?` markers; `save_summary` wraps everything in `try/except Exception` → `[RESULTS-DB] Could not save ...` + None). Settings:
```python
# ── Results database (SQL Server) ─────────────────────────────────────────────
# Also save the Charge Summary rows to SQL Server (core/results_db.py) for the
# results web page (web/). Off by default; a DB error is a log line only.
RESULTS_DB = _env_bool("RESULTS_DB", False)
RESULTS_DB_CONN = _env(
"RESULTS_DB_CONN",
"DRIVER={ODBC Driver 18 for SQL Server};SERVER=MECARE331\\MECARE331;"
"DATABASE=Charge_Validation;Trusted_Connection=yes;"
"TrustServerCertificate=yes;")
```
and `tools/sql/create_tables.sql` = the spec's `CREATE TABLE` wrapped in `IF OBJECT_ID('dbo.ChargeSummary','U') IS NULL BEGIN ... END`.
- [ ] **Step 4: Run** `python tests\test_results_db.py` — Expected: all passed.
### Task 2: `charge_summary.build()` and the flow hook
**Files:**
- Modify: `src/ecw_bot/core/charge_summary.py` (`write()`), `src/ecw_bot/flows/export_medcor_ready.py` (`_read_claim_form` `_conditions`, claim loop, after `_write_charge_summary`)
- Test: `tests/test_results_db.py` (flow cases), `tests/test_charge_summary.py` (unchanged, must stay green)
**Interfaces:**
- Consumes: `results_db.details_for`, `results_db.save_summary`, `cfg.RESULTS_DB`, `cfg.RESULTS_DB_CONN`.
- Produces: `charge_summary.build(report_path, outcomes, assigned_to) -> list` (header + rows); flow `_run_mode() -> str` ("TEST" / "LIVE-WRITE" / "VERIFY"), `_save_results_db(info, outcomes, details) -> None`.
- [ ] **Step 1: Failing tests** in `tests/test_results_db.py`: `_run_mode()` for the three settings combinations; `_save_results_db` with `RESULTS_DB` off calls nothing; on, it calls `charge_summary.build(info["file"], outcomes, cfg.SUMMARY_ASSIGNED_TO)` and `results_db.save_summary(sheet, details, run_at=<datetime>, run_mode=..., conn_str=cfg.RESULTS_DB_CONN)`; a raising `build` prints `[RESULTS-DB]` and does not raise.
- [ ] **Step 2: Run** — Expected: AttributeError (`_run_mode` missing).
- [ ] **Step 3: Implement**: move `write()`'s read+`build_rows` into `build()`, `write()` calls `build()`; in `_read_claim_form` add `"changes": {r.condition_name: r.changes for r in results}` to `data["_conditions"]`; in the claim loop `details = {}` and `details[claim_id] = results_db.details_for(data)`; after `_write_charge_summary(info, outcomes)` call `_save_results_db(info, outcomes, details)`.
- [ ] **Step 4: Run** `tests\test_results_db.py`, `tests\test_charge_summary.py`, `tests\test_claim_boxes.py` — Expected: all passed.
### Task 3: Web page
**Files:**
- Create: `web/__init__.py`, `web/repo.py`, `web/app.py`, `web/templates/index.html`, `run_web.bat`, `tests/test_web_app.py`
**Interfaces:**
- Consumes: `results_db.COLUMNS`, `results_db.TABLE`, `cfg.RESULTS_DB_CONN`.
- Produces: `web.repo.SqlResultsRepo(conn_str, connect=None)` with `runs() -> list[dict]` (`run_at, run_mode, claims, pass_count, fail_count, review_count`, newest first) and `results(run_at, category=None, claim_no=None) -> list[dict]`; `web.app.create_app(repo) -> FastAPI`; module-level `web.app.app`.
- [ ] **Step 1: Failing tests** `tests/test_web_app.py` with a fake repo: `/` lists rows of the latest run, shows Pass/Fail/Review counts and the Charge Summary headers; `?category=Review` passes the filter to the repo; `?category=Bad` → 400; `?claim_no=55-51` searches `5551`… digits only; `?run_at=not-a-date` → 400; `/api/runs` and `/api/results` JSON shapes (dates ISO, money numbers); repo raising → 503 with "Results database not reachable"; no runs → page says no results yet; `SqlResultsRepo.results` builds a parameterized query (fake connection) with `category = ?` and `claim_no LIKE ?`.
- [ ] **Step 2: Run** — Expected: ImportError.
- [ ] **Step 3: Implement** repo, app (`docs_url=None`), template (form: run select, category select, claim search; tiles; table in Charge Summary order with Category colour; `<details>` per row with pretty JSON), `run_web.bat` (`python -m uvicorn web.app:app --host 127.0.0.1 --port 8000`).
- [ ] **Step 4: Run** `python tests\test_web_app.py` — Expected: all passed.
### Task 4: Docs and full suite
- [ ] Context.md: a "Results DB + web page" section (setup: run `tools/sql/create_tables.sql`, grant the Windows login read/write, `RESULTS_DB=1`, `run_web.bat`).
- [ ] Full suite: `bash run_tests.sh` → TOTAL all passed.
- [ ] Report to the operator; commit only when asked.
docs/superpowers/plans/2026-10-14-start-bot-from-web-page.md
+1767
View File
File diff suppressed because it is too large. Load diff
docs/superpowers/plans/2026-10-19-results-portal-redesign.md
+67
View File
@@ -0,0 +1,67 @@
# Results Portal Redesign Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Replace the one-page results table with a Dashboard → Claim status (date cards) → One day (short table + side panel) portal, plus a claim history search, in a navy & blue theme.
**Architecture:** Server-rendered FastAPI + Jinja pages sharing `base.html`. Three new read-only queries in `web/repo.py` group the existing `CV_ChargeSummary` rows by the day a run started. The Bot run card and `/api/*` stay as they are; old `/?run_at=` links redirect to the day page.
**Tech Stack:** FastAPI, Jinja2, pyodbc (SQL Server), vanilla JS, standalone test scripts (`python tests\test_web_app.py`).
**Spec:** `docs/superpowers/specs/2026-10-19-results-portal-redesign-design.md`
## Global Constraints
- Bound to 127.0.0.1; Host allowlist, `no-store` headers, DB errors by kind — unchanged.
- Start / Stop need the page Origin and `X-Run-Token`; page runs are VERIFY only.
- All new pages read-only; every SQL statement parameterized; values never in SQL text.
- DB text is escaped: Jinja autoescape, panel data via `tojson` in an attribute, written with `textContent`.
- No claim ids or patient data in tracked docs; tests use made-up ids.
- No git commits unless the operator asks.
---
### Task 1: Day-wise queries in the repo
**Files:** Modify `web/repo.py`; Test `tests/test_web_app.py`
**Interfaces — Produces:**
- `SqlResultsRepo.days(limit: int = 90) -> list[dict]` — keys `day` (date), `claims`, `runs`, `pass_count`, `fail_count`, `review_count`; newest first.
- `SqlResultsRepo.day_results(day: date, category=None, claim_no=None) -> list[dict]` — full rows (`_SELECT_COLUMNS`) with `run_at >= day AND run_at < day+1`, ordered `run_at, s_no, id`.
- `SqlResultsRepo.claim_history(claim_no: str) -> list[dict]` — `TOP 200` full rows `WHERE claim_no LIKE ?` ordered `run_at DESC, s_no, id`.
- `_coerce` also turns a text `day` into a `date`.
- [ ] Tests: each query's SQL is a parameterized SELECT with the clauses above and the expected params (`[datetime(day), datetime(day)+1 day]`, `["%no%"]`); text `day` coerced.
- [ ] Implement; run `python tests\test_web_app.py`.
### Task 2: Routes
**Files:** Modify `web/app.py`; Test `tests/test_web_app.py`
**Interfaces — Consumes:** Task 1. **Produces:** routes
- `GET /` → `dashboard.html` with `today` (days() row for `date.today()` or zero counts), `mode` ("VERIFY"/"LIVE-WRITE"), `run_token`. `GET /?run_at=<iso>` → 303 to `/day/<date>` (keeps `category`, `claim_no`); bad `run_at` → 400.
- `GET /days` → `days.html` with `days`.
- `GET /day/{day}` (`YYYY-MM-DD`, else 400) with `category`, `claim_no`, `claim` → `day.html` with `day`, `counts` (from `days()` row or computed from the unfiltered rows), `rows` (each with `fields` = list of `[header, display text]` and `details_pretty`), filters.
- `GET /claim?claim_no=` → `claim.html` with `rows` (history) and `claim_no`; empty search → page with the box only.
- FakeRepo in tests gains `days`, `day_results`, `claim_history`.
- [ ] Tests for each route (data shown, links, filters passed, 400s, redirect, DB down 503, hardening incl. new paths).
- [ ] Implement; run tests.
### Task 3: Templates and theme
**Files:** Create `web/templates/base.html`, `dashboard.html`, `days.html`, `day.html`, `claim.html`; delete `web/templates/index.html`.
- `base.html`: `:root` navy palette, header (title, mode pill, breadcrumb block), `{% block content %}`, `{% block scripts %}`, footer PHI note.
- `dashboard.html`: Bot run card (moved from index.html, with the Live claims panel replacing the old Claim status panel; a saved claim opens `/day/<results_run_at date>?claim=<id>`), today tiles, Find a claim form, big Claim status button.
- `days.html`: date cards grid with counts and colour bar; empty state.
- `day.html`: tiles as filter links, claim search, short table (#, Claim #, Time, Payer, Result, Comments), 10-a-page pager, side panel from `data-fields` (`tojson`) via `textContent`; `?claim=` opens it.
- `claim.html`: search box and history table linking to day pages.
- [ ] Tests on rendered HTML (ids, links, no inline `style=`, theme vars, panel wiring, one-scroll lock, Live claims link target).
- [ ] Run `python tests\test_web_app.py` and `python tests\test_run_status.py`.
### Task 4: Browser check
- [ ] Serve the app on a fake repo with sample data (scratch script, made-up ids), open each page in the in-app browser, screenshot, check pager, panel, filters, empty states, narrow width.
- [ ] Full suite.
docs/superpowers/specs/2026-09-15-payer-group-categorization-design.md
+182
View File
@@ -0,0 +1,182 @@
# Payer → Group Name categorization — design
Status: approved by user (Vasanth), 2026-09-15. Not yet implemented.
## Background
This is the next phase of the Salud eCW Charge Entry bot (see `README.md`),
which already automates login, 2FA, claim lookup, and exporting the
"Medcor Ready" claim queue to `Input\Claim Report_<date>.xlsx`.
The next manual step in the real process (shown in
`Salud-Charge Entry_Medcor Ready.mp4`, reviewed 2:45–10:00) is: an operator
renames a copy of that export to "Medcor ready", opens it in a dated network
folder (`...\Charge\Allocation\<Month YYYY>\<MMDDYYYY>\`, alongside sibling
workbooks `Patient`, `Pending with Error`, and `Program`), and manually
categorizes every row by filtering on the `PAYER` column and bulk-filling a
`Group Name` column with one of a small set of values (observed in the video:
`Commercial`, `Dentaquest`, `Medicaid`, `Medicare`, `Program`, plus at least
one more category for payers like `BH COLORADO ACCESS`).
This spec covers automating **only** that categorization step — not the
downstream split into the three sibling workbooks, which stays manual for now
and may be scoped separately later.
## Scope decisions (from user)
- **Scope**: categorize only. Do not build the split into
Patient/Pending with Error/Program workbooks.
- **Mapping source**: derived from the actual current export
(`Input\Claim Report_09-11-2026.xlsx`, 729 rows, 39 distinct `PAYER`
strings), not invented from video frames. Only high-confidence entries are
seeded; everything else is left for a human to fill in (see "Seed mapping
table" below) — this is intentional, not a gap to silently paper over.
- **Unknown payer handling**: flag, never guess. A `PAYER` value with no
entry in the mapping table gets a blank `Group Name`, is logged, and is
called out in the run summary. No fuzzy/substring matching — exact string
match only, so a typo'd or renamed payer can never be silently bucketed
into the wrong group.
- **Integration point**: inside the existing bot, not a standalone script.
Runs automatically as part of `flows/export_medcor_ready.py`.
- **Output location**: a new file, `Output\Medcor Ready_<date>.xlsx`. The
original `Input\Claim Report_<date>.xlsx` is never modified — `claim_file.py`
and everything downstream of it depends on that file's exact column
layout (per `README.md`'s "Reading the claim ids back" section), so this
step must not touch it.
## Architecture
```
flows/export_medcor_ready.py
... existing steps (login → claim lookup → export → claim_file.claim_ids()) ...
→ categorize_by_payer() NEW — reads Input\, writes Output\
core/payer_groups.py NEW — the lookup table + classify()
payer_groups.json NEW — editable config, exact PAYER → Group Name
core/spreadsheet.py EXTENDED — write_xlsx(), stdlib only
```
`categorize_by_payer()` runs after the existing claim-id read-back (the point
where the report is already saved and validated — see
`README.md`'s "Reading the claim ids back"). It:
1. Reads `Input\Claim Report_<date>.xlsx` with the existing `spreadsheet.read_rows()`.
2. For each row, looks up `PAYER` in `payer_groups.classify()`.
3. Writes every original column plus a new `Group Name` column to
`Output\Medcor Ready_<date>.xlsx`.
4. Logs and message-boxes a summary: rows categorized, rows left blank, and
the distinct list of unmapped `PAYER` strings — mirroring the existing
`NOTIFY_AFTER_EXPORT` / `SHOW_CLAIM_IDS` pattern in the same file.
Never fatal: an unmapped payer does not stop the run, the same philosophy
`export_claims_report()` already uses for other soft failures.
## Components
### `src/ecw_bot/core/payer_groups.py`
```python
def classify(payer: str) -> str | None: ... # exact match against the config, or None
def load_mapping() -> dict[str, str]: ... # reads payer_groups.json once
```
Exact-string lookup only. No normalization beyond what's needed to match
eCW's own values (the config keys are the literal `PAYER` strings as eCW
exports them).
### `src/ecw_bot/payer_groups.json`
A flat `{"<exact PAYER string>": "<Group Name>"}` map, hand-editable by
anyone on the team without touching code — same reasoning as `.env` for
settings and `selectors\` for DOM selectors: this data changes for a
different reason (a new payer shows up) than the code around it does.
### `core/spreadsheet.py` — new `write_xlsx()`
The project deliberately has no `openpyxl` dependency (see README: "every
added package is an approval"). `write_xlsx()` follows the same approach
already used for reading — a minimal single-sheet xlsx as a zip of XML, cells
as inline strings (no `sharedStrings.xml`, matching what eCW's own exports
already do, per the existing reader's comments) — built with `zipfile` and
string templates only.
## Data flow
```
Input\Claim Report_<date>.xlsx
→ read_rows() existing
→ for each row: classify(PAYER) new
→ write_xlsx(rows + Group Name) new
→ Output\Medcor Ready_<date>.xlsx
```
## Error handling
- Unmapped payer: blank `Group Name`, logged, summarized — not fatal.
- `Input\` file missing/unreadable: this step is skipped with a clear log
line; it never blocks the rest of the run, since the export already
succeeded by the time this step runs.
- `write_xlsx()` failure (disk full, permission): logged and raised — same
severity as any other unexpected I/O failure in this flow, since silently
losing the categorized output would defeat the point of the step.
## Testing
`tests/test_payer_groups.py`, hand-rolled in the existing style (no pytest):
- Exact-match lookups against a small fixture mapping.
- Unmapped payer → `None`, never a guess.
- Round-trip test for `write_xlsx()`: write a small fixture table, read it
back with the existing `read_rows()`, assert the data matches — the same
"never touched eCW" local-fixture philosophy as `test_claim_form.py`.
## Seed mapping table
Built from the real, current payer list (`Input\Claim Report_09-11-2026.xlsx`,
729 rows). Only high-confidence entries are seeded — this covers roughly 95%
of row volume in that file.
**Mapped (seeded):**
| Payer (exact eCW string) | Group |
|---|---|
| Medicaid | Medicaid |
| DentaQuest | Dentaquest |
| CHP+ DENTAQUEST | Dentaquest |
| AARP Medicare Advantage | Medicare |
| Aetna Medicare ADV | Medicare |
| OUTREACH PROGRAM | Program |
| Anthem Blue Cross Blue Shield | Commercial |
| Cigna HealthCare Of Colorado | Commercial |
| Cigna Healthcare of CO | Commercial |
| Cigna Salud Employee | Commercial |
| Cigna | Commercial |
| Cypress Benefit Admin / Cigna PPO | Commercial |
| United Healthcare | Commercial |
| United Health Care | Commercial |
| United healthcare | Commercial |
| Unitedhealth One | Commercial |
| Oxford Health Insurance UHC | Commercial |
| UMR | Commercial |
| Aetna | Commercial |
| Aetna Alaska Care | Commercial |
| AETNA HEALTH AND LIFE INSURANCE | Commercial |
| BIND BENEFITS INC | Commercial |
| Select Health | Commercial |
**Left unmapped — needs a human decision before first real run (or will be
flagged on first run and filled in afterward):**
BH COLORADO ACCESS (15 rows), Delta Dental of Colorado (3), Devoted (2),
Clinic Sliding Fee Scale (2), Patient (1), Gravie Admin Services (1),
Tricare (1), Rocky Mountain Health Plans (1), Denver Elevate Health Plan (1),
The Kempton Group Administrators Inc (1), Optum Care Network (1), GPA (1),
Northeast Health Partners (1), Delta Dental of Pennsylvania (1),
Delta Dental of Wisconsin (1), Delta Dental (1), Delta Dental of Arkansas (1).
## Out of scope (explicitly)
- Splitting rows into the Patient / Pending with Error / Program sibling
workbooks.
- Writing to the network `Charge\Allocation\...` folder — output stays local
to `Output\`.
- Any change to `Input\Claim Report_<date>.xlsx` or anything that reads it.
docs/superpowers/specs/2026-09-16-condition-framework-design.md
+269
View File
@@ -0,0 +1,269 @@
# Condition Framework — Design
**Status:** Approved by user, 2026-09-16
**Supersedes nothing** — this formalizes the pattern Condition 1 (telehealth)
already established informally in `flows/export_medcor_ready.py`.
## Why
The Charge Validation Process is, by the client's own description, a long
list of per-claim conditions to check and — when a condition's criteria are
met — a correction to apply (Condition 1: a telehealth visit needs POS=11 and
M1=GT). Conditions are being implemented one at a time, and there will be
many more of them. This is a healthcare billing bot: every write it makes
changes what gets billed to a real payer, so the process must be:
- **Reusable** — adding Condition N must not require touching the code for
Conditions 1..N-1, and must follow one obvious, fixed shape every time.
- **Exception-safe** — a bug in one condition's logic must never corrupt
another condition's result on the same claim, crash the whole overnight
batch, or leave a claim half-changed with no record of what happened.
- **Provably correct before it ever touches a real claim** — every condition
must be unit-testable against a fixture, exactly like
`tests/test_telehealth_fix.py` already does for Condition 1.
This spec explicitly rejects driving the live automation with an
LLM-agent-controlled browser (e.g. Playwright MCP) as the *engine* for this
reason: none of the three properties above hold for an engine whose
per-claim behavior is an LLM's live judgment call rather than fixed,
versioned code. Playwright MCP remains valuable as a *development and
debugging aid* (inspecting live eCW DOM, prototyping selectors) — that usage
is unaffected by this spec and requires no code change.
## Non-goals
- Not changing how claims are found, opened, or read (`claim_form.py`,
`export_medcor_ready.py`'s claims-phase — untouched).
- Not building a self-healing/adaptive-selector system. If eCW's DOM shifts
enough to break a selector, that is still a `selectors/claim_form.py`
update, done and tested like every past one (see CHANGELOG "Day 5").
- Not retrying failed conditions automatically. A condition that throws is
logged and skipped for that claim, once, this run.
## Architecture
### New package: `src/ecw_bot/conditions/`
```
src/ecw_bot/conditions/
__init__.py # CONDITIONS: list[module] — the registry
base.py # ConditionResult dataclass + run_all()
telehealth.py # Condition 1 — detect/apply/describe wrapper
```
### The Condition contract
Every condition module in the registry exposes exactly these three
functions. This is the whole interface — nothing else is required, nothing
else is read by the runner.
```python
def detect(claim_id: str, data: dict) -> bool:
"""
Pure. No `page`, no I/O, no side effects. `data` is the same dict
`claim_form.extract()` already produces (plus `progress_notes`, when
OPEN_PROGRESS_NOTES is on). Must be safe to call with a partially
populated dict — return False, never raise, on missing keys.
"""
def apply(page, claim_id: str, data: dict) -> "ConditionResult":
"""
Only called when detect() returned True. Makes the eCW field edits for
this condition and returns what changed. MUST NOT click Save/OK and
MUST NOT show any confirmation dialog itself — the runner's caller owns
both, once, after every condition has had a chance to run. Raising is
allowed and expected to be handled by the caller (see Exception
handling below) — apply() does not need its own try/except for its own
field-write logic, only for optional recovery it wants to attempt
itself.
"""
def describe(result: "ConditionResult") -> str:
"""
One human-readable block for this condition's contribution to the
combined confirmation dialog and the log line — mirrors the per-row
lines `_apply_telehealth_fix` already builds today.
"""
```
### `ConditionResult` (`conditions/base.py`)
```python
@dataclass
class ConditionResult:
condition_name: str
changed: bool
changes: list # same shape as claim_form.set_cpt_pos_modifier()'s
# return value: [{"row", "code", "before", "after"}, ...]
review: str = "" # added 2026-10-05 (unit D) — see below
@classmethod
def none(cls, condition_name: str) -> "ConditionResult":
return cls(condition_name=condition_name, changed=False, changes=[])
@classmethod
def declined(cls, condition_name: str, reason: str) -> "ConditionResult":
return cls(condition_name=condition_name, changed=False, changes=[],
review=reason)
```
**Amendment 2026-10-05 (review items D2/P2, unit D).** `none()` means
"nothing to do — the claim already follows this rule". A condition that
matched the claim but could not act on it (the CPT/HCPCS tab would not
open, an admin code would not add with the grid untouched) now returns
`declined(name, reason)` instead: nothing changed, but the claim is not
clean, and `reason` reaches the Charge Summary as a "Review" row. A
changed result may also carry `review` (a failed add that still moved the
grid). Two optional module hooks join `detect/apply/describe`:
- `needs_review(claim_id, data) -> str | None` — asked only when
`detect()` returned False, for a claim that matched the trigger but
cannot be acted on (vaccine_admin_codes on an unmapped payer, no payer
name, no usable age — never Medicare, which has its own rule).
- `summary_comment(result) -> str` — the Charge Summary comment for a
changed result, built from what actually happened, in place of the
summary's fixed `COMMENTS[name]`.
### The runner (`conditions/base.py`)
```python
def run_all(page, claim_id: str, data: dict) -> list[ConditionResult]:
"""
Runs every registered condition against one claim, in registry order.
Returns only the results where something actually changed — a claim
with nothing to fix returns an empty list, and the caller shows no
dialog and saves nothing.
"""
results = []
for cond in CONDITIONS:
name = cond.__name__.rsplit(".", 1)[-1]
try:
if not cond.detect(claim_id, data):
continue
result = cond.apply(page, claim_id, data)
except Exception as exc:
print(f"[COND {name} {claim_id}] Failed: {exc}")
shot(page, f"condition_{name}_failed_{claim_id}")
continue # this condition's failure must not block the next one
if result.changed:
results.append(result)
return results
```
`shot()` is the existing screenshot helper already used by `open_claim()`'s
diagnostic wrapping (`claim_form.py`) — reused here, not reinvented.
**Amendment 2026-10-05 (review items D2/P2, unit D).** The real signature
is now `run_all(page, claim_id, data, conditions=None, before_apply=None,
report=None)`. Its return value is unchanged — only the changed results —
but that list is not everything that happened, so a caller that passes a
dict as `report` gets the whole account:
| key | contents |
|---|---|
| `fired` | names whose `detect()` returned True |
| `changed` | names with a changed result (= the return value) |
| `failed` | `{"name", "stage", "error"}` per raise; `stage` is `"detect"`, `"apply"` or `"needs_review"`, `error` the exception's first line |
| `review` | `{"name", "reason"}` from every result (changed or not) with a non-empty `review`, and from `needs_review()` of every condition whose `detect()` returned False |
| `comments` | `{name: summary_comment(result)}` for changed results whose module has the hook (a hook that raises is logged and left out — the summary falls back to `COMMENTS[name]`) |
Exceptions are still isolated exactly as in the sketch above: logged,
screenshotted, and the next condition still runs. `detect()` and `apply()`
now each sit in their own try (the sketch's single try is how it began), so
the report can say which stage raised.
### Caller contract (replaces `_apply_telehealth_fix`'s direct call)
`export_medcor_ready.py`'s `_read_claim_form()` changes from calling
`_apply_telehealth_fix(page, claim_id, data)` directly to:
```python
if cfg.APPLY_CORRECTIONS:
results = conditions.run_all(page, claim_id, data)
saved_write = _confirm_and_save(page, claim_id, results)
```
(As built since 2026-10-05: `run_all(..., report=report)` and
`_confirm_and_save(page, claim_id, results, report)`; the report is stored
on the claim's data as `data["_conditions"]` = `{"fired", "saved",
"failed", "review", "comments"}`, which the Charge Summary reads.)
`_confirm_and_save` (new, small, lives in `export_medcor_ready.py` — it is
orchestration, not a condition):
1. If `results` is empty → return `False` immediately, no dialog, no save.
(A claim where nothing fired is not different from today's "no
telehealth mention" case.)
2. Otherwise build one combined message: every fired condition's
`describe()` output, concatenated, in the same
`msgbox.ok_cancel(...)` shape Condition 1 uses today.
3. One Yes/No. Yes → `claim_form.save_claim(page)`. No → log "declined",
return `False` (the caller's existing `finally` still closes with
Cancel, unchanged).
4. Return whether the save actually happened, same as
`_apply_telehealth_fix` returns today — this is what `_read_claim_form`
uses to decide whether its own `finally` should call `close_claim()`.
**Amendment 2026-10-05 (review item P2, unit D):** between 1 and 2 —
and after the SAVE_CORRECTIONS gate — if the report has any `failed` or
`review` entry, the box is never shown: the log says `[COND <id>] Not
offering Save: <reasons> — closing with Cancel` and it returns `False`.
**Why one dialog, not one per condition:** if two conditions both want
changes on the same claim, saving after only reading half the operator's
decision is exactly the kind of half-applied state this spec exists to
prevent. The operator sees the whole picture — every condition that fired,
every field that would change — before any single Save/Cancel decision.
## Exception handling — the actual rules
This is the section the user asked for explicitly, spelled out rather than
left implicit:
| Failure | What happens |
|---|---|
| One condition's `detect()` raises | Caught by `run_all`'s try/except (its own try since 2026-10-05, so the report can say `stage: "detect"`). Logged, screenshotted, skipped, and recorded in the report's `failed` — the claim reads "Review", not clean. Other conditions still run. |
| One condition's `apply()` raises partway through a multi-row edit | Caught by `run_all`. Logged, screenshotted. That condition contributes nothing to `results`. **Corrected 2026-10-05 (review item P2):** this row used to say its partial edits "are never saved, because Save only happens ... for what's in `results`". That was wrong: Save (`saveAllData`) writes the whole form, partial edits included, and the box would have listed only the other conditions. So the raise is recorded in `run_all`'s report (`failed`, stage `"apply"`), and `_confirm_and_save` never offers Save for a claim whose report has a `failed` or `review` entry — the claim is closed with Cancel and reported "Review" in the Charge Summary. |
| `_confirm_and_save`'s dialog itself fails to show (msgbox error) | Not caught here — surfaces to `_read_claim_form`'s existing outer try/except, which logs "Could not read the form" and closes via `finally`. No new behavior needed; the existing outer boundary already covers this. |
| `save_claim()` returns `False` (save click didn't work) | Same as today: logged, `saved_write` stays `False`, `finally` calls `close_claim()` — the claim is left as read, not half-saved. |
| A condition throws on every claim in the batch (systemic bug in that one condition) | By design, this spec does not add batch-level circuit-breaking — each claim's screenshot + log line is enough signal for the operator to notice a pattern across a run's log file. Adding an auto-disable-after-N-failures behavior is explicitly deferred (YAGNI): today's batches run at `CLAIM_PROCESS_LIMIT` scale, small enough that log review is a real diagnostic step, not a chore. |
## Testing
- `tests/test_telehealth_fix.py` — unchanged. It tests
`claim_form.note_mentions_telehealth` and `claim_form.set_cpt_pos_modifier`
directly; those functions do not move.
- `tests/test_conditions_telehealth.py` (new) — tests
`conditions/telehealth.py`'s thin wrapper: `detect()` delegates to
`note_mentions_telehealth` correctly on a plain dict (no page needed);
`apply()` calls `set_cpt_pos_modifier` and returns a populated
`ConditionResult`; `describe()` produces the expected text.
- `tests/test_conditions_runner.py` (new) — tests `run_all()` and
`ConditionResult` in isolation with fake condition modules (no real
telehealth logic involved):
- a condition whose `detect()` returns `False` is never `apply()`-ed
- a condition whose `apply()` raises is skipped, and a *second*,
healthy condition registered after it in `CONDITIONS` still runs and
its result is still returned
- `results` is empty when every condition's `detect()` returns `False`
- a condition's raised exception does not propagate out of `run_all`
## Migration checklist (informs the implementation plan)
1. Create `conditions/base.py` (`ConditionResult`, `run_all`).
2. Create `conditions/telehealth.py` — move `note_mentions_telehealth`-based
detection logic into `detect()`, wrap `set_cpt_pos_modifier` in
`apply()`, write `describe()` from the existing message-building lines
in `_apply_telehealth_fix`.
3. Create `conditions/__init__.py` with `CONDITIONS = [telehealth]`.
4. Add `_confirm_and_save` to `export_medcor_ready.py`; delete
`_apply_telehealth_fix` (its logic is now split across
`conditions/telehealth.py` and `_confirm_and_save`).
5. Update `_read_claim_form` to call `conditions.run_all` +
`_confirm_and_save` instead of `_apply_telehealth_fix` directly.
6. New tests (above) all passing; full existing suite (88+ tests) still
passing unchanged.
7. Condition 2 (whatever it turns out to be) is the first real proof this
spec worked: it should require touching only a new
`conditions/<name>.py` + its test file + one line in `CONDITIONS`.
docs/superpowers/specs/2026-10-07-dental-claims-run-all-conditions-design.md
+137
View File
@@ -0,0 +1,137 @@
# Dental claims run every applicable condition — design
Date: 2026-10-07. Status: approved in chat (operator, 2026-10-07);
implemented 2026-10-12 (operator: "this is our number one condition").
## 2026-10-12 update (as built)
- The rule now also covers #12 (H codes: BH payer → modifier removed;
any other payer → line unchecked + fee 0) and #14 (Medicare: G8510 /
G8431 unchecked + fee 0, #6 not run). Both run on Dental claims like
every other condition; #12's BH path has no modifier column to read on
the Dental grid, so it never fires there.
- A fix the Dental screen cannot hold → **Review** (operator): a condition
module declares `DENTAL_UNSUPPORTED = "<reason>"` and `run_all` records
that reason instead of applying it on a `screen == "dental"` read —
telehealth, medicare_telehealth, vaccine_admin_codes.
- Dental claims are **never saved** in this change (operator), whatever
`SAVE_CORRECTIONS` says; the review-before-close box still shows.
- A Dental claim is read and run through the conditions even with
CHECK_HIGH_RISK and CHECK_SLIDING_FEE both off (it used to come back
blank).
- The Charge Summary applies the same claim-level checks (note read, CPT /
ICD rows read, conditions run) to a Dental read as to UB/HCFA.
- Row selectors gained a third (Dental) alternative, scoped to the Dental
window on screen (`:visible`); `extract()` returns `screen`
("ub" / "hcfa" / "dental").
- Not live-verified yet: the first live run should be SAVE-off on a few
Dental claims (one with DCONS or Dscprit).
## Rule (operator, 2026-10-07)
Every claim is checked against **every condition whose Payer column is
"All" / "All payers"**, **plus** every condition specific to that claim's
payer (Medicare, Medicaid, Dental payers, ...). This holds for every payer
and every claim screen — UB, HCFA and Dental alike.
UB and HCFA claims already work this way: each claim runs the whole
registry (`conditions/__init__.py` `CONDITIONS`) and each condition's own
`detect()` decides from the claim's codes and payer. **The gap is Dental:** a
Dental claim today runs only #15 high risk and #8 sliding fee and never
reaches `run_all`. So, for example, `dummy_codes` (DCONS/Dscprit, which are
Dental codes) can never fire on any claim.
## What a Dental claim will run
| Doc # | Condition | Payer (document) | Module |
|---|---|---|---|
| 5 | J2930/J2001 uncheck + fee 0 | ALL | `j2930_j2001_update` |
| 6 | Depression DX pointer | All | `depression_dx_pointer` |
| 7 | Zero-fee uncheck / priced line check | ALL | `zero_fee_uncheck`, `priced_line_check` |
| 21 | Telehealth POS 2 + modifier | All | `telehealth` |
| 22 | Medicare telehealth G2025 | Medicare | `medicare_telehealth` |
| 18 | Vaccine admin codes | All except Medicare | `vaccine_admin_codes` |
| 8 | DCONS/Dscprit uncheck + fee 0 | All Dental Payer | `dummy_codes` |
| 8 | Sliding fee date check | All Dental Payer | `pages/sliding_fee` (unchanged) |
| 15 | High risk in Billing Notes | Dental Payers | `pages/dental_notes` (unchanged) |
No condition gains a form-type check: each still decides from codes and
payer only. A condition whose codes are not on the claim does nothing.
## Dental screen facts (from Dom.txt, claim screen `.modal[id^="dentalClaim"]`)
- CPT/CDT grid `table[id^="claimCPTCodesTable"]`; real rows are
`tr[ng-repeat^="CPT in CPTCodes"]` (no `CPTROW_` id).
- Per row: checkbox `[ng-click*="updateBillToIns("]` (same as UB/HCFA), code
`[data-fieldname="web_cptcode"]`, pointers `web_icd1`..`web_icd4`, units
`web_units`, fee `web_billedFee` (blur runs `editCPTRow('BILLED_FEE')`),
DOS `DentalClaimCPTSDOS`.
- **No per-row POS and no modifier columns.** POS is claim-level
(`ng-model="ClaimData.pos"`).
- Prog. Notes button `button[ng-click="viewProgNote()"]`
(`dentalClaimProgressNoteBtn…`) — the existing `SEL_PROGRESS_NOTES` matches.
- Insurance grid: `#DentalClaimTbl2` (Bill To, Ins Id, Name, ...).
- Billing Notes: `#dentalClaimNotes` (already read by high risk).
None of the row fields has been exercised live yet; Dom.txt is one capture.
## Design
1. **Selectors** (`selectors/claim_form.py`): add the Dental alternative to
each row selector the helpers already use — `SEL_CPT_ROWS`,
`SEL_CPT_CODE`, `SEL_CPT_ICD1..4`, `SEL_CPT_BILLED_FEE`, `SEL_CPT_UNITS`.
The Dental row selector is scoped under `.modal[id^="dentalClaim"]` so it
cannot match a grid on another screen. Within one row only one alternative
ever matches, the same contract the UB/HCFA pairs rely on.
2. **Reader** (`pages/claim_form.py` `_EXTRACT_JS`): read Dental rows into the
same `cpt` row shape (`code`, `icd1..4`, `units`, `billed_fee`, `billed`,
`sdos`; `pos`/`mod1..4` empty), insurance from `#DentalClaimTbl2`, the
Dental claim number, ICD list, and `notes` from `#dentalClaimNotes`.
One `extract()` for all three screens.
3. **Flow** (`flows/export_medcor_ready.py`): a Dental claim runs high risk +
sliding fee first (unchanged, read-only), then the same path as UB/HCFA:
`extract()`, Prog. Notes (when `OPEN_PROGRESS_NOTES`), `run_all`, per-claim
box, Charge Summary. The Dental results stay on the claim's data so the box
and the summary show both.
4. **A fix the Dental screen cannot make** — e.g. telehealth needs a row
modifier and the Dental grid has none, or POS is claim-level: the condition
returns `declined(...)` with the reason ("Dental screen has no modifier
column"), so the claim is **Review** in the Charge Summary. Never a silent
skip, never a guess.
5. **Charge Summary**: one row per claim combining condition comments with the
Dental findings (HIGH RISK, sliding fee). Category: Review beats Fail beats
Pass, as today.
6. **Saving**: Dental claims are **never saved** in this change, whatever
`SAVE_CORRECTIONS` says. Their corrections are shown and listed as
"Corrections pending (not saved)". Dental save is enabled only after a live
verification-mode run confirms the row writes.
## Error handling
- Dental form not loaded (`wait_claim_loaded`): conditions skipped, claim is
Review — same as HCFA today.
- A Dental row field missing (fee or checkbox not found): the existing helpers
already treat a missing field as unknown, not as zero/unchecked.
- High risk / sliding fee failures keep their current Review handling.
## Testing
- Grow `tests/fixtures/dental_claim.html` from Dom.txt's real Dental grid
(rows, checkbox, fee, pointers, insurance grid, Prog. Notes button) with no
patient data.
- `extract()` on the Dental fixture returns the expected `cpt` and `insurance`.
- Per condition, a Dental case: dummy codes (DCONS/Dscprit) uncheck + zero;
J2001 uncheck + zero; zero-fee uncheck; priced line check; depression
pointer retag on `web_icd*`; telehealth on Dental → declined/Review.
- Flow: a Dental claim gets high risk + sliding fee **and** condition results
in one summary row; never offered Save.
- Full suite stays green.
- Live: one SAVE-off run on Dental test claims (with a DCONS or Dscprit line)
before Dental save is considered.
## Out of scope
- New document conditions not yet built (#3, #4, #9-#14, #16, #19, #20,
#23-#55).
- Saving Dental claims.
- "Special programs" half of #8.
docs/superpowers/specs/2026-10-08-results-db-and-web-page-design.md
+164
View File
@@ -0,0 +1,164 @@
# Charge Summary in SQL Server + results web page — design
Date: 2026-10-08. Status: design approved in chat (operator, 2026-10-08),
spec under review.
## Goal
Every run's Charge Summary rows are also stored in SQL Server, and a simple
local web page (FastAPI) shows them. This is the basic version; later it is
folded into the full portal.
## Decisions (operator, 2026-10-08)
- Server `MECARE331\MECARE331` (SQL Server 2022, on this machine), database
**`Charge_Validation`** (created by the operator).
- The table follows the team's **Charge Summary File Format**
(`\\mecare2\Production\Salud-eCW RPA\Charge Summary File Format.xlsx`) —
the same 24 columns as `core/charge_summary.HEADERS`, plus run columns and
a details JSON. PATIENT, Account Number and Guarantor Name are stored, as
in the Excel file.
- FastAPI for the backend. All needed packages are already installed
(fastapi 0.141, uvicorn, jinja2, httpx, pyodbc, ODBC Driver 18); nothing new
is added to `requirements.txt` for the bot itself.
## Table
`tools/sql/create_tables.sql`, run by the operator in SSMS (the bot never
creates or alters tables):
```sql
USE Charge_Validation;
CREATE TABLE dbo.ChargeSummary (
id INT IDENTITY(1,1) PRIMARY KEY,
run_at DATETIME2 NOT NULL,
run_mode NVARCHAR(20) NOT NULL, -- VERIFY / LIVE-WRITE / TEST
s_no INT NULL,
claim_no NVARCHAR(20) NOT NULL,
service_date DATE NULL,
pvdr NVARCHAR(50) NULL,
patient NVARCHAR(200) NULL,
payer NVARCHAR(200) NULL,
group_name NVARCHAR(100) NULL,
status NVARCHAR(100) NULL,
charges DECIMAL(12,2) NULL,
pmts_adjs DECIMAL(12,2) NULL,
pmts_refunds DECIMAL(12,2) NULL,
adjustment DECIMAL(12,2) NULL,
withheld DECIMAL(12,2) NULL,
balance DECIMAL(12,2) NULL,
visit_type NVARCHAR(100) NULL,
account_number NVARCHAR(50) NULL,
provider_name NVARCHAR(200) NULL,
guarantor_name NVARCHAR(200) NULL,
received_date DATE NULL,
assigned_to NVARCHAR(100) NULL,
completed_on DATE NULL,
comments NVARCHAR(MAX) NULL,
current_status NVARCHAR(200) NULL,
category NVARCHAR(20) NULL,
details_json NVARCHAR(MAX) NULL
);
CREATE INDEX IX_ChargeSummary_run_at ON dbo.ChargeSummary (run_at);
CREATE INDEX IX_ChargeSummary_claim_no ON dbo.ChargeSummary (claim_no);
```
Column map (Excel → DB) lives in one place, `results_db.COLUMNS`, built from
`charge_summary.HEADERS` so the two cannot drift: S.No→s_no, CLAIMS#→claim_no,
SERVICE DATE→service_date, PVDR→pvdr, PATIENT→patient, PAYER→payer, Group
Name→group_name, STATUS→status, CHARGES→charges, PMTS/ADJS→pmts_adjs,
Pmts/Refunds→pmts_refunds, Adjustment→adjustment, WithHeld→withheld,
Balance→balance, Visit Type→visit_type, Account Number→account_number,
Provider Name→provider_name, Guarantor Name→guarantor_name, Recevied
date→received_date, "Assigned To "→assigned_to, Completed On→completed_on,
Comments→comments, Current Status→current_status, Category→category.
Value conversion: money columns strip `$`, `,` and spaces → Decimal; dates
accept `mm/dd/yyyy` (and an Excel date/datetime) → date; blank or
unparseable → NULL (the raw text is not lost: the Excel file still has it).
## Bot side — `src/ecw_bot/core/results_db.py`
- Settings (`settings.py`, from `.env`):
- `RESULTS_DB` (bool, default **off**): write rows to the DB.
- `RESULTS_DB_CONN` (string): ODBC connection string. Default:
`DRIVER={ODBC Driver 18 for SQL Server};SERVER=MECARE331\MECARE331;DATABASE=Charge_Validation;Trusted_Connection=yes;TrustServerCertificate=yes;`
(Windows login, no password). A SQL login's password, if ever used, is put
in `.env` by the operator only.
- `to_record(row: dict, *, run_at, run_mode, details) -> dict`: one Charge
Summary row (header→value) to DB column→value, converted.
- `insert_rows(conn, records) -> int`: one parameterized `INSERT` per row
(`executemany`), one commit; returns rows written. Never builds SQL from
values.
- `save_summary(sheet, details, run_at, run_mode, connect=pyodbc.connect) ->
int | None`: `sheet` is the header+rows list `charge_summary.build_rows`
already returns; `details` maps claim no → details dict. Opens the
connection, inserts, closes. **Any exception is caught**: one
`[RESULTS-DB] Could not save ...` log line, returns None. The run, the Excel
summary and the claims are never affected.
- `details_json` per claim: `{"fired", "changed", "review", "failed",
"comments", "saved", "changes": {condition: [change, ...]},
"dental": {"high_risk", "sliding_fee"}}` — taken from the claim's
`data["_conditions"]` (the flow adds `changes` there) and Dental results.
## Flow changes
- `charge_summary.write()` is split: `charge_summary.build(report_path,
outcomes, assigned_to) -> sheet` (read + `build_rows`), and `write()` saves
that sheet as today. The flow builds once and hands the same sheet to Excel
and to the DB, so both always hold the same rows.
- `_read_claim_form` adds `"changes": {r.condition_name: r.changes}` to
`data["_conditions"]`; the claim loop keeps `details[claim_id]` next to
`outcomes[claim_id]`.
- `_write_charge_summary` calls `results_db.save_summary(...)` after the Excel
write when `RESULTS_DB` is on, and logs `[RESULTS-DB] N row(s) saved`.
- `run_mode`: `TEST` when `TEST_CLAIM_IDS_FILE` is set, else `LIVE-WRITE`
when `SAVE_CORRECTIONS` is on, else `VERIFY` — the same words as the
`[MODE]` banner.
## Web page — `web/`
- `web/app.py` (FastAPI) and `web/templates/index.html` (Jinja2), plain
HTML/CSS, no external scripts or CDNs.
- Read-only: only `SELECT` statements, parameterized.
- `GET /` — page:
- run picker (distinct `run_at`, newest first; default latest run),
- Category filter (All / Pass / Fail / Review) and Claim# search,
- counts per Category for the selected run (Pass / Fail / Review tiles),
- the table in Charge Summary column order, Category coloured,
- each row expands to show `details_json` (conditions fired, before/after
changes, review reasons).
- `GET /api/runs` — runs with counts. `GET /api/results?run_at=&category=&claim_no=`
— rows as JSON (for the later portal).
- Start: `python -m uvicorn web.app:app --host 127.0.0.1 --port 8000`.
Bound to **127.0.0.1 only**: the page shows PHI and is not reachable from
the network. A `run_web.bat` helper does the same.
- The DB connection string is the same `RESULTS_DB_CONN`. If the DB cannot be
reached the page shows a plain error message, not a stack trace.
## Error handling
- DB unreachable / permission denied / table missing during a run → one log
line, run continues, Excel written.
- A value that does not convert → NULL for that column, row still saved.
- Web page: DB errors → HTTP 503 page "Results database not reachable".
## Testing (no real DB)
- `tests/test_results_db.py`: header→column map covers all 24 HEADERS;
money/date conversion (incl. `$1,234.50`, blank, junk); `to_record` output;
`insert_rows` sends one parameterized statement with values in column
order (fake connection records calls); `save_summary` swallows a connect
error and returns None; `details_json` shape.
- `tests/test_charge_summary.py`: `build()` + `write()` split keeps today's
output.
- `tests/test_web_app.py`: FastAPI `TestClient` with a fake repository —
page renders rows and Category counts, filters work, `/api/results` JSON
shape, DB error → 503.
- Full suite stays green. Live: operator runs `create_tables.sql`, sets
`RESULTS_DB=1`, one test run, then opens the page.
## Out of scope (later portal)
Login/roles, editing rows, charts beyond the Category tiles, scheduling,
AI features, hosting on the network.
docs/superpowers/specs/2026-10-14-start-bot-from-web-page-design.md
+176
View File
@@ -0,0 +1,176 @@
# Start the bot from the results web page — design
Date: 2026-10-14. Approved section by section by the operator in chat
(approach A, sections 1–3).
## Goal
The operator starts a Charge Validation run from the local results page
(`run_web.bat` → http://127.0.0.1:8000) instead of the terminal. While it
runs, the page shows live status; it can stop it; when it ends, the page
shows that run's results.
## Decisions (operator, 2026-10-14)
| Question | Answer |
|---|---|
| Controls | Start + live status + Stop (no settings editing on the page) |
| Save mode | **VERIFY only.** A page-started run is always `SAVE_CORRECTIONS=0`, whatever `.env` says. LIVE-WRITE stays terminal-only. |
| Message boxes | **Off** for the run's information boxes (per-claim, review-before-close, "Run the Lookup?", summaries). Boxes that need a person to act in the browser — CAPTCHA, 2FA manual step, security image — **stay on**. |
| Approach | A: the web server starts `run.py` as a separate child process (not a thread, not Task Scheduler). |
Out of scope (YAGNI): editing settings from the page, scheduling, more than
one run at a time, LIVE-WRITE from the page.
## Components
### New: `web/bot_runner.py` — process control only
- `start() -> dict` — starts `python run.py` (the project's own `run.py`,
`sys.executable`, cwd = project root) as a child process with no console
window of its own (`CREATE_NO_WINDOW`); the Chrome window still shows.
The child's stdout/stderr go to `Logs\web_run_<ts>.out`, which catches a
crash before the bot opens its own log; the bot's `Logs\run_<ts>.log`
(named in the lock file) is the status source.
Refuses (raises `AlreadyRunning`) when the lock file names a live pid.
- `stop() -> dict` — `taskkill /PID <pid> /T /F`: the bot and its Chrome
close together. Clears the lock file. Status becomes "Stopped by
operator".
- `status() -> dict` — `running` (bool), `pid`, `started_at`, `log_path`,
`exit_code` (when this web process started it and it has ended),
`stopped_by_operator`.
- The command is fixed. Nothing from the request reaches the command line
or the environment.
**Child environment** — set on the child process, so it wins over `.env`
(`settings.py` loads `.env` with `os.environ.setdefault`):
```
SAVE_CORRECTIONS=0
ECW_KEEP_OPEN=0 # no "Press Enter to close" at the end
QUIET_BOXES=1 # run-flow information boxes go to the log
CONFIRM_BEFORE_LOOKUP=0
```
Everything else follows `.env` (claim file, `CLAIM_PROCESS_LIMIT`,
`RESULTS_DB`, proxy, ...).
### Lock file: `Logs\bot_run.json`
`{"pid", "started_at", "log_path", "source": "web" | "terminal"}`.
- Written by the **bot itself** at the start of `main()` (after
`runlog.start()`, so `log_path` is known) and removed when `main()`
returns — so a terminal run is seen by the page too.
- Before writing it, `main()` refuses to start (exit 2, before login) when
the file names a pid that is alive: two runs would share one eCW login
guard and one Charge Summary.
- A file naming a dead pid is stale: removed and ignored (a killed run
cannot remove its own file).
- `bot_runner.start()` checks it too, so the page answers 409 at once.
### New: `web/run_status.py` — read the run log
Pure functions over log text (no I/O except reading the file tail):
- **Phase** from the last matching line:
`[LOGIN]` / `[CAPTCHA]` → "Logging in"; `[OTP]` / `[2FA]` → "Waiting for
the 2FA approval mail"; `[OK] Logged in` → "Logged in"; `[CLAIMS]` →
"Checking claims"; process ended → "Finished" or a failure text by exit
code.
- **Progress** from `[CLAIMS] --- claim n/N: <id> ---` → `n`, `N`, current
claim id.
- **Counts** from the new `[RESULT]` lines (below) → Pass / Fail / Review.
- **Tail**: the last 30 lines.
- **Exit codes** (from `flows/export_medcor_ready.main`): 0 finished; 2
config / mode gate; 3 login guard refused (its `[GUARD]` line shown); 5
claims phase failed; 6 2FA timed out; other → "ended with code N".
### Changed: `flows/export_medcor_ready.py`
- After each claim's outcome is known in `_process_claims`, one
machine-readable line (terminal runs print it too):
```
[RESULT] claim 3/7 9126000: Review
```
The word is the Category the Charge Summary gets for that claim; a claim
that gets no Charge Summary row (not found in the grid) prints
`Not found`.
- `main()` writes / removes the lock file as above.
### Changed: `core/msgbox.py`
- New setting `QUIET_BOXES` (default off; `settings.py`).
- `info_box(title, msg, needs_person=False)`: with `QUIET_BOXES=1` and
`needs_person=False`, no box — one log line instead:
`[BOX] <title>: <first line of msg>`.
- `ok_cancel` / `confirm` are unchanged (every remaining caller is a
person-needed step: CAPTCHA, 2FA manual, LIVE-WRITE confirm — the last
is never reached in VERIFY). The security-image `info_box` in
`auth/login.py` passes `needs_person=True`.
### Changed: `web/app.py` + `web/templates/index.html`
Endpoints:
| Method | Path | Answer |
|---|---|---|
| POST | `/api/bot/start` | 200 `{status}`; 409 already running; 403 bad Origin/token |
| POST | `/api/bot/stop` | 200 `{status}`; 409 nothing running; 403 bad Origin/token |
| GET | `/api/bot/status` | `{running, phase, progress, counts, tail, started_at, exit_code, message}` |
Page: a "Bot run" card above the results.
- Idle: **[Start run]** and "VERIFY mode — nothing is saved to eCW".
- Running (polls `/api/bot/status` every 2 s): phase, progress bar
"3 / 7", current claim, Pass/Fail/Review chips, collapsible "Live log"
(last 30 lines), **[Stop]** (with a confirm dialog).
- Ended: "Finished in 4m 12s — exit 0" (or the failure text); the results
table reloads on the newest run.
## Security
The page was read-only; Start/Stop are its first actions. The existing
TrustedHost check stops DNS rebinding but not a cross-site POST from
another tab, which could burn eCW login attempts. So every POST needs both:
1. `Origin` (or, when absent, `Referer`) is `http://127.0.0.1:8000` or
`http://localhost:8000` — else 403.
2. Header `X-Run-Token` equals a token made at server start
(`secrets.token_urlsafe(32)`) and embedded in the page — else 403.
Compared with `secrets.compare_digest`.
No password on the page; `.env` credentials are read only by the bot.
Responses keep `no-store`. The page stays bound to 127.0.0.1. The live log
tail can carry PHI, as the results table already does — local only.
## Error handling
- Already running (page or terminal) → 409 "A run is already going
(started HH:MM)".
- Guard refused (exit 3) → the `[GUARD]` line shown ("next attempt
allowed at ...").
- Stop mid-claim: the claim form may stay open in eCW. VERIFY saves
nothing; the next run's `clear_screen` closes it.
- Web server restarted or page closed: the bot keeps running; on reload
the page finds it through the lock file. `exit_code` is unknown then
(the run is shown "Ended" when its pid is gone).
- Log file not there yet → phase "Starting".
## Testing (standalone "N/M passed" scripts)
- `tests/test_bot_runner.py` — a fake child script in place of `run.py`
(injectable command): start → running → stop → stopped; second start →
`AlreadyRunning`; stale lock (dead pid) cleared; the child sees the forced
env (`SAVE_CORRECTIONS=0`, `ECW_KEEP_OPEN=0`, `QUIET_BOXES=1`,
`CONFIRM_BEFORE_LOOKUP=0`) even when the parent env says otherwise.
- `tests/test_run_status.py` — sample log text → phase, progress, counts,
tail; each exit code's text.
- `tests/test_web_app.py` (added cases) — POST without/with wrong Origin or
token → 403; right Origin + token → 200 (runner stubbed); status JSON
shape; the token is in the page.
- `tests/test_msgbox.py` (added cases) — `QUIET_BOXES=1`: info box logged,
not shown; `needs_person=True` still shown.
- Flow test — `[RESULT]` line per claim; `main()` lock file written,
refused on a live one, removed at the end.
docs/superpowers/specs/2026-10-19-results-portal-redesign-design.md
+96
View File
@@ -0,0 +1,96 @@
# Results portal redesign — design
Date: 2026-10-19. Operator requests (voice, Tanglish), choices made on the
visual companion mockups (`.superpowers/brainstorm/828-1792395410/`).
## Problem
The results page (`web/`, FastAPI + Jinja, 127.0.0.1:8000) opens on the
newest run only. A second run (20 claims) hides the first (10 claims) behind
a "Run" dropdown nobody noticed, so earlier runs look lost. The front page
is three quarters claims table (24 columns, sideways slider), which the
operator does not want on the home page. The look is "very basic".
## What the operator chose
| Question | Choice |
|---|---|
| Home page | **Dashboard**: Bot run card (Start / Stop / live progress), today's tiles, one big **Claim status** button |
| Claim status page | **Date cards**, newest first |
| A date with several runs | **All claims of the day together** (every run's rows in one list) |
| One day's page | **Short table + side panel** |
| Colours | **Navy & blue** (navy header, white cards on light grey, blue buttons) |
| Live run's claim list | In the **Bot run card** ("Live claims" side panel) |
| Search one claim over all dates | **Yes** — "Find a claim" on the Dashboard |
| Approach | **A** — server pages (FastAPI + Jinja), one page per screen |
## Pages
All pages share `base.html`: navy header with the title, a mode pill
(VERIFY / LIVE-WRITE from `SAVE_CORRECTIONS`) and a breadcrumb
(Dashboard › Claim status › 16 Oct 2026). Light grey background, white
cards, blue buttons; Pass green, Fail red, Review amber everywhere.
1. **Dashboard `/`**
- Bot run card — the existing Start / Stop / progress / live log, plus a
**Live claims** link opening a side panel with every claim of the
running (or last) run: Waiting / Checking / result. A claim with a
saved result opens `/day/<date of its run>?claim=<no>`.
- Today tiles: claims checked today, Pass, Fail, Review (0 when no run
today).
- **Find a claim** box → `/claim?claim_no=…`.
- Big **Claim status →** button → `/days`.
- No claims table.
2. **Claim status `/days`** — one card per day (last 90 days with a run),
newest first: weekday + date, claims checked, number of runs,
Pass / Fail / Review and a thin colour bar. A card opens `/day/<date>`.
3. **One day `/day/YYYY-MM-DD`**
- Tiles All / Pass / Fail / Review; a tile is a filter link
(`?category=Fail`). Claim # search box (`?claim_no=`).
- Short table, 10 rows a page (client-side pager): #, Claim #, Time
(the run's time), Payer, Result chip, Comments (one line, cut with
an ellipsis). No sideways scroll.
- Row click → side panel with every Charge Summary field, the run's
time and mode, and the conditions/changes JSON.
- `?claim=<no>` opens that claim's panel on load (on its page).
4. **Claim history `/claim?claim_no=…`** — every saved check of claims
matching the number (newest first, up to 200): date, time, mode,
result, comments; each row opens `/day/<date>?claim=<no>`.
`/?run_at=<time>` (old links) redirects (303) to `/day/<date of that run>`.
`/api/runs`, `/api/results`, `/api/bot/*` stay as they are.
## Data
No schema change. `web/repo.py` gains three parameterized SELECTs:
- `days(limit=90)` — `GROUP BY CAST(run_at AS DATE)`: day, claims,
runs (`COUNT(DISTINCT run_at)`), pass / fail / review counts, newest
first.
- `day_results(day, category, claim_no)` — rows with
`run_at >= day AND run_at < day + 1`, ordered by run_at, S.No, id.
- `claim_history(claim_no)` — `TOP 200 … WHERE claim_no LIKE ?` ordered
by run_at DESC.
Rules: a run belongs to the day it **started** (every row of a run shares
its `run_at`), so a run never splits over two cards. The same claim
checked twice in a day shows twice, each with its time; a day's count is
checks. "Not found" claims have no saved row (as today) and appear only in
the live panel. A legacy driver returning the day as text is coerced to a
date, like `run_at` already is.
## Unchanged
127.0.0.1 only; Host allowlist; `no-store` and the other headers; DB
errors reported by kind; Start / Stop need the page's Origin and token;
page runs are VERIFY only. All new pages are read-only. Page text from the
DB goes out escaped (Jinja autoescape; the panel's data as `tojson` in an
attribute, rendered with `textContent`).
## Testing
`tests/test_web_app.py` (fake repo, fake DB connection): each page renders
with its data and links; filters and the bad-input 400s; the redirect;
the three new SQL statements are parameterized and shaped as above; the
hardening, API and Start / Stop cases kept. A browser check of the four
pages on fixture data (Playwright, the in-app browser).
docs/superpowers/specs/2026-10-22-run-tracking-and-portal-sign-in-design.md
+161
View File
@@ -0,0 +1,161 @@
# Run tracking and Portal Sign-in — design
Date: 2026-10-22. Agreed with the operator in chat, question by question
(Q1–Q31, all "yes" to the recommended answers; Q3 changed to "who" = the
Portal User). Terms: `GLOSSARY.md`. Decision record:
`docs/adr/0001-run-events-in-files.md`. Jira: EEWA-31 (Stage 1),
EEWA-32 (Stage 2), EEWA-33 (Stage 3), each blocking the next.
## Goal
See every Run on the Portal: who started it, how long the eCW login took and
where the time went, every Run Error, Claim Error and Warning with its time,
and which claims a Run never reached. Only signed-in Portal Users see the
Portal; every Run records who started (and stopped) it.
## Decisions
| # | Decision |
|---|---|
| Q1 | Three kinds: **Run Error** (the Run stopped), **Claim Error** (one claim not checked), **Warning** (nothing stopped). **Fail** stays a Charge Summary category only. |
| Q2/Q28 | Per **eCW Login Attempt**: start, end, duration, 2FA mail wait, result; steps timed: page open, password typed, captcha, 2FA mail requested (the Log In click), mail received, approval clicked, home page. |
| Q3/Q8 | "Who" = the **Portal User** who signed in; **Started By** / **Stopped By**. |
| Q4 | Events go to `Logs\events_<run>.jsonl` (ADR 0001). |
| Q5/Q25 | Old run logs are not imported. Nothing is deleted. |
| Q6/Q18/Q19/Q22 | A **Runs** page on the Portal: list + detail, detail refreshes every 5 s while its Run is going. |
| Q9 | The Portal stays on 127.0.0.1. |
| Q10–Q17, Q20, Q21 | Sign-in (Stage 2, below). |
| Q14 | Terminal Runs are allowed; Started By = the Windows user. |
| Q15/Q23 | Stage 3: Run Error banner; `started_by` column in `CV_ChargeSummary`; the Excel sheet unchanged. |
| Q24 | Disk check at Run start: system drive under 2 GB = Warning, under 500 MB = Run Error, exit 2, before login. |
| Q26 | Claim Error = search raised / form not read / never finished loading; Charge Summary still says Review. A claim not found in eCW = Warning. |
| Q27 | **Not Reached** = queued claims a Run Error (or Stop) came before. |
## Stage 1 — Run events and the Runs page (EEWA-31)
### `core/run_events.py` (new)
- `start(log_path)` — the events file is the log's name with `run_` →
`events_` and `.log` → `.jsonl` (`run_20261022_141530.log` →
`events_20261022_141530.jsonl`); the Run's id is `20261022_141530`.
Writes `run_start`.
- `emit(kind, **fields)` — one JSON line, `{"t": local ISO time to the
millisecond, "kind": kind, ...}`, flushed at once. **Never raises**: an
event that cannot be written is dropped, the Run carries on.
- Helpers: `warning(code, message, **f)`, `run_error(step, message,
exit_code, exc=None)` (with `traceback.format_exc()` when an exception is
given), `claim_error(claim, step, message)`, `login_step(step, **f)`,
`login_attempt_start()` / `login_attempt_end(result, message="")` (the
module numbers attempts).
- No OTP code, password or eCW username is ever put in an event.
### Event kinds
| kind | fields |
|---|---|
| `run_start` | `source` (portal / terminal), `started_by`, `mode` (VERIFY / LIVE-WRITE), `claim_limit`, `test_sheet` (file name only), `pid`, `log` |
| `disk` | `drive`, `free_mb` |
| `login_attempt_start` | `attempt` |
| `login_step` | `attempt`, `step`: `page_open`, `password`, `captcha_shown`, `captcha_solved`, `captcha_failed`, `twofa_requested`, `mail_received`, `approval_clicked`, `home` |
| `login_attempt_end` | `attempt`, `result`: `ok`, `twofa_timeout`, `captcha_failed`, `restricted`, `failed`; `message` |
| `queue` | `claims` (list) |
| `claim_start` | `n`, `total`, `claim` |
| `claim_result` | `n`, `claim`, `category` (Pass / Fail / Review / Not found) |
| `claim_error` | `claim`, `step` (`search` / `form` / `load`), `message` |
| `warning` | `code` (`not_found`, `fallback`, `dialog`, `cancel_retry`, `form_not_closed`, `disk_low`, `preflight`), `message`, optional `claim` |
| `screenshot` | `path` (a failure shot), optional `claim` |
| `run_error` | `step` (`config`, `disk`, `guard`, `login`, `restricted`, `twofa_timeout`, `preflight`, `claims`, `unexpected`), `exit_code`, `message`, `traceback` |
| `run_end` | `exit_code` |
| `stopped` | `stopped_by` (written by the Portal after Stop; Stage 2 fills the name) |
`started_by`: env `ECW_STARTED_BY` (the Portal sets it, Stage 2), else the
Windows user. `source`: as `bot_run.json` has it (`ECW_RUN_SOURCE`).
### Where the bot emits
- `flows/export_medcor_ready.py`: `main()` starts the events after
`runlog.start()` and writes `run_end` in its `finally`; `_main` writes a
`run_error` at every non-zero return and in every `except`, with the
traceback; the disk check sits before login next to the CDP check;
`_login_with_retry` wraps each `login()` in attempt start / end;
`_process_claims` writes `queue`, `claim_start`, `claim_result`,
`claim_error` (search raised, `_error`, `_not_evaluated`) and the
not-found Warning.
- `auth/login.py` / `captcha.py` / `twofactor.py` / `mailbox.py`: the login
steps.
- `core/fallback.py` `report()`, `core/page_utils.py` `auto_dialog` (the
event only — its behaviour is unchanged, see Context.md §10) and the
failure `shot()`, `pages/claim_form.py` Cancel retries and form-not-closed,
`pages/preflight.py` FAIL: Warnings / screenshots.
- `web/bot_runner.py` `stop()`: appends `stopped` to the Run's events file.
### Runs page
- `web/runs.py` (new): reads `Logs\events_*.jsonl` into Run summaries and a
Run's detail. A Run with no `run_end` is **Running** when `bot_run.json`
names it and its pid is alive, **Stopped** when a `stopped` event is
there, else **Ended without a record** (killed). Result: Done (exit 0),
Run Error (exit and step), Stopped, Running.
- `GET /runs` — newest first: start, Started By, Run Source, mode,
duration, result, eCW Login Attempts + total login time, Pass / Fail /
Review, Claim Errors, Warnings. Filters: from / to date, result.
- `GET /runs/{run_id}` — the login attempts table (each step's time, 2FA
wait = mail received − 2FA requested), Run Errors (traceback folded),
Claim Errors, Warnings, Not Reached claims, the timeline, screenshot
links; refreshes every 5 s while Running.
- `GET /runs/{run_id}/shot/{n}` — the Run's n-th screenshot event, served
only from inside `Screenshots\` and only as `.png`: a saved page's HTML
served from the Portal's origin would run eCW's scripts there, so the
detail page names where the DOM file is instead of linking it.
- A "Runs" link on every page's header.
### Disk check
Settings `DISK_WARN_MB=2048`, `DISK_STOP_MB=500`, on the system drive.
## Stage 2 — Sign-in (EEWA-32)
- `CV_PortalUser` (username, display name, role, PBKDF2-SHA256 hash + salt
+ iterations, must_change, disabled, created / changed) and
`CV_PortalSignIn` (time, username, event: sign_in / sign_out /
wrong_password / locked / timeout) in `Charge_Validation`.
- Roles Admin / Operator / Viewer. Start / Stop: Admin, Operator. Users
page and Sign-in history: Admin.
- `tools\add_portal_user.py` (password with `getpass`). Passwords ≥ 10
characters; change own; Admin reset → must change at next Sign-in;
disable, never delete.
- 5 wrong passwords → 15 minutes locked. Session cookie (HttpOnly,
SameSite=Strict), 30 minutes idle. Every page and `/api` behind it;
127.0.0.1 + Origin / token checks stay.
- The Portal passes the signed-in user to the child as `ECW_STARTED_BY`;
`stop()` writes `stopped_by`. `bot_run.json` keeps `started_by` too, so
the Dashboard's run card says "started by …".
- As built (`web/users.py`, `web/app.py`): sessions in the Portal's memory
(a restart signs everyone out); disabling a user, resetting their
password or changing their role ends / updates their open session at
once; an unknown username reads and costs the same as a wrong password;
the last active Admin cannot be demoted or disabled, nobody disables
themselves; every Admin change is a `CV_PortalSignIn` row with `by_user`;
every form POST carries the page token and is checked for Origin;
`next` after Sign-in never leaves the site.
## Stage 3 — Alerts and `started_by` (EEWA-33)
- Dashboard banner when the last Run ended in a Run Error.
- `CV_ChargeSummary.started_by` (nullable column, written from the Run's
Started By).
- As built: the banner also shows for a Run that ended without a record
(killed or crashed — no `run_end`), never for Done, Stopped or Running;
it names when, Started By, the step, exit code and message, how many
claims were Not Reached, and links the Run. `create_tables.sql` adds the
column to an existing table (`IF COL_LENGTH ... IS NULL ALTER TABLE`).
The bot checks the column before each insert (`results_db.has_column`)
and saves the rows without it while it is missing — a table not yet
updated never costs a Run its rows. The Portal selects it, shows it as
"Started By" in a claim's side panel, and a table without it reads
"Results table needs an update — run create_tables.sql" (SQLSTATE 42S22).
## Out of scope
Network access to the Portal, email / Teams alerts, importing old logs,
deleting old events, editing settings from the Portal.