ecw-charge-validation/CHANGELOG.md

Changelog — Salud eCW Charge Entry Bot

What changed, when, where, and why. The why matters more than the what: most of the decisions below look strange until you know which eCW behaviour forced them, and without that a future change quietly undoes them.

Dates are the working dates on this machine (IST).

---

Day 1 — 2026-09-02

Status at end of day

Phase	State
Login (proxy + 2FA)	working, proven end to end
Navigate to Billing → Claims	working, proven
Claim Lookup filters	working, proven
Lookup results	click proven; Total Counts never captured
Charge entry	not started

Last verified run: Logs/run_20260902_181115.log.

What was built

src/config.py — every setting, selector and timeout in one place, loaded from .env. Written without python-dotenv on purpose: these machines are locked down and each new dependency is one more thing to get approved.

src/auth.py — the login. Navigation retries, the security-image page, CAPTCHA escalation, two-factor, and the landing page.

src/otp.py — reads the second factor out of the mailbox over IMAP and reports what kind of challenge it is, so auth.py can drive the browser accordingly.

src/nav.py — post-login navigation to the Claim Lookup screen.

src/claims.py — the Claim Lookup form: status, date type, date range, balance, Lookup.

src/guard.py — refuses to run when attempts come too close together.

src/ui.py — operator message boxes that always time out.

src/runlog.py — tees the run to Logs/run_<timestamp>.log, flushed per line so a killed run still leaves a readable log.

src/check_net.py, src/check_imap.py — preflight probes. Run these before blaming the bot.

Decisions that are load-bearing

The proxy is per-browser, not system-wide. chromium.launch(proxy=…) in auth.py. This is the single thing that makes the project possible. Measured:

	no proxy	via proxy
eCW portal	HTTP 400	200
approval link	HTTP 400 → "restricted"	200
webmail site (:2096)	200	403, blocked by proxy
mailbox IMAP :993	connects	not proxied at all

A human is deadlocked by that table — proxy on for eCW, off for the mail, on again to confirm, and no ordering works. The bot escapes it by never opening the webmail website: the browser is proxied, and mail is read by imaplib over a raw socket that ignores the proxy entirely. The Windows system proxy stays off.

"Access to the application has been restricted" is an IP check, not a lockout. Same link: 400 unproxied, 200 proxied, seconds apart. So it almost always means the browser lost the proxy — not that the account is locked. This was initially misdiagnosed as rate limiting; the comments and the 30-minute lockout that came from that were corrected.

The second factor is a push-approval link, not a 6-digit code. The mail contains no digits at all. An earlier build assumed a code and "found" 000000, scraped out of color: #000000 in the mail's stylesheet. Fixed by stripping <style>/<script> before any digit hunting, deleting the bare-6-digit fallback, and modelling the challenge as Challenge(kind, value).

cPanel's webmail 2FA does not apply to IMAP. Roundcube now demands a security code; IMAP/POP3/SMTP are exempt — verified live (CAPABILITY offers AUTH=PLAIN, AUTH=LOGIN, no 2FA mechanism). No code change was needed. Noted so nobody later "fixes" the bot by automating Roundcube.

eCW's push mail is delayed, not dropped. Proven by delivery order: the message eCW stamped 11:19:10 arrived after the one stamped 11:26:10 — about seven minutes late, and it has been seen taking ten. An earlier diagnosis claimed eCW simply was not sending, and the "Resend OTP" work built on that made things worse by shortening the first poll to 60s. Now: 300 + 300 + 300s, resend late rather than early, because a second mail only joins the same slow queue. The send-to-deliver lag is logged from IMAP's INTERNALDATE so this stays visible instead of being guessed at again.

The left-nav toggle is a toggle. #jellybean-panelLink4 runs launchLeftPanel(), which flips a class — clicking it on an open panel closes it. Its id is also duplicated in the DOM, so every locator needs .first, and the panel's contents are injected after DOMContentLoaded, so they must be waited for.

Menu and dropdown options are matched by label, never by generated id. claimLookupLink2ngR1 is an ngRepeat index — it means "the second option", not "Claim Dt(s)". If eCW reorders the list, an id-based click selects a different filter and the search still succeeds. A wrong answer with no error is worse than a crash.

Claim Status runs before everything else on the form. It decides whether a claim-based date type can match anything, and whether the Balance row is editable (ng-change="enableInsurance()" sets enableBalanceFlag). It is chosen by option value (108ME) with the label verified after — "Medcor Ready" sits one line from "Medcor Testing" (113ME), and picking the wrong one returns the wrong queue without any error.

Date type must match Claim Status. With "Encounters Without Claims" — rows that have no claim — Claim Dt(s) returns 0, because those rows have no claim date. Measured on the same range: 936 rows on Service Dt(s), 0 on Claim Dt(s). It does not look like a failure: the grid keeps rendering the previous result set while Total Counts silently drops to 0. Hence the warning, and hence Total Counts now goes into the log.

A disabled control is not an inactive one. The Balance row arrives holding > 0.0 with all five controls disabled, and eCW builds its query from the Angular scope rather than from the inputs — so the filter was narrowing every search while being impossible to click. When the row is disabled, _force_clear_balance() clears the models inside $apply. Inside $apply matters: Angular re-renders the inputs, so what the operator confirms is what eCW will search on. Writing el.value would do the opposite.

Dates go in with fill(), not evaluate. fill() fires the input event that ng-model listens for. Setting .value directly updates the display and leaves Angular's scope stale, so eCW would search on the old value. The jQuery calendar is also closed explicitly and checked — left open it covers the grid and a later click can land on a date cell.

Mistakes made and corrected, same day

• Diagnosed the restriction page as rate limiting. It is an IP check. Corrected in config.py, guard.py, and the lockout duration.
• Diagnosed the missing 2FA mail as "eCW never sent it". It was delayed. The resend feature built on that diagnosis initially made the problem worse.
• Added a rule skipping links older than 9 minutes as expired. With a 10-minute mail delay that would discard a link the moment it arrived; it now warns and clicks anyway.
• Built the fallback claims URL as index.jsp/#/…, which loads nothing.

Operational notes

• LOGIN_COOLDOWN_SECONDS=300, LOGIN_MAX_PER_HOUR=4 — roughly two clean attempts per half hour. ECW_FORCE=1 overrides and is usually the wrong reflex.
• CAPTCHA still needs a human, so unattended scheduling is not possible yet.
• Screenshots land in Screenshots/<MM-DD-YYYY>/ and are the input for the next step's selectors.

---

Day 2 — 2026-09-03

The claim lookup and the Excel export were finished, and three of the four bugs below were found in the bot's own screenshots rather than by guessing.

Claim Status first. The lookup was returning 0 rows against a grid showing 936. Cause: Encounters Without Claims returns rows with no claim, so they have no claim date and Claim Dt(s) cannot match. Medcor Ready (108ME) is now selected before anything else, which also enables the Balance row.

wait_for_load_state("networkidle") never returns on this page. eCW's Angular shell polls in the background, so the network is never quiet for the 500 ms Playwright wants. It printed nothing for its full 5-minute timeout, every run looked hung and was killed at exactly that line — which is why the export had never once run. Replaced with a poll of the grid's own "Total Counts" that narrates the wait.

A single changed count is not enough. eCW blanks Total Counts to 0 the instant Lookup is pressed, so the first poll saw 0 != 894 and logged a row count of 0 for a search that returned 894. A figure now has to hold across consecutive reads, and a 0 for twice as long — it is both a legitimate answer and the transient.

download.save_as() copies. Chromium's _pw_downloads\<guid> survives it, byte-identical to the saved report. The orphan sweep at the top of the next export would have adopted it and written a second copy of a report already saved. download.delete() now runs after the save. A real 552 KB xlsx was found sitting in that folder unnamed, from a run killed before save_as.

The page was left scrolled. Playwright scrolls an element into view before clicking it, so reaching the toolbar's Billing menu scrolled the filter form off the top. Reported as "the portal looks broken" when the form was simply above the viewport. The first fix used window.scrollTo(0, 0) and did nothing — on this SPA the document does not scroll, an inner container does. Now scroll_into_view_if_needed() lets Playwright find the real container, and the sweep logs which element it was.

The guard was refusing runs it should have waited out. Five of six consecutive runs died on the cooldown, the last with 3 seconds left. Re-running by hand until the clock clears is exactly the pattern the cooldown exists to discourage, so wait_if_cooling() now sleeps a plain cooldown and proceeds. The burst limit and the restriction lockout still refuse. Writing the test caught a bug in the fix itself: it measured with time.time() while check() measured with datetime.now(), and two clocks for one deadline is how a run waits out its cooldown and is then refused for being a fraction early.

Also: the confirm box before Lookup no longer aborts on a timeout (it only reads, and aborting threw away a completed login), then was turned off entirely; the Balance row is left untouched when it is already empty; and a message box reports the export.

---

Day 3 — 2026-09-07

Restructured into a package. No behaviour change — this was moves and imports, verified rather than assumed.

It had grown to eleven flat modules in src\, 3,253 lines, with claims.py alone at 952 (nearly a third of the project in one file) and config.py mixing 43 DOM selectors in with the date range and the proxy. Those change for different reasons — a selector when eCW ships a build, a setting when the process changes, a flow when the work changes — so in one file, editing a date range and editing a selector were edits to the same place. The charge-entry phases would have pushed claims.py past 1,500 lines.

Now run.py plus src\ecw_bot\ with six layers, none importing a layer above it: settings → selectors → core → auth / pages → flows. Diagnostics moved to tools\, and tests\ exists for the first time.

run.py is the real fix hiding inside the cosmetics. python src\main.py worked only because Python puts a script's own directory on sys.path, so import config resolved and no other way of starting the bot did — not VS Code's "Run File" on a different file, not Task Scheduler, not pytest. The entry point no longer depends on which file you happened to launch.

Verified: 31 files compile with SyntaxWarning as an error (which caught five Windows paths in docstrings needing raw strings), all 24 modules import, an AST pass found 0 undefined names across the split, every one of the old definitions is accounted for in a new home, tests\test_settings.py 3/3, tests\test_guard.py 6/6, and tools\check_imap.py still reaches the mailbox through the new import path.

Not yet run live against eCW. The next run is the first real test of the new layout, and of the scroll and Total Counts fixes from Day 2.

---

Day 4 — 2026-09-15

Automated the payer categorization step. Reviewed against Salud-Charge Entry_Medcor Ready.mp4 (2:45-10:00): after the Medcor Ready export, an operator was filtering the report by PAYER and hand-filling a Group Name column (Commercial, Medicaid, Medicare, Dentaquest, Program, ...) — the same manual, repetitive, rule-based step every other phase of this bot exists to remove.

core/payer_groups.py does the lookup: exact string match against payer_groups.json, on purpose — eCW's own PAYER values already differ by case and spacing for what is really the same carrier (United Healthcare / United Health Care / United healthcare are three separate values in one real export), so anything looser has no safe way to pick which known string a new one is closest to. An unrecognized payer gets a blank Group Name, not a guess, and is logged and message-boxed — this is billing data, and wrong is worse than blank.

core/spreadsheet.py gained a write side (write_xlsx()) to go with its read side, built the same way — stdlib only, no openpyxl — but with the full minimal OOXML part set ([Content_Types].xml, both .rels files, workbook.xml) the reader doesn't need but real Excel does, since this file is meant to be opened by a person.

The categorized output is a new file, Output\Medcor Ready_<date>.xlsx — Input\Claim Report_<date>.xlsx is never modified, because claim_file.py and the charge-entry phase read its columns by position.

Seeded, not complete. The mapping table ships with 23 payers covering roughly 95% of a real export's row volume (Medicaid, DentaQuest, and the clearly-commercial carriers), derived from the actual current export rather than guessed from the video. 17 payers (BH COLORADO ACCESS, the Delta Dental variants, Tricare, Devoted, several TPA names, ...) are deliberately left out — first run will flag them, and filling them in is a one-line JSON edit, no redeploy.

Out of scope for this round: splitting rows into the Patient / Pending with Error / Program sibling workbooks the video also shows, and writing anything to the network Charge\Allocation\... folder — both stay manual for now.

---

Day 5 — 2026-09-15

The bot's first write to eCW. Reviewed the rest of the same video (10:00-45:32): the actual "Charge Validation" work is a per-claim clinical audit — open the UB form, cross-check ICD/CPT/modifiers against the Progress Note and a payer-specific rules sheet, fix one of a handful of standard defect types, save, and log the outcome. That's a much bigger surface than the categorization step, so it's being built one condition at a time rather than all at once — this is the first.

Condition 1: telehealth visit → POS=11, M1=GT. claim_form.py gained three functions: note_mentions_telehealth() (a plain case-insensitive check against the Progress Note's own text — deliberately not narrower, since the note's own heading for this is written inconsistently: TELEHEALTH:, BHTelehealth Visit, Session Setting (PoS): Telehealth), set_cpt_pos_modifier() (sets POS and M1 on every real CPT/HCPCS row — skipping eCW's blank padding rows and the trailing * entry row, the same filter extract() already applies — and reads every field back so a write that silently didn't take is a loud failure, not a quiet gap), and save_claim() (the first place in this codebase that clicks OK instead of Cancel — saveAllData(true)).

Nothing saves without an explicit per-claim confirmation box. This is the first write path this bot has ever had, so the new APPLY_CORRECTIONS setting gates whether the check even runs, and even then every fix stops for an OK/Cancel box naming exactly what changed on which rows before save_claim() is ever called. Declining leaves the claim exactly as it was — the field changes only ever live in the DOM until OK is clicked, so Cancel discards them for free.

The selectors needed for the write (uBClaimPOS*, uBClaimMod1* inside each CPTROW_*) needed no new live-eCW inspection — they fell out of extract()'s own existing id-prefix reader, which already had to name them to read the grid.

tests/test_telehealth_fix.py reuses the existing ub_claim.html / progress_note.html fixtures rather than adding new ones — the real note eCW served on 2026-09-07 already documents a Telehealth visit, so the positive case needs no new fixture, and detection never touches the DOM at all, so the negative case is a plain string. ub_claim.html's Cancel and OK buttons needed a few lines of script to actually hide the modal on click, which they never did before — the original fixture only ever had to test reading, never closing, so this is the first thing to exercise that path either.

Not yet run live. CLAIM_PROCESS_LIMIT=1 is still the default, so the first real run after this lands tests on exactly one real claim — the first live proof that the CPT/HCPCS tab is still the active tab-pane by the time the fix runs (nothing between open_cpt_hcpcs_tab() and here switches away from it, but this is the one assumption the fixture cannot prove, since Bootstrap's tab-pane visibility isn't something a static test page reproduces the same way live Angular does).