ecw-charge-validation/CHANGELOG.md

# Changelog — Salud eCW Charge Entry Bot

What changed, when, where, and **why**. The why matters more than the what: most
of the decisions below look strange until you know which eCW behaviour forced
them, and without that a future change quietly undoes them.

Dates are the working dates on this machine (IST).

---

## Day 1 — 2026-09-02

### Status at end of day

| Phase | State |
|---|---|
| Login (proxy + 2FA) | **working, proven end to end** |
| Navigate to Billing → Claims | **working, proven** |
| Claim Lookup filters | **working, proven** |
| Lookup results | click proven; `Total Counts` never captured |
| Charge entry | not started |

Last verified run: `Logs/run_20260902_181115.log`.

### What was built

**`src/config.py`** — every setting, selector and timeout in one place, loaded
from `.env`. Written without `python-dotenv` on purpose: these machines are
locked down and each new dependency is one more thing to get approved.

**`src/auth.py`** — the login. Navigation retries, the security-image page,
CAPTCHA escalation, two-factor, and the landing page.

**`src/otp.py`** — reads the second factor out of the mailbox over IMAP and
reports *what kind* of challenge it is, so `auth.py` can drive the browser
accordingly.

**`src/nav.py`** — post-login navigation to the Claim Lookup screen.

**`src/claims.py`** — the Claim Lookup form: status, date type, date range,
balance, Lookup.

**`src/guard.py`** — refuses to run when attempts come too close together.

**`src/ui.py`** — operator message boxes that always time out.

**`src/runlog.py`** — tees the run to `Logs/run_<timestamp>.log`, flushed per
line so a killed run still leaves a readable log.

**`src/check_net.py`, `src/check_imap.py`** — preflight probes. Run these before
blaming the bot.

### Decisions that are load-bearing

**The proxy is per-browser, not system-wide.** `chromium.launch(proxy=…)` in
`auth.py`. This is the single thing that makes the project possible. Measured:

| | no proxy | via proxy |
|---|---|---|
| eCW portal | HTTP 400 | **200** |
| approval link | HTTP 400 → "restricted" | **200** |
| webmail site (:2096) | **200** | 403, blocked by proxy |
| mailbox IMAP :993 | **connects** | not proxied at all |

A human is deadlocked by that table — proxy on for eCW, off for the mail, on
again to confirm, and no ordering works. The bot escapes it by never opening the
webmail *website*: the browser is proxied, and mail is read by `imaplib` over a
raw socket that ignores the proxy entirely. The Windows system proxy stays off.

**"Access to the application has been restricted" is an IP check, not a
lockout.** Same link: 400 unproxied, 200 proxied, seconds apart. So it almost
always means the browser lost the proxy — not that the account is locked. This
was initially misdiagnosed as rate limiting; the comments and the 30-minute
lockout that came from that were corrected.

**The second factor is a push-approval link, not a 6-digit code.** The mail
contains no digits at all. An earlier build assumed a code and "found" `000000`,
scraped out of `color: #000000` in the mail's stylesheet. Fixed by stripping
`<style>`/`<script>` before any digit hunting, deleting the bare-6-digit
fallback, and modelling the challenge as `Challenge(kind, value)`.

**cPanel's webmail 2FA does not apply to IMAP.** Roundcube now demands a
security code; IMAP/POP3/SMTP are exempt — verified live (`CAPABILITY` offers
`AUTH=PLAIN`, `AUTH=LOGIN`, no 2FA mechanism). No code change was needed. Noted
so nobody later "fixes" the bot by automating Roundcube.

**eCW's push mail is delayed, not dropped.** Proven by delivery order: the
message eCW stamped `11:19:10` arrived *after* the one stamped `11:26:10` —
about seven minutes late, and it has been seen taking ten. An earlier diagnosis
claimed eCW simply was not sending, and the "Resend OTP" work built on that made
things worse by shortening the first poll to 60s. Now: 300 + 300 + 300s, resend
late rather than early, because a second mail only joins the same slow queue.
The send-to-deliver lag is logged from IMAP's `INTERNALDATE` so this stays
visible instead of being guessed at again.

**The left-nav toggle is a toggle.** `#jellybean-panelLink4` runs
`launchLeftPanel()`, which flips a class — clicking it on an open panel *closes*
it. Its id is also duplicated in the DOM, so every locator needs `.first`, and
the panel's contents are injected after `DOMContentLoaded`, so they must be
waited for.

**Menu and dropdown options are matched by label, never by generated id.**
`claimLookupLink2ngR1` is an ngRepeat *index* — it means "the second option",
not "Claim Dt(s)". If eCW reorders the list, an id-based click selects a
different filter and the search still succeeds. A wrong answer with no error is
worse than a crash.

**Claim Status runs before everything else on the form.** It decides whether a
claim-based date type can match anything, and whether the Balance row is
editable (`ng-change="enableInsurance()"` sets `enableBalanceFlag`). It is
chosen by option *value* (`108ME`) with the *label* verified after — "Medcor
Ready" sits one line from "Medcor Testing" (`113ME`), and picking the wrong one
returns the wrong queue without any error.

**Date type must match Claim Status.** With "Encounters Without Claims" — rows
that have no claim — `Claim Dt(s)` returns 0, because those rows have no claim
date. Measured on the same range: 936 rows on `Service Dt(s)`, 0 on
`Claim Dt(s)`. It does not look like a failure: the grid keeps rendering the
*previous* result set while `Total Counts` silently drops to 0. Hence the
warning, and hence `Total Counts` now goes into the log.

**A disabled control is not an inactive one.** The Balance row arrives holding
`> 0.0` with all five controls `disabled`, and eCW builds its query from the
Angular scope rather than from the inputs — so the filter was narrowing every
search while being impossible to click. When the row is disabled,
`_force_clear_balance()` clears the models inside `$apply`. Inside `$apply`
matters: Angular re-renders the inputs, so what the operator confirms is what
eCW will search on. Writing `el.value` would do the opposite.

**Dates go in with `fill()`, not `evaluate`.** `fill()` fires the `input` event
that `ng-model` listens for. Setting `.value` directly updates the display and
leaves Angular's scope stale, so eCW would search on the old value. The jQuery
calendar is also closed explicitly and checked — left open it covers the grid
and a later click can land on a date cell.

### Mistakes made and corrected, same day

- Diagnosed the restriction page as rate limiting. It is an IP check. Corrected
  in `config.py`, `guard.py`, and the lockout duration.
- Diagnosed the missing 2FA mail as "eCW never sent it". It was delayed. The
  resend feature built on that diagnosis initially made the problem worse.
- Added a rule skipping links older than 9 minutes as expired. With a 10-minute
  mail delay that would discard a link the moment it arrived; it now warns and
  clicks anyway.
- Built the fallback claims URL as `index.jsp/#/…`, which loads nothing.

### Operational notes

- `LOGIN_COOLDOWN_SECONDS=300`, `LOGIN_MAX_PER_HOUR=4` — roughly two clean
  attempts per half hour. `ECW_FORCE=1` overrides and is usually the wrong
  reflex.
- CAPTCHA still needs a human, so unattended scheduling is not possible yet.
- Screenshots land in `Screenshots/<MM-DD-YYYY>/` and are the input for the next
  step's selectors.

---

## Day 2 — 2026-09-03

The claim lookup and the Excel export were finished, and three of the four bugs
below were found in the bot's own screenshots rather than by guessing.

**Claim Status first.** The lookup was returning 0 rows against a grid showing
936. Cause: `Encounters Without Claims` returns rows with no claim, so they have
no claim date and `Claim Dt(s)` cannot match. `Medcor Ready` (`108ME`) is now
selected before anything else, which also enables the Balance row.

**`wait_for_load_state("networkidle")` never returns on this page.** eCW's
Angular shell polls in the background, so the network is never quiet for the
500 ms Playwright wants. It printed nothing for its full 5-minute timeout, every
run looked hung and was killed at exactly that line — which is why the export
had never once run. Replaced with a poll of the grid's own "Total Counts" that
narrates the wait.

**A single changed count is not enough.** eCW blanks Total Counts to 0 the
instant Lookup is pressed, so the first poll saw `0 != 894` and logged a row
count of 0 for a search that returned 894. A figure now has to hold across
consecutive reads, and a `0` for twice as long — it is both a legitimate answer
and the transient.

**`download.save_as()` copies.** Chromium's `_pw_downloads\<guid>` survives it,
byte-identical to the saved report. The orphan sweep at the top of the next
export would have adopted it and written a second copy of a report already
saved. `download.delete()` now runs after the save. A real 552 KB xlsx was found
sitting in that folder unnamed, from a run killed before `save_as`.

**The page was left scrolled.** Playwright scrolls an element into view before
clicking it, so reaching the toolbar's Billing menu scrolled the filter form off
the top. Reported as "the portal looks broken" when the form was simply above
the viewport. The first fix used `window.scrollTo(0, 0)` and **did nothing** —
on this SPA the document does not scroll, an inner container does. Now
`scroll_into_view_if_needed()` lets Playwright find the real container, and the
sweep logs which element it was.

**The guard was refusing runs it should have waited out.** Five of six
consecutive runs died on the cooldown, the last with **3 seconds** left.
Re-running by hand until the clock clears is exactly the pattern the cooldown
exists to discourage, so `wait_if_cooling()` now sleeps a plain cooldown and
proceeds. The burst limit and the restriction lockout still refuse. Writing the
test caught a bug in the fix itself: it measured with `time.time()` while
`check()` measured with `datetime.now()`, and two clocks for one deadline is how
a run waits out its cooldown and is then refused for being a fraction early.

Also: the confirm box before Lookup no longer aborts on a timeout (it only
reads, and aborting threw away a completed login), then was turned off
entirely; the Balance row is left untouched when it is already empty; and a
message box reports the export.

---

## Day 3 — 2026-09-07

**Restructured into a package.** No behaviour change — this was moves and
imports, verified rather than assumed.

It had grown to eleven flat modules in `src\`, 3,253 lines, with `claims.py`
alone at 952 (nearly a third of the project in one file) and `config.py` mixing
43 DOM selectors in with the date range and the proxy. Those change for
different reasons — a selector when eCW ships a build, a setting when the
process changes, a flow when the work changes — so in one file, editing a date
range and editing a selector were edits to the same place. The charge-entry
phases would have pushed `claims.py` past 1,500 lines.

Now `run.py` plus `src\ecw_bot\` with six layers, none importing a layer above
it: `settings` → `selectors` → `core` → `auth` / `pages` → `flows`. Diagnostics
moved to `tools\`, and `tests\` exists for the first time.

`run.py` is the real fix hiding inside the cosmetics. `python src\main.py`
worked *only* because Python puts a script's own directory on `sys.path`, so
`import config` resolved and no other way of starting the bot did — not VS
Code's "Run File" on a different file, not Task Scheduler, not pytest. The
entry point no longer depends on which file you happened to launch.

Verified: 31 files compile with `SyntaxWarning` as an error (which caught five
Windows paths in docstrings needing raw strings), all 24 modules import, an AST
pass found 0 undefined names across the split, every one of the old
definitions is accounted for in a new home, `tests\test_settings.py` 3/3,
`tests\test_guard.py` 6/6, and `tools\check_imap.py` still reaches the mailbox
through the new import path.

**Not yet run live against eCW.** The next run is the first real test of the
new layout, and of the scroll and Total Counts fixes from Day 2.

---

## Day 4 — 2026-09-15

**Automated the payer categorization step.** Reviewed against
`Salud-Charge Entry_Medcor Ready.mp4` (2:45-10:00): after the Medcor Ready
export, an operator was filtering the report by `PAYER` and hand-filling a
`Group Name` column (`Commercial`, `Medicaid`, `Medicare`, `Dentaquest`,
`Program`, ...) — the same manual, repetitive, rule-based step every other
phase of this bot exists to remove.

`core/payer_groups.py` does the lookup: exact string match against
`payer_groups.json`, on purpose — eCW's own PAYER values already differ by
case and spacing for what is really the same carrier (`United Healthcare` /
`United Health Care` / `United healthcare` are three separate values in one
real export), so anything looser has no safe way to pick which known string
a new one is closest to. An unrecognized payer gets a **blank** Group Name,
not a guess, and is logged and message-boxed — this is billing data, and
wrong is worse than blank.

`core/spreadsheet.py` gained a write side (`write_xlsx()`) to go with its
read side, built the same way — stdlib only, no openpyxl — but with the full
minimal OOXML part set (`[Content_Types].xml`, both `.rels` files,
`workbook.xml`) the reader doesn't need but real Excel does, since this file
is meant to be opened by a person.

The categorized output is a **new** file, `Output\Medcor Ready_<date>.xlsx`
— `Input\Claim Report_<date>.xlsx` is never modified, because `claim_file.py`
and the charge-entry phase read its columns by position.

**Seeded, not complete.** The mapping table ships with 23 payers covering
roughly 95% of a real export's row volume (`Medicaid`, `DentaQuest`, and the
clearly-commercial carriers), derived from the actual current export rather
than guessed from the video. 17 payers (`BH COLORADO ACCESS`, the Delta
Dental variants, `Tricare`, `Devoted`, several TPA names, ...) are
deliberately left out — first run will flag them, and filling them in is a
one-line JSON edit, no redeploy.

**Out of scope for this round:** splitting rows into the `Patient` /
`Pending with Error` / `Program` sibling workbooks the video also shows, and
writing anything to the network `Charge\Allocation\...` folder — both stay
manual for now.

---

## Day 5 — 2026-09-15

**The bot's first write to eCW.** Reviewed the rest of the same video
(10:00-45:32): the actual "Charge Validation" work is a per-claim clinical
audit — open the UB form, cross-check ICD/CPT/modifiers against the
Progress Note and a payer-specific rules sheet, fix one of a handful of
standard defect types, save, and log the outcome. That's a much bigger
surface than the categorization step, so it's being built one condition at
a time rather than all at once — this is the first.

**Condition 1: telehealth visit → POS=11, M1=GT.** `claim_form.py` gained
three functions: `note_mentions_telehealth()` (a plain case-insensitive
check against the Progress Note's own text — deliberately not narrower,
since the note's own heading for this is written inconsistently:
`TELEHEALTH:`, `BHTelehealth Visit`, `Session Setting (PoS): Telehealth`),
`set_cpt_pos_modifier()` (sets POS and M1 on every real CPT/HCPCS row —
skipping eCW's blank padding rows and the trailing `*` entry row, the same
filter `extract()` already applies — and reads every field back so a write
that silently didn't take is a loud failure, not a quiet gap), and
`save_claim()` (the first place in this codebase that clicks OK instead of
Cancel — `saveAllData(true)`).

**Nothing saves without an explicit per-claim confirmation box.** This is
the first write path this bot has ever had, so the new `APPLY_CORRECTIONS`
setting gates whether the check even runs, and even then every fix stops
for an OK/Cancel box naming exactly what changed on which rows before
`save_claim()` is ever called. Declining leaves the claim exactly as it
was — the field changes only ever live in the DOM until OK is clicked, so
Cancel discards them for free.

The selectors needed for the write (`uBClaimPOS*`, `uBClaimMod1*` inside
each `CPTROW_*`) needed no new live-eCW inspection — they fell out of
`extract()`'s own existing id-prefix reader, which already had to name them
to read the grid.

`tests/test_telehealth_fix.py` reuses the existing `ub_claim.html` /
`progress_note.html` fixtures rather than adding new ones — the real note
eCW served on 2026-09-07 already documents a Telehealth visit, so the
positive case needs no new fixture, and detection never touches the DOM at
all, so the negative case is a plain string. `ub_claim.html`'s Cancel and
OK buttons needed a few lines of script to actually hide the modal on
click, which they never did before — the original fixture only ever had to
test reading, never closing, so this is the first thing to exercise that
path either.

**Not yet run live.** `CLAIM_PROCESS_LIMIT=1` is still the default, so the
first real run after this lands tests on exactly one real claim — the
first live proof that the CPT/HCPCS tab is still the active tab-pane by
the time the fix runs (nothing between `open_cpt_hcpcs_tab()` and here
switches away from it, but this is the one assumption the fixture cannot
prove, since Bootstrap's tab-pane visibility isn't something a static test
page reproduces the same way live Angular does).