Initial import: eCW Charge Validation bot and results Portal
Snapshot of the working project as of 2026-10-23 (local 2d4d4fc), one commit with no earlier history: the tracked files were de-identified first (synthetic claim ids and patient data only; see Context.md section 7). Credentials, run logs, screenshots, claim reports and client workbooks are gitignored and not included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
ManikandanandClaude Opus 5.5 committed
commit
ba15a574ab
205 files changed
+56772
No files matched your search
+340
@@ -0,0 +1,340 @@
|
||||
# Changelog — Salud eCW Charge Entry Bot
|
||||
|
||||
What changed, when, where, and **why**. The why matters more than the what: most
|
||||
of the decisions below look strange until you know which eCW behaviour forced
|
||||
them, and without that a future change quietly undoes them.
|
||||
|
||||
Dates are the working dates on this machine (IST).
|
||||
|
||||
---
|
||||
|
||||
## Day 1 — 2026-09-02
|
||||
|
||||
### Status at end of day
|
||||
|
||||
| Phase | State |
|
||||
|---|---|
|
||||
| Login (proxy + 2FA) | **working, proven end to end** |
|
||||
| Navigate to Billing → Claims | **working, proven** |
|
||||
| Claim Lookup filters | **working, proven** |
|
||||
| Lookup results | click proven; `Total Counts` never captured |
|
||||
| Charge entry | not started |
|
||||
|
||||
Last verified run: `Logs/run_20260902_181115.log`.
|
||||
|
||||
### What was built
|
||||
|
||||
**`src/config.py`** — every setting, selector and timeout in one place, loaded
|
||||
from `.env`. Written without `python-dotenv` on purpose: these machines are
|
||||
locked down and each new dependency is one more thing to get approved.
|
||||
|
||||
**`src/auth.py`** — the login. Navigation retries, the security-image page,
|
||||
CAPTCHA escalation, two-factor, and the landing page.
|
||||
|
||||
**`src/otp.py`** — reads the second factor out of the mailbox over IMAP and
|
||||
reports *what kind* of challenge it is, so `auth.py` can drive the browser
|
||||
accordingly.
|
||||
|
||||
**`src/nav.py`** — post-login navigation to the Claim Lookup screen.
|
||||
|
||||
**`src/claims.py`** — the Claim Lookup form: status, date type, date range,
|
||||
balance, Lookup.
|
||||
|
||||
**`src/guard.py`** — refuses to run when attempts come too close together.
|
||||
|
||||
**`src/ui.py`** — operator message boxes that always time out.
|
||||
|
||||
**`src/runlog.py`** — tees the run to `Logs/run_<timestamp>.log`, flushed per
|
||||
line so a killed run still leaves a readable log.
|
||||
|
||||
**`src/check_net.py`, `src/check_imap.py`** — preflight probes. Run these before
|
||||
blaming the bot.
|
||||
|
||||
### Decisions that are load-bearing
|
||||
|
||||
**The proxy is per-browser, not system-wide.** `chromium.launch(proxy=…)` in
|
||||
`auth.py`. This is the single thing that makes the project possible. Measured:
|
||||
|
||||
| | no proxy | via proxy |
|
||||
|---|---|---|
|
||||
| eCW portal | HTTP 400 | **200** |
|
||||
| approval link | HTTP 400 → "restricted" | **200** |
|
||||
| webmail site (:2096) | **200** | 403, blocked by proxy |
|
||||
| mailbox IMAP :993 | **connects** | not proxied at all |
|
||||
|
||||
A human is deadlocked by that table — proxy on for eCW, off for the mail, on
|
||||
again to confirm, and no ordering works. The bot escapes it by never opening the
|
||||
webmail *website*: the browser is proxied, and mail is read by `imaplib` over a
|
||||
raw socket that ignores the proxy entirely. The Windows system proxy stays off.
|
||||
|
||||
**"Access to the application has been restricted" is an IP check, not a
|
||||
lockout.** Same link: 400 unproxied, 200 proxied, seconds apart. So it almost
|
||||
always means the browser lost the proxy — not that the account is locked. This
|
||||
was initially misdiagnosed as rate limiting; the comments and the 30-minute
|
||||
lockout that came from that were corrected.
|
||||
|
||||
**The second factor is a push-approval link, not a 6-digit code.** The mail
|
||||
contains no digits at all. An earlier build assumed a code and "found" `000000`,
|
||||
scraped out of `color: #000000` in the mail's stylesheet. Fixed by stripping
|
||||
`<style>`/`<script>` before any digit hunting, deleting the bare-6-digit
|
||||
fallback, and modelling the challenge as `Challenge(kind, value)`.
|
||||
|
||||
**cPanel's webmail 2FA does not apply to IMAP.** Roundcube now demands a
|
||||
security code; IMAP/POP3/SMTP are exempt — verified live (`CAPABILITY` offers
|
||||
`AUTH=PLAIN`, `AUTH=LOGIN`, no 2FA mechanism). No code change was needed. Noted
|
||||
so nobody later "fixes" the bot by automating Roundcube.
|
||||
|
||||
**eCW's push mail is delayed, not dropped.** Proven by delivery order: the
|
||||
message eCW stamped `11:19:10` arrived *after* the one stamped `11:26:10` —
|
||||
about seven minutes late, and it has been seen taking ten. An earlier diagnosis
|
||||
claimed eCW simply was not sending, and the "Resend OTP" work built on that made
|
||||
things worse by shortening the first poll to 60s. Now: 300 + 300 + 300s, resend
|
||||
late rather than early, because a second mail only joins the same slow queue.
|
||||
The send-to-deliver lag is logged from IMAP's `INTERNALDATE` so this stays
|
||||
visible instead of being guessed at again.
|
||||
|
||||
**The left-nav toggle is a toggle.** `#jellybean-panelLink4` runs
|
||||
`launchLeftPanel()`, which flips a class — clicking it on an open panel *closes*
|
||||
it. Its id is also duplicated in the DOM, so every locator needs `.first`, and
|
||||
the panel's contents are injected after `DOMContentLoaded`, so they must be
|
||||
waited for.
|
||||
|
||||
**Menu and dropdown options are matched by label, never by generated id.**
|
||||
`claimLookupLink2ngR1` is an ngRepeat *index* — it means "the second option",
|
||||
not "Claim Dt(s)". If eCW reorders the list, an id-based click selects a
|
||||
different filter and the search still succeeds. A wrong answer with no error is
|
||||
worse than a crash.
|
||||
|
||||
**Claim Status runs before everything else on the form.** It decides whether a
|
||||
claim-based date type can match anything, and whether the Balance row is
|
||||
editable (`ng-change="enableInsurance()"` sets `enableBalanceFlag`). It is
|
||||
chosen by option *value* (`108ME`) with the *label* verified after — "Medcor
|
||||
Ready" sits one line from "Medcor Testing" (`113ME`), and picking the wrong one
|
||||
returns the wrong queue without any error.
|
||||
|
||||
**Date type must match Claim Status.** With "Encounters Without Claims" — rows
|
||||
that have no claim — `Claim Dt(s)` returns 0, because those rows have no claim
|
||||
date. Measured on the same range: 936 rows on `Service Dt(s)`, 0 on
|
||||
`Claim Dt(s)`. It does not look like a failure: the grid keeps rendering the
|
||||
*previous* result set while `Total Counts` silently drops to 0. Hence the
|
||||
warning, and hence `Total Counts` now goes into the log.
|
||||
|
||||
**A disabled control is not an inactive one.** The Balance row arrives holding
|
||||
`> 0.0` with all five controls `disabled`, and eCW builds its query from the
|
||||
Angular scope rather than from the inputs — so the filter was narrowing every
|
||||
search while being impossible to click. When the row is disabled,
|
||||
`_force_clear_balance()` clears the models inside `$apply`. Inside `$apply`
|
||||
matters: Angular re-renders the inputs, so what the operator confirms is what
|
||||
eCW will search on. Writing `el.value` would do the opposite.
|
||||
|
||||
**Dates go in with `fill()`, not `evaluate`.** `fill()` fires the `input` event
|
||||
that `ng-model` listens for. Setting `.value` directly updates the display and
|
||||
leaves Angular's scope stale, so eCW would search on the old value. The jQuery
|
||||
calendar is also closed explicitly and checked — left open it covers the grid
|
||||
and a later click can land on a date cell.
|
||||
|
||||
### Mistakes made and corrected, same day
|
||||
|
||||
- Diagnosed the restriction page as rate limiting. It is an IP check. Corrected
|
||||
in `config.py`, `guard.py`, and the lockout duration.
|
||||
- Diagnosed the missing 2FA mail as "eCW never sent it". It was delayed. The
|
||||
resend feature built on that diagnosis initially made the problem worse.
|
||||
- Added a rule skipping links older than 9 minutes as expired. With a 10-minute
|
||||
mail delay that would discard a link the moment it arrived; it now warns and
|
||||
clicks anyway.
|
||||
- Built the fallback claims URL as `index.jsp/#/…`, which loads nothing.
|
||||
|
||||
### Operational notes
|
||||
|
||||
- `LOGIN_COOLDOWN_SECONDS=300`, `LOGIN_MAX_PER_HOUR=4` — roughly two clean
|
||||
attempts per half hour. `ECW_FORCE=1` overrides and is usually the wrong
|
||||
reflex.
|
||||
- CAPTCHA still needs a human, so unattended scheduling is not possible yet.
|
||||
- Screenshots land in `Screenshots/<MM-DD-YYYY>/` and are the input for the next
|
||||
step's selectors.
|
||||
|
||||
---
|
||||
|
||||
## Day 2 — 2026-09-03
|
||||
|
||||
The claim lookup and the Excel export were finished, and three of the four bugs
|
||||
below were found in the bot's own screenshots rather than by guessing.
|
||||
|
||||
**Claim Status first.** The lookup was returning 0 rows against a grid showing
|
||||
936. Cause: `Encounters Without Claims` returns rows with no claim, so they have
|
||||
no claim date and `Claim Dt(s)` cannot match. `Medcor Ready` (`108ME`) is now
|
||||
selected before anything else, which also enables the Balance row.
|
||||
|
||||
**`wait_for_load_state("networkidle")` never returns on this page.** eCW's
|
||||
Angular shell polls in the background, so the network is never quiet for the
|
||||
500 ms Playwright wants. It printed nothing for its full 5-minute timeout, every
|
||||
run looked hung and was killed at exactly that line — which is why the export
|
||||
had never once run. Replaced with a poll of the grid's own "Total Counts" that
|
||||
narrates the wait.
|
||||
|
||||
**A single changed count is not enough.** eCW blanks Total Counts to 0 the
|
||||
instant Lookup is pressed, so the first poll saw `0 != 894` and logged a row
|
||||
count of 0 for a search that returned 894. A figure now has to hold across
|
||||
consecutive reads, and a `0` for twice as long — it is both a legitimate answer
|
||||
and the transient.
|
||||
|
||||
**`download.save_as()` copies.** Chromium's `_pw_downloads\<guid>` survives it,
|
||||
byte-identical to the saved report. The orphan sweep at the top of the next
|
||||
export would have adopted it and written a second copy of a report already
|
||||
saved. `download.delete()` now runs after the save. A real 552 KB xlsx was found
|
||||
sitting in that folder unnamed, from a run killed before `save_as`.
|
||||
|
||||
**The page was left scrolled.** Playwright scrolls an element into view before
|
||||
clicking it, so reaching the toolbar's Billing menu scrolled the filter form off
|
||||
the top. Reported as "the portal looks broken" when the form was simply above
|
||||
the viewport. The first fix used `window.scrollTo(0, 0)` and **did nothing** —
|
||||
on this SPA the document does not scroll, an inner container does. Now
|
||||
`scroll_into_view_if_needed()` lets Playwright find the real container, and the
|
||||
sweep logs which element it was.
|
||||
|
||||
**The guard was refusing runs it should have waited out.** Five of six
|
||||
consecutive runs died on the cooldown, the last with **3 seconds** left.
|
||||
Re-running by hand until the clock clears is exactly the pattern the cooldown
|
||||
exists to discourage, so `wait_if_cooling()` now sleeps a plain cooldown and
|
||||
proceeds. The burst limit and the restriction lockout still refuse. Writing the
|
||||
test caught a bug in the fix itself: it measured with `time.time()` while
|
||||
`check()` measured with `datetime.now()`, and two clocks for one deadline is how
|
||||
a run waits out its cooldown and is then refused for being a fraction early.
|
||||
|
||||
Also: the confirm box before Lookup no longer aborts on a timeout (it only
|
||||
reads, and aborting threw away a completed login), then was turned off
|
||||
entirely; the Balance row is left untouched when it is already empty; and a
|
||||
message box reports the export.
|
||||
|
||||
---
|
||||
|
||||
## Day 3 — 2026-09-07
|
||||
|
||||
**Restructured into a package.** No behaviour change — this was moves and
|
||||
imports, verified rather than assumed.
|
||||
|
||||
It had grown to eleven flat modules in `src\`, 3,253 lines, with `claims.py`
|
||||
alone at 952 (nearly a third of the project in one file) and `config.py` mixing
|
||||
43 DOM selectors in with the date range and the proxy. Those change for
|
||||
different reasons — a selector when eCW ships a build, a setting when the
|
||||
process changes, a flow when the work changes — so in one file, editing a date
|
||||
range and editing a selector were edits to the same place. The charge-entry
|
||||
phases would have pushed `claims.py` past 1,500 lines.
|
||||
|
||||
Now `run.py` plus `src\ecw_bot\` with six layers, none importing a layer above
|
||||
it: `settings` → `selectors` → `core` → `auth` / `pages` → `flows`. Diagnostics
|
||||
moved to `tools\`, and `tests\` exists for the first time.
|
||||
|
||||
`run.py` is the real fix hiding inside the cosmetics. `python src\main.py`
|
||||
worked *only* because Python puts a script's own directory on `sys.path`, so
|
||||
`import config` resolved and no other way of starting the bot did — not VS
|
||||
Code's "Run File" on a different file, not Task Scheduler, not pytest. The
|
||||
entry point no longer depends on which file you happened to launch.
|
||||
|
||||
Verified: 31 files compile with `SyntaxWarning` as an error (which caught five
|
||||
Windows paths in docstrings needing raw strings), all 24 modules import, an AST
|
||||
pass found 0 undefined names across the split, every one of the old
|
||||
definitions is accounted for in a new home, `tests\test_settings.py` 3/3,
|
||||
`tests\test_guard.py` 6/6, and `tools\check_imap.py` still reaches the mailbox
|
||||
through the new import path.
|
||||
|
||||
**Not yet run live against eCW.** The next run is the first real test of the
|
||||
new layout, and of the scroll and Total Counts fixes from Day 2.
|
||||
|
||||
---
|
||||
|
||||
## Day 4 — 2026-09-15
|
||||
|
||||
**Automated the payer categorization step.** Reviewed against
|
||||
`Salud-Charge Entry_Medcor Ready.mp4` (2:45-10:00): after the Medcor Ready
|
||||
export, an operator was filtering the report by `PAYER` and hand-filling a
|
||||
`Group Name` column (`Commercial`, `Medicaid`, `Medicare`, `Dentaquest`,
|
||||
`Program`, ...) — the same manual, repetitive, rule-based step every other
|
||||
phase of this bot exists to remove.
|
||||
|
||||
`core/payer_groups.py` does the lookup: exact string match against
|
||||
`payer_groups.json`, on purpose — eCW's own PAYER values already differ by
|
||||
case and spacing for what is really the same carrier (`United Healthcare` /
|
||||
`United Health Care` / `United healthcare` are three separate values in one
|
||||
real export), so anything looser has no safe way to pick which known string
|
||||
a new one is closest to. An unrecognized payer gets a **blank** Group Name,
|
||||
not a guess, and is logged and message-boxed — this is billing data, and
|
||||
wrong is worse than blank.
|
||||
|
||||
`core/spreadsheet.py` gained a write side (`write_xlsx()`) to go with its
|
||||
read side, built the same way — stdlib only, no openpyxl — but with the full
|
||||
minimal OOXML part set (`[Content_Types].xml`, both `.rels` files,
|
||||
`workbook.xml`) the reader doesn't need but real Excel does, since this file
|
||||
is meant to be opened by a person.
|
||||
|
||||
The categorized output is a **new** file, `Output\Medcor Ready_<date>.xlsx`
|
||||
— `Input\Claim Report_<date>.xlsx` is never modified, because `claim_file.py`
|
||||
and the charge-entry phase read its columns by position.
|
||||
|
||||
**Seeded, not complete.** The mapping table ships with 23 payers covering
|
||||
roughly 95% of a real export's row volume (`Medicaid`, `DentaQuest`, and the
|
||||
clearly-commercial carriers), derived from the actual current export rather
|
||||
than guessed from the video. 17 payers (`BH COLORADO ACCESS`, the Delta
|
||||
Dental variants, `Tricare`, `Devoted`, several TPA names, ...) are
|
||||
deliberately left out — first run will flag them, and filling them in is a
|
||||
one-line JSON edit, no redeploy.
|
||||
|
||||
**Out of scope for this round:** splitting rows into the `Patient` /
|
||||
`Pending with Error` / `Program` sibling workbooks the video also shows, and
|
||||
writing anything to the network `Charge\Allocation\...` folder — both stay
|
||||
manual for now.
|
||||
|
||||
---
|
||||
|
||||
## Day 5 — 2026-09-15
|
||||
|
||||
**The bot's first write to eCW.** Reviewed the rest of the same video
|
||||
(10:00-45:32): the actual "Charge Validation" work is a per-claim clinical
|
||||
audit — open the UB form, cross-check ICD/CPT/modifiers against the
|
||||
Progress Note and a payer-specific rules sheet, fix one of a handful of
|
||||
standard defect types, save, and log the outcome. That's a much bigger
|
||||
surface than the categorization step, so it's being built one condition at
|
||||
a time rather than all at once — this is the first.
|
||||
|
||||
**Condition 1: telehealth visit → POS=11, M1=GT.** `claim_form.py` gained
|
||||
three functions: `note_mentions_telehealth()` (a plain case-insensitive
|
||||
check against the Progress Note's own text — deliberately not narrower,
|
||||
since the note's own heading for this is written inconsistently:
|
||||
`TELEHEALTH:`, `BHTelehealth Visit`, `Session Setting (PoS): Telehealth`),
|
||||
`set_cpt_pos_modifier()` (sets POS and M1 on every real CPT/HCPCS row —
|
||||
skipping eCW's blank padding rows and the trailing `*` entry row, the same
|
||||
filter `extract()` already applies — and reads every field back so a write
|
||||
that silently didn't take is a loud failure, not a quiet gap), and
|
||||
`save_claim()` (the first place in this codebase that clicks OK instead of
|
||||
Cancel — `saveAllData(true)`).
|
||||
|
||||
**Nothing saves without an explicit per-claim confirmation box.** This is
|
||||
the first write path this bot has ever had, so the new `APPLY_CORRECTIONS`
|
||||
setting gates whether the check even runs, and even then every fix stops
|
||||
for an OK/Cancel box naming exactly what changed on which rows before
|
||||
`save_claim()` is ever called. Declining leaves the claim exactly as it
|
||||
was — the field changes only ever live in the DOM until OK is clicked, so
|
||||
Cancel discards them for free.
|
||||
|
||||
The selectors needed for the write (`uBClaimPOS*`, `uBClaimMod1*` inside
|
||||
each `CPTROW_*`) needed no new live-eCW inspection — they fell out of
|
||||
`extract()`'s own existing id-prefix reader, which already had to name them
|
||||
to read the grid.
|
||||
|
||||
`tests/test_telehealth_fix.py` reuses the existing `ub_claim.html` /
|
||||
`progress_note.html` fixtures rather than adding new ones — the real note
|
||||
eCW served on 2026-09-07 already documents a Telehealth visit, so the
|
||||
positive case needs no new fixture, and detection never touches the DOM at
|
||||
all, so the negative case is a plain string. `ub_claim.html`'s Cancel and
|
||||
OK buttons needed a few lines of script to actually hide the modal on
|
||||
click, which they never did before — the original fixture only ever had to
|
||||
test reading, never closing, so this is the first thing to exercise that
|
||||
path either.
|
||||
|
||||
**Not yet run live.** `CLAIM_PROCESS_LIMIT=1` is still the default, so the
|
||||
first real run after this lands tests on exactly one real claim — the
|
||||
first live proof that the CPT/HCPCS tab is still the active tab-pane by
|
||||
the time the fix runs (nothing between `open_cpt_hcpcs_tab()` and here
|
||||
switches away from it, but this is the one assumption the fixture cannot
|
||||
prove, since Bootstrap's tab-pane visibility isn't something a static test
|
||||
page reproduces the same way live Angular does).
|
||||
Reference in new issue
Block a user