Initial import: eCW Charge Validation bot and results Portal

Snapshot of the working project as of 2026-10-23 (local 2d4d4fc), one
commit with no earlier history: the tracked files were de-identified
first (synthetic claim ids and patient data only; see Context.md
section 7). Credentials, run logs, screenshots, claim reports and client
workbooks are gitignored and not included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
ManikandanandClaude Opus 5.5 committed 2026-10-23 18:38:17 +05:30
commit ba15a574ab
205 files changed
+56772

No files matched your search

@@ -0,0 +1,182 @@
# Payer → Group Name categorization — design
Status: approved by user (Vasanth), 2026-09-15. Not yet implemented.
## Background
This is the next phase of the Salud eCW Charge Entry bot (see `README.md`),
which already automates login, 2FA, claim lookup, and exporting the
"Medcor Ready" claim queue to `Input\Claim Report_<date>.xlsx`.
The next manual step in the real process (shown in
`Salud-Charge Entry_Medcor Ready.mp4`, reviewed 2:45–10:00) is: an operator
renames a copy of that export to "Medcor ready", opens it in a dated network
folder (`...\Charge\Allocation\<Month YYYY>\<MMDDYYYY>\`, alongside sibling
workbooks `Patient`, `Pending with Error`, and `Program`), and manually
categorizes every row by filtering on the `PAYER` column and bulk-filling a
`Group Name` column with one of a small set of values (observed in the video:
`Commercial`, `Dentaquest`, `Medicaid`, `Medicare`, `Program`, plus at least
one more category for payers like `BH COLORADO ACCESS`).
This spec covers automating **only** that categorization step — not the
downstream split into the three sibling workbooks, which stays manual for now
and may be scoped separately later.
## Scope decisions (from user)
- **Scope**: categorize only. Do not build the split into
Patient/Pending with Error/Program workbooks.
- **Mapping source**: derived from the actual current export
(`Input\Claim Report_09-11-2026.xlsx`, 729 rows, 39 distinct `PAYER`
strings), not invented from video frames. Only high-confidence entries are
seeded; everything else is left for a human to fill in (see "Seed mapping
table" below) — this is intentional, not a gap to silently paper over.
- **Unknown payer handling**: flag, never guess. A `PAYER` value with no
entry in the mapping table gets a blank `Group Name`, is logged, and is
called out in the run summary. No fuzzy/substring matching — exact string
match only, so a typo'd or renamed payer can never be silently bucketed
into the wrong group.
- **Integration point**: inside the existing bot, not a standalone script.
Runs automatically as part of `flows/export_medcor_ready.py`.
- **Output location**: a new file, `Output\Medcor Ready_<date>.xlsx`. The
original `Input\Claim Report_<date>.xlsx` is never modified — `claim_file.py`
and everything downstream of it depends on that file's exact column
layout (per `README.md`'s "Reading the claim ids back" section), so this
step must not touch it.
## Architecture
```
flows/export_medcor_ready.py
... existing steps (login → claim lookup → export → claim_file.claim_ids()) ...
→ categorize_by_payer() NEW — reads Input\, writes Output\
core/payer_groups.py NEW — the lookup table + classify()
payer_groups.json NEW — editable config, exact PAYER → Group Name
core/spreadsheet.py EXTENDED — write_xlsx(), stdlib only
```
`categorize_by_payer()` runs after the existing claim-id read-back (the point
where the report is already saved and validated — see
`README.md`'s "Reading the claim ids back"). It:
1. Reads `Input\Claim Report_<date>.xlsx` with the existing `spreadsheet.read_rows()`.
2. For each row, looks up `PAYER` in `payer_groups.classify()`.
3. Writes every original column plus a new `Group Name` column to
`Output\Medcor Ready_<date>.xlsx`.
4. Logs and message-boxes a summary: rows categorized, rows left blank, and
the distinct list of unmapped `PAYER` strings — mirroring the existing
`NOTIFY_AFTER_EXPORT` / `SHOW_CLAIM_IDS` pattern in the same file.
Never fatal: an unmapped payer does not stop the run, the same philosophy
`export_claims_report()` already uses for other soft failures.
## Components
### `src/ecw_bot/core/payer_groups.py`
```python
def classify(payer: str) -> str | None: ... # exact match against the config, or None
def load_mapping() -> dict[str, str]: ... # reads payer_groups.json once
```
Exact-string lookup only. No normalization beyond what's needed to match
eCW's own values (the config keys are the literal `PAYER` strings as eCW
exports them).
### `src/ecw_bot/payer_groups.json`
A flat `{"<exact PAYER string>": "<Group Name>"}` map, hand-editable by
anyone on the team without touching code — same reasoning as `.env` for
settings and `selectors\` for DOM selectors: this data changes for a
different reason (a new payer shows up) than the code around it does.
### `core/spreadsheet.py` — new `write_xlsx()`
The project deliberately has no `openpyxl` dependency (see README: "every
added package is an approval"). `write_xlsx()` follows the same approach
already used for reading — a minimal single-sheet xlsx as a zip of XML, cells
as inline strings (no `sharedStrings.xml`, matching what eCW's own exports
already do, per the existing reader's comments) — built with `zipfile` and
string templates only.
## Data flow
```
Input\Claim Report_<date>.xlsx
→ read_rows() existing
→ for each row: classify(PAYER) new
→ write_xlsx(rows + Group Name) new
→ Output\Medcor Ready_<date>.xlsx
```
## Error handling
- Unmapped payer: blank `Group Name`, logged, summarized — not fatal.
- `Input\` file missing/unreadable: this step is skipped with a clear log
line; it never blocks the rest of the run, since the export already
succeeded by the time this step runs.
- `write_xlsx()` failure (disk full, permission): logged and raised — same
severity as any other unexpected I/O failure in this flow, since silently
losing the categorized output would defeat the point of the step.
## Testing
`tests/test_payer_groups.py`, hand-rolled in the existing style (no pytest):
- Exact-match lookups against a small fixture mapping.
- Unmapped payer → `None`, never a guess.
- Round-trip test for `write_xlsx()`: write a small fixture table, read it
back with the existing `read_rows()`, assert the data matches — the same
"never touched eCW" local-fixture philosophy as `test_claim_form.py`.
## Seed mapping table
Built from the real, current payer list (`Input\Claim Report_09-11-2026.xlsx`,
729 rows). Only high-confidence entries are seeded — this covers roughly 95%
of row volume in that file.
**Mapped (seeded):**
| Payer (exact eCW string) | Group |
|---|---|
| Medicaid | Medicaid |
| DentaQuest | Dentaquest |
| CHP+ DENTAQUEST | Dentaquest |
| AARP Medicare Advantage | Medicare |
| Aetna Medicare ADV | Medicare |
| OUTREACH PROGRAM | Program |
| Anthem Blue Cross Blue Shield | Commercial |
| Cigna HealthCare Of Colorado | Commercial |
| Cigna Healthcare of CO | Commercial |
| Cigna Salud Employee | Commercial |
| Cigna | Commercial |
| Cypress Benefit Admin / Cigna PPO | Commercial |
| United Healthcare | Commercial |
| United Health Care | Commercial |
| United healthcare | Commercial |
| Unitedhealth One | Commercial |
| Oxford Health Insurance UHC | Commercial |
| UMR | Commercial |
| Aetna | Commercial |
| Aetna Alaska Care | Commercial |
| AETNA HEALTH AND LIFE INSURANCE | Commercial |
| BIND BENEFITS INC | Commercial |
| Select Health | Commercial |
**Left unmapped — needs a human decision before first real run (or will be
flagged on first run and filled in afterward):**
BH COLORADO ACCESS (15 rows), Delta Dental of Colorado (3), Devoted (2),
Clinic Sliding Fee Scale (2), Patient (1), Gravie Admin Services (1),
Tricare (1), Rocky Mountain Health Plans (1), Denver Elevate Health Plan (1),
The Kempton Group Administrators Inc (1), Optum Care Network (1), GPA (1),
Northeast Health Partners (1), Delta Dental of Pennsylvania (1),
Delta Dental of Wisconsin (1), Delta Dental (1), Delta Dental of Arkansas (1).
## Out of scope (explicitly)
- Splitting rows into the Patient / Pending with Error / Program sibling
workbooks.
- Writing to the network `Charge\Allocation\...` folder — output stays local
to `Output\`.
- Any change to `Input\Claim Report_<date>.xlsx` or anything that reads it.
@@ -0,0 +1,269 @@
# Condition Framework — Design
**Status:** Approved by user, 2026-09-16
**Supersedes nothing** — this formalizes the pattern Condition 1 (telehealth)
already established informally in `flows/export_medcor_ready.py`.
## Why
The Charge Validation Process is, by the client's own description, a long
list of per-claim conditions to check and — when a condition's criteria are
met — a correction to apply (Condition 1: a telehealth visit needs POS=11 and
M1=GT). Conditions are being implemented one at a time, and there will be
many more of them. This is a healthcare billing bot: every write it makes
changes what gets billed to a real payer, so the process must be:
- **Reusable** — adding Condition N must not require touching the code for
Conditions 1..N-1, and must follow one obvious, fixed shape every time.
- **Exception-safe** — a bug in one condition's logic must never corrupt
another condition's result on the same claim, crash the whole overnight
batch, or leave a claim half-changed with no record of what happened.
- **Provably correct before it ever touches a real claim** — every condition
must be unit-testable against a fixture, exactly like
`tests/test_telehealth_fix.py` already does for Condition 1.
This spec explicitly rejects driving the live automation with an
LLM-agent-controlled browser (e.g. Playwright MCP) as the *engine* for this
reason: none of the three properties above hold for an engine whose
per-claim behavior is an LLM's live judgment call rather than fixed,
versioned code. Playwright MCP remains valuable as a *development and
debugging aid* (inspecting live eCW DOM, prototyping selectors) — that usage
is unaffected by this spec and requires no code change.
## Non-goals
- Not changing how claims are found, opened, or read (`claim_form.py`,
`export_medcor_ready.py`'s claims-phase — untouched).
- Not building a self-healing/adaptive-selector system. If eCW's DOM shifts
enough to break a selector, that is still a `selectors/claim_form.py`
update, done and tested like every past one (see CHANGELOG "Day 5").
- Not retrying failed conditions automatically. A condition that throws is
logged and skipped for that claim, once, this run.
## Architecture
### New package: `src/ecw_bot/conditions/`
```
src/ecw_bot/conditions/
__init__.py # CONDITIONS: list[module] — the registry
base.py # ConditionResult dataclass + run_all()
telehealth.py # Condition 1 — detect/apply/describe wrapper
```
### The Condition contract
Every condition module in the registry exposes exactly these three
functions. This is the whole interface — nothing else is required, nothing
else is read by the runner.
```python
def detect(claim_id: str, data: dict) -> bool:
"""
Pure. No `page`, no I/O, no side effects. `data` is the same dict
`claim_form.extract()` already produces (plus `progress_notes`, when
OPEN_PROGRESS_NOTES is on). Must be safe to call with a partially
populated dict — return False, never raise, on missing keys.
"""
def apply(page, claim_id: str, data: dict) -> "ConditionResult":
"""
Only called when detect() returned True. Makes the eCW field edits for
this condition and returns what changed. MUST NOT click Save/OK and
MUST NOT show any confirmation dialog itself — the runner's caller owns
both, once, after every condition has had a chance to run. Raising is
allowed and expected to be handled by the caller (see Exception
handling below) — apply() does not need its own try/except for its own
field-write logic, only for optional recovery it wants to attempt
itself.
"""
def describe(result: "ConditionResult") -> str:
"""
One human-readable block for this condition's contribution to the
combined confirmation dialog and the log line — mirrors the per-row
lines `_apply_telehealth_fix` already builds today.
"""
```
### `ConditionResult` (`conditions/base.py`)
```python
@dataclass
class ConditionResult:
condition_name: str
changed: bool
changes: list # same shape as claim_form.set_cpt_pos_modifier()'s
# return value: [{"row", "code", "before", "after"}, ...]
review: str = "" # added 2026-10-05 (unit D) — see below
@classmethod
def none(cls, condition_name: str) -> "ConditionResult":
return cls(condition_name=condition_name, changed=False, changes=[])
@classmethod
def declined(cls, condition_name: str, reason: str) -> "ConditionResult":
return cls(condition_name=condition_name, changed=False, changes=[],
review=reason)
```
**Amendment 2026-10-05 (review items D2/P2, unit D).** `none()` means
"nothing to do — the claim already follows this rule". A condition that
matched the claim but could not act on it (the CPT/HCPCS tab would not
open, an admin code would not add with the grid untouched) now returns
`declined(name, reason)` instead: nothing changed, but the claim is not
clean, and `reason` reaches the Charge Summary as a "Review" row. A
changed result may also carry `review` (a failed add that still moved the
grid). Two optional module hooks join `detect/apply/describe`:
- `needs_review(claim_id, data) -> str | None` — asked only when
`detect()` returned False, for a claim that matched the trigger but
cannot be acted on (vaccine_admin_codes on an unmapped payer, no payer
name, no usable age — never Medicare, which has its own rule).
- `summary_comment(result) -> str` — the Charge Summary comment for a
changed result, built from what actually happened, in place of the
summary's fixed `COMMENTS[name]`.
### The runner (`conditions/base.py`)
```python
def run_all(page, claim_id: str, data: dict) -> list[ConditionResult]:
"""
Runs every registered condition against one claim, in registry order.
Returns only the results where something actually changed — a claim
with nothing to fix returns an empty list, and the caller shows no
dialog and saves nothing.
"""
results = []
for cond in CONDITIONS:
name = cond.__name__.rsplit(".", 1)[-1]
try:
if not cond.detect(claim_id, data):
continue
result = cond.apply(page, claim_id, data)
except Exception as exc:
print(f"[COND {name} {claim_id}] Failed: {exc}")
shot(page, f"condition_{name}_failed_{claim_id}")
continue # this condition's failure must not block the next one
if result.changed:
results.append(result)
return results
```
`shot()` is the existing screenshot helper already used by `open_claim()`'s
diagnostic wrapping (`claim_form.py`) — reused here, not reinvented.
**Amendment 2026-10-05 (review items D2/P2, unit D).** The real signature
is now `run_all(page, claim_id, data, conditions=None, before_apply=None,
report=None)`. Its return value is unchanged — only the changed results —
but that list is not everything that happened, so a caller that passes a
dict as `report` gets the whole account:
| key | contents |
|---|---|
| `fired` | names whose `detect()` returned True |
| `changed` | names with a changed result (= the return value) |
| `failed` | `{"name", "stage", "error"}` per raise; `stage` is `"detect"`, `"apply"` or `"needs_review"`, `error` the exception's first line |
| `review` | `{"name", "reason"}` from every result (changed or not) with a non-empty `review`, and from `needs_review()` of every condition whose `detect()` returned False |
| `comments` | `{name: summary_comment(result)}` for changed results whose module has the hook (a hook that raises is logged and left out — the summary falls back to `COMMENTS[name]`) |
Exceptions are still isolated exactly as in the sketch above: logged,
screenshotted, and the next condition still runs. `detect()` and `apply()`
now each sit in their own try (the sketch's single try is how it began), so
the report can say which stage raised.
### Caller contract (replaces `_apply_telehealth_fix`'s direct call)
`export_medcor_ready.py`'s `_read_claim_form()` changes from calling
`_apply_telehealth_fix(page, claim_id, data)` directly to:
```python
if cfg.APPLY_CORRECTIONS:
results = conditions.run_all(page, claim_id, data)
saved_write = _confirm_and_save(page, claim_id, results)
```
(As built since 2026-10-05: `run_all(..., report=report)` and
`_confirm_and_save(page, claim_id, results, report)`; the report is stored
on the claim's data as `data["_conditions"]` = `{"fired", "saved",
"failed", "review", "comments"}`, which the Charge Summary reads.)
`_confirm_and_save` (new, small, lives in `export_medcor_ready.py` — it is
orchestration, not a condition):
1. If `results` is empty → return `False` immediately, no dialog, no save.
(A claim where nothing fired is not different from today's "no
telehealth mention" case.)
2. Otherwise build one combined message: every fired condition's
`describe()` output, concatenated, in the same
`msgbox.ok_cancel(...)` shape Condition 1 uses today.
3. One Yes/No. Yes → `claim_form.save_claim(page)`. No → log "declined",
return `False` (the caller's existing `finally` still closes with
Cancel, unchanged).
4. Return whether the save actually happened, same as
`_apply_telehealth_fix` returns today — this is what `_read_claim_form`
uses to decide whether its own `finally` should call `close_claim()`.
**Amendment 2026-10-05 (review item P2, unit D):** between 1 and 2 —
and after the SAVE_CORRECTIONS gate — if the report has any `failed` or
`review` entry, the box is never shown: the log says `[COND <id>] Not
offering Save: <reasons> — closing with Cancel` and it returns `False`.
**Why one dialog, not one per condition:** if two conditions both want
changes on the same claim, saving after only reading half the operator's
decision is exactly the kind of half-applied state this spec exists to
prevent. The operator sees the whole picture — every condition that fired,
every field that would change — before any single Save/Cancel decision.
## Exception handling — the actual rules
This is the section the user asked for explicitly, spelled out rather than
left implicit:
| Failure | What happens |
|---|---|
| One condition's `detect()` raises | Caught by `run_all`'s try/except (its own try since 2026-10-05, so the report can say `stage: "detect"`). Logged, screenshotted, skipped, and recorded in the report's `failed` — the claim reads "Review", not clean. Other conditions still run. |
| One condition's `apply()` raises partway through a multi-row edit | Caught by `run_all`. Logged, screenshotted. That condition contributes nothing to `results`. **Corrected 2026-10-05 (review item P2):** this row used to say its partial edits "are never saved, because Save only happens ... for what's in `results`". That was wrong: Save (`saveAllData`) writes the whole form, partial edits included, and the box would have listed only the other conditions. So the raise is recorded in `run_all`'s report (`failed`, stage `"apply"`), and `_confirm_and_save` never offers Save for a claim whose report has a `failed` or `review` entry — the claim is closed with Cancel and reported "Review" in the Charge Summary. |
| `_confirm_and_save`'s dialog itself fails to show (msgbox error) | Not caught here — surfaces to `_read_claim_form`'s existing outer try/except, which logs "Could not read the form" and closes via `finally`. No new behavior needed; the existing outer boundary already covers this. |
| `save_claim()` returns `False` (save click didn't work) | Same as today: logged, `saved_write` stays `False`, `finally` calls `close_claim()` — the claim is left as read, not half-saved. |
| A condition throws on every claim in the batch (systemic bug in that one condition) | By design, this spec does not add batch-level circuit-breaking — each claim's screenshot + log line is enough signal for the operator to notice a pattern across a run's log file. Adding an auto-disable-after-N-failures behavior is explicitly deferred (YAGNI): today's batches run at `CLAIM_PROCESS_LIMIT` scale, small enough that log review is a real diagnostic step, not a chore. |
## Testing
- `tests/test_telehealth_fix.py` — unchanged. It tests
`claim_form.note_mentions_telehealth` and `claim_form.set_cpt_pos_modifier`
directly; those functions do not move.
- `tests/test_conditions_telehealth.py` (new) — tests
`conditions/telehealth.py`'s thin wrapper: `detect()` delegates to
`note_mentions_telehealth` correctly on a plain dict (no page needed);
`apply()` calls `set_cpt_pos_modifier` and returns a populated
`ConditionResult`; `describe()` produces the expected text.
- `tests/test_conditions_runner.py` (new) — tests `run_all()` and
`ConditionResult` in isolation with fake condition modules (no real
telehealth logic involved):
- a condition whose `detect()` returns `False` is never `apply()`-ed
- a condition whose `apply()` raises is skipped, and a *second*,
healthy condition registered after it in `CONDITIONS` still runs and
its result is still returned
- `results` is empty when every condition's `detect()` returns `False`
- a condition's raised exception does not propagate out of `run_all`
## Migration checklist (informs the implementation plan)
1. Create `conditions/base.py` (`ConditionResult`, `run_all`).
2. Create `conditions/telehealth.py` — move `note_mentions_telehealth`-based
detection logic into `detect()`, wrap `set_cpt_pos_modifier` in
`apply()`, write `describe()` from the existing message-building lines
in `_apply_telehealth_fix`.
3. Create `conditions/__init__.py` with `CONDITIONS = [telehealth]`.
4. Add `_confirm_and_save` to `export_medcor_ready.py`; delete
`_apply_telehealth_fix` (its logic is now split across
`conditions/telehealth.py` and `_confirm_and_save`).
5. Update `_read_claim_form` to call `conditions.run_all` +
`_confirm_and_save` instead of `_apply_telehealth_fix` directly.
6. New tests (above) all passing; full existing suite (88+ tests) still
passing unchanged.
7. Condition 2 (whatever it turns out to be) is the first real proof this
spec worked: it should require touching only a new
`conditions/<name>.py` + its test file + one line in `CONDITIONS`.
@@ -0,0 +1,137 @@
# Dental claims run every applicable condition — design
Date: 2026-10-07. Status: approved in chat (operator, 2026-10-07);
implemented 2026-10-12 (operator: "this is our number one condition").
## 2026-10-12 update (as built)
- The rule now also covers #12 (H codes: BH payer → modifier removed;
any other payer → line unchecked + fee 0) and #14 (Medicare: G8510 /
G8431 unchecked + fee 0, #6 not run). Both run on Dental claims like
every other condition; #12's BH path has no modifier column to read on
the Dental grid, so it never fires there.
- A fix the Dental screen cannot hold → **Review** (operator): a condition
module declares `DENTAL_UNSUPPORTED = "<reason>"` and `run_all` records
that reason instead of applying it on a `screen == "dental"` read —
telehealth, medicare_telehealth, vaccine_admin_codes.
- Dental claims are **never saved** in this change (operator), whatever
`SAVE_CORRECTIONS` says; the review-before-close box still shows.
- A Dental claim is read and run through the conditions even with
CHECK_HIGH_RISK and CHECK_SLIDING_FEE both off (it used to come back
blank).
- The Charge Summary applies the same claim-level checks (note read, CPT /
ICD rows read, conditions run) to a Dental read as to UB/HCFA.
- Row selectors gained a third (Dental) alternative, scoped to the Dental
window on screen (`:visible`); `extract()` returns `screen`
("ub" / "hcfa" / "dental").
- Not live-verified yet: the first live run should be SAVE-off on a few
Dental claims (one with DCONS or Dscprit).
## Rule (operator, 2026-10-07)
Every claim is checked against **every condition whose Payer column is
"All" / "All payers"**, **plus** every condition specific to that claim's
payer (Medicare, Medicaid, Dental payers, ...). This holds for every payer
and every claim screen — UB, HCFA and Dental alike.
UB and HCFA claims already work this way: each claim runs the whole
registry (`conditions/__init__.py` `CONDITIONS`) and each condition's own
`detect()` decides from the claim's codes and payer. **The gap is Dental:** a
Dental claim today runs only #15 high risk and #8 sliding fee and never
reaches `run_all`. So, for example, `dummy_codes` (DCONS/Dscprit, which are
Dental codes) can never fire on any claim.
## What a Dental claim will run
| Doc # | Condition | Payer (document) | Module |
|---|---|---|---|
| 5 | J2930/J2001 uncheck + fee 0 | ALL | `j2930_j2001_update` |
| 6 | Depression DX pointer | All | `depression_dx_pointer` |
| 7 | Zero-fee uncheck / priced line check | ALL | `zero_fee_uncheck`, `priced_line_check` |
| 21 | Telehealth POS 2 + modifier | All | `telehealth` |
| 22 | Medicare telehealth G2025 | Medicare | `medicare_telehealth` |
| 18 | Vaccine admin codes | All except Medicare | `vaccine_admin_codes` |
| 8 | DCONS/Dscprit uncheck + fee 0 | All Dental Payer | `dummy_codes` |
| 8 | Sliding fee date check | All Dental Payer | `pages/sliding_fee` (unchanged) |
| 15 | High risk in Billing Notes | Dental Payers | `pages/dental_notes` (unchanged) |
No condition gains a form-type check: each still decides from codes and
payer only. A condition whose codes are not on the claim does nothing.
## Dental screen facts (from Dom.txt, claim screen `.modal[id^="dentalClaim"]`)
- CPT/CDT grid `table[id^="claimCPTCodesTable"]`; real rows are
`tr[ng-repeat^="CPT in CPTCodes"]` (no `CPTROW_` id).
- Per row: checkbox `[ng-click*="updateBillToIns("]` (same as UB/HCFA), code
`[data-fieldname="web_cptcode"]`, pointers `web_icd1`..`web_icd4`, units
`web_units`, fee `web_billedFee` (blur runs `editCPTRow('BILLED_FEE')`),
DOS `DentalClaimCPTSDOS`.
- **No per-row POS and no modifier columns.** POS is claim-level
(`ng-model="ClaimData.pos"`).
- Prog. Notes button `button[ng-click="viewProgNote()"]`
(`dentalClaimProgressNoteBtn…`) — the existing `SEL_PROGRESS_NOTES` matches.
- Insurance grid: `#DentalClaimTbl2` (Bill To, Ins Id, Name, ...).
- Billing Notes: `#dentalClaimNotes` (already read by high risk).
None of the row fields has been exercised live yet; Dom.txt is one capture.
## Design
1. **Selectors** (`selectors/claim_form.py`): add the Dental alternative to
each row selector the helpers already use — `SEL_CPT_ROWS`,
`SEL_CPT_CODE`, `SEL_CPT_ICD1..4`, `SEL_CPT_BILLED_FEE`, `SEL_CPT_UNITS`.
The Dental row selector is scoped under `.modal[id^="dentalClaim"]` so it
cannot match a grid on another screen. Within one row only one alternative
ever matches, the same contract the UB/HCFA pairs rely on.
2. **Reader** (`pages/claim_form.py` `_EXTRACT_JS`): read Dental rows into the
same `cpt` row shape (`code`, `icd1..4`, `units`, `billed_fee`, `billed`,
`sdos`; `pos`/`mod1..4` empty), insurance from `#DentalClaimTbl2`, the
Dental claim number, ICD list, and `notes` from `#dentalClaimNotes`.
One `extract()` for all three screens.
3. **Flow** (`flows/export_medcor_ready.py`): a Dental claim runs high risk +
sliding fee first (unchanged, read-only), then the same path as UB/HCFA:
`extract()`, Prog. Notes (when `OPEN_PROGRESS_NOTES`), `run_all`, per-claim
box, Charge Summary. The Dental results stay on the claim's data so the box
and the summary show both.
4. **A fix the Dental screen cannot make** — e.g. telehealth needs a row
modifier and the Dental grid has none, or POS is claim-level: the condition
returns `declined(...)` with the reason ("Dental screen has no modifier
column"), so the claim is **Review** in the Charge Summary. Never a silent
skip, never a guess.
5. **Charge Summary**: one row per claim combining condition comments with the
Dental findings (HIGH RISK, sliding fee). Category: Review beats Fail beats
Pass, as today.
6. **Saving**: Dental claims are **never saved** in this change, whatever
`SAVE_CORRECTIONS` says. Their corrections are shown and listed as
"Corrections pending (not saved)". Dental save is enabled only after a live
verification-mode run confirms the row writes.
## Error handling
- Dental form not loaded (`wait_claim_loaded`): conditions skipped, claim is
Review — same as HCFA today.
- A Dental row field missing (fee or checkbox not found): the existing helpers
already treat a missing field as unknown, not as zero/unchecked.
- High risk / sliding fee failures keep their current Review handling.
## Testing
- Grow `tests/fixtures/dental_claim.html` from Dom.txt's real Dental grid
(rows, checkbox, fee, pointers, insurance grid, Prog. Notes button) with no
patient data.
- `extract()` on the Dental fixture returns the expected `cpt` and `insurance`.
- Per condition, a Dental case: dummy codes (DCONS/Dscprit) uncheck + zero;
J2001 uncheck + zero; zero-fee uncheck; priced line check; depression
pointer retag on `web_icd*`; telehealth on Dental → declined/Review.
- Flow: a Dental claim gets high risk + sliding fee **and** condition results
in one summary row; never offered Save.
- Full suite stays green.
- Live: one SAVE-off run on Dental test claims (with a DCONS or Dscprit line)
before Dental save is considered.
## Out of scope
- New document conditions not yet built (#3, #4, #9-#14, #16, #19, #20,
#23-#55).
- Saving Dental claims.
- "Special programs" half of #8.
@@ -0,0 +1,164 @@
# Charge Summary in SQL Server + results web page — design
Date: 2026-10-08. Status: design approved in chat (operator, 2026-10-08),
spec under review.
## Goal
Every run's Charge Summary rows are also stored in SQL Server, and a simple
local web page (FastAPI) shows them. This is the basic version; later it is
folded into the full portal.
## Decisions (operator, 2026-10-08)
- Server `MECARE331\MECARE331` (SQL Server 2022, on this machine), database
**`Charge_Validation`** (created by the operator).
- The table follows the team's **Charge Summary File Format**
(`\\mecare2\Production\Salud-eCW RPA\Charge Summary File Format.xlsx`) —
the same 24 columns as `core/charge_summary.HEADERS`, plus run columns and
a details JSON. PATIENT, Account Number and Guarantor Name are stored, as
in the Excel file.
- FastAPI for the backend. All needed packages are already installed
(fastapi 0.141, uvicorn, jinja2, httpx, pyodbc, ODBC Driver 18); nothing new
is added to `requirements.txt` for the bot itself.
## Table
`tools/sql/create_tables.sql`, run by the operator in SSMS (the bot never
creates or alters tables):
```sql
USE Charge_Validation;
CREATE TABLE dbo.ChargeSummary (
id INT IDENTITY(1,1) PRIMARY KEY,
run_at DATETIME2 NOT NULL,
run_mode NVARCHAR(20) NOT NULL, -- VERIFY / LIVE-WRITE / TEST
s_no INT NULL,
claim_no NVARCHAR(20) NOT NULL,
service_date DATE NULL,
pvdr NVARCHAR(50) NULL,
patient NVARCHAR(200) NULL,
payer NVARCHAR(200) NULL,
group_name NVARCHAR(100) NULL,
status NVARCHAR(100) NULL,
charges DECIMAL(12,2) NULL,
pmts_adjs DECIMAL(12,2) NULL,
pmts_refunds DECIMAL(12,2) NULL,
adjustment DECIMAL(12,2) NULL,
withheld DECIMAL(12,2) NULL,
balance DECIMAL(12,2) NULL,
visit_type NVARCHAR(100) NULL,
account_number NVARCHAR(50) NULL,
provider_name NVARCHAR(200) NULL,
guarantor_name NVARCHAR(200) NULL,
received_date DATE NULL,
assigned_to NVARCHAR(100) NULL,
completed_on DATE NULL,
comments NVARCHAR(MAX) NULL,
current_status NVARCHAR(200) NULL,
category NVARCHAR(20) NULL,
details_json NVARCHAR(MAX) NULL
);
CREATE INDEX IX_ChargeSummary_run_at ON dbo.ChargeSummary (run_at);
CREATE INDEX IX_ChargeSummary_claim_no ON dbo.ChargeSummary (claim_no);
```
Column map (Excel → DB) lives in one place, `results_db.COLUMNS`, built from
`charge_summary.HEADERS` so the two cannot drift: S.No→s_no, CLAIMS#→claim_no,
SERVICE DATE→service_date, PVDR→pvdr, PATIENT→patient, PAYER→payer, Group
Name→group_name, STATUS→status, CHARGES→charges, PMTS/ADJS→pmts_adjs,
Pmts/Refunds→pmts_refunds, Adjustment→adjustment, WithHeld→withheld,
Balance→balance, Visit Type→visit_type, Account Number→account_number,
Provider Name→provider_name, Guarantor Name→guarantor_name, Recevied
date→received_date, "Assigned To "→assigned_to, Completed On→completed_on,
Comments→comments, Current Status→current_status, Category→category.
Value conversion: money columns strip `$`, `,` and spaces → Decimal; dates
accept `mm/dd/yyyy` (and an Excel date/datetime) → date; blank or
unparseable → NULL (the raw text is not lost: the Excel file still has it).
## Bot side — `src/ecw_bot/core/results_db.py`
- Settings (`settings.py`, from `.env`):
- `RESULTS_DB` (bool, default **off**): write rows to the DB.
- `RESULTS_DB_CONN` (string): ODBC connection string. Default:
`DRIVER={ODBC Driver 18 for SQL Server};SERVER=MECARE331\MECARE331;DATABASE=Charge_Validation;Trusted_Connection=yes;TrustServerCertificate=yes;`
(Windows login, no password). A SQL login's password, if ever used, is put
in `.env` by the operator only.
- `to_record(row: dict, *, run_at, run_mode, details) -> dict`: one Charge
Summary row (header→value) to DB column→value, converted.
- `insert_rows(conn, records) -> int`: one parameterized `INSERT` per row
(`executemany`), one commit; returns rows written. Never builds SQL from
values.
- `save_summary(sheet, details, run_at, run_mode, connect=pyodbc.connect) ->
int | None`: `sheet` is the header+rows list `charge_summary.build_rows`
already returns; `details` maps claim no → details dict. Opens the
connection, inserts, closes. **Any exception is caught**: one
`[RESULTS-DB] Could not save ...` log line, returns None. The run, the Excel
summary and the claims are never affected.
- `details_json` per claim: `{"fired", "changed", "review", "failed",
"comments", "saved", "changes": {condition: [change, ...]},
"dental": {"high_risk", "sliding_fee"}}` — taken from the claim's
`data["_conditions"]` (the flow adds `changes` there) and Dental results.
## Flow changes
- `charge_summary.write()` is split: `charge_summary.build(report_path,
outcomes, assigned_to) -> sheet` (read + `build_rows`), and `write()` saves
that sheet as today. The flow builds once and hands the same sheet to Excel
and to the DB, so both always hold the same rows.
- `_read_claim_form` adds `"changes": {r.condition_name: r.changes}` to
`data["_conditions"]`; the claim loop keeps `details[claim_id]` next to
`outcomes[claim_id]`.
- `_write_charge_summary` calls `results_db.save_summary(...)` after the Excel
write when `RESULTS_DB` is on, and logs `[RESULTS-DB] N row(s) saved`.
- `run_mode`: `TEST` when `TEST_CLAIM_IDS_FILE` is set, else `LIVE-WRITE`
when `SAVE_CORRECTIONS` is on, else `VERIFY` — the same words as the
`[MODE]` banner.
## Web page — `web/`
- `web/app.py` (FastAPI) and `web/templates/index.html` (Jinja2), plain
HTML/CSS, no external scripts or CDNs.
- Read-only: only `SELECT` statements, parameterized.
- `GET /` — page:
- run picker (distinct `run_at`, newest first; default latest run),
- Category filter (All / Pass / Fail / Review) and Claim# search,
- counts per Category for the selected run (Pass / Fail / Review tiles),
- the table in Charge Summary column order, Category coloured,
- each row expands to show `details_json` (conditions fired, before/after
changes, review reasons).
- `GET /api/runs` — runs with counts. `GET /api/results?run_at=&category=&claim_no=`
— rows as JSON (for the later portal).
- Start: `python -m uvicorn web.app:app --host 127.0.0.1 --port 8000`.
Bound to **127.0.0.1 only**: the page shows PHI and is not reachable from
the network. A `run_web.bat` helper does the same.
- The DB connection string is the same `RESULTS_DB_CONN`. If the DB cannot be
reached the page shows a plain error message, not a stack trace.
## Error handling
- DB unreachable / permission denied / table missing during a run → one log
line, run continues, Excel written.
- A value that does not convert → NULL for that column, row still saved.
- Web page: DB errors → HTTP 503 page "Results database not reachable".
## Testing (no real DB)
- `tests/test_results_db.py`: header→column map covers all 24 HEADERS;
money/date conversion (incl. `$1,234.50`, blank, junk); `to_record` output;
`insert_rows` sends one parameterized statement with values in column
order (fake connection records calls); `save_summary` swallows a connect
error and returns None; `details_json` shape.
- `tests/test_charge_summary.py`: `build()` + `write()` split keeps today's
output.
- `tests/test_web_app.py`: FastAPI `TestClient` with a fake repository —
page renders rows and Category counts, filters work, `/api/results` JSON
shape, DB error → 503.
- Full suite stays green. Live: operator runs `create_tables.sql`, sets
`RESULTS_DB=1`, one test run, then opens the page.
## Out of scope (later portal)
Login/roles, editing rows, charts beyond the Category tiles, scheduling,
AI features, hosting on the network.
@@ -0,0 +1,176 @@
# Start the bot from the results web page — design
Date: 2026-10-14. Approved section by section by the operator in chat
(approach A, sections 1–3).
## Goal
The operator starts a Charge Validation run from the local results page
(`run_web.bat` → http://127.0.0.1:8000) instead of the terminal. While it
runs, the page shows live status; it can stop it; when it ends, the page
shows that run's results.
## Decisions (operator, 2026-10-14)
| Question | Answer |
|---|---|
| Controls | Start + live status + Stop (no settings editing on the page) |
| Save mode | **VERIFY only.** A page-started run is always `SAVE_CORRECTIONS=0`, whatever `.env` says. LIVE-WRITE stays terminal-only. |
| Message boxes | **Off** for the run's information boxes (per-claim, review-before-close, "Run the Lookup?", summaries). Boxes that need a person to act in the browser — CAPTCHA, 2FA manual step, security image — **stay on**. |
| Approach | A: the web server starts `run.py` as a separate child process (not a thread, not Task Scheduler). |
Out of scope (YAGNI): editing settings from the page, scheduling, more than
one run at a time, LIVE-WRITE from the page.
## Components
### New: `web/bot_runner.py` — process control only
- `start() -> dict` — starts `python run.py` (the project's own `run.py`,
`sys.executable`, cwd = project root) as a child process with no console
window of its own (`CREATE_NO_WINDOW`); the Chrome window still shows.
The child's stdout/stderr go to `Logs\web_run_<ts>.out`, which catches a
crash before the bot opens its own log; the bot's `Logs\run_<ts>.log`
(named in the lock file) is the status source.
Refuses (raises `AlreadyRunning`) when the lock file names a live pid.
- `stop() -> dict` — `taskkill /PID <pid> /T /F`: the bot and its Chrome
close together. Clears the lock file. Status becomes "Stopped by
operator".
- `status() -> dict` — `running` (bool), `pid`, `started_at`, `log_path`,
`exit_code` (when this web process started it and it has ended),
`stopped_by_operator`.
- The command is fixed. Nothing from the request reaches the command line
or the environment.
**Child environment** — set on the child process, so it wins over `.env`
(`settings.py` loads `.env` with `os.environ.setdefault`):
```
SAVE_CORRECTIONS=0
ECW_KEEP_OPEN=0 # no "Press Enter to close" at the end
QUIET_BOXES=1 # run-flow information boxes go to the log
CONFIRM_BEFORE_LOOKUP=0
```
Everything else follows `.env` (claim file, `CLAIM_PROCESS_LIMIT`,
`RESULTS_DB`, proxy, ...).
### Lock file: `Logs\bot_run.json`
`{"pid", "started_at", "log_path", "source": "web" | "terminal"}`.
- Written by the **bot itself** at the start of `main()` (after
`runlog.start()`, so `log_path` is known) and removed when `main()`
returns — so a terminal run is seen by the page too.
- Before writing it, `main()` refuses to start (exit 2, before login) when
the file names a pid that is alive: two runs would share one eCW login
guard and one Charge Summary.
- A file naming a dead pid is stale: removed and ignored (a killed run
cannot remove its own file).
- `bot_runner.start()` checks it too, so the page answers 409 at once.
### New: `web/run_status.py` — read the run log
Pure functions over log text (no I/O except reading the file tail):
- **Phase** from the last matching line:
`[LOGIN]` / `[CAPTCHA]` → "Logging in"; `[OTP]` / `[2FA]` → "Waiting for
the 2FA approval mail"; `[OK] Logged in` → "Logged in"; `[CLAIMS]` →
"Checking claims"; process ended → "Finished" or a failure text by exit
code.
- **Progress** from `[CLAIMS] --- claim n/N: <id> ---` → `n`, `N`, current
claim id.
- **Counts** from the new `[RESULT]` lines (below) → Pass / Fail / Review.
- **Tail**: the last 30 lines.
- **Exit codes** (from `flows/export_medcor_ready.main`): 0 finished; 2
config / mode gate; 3 login guard refused (its `[GUARD]` line shown); 5
claims phase failed; 6 2FA timed out; other → "ended with code N".
### Changed: `flows/export_medcor_ready.py`
- After each claim's outcome is known in `_process_claims`, one
machine-readable line (terminal runs print it too):
```
[RESULT] claim 3/7 9126000: Review
```
The word is the Category the Charge Summary gets for that claim; a claim
that gets no Charge Summary row (not found in the grid) prints
`Not found`.
- `main()` writes / removes the lock file as above.
### Changed: `core/msgbox.py`
- New setting `QUIET_BOXES` (default off; `settings.py`).
- `info_box(title, msg, needs_person=False)`: with `QUIET_BOXES=1` and
`needs_person=False`, no box — one log line instead:
`[BOX] <title>: <first line of msg>`.
- `ok_cancel` / `confirm` are unchanged (every remaining caller is a
person-needed step: CAPTCHA, 2FA manual, LIVE-WRITE confirm — the last
is never reached in VERIFY). The security-image `info_box` in
`auth/login.py` passes `needs_person=True`.
### Changed: `web/app.py` + `web/templates/index.html`
Endpoints:
| Method | Path | Answer |
|---|---|---|
| POST | `/api/bot/start` | 200 `{status}`; 409 already running; 403 bad Origin/token |
| POST | `/api/bot/stop` | 200 `{status}`; 409 nothing running; 403 bad Origin/token |
| GET | `/api/bot/status` | `{running, phase, progress, counts, tail, started_at, exit_code, message}` |
Page: a "Bot run" card above the results.
- Idle: **[Start run]** and "VERIFY mode — nothing is saved to eCW".
- Running (polls `/api/bot/status` every 2 s): phase, progress bar
"3 / 7", current claim, Pass/Fail/Review chips, collapsible "Live log"
(last 30 lines), **[Stop]** (with a confirm dialog).
- Ended: "Finished in 4m 12s — exit 0" (or the failure text); the results
table reloads on the newest run.
## Security
The page was read-only; Start/Stop are its first actions. The existing
TrustedHost check stops DNS rebinding but not a cross-site POST from
another tab, which could burn eCW login attempts. So every POST needs both:
1. `Origin` (or, when absent, `Referer`) is `http://127.0.0.1:8000` or
`http://localhost:8000` — else 403.
2. Header `X-Run-Token` equals a token made at server start
(`secrets.token_urlsafe(32)`) and embedded in the page — else 403.
Compared with `secrets.compare_digest`.
No password on the page; `.env` credentials are read only by the bot.
Responses keep `no-store`. The page stays bound to 127.0.0.1. The live log
tail can carry PHI, as the results table already does — local only.
## Error handling
- Already running (page or terminal) → 409 "A run is already going
(started HH:MM)".
- Guard refused (exit 3) → the `[GUARD]` line shown ("next attempt
allowed at ...").
- Stop mid-claim: the claim form may stay open in eCW. VERIFY saves
nothing; the next run's `clear_screen` closes it.
- Web server restarted or page closed: the bot keeps running; on reload
the page finds it through the lock file. `exit_code` is unknown then
(the run is shown "Ended" when its pid is gone).
- Log file not there yet → phase "Starting".
## Testing (standalone "N/M passed" scripts)
- `tests/test_bot_runner.py` — a fake child script in place of `run.py`
(injectable command): start → running → stop → stopped; second start →
`AlreadyRunning`; stale lock (dead pid) cleared; the child sees the forced
env (`SAVE_CORRECTIONS=0`, `ECW_KEEP_OPEN=0`, `QUIET_BOXES=1`,
`CONFIRM_BEFORE_LOOKUP=0`) even when the parent env says otherwise.
- `tests/test_run_status.py` — sample log text → phase, progress, counts,
tail; each exit code's text.
- `tests/test_web_app.py` (added cases) — POST without/with wrong Origin or
token → 403; right Origin + token → 200 (runner stubbed); status JSON
shape; the token is in the page.
- `tests/test_msgbox.py` (added cases) — `QUIET_BOXES=1`: info box logged,
not shown; `needs_person=True` still shown.
- Flow test — `[RESULT]` line per claim; `main()` lock file written,
refused on a live one, removed at the end.
@@ -0,0 +1,96 @@
# Results portal redesign — design
Date: 2026-10-19. Operator requests (voice, Tanglish), choices made on the
visual companion mockups (`.superpowers/brainstorm/828-1792395410/`).
## Problem
The results page (`web/`, FastAPI + Jinja, 127.0.0.1:8000) opens on the
newest run only. A second run (20 claims) hides the first (10 claims) behind
a "Run" dropdown nobody noticed, so earlier runs look lost. The front page
is three quarters claims table (24 columns, sideways slider), which the
operator does not want on the home page. The look is "very basic".
## What the operator chose
| Question | Choice |
|---|---|
| Home page | **Dashboard**: Bot run card (Start / Stop / live progress), today's tiles, one big **Claim status** button |
| Claim status page | **Date cards**, newest first |
| A date with several runs | **All claims of the day together** (every run's rows in one list) |
| One day's page | **Short table + side panel** |
| Colours | **Navy & blue** (navy header, white cards on light grey, blue buttons) |
| Live run's claim list | In the **Bot run card** ("Live claims" side panel) |
| Search one claim over all dates | **Yes** — "Find a claim" on the Dashboard |
| Approach | **A** — server pages (FastAPI + Jinja), one page per screen |
## Pages
All pages share `base.html`: navy header with the title, a mode pill
(VERIFY / LIVE-WRITE from `SAVE_CORRECTIONS`) and a breadcrumb
(Dashboard › Claim status › 16 Oct 2026). Light grey background, white
cards, blue buttons; Pass green, Fail red, Review amber everywhere.
1. **Dashboard `/`**
- Bot run card — the existing Start / Stop / progress / live log, plus a
**Live claims** link opening a side panel with every claim of the
running (or last) run: Waiting / Checking / result. A claim with a
saved result opens `/day/<date of its run>?claim=<no>`.
- Today tiles: claims checked today, Pass, Fail, Review (0 when no run
today).
- **Find a claim** box → `/claim?claim_no=…`.
- Big **Claim status →** button → `/days`.
- No claims table.
2. **Claim status `/days`** — one card per day (last 90 days with a run),
newest first: weekday + date, claims checked, number of runs,
Pass / Fail / Review and a thin colour bar. A card opens `/day/<date>`.
3. **One day `/day/YYYY-MM-DD`**
- Tiles All / Pass / Fail / Review; a tile is a filter link
(`?category=Fail`). Claim # search box (`?claim_no=`).
- Short table, 10 rows a page (client-side pager): #, Claim #, Time
(the run's time), Payer, Result chip, Comments (one line, cut with
an ellipsis). No sideways scroll.
- Row click → side panel with every Charge Summary field, the run's
time and mode, and the conditions/changes JSON.
- `?claim=<no>` opens that claim's panel on load (on its page).
4. **Claim history `/claim?claim_no=…`** — every saved check of claims
matching the number (newest first, up to 200): date, time, mode,
result, comments; each row opens `/day/<date>?claim=<no>`.
`/?run_at=<time>` (old links) redirects (303) to `/day/<date of that run>`.
`/api/runs`, `/api/results`, `/api/bot/*` stay as they are.
## Data
No schema change. `web/repo.py` gains three parameterized SELECTs:
- `days(limit=90)` — `GROUP BY CAST(run_at AS DATE)`: day, claims,
runs (`COUNT(DISTINCT run_at)`), pass / fail / review counts, newest
first.
- `day_results(day, category, claim_no)` — rows with
`run_at >= day AND run_at < day + 1`, ordered by run_at, S.No, id.
- `claim_history(claim_no)` — `TOP 200 … WHERE claim_no LIKE ?` ordered
by run_at DESC.
Rules: a run belongs to the day it **started** (every row of a run shares
its `run_at`), so a run never splits over two cards. The same claim
checked twice in a day shows twice, each with its time; a day's count is
checks. "Not found" claims have no saved row (as today) and appear only in
the live panel. A legacy driver returning the day as text is coerced to a
date, like `run_at` already is.
## Unchanged
127.0.0.1 only; Host allowlist; `no-store` and the other headers; DB
errors reported by kind; Start / Stop need the page's Origin and token;
page runs are VERIFY only. All new pages are read-only. Page text from the
DB goes out escaped (Jinja autoescape; the panel's data as `tojson` in an
attribute, rendered with `textContent`).
## Testing
`tests/test_web_app.py` (fake repo, fake DB connection): each page renders
with its data and links; filters and the bad-input 400s; the redirect;
the three new SQL statements are parameterized and shaped as above; the
hardening, API and Start / Stop cases kept. A browser check of the four
pages on fixture data (Playwright, the in-app browser).
@@ -0,0 +1,161 @@
# Run tracking and Portal Sign-in — design
Date: 2026-10-22. Agreed with the operator in chat, question by question
(Q1–Q31, all "yes" to the recommended answers; Q3 changed to "who" = the
Portal User). Terms: `GLOSSARY.md`. Decision record:
`docs/adr/0001-run-events-in-files.md`. Jira: EEWA-31 (Stage 1),
EEWA-32 (Stage 2), EEWA-33 (Stage 3), each blocking the next.
## Goal
See every Run on the Portal: who started it, how long the eCW login took and
where the time went, every Run Error, Claim Error and Warning with its time,
and which claims a Run never reached. Only signed-in Portal Users see the
Portal; every Run records who started (and stopped) it.
## Decisions
| # | Decision |
|---|---|
| Q1 | Three kinds: **Run Error** (the Run stopped), **Claim Error** (one claim not checked), **Warning** (nothing stopped). **Fail** stays a Charge Summary category only. |
| Q2/Q28 | Per **eCW Login Attempt**: start, end, duration, 2FA mail wait, result; steps timed: page open, password typed, captcha, 2FA mail requested (the Log In click), mail received, approval clicked, home page. |
| Q3/Q8 | "Who" = the **Portal User** who signed in; **Started By** / **Stopped By**. |
| Q4 | Events go to `Logs\events_<run>.jsonl` (ADR 0001). |
| Q5/Q25 | Old run logs are not imported. Nothing is deleted. |
| Q6/Q18/Q19/Q22 | A **Runs** page on the Portal: list + detail, detail refreshes every 5 s while its Run is going. |
| Q9 | The Portal stays on 127.0.0.1. |
| Q10–Q17, Q20, Q21 | Sign-in (Stage 2, below). |
| Q14 | Terminal Runs are allowed; Started By = the Windows user. |
| Q15/Q23 | Stage 3: Run Error banner; `started_by` column in `CV_ChargeSummary`; the Excel sheet unchanged. |
| Q24 | Disk check at Run start: system drive under 2 GB = Warning, under 500 MB = Run Error, exit 2, before login. |
| Q26 | Claim Error = search raised / form not read / never finished loading; Charge Summary still says Review. A claim not found in eCW = Warning. |
| Q27 | **Not Reached** = queued claims a Run Error (or Stop) came before. |
## Stage 1 — Run events and the Runs page (EEWA-31)
### `core/run_events.py` (new)
- `start(log_path)` — the events file is the log's name with `run_` →
`events_` and `.log` → `.jsonl` (`run_20261022_141530.log` →
`events_20261022_141530.jsonl`); the Run's id is `20261022_141530`.
Writes `run_start`.
- `emit(kind, **fields)` — one JSON line, `{"t": local ISO time to the
millisecond, "kind": kind, ...}`, flushed at once. **Never raises**: an
event that cannot be written is dropped, the Run carries on.
- Helpers: `warning(code, message, **f)`, `run_error(step, message,
exit_code, exc=None)` (with `traceback.format_exc()` when an exception is
given), `claim_error(claim, step, message)`, `login_step(step, **f)`,
`login_attempt_start()` / `login_attempt_end(result, message="")` (the
module numbers attempts).
- No OTP code, password or eCW username is ever put in an event.
### Event kinds
| kind | fields |
|---|---|
| `run_start` | `source` (portal / terminal), `started_by`, `mode` (VERIFY / LIVE-WRITE), `claim_limit`, `test_sheet` (file name only), `pid`, `log` |
| `disk` | `drive`, `free_mb` |
| `login_attempt_start` | `attempt` |
| `login_step` | `attempt`, `step`: `page_open`, `password`, `captcha_shown`, `captcha_solved`, `captcha_failed`, `twofa_requested`, `mail_received`, `approval_clicked`, `home` |
| `login_attempt_end` | `attempt`, `result`: `ok`, `twofa_timeout`, `captcha_failed`, `restricted`, `failed`; `message` |
| `queue` | `claims` (list) |
| `claim_start` | `n`, `total`, `claim` |
| `claim_result` | `n`, `claim`, `category` (Pass / Fail / Review / Not found) |
| `claim_error` | `claim`, `step` (`search` / `form` / `load`), `message` |
| `warning` | `code` (`not_found`, `fallback`, `dialog`, `cancel_retry`, `form_not_closed`, `disk_low`, `preflight`), `message`, optional `claim` |
| `screenshot` | `path` (a failure shot), optional `claim` |
| `run_error` | `step` (`config`, `disk`, `guard`, `login`, `restricted`, `twofa_timeout`, `preflight`, `claims`, `unexpected`), `exit_code`, `message`, `traceback` |
| `run_end` | `exit_code` |
| `stopped` | `stopped_by` (written by the Portal after Stop; Stage 2 fills the name) |
`started_by`: env `ECW_STARTED_BY` (the Portal sets it, Stage 2), else the
Windows user. `source`: as `bot_run.json` has it (`ECW_RUN_SOURCE`).
### Where the bot emits
- `flows/export_medcor_ready.py`: `main()` starts the events after
`runlog.start()` and writes `run_end` in its `finally`; `_main` writes a
`run_error` at every non-zero return and in every `except`, with the
traceback; the disk check sits before login next to the CDP check;
`_login_with_retry` wraps each `login()` in attempt start / end;
`_process_claims` writes `queue`, `claim_start`, `claim_result`,
`claim_error` (search raised, `_error`, `_not_evaluated`) and the
not-found Warning.
- `auth/login.py` / `captcha.py` / `twofactor.py` / `mailbox.py`: the login
steps.
- `core/fallback.py` `report()`, `core/page_utils.py` `auto_dialog` (the
event only — its behaviour is unchanged, see Context.md §10) and the
failure `shot()`, `pages/claim_form.py` Cancel retries and form-not-closed,
`pages/preflight.py` FAIL: Warnings / screenshots.
- `web/bot_runner.py` `stop()`: appends `stopped` to the Run's events file.
### Runs page
- `web/runs.py` (new): reads `Logs\events_*.jsonl` into Run summaries and a
Run's detail. A Run with no `run_end` is **Running** when `bot_run.json`
names it and its pid is alive, **Stopped** when a `stopped` event is
there, else **Ended without a record** (killed). Result: Done (exit 0),
Run Error (exit and step), Stopped, Running.
- `GET /runs` — newest first: start, Started By, Run Source, mode,
duration, result, eCW Login Attempts + total login time, Pass / Fail /
Review, Claim Errors, Warnings. Filters: from / to date, result.
- `GET /runs/{run_id}` — the login attempts table (each step's time, 2FA
wait = mail received − 2FA requested), Run Errors (traceback folded),
Claim Errors, Warnings, Not Reached claims, the timeline, screenshot
links; refreshes every 5 s while Running.
- `GET /runs/{run_id}/shot/{n}` — the Run's n-th screenshot event, served
only from inside `Screenshots\` and only as `.png`: a saved page's HTML
served from the Portal's origin would run eCW's scripts there, so the
detail page names where the DOM file is instead of linking it.
- A "Runs" link on every page's header.
### Disk check
Settings `DISK_WARN_MB=2048`, `DISK_STOP_MB=500`, on the system drive.
## Stage 2 — Sign-in (EEWA-32)
- `CV_PortalUser` (username, display name, role, PBKDF2-SHA256 hash + salt
+ iterations, must_change, disabled, created / changed) and
`CV_PortalSignIn` (time, username, event: sign_in / sign_out /
wrong_password / locked / timeout) in `Charge_Validation`.
- Roles Admin / Operator / Viewer. Start / Stop: Admin, Operator. Users
page and Sign-in history: Admin.
- `tools\add_portal_user.py` (password with `getpass`). Passwords ≥ 10
characters; change own; Admin reset → must change at next Sign-in;
disable, never delete.
- 5 wrong passwords → 15 minutes locked. Session cookie (HttpOnly,
SameSite=Strict), 30 minutes idle. Every page and `/api` behind it;
127.0.0.1 + Origin / token checks stay.
- The Portal passes the signed-in user to the child as `ECW_STARTED_BY`;
`stop()` writes `stopped_by`. `bot_run.json` keeps `started_by` too, so
the Dashboard's run card says "started by …".
- As built (`web/users.py`, `web/app.py`): sessions in the Portal's memory
(a restart signs everyone out); disabling a user, resetting their
password or changing their role ends / updates their open session at
once; an unknown username reads and costs the same as a wrong password;
the last active Admin cannot be demoted or disabled, nobody disables
themselves; every Admin change is a `CV_PortalSignIn` row with `by_user`;
every form POST carries the page token and is checked for Origin;
`next` after Sign-in never leaves the site.
## Stage 3 — Alerts and `started_by` (EEWA-33)
- Dashboard banner when the last Run ended in a Run Error.
- `CV_ChargeSummary.started_by` (nullable column, written from the Run's
Started By).
- As built: the banner also shows for a Run that ended without a record
(killed or crashed — no `run_end`), never for Done, Stopped or Running;
it names when, Started By, the step, exit code and message, how many
claims were Not Reached, and links the Run. `create_tables.sql` adds the
column to an existing table (`IF COL_LENGTH ... IS NULL ALTER TABLE`).
The bot checks the column before each insert (`results_db.has_column`)
and saves the rows without it while it is missing — a table not yet
updated never costs a Run its rows. The Portal selects it, shows it as
"Started By" in a claim's side panel, and a table without it reads
"Results table needs an update — run create_tables.sql" (SQLSTATE 42S22).
## Out of scope
Network access to the Portal, email / Teams alerts, importing old logs,
deleting old events, editing settings from the Portal.