Initial import: eCW Charge Validation bot and results Portal
Snapshot of the working project as of 2026-10-23 (local 2d4d4fc), one commit with no earlier history: the tracked files were de-identified first (synthetic claim ids and patient data only; see Context.md section 7). Credentials, run logs, screenshots, claim reports and client workbooks are gitignored and not included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
ba15a574ab
205 files changed
+56772
No files matched your search
@@ -0,0 +1,182 @@
|
||||
# Payer → Group Name categorization — design
|
||||
|
||||
Status: approved by user (Vasanth), 2026-09-15. Not yet implemented.
|
||||
|
||||
## Background
|
||||
|
||||
This is the next phase of the Salud eCW Charge Entry bot (see `README.md`),
|
||||
which already automates login, 2FA, claim lookup, and exporting the
|
||||
"Medcor Ready" claim queue to `Input\Claim Report_<date>.xlsx`.
|
||||
|
||||
The next manual step in the real process (shown in
|
||||
`Salud-Charge Entry_Medcor Ready.mp4`, reviewed 2:45–10:00) is: an operator
|
||||
renames a copy of that export to "Medcor ready", opens it in a dated network
|
||||
folder (`...\Charge\Allocation\<Month YYYY>\<MMDDYYYY>\`, alongside sibling
|
||||
workbooks `Patient`, `Pending with Error`, and `Program`), and manually
|
||||
categorizes every row by filtering on the `PAYER` column and bulk-filling a
|
||||
`Group Name` column with one of a small set of values (observed in the video:
|
||||
`Commercial`, `Dentaquest`, `Medicaid`, `Medicare`, `Program`, plus at least
|
||||
one more category for payers like `BH COLORADO ACCESS`).
|
||||
|
||||
This spec covers automating **only** that categorization step — not the
|
||||
downstream split into the three sibling workbooks, which stays manual for now
|
||||
and may be scoped separately later.
|
||||
|
||||
## Scope decisions (from user)
|
||||
|
||||
- **Scope**: categorize only. Do not build the split into
|
||||
Patient/Pending with Error/Program workbooks.
|
||||
- **Mapping source**: derived from the actual current export
|
||||
(`Input\Claim Report_09-11-2026.xlsx`, 729 rows, 39 distinct `PAYER`
|
||||
strings), not invented from video frames. Only high-confidence entries are
|
||||
seeded; everything else is left for a human to fill in (see "Seed mapping
|
||||
table" below) — this is intentional, not a gap to silently paper over.
|
||||
- **Unknown payer handling**: flag, never guess. A `PAYER` value with no
|
||||
entry in the mapping table gets a blank `Group Name`, is logged, and is
|
||||
called out in the run summary. No fuzzy/substring matching — exact string
|
||||
match only, so a typo'd or renamed payer can never be silently bucketed
|
||||
into the wrong group.
|
||||
- **Integration point**: inside the existing bot, not a standalone script.
|
||||
Runs automatically as part of `flows/export_medcor_ready.py`.
|
||||
- **Output location**: a new file, `Output\Medcor Ready_<date>.xlsx`. The
|
||||
original `Input\Claim Report_<date>.xlsx` is never modified — `claim_file.py`
|
||||
and everything downstream of it depends on that file's exact column
|
||||
layout (per `README.md`'s "Reading the claim ids back" section), so this
|
||||
step must not touch it.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
flows/export_medcor_ready.py
|
||||
... existing steps (login → claim lookup → export → claim_file.claim_ids()) ...
|
||||
→ categorize_by_payer() NEW — reads Input\, writes Output\
|
||||
core/payer_groups.py NEW — the lookup table + classify()
|
||||
payer_groups.json NEW — editable config, exact PAYER → Group Name
|
||||
core/spreadsheet.py EXTENDED — write_xlsx(), stdlib only
|
||||
```
|
||||
|
||||
`categorize_by_payer()` runs after the existing claim-id read-back (the point
|
||||
where the report is already saved and validated — see
|
||||
`README.md`'s "Reading the claim ids back"). It:
|
||||
|
||||
1. Reads `Input\Claim Report_<date>.xlsx` with the existing `spreadsheet.read_rows()`.
|
||||
2. For each row, looks up `PAYER` in `payer_groups.classify()`.
|
||||
3. Writes every original column plus a new `Group Name` column to
|
||||
`Output\Medcor Ready_<date>.xlsx`.
|
||||
4. Logs and message-boxes a summary: rows categorized, rows left blank, and
|
||||
the distinct list of unmapped `PAYER` strings — mirroring the existing
|
||||
`NOTIFY_AFTER_EXPORT` / `SHOW_CLAIM_IDS` pattern in the same file.
|
||||
|
||||
Never fatal: an unmapped payer does not stop the run, the same philosophy
|
||||
`export_claims_report()` already uses for other soft failures.
|
||||
|
||||
## Components
|
||||
|
||||
### `src/ecw_bot/core/payer_groups.py`
|
||||
|
||||
```python
|
||||
def classify(payer: str) -> str | None: ... # exact match against the config, or None
|
||||
def load_mapping() -> dict[str, str]: ... # reads payer_groups.json once
|
||||
```
|
||||
|
||||
Exact-string lookup only. No normalization beyond what's needed to match
|
||||
eCW's own values (the config keys are the literal `PAYER` strings as eCW
|
||||
exports them).
|
||||
|
||||
### `src/ecw_bot/payer_groups.json`
|
||||
|
||||
A flat `{"<exact PAYER string>": "<Group Name>"}` map, hand-editable by
|
||||
anyone on the team without touching code — same reasoning as `.env` for
|
||||
settings and `selectors\` for DOM selectors: this data changes for a
|
||||
different reason (a new payer shows up) than the code around it does.
|
||||
|
||||
### `core/spreadsheet.py` — new `write_xlsx()`
|
||||
|
||||
The project deliberately has no `openpyxl` dependency (see README: "every
|
||||
added package is an approval"). `write_xlsx()` follows the same approach
|
||||
already used for reading — a minimal single-sheet xlsx as a zip of XML, cells
|
||||
as inline strings (no `sharedStrings.xml`, matching what eCW's own exports
|
||||
already do, per the existing reader's comments) — built with `zipfile` and
|
||||
string templates only.
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
Input\Claim Report_<date>.xlsx
|
||||
→ read_rows() existing
|
||||
→ for each row: classify(PAYER) new
|
||||
→ write_xlsx(rows + Group Name) new
|
||||
→ Output\Medcor Ready_<date>.xlsx
|
||||
```
|
||||
|
||||
## Error handling
|
||||
|
||||
- Unmapped payer: blank `Group Name`, logged, summarized — not fatal.
|
||||
- `Input\` file missing/unreadable: this step is skipped with a clear log
|
||||
line; it never blocks the rest of the run, since the export already
|
||||
succeeded by the time this step runs.
|
||||
- `write_xlsx()` failure (disk full, permission): logged and raised — same
|
||||
severity as any other unexpected I/O failure in this flow, since silently
|
||||
losing the categorized output would defeat the point of the step.
|
||||
|
||||
## Testing
|
||||
|
||||
`tests/test_payer_groups.py`, hand-rolled in the existing style (no pytest):
|
||||
|
||||
- Exact-match lookups against a small fixture mapping.
|
||||
- Unmapped payer → `None`, never a guess.
|
||||
- Round-trip test for `write_xlsx()`: write a small fixture table, read it
|
||||
back with the existing `read_rows()`, assert the data matches — the same
|
||||
"never touched eCW" local-fixture philosophy as `test_claim_form.py`.
|
||||
|
||||
## Seed mapping table
|
||||
|
||||
Built from the real, current payer list (`Input\Claim Report_09-11-2026.xlsx`,
|
||||
729 rows). Only high-confidence entries are seeded — this covers roughly 95%
|
||||
of row volume in that file.
|
||||
|
||||
**Mapped (seeded):**
|
||||
|
||||
| Payer (exact eCW string) | Group |
|
||||
|---|---|
|
||||
| Medicaid | Medicaid |
|
||||
| DentaQuest | Dentaquest |
|
||||
| CHP+ DENTAQUEST | Dentaquest |
|
||||
| AARP Medicare Advantage | Medicare |
|
||||
| Aetna Medicare ADV | Medicare |
|
||||
| OUTREACH PROGRAM | Program |
|
||||
| Anthem Blue Cross Blue Shield | Commercial |
|
||||
| Cigna HealthCare Of Colorado | Commercial |
|
||||
| Cigna Healthcare of CO | Commercial |
|
||||
| Cigna Salud Employee | Commercial |
|
||||
| Cigna | Commercial |
|
||||
| Cypress Benefit Admin / Cigna PPO | Commercial |
|
||||
| United Healthcare | Commercial |
|
||||
| United Health Care | Commercial |
|
||||
| United healthcare | Commercial |
|
||||
| Unitedhealth One | Commercial |
|
||||
| Oxford Health Insurance UHC | Commercial |
|
||||
| UMR | Commercial |
|
||||
| Aetna | Commercial |
|
||||
| Aetna Alaska Care | Commercial |
|
||||
| AETNA HEALTH AND LIFE INSURANCE | Commercial |
|
||||
| BIND BENEFITS INC | Commercial |
|
||||
| Select Health | Commercial |
|
||||
|
||||
**Left unmapped — needs a human decision before first real run (or will be
|
||||
flagged on first run and filled in afterward):**
|
||||
|
||||
BH COLORADO ACCESS (15 rows), Delta Dental of Colorado (3), Devoted (2),
|
||||
Clinic Sliding Fee Scale (2), Patient (1), Gravie Admin Services (1),
|
||||
Tricare (1), Rocky Mountain Health Plans (1), Denver Elevate Health Plan (1),
|
||||
The Kempton Group Administrators Inc (1), Optum Care Network (1), GPA (1),
|
||||
Northeast Health Partners (1), Delta Dental of Pennsylvania (1),
|
||||
Delta Dental of Wisconsin (1), Delta Dental (1), Delta Dental of Arkansas (1).
|
||||
|
||||
## Out of scope (explicitly)
|
||||
|
||||
- Splitting rows into the Patient / Pending with Error / Program sibling
|
||||
workbooks.
|
||||
- Writing to the network `Charge\Allocation\...` folder — output stays local
|
||||
to `Output\`.
|
||||
- Any change to `Input\Claim Report_<date>.xlsx` or anything that reads it.
|
||||
@@ -0,0 +1,269 @@
|
||||
# Condition Framework — Design
|
||||
|
||||
**Status:** Approved by user, 2026-09-16
|
||||
**Supersedes nothing** — this formalizes the pattern Condition 1 (telehealth)
|
||||
already established informally in `flows/export_medcor_ready.py`.
|
||||
|
||||
## Why
|
||||
|
||||
The Charge Validation Process is, by the client's own description, a long
|
||||
list of per-claim conditions to check and — when a condition's criteria are
|
||||
met — a correction to apply (Condition 1: a telehealth visit needs POS=11 and
|
||||
M1=GT). Conditions are being implemented one at a time, and there will be
|
||||
many more of them. This is a healthcare billing bot: every write it makes
|
||||
changes what gets billed to a real payer, so the process must be:
|
||||
|
||||
- **Reusable** — adding Condition N must not require touching the code for
|
||||
Conditions 1..N-1, and must follow one obvious, fixed shape every time.
|
||||
- **Exception-safe** — a bug in one condition's logic must never corrupt
|
||||
another condition's result on the same claim, crash the whole overnight
|
||||
batch, or leave a claim half-changed with no record of what happened.
|
||||
- **Provably correct before it ever touches a real claim** — every condition
|
||||
must be unit-testable against a fixture, exactly like
|
||||
`tests/test_telehealth_fix.py` already does for Condition 1.
|
||||
|
||||
This spec explicitly rejects driving the live automation with an
|
||||
LLM-agent-controlled browser (e.g. Playwright MCP) as the *engine* for this
|
||||
reason: none of the three properties above hold for an engine whose
|
||||
per-claim behavior is an LLM's live judgment call rather than fixed,
|
||||
versioned code. Playwright MCP remains valuable as a *development and
|
||||
debugging aid* (inspecting live eCW DOM, prototyping selectors) — that usage
|
||||
is unaffected by this spec and requires no code change.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Not changing how claims are found, opened, or read (`claim_form.py`,
|
||||
`export_medcor_ready.py`'s claims-phase — untouched).
|
||||
- Not building a self-healing/adaptive-selector system. If eCW's DOM shifts
|
||||
enough to break a selector, that is still a `selectors/claim_form.py`
|
||||
update, done and tested like every past one (see CHANGELOG "Day 5").
|
||||
- Not retrying failed conditions automatically. A condition that throws is
|
||||
logged and skipped for that claim, once, this run.
|
||||
|
||||
## Architecture
|
||||
|
||||
### New package: `src/ecw_bot/conditions/`
|
||||
|
||||
```
|
||||
src/ecw_bot/conditions/
|
||||
__init__.py # CONDITIONS: list[module] — the registry
|
||||
base.py # ConditionResult dataclass + run_all()
|
||||
telehealth.py # Condition 1 — detect/apply/describe wrapper
|
||||
```
|
||||
|
||||
### The Condition contract
|
||||
|
||||
Every condition module in the registry exposes exactly these three
|
||||
functions. This is the whole interface — nothing else is required, nothing
|
||||
else is read by the runner.
|
||||
|
||||
```python
|
||||
def detect(claim_id: str, data: dict) -> bool:
|
||||
"""
|
||||
Pure. No `page`, no I/O, no side effects. `data` is the same dict
|
||||
`claim_form.extract()` already produces (plus `progress_notes`, when
|
||||
OPEN_PROGRESS_NOTES is on). Must be safe to call with a partially
|
||||
populated dict — return False, never raise, on missing keys.
|
||||
"""
|
||||
|
||||
def apply(page, claim_id: str, data: dict) -> "ConditionResult":
|
||||
"""
|
||||
Only called when detect() returned True. Makes the eCW field edits for
|
||||
this condition and returns what changed. MUST NOT click Save/OK and
|
||||
MUST NOT show any confirmation dialog itself — the runner's caller owns
|
||||
both, once, after every condition has had a chance to run. Raising is
|
||||
allowed and expected to be handled by the caller (see Exception
|
||||
handling below) — apply() does not need its own try/except for its own
|
||||
field-write logic, only for optional recovery it wants to attempt
|
||||
itself.
|
||||
"""
|
||||
|
||||
def describe(result: "ConditionResult") -> str:
|
||||
"""
|
||||
One human-readable block for this condition's contribution to the
|
||||
combined confirmation dialog and the log line — mirrors the per-row
|
||||
lines `_apply_telehealth_fix` already builds today.
|
||||
"""
|
||||
```
|
||||
|
||||
### `ConditionResult` (`conditions/base.py`)
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ConditionResult:
|
||||
condition_name: str
|
||||
changed: bool
|
||||
changes: list # same shape as claim_form.set_cpt_pos_modifier()'s
|
||||
# return value: [{"row", "code", "before", "after"}, ...]
|
||||
|
||||
review: str = "" # added 2026-10-05 (unit D) — see below
|
||||
|
||||
@classmethod
|
||||
def none(cls, condition_name: str) -> "ConditionResult":
|
||||
return cls(condition_name=condition_name, changed=False, changes=[])
|
||||
|
||||
@classmethod
|
||||
def declined(cls, condition_name: str, reason: str) -> "ConditionResult":
|
||||
return cls(condition_name=condition_name, changed=False, changes=[],
|
||||
review=reason)
|
||||
```
|
||||
|
||||
**Amendment 2026-10-05 (review items D2/P2, unit D).** `none()` means
|
||||
"nothing to do — the claim already follows this rule". A condition that
|
||||
matched the claim but could not act on it (the CPT/HCPCS tab would not
|
||||
open, an admin code would not add with the grid untouched) now returns
|
||||
`declined(name, reason)` instead: nothing changed, but the claim is not
|
||||
clean, and `reason` reaches the Charge Summary as a "Review" row. A
|
||||
changed result may also carry `review` (a failed add that still moved the
|
||||
grid). Two optional module hooks join `detect/apply/describe`:
|
||||
|
||||
- `needs_review(claim_id, data) -> str | None` — asked only when
|
||||
`detect()` returned False, for a claim that matched the trigger but
|
||||
cannot be acted on (vaccine_admin_codes on an unmapped payer, no payer
|
||||
name, no usable age — never Medicare, which has its own rule).
|
||||
- `summary_comment(result) -> str` — the Charge Summary comment for a
|
||||
changed result, built from what actually happened, in place of the
|
||||
summary's fixed `COMMENTS[name]`.
|
||||
|
||||
### The runner (`conditions/base.py`)
|
||||
|
||||
```python
|
||||
def run_all(page, claim_id: str, data: dict) -> list[ConditionResult]:
|
||||
"""
|
||||
Runs every registered condition against one claim, in registry order.
|
||||
Returns only the results where something actually changed — a claim
|
||||
with nothing to fix returns an empty list, and the caller shows no
|
||||
dialog and saves nothing.
|
||||
"""
|
||||
results = []
|
||||
for cond in CONDITIONS:
|
||||
name = cond.__name__.rsplit(".", 1)[-1]
|
||||
try:
|
||||
if not cond.detect(claim_id, data):
|
||||
continue
|
||||
result = cond.apply(page, claim_id, data)
|
||||
except Exception as exc:
|
||||
print(f"[COND {name} {claim_id}] Failed: {exc}")
|
||||
shot(page, f"condition_{name}_failed_{claim_id}")
|
||||
continue # this condition's failure must not block the next one
|
||||
if result.changed:
|
||||
results.append(result)
|
||||
return results
|
||||
```
|
||||
|
||||
`shot()` is the existing screenshot helper already used by `open_claim()`'s
|
||||
diagnostic wrapping (`claim_form.py`) — reused here, not reinvented.
|
||||
|
||||
**Amendment 2026-10-05 (review items D2/P2, unit D).** The real signature
|
||||
is now `run_all(page, claim_id, data, conditions=None, before_apply=None,
|
||||
report=None)`. Its return value is unchanged — only the changed results —
|
||||
but that list is not everything that happened, so a caller that passes a
|
||||
dict as `report` gets the whole account:
|
||||
|
||||
| key | contents |
|
||||
|---|---|
|
||||
| `fired` | names whose `detect()` returned True |
|
||||
| `changed` | names with a changed result (= the return value) |
|
||||
| `failed` | `{"name", "stage", "error"}` per raise; `stage` is `"detect"`, `"apply"` or `"needs_review"`, `error` the exception's first line |
|
||||
| `review` | `{"name", "reason"}` from every result (changed or not) with a non-empty `review`, and from `needs_review()` of every condition whose `detect()` returned False |
|
||||
| `comments` | `{name: summary_comment(result)}` for changed results whose module has the hook (a hook that raises is logged and left out — the summary falls back to `COMMENTS[name]`) |
|
||||
|
||||
Exceptions are still isolated exactly as in the sketch above: logged,
|
||||
screenshotted, and the next condition still runs. `detect()` and `apply()`
|
||||
now each sit in their own try (the sketch's single try is how it began), so
|
||||
the report can say which stage raised.
|
||||
|
||||
### Caller contract (replaces `_apply_telehealth_fix`'s direct call)
|
||||
|
||||
`export_medcor_ready.py`'s `_read_claim_form()` changes from calling
|
||||
`_apply_telehealth_fix(page, claim_id, data)` directly to:
|
||||
|
||||
```python
|
||||
if cfg.APPLY_CORRECTIONS:
|
||||
results = conditions.run_all(page, claim_id, data)
|
||||
saved_write = _confirm_and_save(page, claim_id, results)
|
||||
```
|
||||
|
||||
(As built since 2026-10-05: `run_all(..., report=report)` and
|
||||
`_confirm_and_save(page, claim_id, results, report)`; the report is stored
|
||||
on the claim's data as `data["_conditions"]` = `{"fired", "saved",
|
||||
"failed", "review", "comments"}`, which the Charge Summary reads.)
|
||||
|
||||
`_confirm_and_save` (new, small, lives in `export_medcor_ready.py` — it is
|
||||
orchestration, not a condition):
|
||||
|
||||
1. If `results` is empty → return `False` immediately, no dialog, no save.
|
||||
(A claim where nothing fired is not different from today's "no
|
||||
telehealth mention" case.)
|
||||
2. Otherwise build one combined message: every fired condition's
|
||||
`describe()` output, concatenated, in the same
|
||||
`msgbox.ok_cancel(...)` shape Condition 1 uses today.
|
||||
3. One Yes/No. Yes → `claim_form.save_claim(page)`. No → log "declined",
|
||||
return `False` (the caller's existing `finally` still closes with
|
||||
Cancel, unchanged).
|
||||
4. Return whether the save actually happened, same as
|
||||
`_apply_telehealth_fix` returns today — this is what `_read_claim_form`
|
||||
uses to decide whether its own `finally` should call `close_claim()`.
|
||||
|
||||
**Amendment 2026-10-05 (review item P2, unit D):** between 1 and 2 —
|
||||
and after the SAVE_CORRECTIONS gate — if the report has any `failed` or
|
||||
`review` entry, the box is never shown: the log says `[COND <id>] Not
|
||||
offering Save: <reasons> — closing with Cancel` and it returns `False`.
|
||||
|
||||
**Why one dialog, not one per condition:** if two conditions both want
|
||||
changes on the same claim, saving after only reading half the operator's
|
||||
decision is exactly the kind of half-applied state this spec exists to
|
||||
prevent. The operator sees the whole picture — every condition that fired,
|
||||
every field that would change — before any single Save/Cancel decision.
|
||||
|
||||
## Exception handling — the actual rules
|
||||
|
||||
This is the section the user asked for explicitly, spelled out rather than
|
||||
left implicit:
|
||||
|
||||
| Failure | What happens |
|
||||
|---|---|
|
||||
| One condition's `detect()` raises | Caught by `run_all`'s try/except (its own try since 2026-10-05, so the report can say `stage: "detect"`). Logged, screenshotted, skipped, and recorded in the report's `failed` — the claim reads "Review", not clean. Other conditions still run. |
|
||||
| One condition's `apply()` raises partway through a multi-row edit | Caught by `run_all`. Logged, screenshotted. That condition contributes nothing to `results`. **Corrected 2026-10-05 (review item P2):** this row used to say its partial edits "are never saved, because Save only happens ... for what's in `results`". That was wrong: Save (`saveAllData`) writes the whole form, partial edits included, and the box would have listed only the other conditions. So the raise is recorded in `run_all`'s report (`failed`, stage `"apply"`), and `_confirm_and_save` never offers Save for a claim whose report has a `failed` or `review` entry — the claim is closed with Cancel and reported "Review" in the Charge Summary. |
|
||||
| `_confirm_and_save`'s dialog itself fails to show (msgbox error) | Not caught here — surfaces to `_read_claim_form`'s existing outer try/except, which logs "Could not read the form" and closes via `finally`. No new behavior needed; the existing outer boundary already covers this. |
|
||||
| `save_claim()` returns `False` (save click didn't work) | Same as today: logged, `saved_write` stays `False`, `finally` calls `close_claim()` — the claim is left as read, not half-saved. |
|
||||
| A condition throws on every claim in the batch (systemic bug in that one condition) | By design, this spec does not add batch-level circuit-breaking — each claim's screenshot + log line is enough signal for the operator to notice a pattern across a run's log file. Adding an auto-disable-after-N-failures behavior is explicitly deferred (YAGNI): today's batches run at `CLAIM_PROCESS_LIMIT` scale, small enough that log review is a real diagnostic step, not a chore. |
|
||||
|
||||
## Testing
|
||||
|
||||
- `tests/test_telehealth_fix.py` — unchanged. It tests
|
||||
`claim_form.note_mentions_telehealth` and `claim_form.set_cpt_pos_modifier`
|
||||
directly; those functions do not move.
|
||||
- `tests/test_conditions_telehealth.py` (new) — tests
|
||||
`conditions/telehealth.py`'s thin wrapper: `detect()` delegates to
|
||||
`note_mentions_telehealth` correctly on a plain dict (no page needed);
|
||||
`apply()` calls `set_cpt_pos_modifier` and returns a populated
|
||||
`ConditionResult`; `describe()` produces the expected text.
|
||||
- `tests/test_conditions_runner.py` (new) — tests `run_all()` and
|
||||
`ConditionResult` in isolation with fake condition modules (no real
|
||||
telehealth logic involved):
|
||||
- a condition whose `detect()` returns `False` is never `apply()`-ed
|
||||
- a condition whose `apply()` raises is skipped, and a *second*,
|
||||
healthy condition registered after it in `CONDITIONS` still runs and
|
||||
its result is still returned
|
||||
- `results` is empty when every condition's `detect()` returns `False`
|
||||
- a condition's raised exception does not propagate out of `run_all`
|
||||
|
||||
## Migration checklist (informs the implementation plan)
|
||||
|
||||
1. Create `conditions/base.py` (`ConditionResult`, `run_all`).
|
||||
2. Create `conditions/telehealth.py` — move `note_mentions_telehealth`-based
|
||||
detection logic into `detect()`, wrap `set_cpt_pos_modifier` in
|
||||
`apply()`, write `describe()` from the existing message-building lines
|
||||
in `_apply_telehealth_fix`.
|
||||
3. Create `conditions/__init__.py` with `CONDITIONS = [telehealth]`.
|
||||
4. Add `_confirm_and_save` to `export_medcor_ready.py`; delete
|
||||
`_apply_telehealth_fix` (its logic is now split across
|
||||
`conditions/telehealth.py` and `_confirm_and_save`).
|
||||
5. Update `_read_claim_form` to call `conditions.run_all` +
|
||||
`_confirm_and_save` instead of `_apply_telehealth_fix` directly.
|
||||
6. New tests (above) all passing; full existing suite (88+ tests) still
|
||||
passing unchanged.
|
||||
7. Condition 2 (whatever it turns out to be) is the first real proof this
|
||||
spec worked: it should require touching only a new
|
||||
`conditions/<name>.py` + its test file + one line in `CONDITIONS`.
|
||||
@@ -0,0 +1,137 @@
|
||||
# Dental claims run every applicable condition — design
|
||||
|
||||
Date: 2026-10-07. Status: approved in chat (operator, 2026-10-07);
|
||||
implemented 2026-10-12 (operator: "this is our number one condition").
|
||||
|
||||
## 2026-10-12 update (as built)
|
||||
|
||||
- The rule now also covers #12 (H codes: BH payer → modifier removed;
|
||||
any other payer → line unchecked + fee 0) and #14 (Medicare: G8510 /
|
||||
G8431 unchecked + fee 0, #6 not run). Both run on Dental claims like
|
||||
every other condition; #12's BH path has no modifier column to read on
|
||||
the Dental grid, so it never fires there.
|
||||
- A fix the Dental screen cannot hold → **Review** (operator): a condition
|
||||
module declares `DENTAL_UNSUPPORTED = "<reason>"` and `run_all` records
|
||||
that reason instead of applying it on a `screen == "dental"` read —
|
||||
telehealth, medicare_telehealth, vaccine_admin_codes.
|
||||
- Dental claims are **never saved** in this change (operator), whatever
|
||||
`SAVE_CORRECTIONS` says; the review-before-close box still shows.
|
||||
- A Dental claim is read and run through the conditions even with
|
||||
CHECK_HIGH_RISK and CHECK_SLIDING_FEE both off (it used to come back
|
||||
blank).
|
||||
- The Charge Summary applies the same claim-level checks (note read, CPT /
|
||||
ICD rows read, conditions run) to a Dental read as to UB/HCFA.
|
||||
- Row selectors gained a third (Dental) alternative, scoped to the Dental
|
||||
window on screen (`:visible`); `extract()` returns `screen`
|
||||
("ub" / "hcfa" / "dental").
|
||||
- Not live-verified yet: the first live run should be SAVE-off on a few
|
||||
Dental claims (one with DCONS or Dscprit).
|
||||
|
||||
## Rule (operator, 2026-10-07)
|
||||
|
||||
Every claim is checked against **every condition whose Payer column is
|
||||
"All" / "All payers"**, **plus** every condition specific to that claim's
|
||||
payer (Medicare, Medicaid, Dental payers, ...). This holds for every payer
|
||||
and every claim screen — UB, HCFA and Dental alike.
|
||||
|
||||
UB and HCFA claims already work this way: each claim runs the whole
|
||||
registry (`conditions/__init__.py` `CONDITIONS`) and each condition's own
|
||||
`detect()` decides from the claim's codes and payer. **The gap is Dental:** a
|
||||
Dental claim today runs only #15 high risk and #8 sliding fee and never
|
||||
reaches `run_all`. So, for example, `dummy_codes` (DCONS/Dscprit, which are
|
||||
Dental codes) can never fire on any claim.
|
||||
|
||||
## What a Dental claim will run
|
||||
|
||||
| Doc # | Condition | Payer (document) | Module |
|
||||
|---|---|---|---|
|
||||
| 5 | J2930/J2001 uncheck + fee 0 | ALL | `j2930_j2001_update` |
|
||||
| 6 | Depression DX pointer | All | `depression_dx_pointer` |
|
||||
| 7 | Zero-fee uncheck / priced line check | ALL | `zero_fee_uncheck`, `priced_line_check` |
|
||||
| 21 | Telehealth POS 2 + modifier | All | `telehealth` |
|
||||
| 22 | Medicare telehealth G2025 | Medicare | `medicare_telehealth` |
|
||||
| 18 | Vaccine admin codes | All except Medicare | `vaccine_admin_codes` |
|
||||
| 8 | DCONS/Dscprit uncheck + fee 0 | All Dental Payer | `dummy_codes` |
|
||||
| 8 | Sliding fee date check | All Dental Payer | `pages/sliding_fee` (unchanged) |
|
||||
| 15 | High risk in Billing Notes | Dental Payers | `pages/dental_notes` (unchanged) |
|
||||
|
||||
No condition gains a form-type check: each still decides from codes and
|
||||
payer only. A condition whose codes are not on the claim does nothing.
|
||||
|
||||
## Dental screen facts (from Dom.txt, claim screen `.modal[id^="dentalClaim"]`)
|
||||
|
||||
- CPT/CDT grid `table[id^="claimCPTCodesTable"]`; real rows are
|
||||
`tr[ng-repeat^="CPT in CPTCodes"]` (no `CPTROW_` id).
|
||||
- Per row: checkbox `[ng-click*="updateBillToIns("]` (same as UB/HCFA), code
|
||||
`[data-fieldname="web_cptcode"]`, pointers `web_icd1`..`web_icd4`, units
|
||||
`web_units`, fee `web_billedFee` (blur runs `editCPTRow('BILLED_FEE')`),
|
||||
DOS `DentalClaimCPTSDOS`.
|
||||
- **No per-row POS and no modifier columns.** POS is claim-level
|
||||
(`ng-model="ClaimData.pos"`).
|
||||
- Prog. Notes button `button[ng-click="viewProgNote()"]`
|
||||
(`dentalClaimProgressNoteBtn…`) — the existing `SEL_PROGRESS_NOTES` matches.
|
||||
- Insurance grid: `#DentalClaimTbl2` (Bill To, Ins Id, Name, ...).
|
||||
- Billing Notes: `#dentalClaimNotes` (already read by high risk).
|
||||
|
||||
None of the row fields has been exercised live yet; Dom.txt is one capture.
|
||||
|
||||
## Design
|
||||
|
||||
1. **Selectors** (`selectors/claim_form.py`): add the Dental alternative to
|
||||
each row selector the helpers already use — `SEL_CPT_ROWS`,
|
||||
`SEL_CPT_CODE`, `SEL_CPT_ICD1..4`, `SEL_CPT_BILLED_FEE`, `SEL_CPT_UNITS`.
|
||||
The Dental row selector is scoped under `.modal[id^="dentalClaim"]` so it
|
||||
cannot match a grid on another screen. Within one row only one alternative
|
||||
ever matches, the same contract the UB/HCFA pairs rely on.
|
||||
2. **Reader** (`pages/claim_form.py` `_EXTRACT_JS`): read Dental rows into the
|
||||
same `cpt` row shape (`code`, `icd1..4`, `units`, `billed_fee`, `billed`,
|
||||
`sdos`; `pos`/`mod1..4` empty), insurance from `#DentalClaimTbl2`, the
|
||||
Dental claim number, ICD list, and `notes` from `#dentalClaimNotes`.
|
||||
One `extract()` for all three screens.
|
||||
3. **Flow** (`flows/export_medcor_ready.py`): a Dental claim runs high risk +
|
||||
sliding fee first (unchanged, read-only), then the same path as UB/HCFA:
|
||||
`extract()`, Prog. Notes (when `OPEN_PROGRESS_NOTES`), `run_all`, per-claim
|
||||
box, Charge Summary. The Dental results stay on the claim's data so the box
|
||||
and the summary show both.
|
||||
4. **A fix the Dental screen cannot make** — e.g. telehealth needs a row
|
||||
modifier and the Dental grid has none, or POS is claim-level: the condition
|
||||
returns `declined(...)` with the reason ("Dental screen has no modifier
|
||||
column"), so the claim is **Review** in the Charge Summary. Never a silent
|
||||
skip, never a guess.
|
||||
5. **Charge Summary**: one row per claim combining condition comments with the
|
||||
Dental findings (HIGH RISK, sliding fee). Category: Review beats Fail beats
|
||||
Pass, as today.
|
||||
6. **Saving**: Dental claims are **never saved** in this change, whatever
|
||||
`SAVE_CORRECTIONS` says. Their corrections are shown and listed as
|
||||
"Corrections pending (not saved)". Dental save is enabled only after a live
|
||||
verification-mode run confirms the row writes.
|
||||
|
||||
## Error handling
|
||||
|
||||
- Dental form not loaded (`wait_claim_loaded`): conditions skipped, claim is
|
||||
Review — same as HCFA today.
|
||||
- A Dental row field missing (fee or checkbox not found): the existing helpers
|
||||
already treat a missing field as unknown, not as zero/unchecked.
|
||||
- High risk / sliding fee failures keep their current Review handling.
|
||||
|
||||
## Testing
|
||||
|
||||
- Grow `tests/fixtures/dental_claim.html` from Dom.txt's real Dental grid
|
||||
(rows, checkbox, fee, pointers, insurance grid, Prog. Notes button) with no
|
||||
patient data.
|
||||
- `extract()` on the Dental fixture returns the expected `cpt` and `insurance`.
|
||||
- Per condition, a Dental case: dummy codes (DCONS/Dscprit) uncheck + zero;
|
||||
J2001 uncheck + zero; zero-fee uncheck; priced line check; depression
|
||||
pointer retag on `web_icd*`; telehealth on Dental → declined/Review.
|
||||
- Flow: a Dental claim gets high risk + sliding fee **and** condition results
|
||||
in one summary row; never offered Save.
|
||||
- Full suite stays green.
|
||||
- Live: one SAVE-off run on Dental test claims (with a DCONS or Dscprit line)
|
||||
before Dental save is considered.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- New document conditions not yet built (#3, #4, #9-#14, #16, #19, #20,
|
||||
#23-#55).
|
||||
- Saving Dental claims.
|
||||
- "Special programs" half of #8.
|
||||
@@ -0,0 +1,164 @@
|
||||
# Charge Summary in SQL Server + results web page — design
|
||||
|
||||
Date: 2026-10-08. Status: design approved in chat (operator, 2026-10-08),
|
||||
spec under review.
|
||||
|
||||
## Goal
|
||||
|
||||
Every run's Charge Summary rows are also stored in SQL Server, and a simple
|
||||
local web page (FastAPI) shows them. This is the basic version; later it is
|
||||
folded into the full portal.
|
||||
|
||||
## Decisions (operator, 2026-10-08)
|
||||
|
||||
- Server `MECARE331\MECARE331` (SQL Server 2022, on this machine), database
|
||||
**`Charge_Validation`** (created by the operator).
|
||||
- The table follows the team's **Charge Summary File Format**
|
||||
(`\\mecare2\Production\Salud-eCW RPA\Charge Summary File Format.xlsx`) —
|
||||
the same 24 columns as `core/charge_summary.HEADERS`, plus run columns and
|
||||
a details JSON. PATIENT, Account Number and Guarantor Name are stored, as
|
||||
in the Excel file.
|
||||
- FastAPI for the backend. All needed packages are already installed
|
||||
(fastapi 0.141, uvicorn, jinja2, httpx, pyodbc, ODBC Driver 18); nothing new
|
||||
is added to `requirements.txt` for the bot itself.
|
||||
|
||||
## Table
|
||||
|
||||
`tools/sql/create_tables.sql`, run by the operator in SSMS (the bot never
|
||||
creates or alters tables):
|
||||
|
||||
```sql
|
||||
USE Charge_Validation;
|
||||
CREATE TABLE dbo.ChargeSummary (
|
||||
id INT IDENTITY(1,1) PRIMARY KEY,
|
||||
run_at DATETIME2 NOT NULL,
|
||||
run_mode NVARCHAR(20) NOT NULL, -- VERIFY / LIVE-WRITE / TEST
|
||||
s_no INT NULL,
|
||||
claim_no NVARCHAR(20) NOT NULL,
|
||||
service_date DATE NULL,
|
||||
pvdr NVARCHAR(50) NULL,
|
||||
patient NVARCHAR(200) NULL,
|
||||
payer NVARCHAR(200) NULL,
|
||||
group_name NVARCHAR(100) NULL,
|
||||
status NVARCHAR(100) NULL,
|
||||
charges DECIMAL(12,2) NULL,
|
||||
pmts_adjs DECIMAL(12,2) NULL,
|
||||
pmts_refunds DECIMAL(12,2) NULL,
|
||||
adjustment DECIMAL(12,2) NULL,
|
||||
withheld DECIMAL(12,2) NULL,
|
||||
balance DECIMAL(12,2) NULL,
|
||||
visit_type NVARCHAR(100) NULL,
|
||||
account_number NVARCHAR(50) NULL,
|
||||
provider_name NVARCHAR(200) NULL,
|
||||
guarantor_name NVARCHAR(200) NULL,
|
||||
received_date DATE NULL,
|
||||
assigned_to NVARCHAR(100) NULL,
|
||||
completed_on DATE NULL,
|
||||
comments NVARCHAR(MAX) NULL,
|
||||
current_status NVARCHAR(200) NULL,
|
||||
category NVARCHAR(20) NULL,
|
||||
details_json NVARCHAR(MAX) NULL
|
||||
);
|
||||
CREATE INDEX IX_ChargeSummary_run_at ON dbo.ChargeSummary (run_at);
|
||||
CREATE INDEX IX_ChargeSummary_claim_no ON dbo.ChargeSummary (claim_no);
|
||||
```
|
||||
|
||||
Column map (Excel → DB) lives in one place, `results_db.COLUMNS`, built from
|
||||
`charge_summary.HEADERS` so the two cannot drift: S.No→s_no, CLAIMS#→claim_no,
|
||||
SERVICE DATE→service_date, PVDR→pvdr, PATIENT→patient, PAYER→payer, Group
|
||||
Name→group_name, STATUS→status, CHARGES→charges, PMTS/ADJS→pmts_adjs,
|
||||
Pmts/Refunds→pmts_refunds, Adjustment→adjustment, WithHeld→withheld,
|
||||
Balance→balance, Visit Type→visit_type, Account Number→account_number,
|
||||
Provider Name→provider_name, Guarantor Name→guarantor_name, Recevied
|
||||
date→received_date, "Assigned To "→assigned_to, Completed On→completed_on,
|
||||
Comments→comments, Current Status→current_status, Category→category.
|
||||
|
||||
Value conversion: money columns strip `$`, `,` and spaces → Decimal; dates
|
||||
accept `mm/dd/yyyy` (and an Excel date/datetime) → date; blank or
|
||||
unparseable → NULL (the raw text is not lost: the Excel file still has it).
|
||||
|
||||
## Bot side — `src/ecw_bot/core/results_db.py`
|
||||
|
||||
- Settings (`settings.py`, from `.env`):
|
||||
- `RESULTS_DB` (bool, default **off**): write rows to the DB.
|
||||
- `RESULTS_DB_CONN` (string): ODBC connection string. Default:
|
||||
`DRIVER={ODBC Driver 18 for SQL Server};SERVER=MECARE331\MECARE331;DATABASE=Charge_Validation;Trusted_Connection=yes;TrustServerCertificate=yes;`
|
||||
(Windows login, no password). A SQL login's password, if ever used, is put
|
||||
in `.env` by the operator only.
|
||||
- `to_record(row: dict, *, run_at, run_mode, details) -> dict`: one Charge
|
||||
Summary row (header→value) to DB column→value, converted.
|
||||
- `insert_rows(conn, records) -> int`: one parameterized `INSERT` per row
|
||||
(`executemany`), one commit; returns rows written. Never builds SQL from
|
||||
values.
|
||||
- `save_summary(sheet, details, run_at, run_mode, connect=pyodbc.connect) ->
|
||||
int | None`: `sheet` is the header+rows list `charge_summary.build_rows`
|
||||
already returns; `details` maps claim no → details dict. Opens the
|
||||
connection, inserts, closes. **Any exception is caught**: one
|
||||
`[RESULTS-DB] Could not save ...` log line, returns None. The run, the Excel
|
||||
summary and the claims are never affected.
|
||||
- `details_json` per claim: `{"fired", "changed", "review", "failed",
|
||||
"comments", "saved", "changes": {condition: [change, ...]},
|
||||
"dental": {"high_risk", "sliding_fee"}}` — taken from the claim's
|
||||
`data["_conditions"]` (the flow adds `changes` there) and Dental results.
|
||||
|
||||
## Flow changes
|
||||
|
||||
- `charge_summary.write()` is split: `charge_summary.build(report_path,
|
||||
outcomes, assigned_to) -> sheet` (read + `build_rows`), and `write()` saves
|
||||
that sheet as today. The flow builds once and hands the same sheet to Excel
|
||||
and to the DB, so both always hold the same rows.
|
||||
- `_read_claim_form` adds `"changes": {r.condition_name: r.changes}` to
|
||||
`data["_conditions"]`; the claim loop keeps `details[claim_id]` next to
|
||||
`outcomes[claim_id]`.
|
||||
- `_write_charge_summary` calls `results_db.save_summary(...)` after the Excel
|
||||
write when `RESULTS_DB` is on, and logs `[RESULTS-DB] N row(s) saved`.
|
||||
- `run_mode`: `TEST` when `TEST_CLAIM_IDS_FILE` is set, else `LIVE-WRITE`
|
||||
when `SAVE_CORRECTIONS` is on, else `VERIFY` — the same words as the
|
||||
`[MODE]` banner.
|
||||
|
||||
## Web page — `web/`
|
||||
|
||||
- `web/app.py` (FastAPI) and `web/templates/index.html` (Jinja2), plain
|
||||
HTML/CSS, no external scripts or CDNs.
|
||||
- Read-only: only `SELECT` statements, parameterized.
|
||||
- `GET /` — page:
|
||||
- run picker (distinct `run_at`, newest first; default latest run),
|
||||
- Category filter (All / Pass / Fail / Review) and Claim# search,
|
||||
- counts per Category for the selected run (Pass / Fail / Review tiles),
|
||||
- the table in Charge Summary column order, Category coloured,
|
||||
- each row expands to show `details_json` (conditions fired, before/after
|
||||
changes, review reasons).
|
||||
- `GET /api/runs` — runs with counts. `GET /api/results?run_at=&category=&claim_no=`
|
||||
— rows as JSON (for the later portal).
|
||||
- Start: `python -m uvicorn web.app:app --host 127.0.0.1 --port 8000`.
|
||||
Bound to **127.0.0.1 only**: the page shows PHI and is not reachable from
|
||||
the network. A `run_web.bat` helper does the same.
|
||||
- The DB connection string is the same `RESULTS_DB_CONN`. If the DB cannot be
|
||||
reached the page shows a plain error message, not a stack trace.
|
||||
|
||||
## Error handling
|
||||
|
||||
- DB unreachable / permission denied / table missing during a run → one log
|
||||
line, run continues, Excel written.
|
||||
- A value that does not convert → NULL for that column, row still saved.
|
||||
- Web page: DB errors → HTTP 503 page "Results database not reachable".
|
||||
|
||||
## Testing (no real DB)
|
||||
|
||||
- `tests/test_results_db.py`: header→column map covers all 24 HEADERS;
|
||||
money/date conversion (incl. `$1,234.50`, blank, junk); `to_record` output;
|
||||
`insert_rows` sends one parameterized statement with values in column
|
||||
order (fake connection records calls); `save_summary` swallows a connect
|
||||
error and returns None; `details_json` shape.
|
||||
- `tests/test_charge_summary.py`: `build()` + `write()` split keeps today's
|
||||
output.
|
||||
- `tests/test_web_app.py`: FastAPI `TestClient` with a fake repository —
|
||||
page renders rows and Category counts, filters work, `/api/results` JSON
|
||||
shape, DB error → 503.
|
||||
- Full suite stays green. Live: operator runs `create_tables.sql`, sets
|
||||
`RESULTS_DB=1`, one test run, then opens the page.
|
||||
|
||||
## Out of scope (later portal)
|
||||
|
||||
Login/roles, editing rows, charts beyond the Category tiles, scheduling,
|
||||
AI features, hosting on the network.
|
||||
@@ -0,0 +1,176 @@
|
||||
# Start the bot from the results web page — design
|
||||
|
||||
Date: 2026-10-14. Approved section by section by the operator in chat
|
||||
(approach A, sections 1–3).
|
||||
|
||||
## Goal
|
||||
|
||||
The operator starts a Charge Validation run from the local results page
|
||||
(`run_web.bat` → http://127.0.0.1:8000) instead of the terminal. While it
|
||||
runs, the page shows live status; it can stop it; when it ends, the page
|
||||
shows that run's results.
|
||||
|
||||
## Decisions (operator, 2026-10-14)
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Controls | Start + live status + Stop (no settings editing on the page) |
|
||||
| Save mode | **VERIFY only.** A page-started run is always `SAVE_CORRECTIONS=0`, whatever `.env` says. LIVE-WRITE stays terminal-only. |
|
||||
| Message boxes | **Off** for the run's information boxes (per-claim, review-before-close, "Run the Lookup?", summaries). Boxes that need a person to act in the browser — CAPTCHA, 2FA manual step, security image — **stay on**. |
|
||||
| Approach | A: the web server starts `run.py` as a separate child process (not a thread, not Task Scheduler). |
|
||||
|
||||
Out of scope (YAGNI): editing settings from the page, scheduling, more than
|
||||
one run at a time, LIVE-WRITE from the page.
|
||||
|
||||
## Components
|
||||
|
||||
### New: `web/bot_runner.py` — process control only
|
||||
|
||||
- `start() -> dict` — starts `python run.py` (the project's own `run.py`,
|
||||
`sys.executable`, cwd = project root) as a child process with no console
|
||||
window of its own (`CREATE_NO_WINDOW`); the Chrome window still shows.
|
||||
The child's stdout/stderr go to `Logs\web_run_<ts>.out`, which catches a
|
||||
crash before the bot opens its own log; the bot's `Logs\run_<ts>.log`
|
||||
(named in the lock file) is the status source.
|
||||
Refuses (raises `AlreadyRunning`) when the lock file names a live pid.
|
||||
- `stop() -> dict` — `taskkill /PID <pid> /T /F`: the bot and its Chrome
|
||||
close together. Clears the lock file. Status becomes "Stopped by
|
||||
operator".
|
||||
- `status() -> dict` — `running` (bool), `pid`, `started_at`, `log_path`,
|
||||
`exit_code` (when this web process started it and it has ended),
|
||||
`stopped_by_operator`.
|
||||
- The command is fixed. Nothing from the request reaches the command line
|
||||
or the environment.
|
||||
|
||||
**Child environment** — set on the child process, so it wins over `.env`
|
||||
(`settings.py` loads `.env` with `os.environ.setdefault`):
|
||||
|
||||
```
|
||||
SAVE_CORRECTIONS=0
|
||||
ECW_KEEP_OPEN=0 # no "Press Enter to close" at the end
|
||||
QUIET_BOXES=1 # run-flow information boxes go to the log
|
||||
CONFIRM_BEFORE_LOOKUP=0
|
||||
```
|
||||
|
||||
Everything else follows `.env` (claim file, `CLAIM_PROCESS_LIMIT`,
|
||||
`RESULTS_DB`, proxy, ...).
|
||||
|
||||
### Lock file: `Logs\bot_run.json`
|
||||
|
||||
`{"pid", "started_at", "log_path", "source": "web" | "terminal"}`.
|
||||
|
||||
- Written by the **bot itself** at the start of `main()` (after
|
||||
`runlog.start()`, so `log_path` is known) and removed when `main()`
|
||||
returns — so a terminal run is seen by the page too.
|
||||
- Before writing it, `main()` refuses to start (exit 2, before login) when
|
||||
the file names a pid that is alive: two runs would share one eCW login
|
||||
guard and one Charge Summary.
|
||||
- A file naming a dead pid is stale: removed and ignored (a killed run
|
||||
cannot remove its own file).
|
||||
- `bot_runner.start()` checks it too, so the page answers 409 at once.
|
||||
|
||||
### New: `web/run_status.py` — read the run log
|
||||
|
||||
Pure functions over log text (no I/O except reading the file tail):
|
||||
|
||||
- **Phase** from the last matching line:
|
||||
`[LOGIN]` / `[CAPTCHA]` → "Logging in"; `[OTP]` / `[2FA]` → "Waiting for
|
||||
the 2FA approval mail"; `[OK] Logged in` → "Logged in"; `[CLAIMS]` →
|
||||
"Checking claims"; process ended → "Finished" or a failure text by exit
|
||||
code.
|
||||
- **Progress** from `[CLAIMS] --- claim n/N: <id> ---` → `n`, `N`, current
|
||||
claim id.
|
||||
- **Counts** from the new `[RESULT]` lines (below) → Pass / Fail / Review.
|
||||
- **Tail**: the last 30 lines.
|
||||
- **Exit codes** (from `flows/export_medcor_ready.main`): 0 finished; 2
|
||||
config / mode gate; 3 login guard refused (its `[GUARD]` line shown); 5
|
||||
claims phase failed; 6 2FA timed out; other → "ended with code N".
|
||||
|
||||
### Changed: `flows/export_medcor_ready.py`
|
||||
|
||||
- After each claim's outcome is known in `_process_claims`, one
|
||||
machine-readable line (terminal runs print it too):
|
||||
|
||||
```
|
||||
[RESULT] claim 3/7 9126000: Review
|
||||
```
|
||||
|
||||
The word is the Category the Charge Summary gets for that claim; a claim
|
||||
that gets no Charge Summary row (not found in the grid) prints
|
||||
`Not found`.
|
||||
- `main()` writes / removes the lock file as above.
|
||||
|
||||
### Changed: `core/msgbox.py`
|
||||
|
||||
- New setting `QUIET_BOXES` (default off; `settings.py`).
|
||||
- `info_box(title, msg, needs_person=False)`: with `QUIET_BOXES=1` and
|
||||
`needs_person=False`, no box — one log line instead:
|
||||
`[BOX] <title>: <first line of msg>`.
|
||||
- `ok_cancel` / `confirm` are unchanged (every remaining caller is a
|
||||
person-needed step: CAPTCHA, 2FA manual, LIVE-WRITE confirm — the last
|
||||
is never reached in VERIFY). The security-image `info_box` in
|
||||
`auth/login.py` passes `needs_person=True`.
|
||||
|
||||
### Changed: `web/app.py` + `web/templates/index.html`
|
||||
|
||||
Endpoints:
|
||||
|
||||
| Method | Path | Answer |
|
||||
|---|---|---|
|
||||
| POST | `/api/bot/start` | 200 `{status}`; 409 already running; 403 bad Origin/token |
|
||||
| POST | `/api/bot/stop` | 200 `{status}`; 409 nothing running; 403 bad Origin/token |
|
||||
| GET | `/api/bot/status` | `{running, phase, progress, counts, tail, started_at, exit_code, message}` |
|
||||
|
||||
Page: a "Bot run" card above the results.
|
||||
- Idle: **[Start run]** and "VERIFY mode — nothing is saved to eCW".
|
||||
- Running (polls `/api/bot/status` every 2 s): phase, progress bar
|
||||
"3 / 7", current claim, Pass/Fail/Review chips, collapsible "Live log"
|
||||
(last 30 lines), **[Stop]** (with a confirm dialog).
|
||||
- Ended: "Finished in 4m 12s — exit 0" (or the failure text); the results
|
||||
table reloads on the newest run.
|
||||
|
||||
## Security
|
||||
|
||||
The page was read-only; Start/Stop are its first actions. The existing
|
||||
TrustedHost check stops DNS rebinding but not a cross-site POST from
|
||||
another tab, which could burn eCW login attempts. So every POST needs both:
|
||||
|
||||
1. `Origin` (or, when absent, `Referer`) is `http://127.0.0.1:8000` or
|
||||
`http://localhost:8000` — else 403.
|
||||
2. Header `X-Run-Token` equals a token made at server start
|
||||
(`secrets.token_urlsafe(32)`) and embedded in the page — else 403.
|
||||
Compared with `secrets.compare_digest`.
|
||||
|
||||
No password on the page; `.env` credentials are read only by the bot.
|
||||
Responses keep `no-store`. The page stays bound to 127.0.0.1. The live log
|
||||
tail can carry PHI, as the results table already does — local only.
|
||||
|
||||
## Error handling
|
||||
|
||||
- Already running (page or terminal) → 409 "A run is already going
|
||||
(started HH:MM)".
|
||||
- Guard refused (exit 3) → the `[GUARD]` line shown ("next attempt
|
||||
allowed at ...").
|
||||
- Stop mid-claim: the claim form may stay open in eCW. VERIFY saves
|
||||
nothing; the next run's `clear_screen` closes it.
|
||||
- Web server restarted or page closed: the bot keeps running; on reload
|
||||
the page finds it through the lock file. `exit_code` is unknown then
|
||||
(the run is shown "Ended" when its pid is gone).
|
||||
- Log file not there yet → phase "Starting".
|
||||
|
||||
## Testing (standalone "N/M passed" scripts)
|
||||
|
||||
- `tests/test_bot_runner.py` — a fake child script in place of `run.py`
|
||||
(injectable command): start → running → stop → stopped; second start →
|
||||
`AlreadyRunning`; stale lock (dead pid) cleared; the child sees the forced
|
||||
env (`SAVE_CORRECTIONS=0`, `ECW_KEEP_OPEN=0`, `QUIET_BOXES=1`,
|
||||
`CONFIRM_BEFORE_LOOKUP=0`) even when the parent env says otherwise.
|
||||
- `tests/test_run_status.py` — sample log text → phase, progress, counts,
|
||||
tail; each exit code's text.
|
||||
- `tests/test_web_app.py` (added cases) — POST without/with wrong Origin or
|
||||
token → 403; right Origin + token → 200 (runner stubbed); status JSON
|
||||
shape; the token is in the page.
|
||||
- `tests/test_msgbox.py` (added cases) — `QUIET_BOXES=1`: info box logged,
|
||||
not shown; `needs_person=True` still shown.
|
||||
- Flow test — `[RESULT]` line per claim; `main()` lock file written,
|
||||
refused on a live one, removed at the end.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Results portal redesign — design
|
||||
|
||||
Date: 2026-10-19. Operator requests (voice, Tanglish), choices made on the
|
||||
visual companion mockups (`.superpowers/brainstorm/828-1792395410/`).
|
||||
|
||||
## Problem
|
||||
|
||||
The results page (`web/`, FastAPI + Jinja, 127.0.0.1:8000) opens on the
|
||||
newest run only. A second run (20 claims) hides the first (10 claims) behind
|
||||
a "Run" dropdown nobody noticed, so earlier runs look lost. The front page
|
||||
is three quarters claims table (24 columns, sideways slider), which the
|
||||
operator does not want on the home page. The look is "very basic".
|
||||
|
||||
## What the operator chose
|
||||
|
||||
| Question | Choice |
|
||||
|---|---|
|
||||
| Home page | **Dashboard**: Bot run card (Start / Stop / live progress), today's tiles, one big **Claim status** button |
|
||||
| Claim status page | **Date cards**, newest first |
|
||||
| A date with several runs | **All claims of the day together** (every run's rows in one list) |
|
||||
| One day's page | **Short table + side panel** |
|
||||
| Colours | **Navy & blue** (navy header, white cards on light grey, blue buttons) |
|
||||
| Live run's claim list | In the **Bot run card** ("Live claims" side panel) |
|
||||
| Search one claim over all dates | **Yes** — "Find a claim" on the Dashboard |
|
||||
| Approach | **A** — server pages (FastAPI + Jinja), one page per screen |
|
||||
|
||||
## Pages
|
||||
|
||||
All pages share `base.html`: navy header with the title, a mode pill
|
||||
(VERIFY / LIVE-WRITE from `SAVE_CORRECTIONS`) and a breadcrumb
|
||||
(Dashboard › Claim status › 16 Oct 2026). Light grey background, white
|
||||
cards, blue buttons; Pass green, Fail red, Review amber everywhere.
|
||||
|
||||
1. **Dashboard `/`**
|
||||
- Bot run card — the existing Start / Stop / progress / live log, plus a
|
||||
**Live claims** link opening a side panel with every claim of the
|
||||
running (or last) run: Waiting / Checking / result. A claim with a
|
||||
saved result opens `/day/<date of its run>?claim=<no>`.
|
||||
- Today tiles: claims checked today, Pass, Fail, Review (0 when no run
|
||||
today).
|
||||
- **Find a claim** box → `/claim?claim_no=…`.
|
||||
- Big **Claim status →** button → `/days`.
|
||||
- No claims table.
|
||||
2. **Claim status `/days`** — one card per day (last 90 days with a run),
|
||||
newest first: weekday + date, claims checked, number of runs,
|
||||
Pass / Fail / Review and a thin colour bar. A card opens `/day/<date>`.
|
||||
3. **One day `/day/YYYY-MM-DD`**
|
||||
- Tiles All / Pass / Fail / Review; a tile is a filter link
|
||||
(`?category=Fail`). Claim # search box (`?claim_no=`).
|
||||
- Short table, 10 rows a page (client-side pager): #, Claim #, Time
|
||||
(the run's time), Payer, Result chip, Comments (one line, cut with
|
||||
an ellipsis). No sideways scroll.
|
||||
- Row click → side panel with every Charge Summary field, the run's
|
||||
time and mode, and the conditions/changes JSON.
|
||||
- `?claim=<no>` opens that claim's panel on load (on its page).
|
||||
4. **Claim history `/claim?claim_no=…`** — every saved check of claims
|
||||
matching the number (newest first, up to 200): date, time, mode,
|
||||
result, comments; each row opens `/day/<date>?claim=<no>`.
|
||||
|
||||
`/?run_at=<time>` (old links) redirects (303) to `/day/<date of that run>`.
|
||||
`/api/runs`, `/api/results`, `/api/bot/*` stay as they are.
|
||||
|
||||
## Data
|
||||
|
||||
No schema change. `web/repo.py` gains three parameterized SELECTs:
|
||||
|
||||
- `days(limit=90)` — `GROUP BY CAST(run_at AS DATE)`: day, claims,
|
||||
runs (`COUNT(DISTINCT run_at)`), pass / fail / review counts, newest
|
||||
first.
|
||||
- `day_results(day, category, claim_no)` — rows with
|
||||
`run_at >= day AND run_at < day + 1`, ordered by run_at, S.No, id.
|
||||
- `claim_history(claim_no)` — `TOP 200 … WHERE claim_no LIKE ?` ordered
|
||||
by run_at DESC.
|
||||
|
||||
Rules: a run belongs to the day it **started** (every row of a run shares
|
||||
its `run_at`), so a run never splits over two cards. The same claim
|
||||
checked twice in a day shows twice, each with its time; a day's count is
|
||||
checks. "Not found" claims have no saved row (as today) and appear only in
|
||||
the live panel. A legacy driver returning the day as text is coerced to a
|
||||
date, like `run_at` already is.
|
||||
|
||||
## Unchanged
|
||||
|
||||
127.0.0.1 only; Host allowlist; `no-store` and the other headers; DB
|
||||
errors reported by kind; Start / Stop need the page's Origin and token;
|
||||
page runs are VERIFY only. All new pages are read-only. Page text from the
|
||||
DB goes out escaped (Jinja autoescape; the panel's data as `tojson` in an
|
||||
attribute, rendered with `textContent`).
|
||||
|
||||
## Testing
|
||||
|
||||
`tests/test_web_app.py` (fake repo, fake DB connection): each page renders
|
||||
with its data and links; filters and the bad-input 400s; the redirect;
|
||||
the three new SQL statements are parameterized and shaped as above; the
|
||||
hardening, API and Start / Stop cases kept. A browser check of the four
|
||||
pages on fixture data (Playwright, the in-app browser).
|
||||
@@ -0,0 +1,161 @@
|
||||
# Run tracking and Portal Sign-in — design
|
||||
|
||||
Date: 2026-10-22. Agreed with the operator in chat, question by question
|
||||
(Q1–Q31, all "yes" to the recommended answers; Q3 changed to "who" = the
|
||||
Portal User). Terms: `GLOSSARY.md`. Decision record:
|
||||
`docs/adr/0001-run-events-in-files.md`. Jira: EEWA-31 (Stage 1),
|
||||
EEWA-32 (Stage 2), EEWA-33 (Stage 3), each blocking the next.
|
||||
|
||||
## Goal
|
||||
|
||||
See every Run on the Portal: who started it, how long the eCW login took and
|
||||
where the time went, every Run Error, Claim Error and Warning with its time,
|
||||
and which claims a Run never reached. Only signed-in Portal Users see the
|
||||
Portal; every Run records who started (and stopped) it.
|
||||
|
||||
## Decisions
|
||||
|
||||
| # | Decision |
|
||||
|---|---|
|
||||
| Q1 | Three kinds: **Run Error** (the Run stopped), **Claim Error** (one claim not checked), **Warning** (nothing stopped). **Fail** stays a Charge Summary category only. |
|
||||
| Q2/Q28 | Per **eCW Login Attempt**: start, end, duration, 2FA mail wait, result; steps timed: page open, password typed, captcha, 2FA mail requested (the Log In click), mail received, approval clicked, home page. |
|
||||
| Q3/Q8 | "Who" = the **Portal User** who signed in; **Started By** / **Stopped By**. |
|
||||
| Q4 | Events go to `Logs\events_<run>.jsonl` (ADR 0001). |
|
||||
| Q5/Q25 | Old run logs are not imported. Nothing is deleted. |
|
||||
| Q6/Q18/Q19/Q22 | A **Runs** page on the Portal: list + detail, detail refreshes every 5 s while its Run is going. |
|
||||
| Q9 | The Portal stays on 127.0.0.1. |
|
||||
| Q10–Q17, Q20, Q21 | Sign-in (Stage 2, below). |
|
||||
| Q14 | Terminal Runs are allowed; Started By = the Windows user. |
|
||||
| Q15/Q23 | Stage 3: Run Error banner; `started_by` column in `CV_ChargeSummary`; the Excel sheet unchanged. |
|
||||
| Q24 | Disk check at Run start: system drive under 2 GB = Warning, under 500 MB = Run Error, exit 2, before login. |
|
||||
| Q26 | Claim Error = search raised / form not read / never finished loading; Charge Summary still says Review. A claim not found in eCW = Warning. |
|
||||
| Q27 | **Not Reached** = queued claims a Run Error (or Stop) came before. |
|
||||
|
||||
## Stage 1 — Run events and the Runs page (EEWA-31)
|
||||
|
||||
### `core/run_events.py` (new)
|
||||
|
||||
- `start(log_path)` — the events file is the log's name with `run_` →
|
||||
`events_` and `.log` → `.jsonl` (`run_20261022_141530.log` →
|
||||
`events_20261022_141530.jsonl`); the Run's id is `20261022_141530`.
|
||||
Writes `run_start`.
|
||||
- `emit(kind, **fields)` — one JSON line, `{"t": local ISO time to the
|
||||
millisecond, "kind": kind, ...}`, flushed at once. **Never raises**: an
|
||||
event that cannot be written is dropped, the Run carries on.
|
||||
- Helpers: `warning(code, message, **f)`, `run_error(step, message,
|
||||
exit_code, exc=None)` (with `traceback.format_exc()` when an exception is
|
||||
given), `claim_error(claim, step, message)`, `login_step(step, **f)`,
|
||||
`login_attempt_start()` / `login_attempt_end(result, message="")` (the
|
||||
module numbers attempts).
|
||||
- No OTP code, password or eCW username is ever put in an event.
|
||||
|
||||
### Event kinds
|
||||
|
||||
| kind | fields |
|
||||
|---|---|
|
||||
| `run_start` | `source` (portal / terminal), `started_by`, `mode` (VERIFY / LIVE-WRITE), `claim_limit`, `test_sheet` (file name only), `pid`, `log` |
|
||||
| `disk` | `drive`, `free_mb` |
|
||||
| `login_attempt_start` | `attempt` |
|
||||
| `login_step` | `attempt`, `step`: `page_open`, `password`, `captcha_shown`, `captcha_solved`, `captcha_failed`, `twofa_requested`, `mail_received`, `approval_clicked`, `home` |
|
||||
| `login_attempt_end` | `attempt`, `result`: `ok`, `twofa_timeout`, `captcha_failed`, `restricted`, `failed`; `message` |
|
||||
| `queue` | `claims` (list) |
|
||||
| `claim_start` | `n`, `total`, `claim` |
|
||||
| `claim_result` | `n`, `claim`, `category` (Pass / Fail / Review / Not found) |
|
||||
| `claim_error` | `claim`, `step` (`search` / `form` / `load`), `message` |
|
||||
| `warning` | `code` (`not_found`, `fallback`, `dialog`, `cancel_retry`, `form_not_closed`, `disk_low`, `preflight`), `message`, optional `claim` |
|
||||
| `screenshot` | `path` (a failure shot), optional `claim` |
|
||||
| `run_error` | `step` (`config`, `disk`, `guard`, `login`, `restricted`, `twofa_timeout`, `preflight`, `claims`, `unexpected`), `exit_code`, `message`, `traceback` |
|
||||
| `run_end` | `exit_code` |
|
||||
| `stopped` | `stopped_by` (written by the Portal after Stop; Stage 2 fills the name) |
|
||||
|
||||
`started_by`: env `ECW_STARTED_BY` (the Portal sets it, Stage 2), else the
|
||||
Windows user. `source`: as `bot_run.json` has it (`ECW_RUN_SOURCE`).
|
||||
|
||||
### Where the bot emits
|
||||
|
||||
- `flows/export_medcor_ready.py`: `main()` starts the events after
|
||||
`runlog.start()` and writes `run_end` in its `finally`; `_main` writes a
|
||||
`run_error` at every non-zero return and in every `except`, with the
|
||||
traceback; the disk check sits before login next to the CDP check;
|
||||
`_login_with_retry` wraps each `login()` in attempt start / end;
|
||||
`_process_claims` writes `queue`, `claim_start`, `claim_result`,
|
||||
`claim_error` (search raised, `_error`, `_not_evaluated`) and the
|
||||
not-found Warning.
|
||||
- `auth/login.py` / `captcha.py` / `twofactor.py` / `mailbox.py`: the login
|
||||
steps.
|
||||
- `core/fallback.py` `report()`, `core/page_utils.py` `auto_dialog` (the
|
||||
event only — its behaviour is unchanged, see Context.md §10) and the
|
||||
failure `shot()`, `pages/claim_form.py` Cancel retries and form-not-closed,
|
||||
`pages/preflight.py` FAIL: Warnings / screenshots.
|
||||
- `web/bot_runner.py` `stop()`: appends `stopped` to the Run's events file.
|
||||
|
||||
### Runs page
|
||||
|
||||
- `web/runs.py` (new): reads `Logs\events_*.jsonl` into Run summaries and a
|
||||
Run's detail. A Run with no `run_end` is **Running** when `bot_run.json`
|
||||
names it and its pid is alive, **Stopped** when a `stopped` event is
|
||||
there, else **Ended without a record** (killed). Result: Done (exit 0),
|
||||
Run Error (exit and step), Stopped, Running.
|
||||
- `GET /runs` — newest first: start, Started By, Run Source, mode,
|
||||
duration, result, eCW Login Attempts + total login time, Pass / Fail /
|
||||
Review, Claim Errors, Warnings. Filters: from / to date, result.
|
||||
- `GET /runs/{run_id}` — the login attempts table (each step's time, 2FA
|
||||
wait = mail received − 2FA requested), Run Errors (traceback folded),
|
||||
Claim Errors, Warnings, Not Reached claims, the timeline, screenshot
|
||||
links; refreshes every 5 s while Running.
|
||||
- `GET /runs/{run_id}/shot/{n}` — the Run's n-th screenshot event, served
|
||||
only from inside `Screenshots\` and only as `.png`: a saved page's HTML
|
||||
served from the Portal's origin would run eCW's scripts there, so the
|
||||
detail page names where the DOM file is instead of linking it.
|
||||
- A "Runs" link on every page's header.
|
||||
|
||||
### Disk check
|
||||
|
||||
Settings `DISK_WARN_MB=2048`, `DISK_STOP_MB=500`, on the system drive.
|
||||
|
||||
## Stage 2 — Sign-in (EEWA-32)
|
||||
|
||||
- `CV_PortalUser` (username, display name, role, PBKDF2-SHA256 hash + salt
|
||||
+ iterations, must_change, disabled, created / changed) and
|
||||
`CV_PortalSignIn` (time, username, event: sign_in / sign_out /
|
||||
wrong_password / locked / timeout) in `Charge_Validation`.
|
||||
- Roles Admin / Operator / Viewer. Start / Stop: Admin, Operator. Users
|
||||
page and Sign-in history: Admin.
|
||||
- `tools\add_portal_user.py` (password with `getpass`). Passwords ≥ 10
|
||||
characters; change own; Admin reset → must change at next Sign-in;
|
||||
disable, never delete.
|
||||
- 5 wrong passwords → 15 minutes locked. Session cookie (HttpOnly,
|
||||
SameSite=Strict), 30 minutes idle. Every page and `/api` behind it;
|
||||
127.0.0.1 + Origin / token checks stay.
|
||||
- The Portal passes the signed-in user to the child as `ECW_STARTED_BY`;
|
||||
`stop()` writes `stopped_by`. `bot_run.json` keeps `started_by` too, so
|
||||
the Dashboard's run card says "started by …".
|
||||
- As built (`web/users.py`, `web/app.py`): sessions in the Portal's memory
|
||||
(a restart signs everyone out); disabling a user, resetting their
|
||||
password or changing their role ends / updates their open session at
|
||||
once; an unknown username reads and costs the same as a wrong password;
|
||||
the last active Admin cannot be demoted or disabled, nobody disables
|
||||
themselves; every Admin change is a `CV_PortalSignIn` row with `by_user`;
|
||||
every form POST carries the page token and is checked for Origin;
|
||||
`next` after Sign-in never leaves the site.
|
||||
|
||||
## Stage 3 — Alerts and `started_by` (EEWA-33)
|
||||
|
||||
- Dashboard banner when the last Run ended in a Run Error.
|
||||
- `CV_ChargeSummary.started_by` (nullable column, written from the Run's
|
||||
Started By).
|
||||
- As built: the banner also shows for a Run that ended without a record
|
||||
(killed or crashed — no `run_end`), never for Done, Stopped or Running;
|
||||
it names when, Started By, the step, exit code and message, how many
|
||||
claims were Not Reached, and links the Run. `create_tables.sql` adds the
|
||||
column to an existing table (`IF COL_LENGTH ... IS NULL ALTER TABLE`).
|
||||
The bot checks the column before each insert (`results_db.has_column`)
|
||||
and saves the rows without it while it is missing — a table not yet
|
||||
updated never costs a Run its rows. The Portal selects it, shows it as
|
||||
"Started By" in a claim's side panel, and a table without it reads
|
||||
"Results table needs an update — run create_tables.sql" (SQLSTATE 42S22).
|
||||
|
||||
## Out of scope
|
||||
|
||||
Network access to the Portal, email / Teams alerts, importing old logs,
|
||||
deleting old events, editing settings from the Portal.
|
||||
Reference in new issue
Block a user