Initial import: eCW Charge Validation bot and results Portal
Snapshot of the working project as of 2026-10-23 (local 2d4d4fc), one commit with no earlier history: the tracked files were de-identified first (synthetic claim ids and patient data only; see Context.md section 7). Credentials, run logs, screenshots, claim reports and client workbooks are gitignored and not included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
ManikandanandClaude Opus 5.5 committed
commit
ba15a574ab
205 files changed
+56772
No files matched your search
@@ -0,0 +1,910 @@
|
||||
# Payer Group Categorization Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** After the bot exports the Medcor Ready claim report, automatically add a `Group Name` column (Commercial / Medicaid / Medicare / Dentaquest / Program / …) based on each row's `PAYER`, writing the result to a new file so the existing pipeline's file layout is untouched.
|
||||
|
||||
**Architecture:** A new `core/payer_groups.py` does an exact-string lookup against a hand-editable `payer_groups.json`. A new `core/spreadsheet.write_xlsx()` (stdlib only, mirroring the existing reader) writes the categorized rows to `Output\Medcor Ready_<date>.xlsx`. `flows/export_medcor_ready.py` wires the two together as a new, non-fatal step right after the existing claim-id read-back.
|
||||
|
||||
**Tech Stack:** Python 3.13, stdlib only (`zipfile`, `xml.etree`, `json`, `xml.sax.saxutils`) — no new third-party dependencies, matching the project's existing "no openpyxl" constraint.
|
||||
|
||||
**Spec:** [docs/superpowers/specs/2026-09-15-payer-group-categorization-design.md](../specs/2026-09-15-payer-group-categorization-design.md)
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- No new third-party dependencies (project has none beyond `playwright`; `spreadsheet.py`'s whole existence is to avoid `openpyxl`).
|
||||
- Tests are hand-rolled `case_*` functions collected via `globals()` and run by a `main()` — no pytest, matching every existing file under `tests\`.
|
||||
- `Input\Claim Report_<date>.xlsx` must never be modified by this feature — `claim_file.py` and the charge-entry phase depend on its exact column layout.
|
||||
- Payer matching is **exact string, case-sensitive, no fuzzy/substring logic** — an unmapped payer must be flagged, never guessed (see spec's "Unknown payer handling").
|
||||
- Every new step must be non-fatal to the run — matches the existing philosophy in `flows/export_medcor_ready.py` (`_read_back_claim_ids`, `_read_claim_form`).
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Initialize git
|
||||
|
||||
This project currently has no git history at all (verified: `git status` outside a repo). The plan's commit-per-task workflow needs a repo to commit into, so this task creates one and takes a single baseline commit of the existing code before any feature work starts. This is additive only — no existing file is changed.
|
||||
|
||||
**Files:**
|
||||
- Create: `D:\Ecw Process\.git\` (via `git init`)
|
||||
|
||||
**Interfaces:** none — infrastructure only.
|
||||
|
||||
- [ ] **Step 1: Confirm there is no existing git repo**
|
||||
|
||||
Run: `git -C "D:\Ecw Process" status`
|
||||
Expected: `fatal: not a git repository (or any of the parent directories): .git`
|
||||
|
||||
- [ ] **Step 2: Initialize the repository**
|
||||
|
||||
Run: `git -C "D:\Ecw Process" init`
|
||||
Expected: `Initialized empty Git repository in D:/Ecw Process/.git/`
|
||||
|
||||
- [ ] **Step 3: Check `.gitignore` already covers the right things**
|
||||
|
||||
Run: `cat "D:\Ecw Process\.gitignore"`
|
||||
Expected: it already excludes `.env`, `Logs\`, `Screenshots\`, `Output\`, `Input\` (per `README.md`'s Layout section, all four are described as gitignored). If any of those four is missing, add it before continuing — this baseline commit must not include patient data or credentials.
|
||||
|
||||
- [ ] **Step 4: Stage and commit the current codebase as a baseline**
|
||||
|
||||
```bash
|
||||
cd "D:\Ecw Process"
|
||||
git add -A
|
||||
git status
|
||||
```
|
||||
|
||||
Read the `git status` output before committing: confirm nothing under `Input\`, `Output\`, `Logs\`, `Screenshots\`, or `.env` is staged. If any of those appear, fix `.gitignore` and re-run `git add -A`.
|
||||
|
||||
```bash
|
||||
git commit -m "chore: initialize git repository (baseline)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: `write_xlsx()` in `core/spreadsheet.py`
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ecw_bot/core/spreadsheet.py`
|
||||
- Test: `tests/test_spreadsheet_write.py` (new)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `spreadsheet.write_xlsx(path, rows: list[list]) -> None` — `rows` is a header row plus data rows, each a list of values (any type, written as `str()`). Raises on I/O failure. Creates parent directories if needed.
|
||||
- Consumes: nothing new — sits next to the existing `read_rows(path) -> list`.
|
||||
|
||||
- [ ] **Step 1: Write the failing round-trip test**
|
||||
|
||||
Create `tests/test_spreadsheet_write.py`:
|
||||
|
||||
```python
|
||||
"""
|
||||
write_xlsx(): the write side of core/spreadsheet.py.
|
||||
|
||||
python tests\\test_spreadsheet_write.py
|
||||
|
||||
Round-trips through the existing `read_rows()` reader rather than asserting
|
||||
on raw XML — what matters is that a file this writes is a file this project's
|
||||
own reader (and real Excel) can open, not the exact bytes.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import pathlib
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent / "src"))
|
||||
|
||||
from ecw_bot.core import spreadsheet # noqa: E402
|
||||
|
||||
TMP = pathlib.Path(tempfile.mkdtemp())
|
||||
|
||||
|
||||
def case_write_then_read_round_trips():
|
||||
rows = [
|
||||
["CLAIMS#", "PAYER", "Group Name"],
|
||||
["9107000", "Medicaid", "Medicaid"],
|
||||
["9106000", "DentaQuest", "Dentaquest"],
|
||||
]
|
||||
path = TMP / "roundtrip.xlsx"
|
||||
spreadsheet.write_xlsx(path, rows)
|
||||
back = spreadsheet.read_rows(path)
|
||||
assert back == rows, back
|
||||
|
||||
|
||||
def case_special_characters_are_escaped():
|
||||
rows = [["PATIENT", "NOTE"],
|
||||
["Smith & Sons", "A < B > C"],
|
||||
["O'Brien", '"quoted"']]
|
||||
path = TMP / "escaped.xlsx"
|
||||
spreadsheet.write_xlsx(path, rows)
|
||||
back = spreadsheet.read_rows(path)
|
||||
assert back == rows, back
|
||||
|
||||
|
||||
def case_wide_row_uses_correct_column_letters():
|
||||
"""27 columns exercises the single-letter -> AA rollover."""
|
||||
header = [f"col{i}" for i in range(27)]
|
||||
data = [str(i) for i in range(27)]
|
||||
path = TMP / "wide.xlsx"
|
||||
spreadsheet.write_xlsx(path, [header, data])
|
||||
back = spreadsheet.read_rows(path)
|
||||
assert back[1][26] == "26", back[1] # column AA (index 26)
|
||||
assert back[1] == data, back[1]
|
||||
|
||||
|
||||
def case_creates_parent_directory():
|
||||
path = TMP / "nested" / "new_dir" / "file.xlsx"
|
||||
assert not path.parent.exists()
|
||||
spreadsheet.write_xlsx(path, [["A"], ["1"]])
|
||||
assert path.exists()
|
||||
|
||||
|
||||
def case_non_string_values_are_stringified():
|
||||
rows = [["N"], [5], [3.5]]
|
||||
path = TMP / "numbers.xlsx"
|
||||
spreadsheet.write_xlsx(path, rows)
|
||||
back = spreadsheet.read_rows(path)
|
||||
assert back == [["N"], ["5"], ["3.5"]], back
|
||||
|
||||
|
||||
CASES = [v for k, v in sorted(globals().items()) if k.startswith("case_")]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
failures = 0
|
||||
for fn in CASES:
|
||||
name = fn.__name__[5:].replace("_", " ")
|
||||
try:
|
||||
fn()
|
||||
print(f" ok {name}")
|
||||
except AssertionError as exc:
|
||||
print(f" FAIL {name}: {exc}")
|
||||
failures += 1
|
||||
except Exception as exc:
|
||||
print(f" ERROR {name}: {type(exc).__name__}: {exc}")
|
||||
failures += 1
|
||||
print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run it to verify it fails**
|
||||
|
||||
Run: `python tests\test_spreadsheet_write.py`
|
||||
Expected: `ERROR` on every case with `AttributeError: module 'ecw_bot.core.spreadsheet' has no attribute 'write_xlsx'`
|
||||
|
||||
- [ ] **Step 3: Implement `write_xlsx()` in `core/spreadsheet.py`**
|
||||
|
||||
Add these imports at the top of `src/ecw_bot/core/spreadsheet.py` (alongside the existing `import csv / re / zipfile` block):
|
||||
|
||||
```python
|
||||
from xml.sax.saxutils import escape
|
||||
```
|
||||
|
||||
Append this to the end of `src/ecw_bot/core/spreadsheet.py`:
|
||||
|
||||
```python
|
||||
# =============================================================================
|
||||
# Writing (the categorization step's output)
|
||||
# =============================================================================
|
||||
# The read side above has no dependency on openpyxl; the write side keeps that
|
||||
# constraint too. What follows is the minimal set of OOXML parts real Excel
|
||||
# actually requires to open a file without a "repair" prompt — more than the
|
||||
# reader above needs (it only ever looks at xl/worksheets/sheet*.xml), because
|
||||
# this file is meant to be opened by a person, not just this codebase.
|
||||
|
||||
_CONTENT_TYPES = (
|
||||
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
|
||||
'<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
|
||||
'<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>'
|
||||
'<Default Extension="xml" ContentType="application/xml"/>'
|
||||
'<Override PartName="/xl/workbook.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml"/>'
|
||||
'<Override PartName="/xl/worksheets/sheet1.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml"/>'
|
||||
'</Types>'
|
||||
)
|
||||
|
||||
_ROOT_RELS = (
|
||||
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
|
||||
'<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
|
||||
'<Relationship Id="rId1" '
|
||||
'Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" '
|
||||
'Target="xl/workbook.xml"/>'
|
||||
'</Relationships>'
|
||||
)
|
||||
|
||||
_WORKBOOK = (
|
||||
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
|
||||
'<workbook xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main" '
|
||||
'xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">'
|
||||
'<sheets><sheet name="Sheet1" sheetId="1" r:id="rId1"/></sheets>'
|
||||
'</workbook>'
|
||||
)
|
||||
|
||||
_WORKBOOK_RELS = (
|
||||
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
|
||||
'<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
|
||||
'<Relationship Id="rId1" '
|
||||
'Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" '
|
||||
'Target="worksheets/sheet1.xml"/>'
|
||||
'</Relationships>'
|
||||
)
|
||||
|
||||
|
||||
def _col_letter(idx: int) -> str:
|
||||
"""0 -> 'A', 25 -> 'Z', 26 -> 'AA'. Inverse of `_col_index` above."""
|
||||
letters = ""
|
||||
idx += 1
|
||||
while idx > 0:
|
||||
idx, rem = divmod(idx - 1, 26)
|
||||
letters = chr(65 + rem) + letters
|
||||
return letters
|
||||
|
||||
|
||||
def write_xlsx(path, rows: list) -> None:
|
||||
"""
|
||||
Write `rows` (a header row plus data rows, each a list of values) as a
|
||||
minimal single-sheet xlsx.
|
||||
|
||||
Inline `t="str"` cells, no sharedStrings — the same shape `_read_xlsx`
|
||||
above already parses, and what eCW's own export already sends (see the
|
||||
module docstring). Every value is written through `str()`, so callers can
|
||||
pass numbers or None without pre-formatting.
|
||||
"""
|
||||
path = Path(path)
|
||||
body = []
|
||||
for r, row in enumerate(rows, start=1):
|
||||
cells = []
|
||||
for c, value in enumerate(row):
|
||||
text = escape(str(value))
|
||||
cells.append(f'<c r="{_col_letter(c)}{r}" t="str"><v>{text}</v></c>')
|
||||
body.append(f'<row r="{r}">{"".join(cells)}</row>')
|
||||
sheet = (
|
||||
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
|
||||
'<worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main">'
|
||||
f'<sheetData>{"".join(body)}</sheetData></worksheet>'
|
||||
)
|
||||
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
zf.writestr("[Content_Types].xml", _CONTENT_TYPES)
|
||||
zf.writestr("_rels/.rels", _ROOT_RELS)
|
||||
zf.writestr("xl/workbook.xml", _WORKBOOK)
|
||||
zf.writestr("xl/_rels/workbook.xml.rels", _WORKBOOK_RELS)
|
||||
zf.writestr("xl/worksheets/sheet1.xml", sheet)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the test to verify it passes**
|
||||
|
||||
Run: `python tests\test_spreadsheet_write.py`
|
||||
Expected: `5/5 passed`
|
||||
|
||||
- [ ] **Step 5: Sanity-check the file actually opens in Excel/WPS**
|
||||
|
||||
This machine has Excel/WPS installed (seen in the video). Manually:
|
||||
|
||||
```bash
|
||||
python -c "import sys; sys.path.insert(0,'src'); from ecw_bot.core import spreadsheet as s; s.write_xlsx('Output/_manual_check.xlsx', [['A','B'],['1','2']])"
|
||||
```
|
||||
|
||||
Open `Output\_manual_check.xlsx` by hand and confirm it opens with no "repair" prompt and shows the two cells. Delete the file afterward — it's a throwaway check, not a fixture.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
cd "D:\Ecw Process"
|
||||
git add src/ecw_bot/core/spreadsheet.py tests/test_spreadsheet_write.py
|
||||
git commit -m "feat: add write_xlsx() to core/spreadsheet.py"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: `core/payer_groups.py` + seed `payer_groups.json`
|
||||
|
||||
**Files:**
|
||||
- Create: `src/ecw_bot/core/payer_groups.py`
|
||||
- Create: `src/ecw_bot/payer_groups.json`
|
||||
- Test: `tests/test_payer_groups.py` (new)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `payer_groups.load_mapping(path=None) -> dict[str, str]`, `payer_groups.classify(payer: str, mapping: dict) -> str | None`.
|
||||
- Consumes: nothing from Task 2.
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Create `tests/test_payer_groups.py`:
|
||||
|
||||
```python
|
||||
"""
|
||||
Payer -> Group Name lookup.
|
||||
|
||||
python tests\\test_payer_groups.py
|
||||
|
||||
Exact-string match only, on purpose: eCW's own PAYER values already differ by
|
||||
case and spacing for what is really the same carrier (`United Healthcare` /
|
||||
`United Health Care` / `United healthcare` all appear in one real export), so
|
||||
"close enough" matching would have to guess which of several real, distinct
|
||||
strings a new one is closest to. Billing data does not get a guess — an
|
||||
unrecognized string must come back None and get a human's attention.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import json
|
||||
import pathlib
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent / "src"))
|
||||
|
||||
from ecw_bot.core import payer_groups # noqa: E402
|
||||
|
||||
TMP = pathlib.Path(tempfile.mkdtemp())
|
||||
|
||||
MAPPING = {
|
||||
"Medicaid": "Medicaid",
|
||||
"DentaQuest": "Dentaquest",
|
||||
"Anthem Blue Cross Blue Shield": "Commercial",
|
||||
}
|
||||
|
||||
|
||||
def case_exact_match_returns_group():
|
||||
assert payer_groups.classify("Medicaid", MAPPING) == "Medicaid"
|
||||
assert payer_groups.classify("DentaQuest", MAPPING) == "Dentaquest"
|
||||
|
||||
|
||||
def case_unmapped_payer_returns_none():
|
||||
assert payer_groups.classify("Some New Payer LLC", MAPPING) is None
|
||||
|
||||
|
||||
def case_case_sensitive_no_fuzzy_matching():
|
||||
"""Different case is a different string, on purpose — see module docstring."""
|
||||
assert payer_groups.classify("medicaid", MAPPING) is None
|
||||
assert payer_groups.classify("MEDICAID", MAPPING) is None
|
||||
|
||||
|
||||
def case_blank_or_none_payer_is_unmapped():
|
||||
assert payer_groups.classify("", MAPPING) is None
|
||||
assert payer_groups.classify(None, MAPPING) is None
|
||||
|
||||
|
||||
def case_load_mapping_reads_json_file():
|
||||
path = TMP / "mapping.json"
|
||||
path.write_text(json.dumps({"Medicaid": "Medicaid"}), encoding="utf-8")
|
||||
loaded = payer_groups.load_mapping(path)
|
||||
assert loaded == {"Medicaid": "Medicaid"}, loaded
|
||||
|
||||
|
||||
def case_load_mapping_missing_file_raises():
|
||||
try:
|
||||
payer_groups.load_mapping(TMP / "does_not_exist.json")
|
||||
except FileNotFoundError as exc:
|
||||
assert "does_not_exist.json" in str(exc), exc
|
||||
return
|
||||
raise AssertionError("expected FileNotFoundError")
|
||||
|
||||
|
||||
def case_seed_config_loads_and_covers_medicaid_and_dentaquest():
|
||||
"""The real config file this ships with, not a fixture."""
|
||||
mapping = payer_groups.load_mapping()
|
||||
assert mapping.get("Medicaid") == "Medicaid", mapping.get("Medicaid")
|
||||
assert mapping.get("DentaQuest") == "Dentaquest", mapping.get("DentaQuest")
|
||||
|
||||
|
||||
CASES = [v for k, v in sorted(globals().items()) if k.startswith("case_")]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
failures = 0
|
||||
for fn in CASES:
|
||||
name = fn.__name__[5:].replace("_", " ")
|
||||
try:
|
||||
fn()
|
||||
print(f" ok {name}")
|
||||
except AssertionError as exc:
|
||||
print(f" FAIL {name}: {exc}")
|
||||
failures += 1
|
||||
except Exception as exc:
|
||||
print(f" ERROR {name}: {type(exc).__name__}: {exc}")
|
||||
failures += 1
|
||||
print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run it to verify it fails**
|
||||
|
||||
Run: `python tests\test_payer_groups.py`
|
||||
Expected: `ERROR` on every case — `ModuleNotFoundError: No module named 'ecw_bot.core.payer_groups'`
|
||||
|
||||
- [ ] **Step 3: Create the seed config, `src/ecw_bot/payer_groups.json`**
|
||||
|
||||
```json
|
||||
{
|
||||
"Medicaid": "Medicaid",
|
||||
"DentaQuest": "Dentaquest",
|
||||
"CHP+ DENTAQUEST": "Dentaquest",
|
||||
"AARP Medicare Advantage": "Medicare",
|
||||
"Aetna Medicare ADV": "Medicare",
|
||||
"OUTREACH PROGRAM": "Program",
|
||||
"Anthem Blue Cross Blue Shield": "Commercial",
|
||||
"Cigna HealthCare Of Colorado": "Commercial",
|
||||
"Cigna Healthcare of CO": "Commercial",
|
||||
"Cigna Salud Employee": "Commercial",
|
||||
"Cigna": "Commercial",
|
||||
"Cypress Benefit Admin / Cigna PPO": "Commercial",
|
||||
"United Healthcare": "Commercial",
|
||||
"United Health Care": "Commercial",
|
||||
"United healthcare": "Commercial",
|
||||
"Unitedhealth One": "Commercial",
|
||||
"Oxford Health Insurance UHC": "Commercial",
|
||||
"UMR": "Commercial",
|
||||
"Aetna": "Commercial",
|
||||
"Aetna Alaska Care": "Commercial",
|
||||
"AETNA HEALTH AND LIFE INSURANCE": "Commercial",
|
||||
"BIND BENEFITS INC": "Commercial",
|
||||
"Select Health": "Commercial"
|
||||
}
|
||||
```
|
||||
|
||||
This is the "Mapped (seeded)" table from the spec, byte-for-byte. The 17 payers in the spec's "Left unmapped" list are deliberately absent — `classify()` returning `None` for them, and the flow logging/flagging that, is the intended behavior until a human adds them.
|
||||
|
||||
- [ ] **Step 4: Implement `core/payer_groups.py`**
|
||||
|
||||
```python
|
||||
"""
|
||||
Payer -> Group Name lookup for the categorization step.
|
||||
|
||||
Exact-string match against a small hand-edited table (`payer_groups.json`,
|
||||
next to this package) — no fuzzy or substring matching. eCW's own PAYER
|
||||
values already differ by case and spacing for what is really the same
|
||||
carrier (see `tests/test_payer_groups.py`'s docstring), so "close enough"
|
||||
matching has no safe way to pick which known string a new one is closest to.
|
||||
An unrecognized payer must come back as None and get a human's attention —
|
||||
never a guess, on billing data.
|
||||
|
||||
Adding a payer is a one-line edit to `payer_groups.json`, no code change.
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
_CONFIG_PATH = Path(__file__).resolve().parent.parent / "payer_groups.json"
|
||||
|
||||
|
||||
def load_mapping(path=None) -> dict:
|
||||
"""The payer -> group table. `path` defaults to the shipped config."""
|
||||
p = Path(path) if path else _CONFIG_PATH
|
||||
if not p.exists():
|
||||
raise FileNotFoundError(
|
||||
f"{p} not found — the payer group table is missing")
|
||||
return json.loads(p.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def classify(payer: str, mapping: dict) -> str:
|
||||
"""The Group Name for this exact PAYER string, or None if unmapped."""
|
||||
return mapping.get((payer or "").strip())
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run the tests to verify they pass**
|
||||
|
||||
Run: `python tests\test_payer_groups.py`
|
||||
Expected: `7/7 passed`
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
cd "D:\Ecw Process"
|
||||
git add src/ecw_bot/core/payer_groups.py src/ecw_bot/payer_groups.json tests/test_payer_groups.py
|
||||
git commit -m "feat: add payer_groups lookup module and seed config"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Wire the categorization step into the export flow
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ecw_bot/settings.py`
|
||||
- Modify: `.env.example`
|
||||
- Modify: `src/ecw_bot/flows/export_medcor_ready.py`
|
||||
- Test: `tests/test_categorize_by_payer.py` (new)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `spreadsheet.read_rows`, `spreadsheet.write_xlsx` (Task 2); `payer_groups.load_mapping`, `payer_groups.classify` (Task 3).
|
||||
- Produces: `export_medcor_ready._categorize_by_payer(saved: Path | None) -> Path | None` — writes `Output\Medcor Ready_<date>.xlsx` and returns its path, or `None` if the step could not run (never raises).
|
||||
|
||||
- [ ] **Step 1: Add the new setting**
|
||||
|
||||
In `src/ecw_bot/settings.py`, immediately after the `SHOW_CLAIM_IDS` setting (currently ending around line 311, just before the `# ── Per-claim processing ──` section header), add:
|
||||
|
||||
```python
|
||||
# After categorizing by payer, tell the operator about any PAYER value with
|
||||
# no entry in payer_groups.json — those rows are saved with a blank Group
|
||||
# Name rather than a guess. Informational only, same reasoning as
|
||||
# NOTIFY_AFTER_EXPORT above. Set 0 for unattended runs; the full list still
|
||||
# goes to the log either way.
|
||||
NOTIFY_UNMAPPED_PAYERS = _env_bool("NOTIFY_UNMAPPED_PAYERS", True)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add the matching line to `.env.example`**
|
||||
|
||||
Open `.env.example`, find the block documenting `SHOW_CLAIM_IDS` (search for that key), and add directly after it:
|
||||
|
||||
```
|
||||
# After categorizing the export by payer, show a box listing any PAYER value
|
||||
# with no entry in src/ecw_bot/payer_groups.json (those rows get a blank
|
||||
# Group Name, not a guess). 1 = show it, 0 = log only.
|
||||
NOTIFY_UNMAPPED_PAYERS=1
|
||||
```
|
||||
|
||||
Match whatever `1`/`0` convention the surrounding boolean settings in that file already use — check `SHOW_CLAIM_IDS`'s line right above it and mirror its exact style so the two files can't drift the way the earlier `OTP_WAIT_SECONDS` mismatch did.
|
||||
|
||||
- [ ] **Step 3: Write the failing test for `_categorize_by_payer`**
|
||||
|
||||
Create `tests/test_categorize_by_payer.py`:
|
||||
|
||||
```python
|
||||
"""
|
||||
export_medcor_ready._categorize_by_payer(): the payer -> Group Name step.
|
||||
|
||||
python tests\\test_categorize_by_payer.py
|
||||
|
||||
Builds a real xlsx (same helper shape as test_claim_file.py's `xlsx()`) so
|
||||
this exercises the actual read -> classify -> write path, not a mock of it.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import zipfile
|
||||
import pathlib
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent / "src"))
|
||||
|
||||
from ecw_bot.flows import export_medcor_ready as flow # noqa: E402
|
||||
from ecw_bot.core import spreadsheet # noqa: E402
|
||||
from ecw_bot import settings as cfg # noqa: E402
|
||||
|
||||
TMP = pathlib.Path(tempfile.mkdtemp())
|
||||
NS = 'xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"'
|
||||
|
||||
|
||||
def xlsx(header, data_rows, name="t.xlsx") -> pathlib.Path:
|
||||
"""A minimal real xlsx: header + data rows, both plain string lists."""
|
||||
rows = [header] + data_rows
|
||||
body = []
|
||||
for r, row in enumerate(rows, start=1):
|
||||
cells = "".join(
|
||||
f'<c r="{chr(65 + c)}{r}" t="str"><v>{v}</v></c>'
|
||||
for c, v in enumerate(row))
|
||||
body.append(f'<row r="{r}">{cells}</row>')
|
||||
sheet = (f'<?xml version="1.0"?><worksheet {NS}><sheetData>'
|
||||
f'{"".join(body)}</sheetData></worksheet>')
|
||||
path = TMP / name
|
||||
with zipfile.ZipFile(path, "w") as z:
|
||||
z.writestr("xl/worksheets/sheet1.xml", sheet)
|
||||
return path
|
||||
|
||||
|
||||
HEADER = ["CLAIMS#", "PAYER", "CHARGES"]
|
||||
|
||||
|
||||
def case_categorizes_known_payers():
|
||||
saved = xlsx(HEADER, [
|
||||
["9107000", "Medicaid", "275.00"],
|
||||
["9106000", "DentaQuest", "117.00"],
|
||||
], name="known.xlsx")
|
||||
out = flow._categorize_by_payer(saved)
|
||||
assert out is not None and out.exists(), out
|
||||
rows = spreadsheet.read_rows(out)
|
||||
assert rows[0] == HEADER + ["Group Name"], rows[0]
|
||||
assert rows[1] == ["9107000", "Medicaid", "275.00", "Medicaid"], rows[1]
|
||||
assert rows[2] == ["9106000", "DentaQuest", "117.00", "Dentaquest"], rows[2]
|
||||
|
||||
|
||||
def case_unmapped_payer_gets_blank_group_not_a_guess():
|
||||
saved = xlsx(HEADER, [
|
||||
["9108000", "Some Brand New Payer", "50.00"],
|
||||
], name="unmapped.xlsx")
|
||||
out = flow._categorize_by_payer(saved)
|
||||
rows = spreadsheet.read_rows(out)
|
||||
assert rows[1][-1] == "", rows[1] # blank, not a guessed group
|
||||
|
||||
|
||||
def case_none_input_is_a_no_op():
|
||||
assert flow._categorize_by_payer(None) is None
|
||||
|
||||
|
||||
def case_missing_payer_column_does_not_raise():
|
||||
saved = xlsx(["CLAIMS#", "CHARGES"], [["9108000", "50.00"]],
|
||||
name="nopayer.xlsx")
|
||||
assert flow._categorize_by_payer(saved) is None # skipped, not a crash
|
||||
|
||||
|
||||
CASES = [v for k, v in sorted(globals().items()) if k.startswith("case_")]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
failures = 0
|
||||
for fn in CASES:
|
||||
name = fn.__name__[5:].replace("_", " ")
|
||||
try:
|
||||
fn()
|
||||
print(f" ok {name}")
|
||||
except AssertionError as exc:
|
||||
print(f" FAIL {name}: {exc}")
|
||||
failures += 1
|
||||
except Exception as exc:
|
||||
print(f" ERROR {name}: {type(exc).__name__}: {exc}")
|
||||
failures += 1
|
||||
print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run it to verify it fails**
|
||||
|
||||
Run: `python tests\test_categorize_by_payer.py`
|
||||
Expected: `ERROR` on every case — `AttributeError: module 'ecw_bot.flows.export_medcor_ready' has no attribute '_categorize_by_payer'`
|
||||
|
||||
- [ ] **Step 5: Implement `_categorize_by_payer` in `flows/export_medcor_ready.py`**
|
||||
|
||||
Change the import line near the top (currently `from ..core import guard, runlog, msgbox`) to:
|
||||
|
||||
```python
|
||||
from ..core import guard, runlog, msgbox, spreadsheet, payer_groups
|
||||
```
|
||||
|
||||
Add this new function directly after `_read_back_claim_ids` (which ends at the `return info` around line 85), before `_read_claim_form`:
|
||||
|
||||
```python
|
||||
def _categorize_by_payer(saved):
|
||||
"""
|
||||
Add a Group Name column (Commercial / Medicaid / Medicare / Dentaquest /
|
||||
Program / ...) next to the saved report, based on each row's PAYER, and
|
||||
write it to Output\\Medcor Ready_<date>.xlsx.
|
||||
|
||||
Never touches `saved` itself: claim_file.py and everything after it reads
|
||||
that file's columns by position (see claim_file.py's module docstring),
|
||||
so this always writes a SEPARATE file rather than editing it in place.
|
||||
|
||||
Never fatal. An unmapped payer, a missing PAYER column, or a write
|
||||
failure is worth reporting loudly, not worth losing the already-saved
|
||||
report over — same reasoning as `_read_back_claim_ids` above.
|
||||
"""
|
||||
if saved is None:
|
||||
return None
|
||||
try:
|
||||
rows = spreadsheet.read_rows(saved)
|
||||
except Exception as exc:
|
||||
print(f"[CATEGORIZE] Could not read {saved.name}: {exc}")
|
||||
return None
|
||||
if not rows:
|
||||
print(f"[CATEGORIZE] {saved.name} has no rows — skipping")
|
||||
return None
|
||||
|
||||
header, data = rows[0], rows[1:]
|
||||
if "PAYER" not in header:
|
||||
print(f"[CATEGORIZE] No PAYER column in {saved.name} — skipping. "
|
||||
f"Header reads: {header}")
|
||||
return None
|
||||
payer_col = header.index("PAYER")
|
||||
|
||||
try:
|
||||
mapping = payer_groups.load_mapping()
|
||||
except Exception as exc:
|
||||
print(f"[CATEGORIZE] Could not load payer_groups.json: {exc}")
|
||||
return None
|
||||
|
||||
out_rows = [header + ["Group Name"]]
|
||||
unmapped = {}
|
||||
mapped_count = 0
|
||||
for row in data:
|
||||
payer = row[payer_col] if payer_col < len(row) else ""
|
||||
group = payer_groups.classify(payer, mapping)
|
||||
if group:
|
||||
mapped_count += 1
|
||||
else:
|
||||
unmapped[payer] = unmapped.get(payer, 0) + 1
|
||||
out_rows.append(list(row) + [group or ""])
|
||||
|
||||
out_path = cfg.ROOT / "Output" / f"Medcor Ready_{cfg.TODAY}.xlsx"
|
||||
try:
|
||||
spreadsheet.write_xlsx(out_path, out_rows)
|
||||
except Exception as exc:
|
||||
print(f"[CATEGORIZE] Could not write {out_path.name}: {exc}")
|
||||
return None
|
||||
|
||||
print(f"[CATEGORIZE] {mapped_count} of {len(data)} row(s) categorized "
|
||||
f"-> {out_path.name}")
|
||||
if unmapped:
|
||||
listed = sorted(unmapped.items(), key=lambda kv: -kv[1])
|
||||
print(f"[CATEGORIZE] {sum(unmapped.values())} row(s) with no group "
|
||||
f"in payer_groups.json:")
|
||||
for payer, count in listed:
|
||||
print(f"[CATEGORIZE] {count:4d} {payer!r}")
|
||||
if cfg.NOTIFY_UNMAPPED_PAYERS:
|
||||
body = (f"{sum(unmapped.values())} row(s)-ku Group Name "
|
||||
f"kidaikkala.\n\n"
|
||||
f"{out_path.name} file-la '' (blank) Group Name-oda "
|
||||
f"irukkum. Idha payer_groups.json-la add pannunga:\n\n"
|
||||
+ "\n".join(f" {p} ({n} row)" for p, n in listed[:15]))
|
||||
if len(listed) > 15:
|
||||
body += f"\n ... innum {len(listed) - 15}"
|
||||
msgbox.info_box("eCW Bot — Group Name podala", body)
|
||||
|
||||
return out_path
|
||||
```
|
||||
|
||||
Add the call into `_claims_phase` — change:
|
||||
|
||||
```python
|
||||
saved = claim_report.export_claims_report(page)
|
||||
info = _read_back_claim_ids(saved)
|
||||
if info:
|
||||
_process_claims(page, info)
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```python
|
||||
saved = claim_report.export_claims_report(page)
|
||||
info = _read_back_claim_ids(saved)
|
||||
_categorize_by_payer(saved)
|
||||
if info:
|
||||
_process_claims(page, info)
|
||||
```
|
||||
|
||||
Finally, update the module's own flow docstring at the top of the file — change:
|
||||
|
||||
```
|
||||
→ Input\Claim Report_<MM-DD-YYYY>.xlsx
|
||||
→ read the file back, pull the CLAIMS# column
|
||||
→ search each claim id on its own (CLAIM_PROCESS_LIMIT)
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```
|
||||
→ Input\Claim Report_<MM-DD-YYYY>.xlsx
|
||||
→ read the file back, pull the CLAIMS# column
|
||||
→ categorize by payer → Output\Medcor Ready_<MM-DD-YYYY>.xlsx
|
||||
→ search each claim id on its own (CLAIM_PROCESS_LIMIT)
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the test to verify it passes**
|
||||
|
||||
Run: `python tests\test_categorize_by_payer.py`
|
||||
Expected: `4/4 passed`
|
||||
|
||||
- [ ] **Step 7: Run the full existing test suite to confirm nothing else broke**
|
||||
|
||||
```bash
|
||||
python tests\test_settings.py
|
||||
python tests\test_guard.py
|
||||
python tests\test_dead_end.py
|
||||
python tests\test_claim_file.py
|
||||
python tests\test_claim_form.py
|
||||
python tests\test_progress_note.py
|
||||
python tests\test_input_archive.py
|
||||
python tests\test_spreadsheet_write.py
|
||||
python tests\test_payer_groups.py
|
||||
python tests\test_categorize_by_payer.py
|
||||
```
|
||||
|
||||
Expected: every one reports all cases passed. `test_settings.py` in particular asserts the module graph stays acyclic and that no selector leaks into `settings` — confirm it still passes since this task touched `settings.py`.
|
||||
|
||||
- [ ] **Step 8: Commit**
|
||||
|
||||
```bash
|
||||
cd "D:\Ecw Process"
|
||||
git add src/ecw_bot/settings.py .env.example src/ecw_bot/flows/export_medcor_ready.py tests/test_categorize_by_payer.py
|
||||
git commit -m "feat: categorize Medcor Ready export by payer after each run"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Document it
|
||||
|
||||
**Files:**
|
||||
- Modify: `README.md`
|
||||
- Modify: `CHANGELOG.md`
|
||||
|
||||
**Interfaces:** none — documentation only.
|
||||
|
||||
- [ ] **Step 1: Add a short section to `README.md`**
|
||||
|
||||
Find the "### Reading the claim ids back" section (the last subsection under "## Claim lookup", ending just before "## Known gaps"). Add a new subsection directly after it:
|
||||
|
||||
```markdown
|
||||
### Categorizing by payer
|
||||
|
||||
Right after the claim ids are read back, every row of the just-saved report
|
||||
gets a `Group Name` (`Commercial`, `Medicaid`, `Medicare`, `Dentaquest`,
|
||||
`Program`, ...) based on its `PAYER`, and the result is saved to
|
||||
`Output\Medcor Ready_<MM-DD-YYYY>.xlsx`. The original `Input\` file is never
|
||||
touched — `claim_file.py` and the charge-entry phase depend on its exact
|
||||
column layout.
|
||||
|
||||
The mapping is `src\ecw_bot\payer_groups.json` — a flat, hand-editable
|
||||
`{"exact PAYER string": "Group Name"}` table. **Exact match only, no fuzzy
|
||||
matching**: eCW's own PAYER values already differ by case and spacing for
|
||||
what is really the same carrier (`United Healthcare` / `United Health Care`
|
||||
/ `United healthcare` are three separate real values), so a PAYER string
|
||||
with no entry gets a **blank** Group Name rather than a guess. Unmapped rows
|
||||
are logged and, unless `NOTIFY_UNMAPPED_PAYERS=0`, shown in a message box —
|
||||
adding one is a one-line edit to the JSON file, no code change.
|
||||
|
||||
Never fatal: a missing PAYER column, an unreadable report, or a write
|
||||
failure is logged and the run continues — the export already succeeded by
|
||||
the time this step runs.
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add a CHANGELOG entry**
|
||||
|
||||
Append to the end of `CHANGELOG.md`:
|
||||
|
||||
```markdown
|
||||
|
||||
---
|
||||
|
||||
## Day 4 — 2026-09-15
|
||||
|
||||
**Automated the payer categorization step.** Reviewed against
|
||||
`Salud-Charge Entry_Medcor Ready.mp4` (2:45-10:00): after the Medcor Ready
|
||||
export, an operator was filtering the report by `PAYER` and hand-filling a
|
||||
`Group Name` column (`Commercial`, `Medicaid`, `Medicare`, `Dentaquest`,
|
||||
`Program`, ...) — the same manual, repetitive, rule-based step every other
|
||||
phase of this bot exists to remove.
|
||||
|
||||
`core/payer_groups.py` does the lookup: exact string match against
|
||||
`payer_groups.json`, on purpose — eCW's own PAYER values already differ by
|
||||
case and spacing for what is really the same carrier (`United Healthcare` /
|
||||
`United Health Care` / `United healthcare` are three separate values in one
|
||||
real export), so anything looser has no safe way to pick which known string
|
||||
a new one is closest to. An unrecognized payer gets a **blank** Group Name,
|
||||
not a guess, and is logged and message-boxed — this is billing data, and
|
||||
wrong is worse than blank.
|
||||
|
||||
`core/spreadsheet.py` gained a write side (`write_xlsx()`) to go with its
|
||||
read side, built the same way — stdlib only, no openpyxl — but with the full
|
||||
minimal OOXML part set (`[Content_Types].xml`, both `.rels` files,
|
||||
`workbook.xml`) the reader doesn't need but real Excel does, since this file
|
||||
is meant to be opened by a person.
|
||||
|
||||
The categorized output is a **new** file, `Output\Medcor Ready_<date>.xlsx`
|
||||
— `Input\Claim Report_<date>.xlsx` is never modified, because `claim_file.py`
|
||||
and the charge-entry phase read its columns by position.
|
||||
|
||||
**Seeded, not complete.** The mapping table ships with 23 payers covering
|
||||
roughly 95% of a real export's row volume (`Medicaid`, `DentaQuest`, and the
|
||||
clearly-commercial carriers), derived from the actual current export rather
|
||||
than guessed from the video. 17 payers (`BH COLORADO ACCESS`, the Delta
|
||||
Dental variants, `Tricare`, `Devoted`, several TPA names, ...) are
|
||||
deliberately left out — first run will flag them, and filling them in is a
|
||||
one-line JSON edit, no redeploy.
|
||||
|
||||
**Out of scope for this round:** splitting rows into the `Patient` /
|
||||
`Pending with Error` / `Program` sibling workbooks the video also shows, and
|
||||
writing anything to the network `Charge\Allocation\...` folder — both stay
|
||||
manual for now.
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
cd "D:\Ecw Process"
|
||||
git add README.md CHANGELOG.md
|
||||
git commit -m "docs: document the payer categorization step"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-review notes
|
||||
|
||||
- **Spec coverage:** architecture (Task 2-4), components (`payer_groups.py` Task 3, `write_xlsx` Task 2, flow wiring Task 4), data flow (Task 4 Step 5), error handling (unmapped payer / missing file / write failure, all in Task 4 Step 5 and covered by tests), testing (every task has its own `test_*.py`), seed mapping table (Task 3 Step 3, copied verbatim from the spec). All covered.
|
||||
- **Placeholder scan:** no TBD/TODO; every step has real, complete code.
|
||||
- **Type consistency:** `classify(payer, mapping)` signature is identical everywhere it's called (Task 3 tests, Task 4's `_categorize_by_payer`); `load_mapping(path=None)` likewise.
|
||||
- **Scope:** single subsystem (categorization only), matches the spec's explicit out-of-scope list — no split-into-3-files logic anywhere in this plan.
|
||||
@@ -0,0 +1,108 @@
|
||||
# Results DB + Web Page Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Store every run's Charge Summary rows in SQL Server (`Charge_Validation.dbo.ChargeSummary`) and show them on a local FastAPI page.
|
||||
|
||||
**Architecture:** A pure module `core/results_db.py` turns Charge Summary rows (the same rows the Excel file gets) into typed DB records and inserts them with one parameterized statement; the flow calls it after the Excel write, behind `RESULTS_DB`, and any DB error is a log line only. A separate `web/` FastAPI app reads the table through a small repository class and renders one Jinja2 page plus two JSON endpoints, bound to 127.0.0.1.
|
||||
|
||||
**Tech Stack:** Python 3.13, pyodbc + ODBC Driver 18 for SQL Server, FastAPI 0.141 + Jinja2 + uvicorn, Starlette TestClient (httpx). All already installed.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-10-08-results-db-and-web-page-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Server `MECARE331\MECARE331`, database `Charge_Validation`, table `dbo.ChargeSummary`.
|
||||
- Default connection: `DRIVER={ODBC Driver 18 for SQL Server};SERVER=MECARE331\MECARE331;DATABASE=Charge_Validation;Trusted_Connection=yes;TrustServerCertificate=yes;`
|
||||
- `RESULTS_DB` default **off**. The bot never creates/alters tables; the operator runs `tools/sql/create_tables.sql` in SSMS.
|
||||
- A DB failure never stops the run and never touches the Excel summary.
|
||||
- Only parameterized SQL; web is read-only (`SELECT` only).
|
||||
- Web bound to `127.0.0.1` only. No external scripts/CDNs.
|
||||
- No new entries in `requirements.txt`. Tests are standalone scripts (`python tests\test_x.py`, prints `N/M passed`), no pytest, no real DB.
|
||||
- Never commit unless the operator asks; never commit `run.py`, `Input/`, `Output/`, `Logs/`, `Screenshots/`, `Dom.txt`, client `.xlsx`.
|
||||
|
||||
## Rulings (plan vs spec)
|
||||
|
||||
- Ruling: the flow builds the sheet once more for the DB (`charge_summary.build`) instead of threading one sheet through `write()` — keeps `write()` and its tests unchanged; both builds read the same report and outcomes on the same day, so the rows are identical — cost if wrong: a row differing between Excel and DB, visible on the page.
|
||||
- Ruling: only rows the run filled in (non-blank Category) are saved — unprocessed / not-found rows are blank in the Excel too and would only clutter the page — cost if wrong: the page lacks blank rows the Excel has.
|
||||
|
||||
## File map
|
||||
|
||||
| File | Responsibility |
|
||||
|---|---|
|
||||
| `tools/sql/create_tables.sql` | Table + indexes, run by the operator |
|
||||
| `src/ecw_bot/settings.py` | `RESULTS_DB`, `RESULTS_DB_CONN` |
|
||||
| `src/ecw_bot/core/results_db.py` | Column map, value conversion, `details_for`, `insert_rows`, `save_summary` |
|
||||
| `src/ecw_bot/core/charge_summary.py` | `build()` split out of `write()` |
|
||||
| `src/ecw_bot/flows/export_medcor_ready.py` | `changes` in `_conditions`, `details` dict, `_run_mode()`, `_save_results_db()` |
|
||||
| `web/__init__.py`, `web/repo.py`, `web/app.py`, `web/templates/index.html` | Read-only page + JSON API |
|
||||
| `run_web.bat` | Start the page on 127.0.0.1:8000 |
|
||||
| `tests/test_results_db.py`, `tests/test_web_app.py` | Tests |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: SQL script, settings, `results_db` module
|
||||
|
||||
**Files:**
|
||||
- Create: `tools/sql/create_tables.sql`, `src/ecw_bot/core/results_db.py`, `tests/test_results_db.py`
|
||||
- Modify: `src/ecw_bot/settings.py` (after `SUMMARY_ASSIGNED_TO`)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `results_db.COLUMNS: list[tuple[str, str]]` (header, db column), `DB_COLUMNS: list[str]`, `TABLE = "dbo.ChargeSummary"`, `to_money(v) -> Decimal|None`, `to_date(v) -> date|None`, `to_record(row: dict, *, run_at, run_mode, details) -> dict`, `details_for(data: dict) -> dict`, `insert_rows(conn, records) -> int`, `save_summary(sheet, details, *, run_at, run_mode, conn_str, connect=None) -> int|None`; settings `RESULTS_DB: bool`, `RESULTS_DB_CONN: str`.
|
||||
|
||||
- [ ] **Step 1: Write the failing tests** — `tests/test_results_db.py` with cases: every HEADER mapped (24) and `DB_COLUMNS` = run_at, run_mode, 24 columns, details_json; `to_money("$1,234.50") == Decimal("1234.50")`, `"(12.00)"` → -12.00, blank/"abc"/"NaN"/"1e20" → None, `365` → 365.00; `to_date("7/31/2026") == date(2026,7,31)`, ISO and datetime accepted, blank/junk → None; `to_record` converts types, blank text → None, `details_json` is JSON or None; `details_for` copies `_conditions` keys, Dental results, error keys; `insert_rows` sends one `INSERT INTO dbo.ChargeSummary (...) VALUES (?, ...)` with values in `DB_COLUMNS` order and commits once; `save_summary` skips blank-Category rows, closes the connection, returns the count; a raising `connect` returns None and prints `[RESULTS-DB]`. (Full code: see the test file written in Step 1 of execution.)
|
||||
|
||||
- [ ] **Step 2: Run** `python tests\test_results_db.py` — Expected: ImportError (no `results_db`).
|
||||
|
||||
- [ ] **Step 3: Implement** `results_db.py` per the Interfaces above (money: strip `$ ,`, `(x)` negative, finite and `< 10**10`, quantize 0.01; dates: `mm/dd/yyyy`, `yyyy-mm-dd`, `date`/`datetime`; `INSERT` built only from `DB_COLUMNS` names and `?` markers; `save_summary` wraps everything in `try/except Exception` → `[RESULTS-DB] Could not save ...` + None). Settings:
|
||||
|
||||
```python
|
||||
# ── Results database (SQL Server) ─────────────────────────────────────────────
|
||||
# Also save the Charge Summary rows to SQL Server (core/results_db.py) for the
|
||||
# results web page (web/). Off by default; a DB error is a log line only.
|
||||
RESULTS_DB = _env_bool("RESULTS_DB", False)
|
||||
RESULTS_DB_CONN = _env(
|
||||
"RESULTS_DB_CONN",
|
||||
"DRIVER={ODBC Driver 18 for SQL Server};SERVER=MECARE331\\MECARE331;"
|
||||
"DATABASE=Charge_Validation;Trusted_Connection=yes;"
|
||||
"TrustServerCertificate=yes;")
|
||||
```
|
||||
|
||||
and `tools/sql/create_tables.sql` = the spec's `CREATE TABLE` wrapped in `IF OBJECT_ID('dbo.ChargeSummary','U') IS NULL BEGIN ... END`.
|
||||
|
||||
- [ ] **Step 4: Run** `python tests\test_results_db.py` — Expected: all passed.
|
||||
|
||||
### Task 2: `charge_summary.build()` and the flow hook
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ecw_bot/core/charge_summary.py` (`write()`), `src/ecw_bot/flows/export_medcor_ready.py` (`_read_claim_form` `_conditions`, claim loop, after `_write_charge_summary`)
|
||||
- Test: `tests/test_results_db.py` (flow cases), `tests/test_charge_summary.py` (unchanged, must stay green)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `results_db.details_for`, `results_db.save_summary`, `cfg.RESULTS_DB`, `cfg.RESULTS_DB_CONN`.
|
||||
- Produces: `charge_summary.build(report_path, outcomes, assigned_to) -> list` (header + rows); flow `_run_mode() -> str` ("TEST" / "LIVE-WRITE" / "VERIFY"), `_save_results_db(info, outcomes, details) -> None`.
|
||||
|
||||
- [ ] **Step 1: Failing tests** in `tests/test_results_db.py`: `_run_mode()` for the three settings combinations; `_save_results_db` with `RESULTS_DB` off calls nothing; on, it calls `charge_summary.build(info["file"], outcomes, cfg.SUMMARY_ASSIGNED_TO)` and `results_db.save_summary(sheet, details, run_at=<datetime>, run_mode=..., conn_str=cfg.RESULTS_DB_CONN)`; a raising `build` prints `[RESULTS-DB]` and does not raise.
|
||||
- [ ] **Step 2: Run** — Expected: AttributeError (`_run_mode` missing).
|
||||
- [ ] **Step 3: Implement**: move `write()`'s read+`build_rows` into `build()`, `write()` calls `build()`; in `_read_claim_form` add `"changes": {r.condition_name: r.changes for r in results}` to `data["_conditions"]`; in the claim loop `details = {}` and `details[claim_id] = results_db.details_for(data)`; after `_write_charge_summary(info, outcomes)` call `_save_results_db(info, outcomes, details)`.
|
||||
- [ ] **Step 4: Run** `tests\test_results_db.py`, `tests\test_charge_summary.py`, `tests\test_claim_boxes.py` — Expected: all passed.
|
||||
|
||||
### Task 3: Web page
|
||||
|
||||
**Files:**
|
||||
- Create: `web/__init__.py`, `web/repo.py`, `web/app.py`, `web/templates/index.html`, `run_web.bat`, `tests/test_web_app.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `results_db.COLUMNS`, `results_db.TABLE`, `cfg.RESULTS_DB_CONN`.
|
||||
- Produces: `web.repo.SqlResultsRepo(conn_str, connect=None)` with `runs() -> list[dict]` (`run_at, run_mode, claims, pass_count, fail_count, review_count`, newest first) and `results(run_at, category=None, claim_no=None) -> list[dict]`; `web.app.create_app(repo) -> FastAPI`; module-level `web.app.app`.
|
||||
|
||||
- [ ] **Step 1: Failing tests** `tests/test_web_app.py` with a fake repo: `/` lists rows of the latest run, shows Pass/Fail/Review counts and the Charge Summary headers; `?category=Review` passes the filter to the repo; `?category=Bad` → 400; `?claim_no=55-51` searches `5551`… digits only; `?run_at=not-a-date` → 400; `/api/runs` and `/api/results` JSON shapes (dates ISO, money numbers); repo raising → 503 with "Results database not reachable"; no runs → page says no results yet; `SqlResultsRepo.results` builds a parameterized query (fake connection) with `category = ?` and `claim_no LIKE ?`.
|
||||
- [ ] **Step 2: Run** — Expected: ImportError.
|
||||
- [ ] **Step 3: Implement** repo, app (`docs_url=None`), template (form: run select, category select, claim search; tiles; table in Charge Summary order with Category colour; `<details>` per row with pretty JSON), `run_web.bat` (`python -m uvicorn web.app:app --host 127.0.0.1 --port 8000`).
|
||||
- [ ] **Step 4: Run** `python tests\test_web_app.py` — Expected: all passed.
|
||||
|
||||
### Task 4: Docs and full suite
|
||||
|
||||
- [ ] Context.md: a "Results DB + web page" section (setup: run `tools/sql/create_tables.sql`, grant the Windows login read/write, `RESULTS_DB=1`, `run_web.bat`).
|
||||
- [ ] Full suite: `bash run_tests.sh` → TOTAL all passed.
|
||||
- [ ] Report to the operator; commit only when asked.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,67 @@
|
||||
# Results Portal Redesign Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace the one-page results table with a Dashboard → Claim status (date cards) → One day (short table + side panel) portal, plus a claim history search, in a navy & blue theme.
|
||||
|
||||
**Architecture:** Server-rendered FastAPI + Jinja pages sharing `base.html`. Three new read-only queries in `web/repo.py` group the existing `CV_ChargeSummary` rows by the day a run started. The Bot run card and `/api/*` stay as they are; old `/?run_at=` links redirect to the day page.
|
||||
|
||||
**Tech Stack:** FastAPI, Jinja2, pyodbc (SQL Server), vanilla JS, standalone test scripts (`python tests\test_web_app.py`).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-10-19-results-portal-redesign-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Bound to 127.0.0.1; Host allowlist, `no-store` headers, DB errors by kind — unchanged.
|
||||
- Start / Stop need the page Origin and `X-Run-Token`; page runs are VERIFY only.
|
||||
- All new pages read-only; every SQL statement parameterized; values never in SQL text.
|
||||
- DB text is escaped: Jinja autoescape, panel data via `tojson` in an attribute, written with `textContent`.
|
||||
- No claim ids or patient data in tracked docs; tests use made-up ids.
|
||||
- No git commits unless the operator asks.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Day-wise queries in the repo
|
||||
|
||||
**Files:** Modify `web/repo.py`; Test `tests/test_web_app.py`
|
||||
|
||||
**Interfaces — Produces:**
|
||||
- `SqlResultsRepo.days(limit: int = 90) -> list[dict]` — keys `day` (date), `claims`, `runs`, `pass_count`, `fail_count`, `review_count`; newest first.
|
||||
- `SqlResultsRepo.day_results(day: date, category=None, claim_no=None) -> list[dict]` — full rows (`_SELECT_COLUMNS`) with `run_at >= day AND run_at < day+1`, ordered `run_at, s_no, id`.
|
||||
- `SqlResultsRepo.claim_history(claim_no: str) -> list[dict]` — `TOP 200` full rows `WHERE claim_no LIKE ?` ordered `run_at DESC, s_no, id`.
|
||||
- `_coerce` also turns a text `day` into a `date`.
|
||||
|
||||
- [ ] Tests: each query's SQL is a parameterized SELECT with the clauses above and the expected params (`[datetime(day), datetime(day)+1 day]`, `["%no%"]`); text `day` coerced.
|
||||
- [ ] Implement; run `python tests\test_web_app.py`.
|
||||
|
||||
### Task 2: Routes
|
||||
|
||||
**Files:** Modify `web/app.py`; Test `tests/test_web_app.py`
|
||||
|
||||
**Interfaces — Consumes:** Task 1. **Produces:** routes
|
||||
- `GET /` → `dashboard.html` with `today` (days() row for `date.today()` or zero counts), `mode` ("VERIFY"/"LIVE-WRITE"), `run_token`. `GET /?run_at=<iso>` → 303 to `/day/<date>` (keeps `category`, `claim_no`); bad `run_at` → 400.
|
||||
- `GET /days` → `days.html` with `days`.
|
||||
- `GET /day/{day}` (`YYYY-MM-DD`, else 400) with `category`, `claim_no`, `claim` → `day.html` with `day`, `counts` (from `days()` row or computed from the unfiltered rows), `rows` (each with `fields` = list of `[header, display text]` and `details_pretty`), filters.
|
||||
- `GET /claim?claim_no=` → `claim.html` with `rows` (history) and `claim_no`; empty search → page with the box only.
|
||||
- FakeRepo in tests gains `days`, `day_results`, `claim_history`.
|
||||
|
||||
- [ ] Tests for each route (data shown, links, filters passed, 400s, redirect, DB down 503, hardening incl. new paths).
|
||||
- [ ] Implement; run tests.
|
||||
|
||||
### Task 3: Templates and theme
|
||||
|
||||
**Files:** Create `web/templates/base.html`, `dashboard.html`, `days.html`, `day.html`, `claim.html`; delete `web/templates/index.html`.
|
||||
|
||||
- `base.html`: `:root` navy palette, header (title, mode pill, breadcrumb block), `{% block content %}`, `{% block scripts %}`, footer PHI note.
|
||||
- `dashboard.html`: Bot run card (moved from index.html, with the Live claims panel replacing the old Claim status panel; a saved claim opens `/day/<results_run_at date>?claim=<id>`), today tiles, Find a claim form, big Claim status button.
|
||||
- `days.html`: date cards grid with counts and colour bar; empty state.
|
||||
- `day.html`: tiles as filter links, claim search, short table (#, Claim #, Time, Payer, Result, Comments), 10-a-page pager, side panel from `data-fields` (`tojson`) via `textContent`; `?claim=` opens it.
|
||||
- `claim.html`: search box and history table linking to day pages.
|
||||
|
||||
- [ ] Tests on rendered HTML (ids, links, no inline `style=`, theme vars, panel wiring, one-scroll lock, Live claims link target).
|
||||
- [ ] Run `python tests\test_web_app.py` and `python tests\test_run_status.py`.
|
||||
|
||||
### Task 4: Browser check
|
||||
|
||||
- [ ] Serve the app on a fake repo with sample data (scratch script, made-up ids), open each page in the in-app browser, screenshot, check pager, panel, filters, empty states, narrow width.
|
||||
- [ ] Full suite.
|
||||
@@ -0,0 +1,182 @@
|
||||
# Payer → Group Name categorization — design
|
||||
|
||||
Status: approved by user (Vasanth), 2026-09-15. Not yet implemented.
|
||||
|
||||
## Background
|
||||
|
||||
This is the next phase of the Salud eCW Charge Entry bot (see `README.md`),
|
||||
which already automates login, 2FA, claim lookup, and exporting the
|
||||
"Medcor Ready" claim queue to `Input\Claim Report_<date>.xlsx`.
|
||||
|
||||
The next manual step in the real process (shown in
|
||||
`Salud-Charge Entry_Medcor Ready.mp4`, reviewed 2:45–10:00) is: an operator
|
||||
renames a copy of that export to "Medcor ready", opens it in a dated network
|
||||
folder (`...\Charge\Allocation\<Month YYYY>\<MMDDYYYY>\`, alongside sibling
|
||||
workbooks `Patient`, `Pending with Error`, and `Program`), and manually
|
||||
categorizes every row by filtering on the `PAYER` column and bulk-filling a
|
||||
`Group Name` column with one of a small set of values (observed in the video:
|
||||
`Commercial`, `Dentaquest`, `Medicaid`, `Medicare`, `Program`, plus at least
|
||||
one more category for payers like `BH COLORADO ACCESS`).
|
||||
|
||||
This spec covers automating **only** that categorization step — not the
|
||||
downstream split into the three sibling workbooks, which stays manual for now
|
||||
and may be scoped separately later.
|
||||
|
||||
## Scope decisions (from user)
|
||||
|
||||
- **Scope**: categorize only. Do not build the split into
|
||||
Patient/Pending with Error/Program workbooks.
|
||||
- **Mapping source**: derived from the actual current export
|
||||
(`Input\Claim Report_09-11-2026.xlsx`, 729 rows, 39 distinct `PAYER`
|
||||
strings), not invented from video frames. Only high-confidence entries are
|
||||
seeded; everything else is left for a human to fill in (see "Seed mapping
|
||||
table" below) — this is intentional, not a gap to silently paper over.
|
||||
- **Unknown payer handling**: flag, never guess. A `PAYER` value with no
|
||||
entry in the mapping table gets a blank `Group Name`, is logged, and is
|
||||
called out in the run summary. No fuzzy/substring matching — exact string
|
||||
match only, so a typo'd or renamed payer can never be silently bucketed
|
||||
into the wrong group.
|
||||
- **Integration point**: inside the existing bot, not a standalone script.
|
||||
Runs automatically as part of `flows/export_medcor_ready.py`.
|
||||
- **Output location**: a new file, `Output\Medcor Ready_<date>.xlsx`. The
|
||||
original `Input\Claim Report_<date>.xlsx` is never modified — `claim_file.py`
|
||||
and everything downstream of it depends on that file's exact column
|
||||
layout (per `README.md`'s "Reading the claim ids back" section), so this
|
||||
step must not touch it.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
flows/export_medcor_ready.py
|
||||
... existing steps (login → claim lookup → export → claim_file.claim_ids()) ...
|
||||
→ categorize_by_payer() NEW — reads Input\, writes Output\
|
||||
core/payer_groups.py NEW — the lookup table + classify()
|
||||
payer_groups.json NEW — editable config, exact PAYER → Group Name
|
||||
core/spreadsheet.py EXTENDED — write_xlsx(), stdlib only
|
||||
```
|
||||
|
||||
`categorize_by_payer()` runs after the existing claim-id read-back (the point
|
||||
where the report is already saved and validated — see
|
||||
`README.md`'s "Reading the claim ids back"). It:
|
||||
|
||||
1. Reads `Input\Claim Report_<date>.xlsx` with the existing `spreadsheet.read_rows()`.
|
||||
2. For each row, looks up `PAYER` in `payer_groups.classify()`.
|
||||
3. Writes every original column plus a new `Group Name` column to
|
||||
`Output\Medcor Ready_<date>.xlsx`.
|
||||
4. Logs and message-boxes a summary: rows categorized, rows left blank, and
|
||||
the distinct list of unmapped `PAYER` strings — mirroring the existing
|
||||
`NOTIFY_AFTER_EXPORT` / `SHOW_CLAIM_IDS` pattern in the same file.
|
||||
|
||||
Never fatal: an unmapped payer does not stop the run, the same philosophy
|
||||
`export_claims_report()` already uses for other soft failures.
|
||||
|
||||
## Components
|
||||
|
||||
### `src/ecw_bot/core/payer_groups.py`
|
||||
|
||||
```python
|
||||
def classify(payer: str) -> str | None: ... # exact match against the config, or None
|
||||
def load_mapping() -> dict[str, str]: ... # reads payer_groups.json once
|
||||
```
|
||||
|
||||
Exact-string lookup only. No normalization beyond what's needed to match
|
||||
eCW's own values (the config keys are the literal `PAYER` strings as eCW
|
||||
exports them).
|
||||
|
||||
### `src/ecw_bot/payer_groups.json`
|
||||
|
||||
A flat `{"<exact PAYER string>": "<Group Name>"}` map, hand-editable by
|
||||
anyone on the team without touching code — same reasoning as `.env` for
|
||||
settings and `selectors\` for DOM selectors: this data changes for a
|
||||
different reason (a new payer shows up) than the code around it does.
|
||||
|
||||
### `core/spreadsheet.py` — new `write_xlsx()`
|
||||
|
||||
The project deliberately has no `openpyxl` dependency (see README: "every
|
||||
added package is an approval"). `write_xlsx()` follows the same approach
|
||||
already used for reading — a minimal single-sheet xlsx as a zip of XML, cells
|
||||
as inline strings (no `sharedStrings.xml`, matching what eCW's own exports
|
||||
already do, per the existing reader's comments) — built with `zipfile` and
|
||||
string templates only.
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
Input\Claim Report_<date>.xlsx
|
||||
→ read_rows() existing
|
||||
→ for each row: classify(PAYER) new
|
||||
→ write_xlsx(rows + Group Name) new
|
||||
→ Output\Medcor Ready_<date>.xlsx
|
||||
```
|
||||
|
||||
## Error handling
|
||||
|
||||
- Unmapped payer: blank `Group Name`, logged, summarized — not fatal.
|
||||
- `Input\` file missing/unreadable: this step is skipped with a clear log
|
||||
line; it never blocks the rest of the run, since the export already
|
||||
succeeded by the time this step runs.
|
||||
- `write_xlsx()` failure (disk full, permission): logged and raised — same
|
||||
severity as any other unexpected I/O failure in this flow, since silently
|
||||
losing the categorized output would defeat the point of the step.
|
||||
|
||||
## Testing
|
||||
|
||||
`tests/test_payer_groups.py`, hand-rolled in the existing style (no pytest):
|
||||
|
||||
- Exact-match lookups against a small fixture mapping.
|
||||
- Unmapped payer → `None`, never a guess.
|
||||
- Round-trip test for `write_xlsx()`: write a small fixture table, read it
|
||||
back with the existing `read_rows()`, assert the data matches — the same
|
||||
"never touched eCW" local-fixture philosophy as `test_claim_form.py`.
|
||||
|
||||
## Seed mapping table
|
||||
|
||||
Built from the real, current payer list (`Input\Claim Report_09-11-2026.xlsx`,
|
||||
729 rows). Only high-confidence entries are seeded — this covers roughly 95%
|
||||
of row volume in that file.
|
||||
|
||||
**Mapped (seeded):**
|
||||
|
||||
| Payer (exact eCW string) | Group |
|
||||
|---|---|
|
||||
| Medicaid | Medicaid |
|
||||
| DentaQuest | Dentaquest |
|
||||
| CHP+ DENTAQUEST | Dentaquest |
|
||||
| AARP Medicare Advantage | Medicare |
|
||||
| Aetna Medicare ADV | Medicare |
|
||||
| OUTREACH PROGRAM | Program |
|
||||
| Anthem Blue Cross Blue Shield | Commercial |
|
||||
| Cigna HealthCare Of Colorado | Commercial |
|
||||
| Cigna Healthcare of CO | Commercial |
|
||||
| Cigna Salud Employee | Commercial |
|
||||
| Cigna | Commercial |
|
||||
| Cypress Benefit Admin / Cigna PPO | Commercial |
|
||||
| United Healthcare | Commercial |
|
||||
| United Health Care | Commercial |
|
||||
| United healthcare | Commercial |
|
||||
| Unitedhealth One | Commercial |
|
||||
| Oxford Health Insurance UHC | Commercial |
|
||||
| UMR | Commercial |
|
||||
| Aetna | Commercial |
|
||||
| Aetna Alaska Care | Commercial |
|
||||
| AETNA HEALTH AND LIFE INSURANCE | Commercial |
|
||||
| BIND BENEFITS INC | Commercial |
|
||||
| Select Health | Commercial |
|
||||
|
||||
**Left unmapped — needs a human decision before first real run (or will be
|
||||
flagged on first run and filled in afterward):**
|
||||
|
||||
BH COLORADO ACCESS (15 rows), Delta Dental of Colorado (3), Devoted (2),
|
||||
Clinic Sliding Fee Scale (2), Patient (1), Gravie Admin Services (1),
|
||||
Tricare (1), Rocky Mountain Health Plans (1), Denver Elevate Health Plan (1),
|
||||
The Kempton Group Administrators Inc (1), Optum Care Network (1), GPA (1),
|
||||
Northeast Health Partners (1), Delta Dental of Pennsylvania (1),
|
||||
Delta Dental of Wisconsin (1), Delta Dental (1), Delta Dental of Arkansas (1).
|
||||
|
||||
## Out of scope (explicitly)
|
||||
|
||||
- Splitting rows into the Patient / Pending with Error / Program sibling
|
||||
workbooks.
|
||||
- Writing to the network `Charge\Allocation\...` folder — output stays local
|
||||
to `Output\`.
|
||||
- Any change to `Input\Claim Report_<date>.xlsx` or anything that reads it.
|
||||
@@ -0,0 +1,269 @@
|
||||
# Condition Framework — Design
|
||||
|
||||
**Status:** Approved by user, 2026-09-16
|
||||
**Supersedes nothing** — this formalizes the pattern Condition 1 (telehealth)
|
||||
already established informally in `flows/export_medcor_ready.py`.
|
||||
|
||||
## Why
|
||||
|
||||
The Charge Validation Process is, by the client's own description, a long
|
||||
list of per-claim conditions to check and — when a condition's criteria are
|
||||
met — a correction to apply (Condition 1: a telehealth visit needs POS=11 and
|
||||
M1=GT). Conditions are being implemented one at a time, and there will be
|
||||
many more of them. This is a healthcare billing bot: every write it makes
|
||||
changes what gets billed to a real payer, so the process must be:
|
||||
|
||||
- **Reusable** — adding Condition N must not require touching the code for
|
||||
Conditions 1..N-1, and must follow one obvious, fixed shape every time.
|
||||
- **Exception-safe** — a bug in one condition's logic must never corrupt
|
||||
another condition's result on the same claim, crash the whole overnight
|
||||
batch, or leave a claim half-changed with no record of what happened.
|
||||
- **Provably correct before it ever touches a real claim** — every condition
|
||||
must be unit-testable against a fixture, exactly like
|
||||
`tests/test_telehealth_fix.py` already does for Condition 1.
|
||||
|
||||
This spec explicitly rejects driving the live automation with an
|
||||
LLM-agent-controlled browser (e.g. Playwright MCP) as the *engine* for this
|
||||
reason: none of the three properties above hold for an engine whose
|
||||
per-claim behavior is an LLM's live judgment call rather than fixed,
|
||||
versioned code. Playwright MCP remains valuable as a *development and
|
||||
debugging aid* (inspecting live eCW DOM, prototyping selectors) — that usage
|
||||
is unaffected by this spec and requires no code change.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Not changing how claims are found, opened, or read (`claim_form.py`,
|
||||
`export_medcor_ready.py`'s claims-phase — untouched).
|
||||
- Not building a self-healing/adaptive-selector system. If eCW's DOM shifts
|
||||
enough to break a selector, that is still a `selectors/claim_form.py`
|
||||
update, done and tested like every past one (see CHANGELOG "Day 5").
|
||||
- Not retrying failed conditions automatically. A condition that throws is
|
||||
logged and skipped for that claim, once, this run.
|
||||
|
||||
## Architecture
|
||||
|
||||
### New package: `src/ecw_bot/conditions/`
|
||||
|
||||
```
|
||||
src/ecw_bot/conditions/
|
||||
__init__.py # CONDITIONS: list[module] — the registry
|
||||
base.py # ConditionResult dataclass + run_all()
|
||||
telehealth.py # Condition 1 — detect/apply/describe wrapper
|
||||
```
|
||||
|
||||
### The Condition contract
|
||||
|
||||
Every condition module in the registry exposes exactly these three
|
||||
functions. This is the whole interface — nothing else is required, nothing
|
||||
else is read by the runner.
|
||||
|
||||
```python
|
||||
def detect(claim_id: str, data: dict) -> bool:
|
||||
"""
|
||||
Pure. No `page`, no I/O, no side effects. `data` is the same dict
|
||||
`claim_form.extract()` already produces (plus `progress_notes`, when
|
||||
OPEN_PROGRESS_NOTES is on). Must be safe to call with a partially
|
||||
populated dict — return False, never raise, on missing keys.
|
||||
"""
|
||||
|
||||
def apply(page, claim_id: str, data: dict) -> "ConditionResult":
|
||||
"""
|
||||
Only called when detect() returned True. Makes the eCW field edits for
|
||||
this condition and returns what changed. MUST NOT click Save/OK and
|
||||
MUST NOT show any confirmation dialog itself — the runner's caller owns
|
||||
both, once, after every condition has had a chance to run. Raising is
|
||||
allowed and expected to be handled by the caller (see Exception
|
||||
handling below) — apply() does not need its own try/except for its own
|
||||
field-write logic, only for optional recovery it wants to attempt
|
||||
itself.
|
||||
"""
|
||||
|
||||
def describe(result: "ConditionResult") -> str:
|
||||
"""
|
||||
One human-readable block for this condition's contribution to the
|
||||
combined confirmation dialog and the log line — mirrors the per-row
|
||||
lines `_apply_telehealth_fix` already builds today.
|
||||
"""
|
||||
```
|
||||
|
||||
### `ConditionResult` (`conditions/base.py`)
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ConditionResult:
|
||||
condition_name: str
|
||||
changed: bool
|
||||
changes: list # same shape as claim_form.set_cpt_pos_modifier()'s
|
||||
# return value: [{"row", "code", "before", "after"}, ...]
|
||||
|
||||
review: str = "" # added 2026-10-05 (unit D) — see below
|
||||
|
||||
@classmethod
|
||||
def none(cls, condition_name: str) -> "ConditionResult":
|
||||
return cls(condition_name=condition_name, changed=False, changes=[])
|
||||
|
||||
@classmethod
|
||||
def declined(cls, condition_name: str, reason: str) -> "ConditionResult":
|
||||
return cls(condition_name=condition_name, changed=False, changes=[],
|
||||
review=reason)
|
||||
```
|
||||
|
||||
**Amendment 2026-10-05 (review items D2/P2, unit D).** `none()` means
|
||||
"nothing to do — the claim already follows this rule". A condition that
|
||||
matched the claim but could not act on it (the CPT/HCPCS tab would not
|
||||
open, an admin code would not add with the grid untouched) now returns
|
||||
`declined(name, reason)` instead: nothing changed, but the claim is not
|
||||
clean, and `reason` reaches the Charge Summary as a "Review" row. A
|
||||
changed result may also carry `review` (a failed add that still moved the
|
||||
grid). Two optional module hooks join `detect/apply/describe`:
|
||||
|
||||
- `needs_review(claim_id, data) -> str | None` — asked only when
|
||||
`detect()` returned False, for a claim that matched the trigger but
|
||||
cannot be acted on (vaccine_admin_codes on an unmapped payer, no payer
|
||||
name, no usable age — never Medicare, which has its own rule).
|
||||
- `summary_comment(result) -> str` — the Charge Summary comment for a
|
||||
changed result, built from what actually happened, in place of the
|
||||
summary's fixed `COMMENTS[name]`.
|
||||
|
||||
### The runner (`conditions/base.py`)
|
||||
|
||||
```python
|
||||
def run_all(page, claim_id: str, data: dict) -> list[ConditionResult]:
|
||||
"""
|
||||
Runs every registered condition against one claim, in registry order.
|
||||
Returns only the results where something actually changed — a claim
|
||||
with nothing to fix returns an empty list, and the caller shows no
|
||||
dialog and saves nothing.
|
||||
"""
|
||||
results = []
|
||||
for cond in CONDITIONS:
|
||||
name = cond.__name__.rsplit(".", 1)[-1]
|
||||
try:
|
||||
if not cond.detect(claim_id, data):
|
||||
continue
|
||||
result = cond.apply(page, claim_id, data)
|
||||
except Exception as exc:
|
||||
print(f"[COND {name} {claim_id}] Failed: {exc}")
|
||||
shot(page, f"condition_{name}_failed_{claim_id}")
|
||||
continue # this condition's failure must not block the next one
|
||||
if result.changed:
|
||||
results.append(result)
|
||||
return results
|
||||
```
|
||||
|
||||
`shot()` is the existing screenshot helper already used by `open_claim()`'s
|
||||
diagnostic wrapping (`claim_form.py`) — reused here, not reinvented.
|
||||
|
||||
**Amendment 2026-10-05 (review items D2/P2, unit D).** The real signature
|
||||
is now `run_all(page, claim_id, data, conditions=None, before_apply=None,
|
||||
report=None)`. Its return value is unchanged — only the changed results —
|
||||
but that list is not everything that happened, so a caller that passes a
|
||||
dict as `report` gets the whole account:
|
||||
|
||||
| key | contents |
|
||||
|---|---|
|
||||
| `fired` | names whose `detect()` returned True |
|
||||
| `changed` | names with a changed result (= the return value) |
|
||||
| `failed` | `{"name", "stage", "error"}` per raise; `stage` is `"detect"`, `"apply"` or `"needs_review"`, `error` the exception's first line |
|
||||
| `review` | `{"name", "reason"}` from every result (changed or not) with a non-empty `review`, and from `needs_review()` of every condition whose `detect()` returned False |
|
||||
| `comments` | `{name: summary_comment(result)}` for changed results whose module has the hook (a hook that raises is logged and left out — the summary falls back to `COMMENTS[name]`) |
|
||||
|
||||
Exceptions are still isolated exactly as in the sketch above: logged,
|
||||
screenshotted, and the next condition still runs. `detect()` and `apply()`
|
||||
now each sit in their own try (the sketch's single try is how it began), so
|
||||
the report can say which stage raised.
|
||||
|
||||
### Caller contract (replaces `_apply_telehealth_fix`'s direct call)
|
||||
|
||||
`export_medcor_ready.py`'s `_read_claim_form()` changes from calling
|
||||
`_apply_telehealth_fix(page, claim_id, data)` directly to:
|
||||
|
||||
```python
|
||||
if cfg.APPLY_CORRECTIONS:
|
||||
results = conditions.run_all(page, claim_id, data)
|
||||
saved_write = _confirm_and_save(page, claim_id, results)
|
||||
```
|
||||
|
||||
(As built since 2026-10-05: `run_all(..., report=report)` and
|
||||
`_confirm_and_save(page, claim_id, results, report)`; the report is stored
|
||||
on the claim's data as `data["_conditions"]` = `{"fired", "saved",
|
||||
"failed", "review", "comments"}`, which the Charge Summary reads.)
|
||||
|
||||
`_confirm_and_save` (new, small, lives in `export_medcor_ready.py` — it is
|
||||
orchestration, not a condition):
|
||||
|
||||
1. If `results` is empty → return `False` immediately, no dialog, no save.
|
||||
(A claim where nothing fired is not different from today's "no
|
||||
telehealth mention" case.)
|
||||
2. Otherwise build one combined message: every fired condition's
|
||||
`describe()` output, concatenated, in the same
|
||||
`msgbox.ok_cancel(...)` shape Condition 1 uses today.
|
||||
3. One Yes/No. Yes → `claim_form.save_claim(page)`. No → log "declined",
|
||||
return `False` (the caller's existing `finally` still closes with
|
||||
Cancel, unchanged).
|
||||
4. Return whether the save actually happened, same as
|
||||
`_apply_telehealth_fix` returns today — this is what `_read_claim_form`
|
||||
uses to decide whether its own `finally` should call `close_claim()`.
|
||||
|
||||
**Amendment 2026-10-05 (review item P2, unit D):** between 1 and 2 —
|
||||
and after the SAVE_CORRECTIONS gate — if the report has any `failed` or
|
||||
`review` entry, the box is never shown: the log says `[COND <id>] Not
|
||||
offering Save: <reasons> — closing with Cancel` and it returns `False`.
|
||||
|
||||
**Why one dialog, not one per condition:** if two conditions both want
|
||||
changes on the same claim, saving after only reading half the operator's
|
||||
decision is exactly the kind of half-applied state this spec exists to
|
||||
prevent. The operator sees the whole picture — every condition that fired,
|
||||
every field that would change — before any single Save/Cancel decision.
|
||||
|
||||
## Exception handling — the actual rules
|
||||
|
||||
This is the section the user asked for explicitly, spelled out rather than
|
||||
left implicit:
|
||||
|
||||
| Failure | What happens |
|
||||
|---|---|
|
||||
| One condition's `detect()` raises | Caught by `run_all`'s try/except (its own try since 2026-10-05, so the report can say `stage: "detect"`). Logged, screenshotted, skipped, and recorded in the report's `failed` — the claim reads "Review", not clean. Other conditions still run. |
|
||||
| One condition's `apply()` raises partway through a multi-row edit | Caught by `run_all`. Logged, screenshotted. That condition contributes nothing to `results`. **Corrected 2026-10-05 (review item P2):** this row used to say its partial edits "are never saved, because Save only happens ... for what's in `results`". That was wrong: Save (`saveAllData`) writes the whole form, partial edits included, and the box would have listed only the other conditions. So the raise is recorded in `run_all`'s report (`failed`, stage `"apply"`), and `_confirm_and_save` never offers Save for a claim whose report has a `failed` or `review` entry — the claim is closed with Cancel and reported "Review" in the Charge Summary. |
|
||||
| `_confirm_and_save`'s dialog itself fails to show (msgbox error) | Not caught here — surfaces to `_read_claim_form`'s existing outer try/except, which logs "Could not read the form" and closes via `finally`. No new behavior needed; the existing outer boundary already covers this. |
|
||||
| `save_claim()` returns `False` (save click didn't work) | Same as today: logged, `saved_write` stays `False`, `finally` calls `close_claim()` — the claim is left as read, not half-saved. |
|
||||
| A condition throws on every claim in the batch (systemic bug in that one condition) | By design, this spec does not add batch-level circuit-breaking — each claim's screenshot + log line is enough signal for the operator to notice a pattern across a run's log file. Adding an auto-disable-after-N-failures behavior is explicitly deferred (YAGNI): today's batches run at `CLAIM_PROCESS_LIMIT` scale, small enough that log review is a real diagnostic step, not a chore. |
|
||||
|
||||
## Testing
|
||||
|
||||
- `tests/test_telehealth_fix.py` — unchanged. It tests
|
||||
`claim_form.note_mentions_telehealth` and `claim_form.set_cpt_pos_modifier`
|
||||
directly; those functions do not move.
|
||||
- `tests/test_conditions_telehealth.py` (new) — tests
|
||||
`conditions/telehealth.py`'s thin wrapper: `detect()` delegates to
|
||||
`note_mentions_telehealth` correctly on a plain dict (no page needed);
|
||||
`apply()` calls `set_cpt_pos_modifier` and returns a populated
|
||||
`ConditionResult`; `describe()` produces the expected text.
|
||||
- `tests/test_conditions_runner.py` (new) — tests `run_all()` and
|
||||
`ConditionResult` in isolation with fake condition modules (no real
|
||||
telehealth logic involved):
|
||||
- a condition whose `detect()` returns `False` is never `apply()`-ed
|
||||
- a condition whose `apply()` raises is skipped, and a *second*,
|
||||
healthy condition registered after it in `CONDITIONS` still runs and
|
||||
its result is still returned
|
||||
- `results` is empty when every condition's `detect()` returns `False`
|
||||
- a condition's raised exception does not propagate out of `run_all`
|
||||
|
||||
## Migration checklist (informs the implementation plan)
|
||||
|
||||
1. Create `conditions/base.py` (`ConditionResult`, `run_all`).
|
||||
2. Create `conditions/telehealth.py` — move `note_mentions_telehealth`-based
|
||||
detection logic into `detect()`, wrap `set_cpt_pos_modifier` in
|
||||
`apply()`, write `describe()` from the existing message-building lines
|
||||
in `_apply_telehealth_fix`.
|
||||
3. Create `conditions/__init__.py` with `CONDITIONS = [telehealth]`.
|
||||
4. Add `_confirm_and_save` to `export_medcor_ready.py`; delete
|
||||
`_apply_telehealth_fix` (its logic is now split across
|
||||
`conditions/telehealth.py` and `_confirm_and_save`).
|
||||
5. Update `_read_claim_form` to call `conditions.run_all` +
|
||||
`_confirm_and_save` instead of `_apply_telehealth_fix` directly.
|
||||
6. New tests (above) all passing; full existing suite (88+ tests) still
|
||||
passing unchanged.
|
||||
7. Condition 2 (whatever it turns out to be) is the first real proof this
|
||||
spec worked: it should require touching only a new
|
||||
`conditions/<name>.py` + its test file + one line in `CONDITIONS`.
|
||||
@@ -0,0 +1,137 @@
|
||||
# Dental claims run every applicable condition — design
|
||||
|
||||
Date: 2026-10-07. Status: approved in chat (operator, 2026-10-07);
|
||||
implemented 2026-10-12 (operator: "this is our number one condition").
|
||||
|
||||
## 2026-10-12 update (as built)
|
||||
|
||||
- The rule now also covers #12 (H codes: BH payer → modifier removed;
|
||||
any other payer → line unchecked + fee 0) and #14 (Medicare: G8510 /
|
||||
G8431 unchecked + fee 0, #6 not run). Both run on Dental claims like
|
||||
every other condition; #12's BH path has no modifier column to read on
|
||||
the Dental grid, so it never fires there.
|
||||
- A fix the Dental screen cannot hold → **Review** (operator): a condition
|
||||
module declares `DENTAL_UNSUPPORTED = "<reason>"` and `run_all` records
|
||||
that reason instead of applying it on a `screen == "dental"` read —
|
||||
telehealth, medicare_telehealth, vaccine_admin_codes.
|
||||
- Dental claims are **never saved** in this change (operator), whatever
|
||||
`SAVE_CORRECTIONS` says; the review-before-close box still shows.
|
||||
- A Dental claim is read and run through the conditions even with
|
||||
CHECK_HIGH_RISK and CHECK_SLIDING_FEE both off (it used to come back
|
||||
blank).
|
||||
- The Charge Summary applies the same claim-level checks (note read, CPT /
|
||||
ICD rows read, conditions run) to a Dental read as to UB/HCFA.
|
||||
- Row selectors gained a third (Dental) alternative, scoped to the Dental
|
||||
window on screen (`:visible`); `extract()` returns `screen`
|
||||
("ub" / "hcfa" / "dental").
|
||||
- Not live-verified yet: the first live run should be SAVE-off on a few
|
||||
Dental claims (one with DCONS or Dscprit).
|
||||
|
||||
## Rule (operator, 2026-10-07)
|
||||
|
||||
Every claim is checked against **every condition whose Payer column is
|
||||
"All" / "All payers"**, **plus** every condition specific to that claim's
|
||||
payer (Medicare, Medicaid, Dental payers, ...). This holds for every payer
|
||||
and every claim screen — UB, HCFA and Dental alike.
|
||||
|
||||
UB and HCFA claims already work this way: each claim runs the whole
|
||||
registry (`conditions/__init__.py` `CONDITIONS`) and each condition's own
|
||||
`detect()` decides from the claim's codes and payer. **The gap is Dental:** a
|
||||
Dental claim today runs only #15 high risk and #8 sliding fee and never
|
||||
reaches `run_all`. So, for example, `dummy_codes` (DCONS/Dscprit, which are
|
||||
Dental codes) can never fire on any claim.
|
||||
|
||||
## What a Dental claim will run
|
||||
|
||||
| Doc # | Condition | Payer (document) | Module |
|
||||
|---|---|---|---|
|
||||
| 5 | J2930/J2001 uncheck + fee 0 | ALL | `j2930_j2001_update` |
|
||||
| 6 | Depression DX pointer | All | `depression_dx_pointer` |
|
||||
| 7 | Zero-fee uncheck / priced line check | ALL | `zero_fee_uncheck`, `priced_line_check` |
|
||||
| 21 | Telehealth POS 2 + modifier | All | `telehealth` |
|
||||
| 22 | Medicare telehealth G2025 | Medicare | `medicare_telehealth` |
|
||||
| 18 | Vaccine admin codes | All except Medicare | `vaccine_admin_codes` |
|
||||
| 8 | DCONS/Dscprit uncheck + fee 0 | All Dental Payer | `dummy_codes` |
|
||||
| 8 | Sliding fee date check | All Dental Payer | `pages/sliding_fee` (unchanged) |
|
||||
| 15 | High risk in Billing Notes | Dental Payers | `pages/dental_notes` (unchanged) |
|
||||
|
||||
No condition gains a form-type check: each still decides from codes and
|
||||
payer only. A condition whose codes are not on the claim does nothing.
|
||||
|
||||
## Dental screen facts (from Dom.txt, claim screen `.modal[id^="dentalClaim"]`)
|
||||
|
||||
- CPT/CDT grid `table[id^="claimCPTCodesTable"]`; real rows are
|
||||
`tr[ng-repeat^="CPT in CPTCodes"]` (no `CPTROW_` id).
|
||||
- Per row: checkbox `[ng-click*="updateBillToIns("]` (same as UB/HCFA), code
|
||||
`[data-fieldname="web_cptcode"]`, pointers `web_icd1`..`web_icd4`, units
|
||||
`web_units`, fee `web_billedFee` (blur runs `editCPTRow('BILLED_FEE')`),
|
||||
DOS `DentalClaimCPTSDOS`.
|
||||
- **No per-row POS and no modifier columns.** POS is claim-level
|
||||
(`ng-model="ClaimData.pos"`).
|
||||
- Prog. Notes button `button[ng-click="viewProgNote()"]`
|
||||
(`dentalClaimProgressNoteBtn…`) — the existing `SEL_PROGRESS_NOTES` matches.
|
||||
- Insurance grid: `#DentalClaimTbl2` (Bill To, Ins Id, Name, ...).
|
||||
- Billing Notes: `#dentalClaimNotes` (already read by high risk).
|
||||
|
||||
None of the row fields has been exercised live yet; Dom.txt is one capture.
|
||||
|
||||
## Design
|
||||
|
||||
1. **Selectors** (`selectors/claim_form.py`): add the Dental alternative to
|
||||
each row selector the helpers already use — `SEL_CPT_ROWS`,
|
||||
`SEL_CPT_CODE`, `SEL_CPT_ICD1..4`, `SEL_CPT_BILLED_FEE`, `SEL_CPT_UNITS`.
|
||||
The Dental row selector is scoped under `.modal[id^="dentalClaim"]` so it
|
||||
cannot match a grid on another screen. Within one row only one alternative
|
||||
ever matches, the same contract the UB/HCFA pairs rely on.
|
||||
2. **Reader** (`pages/claim_form.py` `_EXTRACT_JS`): read Dental rows into the
|
||||
same `cpt` row shape (`code`, `icd1..4`, `units`, `billed_fee`, `billed`,
|
||||
`sdos`; `pos`/`mod1..4` empty), insurance from `#DentalClaimTbl2`, the
|
||||
Dental claim number, ICD list, and `notes` from `#dentalClaimNotes`.
|
||||
One `extract()` for all three screens.
|
||||
3. **Flow** (`flows/export_medcor_ready.py`): a Dental claim runs high risk +
|
||||
sliding fee first (unchanged, read-only), then the same path as UB/HCFA:
|
||||
`extract()`, Prog. Notes (when `OPEN_PROGRESS_NOTES`), `run_all`, per-claim
|
||||
box, Charge Summary. The Dental results stay on the claim's data so the box
|
||||
and the summary show both.
|
||||
4. **A fix the Dental screen cannot make** — e.g. telehealth needs a row
|
||||
modifier and the Dental grid has none, or POS is claim-level: the condition
|
||||
returns `declined(...)` with the reason ("Dental screen has no modifier
|
||||
column"), so the claim is **Review** in the Charge Summary. Never a silent
|
||||
skip, never a guess.
|
||||
5. **Charge Summary**: one row per claim combining condition comments with the
|
||||
Dental findings (HIGH RISK, sliding fee). Category: Review beats Fail beats
|
||||
Pass, as today.
|
||||
6. **Saving**: Dental claims are **never saved** in this change, whatever
|
||||
`SAVE_CORRECTIONS` says. Their corrections are shown and listed as
|
||||
"Corrections pending (not saved)". Dental save is enabled only after a live
|
||||
verification-mode run confirms the row writes.
|
||||
|
||||
## Error handling
|
||||
|
||||
- Dental form not loaded (`wait_claim_loaded`): conditions skipped, claim is
|
||||
Review — same as HCFA today.
|
||||
- A Dental row field missing (fee or checkbox not found): the existing helpers
|
||||
already treat a missing field as unknown, not as zero/unchecked.
|
||||
- High risk / sliding fee failures keep their current Review handling.
|
||||
|
||||
## Testing
|
||||
|
||||
- Grow `tests/fixtures/dental_claim.html` from Dom.txt's real Dental grid
|
||||
(rows, checkbox, fee, pointers, insurance grid, Prog. Notes button) with no
|
||||
patient data.
|
||||
- `extract()` on the Dental fixture returns the expected `cpt` and `insurance`.
|
||||
- Per condition, a Dental case: dummy codes (DCONS/Dscprit) uncheck + zero;
|
||||
J2001 uncheck + zero; zero-fee uncheck; priced line check; depression
|
||||
pointer retag on `web_icd*`; telehealth on Dental → declined/Review.
|
||||
- Flow: a Dental claim gets high risk + sliding fee **and** condition results
|
||||
in one summary row; never offered Save.
|
||||
- Full suite stays green.
|
||||
- Live: one SAVE-off run on Dental test claims (with a DCONS or Dscprit line)
|
||||
before Dental save is considered.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- New document conditions not yet built (#3, #4, #9-#14, #16, #19, #20,
|
||||
#23-#55).
|
||||
- Saving Dental claims.
|
||||
- "Special programs" half of #8.
|
||||
@@ -0,0 +1,164 @@
|
||||
# Charge Summary in SQL Server + results web page — design
|
||||
|
||||
Date: 2026-10-08. Status: design approved in chat (operator, 2026-10-08),
|
||||
spec under review.
|
||||
|
||||
## Goal
|
||||
|
||||
Every run's Charge Summary rows are also stored in SQL Server, and a simple
|
||||
local web page (FastAPI) shows them. This is the basic version; later it is
|
||||
folded into the full portal.
|
||||
|
||||
## Decisions (operator, 2026-10-08)
|
||||
|
||||
- Server `MECARE331\MECARE331` (SQL Server 2022, on this machine), database
|
||||
**`Charge_Validation`** (created by the operator).
|
||||
- The table follows the team's **Charge Summary File Format**
|
||||
(`\\mecare2\Production\Salud-eCW RPA\Charge Summary File Format.xlsx`) —
|
||||
the same 24 columns as `core/charge_summary.HEADERS`, plus run columns and
|
||||
a details JSON. PATIENT, Account Number and Guarantor Name are stored, as
|
||||
in the Excel file.
|
||||
- FastAPI for the backend. All needed packages are already installed
|
||||
(fastapi 0.141, uvicorn, jinja2, httpx, pyodbc, ODBC Driver 18); nothing new
|
||||
is added to `requirements.txt` for the bot itself.
|
||||
|
||||
## Table
|
||||
|
||||
`tools/sql/create_tables.sql`, run by the operator in SSMS (the bot never
|
||||
creates or alters tables):
|
||||
|
||||
```sql
|
||||
USE Charge_Validation;
|
||||
CREATE TABLE dbo.ChargeSummary (
|
||||
id INT IDENTITY(1,1) PRIMARY KEY,
|
||||
run_at DATETIME2 NOT NULL,
|
||||
run_mode NVARCHAR(20) NOT NULL, -- VERIFY / LIVE-WRITE / TEST
|
||||
s_no INT NULL,
|
||||
claim_no NVARCHAR(20) NOT NULL,
|
||||
service_date DATE NULL,
|
||||
pvdr NVARCHAR(50) NULL,
|
||||
patient NVARCHAR(200) NULL,
|
||||
payer NVARCHAR(200) NULL,
|
||||
group_name NVARCHAR(100) NULL,
|
||||
status NVARCHAR(100) NULL,
|
||||
charges DECIMAL(12,2) NULL,
|
||||
pmts_adjs DECIMAL(12,2) NULL,
|
||||
pmts_refunds DECIMAL(12,2) NULL,
|
||||
adjustment DECIMAL(12,2) NULL,
|
||||
withheld DECIMAL(12,2) NULL,
|
||||
balance DECIMAL(12,2) NULL,
|
||||
visit_type NVARCHAR(100) NULL,
|
||||
account_number NVARCHAR(50) NULL,
|
||||
provider_name NVARCHAR(200) NULL,
|
||||
guarantor_name NVARCHAR(200) NULL,
|
||||
received_date DATE NULL,
|
||||
assigned_to NVARCHAR(100) NULL,
|
||||
completed_on DATE NULL,
|
||||
comments NVARCHAR(MAX) NULL,
|
||||
current_status NVARCHAR(200) NULL,
|
||||
category NVARCHAR(20) NULL,
|
||||
details_json NVARCHAR(MAX) NULL
|
||||
);
|
||||
CREATE INDEX IX_ChargeSummary_run_at ON dbo.ChargeSummary (run_at);
|
||||
CREATE INDEX IX_ChargeSummary_claim_no ON dbo.ChargeSummary (claim_no);
|
||||
```
|
||||
|
||||
Column map (Excel → DB) lives in one place, `results_db.COLUMNS`, built from
|
||||
`charge_summary.HEADERS` so the two cannot drift: S.No→s_no, CLAIMS#→claim_no,
|
||||
SERVICE DATE→service_date, PVDR→pvdr, PATIENT→patient, PAYER→payer, Group
|
||||
Name→group_name, STATUS→status, CHARGES→charges, PMTS/ADJS→pmts_adjs,
|
||||
Pmts/Refunds→pmts_refunds, Adjustment→adjustment, WithHeld→withheld,
|
||||
Balance→balance, Visit Type→visit_type, Account Number→account_number,
|
||||
Provider Name→provider_name, Guarantor Name→guarantor_name, Recevied
|
||||
date→received_date, "Assigned To "→assigned_to, Completed On→completed_on,
|
||||
Comments→comments, Current Status→current_status, Category→category.
|
||||
|
||||
Value conversion: money columns strip `$`, `,` and spaces → Decimal; dates
|
||||
accept `mm/dd/yyyy` (and an Excel date/datetime) → date; blank or
|
||||
unparseable → NULL (the raw text is not lost: the Excel file still has it).
|
||||
|
||||
## Bot side — `src/ecw_bot/core/results_db.py`
|
||||
|
||||
- Settings (`settings.py`, from `.env`):
|
||||
- `RESULTS_DB` (bool, default **off**): write rows to the DB.
|
||||
- `RESULTS_DB_CONN` (string): ODBC connection string. Default:
|
||||
`DRIVER={ODBC Driver 18 for SQL Server};SERVER=MECARE331\MECARE331;DATABASE=Charge_Validation;Trusted_Connection=yes;TrustServerCertificate=yes;`
|
||||
(Windows login, no password). A SQL login's password, if ever used, is put
|
||||
in `.env` by the operator only.
|
||||
- `to_record(row: dict, *, run_at, run_mode, details) -> dict`: one Charge
|
||||
Summary row (header→value) to DB column→value, converted.
|
||||
- `insert_rows(conn, records) -> int`: one parameterized `INSERT` per row
|
||||
(`executemany`), one commit; returns rows written. Never builds SQL from
|
||||
values.
|
||||
- `save_summary(sheet, details, run_at, run_mode, connect=pyodbc.connect) ->
|
||||
int | None`: `sheet` is the header+rows list `charge_summary.build_rows`
|
||||
already returns; `details` maps claim no → details dict. Opens the
|
||||
connection, inserts, closes. **Any exception is caught**: one
|
||||
`[RESULTS-DB] Could not save ...` log line, returns None. The run, the Excel
|
||||
summary and the claims are never affected.
|
||||
- `details_json` per claim: `{"fired", "changed", "review", "failed",
|
||||
"comments", "saved", "changes": {condition: [change, ...]},
|
||||
"dental": {"high_risk", "sliding_fee"}}` — taken from the claim's
|
||||
`data["_conditions"]` (the flow adds `changes` there) and Dental results.
|
||||
|
||||
## Flow changes
|
||||
|
||||
- `charge_summary.write()` is split: `charge_summary.build(report_path,
|
||||
outcomes, assigned_to) -> sheet` (read + `build_rows`), and `write()` saves
|
||||
that sheet as today. The flow builds once and hands the same sheet to Excel
|
||||
and to the DB, so both always hold the same rows.
|
||||
- `_read_claim_form` adds `"changes": {r.condition_name: r.changes}` to
|
||||
`data["_conditions"]`; the claim loop keeps `details[claim_id]` next to
|
||||
`outcomes[claim_id]`.
|
||||
- `_write_charge_summary` calls `results_db.save_summary(...)` after the Excel
|
||||
write when `RESULTS_DB` is on, and logs `[RESULTS-DB] N row(s) saved`.
|
||||
- `run_mode`: `TEST` when `TEST_CLAIM_IDS_FILE` is set, else `LIVE-WRITE`
|
||||
when `SAVE_CORRECTIONS` is on, else `VERIFY` — the same words as the
|
||||
`[MODE]` banner.
|
||||
|
||||
## Web page — `web/`
|
||||
|
||||
- `web/app.py` (FastAPI) and `web/templates/index.html` (Jinja2), plain
|
||||
HTML/CSS, no external scripts or CDNs.
|
||||
- Read-only: only `SELECT` statements, parameterized.
|
||||
- `GET /` — page:
|
||||
- run picker (distinct `run_at`, newest first; default latest run),
|
||||
- Category filter (All / Pass / Fail / Review) and Claim# search,
|
||||
- counts per Category for the selected run (Pass / Fail / Review tiles),
|
||||
- the table in Charge Summary column order, Category coloured,
|
||||
- each row expands to show `details_json` (conditions fired, before/after
|
||||
changes, review reasons).
|
||||
- `GET /api/runs` — runs with counts. `GET /api/results?run_at=&category=&claim_no=`
|
||||
— rows as JSON (for the later portal).
|
||||
- Start: `python -m uvicorn web.app:app --host 127.0.0.1 --port 8000`.
|
||||
Bound to **127.0.0.1 only**: the page shows PHI and is not reachable from
|
||||
the network. A `run_web.bat` helper does the same.
|
||||
- The DB connection string is the same `RESULTS_DB_CONN`. If the DB cannot be
|
||||
reached the page shows a plain error message, not a stack trace.
|
||||
|
||||
## Error handling
|
||||
|
||||
- DB unreachable / permission denied / table missing during a run → one log
|
||||
line, run continues, Excel written.
|
||||
- A value that does not convert → NULL for that column, row still saved.
|
||||
- Web page: DB errors → HTTP 503 page "Results database not reachable".
|
||||
|
||||
## Testing (no real DB)
|
||||
|
||||
- `tests/test_results_db.py`: header→column map covers all 24 HEADERS;
|
||||
money/date conversion (incl. `$1,234.50`, blank, junk); `to_record` output;
|
||||
`insert_rows` sends one parameterized statement with values in column
|
||||
order (fake connection records calls); `save_summary` swallows a connect
|
||||
error and returns None; `details_json` shape.
|
||||
- `tests/test_charge_summary.py`: `build()` + `write()` split keeps today's
|
||||
output.
|
||||
- `tests/test_web_app.py`: FastAPI `TestClient` with a fake repository —
|
||||
page renders rows and Category counts, filters work, `/api/results` JSON
|
||||
shape, DB error → 503.
|
||||
- Full suite stays green. Live: operator runs `create_tables.sql`, sets
|
||||
`RESULTS_DB=1`, one test run, then opens the page.
|
||||
|
||||
## Out of scope (later portal)
|
||||
|
||||
Login/roles, editing rows, charts beyond the Category tiles, scheduling,
|
||||
AI features, hosting on the network.
|
||||
@@ -0,0 +1,176 @@
|
||||
# Start the bot from the results web page — design
|
||||
|
||||
Date: 2026-10-14. Approved section by section by the operator in chat
|
||||
(approach A, sections 1–3).
|
||||
|
||||
## Goal
|
||||
|
||||
The operator starts a Charge Validation run from the local results page
|
||||
(`run_web.bat` → http://127.0.0.1:8000) instead of the terminal. While it
|
||||
runs, the page shows live status; it can stop it; when it ends, the page
|
||||
shows that run's results.
|
||||
|
||||
## Decisions (operator, 2026-10-14)
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| Controls | Start + live status + Stop (no settings editing on the page) |
|
||||
| Save mode | **VERIFY only.** A page-started run is always `SAVE_CORRECTIONS=0`, whatever `.env` says. LIVE-WRITE stays terminal-only. |
|
||||
| Message boxes | **Off** for the run's information boxes (per-claim, review-before-close, "Run the Lookup?", summaries). Boxes that need a person to act in the browser — CAPTCHA, 2FA manual step, security image — **stay on**. |
|
||||
| Approach | A: the web server starts `run.py` as a separate child process (not a thread, not Task Scheduler). |
|
||||
|
||||
Out of scope (YAGNI): editing settings from the page, scheduling, more than
|
||||
one run at a time, LIVE-WRITE from the page.
|
||||
|
||||
## Components
|
||||
|
||||
### New: `web/bot_runner.py` — process control only
|
||||
|
||||
- `start() -> dict` — starts `python run.py` (the project's own `run.py`,
|
||||
`sys.executable`, cwd = project root) as a child process with no console
|
||||
window of its own (`CREATE_NO_WINDOW`); the Chrome window still shows.
|
||||
The child's stdout/stderr go to `Logs\web_run_<ts>.out`, which catches a
|
||||
crash before the bot opens its own log; the bot's `Logs\run_<ts>.log`
|
||||
(named in the lock file) is the status source.
|
||||
Refuses (raises `AlreadyRunning`) when the lock file names a live pid.
|
||||
- `stop() -> dict` — `taskkill /PID <pid> /T /F`: the bot and its Chrome
|
||||
close together. Clears the lock file. Status becomes "Stopped by
|
||||
operator".
|
||||
- `status() -> dict` — `running` (bool), `pid`, `started_at`, `log_path`,
|
||||
`exit_code` (when this web process started it and it has ended),
|
||||
`stopped_by_operator`.
|
||||
- The command is fixed. Nothing from the request reaches the command line
|
||||
or the environment.
|
||||
|
||||
**Child environment** — set on the child process, so it wins over `.env`
|
||||
(`settings.py` loads `.env` with `os.environ.setdefault`):
|
||||
|
||||
```
|
||||
SAVE_CORRECTIONS=0
|
||||
ECW_KEEP_OPEN=0 # no "Press Enter to close" at the end
|
||||
QUIET_BOXES=1 # run-flow information boxes go to the log
|
||||
CONFIRM_BEFORE_LOOKUP=0
|
||||
```
|
||||
|
||||
Everything else follows `.env` (claim file, `CLAIM_PROCESS_LIMIT`,
|
||||
`RESULTS_DB`, proxy, ...).
|
||||
|
||||
### Lock file: `Logs\bot_run.json`
|
||||
|
||||
`{"pid", "started_at", "log_path", "source": "web" | "terminal"}`.
|
||||
|
||||
- Written by the **bot itself** at the start of `main()` (after
|
||||
`runlog.start()`, so `log_path` is known) and removed when `main()`
|
||||
returns — so a terminal run is seen by the page too.
|
||||
- Before writing it, `main()` refuses to start (exit 2, before login) when
|
||||
the file names a pid that is alive: two runs would share one eCW login
|
||||
guard and one Charge Summary.
|
||||
- A file naming a dead pid is stale: removed and ignored (a killed run
|
||||
cannot remove its own file).
|
||||
- `bot_runner.start()` checks it too, so the page answers 409 at once.
|
||||
|
||||
### New: `web/run_status.py` — read the run log
|
||||
|
||||
Pure functions over log text (no I/O except reading the file tail):
|
||||
|
||||
- **Phase** from the last matching line:
|
||||
`[LOGIN]` / `[CAPTCHA]` → "Logging in"; `[OTP]` / `[2FA]` → "Waiting for
|
||||
the 2FA approval mail"; `[OK] Logged in` → "Logged in"; `[CLAIMS]` →
|
||||
"Checking claims"; process ended → "Finished" or a failure text by exit
|
||||
code.
|
||||
- **Progress** from `[CLAIMS] --- claim n/N: <id> ---` → `n`, `N`, current
|
||||
claim id.
|
||||
- **Counts** from the new `[RESULT]` lines (below) → Pass / Fail / Review.
|
||||
- **Tail**: the last 30 lines.
|
||||
- **Exit codes** (from `flows/export_medcor_ready.main`): 0 finished; 2
|
||||
config / mode gate; 3 login guard refused (its `[GUARD]` line shown); 5
|
||||
claims phase failed; 6 2FA timed out; other → "ended with code N".
|
||||
|
||||
### Changed: `flows/export_medcor_ready.py`
|
||||
|
||||
- After each claim's outcome is known in `_process_claims`, one
|
||||
machine-readable line (terminal runs print it too):
|
||||
|
||||
```
|
||||
[RESULT] claim 3/7 9126000: Review
|
||||
```
|
||||
|
||||
The word is the Category the Charge Summary gets for that claim; a claim
|
||||
that gets no Charge Summary row (not found in the grid) prints
|
||||
`Not found`.
|
||||
- `main()` writes / removes the lock file as above.
|
||||
|
||||
### Changed: `core/msgbox.py`
|
||||
|
||||
- New setting `QUIET_BOXES` (default off; `settings.py`).
|
||||
- `info_box(title, msg, needs_person=False)`: with `QUIET_BOXES=1` and
|
||||
`needs_person=False`, no box — one log line instead:
|
||||
`[BOX] <title>: <first line of msg>`.
|
||||
- `ok_cancel` / `confirm` are unchanged (every remaining caller is a
|
||||
person-needed step: CAPTCHA, 2FA manual, LIVE-WRITE confirm — the last
|
||||
is never reached in VERIFY). The security-image `info_box` in
|
||||
`auth/login.py` passes `needs_person=True`.
|
||||
|
||||
### Changed: `web/app.py` + `web/templates/index.html`
|
||||
|
||||
Endpoints:
|
||||
|
||||
| Method | Path | Answer |
|
||||
|---|---|---|
|
||||
| POST | `/api/bot/start` | 200 `{status}`; 409 already running; 403 bad Origin/token |
|
||||
| POST | `/api/bot/stop` | 200 `{status}`; 409 nothing running; 403 bad Origin/token |
|
||||
| GET | `/api/bot/status` | `{running, phase, progress, counts, tail, started_at, exit_code, message}` |
|
||||
|
||||
Page: a "Bot run" card above the results.
|
||||
- Idle: **[Start run]** and "VERIFY mode — nothing is saved to eCW".
|
||||
- Running (polls `/api/bot/status` every 2 s): phase, progress bar
|
||||
"3 / 7", current claim, Pass/Fail/Review chips, collapsible "Live log"
|
||||
(last 30 lines), **[Stop]** (with a confirm dialog).
|
||||
- Ended: "Finished in 4m 12s — exit 0" (or the failure text); the results
|
||||
table reloads on the newest run.
|
||||
|
||||
## Security
|
||||
|
||||
The page was read-only; Start/Stop are its first actions. The existing
|
||||
TrustedHost check stops DNS rebinding but not a cross-site POST from
|
||||
another tab, which could burn eCW login attempts. So every POST needs both:
|
||||
|
||||
1. `Origin` (or, when absent, `Referer`) is `http://127.0.0.1:8000` or
|
||||
`http://localhost:8000` — else 403.
|
||||
2. Header `X-Run-Token` equals a token made at server start
|
||||
(`secrets.token_urlsafe(32)`) and embedded in the page — else 403.
|
||||
Compared with `secrets.compare_digest`.
|
||||
|
||||
No password on the page; `.env` credentials are read only by the bot.
|
||||
Responses keep `no-store`. The page stays bound to 127.0.0.1. The live log
|
||||
tail can carry PHI, as the results table already does — local only.
|
||||
|
||||
## Error handling
|
||||
|
||||
- Already running (page or terminal) → 409 "A run is already going
|
||||
(started HH:MM)".
|
||||
- Guard refused (exit 3) → the `[GUARD]` line shown ("next attempt
|
||||
allowed at ...").
|
||||
- Stop mid-claim: the claim form may stay open in eCW. VERIFY saves
|
||||
nothing; the next run's `clear_screen` closes it.
|
||||
- Web server restarted or page closed: the bot keeps running; on reload
|
||||
the page finds it through the lock file. `exit_code` is unknown then
|
||||
(the run is shown "Ended" when its pid is gone).
|
||||
- Log file not there yet → phase "Starting".
|
||||
|
||||
## Testing (standalone "N/M passed" scripts)
|
||||
|
||||
- `tests/test_bot_runner.py` — a fake child script in place of `run.py`
|
||||
(injectable command): start → running → stop → stopped; second start →
|
||||
`AlreadyRunning`; stale lock (dead pid) cleared; the child sees the forced
|
||||
env (`SAVE_CORRECTIONS=0`, `ECW_KEEP_OPEN=0`, `QUIET_BOXES=1`,
|
||||
`CONFIRM_BEFORE_LOOKUP=0`) even when the parent env says otherwise.
|
||||
- `tests/test_run_status.py` — sample log text → phase, progress, counts,
|
||||
tail; each exit code's text.
|
||||
- `tests/test_web_app.py` (added cases) — POST without/with wrong Origin or
|
||||
token → 403; right Origin + token → 200 (runner stubbed); status JSON
|
||||
shape; the token is in the page.
|
||||
- `tests/test_msgbox.py` (added cases) — `QUIET_BOXES=1`: info box logged,
|
||||
not shown; `needs_person=True` still shown.
|
||||
- Flow test — `[RESULT]` line per claim; `main()` lock file written,
|
||||
refused on a live one, removed at the end.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Results portal redesign — design
|
||||
|
||||
Date: 2026-10-19. Operator requests (voice, Tanglish), choices made on the
|
||||
visual companion mockups (`.superpowers/brainstorm/828-1792395410/`).
|
||||
|
||||
## Problem
|
||||
|
||||
The results page (`web/`, FastAPI + Jinja, 127.0.0.1:8000) opens on the
|
||||
newest run only. A second run (20 claims) hides the first (10 claims) behind
|
||||
a "Run" dropdown nobody noticed, so earlier runs look lost. The front page
|
||||
is three quarters claims table (24 columns, sideways slider), which the
|
||||
operator does not want on the home page. The look is "very basic".
|
||||
|
||||
## What the operator chose
|
||||
|
||||
| Question | Choice |
|
||||
|---|---|
|
||||
| Home page | **Dashboard**: Bot run card (Start / Stop / live progress), today's tiles, one big **Claim status** button |
|
||||
| Claim status page | **Date cards**, newest first |
|
||||
| A date with several runs | **All claims of the day together** (every run's rows in one list) |
|
||||
| One day's page | **Short table + side panel** |
|
||||
| Colours | **Navy & blue** (navy header, white cards on light grey, blue buttons) |
|
||||
| Live run's claim list | In the **Bot run card** ("Live claims" side panel) |
|
||||
| Search one claim over all dates | **Yes** — "Find a claim" on the Dashboard |
|
||||
| Approach | **A** — server pages (FastAPI + Jinja), one page per screen |
|
||||
|
||||
## Pages
|
||||
|
||||
All pages share `base.html`: navy header with the title, a mode pill
|
||||
(VERIFY / LIVE-WRITE from `SAVE_CORRECTIONS`) and a breadcrumb
|
||||
(Dashboard › Claim status › 16 Oct 2026). Light grey background, white
|
||||
cards, blue buttons; Pass green, Fail red, Review amber everywhere.
|
||||
|
||||
1. **Dashboard `/`**
|
||||
- Bot run card — the existing Start / Stop / progress / live log, plus a
|
||||
**Live claims** link opening a side panel with every claim of the
|
||||
running (or last) run: Waiting / Checking / result. A claim with a
|
||||
saved result opens `/day/<date of its run>?claim=<no>`.
|
||||
- Today tiles: claims checked today, Pass, Fail, Review (0 when no run
|
||||
today).
|
||||
- **Find a claim** box → `/claim?claim_no=…`.
|
||||
- Big **Claim status →** button → `/days`.
|
||||
- No claims table.
|
||||
2. **Claim status `/days`** — one card per day (last 90 days with a run),
|
||||
newest first: weekday + date, claims checked, number of runs,
|
||||
Pass / Fail / Review and a thin colour bar. A card opens `/day/<date>`.
|
||||
3. **One day `/day/YYYY-MM-DD`**
|
||||
- Tiles All / Pass / Fail / Review; a tile is a filter link
|
||||
(`?category=Fail`). Claim # search box (`?claim_no=`).
|
||||
- Short table, 10 rows a page (client-side pager): #, Claim #, Time
|
||||
(the run's time), Payer, Result chip, Comments (one line, cut with
|
||||
an ellipsis). No sideways scroll.
|
||||
- Row click → side panel with every Charge Summary field, the run's
|
||||
time and mode, and the conditions/changes JSON.
|
||||
- `?claim=<no>` opens that claim's panel on load (on its page).
|
||||
4. **Claim history `/claim?claim_no=…`** — every saved check of claims
|
||||
matching the number (newest first, up to 200): date, time, mode,
|
||||
result, comments; each row opens `/day/<date>?claim=<no>`.
|
||||
|
||||
`/?run_at=<time>` (old links) redirects (303) to `/day/<date of that run>`.
|
||||
`/api/runs`, `/api/results`, `/api/bot/*` stay as they are.
|
||||
|
||||
## Data
|
||||
|
||||
No schema change. `web/repo.py` gains three parameterized SELECTs:
|
||||
|
||||
- `days(limit=90)` — `GROUP BY CAST(run_at AS DATE)`: day, claims,
|
||||
runs (`COUNT(DISTINCT run_at)`), pass / fail / review counts, newest
|
||||
first.
|
||||
- `day_results(day, category, claim_no)` — rows with
|
||||
`run_at >= day AND run_at < day + 1`, ordered by run_at, S.No, id.
|
||||
- `claim_history(claim_no)` — `TOP 200 … WHERE claim_no LIKE ?` ordered
|
||||
by run_at DESC.
|
||||
|
||||
Rules: a run belongs to the day it **started** (every row of a run shares
|
||||
its `run_at`), so a run never splits over two cards. The same claim
|
||||
checked twice in a day shows twice, each with its time; a day's count is
|
||||
checks. "Not found" claims have no saved row (as today) and appear only in
|
||||
the live panel. A legacy driver returning the day as text is coerced to a
|
||||
date, like `run_at` already is.
|
||||
|
||||
## Unchanged
|
||||
|
||||
127.0.0.1 only; Host allowlist; `no-store` and the other headers; DB
|
||||
errors reported by kind; Start / Stop need the page's Origin and token;
|
||||
page runs are VERIFY only. All new pages are read-only. Page text from the
|
||||
DB goes out escaped (Jinja autoescape; the panel's data as `tojson` in an
|
||||
attribute, rendered with `textContent`).
|
||||
|
||||
## Testing
|
||||
|
||||
`tests/test_web_app.py` (fake repo, fake DB connection): each page renders
|
||||
with its data and links; filters and the bad-input 400s; the redirect;
|
||||
the three new SQL statements are parameterized and shaped as above; the
|
||||
hardening, API and Start / Stop cases kept. A browser check of the four
|
||||
pages on fixture data (Playwright, the in-app browser).
|
||||
@@ -0,0 +1,161 @@
|
||||
# Run tracking and Portal Sign-in — design
|
||||
|
||||
Date: 2026-10-22. Agreed with the operator in chat, question by question
|
||||
(Q1–Q31, all "yes" to the recommended answers; Q3 changed to "who" = the
|
||||
Portal User). Terms: `GLOSSARY.md`. Decision record:
|
||||
`docs/adr/0001-run-events-in-files.md`. Jira: EEWA-31 (Stage 1),
|
||||
EEWA-32 (Stage 2), EEWA-33 (Stage 3), each blocking the next.
|
||||
|
||||
## Goal
|
||||
|
||||
See every Run on the Portal: who started it, how long the eCW login took and
|
||||
where the time went, every Run Error, Claim Error and Warning with its time,
|
||||
and which claims a Run never reached. Only signed-in Portal Users see the
|
||||
Portal; every Run records who started (and stopped) it.
|
||||
|
||||
## Decisions
|
||||
|
||||
| # | Decision |
|
||||
|---|---|
|
||||
| Q1 | Three kinds: **Run Error** (the Run stopped), **Claim Error** (one claim not checked), **Warning** (nothing stopped). **Fail** stays a Charge Summary category only. |
|
||||
| Q2/Q28 | Per **eCW Login Attempt**: start, end, duration, 2FA mail wait, result; steps timed: page open, password typed, captcha, 2FA mail requested (the Log In click), mail received, approval clicked, home page. |
|
||||
| Q3/Q8 | "Who" = the **Portal User** who signed in; **Started By** / **Stopped By**. |
|
||||
| Q4 | Events go to `Logs\events_<run>.jsonl` (ADR 0001). |
|
||||
| Q5/Q25 | Old run logs are not imported. Nothing is deleted. |
|
||||
| Q6/Q18/Q19/Q22 | A **Runs** page on the Portal: list + detail, detail refreshes every 5 s while its Run is going. |
|
||||
| Q9 | The Portal stays on 127.0.0.1. |
|
||||
| Q10–Q17, Q20, Q21 | Sign-in (Stage 2, below). |
|
||||
| Q14 | Terminal Runs are allowed; Started By = the Windows user. |
|
||||
| Q15/Q23 | Stage 3: Run Error banner; `started_by` column in `CV_ChargeSummary`; the Excel sheet unchanged. |
|
||||
| Q24 | Disk check at Run start: system drive under 2 GB = Warning, under 500 MB = Run Error, exit 2, before login. |
|
||||
| Q26 | Claim Error = search raised / form not read / never finished loading; Charge Summary still says Review. A claim not found in eCW = Warning. |
|
||||
| Q27 | **Not Reached** = queued claims a Run Error (or Stop) came before. |
|
||||
|
||||
## Stage 1 — Run events and the Runs page (EEWA-31)
|
||||
|
||||
### `core/run_events.py` (new)
|
||||
|
||||
- `start(log_path)` — the events file is the log's name with `run_` →
|
||||
`events_` and `.log` → `.jsonl` (`run_20261022_141530.log` →
|
||||
`events_20261022_141530.jsonl`); the Run's id is `20261022_141530`.
|
||||
Writes `run_start`.
|
||||
- `emit(kind, **fields)` — one JSON line, `{"t": local ISO time to the
|
||||
millisecond, "kind": kind, ...}`, flushed at once. **Never raises**: an
|
||||
event that cannot be written is dropped, the Run carries on.
|
||||
- Helpers: `warning(code, message, **f)`, `run_error(step, message,
|
||||
exit_code, exc=None)` (with `traceback.format_exc()` when an exception is
|
||||
given), `claim_error(claim, step, message)`, `login_step(step, **f)`,
|
||||
`login_attempt_start()` / `login_attempt_end(result, message="")` (the
|
||||
module numbers attempts).
|
||||
- No OTP code, password or eCW username is ever put in an event.
|
||||
|
||||
### Event kinds
|
||||
|
||||
| kind | fields |
|
||||
|---|---|
|
||||
| `run_start` | `source` (portal / terminal), `started_by`, `mode` (VERIFY / LIVE-WRITE), `claim_limit`, `test_sheet` (file name only), `pid`, `log` |
|
||||
| `disk` | `drive`, `free_mb` |
|
||||
| `login_attempt_start` | `attempt` |
|
||||
| `login_step` | `attempt`, `step`: `page_open`, `password`, `captcha_shown`, `captcha_solved`, `captcha_failed`, `twofa_requested`, `mail_received`, `approval_clicked`, `home` |
|
||||
| `login_attempt_end` | `attempt`, `result`: `ok`, `twofa_timeout`, `captcha_failed`, `restricted`, `failed`; `message` |
|
||||
| `queue` | `claims` (list) |
|
||||
| `claim_start` | `n`, `total`, `claim` |
|
||||
| `claim_result` | `n`, `claim`, `category` (Pass / Fail / Review / Not found) |
|
||||
| `claim_error` | `claim`, `step` (`search` / `form` / `load`), `message` |
|
||||
| `warning` | `code` (`not_found`, `fallback`, `dialog`, `cancel_retry`, `form_not_closed`, `disk_low`, `preflight`), `message`, optional `claim` |
|
||||
| `screenshot` | `path` (a failure shot), optional `claim` |
|
||||
| `run_error` | `step` (`config`, `disk`, `guard`, `login`, `restricted`, `twofa_timeout`, `preflight`, `claims`, `unexpected`), `exit_code`, `message`, `traceback` |
|
||||
| `run_end` | `exit_code` |
|
||||
| `stopped` | `stopped_by` (written by the Portal after Stop; Stage 2 fills the name) |
|
||||
|
||||
`started_by`: env `ECW_STARTED_BY` (the Portal sets it, Stage 2), else the
|
||||
Windows user. `source`: as `bot_run.json` has it (`ECW_RUN_SOURCE`).
|
||||
|
||||
### Where the bot emits
|
||||
|
||||
- `flows/export_medcor_ready.py`: `main()` starts the events after
|
||||
`runlog.start()` and writes `run_end` in its `finally`; `_main` writes a
|
||||
`run_error` at every non-zero return and in every `except`, with the
|
||||
traceback; the disk check sits before login next to the CDP check;
|
||||
`_login_with_retry` wraps each `login()` in attempt start / end;
|
||||
`_process_claims` writes `queue`, `claim_start`, `claim_result`,
|
||||
`claim_error` (search raised, `_error`, `_not_evaluated`) and the
|
||||
not-found Warning.
|
||||
- `auth/login.py` / `captcha.py` / `twofactor.py` / `mailbox.py`: the login
|
||||
steps.
|
||||
- `core/fallback.py` `report()`, `core/page_utils.py` `auto_dialog` (the
|
||||
event only — its behaviour is unchanged, see Context.md §10) and the
|
||||
failure `shot()`, `pages/claim_form.py` Cancel retries and form-not-closed,
|
||||
`pages/preflight.py` FAIL: Warnings / screenshots.
|
||||
- `web/bot_runner.py` `stop()`: appends `stopped` to the Run's events file.
|
||||
|
||||
### Runs page
|
||||
|
||||
- `web/runs.py` (new): reads `Logs\events_*.jsonl` into Run summaries and a
|
||||
Run's detail. A Run with no `run_end` is **Running** when `bot_run.json`
|
||||
names it and its pid is alive, **Stopped** when a `stopped` event is
|
||||
there, else **Ended without a record** (killed). Result: Done (exit 0),
|
||||
Run Error (exit and step), Stopped, Running.
|
||||
- `GET /runs` — newest first: start, Started By, Run Source, mode,
|
||||
duration, result, eCW Login Attempts + total login time, Pass / Fail /
|
||||
Review, Claim Errors, Warnings. Filters: from / to date, result.
|
||||
- `GET /runs/{run_id}` — the login attempts table (each step's time, 2FA
|
||||
wait = mail received − 2FA requested), Run Errors (traceback folded),
|
||||
Claim Errors, Warnings, Not Reached claims, the timeline, screenshot
|
||||
links; refreshes every 5 s while Running.
|
||||
- `GET /runs/{run_id}/shot/{n}` — the Run's n-th screenshot event, served
|
||||
only from inside `Screenshots\` and only as `.png`: a saved page's HTML
|
||||
served from the Portal's origin would run eCW's scripts there, so the
|
||||
detail page names where the DOM file is instead of linking it.
|
||||
- A "Runs" link on every page's header.
|
||||
|
||||
### Disk check
|
||||
|
||||
Settings `DISK_WARN_MB=2048`, `DISK_STOP_MB=500`, on the system drive.
|
||||
|
||||
## Stage 2 — Sign-in (EEWA-32)
|
||||
|
||||
- `CV_PortalUser` (username, display name, role, PBKDF2-SHA256 hash + salt
|
||||
+ iterations, must_change, disabled, created / changed) and
|
||||
`CV_PortalSignIn` (time, username, event: sign_in / sign_out /
|
||||
wrong_password / locked / timeout) in `Charge_Validation`.
|
||||
- Roles Admin / Operator / Viewer. Start / Stop: Admin, Operator. Users
|
||||
page and Sign-in history: Admin.
|
||||
- `tools\add_portal_user.py` (password with `getpass`). Passwords ≥ 10
|
||||
characters; change own; Admin reset → must change at next Sign-in;
|
||||
disable, never delete.
|
||||
- 5 wrong passwords → 15 minutes locked. Session cookie (HttpOnly,
|
||||
SameSite=Strict), 30 minutes idle. Every page and `/api` behind it;
|
||||
127.0.0.1 + Origin / token checks stay.
|
||||
- The Portal passes the signed-in user to the child as `ECW_STARTED_BY`;
|
||||
`stop()` writes `stopped_by`. `bot_run.json` keeps `started_by` too, so
|
||||
the Dashboard's run card says "started by …".
|
||||
- As built (`web/users.py`, `web/app.py`): sessions in the Portal's memory
|
||||
(a restart signs everyone out); disabling a user, resetting their
|
||||
password or changing their role ends / updates their open session at
|
||||
once; an unknown username reads and costs the same as a wrong password;
|
||||
the last active Admin cannot be demoted or disabled, nobody disables
|
||||
themselves; every Admin change is a `CV_PortalSignIn` row with `by_user`;
|
||||
every form POST carries the page token and is checked for Origin;
|
||||
`next` after Sign-in never leaves the site.
|
||||
|
||||
## Stage 3 — Alerts and `started_by` (EEWA-33)
|
||||
|
||||
- Dashboard banner when the last Run ended in a Run Error.
|
||||
- `CV_ChargeSummary.started_by` (nullable column, written from the Run's
|
||||
Started By).
|
||||
- As built: the banner also shows for a Run that ended without a record
|
||||
(killed or crashed — no `run_end`), never for Done, Stopped or Running;
|
||||
it names when, Started By, the step, exit code and message, how many
|
||||
claims were Not Reached, and links the Run. `create_tables.sql` adds the
|
||||
column to an existing table (`IF COL_LENGTH ... IS NULL ALTER TABLE`).
|
||||
The bot checks the column before each insert (`results_db.has_column`)
|
||||
and saves the rows without it while it is missing — a table not yet
|
||||
updated never costs a Run its rows. The Portal selects it, shows it as
|
||||
"Started By" in a claim's side panel, and a table without it reads
|
||||
"Results table needs an update — run create_tables.sql" (SQLSTATE 42S22).
|
||||
|
||||
## Out of scope
|
||||
|
||||
Network access to the Portal, email / Teams alerts, importing old logs,
|
||||
deleting old events, editing settings from the Portal.
|
||||
Reference in new issue
Block a user