Initial import: eCW Charge Validation bot and results Portal
Snapshot of the working project as of 2026-10-23 (local 2d4d4fc), one commit with no earlier history: the tracked files were de-identified first (synthetic claim ids and patient data only; see Context.md section 7). Credentials, run logs, screenshots, claim reports and client workbooks are gitignored and not included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
ManikandanandClaude Opus 5.5 committed
commit
ba15a574ab
205 files changed
+56772
No files matched your search
@@ -0,0 +1,910 @@
|
||||
# Payer Group Categorization Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** After the bot exports the Medcor Ready claim report, automatically add a `Group Name` column (Commercial / Medicaid / Medicare / Dentaquest / Program / …) based on each row's `PAYER`, writing the result to a new file so the existing pipeline's file layout is untouched.
|
||||
|
||||
**Architecture:** A new `core/payer_groups.py` does an exact-string lookup against a hand-editable `payer_groups.json`. A new `core/spreadsheet.write_xlsx()` (stdlib only, mirroring the existing reader) writes the categorized rows to `Output\Medcor Ready_<date>.xlsx`. `flows/export_medcor_ready.py` wires the two together as a new, non-fatal step right after the existing claim-id read-back.
|
||||
|
||||
**Tech Stack:** Python 3.13, stdlib only (`zipfile`, `xml.etree`, `json`, `xml.sax.saxutils`) — no new third-party dependencies, matching the project's existing "no openpyxl" constraint.
|
||||
|
||||
**Spec:** [docs/superpowers/specs/2026-09-15-payer-group-categorization-design.md](../specs/2026-09-15-payer-group-categorization-design.md)
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- No new third-party dependencies (project has none beyond `playwright`; `spreadsheet.py`'s whole existence is to avoid `openpyxl`).
|
||||
- Tests are hand-rolled `case_*` functions collected via `globals()` and run by a `main()` — no pytest, matching every existing file under `tests\`.
|
||||
- `Input\Claim Report_<date>.xlsx` must never be modified by this feature — `claim_file.py` and the charge-entry phase depend on its exact column layout.
|
||||
- Payer matching is **exact string, case-sensitive, no fuzzy/substring logic** — an unmapped payer must be flagged, never guessed (see spec's "Unknown payer handling").
|
||||
- Every new step must be non-fatal to the run — matches the existing philosophy in `flows/export_medcor_ready.py` (`_read_back_claim_ids`, `_read_claim_form`).
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Initialize git
|
||||
|
||||
This project currently has no git history at all (verified: `git status` outside a repo). The plan's commit-per-task workflow needs a repo to commit into, so this task creates one and takes a single baseline commit of the existing code before any feature work starts. This is additive only — no existing file is changed.
|
||||
|
||||
**Files:**
|
||||
- Create: `D:\Ecw Process\.git\` (via `git init`)
|
||||
|
||||
**Interfaces:** none — infrastructure only.
|
||||
|
||||
- [ ] **Step 1: Confirm there is no existing git repo**
|
||||
|
||||
Run: `git -C "D:\Ecw Process" status`
|
||||
Expected: `fatal: not a git repository (or any of the parent directories): .git`
|
||||
|
||||
- [ ] **Step 2: Initialize the repository**
|
||||
|
||||
Run: `git -C "D:\Ecw Process" init`
|
||||
Expected: `Initialized empty Git repository in D:/Ecw Process/.git/`
|
||||
|
||||
- [ ] **Step 3: Check `.gitignore` already covers the right things**
|
||||
|
||||
Run: `cat "D:\Ecw Process\.gitignore"`
|
||||
Expected: it already excludes `.env`, `Logs\`, `Screenshots\`, `Output\`, `Input\` (per `README.md`'s Layout section, all four are described as gitignored). If any of those four is missing, add it before continuing — this baseline commit must not include patient data or credentials.
|
||||
|
||||
- [ ] **Step 4: Stage and commit the current codebase as a baseline**
|
||||
|
||||
```bash
|
||||
cd "D:\Ecw Process"
|
||||
git add -A
|
||||
git status
|
||||
```
|
||||
|
||||
Read the `git status` output before committing: confirm nothing under `Input\`, `Output\`, `Logs\`, `Screenshots\`, or `.env` is staged. If any of those appear, fix `.gitignore` and re-run `git add -A`.
|
||||
|
||||
```bash
|
||||
git commit -m "chore: initialize git repository (baseline)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: `write_xlsx()` in `core/spreadsheet.py`
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ecw_bot/core/spreadsheet.py`
|
||||
- Test: `tests/test_spreadsheet_write.py` (new)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `spreadsheet.write_xlsx(path, rows: list[list]) -> None` — `rows` is a header row plus data rows, each a list of values (any type, written as `str()`). Raises on I/O failure. Creates parent directories if needed.
|
||||
- Consumes: nothing new — sits next to the existing `read_rows(path) -> list`.
|
||||
|
||||
- [ ] **Step 1: Write the failing round-trip test**
|
||||
|
||||
Create `tests/test_spreadsheet_write.py`:
|
||||
|
||||
```python
|
||||
"""
|
||||
write_xlsx(): the write side of core/spreadsheet.py.
|
||||
|
||||
python tests\\test_spreadsheet_write.py
|
||||
|
||||
Round-trips through the existing `read_rows()` reader rather than asserting
|
||||
on raw XML — what matters is that a file this writes is a file this project's
|
||||
own reader (and real Excel) can open, not the exact bytes.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import pathlib
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent / "src"))
|
||||
|
||||
from ecw_bot.core import spreadsheet # noqa: E402
|
||||
|
||||
TMP = pathlib.Path(tempfile.mkdtemp())
|
||||
|
||||
|
||||
def case_write_then_read_round_trips():
|
||||
rows = [
|
||||
["CLAIMS#", "PAYER", "Group Name"],
|
||||
["9107000", "Medicaid", "Medicaid"],
|
||||
["9106000", "DentaQuest", "Dentaquest"],
|
||||
]
|
||||
path = TMP / "roundtrip.xlsx"
|
||||
spreadsheet.write_xlsx(path, rows)
|
||||
back = spreadsheet.read_rows(path)
|
||||
assert back == rows, back
|
||||
|
||||
|
||||
def case_special_characters_are_escaped():
|
||||
rows = [["PATIENT", "NOTE"],
|
||||
["Smith & Sons", "A < B > C"],
|
||||
["O'Brien", '"quoted"']]
|
||||
path = TMP / "escaped.xlsx"
|
||||
spreadsheet.write_xlsx(path, rows)
|
||||
back = spreadsheet.read_rows(path)
|
||||
assert back == rows, back
|
||||
|
||||
|
||||
def case_wide_row_uses_correct_column_letters():
|
||||
"""27 columns exercises the single-letter -> AA rollover."""
|
||||
header = [f"col{i}" for i in range(27)]
|
||||
data = [str(i) for i in range(27)]
|
||||
path = TMP / "wide.xlsx"
|
||||
spreadsheet.write_xlsx(path, [header, data])
|
||||
back = spreadsheet.read_rows(path)
|
||||
assert back[1][26] == "26", back[1] # column AA (index 26)
|
||||
assert back[1] == data, back[1]
|
||||
|
||||
|
||||
def case_creates_parent_directory():
|
||||
path = TMP / "nested" / "new_dir" / "file.xlsx"
|
||||
assert not path.parent.exists()
|
||||
spreadsheet.write_xlsx(path, [["A"], ["1"]])
|
||||
assert path.exists()
|
||||
|
||||
|
||||
def case_non_string_values_are_stringified():
|
||||
rows = [["N"], [5], [3.5]]
|
||||
path = TMP / "numbers.xlsx"
|
||||
spreadsheet.write_xlsx(path, rows)
|
||||
back = spreadsheet.read_rows(path)
|
||||
assert back == [["N"], ["5"], ["3.5"]], back
|
||||
|
||||
|
||||
CASES = [v for k, v in sorted(globals().items()) if k.startswith("case_")]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
failures = 0
|
||||
for fn in CASES:
|
||||
name = fn.__name__[5:].replace("_", " ")
|
||||
try:
|
||||
fn()
|
||||
print(f" ok {name}")
|
||||
except AssertionError as exc:
|
||||
print(f" FAIL {name}: {exc}")
|
||||
failures += 1
|
||||
except Exception as exc:
|
||||
print(f" ERROR {name}: {type(exc).__name__}: {exc}")
|
||||
failures += 1
|
||||
print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run it to verify it fails**
|
||||
|
||||
Run: `python tests\test_spreadsheet_write.py`
|
||||
Expected: `ERROR` on every case with `AttributeError: module 'ecw_bot.core.spreadsheet' has no attribute 'write_xlsx'`
|
||||
|
||||
- [ ] **Step 3: Implement `write_xlsx()` in `core/spreadsheet.py`**
|
||||
|
||||
Add these imports at the top of `src/ecw_bot/core/spreadsheet.py` (alongside the existing `import csv / re / zipfile` block):
|
||||
|
||||
```python
|
||||
from xml.sax.saxutils import escape
|
||||
```
|
||||
|
||||
Append this to the end of `src/ecw_bot/core/spreadsheet.py`:
|
||||
|
||||
```python
|
||||
# =============================================================================
|
||||
# Writing (the categorization step's output)
|
||||
# =============================================================================
|
||||
# The read side above has no dependency on openpyxl; the write side keeps that
|
||||
# constraint too. What follows is the minimal set of OOXML parts real Excel
|
||||
# actually requires to open a file without a "repair" prompt — more than the
|
||||
# reader above needs (it only ever looks at xl/worksheets/sheet*.xml), because
|
||||
# this file is meant to be opened by a person, not just this codebase.
|
||||
|
||||
_CONTENT_TYPES = (
|
||||
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
|
||||
'<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
|
||||
'<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>'
|
||||
'<Default Extension="xml" ContentType="application/xml"/>'
|
||||
'<Override PartName="/xl/workbook.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml"/>'
|
||||
'<Override PartName="/xl/worksheets/sheet1.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml"/>'
|
||||
'</Types>'
|
||||
)
|
||||
|
||||
_ROOT_RELS = (
|
||||
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
|
||||
'<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
|
||||
'<Relationship Id="rId1" '
|
||||
'Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" '
|
||||
'Target="xl/workbook.xml"/>'
|
||||
'</Relationships>'
|
||||
)
|
||||
|
||||
_WORKBOOK = (
|
||||
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
|
||||
'<workbook xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main" '
|
||||
'xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships">'
|
||||
'<sheets><sheet name="Sheet1" sheetId="1" r:id="rId1"/></sheets>'
|
||||
'</workbook>'
|
||||
)
|
||||
|
||||
_WORKBOOK_RELS = (
|
||||
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
|
||||
'<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
|
||||
'<Relationship Id="rId1" '
|
||||
'Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" '
|
||||
'Target="worksheets/sheet1.xml"/>'
|
||||
'</Relationships>'
|
||||
)
|
||||
|
||||
|
||||
def _col_letter(idx: int) -> str:
|
||||
"""0 -> 'A', 25 -> 'Z', 26 -> 'AA'. Inverse of `_col_index` above."""
|
||||
letters = ""
|
||||
idx += 1
|
||||
while idx > 0:
|
||||
idx, rem = divmod(idx - 1, 26)
|
||||
letters = chr(65 + rem) + letters
|
||||
return letters
|
||||
|
||||
|
||||
def write_xlsx(path, rows: list) -> None:
|
||||
"""
|
||||
Write `rows` (a header row plus data rows, each a list of values) as a
|
||||
minimal single-sheet xlsx.
|
||||
|
||||
Inline `t="str"` cells, no sharedStrings — the same shape `_read_xlsx`
|
||||
above already parses, and what eCW's own export already sends (see the
|
||||
module docstring). Every value is written through `str()`, so callers can
|
||||
pass numbers or None without pre-formatting.
|
||||
"""
|
||||
path = Path(path)
|
||||
body = []
|
||||
for r, row in enumerate(rows, start=1):
|
||||
cells = []
|
||||
for c, value in enumerate(row):
|
||||
text = escape(str(value))
|
||||
cells.append(f'<c r="{_col_letter(c)}{r}" t="str"><v>{text}</v></c>')
|
||||
body.append(f'<row r="{r}">{"".join(cells)}</row>')
|
||||
sheet = (
|
||||
'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
|
||||
'<worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main">'
|
||||
f'<sheetData>{"".join(body)}</sheetData></worksheet>'
|
||||
)
|
||||
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
zf.writestr("[Content_Types].xml", _CONTENT_TYPES)
|
||||
zf.writestr("_rels/.rels", _ROOT_RELS)
|
||||
zf.writestr("xl/workbook.xml", _WORKBOOK)
|
||||
zf.writestr("xl/_rels/workbook.xml.rels", _WORKBOOK_RELS)
|
||||
zf.writestr("xl/worksheets/sheet1.xml", sheet)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the test to verify it passes**
|
||||
|
||||
Run: `python tests\test_spreadsheet_write.py`
|
||||
Expected: `5/5 passed`
|
||||
|
||||
- [ ] **Step 5: Sanity-check the file actually opens in Excel/WPS**
|
||||
|
||||
This machine has Excel/WPS installed (seen in the video). Manually:
|
||||
|
||||
```bash
|
||||
python -c "import sys; sys.path.insert(0,'src'); from ecw_bot.core import spreadsheet as s; s.write_xlsx('Output/_manual_check.xlsx', [['A','B'],['1','2']])"
|
||||
```
|
||||
|
||||
Open `Output\_manual_check.xlsx` by hand and confirm it opens with no "repair" prompt and shows the two cells. Delete the file afterward — it's a throwaway check, not a fixture.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
cd "D:\Ecw Process"
|
||||
git add src/ecw_bot/core/spreadsheet.py tests/test_spreadsheet_write.py
|
||||
git commit -m "feat: add write_xlsx() to core/spreadsheet.py"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: `core/payer_groups.py` + seed `payer_groups.json`
|
||||
|
||||
**Files:**
|
||||
- Create: `src/ecw_bot/core/payer_groups.py`
|
||||
- Create: `src/ecw_bot/payer_groups.json`
|
||||
- Test: `tests/test_payer_groups.py` (new)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `payer_groups.load_mapping(path=None) -> dict[str, str]`, `payer_groups.classify(payer: str, mapping: dict) -> str | None`.
|
||||
- Consumes: nothing from Task 2.
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Create `tests/test_payer_groups.py`:
|
||||
|
||||
```python
|
||||
"""
|
||||
Payer -> Group Name lookup.
|
||||
|
||||
python tests\\test_payer_groups.py
|
||||
|
||||
Exact-string match only, on purpose: eCW's own PAYER values already differ by
|
||||
case and spacing for what is really the same carrier (`United Healthcare` /
|
||||
`United Health Care` / `United healthcare` all appear in one real export), so
|
||||
"close enough" matching would have to guess which of several real, distinct
|
||||
strings a new one is closest to. Billing data does not get a guess — an
|
||||
unrecognized string must come back None and get a human's attention.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import json
|
||||
import pathlib
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent / "src"))
|
||||
|
||||
from ecw_bot.core import payer_groups # noqa: E402
|
||||
|
||||
TMP = pathlib.Path(tempfile.mkdtemp())
|
||||
|
||||
MAPPING = {
|
||||
"Medicaid": "Medicaid",
|
||||
"DentaQuest": "Dentaquest",
|
||||
"Anthem Blue Cross Blue Shield": "Commercial",
|
||||
}
|
||||
|
||||
|
||||
def case_exact_match_returns_group():
|
||||
assert payer_groups.classify("Medicaid", MAPPING) == "Medicaid"
|
||||
assert payer_groups.classify("DentaQuest", MAPPING) == "Dentaquest"
|
||||
|
||||
|
||||
def case_unmapped_payer_returns_none():
|
||||
assert payer_groups.classify("Some New Payer LLC", MAPPING) is None
|
||||
|
||||
|
||||
def case_case_sensitive_no_fuzzy_matching():
|
||||
"""Different case is a different string, on purpose — see module docstring."""
|
||||
assert payer_groups.classify("medicaid", MAPPING) is None
|
||||
assert payer_groups.classify("MEDICAID", MAPPING) is None
|
||||
|
||||
|
||||
def case_blank_or_none_payer_is_unmapped():
|
||||
assert payer_groups.classify("", MAPPING) is None
|
||||
assert payer_groups.classify(None, MAPPING) is None
|
||||
|
||||
|
||||
def case_load_mapping_reads_json_file():
|
||||
path = TMP / "mapping.json"
|
||||
path.write_text(json.dumps({"Medicaid": "Medicaid"}), encoding="utf-8")
|
||||
loaded = payer_groups.load_mapping(path)
|
||||
assert loaded == {"Medicaid": "Medicaid"}, loaded
|
||||
|
||||
|
||||
def case_load_mapping_missing_file_raises():
|
||||
try:
|
||||
payer_groups.load_mapping(TMP / "does_not_exist.json")
|
||||
except FileNotFoundError as exc:
|
||||
assert "does_not_exist.json" in str(exc), exc
|
||||
return
|
||||
raise AssertionError("expected FileNotFoundError")
|
||||
|
||||
|
||||
def case_seed_config_loads_and_covers_medicaid_and_dentaquest():
|
||||
"""The real config file this ships with, not a fixture."""
|
||||
mapping = payer_groups.load_mapping()
|
||||
assert mapping.get("Medicaid") == "Medicaid", mapping.get("Medicaid")
|
||||
assert mapping.get("DentaQuest") == "Dentaquest", mapping.get("DentaQuest")
|
||||
|
||||
|
||||
CASES = [v for k, v in sorted(globals().items()) if k.startswith("case_")]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
failures = 0
|
||||
for fn in CASES:
|
||||
name = fn.__name__[5:].replace("_", " ")
|
||||
try:
|
||||
fn()
|
||||
print(f" ok {name}")
|
||||
except AssertionError as exc:
|
||||
print(f" FAIL {name}: {exc}")
|
||||
failures += 1
|
||||
except Exception as exc:
|
||||
print(f" ERROR {name}: {type(exc).__name__}: {exc}")
|
||||
failures += 1
|
||||
print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run it to verify it fails**
|
||||
|
||||
Run: `python tests\test_payer_groups.py`
|
||||
Expected: `ERROR` on every case — `ModuleNotFoundError: No module named 'ecw_bot.core.payer_groups'`
|
||||
|
||||
- [ ] **Step 3: Create the seed config, `src/ecw_bot/payer_groups.json`**
|
||||
|
||||
```json
|
||||
{
|
||||
"Medicaid": "Medicaid",
|
||||
"DentaQuest": "Dentaquest",
|
||||
"CHP+ DENTAQUEST": "Dentaquest",
|
||||
"AARP Medicare Advantage": "Medicare",
|
||||
"Aetna Medicare ADV": "Medicare",
|
||||
"OUTREACH PROGRAM": "Program",
|
||||
"Anthem Blue Cross Blue Shield": "Commercial",
|
||||
"Cigna HealthCare Of Colorado": "Commercial",
|
||||
"Cigna Healthcare of CO": "Commercial",
|
||||
"Cigna Salud Employee": "Commercial",
|
||||
"Cigna": "Commercial",
|
||||
"Cypress Benefit Admin / Cigna PPO": "Commercial",
|
||||
"United Healthcare": "Commercial",
|
||||
"United Health Care": "Commercial",
|
||||
"United healthcare": "Commercial",
|
||||
"Unitedhealth One": "Commercial",
|
||||
"Oxford Health Insurance UHC": "Commercial",
|
||||
"UMR": "Commercial",
|
||||
"Aetna": "Commercial",
|
||||
"Aetna Alaska Care": "Commercial",
|
||||
"AETNA HEALTH AND LIFE INSURANCE": "Commercial",
|
||||
"BIND BENEFITS INC": "Commercial",
|
||||
"Select Health": "Commercial"
|
||||
}
|
||||
```
|
||||
|
||||
This is the "Mapped (seeded)" table from the spec, byte-for-byte. The 17 payers in the spec's "Left unmapped" list are deliberately absent — `classify()` returning `None` for them, and the flow logging/flagging that, is the intended behavior until a human adds them.
|
||||
|
||||
- [ ] **Step 4: Implement `core/payer_groups.py`**
|
||||
|
||||
```python
|
||||
"""
|
||||
Payer -> Group Name lookup for the categorization step.
|
||||
|
||||
Exact-string match against a small hand-edited table (`payer_groups.json`,
|
||||
next to this package) — no fuzzy or substring matching. eCW's own PAYER
|
||||
values already differ by case and spacing for what is really the same
|
||||
carrier (see `tests/test_payer_groups.py`'s docstring), so "close enough"
|
||||
matching has no safe way to pick which known string a new one is closest to.
|
||||
An unrecognized payer must come back as None and get a human's attention —
|
||||
never a guess, on billing data.
|
||||
|
||||
Adding a payer is a one-line edit to `payer_groups.json`, no code change.
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
_CONFIG_PATH = Path(__file__).resolve().parent.parent / "payer_groups.json"
|
||||
|
||||
|
||||
def load_mapping(path=None) -> dict:
|
||||
"""The payer -> group table. `path` defaults to the shipped config."""
|
||||
p = Path(path) if path else _CONFIG_PATH
|
||||
if not p.exists():
|
||||
raise FileNotFoundError(
|
||||
f"{p} not found — the payer group table is missing")
|
||||
return json.loads(p.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def classify(payer: str, mapping: dict) -> str:
|
||||
"""The Group Name for this exact PAYER string, or None if unmapped."""
|
||||
return mapping.get((payer or "").strip())
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run the tests to verify they pass**
|
||||
|
||||
Run: `python tests\test_payer_groups.py`
|
||||
Expected: `7/7 passed`
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
cd "D:\Ecw Process"
|
||||
git add src/ecw_bot/core/payer_groups.py src/ecw_bot/payer_groups.json tests/test_payer_groups.py
|
||||
git commit -m "feat: add payer_groups lookup module and seed config"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Wire the categorization step into the export flow
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ecw_bot/settings.py`
|
||||
- Modify: `.env.example`
|
||||
- Modify: `src/ecw_bot/flows/export_medcor_ready.py`
|
||||
- Test: `tests/test_categorize_by_payer.py` (new)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `spreadsheet.read_rows`, `spreadsheet.write_xlsx` (Task 2); `payer_groups.load_mapping`, `payer_groups.classify` (Task 3).
|
||||
- Produces: `export_medcor_ready._categorize_by_payer(saved: Path | None) -> Path | None` — writes `Output\Medcor Ready_<date>.xlsx` and returns its path, or `None` if the step could not run (never raises).
|
||||
|
||||
- [ ] **Step 1: Add the new setting**
|
||||
|
||||
In `src/ecw_bot/settings.py`, immediately after the `SHOW_CLAIM_IDS` setting (currently ending around line 311, just before the `# ── Per-claim processing ──` section header), add:
|
||||
|
||||
```python
|
||||
# After categorizing by payer, tell the operator about any PAYER value with
|
||||
# no entry in payer_groups.json — those rows are saved with a blank Group
|
||||
# Name rather than a guess. Informational only, same reasoning as
|
||||
# NOTIFY_AFTER_EXPORT above. Set 0 for unattended runs; the full list still
|
||||
# goes to the log either way.
|
||||
NOTIFY_UNMAPPED_PAYERS = _env_bool("NOTIFY_UNMAPPED_PAYERS", True)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add the matching line to `.env.example`**
|
||||
|
||||
Open `.env.example`, find the block documenting `SHOW_CLAIM_IDS` (search for that key), and add directly after it:
|
||||
|
||||
```
|
||||
# After categorizing the export by payer, show a box listing any PAYER value
|
||||
# with no entry in src/ecw_bot/payer_groups.json (those rows get a blank
|
||||
# Group Name, not a guess). 1 = show it, 0 = log only.
|
||||
NOTIFY_UNMAPPED_PAYERS=1
|
||||
```
|
||||
|
||||
Match whatever `1`/`0` convention the surrounding boolean settings in that file already use — check `SHOW_CLAIM_IDS`'s line right above it and mirror its exact style so the two files can't drift the way the earlier `OTP_WAIT_SECONDS` mismatch did.
|
||||
|
||||
- [ ] **Step 3: Write the failing test for `_categorize_by_payer`**
|
||||
|
||||
Create `tests/test_categorize_by_payer.py`:
|
||||
|
||||
```python
|
||||
"""
|
||||
export_medcor_ready._categorize_by_payer(): the payer -> Group Name step.
|
||||
|
||||
python tests\\test_categorize_by_payer.py
|
||||
|
||||
Builds a real xlsx (same helper shape as test_claim_file.py's `xlsx()`) so
|
||||
this exercises the actual read -> classify -> write path, not a mock of it.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import zipfile
|
||||
import pathlib
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent / "src"))
|
||||
|
||||
from ecw_bot.flows import export_medcor_ready as flow # noqa: E402
|
||||
from ecw_bot.core import spreadsheet # noqa: E402
|
||||
from ecw_bot import settings as cfg # noqa: E402
|
||||
|
||||
TMP = pathlib.Path(tempfile.mkdtemp())
|
||||
NS = 'xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"'
|
||||
|
||||
|
||||
def xlsx(header, data_rows, name="t.xlsx") -> pathlib.Path:
|
||||
"""A minimal real xlsx: header + data rows, both plain string lists."""
|
||||
rows = [header] + data_rows
|
||||
body = []
|
||||
for r, row in enumerate(rows, start=1):
|
||||
cells = "".join(
|
||||
f'<c r="{chr(65 + c)}{r}" t="str"><v>{v}</v></c>'
|
||||
for c, v in enumerate(row))
|
||||
body.append(f'<row r="{r}">{cells}</row>')
|
||||
sheet = (f'<?xml version="1.0"?><worksheet {NS}><sheetData>'
|
||||
f'{"".join(body)}</sheetData></worksheet>')
|
||||
path = TMP / name
|
||||
with zipfile.ZipFile(path, "w") as z:
|
||||
z.writestr("xl/worksheets/sheet1.xml", sheet)
|
||||
return path
|
||||
|
||||
|
||||
HEADER = ["CLAIMS#", "PAYER", "CHARGES"]
|
||||
|
||||
|
||||
def case_categorizes_known_payers():
|
||||
saved = xlsx(HEADER, [
|
||||
["9107000", "Medicaid", "275.00"],
|
||||
["9106000", "DentaQuest", "117.00"],
|
||||
], name="known.xlsx")
|
||||
out = flow._categorize_by_payer(saved)
|
||||
assert out is not None and out.exists(), out
|
||||
rows = spreadsheet.read_rows(out)
|
||||
assert rows[0] == HEADER + ["Group Name"], rows[0]
|
||||
assert rows[1] == ["9107000", "Medicaid", "275.00", "Medicaid"], rows[1]
|
||||
assert rows[2] == ["9106000", "DentaQuest", "117.00", "Dentaquest"], rows[2]
|
||||
|
||||
|
||||
def case_unmapped_payer_gets_blank_group_not_a_guess():
|
||||
saved = xlsx(HEADER, [
|
||||
["9108000", "Some Brand New Payer", "50.00"],
|
||||
], name="unmapped.xlsx")
|
||||
out = flow._categorize_by_payer(saved)
|
||||
rows = spreadsheet.read_rows(out)
|
||||
assert rows[1][-1] == "", rows[1] # blank, not a guessed group
|
||||
|
||||
|
||||
def case_none_input_is_a_no_op():
|
||||
assert flow._categorize_by_payer(None) is None
|
||||
|
||||
|
||||
def case_missing_payer_column_does_not_raise():
|
||||
saved = xlsx(["CLAIMS#", "CHARGES"], [["9108000", "50.00"]],
|
||||
name="nopayer.xlsx")
|
||||
assert flow._categorize_by_payer(saved) is None # skipped, not a crash
|
||||
|
||||
|
||||
CASES = [v for k, v in sorted(globals().items()) if k.startswith("case_")]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
failures = 0
|
||||
for fn in CASES:
|
||||
name = fn.__name__[5:].replace("_", " ")
|
||||
try:
|
||||
fn()
|
||||
print(f" ok {name}")
|
||||
except AssertionError as exc:
|
||||
print(f" FAIL {name}: {exc}")
|
||||
failures += 1
|
||||
except Exception as exc:
|
||||
print(f" ERROR {name}: {type(exc).__name__}: {exc}")
|
||||
failures += 1
|
||||
print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run it to verify it fails**
|
||||
|
||||
Run: `python tests\test_categorize_by_payer.py`
|
||||
Expected: `ERROR` on every case — `AttributeError: module 'ecw_bot.flows.export_medcor_ready' has no attribute '_categorize_by_payer'`
|
||||
|
||||
- [ ] **Step 5: Implement `_categorize_by_payer` in `flows/export_medcor_ready.py`**
|
||||
|
||||
Change the import line near the top (currently `from ..core import guard, runlog, msgbox`) to:
|
||||
|
||||
```python
|
||||
from ..core import guard, runlog, msgbox, spreadsheet, payer_groups
|
||||
```
|
||||
|
||||
Add this new function directly after `_read_back_claim_ids` (which ends at the `return info` around line 85), before `_read_claim_form`:
|
||||
|
||||
```python
|
||||
def _categorize_by_payer(saved):
|
||||
"""
|
||||
Add a Group Name column (Commercial / Medicaid / Medicare / Dentaquest /
|
||||
Program / ...) next to the saved report, based on each row's PAYER, and
|
||||
write it to Output\\Medcor Ready_<date>.xlsx.
|
||||
|
||||
Never touches `saved` itself: claim_file.py and everything after it reads
|
||||
that file's columns by position (see claim_file.py's module docstring),
|
||||
so this always writes a SEPARATE file rather than editing it in place.
|
||||
|
||||
Never fatal. An unmapped payer, a missing PAYER column, or a write
|
||||
failure is worth reporting loudly, not worth losing the already-saved
|
||||
report over — same reasoning as `_read_back_claim_ids` above.
|
||||
"""
|
||||
if saved is None:
|
||||
return None
|
||||
try:
|
||||
rows = spreadsheet.read_rows(saved)
|
||||
except Exception as exc:
|
||||
print(f"[CATEGORIZE] Could not read {saved.name}: {exc}")
|
||||
return None
|
||||
if not rows:
|
||||
print(f"[CATEGORIZE] {saved.name} has no rows — skipping")
|
||||
return None
|
||||
|
||||
header, data = rows[0], rows[1:]
|
||||
if "PAYER" not in header:
|
||||
print(f"[CATEGORIZE] No PAYER column in {saved.name} — skipping. "
|
||||
f"Header reads: {header}")
|
||||
return None
|
||||
payer_col = header.index("PAYER")
|
||||
|
||||
try:
|
||||
mapping = payer_groups.load_mapping()
|
||||
except Exception as exc:
|
||||
print(f"[CATEGORIZE] Could not load payer_groups.json: {exc}")
|
||||
return None
|
||||
|
||||
out_rows = [header + ["Group Name"]]
|
||||
unmapped = {}
|
||||
mapped_count = 0
|
||||
for row in data:
|
||||
payer = row[payer_col] if payer_col < len(row) else ""
|
||||
group = payer_groups.classify(payer, mapping)
|
||||
if group:
|
||||
mapped_count += 1
|
||||
else:
|
||||
unmapped[payer] = unmapped.get(payer, 0) + 1
|
||||
out_rows.append(list(row) + [group or ""])
|
||||
|
||||
out_path = cfg.ROOT / "Output" / f"Medcor Ready_{cfg.TODAY}.xlsx"
|
||||
try:
|
||||
spreadsheet.write_xlsx(out_path, out_rows)
|
||||
except Exception as exc:
|
||||
print(f"[CATEGORIZE] Could not write {out_path.name}: {exc}")
|
||||
return None
|
||||
|
||||
print(f"[CATEGORIZE] {mapped_count} of {len(data)} row(s) categorized "
|
||||
f"-> {out_path.name}")
|
||||
if unmapped:
|
||||
listed = sorted(unmapped.items(), key=lambda kv: -kv[1])
|
||||
print(f"[CATEGORIZE] {sum(unmapped.values())} row(s) with no group "
|
||||
f"in payer_groups.json:")
|
||||
for payer, count in listed:
|
||||
print(f"[CATEGORIZE] {count:4d} {payer!r}")
|
||||
if cfg.NOTIFY_UNMAPPED_PAYERS:
|
||||
body = (f"{sum(unmapped.values())} row(s)-ku Group Name "
|
||||
f"kidaikkala.\n\n"
|
||||
f"{out_path.name} file-la '' (blank) Group Name-oda "
|
||||
f"irukkum. Idha payer_groups.json-la add pannunga:\n\n"
|
||||
+ "\n".join(f" {p} ({n} row)" for p, n in listed[:15]))
|
||||
if len(listed) > 15:
|
||||
body += f"\n ... innum {len(listed) - 15}"
|
||||
msgbox.info_box("eCW Bot — Group Name podala", body)
|
||||
|
||||
return out_path
|
||||
```
|
||||
|
||||
Add the call into `_claims_phase` — change:
|
||||
|
||||
```python
|
||||
saved = claim_report.export_claims_report(page)
|
||||
info = _read_back_claim_ids(saved)
|
||||
if info:
|
||||
_process_claims(page, info)
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```python
|
||||
saved = claim_report.export_claims_report(page)
|
||||
info = _read_back_claim_ids(saved)
|
||||
_categorize_by_payer(saved)
|
||||
if info:
|
||||
_process_claims(page, info)
|
||||
```
|
||||
|
||||
Finally, update the module's own flow docstring at the top of the file — change:
|
||||
|
||||
```
|
||||
→ Input\Claim Report_<MM-DD-YYYY>.xlsx
|
||||
→ read the file back, pull the CLAIMS# column
|
||||
→ search each claim id on its own (CLAIM_PROCESS_LIMIT)
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```
|
||||
→ Input\Claim Report_<MM-DD-YYYY>.xlsx
|
||||
→ read the file back, pull the CLAIMS# column
|
||||
→ categorize by payer → Output\Medcor Ready_<MM-DD-YYYY>.xlsx
|
||||
→ search each claim id on its own (CLAIM_PROCESS_LIMIT)
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the test to verify it passes**
|
||||
|
||||
Run: `python tests\test_categorize_by_payer.py`
|
||||
Expected: `4/4 passed`
|
||||
|
||||
- [ ] **Step 7: Run the full existing test suite to confirm nothing else broke**
|
||||
|
||||
```bash
|
||||
python tests\test_settings.py
|
||||
python tests\test_guard.py
|
||||
python tests\test_dead_end.py
|
||||
python tests\test_claim_file.py
|
||||
python tests\test_claim_form.py
|
||||
python tests\test_progress_note.py
|
||||
python tests\test_input_archive.py
|
||||
python tests\test_spreadsheet_write.py
|
||||
python tests\test_payer_groups.py
|
||||
python tests\test_categorize_by_payer.py
|
||||
```
|
||||
|
||||
Expected: every one reports all cases passed. `test_settings.py` in particular asserts the module graph stays acyclic and that no selector leaks into `settings` — confirm it still passes since this task touched `settings.py`.
|
||||
|
||||
- [ ] **Step 8: Commit**
|
||||
|
||||
```bash
|
||||
cd "D:\Ecw Process"
|
||||
git add src/ecw_bot/settings.py .env.example src/ecw_bot/flows/export_medcor_ready.py tests/test_categorize_by_payer.py
|
||||
git commit -m "feat: categorize Medcor Ready export by payer after each run"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Document it
|
||||
|
||||
**Files:**
|
||||
- Modify: `README.md`
|
||||
- Modify: `CHANGELOG.md`
|
||||
|
||||
**Interfaces:** none — documentation only.
|
||||
|
||||
- [ ] **Step 1: Add a short section to `README.md`**
|
||||
|
||||
Find the "### Reading the claim ids back" section (the last subsection under "## Claim lookup", ending just before "## Known gaps"). Add a new subsection directly after it:
|
||||
|
||||
```markdown
|
||||
### Categorizing by payer
|
||||
|
||||
Right after the claim ids are read back, every row of the just-saved report
|
||||
gets a `Group Name` (`Commercial`, `Medicaid`, `Medicare`, `Dentaquest`,
|
||||
`Program`, ...) based on its `PAYER`, and the result is saved to
|
||||
`Output\Medcor Ready_<MM-DD-YYYY>.xlsx`. The original `Input\` file is never
|
||||
touched — `claim_file.py` and the charge-entry phase depend on its exact
|
||||
column layout.
|
||||
|
||||
The mapping is `src\ecw_bot\payer_groups.json` — a flat, hand-editable
|
||||
`{"exact PAYER string": "Group Name"}` table. **Exact match only, no fuzzy
|
||||
matching**: eCW's own PAYER values already differ by case and spacing for
|
||||
what is really the same carrier (`United Healthcare` / `United Health Care`
|
||||
/ `United healthcare` are three separate real values), so a PAYER string
|
||||
with no entry gets a **blank** Group Name rather than a guess. Unmapped rows
|
||||
are logged and, unless `NOTIFY_UNMAPPED_PAYERS=0`, shown in a message box —
|
||||
adding one is a one-line edit to the JSON file, no code change.
|
||||
|
||||
Never fatal: a missing PAYER column, an unreadable report, or a write
|
||||
failure is logged and the run continues — the export already succeeded by
|
||||
the time this step runs.
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add a CHANGELOG entry**
|
||||
|
||||
Append to the end of `CHANGELOG.md`:
|
||||
|
||||
```markdown
|
||||
|
||||
---
|
||||
|
||||
## Day 4 — 2026-09-15
|
||||
|
||||
**Automated the payer categorization step.** Reviewed against
|
||||
`Salud-Charge Entry_Medcor Ready.mp4` (2:45-10:00): after the Medcor Ready
|
||||
export, an operator was filtering the report by `PAYER` and hand-filling a
|
||||
`Group Name` column (`Commercial`, `Medicaid`, `Medicare`, `Dentaquest`,
|
||||
`Program`, ...) — the same manual, repetitive, rule-based step every other
|
||||
phase of this bot exists to remove.
|
||||
|
||||
`core/payer_groups.py` does the lookup: exact string match against
|
||||
`payer_groups.json`, on purpose — eCW's own PAYER values already differ by
|
||||
case and spacing for what is really the same carrier (`United Healthcare` /
|
||||
`United Health Care` / `United healthcare` are three separate values in one
|
||||
real export), so anything looser has no safe way to pick which known string
|
||||
a new one is closest to. An unrecognized payer gets a **blank** Group Name,
|
||||
not a guess, and is logged and message-boxed — this is billing data, and
|
||||
wrong is worse than blank.
|
||||
|
||||
`core/spreadsheet.py` gained a write side (`write_xlsx()`) to go with its
|
||||
read side, built the same way — stdlib only, no openpyxl — but with the full
|
||||
minimal OOXML part set (`[Content_Types].xml`, both `.rels` files,
|
||||
`workbook.xml`) the reader doesn't need but real Excel does, since this file
|
||||
is meant to be opened by a person.
|
||||
|
||||
The categorized output is a **new** file, `Output\Medcor Ready_<date>.xlsx`
|
||||
— `Input\Claim Report_<date>.xlsx` is never modified, because `claim_file.py`
|
||||
and the charge-entry phase read its columns by position.
|
||||
|
||||
**Seeded, not complete.** The mapping table ships with 23 payers covering
|
||||
roughly 95% of a real export's row volume (`Medicaid`, `DentaQuest`, and the
|
||||
clearly-commercial carriers), derived from the actual current export rather
|
||||
than guessed from the video. 17 payers (`BH COLORADO ACCESS`, the Delta
|
||||
Dental variants, `Tricare`, `Devoted`, several TPA names, ...) are
|
||||
deliberately left out — first run will flag them, and filling them in is a
|
||||
one-line JSON edit, no redeploy.
|
||||
|
||||
**Out of scope for this round:** splitting rows into the `Patient` /
|
||||
`Pending with Error` / `Program` sibling workbooks the video also shows, and
|
||||
writing anything to the network `Charge\Allocation\...` folder — both stay
|
||||
manual for now.
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
cd "D:\Ecw Process"
|
||||
git add README.md CHANGELOG.md
|
||||
git commit -m "docs: document the payer categorization step"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-review notes
|
||||
|
||||
- **Spec coverage:** architecture (Task 2-4), components (`payer_groups.py` Task 3, `write_xlsx` Task 2, flow wiring Task 4), data flow (Task 4 Step 5), error handling (unmapped payer / missing file / write failure, all in Task 4 Step 5 and covered by tests), testing (every task has its own `test_*.py`), seed mapping table (Task 3 Step 3, copied verbatim from the spec). All covered.
|
||||
- **Placeholder scan:** no TBD/TODO; every step has real, complete code.
|
||||
- **Type consistency:** `classify(payer, mapping)` signature is identical everywhere it's called (Task 3 tests, Task 4's `_categorize_by_payer`); `load_mapping(path=None)` likewise.
|
||||
- **Scope:** single subsystem (categorization only), matches the spec's explicit out-of-scope list — no split-into-3-files logic anywhere in this plan.
|
||||
@@ -0,0 +1,108 @@
|
||||
# Results DB + Web Page Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Store every run's Charge Summary rows in SQL Server (`Charge_Validation.dbo.ChargeSummary`) and show them on a local FastAPI page.
|
||||
|
||||
**Architecture:** A pure module `core/results_db.py` turns Charge Summary rows (the same rows the Excel file gets) into typed DB records and inserts them with one parameterized statement; the flow calls it after the Excel write, behind `RESULTS_DB`, and any DB error is a log line only. A separate `web/` FastAPI app reads the table through a small repository class and renders one Jinja2 page plus two JSON endpoints, bound to 127.0.0.1.
|
||||
|
||||
**Tech Stack:** Python 3.13, pyodbc + ODBC Driver 18 for SQL Server, FastAPI 0.141 + Jinja2 + uvicorn, Starlette TestClient (httpx). All already installed.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-10-08-results-db-and-web-page-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Server `MECARE331\MECARE331`, database `Charge_Validation`, table `dbo.ChargeSummary`.
|
||||
- Default connection: `DRIVER={ODBC Driver 18 for SQL Server};SERVER=MECARE331\MECARE331;DATABASE=Charge_Validation;Trusted_Connection=yes;TrustServerCertificate=yes;`
|
||||
- `RESULTS_DB` default **off**. The bot never creates/alters tables; the operator runs `tools/sql/create_tables.sql` in SSMS.
|
||||
- A DB failure never stops the run and never touches the Excel summary.
|
||||
- Only parameterized SQL; web is read-only (`SELECT` only).
|
||||
- Web bound to `127.0.0.1` only. No external scripts/CDNs.
|
||||
- No new entries in `requirements.txt`. Tests are standalone scripts (`python tests\test_x.py`, prints `N/M passed`), no pytest, no real DB.
|
||||
- Never commit unless the operator asks; never commit `run.py`, `Input/`, `Output/`, `Logs/`, `Screenshots/`, `Dom.txt`, client `.xlsx`.
|
||||
|
||||
## Rulings (plan vs spec)
|
||||
|
||||
- Ruling: the flow builds the sheet once more for the DB (`charge_summary.build`) instead of threading one sheet through `write()` — keeps `write()` and its tests unchanged; both builds read the same report and outcomes on the same day, so the rows are identical — cost if wrong: a row differing between Excel and DB, visible on the page.
|
||||
- Ruling: only rows the run filled in (non-blank Category) are saved — unprocessed / not-found rows are blank in the Excel too and would only clutter the page — cost if wrong: the page lacks blank rows the Excel has.
|
||||
|
||||
## File map
|
||||
|
||||
| File | Responsibility |
|
||||
|---|---|
|
||||
| `tools/sql/create_tables.sql` | Table + indexes, run by the operator |
|
||||
| `src/ecw_bot/settings.py` | `RESULTS_DB`, `RESULTS_DB_CONN` |
|
||||
| `src/ecw_bot/core/results_db.py` | Column map, value conversion, `details_for`, `insert_rows`, `save_summary` |
|
||||
| `src/ecw_bot/core/charge_summary.py` | `build()` split out of `write()` |
|
||||
| `src/ecw_bot/flows/export_medcor_ready.py` | `changes` in `_conditions`, `details` dict, `_run_mode()`, `_save_results_db()` |
|
||||
| `web/__init__.py`, `web/repo.py`, `web/app.py`, `web/templates/index.html` | Read-only page + JSON API |
|
||||
| `run_web.bat` | Start the page on 127.0.0.1:8000 |
|
||||
| `tests/test_results_db.py`, `tests/test_web_app.py` | Tests |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: SQL script, settings, `results_db` module
|
||||
|
||||
**Files:**
|
||||
- Create: `tools/sql/create_tables.sql`, `src/ecw_bot/core/results_db.py`, `tests/test_results_db.py`
|
||||
- Modify: `src/ecw_bot/settings.py` (after `SUMMARY_ASSIGNED_TO`)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `results_db.COLUMNS: list[tuple[str, str]]` (header, db column), `DB_COLUMNS: list[str]`, `TABLE = "dbo.ChargeSummary"`, `to_money(v) -> Decimal|None`, `to_date(v) -> date|None`, `to_record(row: dict, *, run_at, run_mode, details) -> dict`, `details_for(data: dict) -> dict`, `insert_rows(conn, records) -> int`, `save_summary(sheet, details, *, run_at, run_mode, conn_str, connect=None) -> int|None`; settings `RESULTS_DB: bool`, `RESULTS_DB_CONN: str`.
|
||||
|
||||
- [ ] **Step 1: Write the failing tests** — `tests/test_results_db.py` with cases: every HEADER mapped (24) and `DB_COLUMNS` = run_at, run_mode, 24 columns, details_json; `to_money("$1,234.50") == Decimal("1234.50")`, `"(12.00)"` → -12.00, blank/"abc"/"NaN"/"1e20" → None, `365` → 365.00; `to_date("7/31/2026") == date(2026,7,31)`, ISO and datetime accepted, blank/junk → None; `to_record` converts types, blank text → None, `details_json` is JSON or None; `details_for` copies `_conditions` keys, Dental results, error keys; `insert_rows` sends one `INSERT INTO dbo.ChargeSummary (...) VALUES (?, ...)` with values in `DB_COLUMNS` order and commits once; `save_summary` skips blank-Category rows, closes the connection, returns the count; a raising `connect` returns None and prints `[RESULTS-DB]`. (Full code: see the test file written in Step 1 of execution.)
|
||||
|
||||
- [ ] **Step 2: Run** `python tests\test_results_db.py` — Expected: ImportError (no `results_db`).
|
||||
|
||||
- [ ] **Step 3: Implement** `results_db.py` per the Interfaces above (money: strip `$ ,`, `(x)` negative, finite and `< 10**10`, quantize 0.01; dates: `mm/dd/yyyy`, `yyyy-mm-dd`, `date`/`datetime`; `INSERT` built only from `DB_COLUMNS` names and `?` markers; `save_summary` wraps everything in `try/except Exception` → `[RESULTS-DB] Could not save ...` + None). Settings:
|
||||
|
||||
```python
|
||||
# ── Results database (SQL Server) ─────────────────────────────────────────────
|
||||
# Also save the Charge Summary rows to SQL Server (core/results_db.py) for the
|
||||
# results web page (web/). Off by default; a DB error is a log line only.
|
||||
RESULTS_DB = _env_bool("RESULTS_DB", False)
|
||||
RESULTS_DB_CONN = _env(
|
||||
"RESULTS_DB_CONN",
|
||||
"DRIVER={ODBC Driver 18 for SQL Server};SERVER=MECARE331\\MECARE331;"
|
||||
"DATABASE=Charge_Validation;Trusted_Connection=yes;"
|
||||
"TrustServerCertificate=yes;")
|
||||
```
|
||||
|
||||
and `tools/sql/create_tables.sql` = the spec's `CREATE TABLE` wrapped in `IF OBJECT_ID('dbo.ChargeSummary','U') IS NULL BEGIN ... END`.
|
||||
|
||||
- [ ] **Step 4: Run** `python tests\test_results_db.py` — Expected: all passed.
|
||||
|
||||
### Task 2: `charge_summary.build()` and the flow hook
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ecw_bot/core/charge_summary.py` (`write()`), `src/ecw_bot/flows/export_medcor_ready.py` (`_read_claim_form` `_conditions`, claim loop, after `_write_charge_summary`)
|
||||
- Test: `tests/test_results_db.py` (flow cases), `tests/test_charge_summary.py` (unchanged, must stay green)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `results_db.details_for`, `results_db.save_summary`, `cfg.RESULTS_DB`, `cfg.RESULTS_DB_CONN`.
|
||||
- Produces: `charge_summary.build(report_path, outcomes, assigned_to) -> list` (header + rows); flow `_run_mode() -> str` ("TEST" / "LIVE-WRITE" / "VERIFY"), `_save_results_db(info, outcomes, details) -> None`.
|
||||
|
||||
- [ ] **Step 1: Failing tests** in `tests/test_results_db.py`: `_run_mode()` for the three settings combinations; `_save_results_db` with `RESULTS_DB` off calls nothing; on, it calls `charge_summary.build(info["file"], outcomes, cfg.SUMMARY_ASSIGNED_TO)` and `results_db.save_summary(sheet, details, run_at=<datetime>, run_mode=..., conn_str=cfg.RESULTS_DB_CONN)`; a raising `build` prints `[RESULTS-DB]` and does not raise.
|
||||
- [ ] **Step 2: Run** — Expected: AttributeError (`_run_mode` missing).
|
||||
- [ ] **Step 3: Implement**: move `write()`'s read+`build_rows` into `build()`, `write()` calls `build()`; in `_read_claim_form` add `"changes": {r.condition_name: r.changes for r in results}` to `data["_conditions"]`; in the claim loop `details = {}` and `details[claim_id] = results_db.details_for(data)`; after `_write_charge_summary(info, outcomes)` call `_save_results_db(info, outcomes, details)`.
|
||||
- [ ] **Step 4: Run** `tests\test_results_db.py`, `tests\test_charge_summary.py`, `tests\test_claim_boxes.py` — Expected: all passed.
|
||||
|
||||
### Task 3: Web page
|
||||
|
||||
**Files:**
|
||||
- Create: `web/__init__.py`, `web/repo.py`, `web/app.py`, `web/templates/index.html`, `run_web.bat`, `tests/test_web_app.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `results_db.COLUMNS`, `results_db.TABLE`, `cfg.RESULTS_DB_CONN`.
|
||||
- Produces: `web.repo.SqlResultsRepo(conn_str, connect=None)` with `runs() -> list[dict]` (`run_at, run_mode, claims, pass_count, fail_count, review_count`, newest first) and `results(run_at, category=None, claim_no=None) -> list[dict]`; `web.app.create_app(repo) -> FastAPI`; module-level `web.app.app`.
|
||||
|
||||
- [ ] **Step 1: Failing tests** `tests/test_web_app.py` with a fake repo: `/` lists rows of the latest run, shows Pass/Fail/Review counts and the Charge Summary headers; `?category=Review` passes the filter to the repo; `?category=Bad` → 400; `?claim_no=55-51` searches `5551`… digits only; `?run_at=not-a-date` → 400; `/api/runs` and `/api/results` JSON shapes (dates ISO, money numbers); repo raising → 503 with "Results database not reachable"; no runs → page says no results yet; `SqlResultsRepo.results` builds a parameterized query (fake connection) with `category = ?` and `claim_no LIKE ?`.
|
||||
- [ ] **Step 2: Run** — Expected: ImportError.
|
||||
- [ ] **Step 3: Implement** repo, app (`docs_url=None`), template (form: run select, category select, claim search; tiles; table in Charge Summary order with Category colour; `<details>` per row with pretty JSON), `run_web.bat` (`python -m uvicorn web.app:app --host 127.0.0.1 --port 8000`).
|
||||
- [ ] **Step 4: Run** `python tests\test_web_app.py` — Expected: all passed.
|
||||
|
||||
### Task 4: Docs and full suite
|
||||
|
||||
- [ ] Context.md: a "Results DB + web page" section (setup: run `tools/sql/create_tables.sql`, grant the Windows login read/write, `RESULTS_DB=1`, `run_web.bat`).
|
||||
- [ ] Full suite: `bash run_tests.sh` → TOTAL all passed.
|
||||
- [ ] Report to the operator; commit only when asked.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,67 @@
|
||||
# Results Portal Redesign Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace the one-page results table with a Dashboard → Claim status (date cards) → One day (short table + side panel) portal, plus a claim history search, in a navy & blue theme.
|
||||
|
||||
**Architecture:** Server-rendered FastAPI + Jinja pages sharing `base.html`. Three new read-only queries in `web/repo.py` group the existing `CV_ChargeSummary` rows by the day a run started. The Bot run card and `/api/*` stay as they are; old `/?run_at=` links redirect to the day page.
|
||||
|
||||
**Tech Stack:** FastAPI, Jinja2, pyodbc (SQL Server), vanilla JS, standalone test scripts (`python tests\test_web_app.py`).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-10-19-results-portal-redesign-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Bound to 127.0.0.1; Host allowlist, `no-store` headers, DB errors by kind — unchanged.
|
||||
- Start / Stop need the page Origin and `X-Run-Token`; page runs are VERIFY only.
|
||||
- All new pages read-only; every SQL statement parameterized; values never in SQL text.
|
||||
- DB text is escaped: Jinja autoescape, panel data via `tojson` in an attribute, written with `textContent`.
|
||||
- No claim ids or patient data in tracked docs; tests use made-up ids.
|
||||
- No git commits unless the operator asks.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Day-wise queries in the repo
|
||||
|
||||
**Files:** Modify `web/repo.py`; Test `tests/test_web_app.py`
|
||||
|
||||
**Interfaces — Produces:**
|
||||
- `SqlResultsRepo.days(limit: int = 90) -> list[dict]` — keys `day` (date), `claims`, `runs`, `pass_count`, `fail_count`, `review_count`; newest first.
|
||||
- `SqlResultsRepo.day_results(day: date, category=None, claim_no=None) -> list[dict]` — full rows (`_SELECT_COLUMNS`) with `run_at >= day AND run_at < day+1`, ordered `run_at, s_no, id`.
|
||||
- `SqlResultsRepo.claim_history(claim_no: str) -> list[dict]` — `TOP 200` full rows `WHERE claim_no LIKE ?` ordered `run_at DESC, s_no, id`.
|
||||
- `_coerce` also turns a text `day` into a `date`.
|
||||
|
||||
- [ ] Tests: each query's SQL is a parameterized SELECT with the clauses above and the expected params (`[datetime(day), datetime(day)+1 day]`, `["%no%"]`); text `day` coerced.
|
||||
- [ ] Implement; run `python tests\test_web_app.py`.
|
||||
|
||||
### Task 2: Routes
|
||||
|
||||
**Files:** Modify `web/app.py`; Test `tests/test_web_app.py`
|
||||
|
||||
**Interfaces — Consumes:** Task 1. **Produces:** routes
|
||||
- `GET /` → `dashboard.html` with `today` (days() row for `date.today()` or zero counts), `mode` ("VERIFY"/"LIVE-WRITE"), `run_token`. `GET /?run_at=<iso>` → 303 to `/day/<date>` (keeps `category`, `claim_no`); bad `run_at` → 400.
|
||||
- `GET /days` → `days.html` with `days`.
|
||||
- `GET /day/{day}` (`YYYY-MM-DD`, else 400) with `category`, `claim_no`, `claim` → `day.html` with `day`, `counts` (from `days()` row or computed from the unfiltered rows), `rows` (each with `fields` = list of `[header, display text]` and `details_pretty`), filters.
|
||||
- `GET /claim?claim_no=` → `claim.html` with `rows` (history) and `claim_no`; empty search → page with the box only.
|
||||
- FakeRepo in tests gains `days`, `day_results`, `claim_history`.
|
||||
|
||||
- [ ] Tests for each route (data shown, links, filters passed, 400s, redirect, DB down 503, hardening incl. new paths).
|
||||
- [ ] Implement; run tests.
|
||||
|
||||
### Task 3: Templates and theme
|
||||
|
||||
**Files:** Create `web/templates/base.html`, `dashboard.html`, `days.html`, `day.html`, `claim.html`; delete `web/templates/index.html`.
|
||||
|
||||
- `base.html`: `:root` navy palette, header (title, mode pill, breadcrumb block), `{% block content %}`, `{% block scripts %}`, footer PHI note.
|
||||
- `dashboard.html`: Bot run card (moved from index.html, with the Live claims panel replacing the old Claim status panel; a saved claim opens `/day/<results_run_at date>?claim=<id>`), today tiles, Find a claim form, big Claim status button.
|
||||
- `days.html`: date cards grid with counts and colour bar; empty state.
|
||||
- `day.html`: tiles as filter links, claim search, short table (#, Claim #, Time, Payer, Result, Comments), 10-a-page pager, side panel from `data-fields` (`tojson`) via `textContent`; `?claim=` opens it.
|
||||
- `claim.html`: search box and history table linking to day pages.
|
||||
|
||||
- [ ] Tests on rendered HTML (ids, links, no inline `style=`, theme vars, panel wiring, one-scroll lock, Live claims link target).
|
||||
- [ ] Run `python tests\test_web_app.py` and `python tests\test_run_status.py`.
|
||||
|
||||
### Task 4: Browser check
|
||||
|
||||
- [ ] Serve the app on a fake repo with sample data (scratch script, made-up ids), open each page in the in-app browser, screenshot, check pager, panel, filters, empty states, narrow width.
|
||||
- [ ] Full suite.
|
||||
Reference in new issue
Block a user