automation-tools/ecw-charge-validation
8
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
840272032a331482fdc07e973b1ee855902ac629
ecw-charge-validation/docs/superpowers/plans/2026-10-14-start-bot-from-web-page.md
T
ManikandanandClaude Opus 5.5 ba15a574ab Initial import: eCW Charge Validation bot and results Portal 
Snapshot of the working project as of 2026-10-23 (local 2d4d4fc), one
commit with no earlier history: the tracked files were de-identified
first (synthetic claim ids and patient data only; see Context.md
section 7). Credentials, run logs, screenshots, claim reports and client
workbooks are gitignored and not included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 18:38:17 +05:30

1768 lines
63 KiB
Markdown
Raw Blame History

# Start the Bot from the Web Page — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** The operator starts a VERIFY run of the eCW bot from the local results page, watches it live, can stop it, and sees its results when it ends.
**Architecture:** The bot records each run in `Logs\bot_run.json` (pid, log path, exit code) and refuses a second live run. The web server starts `run.py` as a child process with forced VERIFY / quiet-box settings, kills its process tree on Stop, and reports status by parsing the bot's own run log. Start/Stop are POSTs guarded by an Origin check and a per-server token.
**Tech Stack:** Python 3.13 (`C:\Python313\python.exe`, no venv), FastAPI + Jinja2 + Starlette TestClient, `subprocess`, `ctypes` (kernel32), plain HTML/JS in `web/templates/index.html`.
**Spec:** `docs/superpowers/specs/2026-10-14-start-bot-from-web-page-design.md`
## Global Constraints
- A page-started run is ALWAYS VERIFY: child env `SAVE_CORRECTIONS=0`, whatever `.env` says. LIVE-WRITE stays terminal-only.
- Child env forced exactly: `SAVE_CORRECTIONS=0`, `ECW_KEEP_OPEN=0`, `QUIET_BOXES=1`, `CONFIRM_BEFORE_LOOKUP=0`, `ECW_RUN_SOURCE=web`. Nothing from an HTTP request reaches the command line or the environment.
- Boxes that need a person (CAPTCHA, 2FA manual step, security image, LIVE-WRITE confirm) are never quieted.
- Every POST: `Origin` (or `Referer` when Origin is absent) must be `http://127.0.0.1:8000` or `http://localhost:8000`, AND header `X-Run-Token` must equal the server's token (`secrets.compare_digest`). Else 403.
- The page stays bound to 127.0.0.1; responses keep `no-store`; page text from the log is put in the DOM with `textContent`, never `innerHTML`.
- Never `os.kill(pid, 0)` on Windows — it terminates the process. Liveness is `OpenProcess` + `GetExitCodeProcess`.
- Tests are standalone scripts printing `N/M passed` (`python tests\<name>.py`); a test never writes into the real `Logs\` folder.
- **Commits:** the operator's rule is "never commit unless asked at that moment". Steps below say *stage*; the final task asks the operator before committing.
- `run.py` and `tests/test_guard.py` have the operator's own uncommitted edits — never stage them.
## File Structure
| File | Responsibility |
|---|---|
| `src/ecw_bot/core/run_lock.py` (new) | The run record `Logs\bot_run.json`: read / acquire / finish / mark_stopped, `pid_alive()` |
| `src/ecw_bot/flows/export_medcor_ready.py` (modify) | `main()` takes and finishes the run record; `[RESULT]` line per claim |
| `src/ecw_bot/settings.py` (modify) | `QUIET_BOXES` setting |
| `src/ecw_bot/core/msgbox.py` (modify) | `info(..., needs_person=False)` / `info_box(..., needs_person=False)` honour `QUIET_BOXES` |
| `src/ecw_bot/auth/login.py` (modify) | security-image `info_box` passes `needs_person=True` |
| `web/run_status.py` (new) | Parse run-log text → phase, progress, counts, tail, message |
| `web/bot_runner.py` (new) | Start / stop / status of the child bot process |
| `web/app.py` (modify) | `/api/bot/start`, `/api/bot/stop`, `/api/bot/status`, token + Origin guard |
| `web/templates/index.html` (modify) | "Bot run" card + polling JS |
| tests (new) `tests/test_run_lock.py`, `tests/test_run_status.py`, `tests/test_bot_runner.py` | |
| tests (modify) `tests/test_msgbox.py`, `tests/test_web_app.py`, `tests/test_mode_banner.py`, `tests/test_test_claim_ids_file.py`, `tests/test_clean_between_claims.py` | |
---
### Task 1: The run record (`core/run_lock.py`)
**Files:**
- Create: `src/ecw_bot/core/run_lock.py`
- Test: `tests/test_run_lock.py`
**Interfaces:**
- Produces: `run_lock.FILE_NAME = "bot_run.json"`, `run_lock.path() -> Path` (`cfg.LOGS / FILE_NAME`, read at call time), `class Busy(Exception)`, `pid_alive(pid) -> bool`, `read() -> dict | None`, `is_live(rec) -> bool`, `acquire(log_path, source=None) -> dict`, `finish(exit_code) -> None` (never raises), `mark_stopped(pid) -> None`.
- Record keys: `pid` (int), `started_at` (ISO seconds str), `log_path` (str), `source` (`"web"`/`"terminal"`), `ended_at` (ISO str or None), `exit_code` (int or None), `stopped_by_operator` (bool).
- [ ] **Step 1: Write the failing test** — `tests/test_run_lock.py`:
```python
r"""
core/run_lock.py — the record of the latest bot run, Logs\bot_run.json
(spec 2026-10-14-start-bot-from-web-page-design.md).
python tests\test_run_lock.py
Every case runs with cfg.LOGS pointed at a temp folder.
"""
import os
import sys
import json
import pathlib
import tempfile
import subprocess
ROOT = pathlib.Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT / "src"))
from ecw_bot import settings as cfg # noqa: E402
from ecw_bot.core import run_lock # noqa: E402
CASES = []
def case(fn):
CASES.append(fn)
return fn
def _dead_pid() -> int:
p = subprocess.Popen([sys.executable, "-c", "pass"])
p.wait()
return p.pid
@case
def pid_alive_is_true_for_this_python_process():
assert run_lock.pid_alive(os.getpid()) is True
@case
def pid_alive_is_false_for_junk_and_ended_processes():
for junk in (None, "", "x", 0, -5):
assert run_lock.pid_alive(junk) is False, junk
assert run_lock.pid_alive(_dead_pid()) is False
@case
def pid_alive_is_false_for_a_live_process_that_is_not_python():
"""A stale record's pid reused by another program must not read as a
live bot run."""
p = subprocess.Popen(["ping", "-n", "6", "127.0.0.1"],
stdout=subprocess.DEVNULL)
try:
assert run_lock.pid_alive(p.pid) is False
finally:
p.kill()
p.wait()
@case
def acquire_writes_this_process_record():
rec = run_lock.acquire(pathlib.Path("Logs/run_x.log"), source="web")
assert rec["pid"] == os.getpid() and rec["source"] == "web", rec
assert rec["ended_at"] is None and rec["exit_code"] is None, rec
on_disk = json.loads(run_lock.path().read_text(encoding="utf-8"))
assert on_disk == rec, on_disk
@case
def acquire_source_defaults_to_the_env_then_terminal():
os.environ.pop("ECW_RUN_SOURCE", None)
assert run_lock.acquire("a.log")["source"] == "terminal"
run_lock.finish(0)
os.environ["ECW_RUN_SOURCE"] = "web"
try:
assert run_lock.acquire("a.log")["source"] == "web"
finally:
os.environ.pop("ECW_RUN_SOURCE", None)
@case
def a_live_record_refuses_a_second_run():
run_lock.acquire("a.log")
try:
run_lock.acquire("b.log")
except run_lock.Busy as exc:
assert "already going" in str(exc), exc
else:
raise AssertionError("second acquire was allowed")
@case
def a_stale_record_of_a_dead_pid_is_replaced():
run_lock.path().parent.mkdir(parents=True, exist_ok=True)
run_lock.path().write_text(json.dumps(
{"pid": _dead_pid(), "started_at": "2026-10-14T09:00:00",
"log_path": "old.log", "ended_at": None}), encoding="utf-8")
rec = run_lock.acquire("new.log")
assert rec["log_path"] == "new.log", rec
@case
def an_ended_record_does_not_block():
run_lock.acquire("a.log")
run_lock.finish(0)
assert run_lock.acquire("b.log")["log_path"] == "b.log"
@case
def finish_records_the_exit_code_and_end_time():
run_lock.acquire("a.log")
run_lock.finish(5)
rec = run_lock.read()
assert rec["exit_code"] == 5 and rec["ended_at"], rec
assert run_lock.is_live(rec) is False
@case
def finish_leaves_another_process_record_alone():
run_lock.path().parent.mkdir(parents=True, exist_ok=True)
other = {"pid": os.getpid() + 100000, "ended_at": None,
"exit_code": None}
run_lock.path().write_text(json.dumps(other), encoding="utf-8")
run_lock.finish(0)
assert run_lock.read() == other
@case
def finish_never_raises_without_a_record():
run_lock.finish(0) # no file at all
run_lock.path().parent.mkdir(parents=True, exist_ok=True)
run_lock.path().write_text("{not json", encoding="utf-8")
run_lock.finish(0)
assert run_lock.read() is None
@case
def mark_stopped_ends_the_named_run():
rec = run_lock.acquire("a.log")
run_lock.mark_stopped(rec["pid"])
rec = run_lock.read()
assert rec["stopped_by_operator"] is True and rec["ended_at"], rec
assert rec["exit_code"] is None, rec
def main() -> int:
failures = 0
orig = cfg.LOGS
for fn in CASES:
name = fn.__name__.replace("_", " ")
with tempfile.TemporaryDirectory() as tmp:
cfg.LOGS = pathlib.Path(tmp) / "Logs"
try:
fn()
print(f" ok {name}")
except AssertionError as exc:
print(f" FAIL {name}: {exc}")
failures += 1
except Exception as exc:
print(f" ERROR {name}: {type(exc).__name__}: {exc}")
failures += 1
finally:
cfg.LOGS = orig
print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(main())
```
- [ ] **Step 2: Run it to see it fail**
Run: `python tests\test_run_lock.py`
Expected: `ModuleNotFoundError: No module named 'ecw_bot.core.run_lock'`
- [ ] **Step 3: Write `src/ecw_bot/core/run_lock.py`**
```python
r"""
The run record — `Logs\bot_run.json`, the latest bot run (spec
docs/superpowers/specs/2026-10-14-start-bot-from-web-page-design.md).
Written by the bot itself at the start of `main()` and finished when
`main()` returns, so the results page sees a run started in the terminal
as well as one it started. While a run is live a second `main()` refuses to
start: two runs would share one eCW login guard and one Charge Summary.
{"pid", "started_at", "log_path", "source": "web" | "terminal",
"ended_at", "exit_code", "stopped_by_operator"}
Live = no `ended_at` and the pid is a running python process. A killed run
cannot finish its own record; its pid is gone, so it reads as ended.
"""
import os
import json
import ctypes
from ctypes import wintypes
from datetime import datetime
from .. import settings as cfg
FILE_NAME = "bot_run.json"
_PROCESS_QUERY_LIMITED_INFORMATION = 0x1000
_STILL_ACTIVE = 259
class Busy(Exception):
"""Another bot run is live."""
def path():
"""Read at call time: tests point cfg.LOGS elsewhere."""
return cfg.LOGS / FILE_NAME
def _now() -> str:
return datetime.now().isoformat(timespec="seconds")
def pid_alive(pid) -> bool:
"""
A running process with this pid whose program is python.
Never `os.kill(pid, 0)`: on Windows that TERMINATES the process.
The program-name check keeps a stale record whose pid Windows gave to
some other program from reading as a live bot run.
"""
try:
pid = int(pid)
except (TypeError, ValueError):
return False
if pid <= 0:
return False
if os.name != "nt":
try:
os.kill(pid, 0)
return True
except OSError:
return False
k32 = ctypes.WinDLL("kernel32", use_last_error=True)
k32.OpenProcess.restype = wintypes.HANDLE
k32.OpenProcess.argtypes = [wintypes.DWORD, wintypes.BOOL, wintypes.DWORD]
k32.GetExitCodeProcess.argtypes = [wintypes.HANDLE,
ctypes.POINTER(wintypes.DWORD)]
k32.QueryFullProcessImageNameW.argtypes = [
wintypes.HANDLE, wintypes.DWORD, wintypes.LPWSTR,
ctypes.POINTER(wintypes.DWORD)]
k32.CloseHandle.argtypes = [wintypes.HANDLE]
handle = k32.OpenProcess(_PROCESS_QUERY_LIMITED_INFORMATION, False, pid)
if not handle:
return False
try:
code = wintypes.DWORD()
if not k32.GetExitCodeProcess(handle, ctypes.byref(code)):
return False
if code.value != _STILL_ACTIVE:
return False
buf = ctypes.create_unicode_buffer(1024)
size = wintypes.DWORD(len(buf))
if not k32.QueryFullProcessImageNameW(handle, 0, buf,
ctypes.byref(size)):
return True
return "python" in os.path.basename(buf.value).lower()
finally:
k32.CloseHandle(handle)
def read():
"""The record, or None when there is none or it cannot be read."""
try:
data = json.loads(path().read_text(encoding="utf-8"))
except (OSError, ValueError):
return None
return data if isinstance(data, dict) else None
def is_live(rec) -> bool:
return bool(rec) and not rec.get("ended_at") and pid_alive(rec.get("pid"))
def _write(rec: dict) -> None:
target = path()
target.parent.mkdir(parents=True, exist_ok=True)
tmp = target.with_suffix(".tmp")
tmp.write_text(json.dumps(rec, indent=2), encoding="utf-8")
os.replace(tmp, target)
def acquire(log_path, source=None) -> dict:
"""This process's record, written; raises Busy over a live one."""
rec = read()
if is_live(rec):
raise Busy(f"A bot run is already going (pid {rec.get('pid')}, "
f"started {rec.get('started_at') or '?'}) — not starting "
f"a second one")
rec = {"pid": os.getpid(), "started_at": _now(),
"log_path": str(log_path),
"source": source or os.environ.get("ECW_RUN_SOURCE") or "terminal",
"ended_at": None, "exit_code": None,
"stopped_by_operator": False}
_write(rec)
return rec
def finish(exit_code) -> None:
"""End this process's own record. Never raises."""
try:
rec = read()
if not rec or rec.get("pid") != os.getpid():
return
rec.update(ended_at=_now(), exit_code=exit_code)
_write(rec)
except Exception:
pass
def mark_stopped(pid) -> None:
"""The page killed run `pid`: say so on its record."""
rec = read()
if not rec or rec.get("pid") != pid or rec.get("ended_at"):
return
rec.update(ended_at=_now(), exit_code=None, stopped_by_operator=True)
_write(rec)
```
- [ ] **Step 4: Run it to see it pass**
Run: `python tests\test_run_lock.py`
Expected: `12/12 passed`
- [ ] **Step 5: Stage**
`git add src/ecw_bot/core/run_lock.py tests/test_run_lock.py`
---
### Task 2: The bot takes the run record; one `[RESULT]` line per claim
**Files:**
- Modify: `src/ecw_bot/flows/export_medcor_ready.py` (imports ~line 49; `main()` ~line 1315; `_process_claims` after `outcomes[claim_id] = outcome` ~line 957)
- Modify: `tests/test_mode_banner.py` (`_run_main`), `tests/test_test_claim_ids_file.py` (`_PATCHED` + the main() case), `tests/test_clean_between_claims.py` (`_run`)
- Test: `tests/test_run_lock.py` (add cases)
**Interfaces:**
- Consumes: `run_lock.acquire`, `run_lock.finish`, `run_lock.Busy` (Task 1).
- Produces: `main() -> int` unchanged signature; new `_main(log_path) -> int` holds the old body. Log line format consumed by Task 4: `[RESULT] claim {n}/{N} {claim_id}: {Category or "Not found"}`. A busy record → `[RUN] <Busy message>` and exit code 2.
- [ ] **Step 1: Write the failing tests** — append to `tests/test_run_lock.py` before `def main()`:
```python
# ── the flow: main() and the [RESULT] line ───────────────────────────────────
import io # noqa: E402
import contextlib # noqa: E402
from ecw_bot.flows import export_medcor_ready as flow # noqa: E402
def _main_with(inner):
"""flow.main() with runlog stubbed and `_main` replaced by `inner`."""
orig = (flow.runlog.start, flow._main)
flow.runlog.start = lambda: pathlib.Path("Logs/run_test.log")
flow._main = inner
out = io.StringIO()
try:
with contextlib.redirect_stdout(out):
return flow.main(), out.getvalue()
finally:
flow.runlog.start, flow._main = orig
@case
def main_records_the_run_and_its_exit_code():
seen = {}
def inner(log_path):
seen["live"] = run_lock.is_live(run_lock.read())
seen["log"] = str(log_path)
return 5
code, _ = _main_with(inner)
assert code == 5
assert seen["live"] is True and seen["log"].endswith("run_test.log"), seen
rec = run_lock.read()
assert rec["exit_code"] == 5 and rec["ended_at"], rec
@case
def main_finishes_the_record_when_the_run_raises():
def inner(log_path):
raise RuntimeError("boom")
try:
_main_with(inner)
except RuntimeError:
pass
rec = run_lock.read()
assert rec["ended_at"] and rec["exit_code"] == 1, rec
@case
def main_refuses_a_second_run_before_login():
run_lock.acquire("other.log") # this process: live python
called = []
code, out = _main_with(lambda log_path: called.append(1) or 0)
assert code == 2 and called == [], (code, out)
assert "[RUN] A bot run is already going" in out, out
@case
def each_claim_prints_its_result_line():
orig = (cfg.SHOW_EACH_CLAIM, cfg.SHOW_CLAIM_IDS, cfg.CLAIM_PROCESS_LIMIT,
flow.claim_results.search_claim, flow._read_claim_form,
flow.claim_filters.clear_claim_no, flow._write_charge_summary,
flow._save_results_db, flow.claim_form.clear_screen)
cfg.SHOW_EACH_CLAIM = False
cfg.SHOW_CLAIM_IDS = False
cfg.CLAIM_PROCESS_LIMIT = 0
def search(page, cid):
return {"id": cid, "count": 1, "found": cid != "3", "seconds": 1.0}
flow.claim_results.search_claim = search
flow._read_claim_form = lambda page, cid, head: {
"claim_no": cid,
"cpt": [{"code": "99213", "checked": True, "fee": "100.00"}],
"icd": [{"code": "Z00.00"}],
"progress_notes": {"opened": "modal", "text": "Office visit."},
"_conditions": {"fired": [], "saved": False, "failed": [],
"review": [], "comments": {}}}
flow.claim_filters.clear_claim_no = lambda page: None
flow._write_charge_summary = lambda info, o: None
flow._save_results_db = lambda info, o, d: None
flow.claim_form.clear_screen = lambda page, cid, keep=(): []
out = io.StringIO()
try:
with contextlib.redirect_stdout(out):
flow._process_claims(None, {"ids": ["1", "3"], "file": None})
finally:
(cfg.SHOW_EACH_CLAIM, cfg.SHOW_CLAIM_IDS, cfg.CLAIM_PROCESS_LIMIT,
flow.claim_results.search_claim, flow._read_claim_form,
flow.claim_filters.clear_claim_no, flow._write_charge_summary,
flow._save_results_db, flow.claim_form.clear_screen) = orig
text = out.getvalue()
assert "[RESULT] claim 1/2 1: Pass" in text, text
assert "[RESULT] claim 2/2 3: Not found" in text, text
```
- [ ] **Step 2: Run to see the new cases fail**
Run: `python tests\test_run_lock.py`
Expected: the four new cases FAIL/ERROR (`flow` has no `_main`; no `[RESULT]` line); the first 12 pass.
- [ ] **Step 3: Implement in `src/ecw_bot/flows/export_medcor_ready.py`**
Add `run_lock` to the `from ..core import (guard, runlog, msgbox, ...)` import list (keep alphabetical-ish placement next to `runlog`).
Replace the top of `main()`:
```python
def main() -> int:
log_path = runlog.start()
```
with:
```python
def main() -> int:
"""
The run record first (`core/run_lock.py`, 2026-10-14): a second live
run refuses before login — exit 2, nothing spent — and every way out
of `_main()`, a raise included, ends the record with its exit code, so
the results page can say how the run ended.
"""
log_path = runlog.start()
try:
run_lock.acquire(log_path)
except run_lock.Busy as exc:
print(f"[RUN] {exc}")
return 2
code = 1
try:
code = _main(log_path)
return code
finally:
run_lock.finish(code)
def _main(log_path) -> int:
```
(The rest of the old body — from `problems = cfg.validate()` to the end — stays exactly as it is, now inside `_main`.)
In `_process_claims`, right after `outcomes[claim_id] = outcome`:
```python
# One fixed-format line per claim for the results page's live
# status (web/run_status.py, 2026-10-14). {} = no Charge Summary
# row: the claim was not found in the grid.
print(f"[RESULT] claim {n}/{len(todo)} {claim_id}: "
f"{(outcome or {}).get('Category') or 'Not found'}")
```
- [ ] **Step 4: Keep the existing main() tests out of the real `Logs\`**
`tests/test_mode_banner.py`, in `_run_main`, extend the saved/stubbed tuple:
```python
orig = (flow.runlog.start, flow.sync_playwright, flow._login_with_retry,
msgbox.ok_cancel, flow.run_lock.acquire, flow.run_lock.finish)
flow.runlog.start = lambda: pathlib.Path("unused.log")
flow.sync_playwright = _reached_login
flow._login_with_retry = _reached_login
msgbox.ok_cancel = _box
flow.run_lock.acquire = lambda log_path, source=None: {}
flow.run_lock.finish = lambda code: None
```
and the restore line:
```python
(flow.runlog.start, flow.sync_playwright, flow._login_with_retry,
msgbox.ok_cancel, flow.run_lock.acquire, flow.run_lock.finish) = orig
```
`tests/test_test_claim_ids_file.py`: add `(flow.run_lock, "acquire"), (flow.run_lock, "finish"),` to `_PATCHED`, and in the case that calls `flow.main()` (next to `flow.runlog.start = lambda: pathlib.Path("unused.log")`) add:
```python
flow.run_lock.acquire = lambda log_path, source=None: {}
flow.run_lock.finish = lambda code: None
```
`tests/test_clean_between_claims.py` needs nothing (it calls `_process_claims`, not `main`).
- [ ] **Step 5: Run the affected tests**
Run: `python tests\test_run_lock.py` → `16/16 passed`
Run: `python tests\test_mode_banner.py` → all passed
Run: `python tests\test_test_claim_ids_file.py` → all passed
Run: `python tests\test_clean_between_claims.py` → `20/20 passed`
Then: `dir Logs\bot_run.json` must NOT exist (tests did not write the real Logs). If it exists from a test, find which test wrote it and stub it there.
- [ ] **Step 6: Stage**
`git add src/ecw_bot/flows/export_medcor_ready.py tests/test_run_lock.py tests/test_mode_banner.py tests/test_test_claim_ids_file.py`
---
### Task 3: `QUIET_BOXES` — information boxes to the log
**Files:**
- Modify: `src/ecw_bot/settings.py` (next to `SHOW_EACH_CLAIM`, ~line 480)
- Modify: `src/ecw_bot/core/msgbox.py` (`info`, `info_box`)
- Modify: `src/ecw_bot/auth/login.py` (security-image `msgbox.info_box`, ~line 93)
- Test: `tests/test_msgbox.py` (add cases)
**Interfaces:**
- Produces: `cfg.QUIET_BOXES: bool` (env `QUIET_BOXES`, default False); `msgbox.info(title, msg, timeout_s, needs_person=False) -> str`; `msgbox.info_box(title, msg, needs_person=False) -> None`. Quiet log line: `[BOX] {title}: {first line of msg}`; `info()` returns `OK` when quiet.
- [ ] **Step 1: Write the failing tests** — add to `tests/test_msgbox.py` before its `main()`:
```python
@case
def quiet_boxes_send_an_info_box_to_the_log_not_the_screen():
import io
import contextlib
shown = []
orig = (msgbox._raw_call, msgbox.cfg.QUIET_BOXES)
msgbox._raw_call = lambda *a: shown.append(a) or 1
msgbox.cfg.QUIET_BOXES = True
out = io.StringIO()
try:
with contextlib.redirect_stdout(out):
msgbox.info_box("eCW Bot — Claim 1", "Summary : Pass\nmore")
rc = msgbox.info("eCW Bot — CSV report", "Saved.", 5)
finally:
msgbox._raw_call, msgbox.cfg.QUIET_BOXES = orig
assert shown == [], shown
assert rc == msgbox.OK
text = out.getvalue()
assert "[BOX] eCW Bot — Claim 1: Summary : Pass" in text, text
assert "more" not in text, text
@case
def quiet_boxes_still_show_a_box_that_needs_a_person():
shown = []
orig = (msgbox._raw_call, msgbox.cfg.QUIET_BOXES)
msgbox._raw_call = lambda *a: shown.append(a) or 1
msgbox.cfg.QUIET_BOXES = True
try:
msgbox.info_box("eCW Bot — Security Image", "Click Continue",
needs_person=True)
msgbox.confirm("eCW Bot — CAPTCHA", "Type it", 5)
finally:
msgbox._raw_call, msgbox.cfg.QUIET_BOXES = orig
assert len(shown) == 2, shown
@case
def without_quiet_boxes_an_info_box_is_shown():
shown = []
orig = (msgbox._raw_call, msgbox.cfg.QUIET_BOXES)
msgbox._raw_call = lambda *a: shown.append(a) or 1
msgbox.cfg.QUIET_BOXES = False
try:
msgbox.info_box("eCW Bot — Claim 1", "Summary : Pass")
finally:
msgbox._raw_call, msgbox.cfg.QUIET_BOXES = orig
assert len(shown) == 1, shown
```
- [ ] **Step 2: Run to see them fail**
Run: `python tests\test_msgbox.py`
Expected: the new cases ERROR with `AttributeError: ... 'QUIET_BOXES'` / unexpected keyword `needs_person`.
- [ ] **Step 3: Implement**
`src/ecw_bot/settings.py`, after the `SHOW_EACH_CLAIM = ...` line:
```python
# Information boxes go to the log as one "[BOX] title: first line" line
# instead of the screen. The results page sets it for the runs it starts
# (2026-10-14). Boxes a person must act on — CAPTCHA, 2FA manual step,
# security image, LIVE-WRITE confirm — are never quieted.
QUIET_BOXES = _env_bool("QUIET_BOXES", False)
```
`src/ecw_bot/core/msgbox.py` — replace `info()`:
```python
def info(title: str, msg: str, timeout_s: int,
needs_person: bool = False) -> str:
"""Informational box. Returns OK or TIMEOUT.
With QUIET_BOXES on (a run the results page started), a box that does
not need a person is one log line instead, and answers OK."""
if cfg.QUIET_BOXES and not needs_person:
first = (msg or "").strip().splitlines()[0] if (msg or "").strip() \
else ""
print(f"[BOX] {titled(title)}: {first}")
return OK
try:
rc = _raw_call(msg, titled(title), _MB_OK | _MB_ICONINFO | _MB_TOPMOST, timeout_s)
except Exception:
return TIMEOUT
return TIMEOUT if rc == _IDTIMEOUT else OK
```
and `info_box()`:
```python
def info_box(title: str, msg: str, needs_person: bool = False) -> None:
"""Informational box on the configured timeout. Never blocks the run.
`needs_person=True` for a box a person must act on (it is never
quieted by QUIET_BOXES)."""
title = titled(title)
if info(title, msg, cfg.MSGBOX_TIMEOUT_SECONDS,
needs_person=needs_person) == TIMEOUT:
print(f"[UI] '{title}' timed out with nobody at the desk — continuing")
```
(`info()` references `cfg` and `titled`, defined lower in the module — fine at call time.)
`src/ecw_bot/auth/login.py`, the security-image box: add `needs_person=True` as the last argument of that `msgbox.info_box(...)` call.
- [ ] **Step 4: Run**
Run: `python tests\test_msgbox.py` → all passed.
Run: `python tests\test_test_claim_ids_file.py` → all passed (it stubs `msgbox.info_box` with `lambda t, m:` — flow calls never pass `needs_person`, so that still works).
- [ ] **Step 5: Stage**
`git add src/ecw_bot/settings.py src/ecw_bot/core/msgbox.py src/ecw_bot/auth/login.py tests/test_msgbox.py`
---
### Task 4: Read the run log (`web/run_status.py`)
**Files:**
- Create: `web/run_status.py`
- Test: `tests/test_run_status.py`
**Interfaces:**
- Consumes: log line formats — `[LOGIN]`, `[CAPTCHA]`, `[OTP]`, `[2FA]`, `[OK] Logged in`, `[CLAIMS] --- claim n/N: <id> ---`, `[RESULT] claim n/N <id>: <Category>` (Task 2), `[GUARD] ...`; exit codes 0/2/3/5/6.
- Produces: `TAIL_LINES = 30`; `parse(text) -> {"phase", "progress": [n, N] | None, "current": str | None, "counts": {"Pass","Fail","Review","Not found": int}, "guard": str | None, "tail": [str]}`; `exit_text(code, stopped=False) -> str`; `read_tail(path, max_bytes=256_000) -> str`; `summarize(runner_status: dict) -> dict` (runner status keys from Task 5 plus `phase`, `progress`, `current`, `counts`, `tail`, `message`, `elapsed_s`).
- [ ] **Step 1: Write the failing test** — `tests/test_run_status.py`:
```python
r"""
web/run_status.py — the bot's run log read into the page's live status.
python tests\test_run_status.py
"""
import sys
import pathlib
import tempfile
ROOT = pathlib.Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(ROOT / "src"))
from web import run_status as rs # noqa: E402
CASES = []
def case(fn):
CASES.append(fn)
return fn
LOG = """\
======================================================================
2026-10-14 10:00:00 — run start
[CONFIG] user=x otp=imap
[LOGIN] Opening the login page
[OTP] Waiting for the approval mail
[OK] Logged in in 95s
[CLAIMS] --- claim 1/3: 9124000 ---
[RESULT] claim 1/3 9124000: Pass
[CLAIMS] --- claim 2/3: 9119000 ---
[CLAIMS] --- claim 2/3 done, moving to next ---
[RESULT] claim 2/3 9119000: Fail
[CLAIMS] --- claim 3/3: 9126000 ---
"""
@case
def phase_follows_the_last_tagged_line():
assert rs.parse("[LOGIN] x")["phase"] == "Logging in"
assert rs.parse("[LOGIN] x\n[2FA] y")["phase"] == \
"Waiting for the 2FA approval mail"
assert rs.parse("[OTP] y\n[OK] Logged in in 3s")["phase"] == "Logged in"
assert rs.parse(LOG)["phase"] == "Checking claims"
assert rs.parse("")["phase"] == "Starting"
@case
def progress_and_current_claim_come_from_the_claim_header():
p = rs.parse(LOG)
assert p["progress"] == [3, 3] and p["current"] == "9126000", p
@case
def a_done_line_is_not_a_claim_header():
p = rs.parse("[CLAIMS] --- claim 2/3 done, moving to next ---")
assert p["progress"] is None, p
@case
def counts_come_from_result_lines():
c = rs.parse(LOG + "[RESULT] claim 3/3 9126000: Review\n")["counts"]
assert c == {"Pass": 1, "Fail": 1, "Review": 1, "Not found": 0}, c
@case
def tail_is_the_last_lines():
text = "\n".join(f"line {i}" for i in range(100))
tail = rs.parse(text)["tail"]
assert len(tail) == rs.TAIL_LINES and tail[-1] == "line 99", tail
@case
def guard_line_is_kept():
p = rs.parse("[GUARD] Refused: next attempt allowed at 10:40")
assert p["guard"] == "[GUARD] Refused: next attempt allowed at 10:40"
@case
def exit_codes_read_as_plain_words():
assert rs.exit_text(0) == "Finished"
assert "setting" in rs.exit_text(2)
assert "guard" in rs.exit_text(3)
assert "claims step failed" in rs.exit_text(5)
assert "2FA" in rs.exit_text(6)
assert rs.exit_text(9) == "Ended with code 9"
assert rs.exit_text(None, stopped=True) == "Stopped by operator"
assert "no exit code" in rs.exit_text(None)
@case
def read_tail_reads_the_end_of_a_file_and_missing_is_empty():
with tempfile.TemporaryDirectory() as tmp:
p = pathlib.Path(tmp) / "run.log"
p.write_text("a\n" * 1000 + "LAST\n", encoding="utf-8")
assert rs.read_tail(p, max_bytes=100).endswith("LAST\n")
assert rs.read_tail(pathlib.Path(tmp) / "none.log") == ""
assert rs.read_tail(None) == ""
@case
def summarize_running_uses_the_phase_as_the_message():
with tempfile.TemporaryDirectory() as tmp:
p = pathlib.Path(tmp) / "run.log"
p.write_text(LOG, encoding="utf-8")
s = rs.summarize({"running": True, "starting": False,
"log_path": str(p), "out_path": None,
"exit_code": None, "stopped_by_operator": False,
"started_at": "2026-10-14T10:00:00",
"ended_at": None})
assert s["message"] == "Checking claims", s
assert s["progress"] == [3, 3] and "log_path" not in s, s
@case
def summarize_ended_says_how_and_how_long():
with tempfile.TemporaryDirectory() as tmp:
p = pathlib.Path(tmp) / "run.log"
p.write_text("[GUARD] Refused: next attempt allowed at 10:40\n",
encoding="utf-8")
s = rs.summarize({"running": False, "starting": False,
"log_path": str(p), "out_path": None,
"exit_code": 3, "stopped_by_operator": False,
"started_at": "2026-10-14T10:00:00",
"ended_at": "2026-10-14T10:04:12"})
assert s["message"].startswith("Login guard refused"), s
assert "next attempt allowed at 10:40" in s["message"], s
assert s["elapsed_s"] == 252, s
@case
def summarize_starting_reads_the_child_output_file():
with tempfile.TemporaryDirectory() as tmp:
out = pathlib.Path(tmp) / "web_run.out"
out.write_text("Traceback: boom\n", encoding="utf-8")
s = rs.summarize({"running": True, "starting": True,
"log_path": None, "out_path": str(out),
"exit_code": None, "stopped_by_operator": False,
"started_at": None, "ended_at": None})
assert s["message"] == "Starting" and s["tail"] == ["Traceback: boom"], s
def main() -> int:
failures = 0
for fn in CASES:
name = fn.__name__.replace("_", " ")
try:
fn()
print(f" ok {name}")
except AssertionError as exc:
print(f" FAIL {name}: {exc}")
failures += 1
except Exception as exc:
print(f" ERROR {name}: {type(exc).__name__}: {exc}")
failures += 1
print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(main())
```
- [ ] **Step 2: Run to see it fail**
Run: `python tests\test_run_status.py`
Expected: `ImportError: cannot import name 'run_status' from 'web'`
- [ ] **Step 3: Write `web/run_status.py`**
```python
r"""
The bot's run log, read into the results page's live status (spec
docs/superpowers/specs/2026-10-14-start-bot-from-web-page-design.md).
Everything comes from lines the bot already prints, plus its one
fixed-format line per claim, `[RESULT] claim n/N <id>: <Category>`.
The log can hold patient data — the page shows it on 127.0.0.1 only, as it
shows the results table.
"""
import re
from datetime import datetime
TAIL_LINES = 30
_CLAIM = re.compile(r"^\[CLAIMS\] --- claim (\d+)/(\d+): (\S+) ---$")
_RESULT = re.compile(r"^\[RESULT\] claim (\d+)/(\d+) (\S+): (.+)$")
_PHASES = (
(("[LOGIN]", "[CAPTCHA]"), "Logging in"),
(("[OTP]", "[2FA]"), "Waiting for the 2FA approval mail"),
(("[OK] Logged in",), "Logged in"),
(("[CLAIMS]", "[RESULT]"), "Checking claims"),
)
_EXIT = {
0: "Finished",
2: "Stopped before login — a setting needs fixing, or another run is "
"going (see the log)",
3: "Login guard refused the login — too many attempts this hour",
5: "The claims step failed (see the log)",
6: "The 2FA approval mail did not arrive in time",
}
def parse(text: str) -> dict:
phase, progress, current, guard = "Starting", None, None, None
counts = {"Pass": 0, "Fail": 0, "Review": 0, "Not found": 0}
lines = (text or "").splitlines()
for raw in lines:
line = raw.strip()
for prefixes, name in _PHASES:
if line.startswith(prefixes):
phase = name
m = _CLAIM.match(line)
if m:
progress = [int(m.group(1)), int(m.group(2))]
current = m.group(3)
m = _RESULT.match(line)
if m:
cat = m.group(4).strip()
counts[cat] = counts.get(cat, 0) + 1
if line.startswith("[GUARD]"):
guard = line
return {"phase": phase, "progress": progress, "current": current,
"counts": counts, "guard": guard,
"tail": [ln.rstrip() for ln in lines[-TAIL_LINES:]]}
def exit_text(code, stopped: bool = False) -> str:
if stopped:
return "Stopped by operator"
if code is None:
return "Ended (no exit code — the run was stopped or crashed)"
return _EXIT.get(code, f"Ended with code {code}")
def read_tail(path, max_bytes: int = 256_000) -> str:
"""The end of a text file; '' when there is none."""
if not path:
return ""
try:
with open(path, "rb") as fh:
fh.seek(0, 2)
size = fh.tell()
fh.seek(max(0, size - max_bytes))
return fh.read().decode("utf-8", errors="replace")
except OSError:
return ""
def _elapsed(started, ended):
try:
a = datetime.fromisoformat(started)
b = datetime.fromisoformat(ended) if ended else datetime.now()
return int((b - a).total_seconds())
except (TypeError, ValueError):
return None
def summarize(st: dict) -> dict:
"""The runner's status (web/bot_runner.py) plus what the log says.
File paths are not sent to the page."""
text = read_tail(st.get("log_path")) or read_tail(st.get("out_path"))
p = parse(text)
if st.get("running"):
message = p["phase"]
else:
message = exit_text(st.get("exit_code"),
st.get("stopped_by_operator", False))
if st.get("exit_code") == 3 and p["guard"]:
message += f" — {p['guard']}"
out = {k: v for k, v in st.items() if k not in ("log_path", "out_path")}
out.update(phase=p["phase"], progress=p["progress"],
current=p["current"], counts=p["counts"], tail=p["tail"],
message=message,
elapsed_s=_elapsed(st.get("started_at"), st.get("ended_at")))
return out
```
- [ ] **Step 4: Run**
Run: `python tests\test_run_status.py` → `11/11 passed`
- [ ] **Step 5: Stage**
`git add web/run_status.py tests/test_run_status.py`
---
### Task 5: Start / stop the child process (`web/bot_runner.py`)
**Files:**
- Create: `web/bot_runner.py`
- Test: `tests/test_bot_runner.py`
**Interfaces:**
- Consumes: `run_lock.read`, `run_lock.is_live`, `run_lock.mark_stopped` (Task 1).
- Produces: `FORCED_ENV: dict`; `class AlreadyRunning(Exception)`; `class NotRunning(Exception)`; `class BotRunner(command=None, root=ROOT)` with `start() -> dict`, `stop() -> dict`, `status() -> dict`. Status keys: `running, starting, pid, started_at, ended_at, exit_code, stopped_by_operator, source, log_path, out_path`.
- [ ] **Step 1: Write the failing test** — `tests/test_bot_runner.py`:
```python
r"""
web/bot_runner.py — start, watch and stop the bot as a child process. A
fake child script stands in for run.py: it takes the run record the way the
bot does, writes the env it was given, and waits.
python tests\test_bot_runner.py
"""
import os
import sys
import json
import time
import pathlib
import tempfile
ROOT = pathlib.Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(ROOT / "src"))
from ecw_bot import settings as cfg # noqa: E402
from ecw_bot.core import run_lock # noqa: E402
from web import bot_runner as br # noqa: E402
CASES = []
def case(fn):
CASES.append(fn)
return fn
# The child: like run.py, it takes the record (cfg.LOGS from argv), dumps
# the env keys under test, then sleeps until killed or `seconds` pass.
CHILD = r'''
import os, sys, json, time, pathlib
sys.path.insert(0, sys.argv[1])
from ecw_bot import settings as cfg
from ecw_bot.core import run_lock
cfg.LOGS = pathlib.Path(sys.argv[2])
log = cfg.LOGS / "run_child.log"
log.write_text("[LOGIN] fake\n", encoding="utf-8")
run_lock.acquire(log)
keys = ["SAVE_CORRECTIONS", "ECW_KEEP_OPEN", "QUIET_BOXES",
"CONFIRM_BEFORE_LOOKUP", "ECW_RUN_SOURCE"]
(cfg.LOGS / "env.json").write_text(
json.dumps({k: os.environ.get(k) for k in keys}), encoding="utf-8")
time.sleep(float(sys.argv[3]))
run_lock.finish(0)
'''
def _runner(tmp, seconds=30):
script = pathlib.Path(tmp) / "child.py"
script.write_text(CHILD, encoding="utf-8")
cmd = [sys.executable, str(script), str(ROOT / "src"), str(cfg.LOGS),
str(seconds)]
return br.BotRunner(command=cmd, root=ROOT)
def _wait(pred, timeout=15):
end = time.monotonic() + timeout
while time.monotonic() < end:
if pred():
return True
time.sleep(0.2)
return False
@case
def start_runs_the_child_with_the_forced_env(tmp):
os.environ["SAVE_CORRECTIONS"] = "1" # the parent says LIVE-WRITE
try:
r = _runner(tmp)
st = r.start()
assert st["running"] is True, st
assert _wait(lambda: (cfg.LOGS / "env.json").exists())
env = json.loads((cfg.LOGS / "env.json").read_text(encoding="utf-8"))
assert env == {"SAVE_CORRECTIONS": "0", "ECW_KEEP_OPEN": "0",
"QUIET_BOXES": "1", "CONFIRM_BEFORE_LOOKUP": "0",
"ECW_RUN_SOURCE": "web"}, env
assert _wait(lambda: r.status()["starting"] is False)
st = r.status()
assert st["running"] is True and st["source"] == "web", st
assert st["log_path"].endswith("run_child.log"), st
finally:
os.environ.pop("SAVE_CORRECTIONS", None)
try:
r.stop()
except br.NotRunning:
pass
@case
def a_second_start_is_refused(tmp):
r = _runner(tmp)
r.start()
try:
assert _wait(lambda: run_lock.is_live(run_lock.read()))
for other in (r, _runner(tmp)):
try:
other.start()
except br.AlreadyRunning as exc:
assert "already going" in str(exc), exc
else:
raise AssertionError("second start allowed")
finally:
r.stop()
@case
def stop_kills_the_run_and_marks_it_stopped(tmp):
r = _runner(tmp)
r.start()
assert _wait(lambda: run_lock.is_live(run_lock.read()))
pid = run_lock.read()["pid"]
st = r.stop()
assert st["running"] is False, st
assert st["stopped_by_operator"] is True, st
assert run_lock.pid_alive(pid) is False
@case
def stop_with_nothing_running_says_so(tmp):
try:
_runner(tmp).stop()
except br.NotRunning:
pass
else:
raise AssertionError("stop of nothing did not raise")
@case
def a_run_that_ends_reports_its_exit_code(tmp):
r = _runner(tmp, seconds=0.5)
r.start()
assert _wait(lambda: r.status()["running"] is False)
st = r.status()
assert st["exit_code"] == 0 and st["stopped_by_operator"] is False, st
@case
def a_terminal_run_is_seen_and_blocks_start(tmp):
"""The record of a live run this runner did not start (this test
process stands in for a terminal run)."""
run_lock.acquire("terminal.log", source="terminal")
r = _runner(tmp)
st = r.status()
assert st["running"] is True and st["source"] == "terminal", st
try:
r.start()
except br.AlreadyRunning:
pass
else:
raise AssertionError("started over a terminal run")
run_lock.finish(0)
def main() -> int:
failures = 0
orig = cfg.LOGS
for fn in CASES:
name = fn.__name__.replace("_", " ")
with tempfile.TemporaryDirectory() as tmp:
cfg.LOGS = pathlib.Path(tmp) / "Logs"
cfg.LOGS.mkdir()
try:
fn(tmp)
print(f" ok {name}")
except AssertionError as exc:
print(f" FAIL {name}: {exc}")
failures += 1
except Exception as exc:
print(f" ERROR {name}: {type(exc).__name__}: {exc}")
failures += 1
finally:
cfg.LOGS = orig
print(f"\n{len(CASES) - failures}/{len(CASES)} passed")
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(main())
```
- [ ] **Step 2: Run to see it fail**
Run: `python tests\test_bot_runner.py`
Expected: `ImportError: cannot import name 'bot_runner' from 'web'`
- [ ] **Step 3: Write `web/bot_runner.py`**
```python
r"""
Start, watch and stop the bot from the results page (spec
docs/superpowers/specs/2026-10-14-start-bot-from-web-page-design.md).
The bot runs as its own process — `python run.py`, the same entry point as
the terminal — so a hung or crashed run never takes the web server with
it, and closing the page does not stop the run. The command and its
environment are fixed here; nothing from a request reaches them.
The run itself says whether it is live: `Logs\bot_run.json`
(`core/run_lock.py`), which the bot writes at start and finishes at the
end, terminal runs included.
"""
import os
import sys
import pathlib
import subprocess
from datetime import datetime
from ecw_bot import settings as cfg
from ecw_bot.core import run_lock
ROOT = pathlib.Path(__file__).resolve().parent.parent
# A page-started run: VERIFY whatever .env says, no "Press Enter", no
# information boxes (the page shows the run), no "Run the Lookup?" box.
FORCED_ENV = {
"SAVE_CORRECTIONS": "0",
"ECW_KEEP_OPEN": "0",
"QUIET_BOXES": "1",
"CONFIRM_BEFORE_LOOKUP": "0",
"ECW_RUN_SOURCE": "web",
}
_NO_WINDOW = getattr(subprocess, "CREATE_NO_WINDOW", 0)
class AlreadyRunning(Exception):
pass
class NotRunning(Exception):
pass
class BotRunner:
def __init__(self, command=None, root=ROOT):
self.command = list(command or [sys.executable, str(root / "run.py")])
self.root = pathlib.Path(root)
self._proc = None
self._out_path = None
def _child_running(self) -> bool:
return self._proc is not None and self._proc.poll() is None
def start(self) -> dict:
rec = run_lock.read()
if run_lock.is_live(rec):
raise AlreadyRunning(
f"A run is already going (started "
f"{(rec or {}).get('started_at') or '?'})")
if self._child_running():
raise AlreadyRunning("A run is already going (still starting)")
env = dict(os.environ)
env.update(FORCED_ENV)
cfg.LOGS.mkdir(parents=True, exist_ok=True)
self._out_path = cfg.LOGS / f"web_run_{datetime.now():%Y%m%d_%H%M%S}.out"
with open(self._out_path, "ab") as out:
self._proc = subprocess.Popen(
self.command, cwd=str(self.root), env=env,
stdin=subprocess.DEVNULL, stdout=out,
stderr=subprocess.STDOUT, creationflags=_NO_WINDOW)
print(f"[WEB] Bot run started from the page (pid {self._proc.pid})")
return self.status()
def stop(self) -> dict:
rec = run_lock.read()
pids = []
if run_lock.is_live(rec):
pids.append(int(rec["pid"]))
if self._child_running():
pids.append(self._proc.pid)
if not pids:
raise NotRunning("No run is going")
for pid in dict.fromkeys(pids):
# /T: the bot's Chrome goes with it.
subprocess.run(["taskkill", "/PID", str(pid), "/T", "/F"],
capture_output=True, creationflags=_NO_WINDOW)
if self._proc is not None:
try:
self._proc.wait(timeout=15)
except subprocess.TimeoutExpired:
pass
if rec and rec.get("pid") in pids:
run_lock.mark_stopped(rec["pid"])
print(f"[WEB] Bot run stopped from the page (pid "
f"{', '.join(str(p) for p in dict.fromkeys(pids))})")
return self.status()
def status(self) -> dict:
rec = run_lock.read() or {}
ours = self._proc is not None and rec.get("pid") == self._proc.pid
if self._proc is not None and not ours:
# The child has not written its record yet — or ended before
# it could (a crash at import, a refused second run).
if self._child_running():
return {"running": True, "starting": True, "pid": None,
"started_at": None, "ended_at": None,
"exit_code": None, "stopped_by_operator": False,
"source": "web", "log_path": None,
"out_path": str(self._out_path)}
if not run_lock.is_live(rec):
return {"running": False, "starting": False, "pid": None,
"started_at": None, "ended_at": None,
"exit_code": self._proc.returncode,
"stopped_by_operator": False, "source": "web",
"log_path": None, "out_path": str(self._out_path)}
exit_code = rec.get("exit_code")
if (ours and exit_code is None and not self._child_running()
and not rec.get("stopped_by_operator")):
exit_code = self._proc.returncode
return {"running": run_lock.is_live(rec), "starting": False,
"pid": rec.get("pid"), "started_at": rec.get("started_at"),
"ended_at": rec.get("ended_at"), "exit_code": exit_code,
"stopped_by_operator": bool(rec.get("stopped_by_operator")),
"source": rec.get("source"), "log_path": rec.get("log_path"),
"out_path": str(self._out_path) if ours and self._out_path
else None}
```
- [ ] **Step 4: Run**
Run: `python tests\test_bot_runner.py` → `6/6 passed`
- [ ] **Step 5: Stage**
`git add web/bot_runner.py tests/test_bot_runner.py`
---
### Task 6: Endpoints, guard and the "Bot run" card
**Files:**
- Modify: `web/app.py` (imports; `create_app`; module docstring)
- Modify: `web/templates/index.html`
- Test: `tests/test_web_app.py` (add cases + `FakeRunner`)
**Interfaces:**
- Consumes: `BotRunner`, `AlreadyRunning`, `NotRunning` (Task 5); `run_status.summarize` (Task 4).
- Produces: `create_app(repo, allowed_hosts=ALLOWED_HOSTS, runner=None, allowed_origins=ALLOWED_ORIGINS)`; `app.state.run_token: str`; `ALLOWED_ORIGINS = ("http://127.0.0.1:8000", "http://localhost:8000")`; routes `POST /api/bot/start`, `POST /api/bot/stop`, `GET /api/bot/status`; template variable `run_token`.
- [ ] **Step 1: Write the failing tests** — in `tests/test_web_app.py`, after `_client`:
```python
class FakeRunner:
def __init__(self, running=False):
self.running = running
self.calls = []
def _st(self):
return {"running": self.running, "starting": False, "pid": 1,
"started_at": "2026-10-14T10:00:00", "ended_at": None,
"exit_code": None, "stopped_by_operator": False,
"source": "web", "log_path": None, "out_path": None}
def start(self):
self.calls.append("start")
if self.running:
raise web_app.AlreadyRunning("A run is already going (started x)")
self.running = True
return self._st()
def stop(self):
self.calls.append("stop")
if not self.running:
raise web_app.NotRunning("No run is going")
self.running = False
return self._st()
def status(self):
return self._st()
def _bot_client(runner=None):
runner = runner or FakeRunner()
app = web_app.create_app(FakeRepo(), runner=runner)
return (TestClient(app, base_url="http://127.0.0.1:8000"), runner,
app.state.run_token)
_GOOD_ORIGIN = {"Origin": "http://127.0.0.1:8000"}
```
and the cases (before `def main()`):
```python
@case
def the_page_carries_the_run_token_and_the_start_button():
client, _, token = _bot_client()
page = client.get("/").text
assert f'content="{token}"' in page
assert 'id="br-start"' in page and "VERIFY mode" in page
@case
def start_without_the_token_or_origin_is_forbidden():
client, runner, token = _bot_client()
for headers in ({}, {"X-Run-Token": token},
{**_GOOD_ORIGIN},
{**_GOOD_ORIGIN, "X-Run-Token": "wrong"},
{"Origin": "http://evil.example", "X-Run-Token": token},
{"Referer": "http://evil.example/x",
"X-Run-Token": token}):
r = client.post("/api/bot/start", headers=headers)
assert r.status_code == 403, (headers, r.status_code)
assert runner.calls == [], runner.calls
@case
def start_and_stop_with_origin_and_token():
client, runner, token = _bot_client()
h = {**_GOOD_ORIGIN, "X-Run-Token": token}
r = client.post("/api/bot/start", headers=h)
assert r.status_code == 200 and r.json()["running"] is True, r.text
r = client.post("/api/bot/stop", headers=h)
assert r.status_code == 200 and r.json()["running"] is False, r.text
assert runner.calls == ["start", "stop"]
@case
def a_referer_from_the_page_counts_when_origin_is_absent():
client, runner, token = _bot_client()
r = client.post("/api/bot/start", headers={
"Referer": "http://localhost:8000/?run_at=x", "X-Run-Token": token})
assert r.status_code == 200, r.text
@case
def start_while_running_is_409_and_stop_of_nothing_is_409():
client, runner, token = _bot_client(FakeRunner(running=True))
h = {**_GOOD_ORIGIN, "X-Run-Token": token}
r = client.post("/api/bot/start", headers=h)
assert r.status_code == 409 and "already going" in r.json()["detail"]
client.post("/api/bot/stop", headers=h)
r = client.post("/api/bot/stop", headers=h)
assert r.status_code == 409 and "No run" in r.json()["detail"]
@case
def status_has_the_page_fields_and_no_file_paths():
client, _, _ = _bot_client(FakeRunner(running=True))
s = client.get("/api/bot/status").json()
for key in ("running", "phase", "progress", "counts", "tail",
"message", "started_at", "exit_code"):
assert key in s, (key, s)
assert "log_path" not in s and "out_path" not in s, s
```
- [ ] **Step 2: Run to see them fail**
Run: `python tests\test_web_app.py`
Expected: new cases ERROR (`create_app() got an unexpected keyword argument 'runner'`).
- [ ] **Step 3: Implement `web/app.py`**
Imports (after `from fastapi.templating import Jinja2Templates`):
```python
import secrets # noqa: E402
from urllib.parse import urlsplit # noqa: E402
```
and after `from .repo import SqlResultsRepo`:
```python
from . import run_status # noqa: E402
from .bot_runner import (AlreadyRunning, BotRunner, # noqa: E402
NotRunning)
```
After `ALLOWED_HOSTS = ...`:
```python
# The only pages allowed to Start / Stop a run: this one (2026-10-14). A
# page on another site in the same browser could otherwise POST here.
ALLOWED_ORIGINS = ("http://127.0.0.1:8000", "http://localhost:8000")
```
Add near the other exception classes:
```python
class Forbidden(Exception):
pass
```
Change the signature and add, at the top of `create_app` after `app = FastAPI(...)`:
```python
def create_app(repo, allowed_hosts=ALLOWED_HOSTS, runner=None,
allowed_origins=ALLOWED_ORIGINS) -> FastAPI:
app = FastAPI(title="eCW Charge Validation — Results",
docs_url=None, redoc_url=None, openapi_url=None)
runner = runner if runner is not None else BotRunner()
# Made per server start and put in the page: a page on another site
# cannot read it, so it cannot send it.
app.state.run_token = secrets.token_urlsafe(32)
def _check_writer(request: Request) -> None:
origin = request.headers.get("origin")
if not origin:
ref = urlsplit(request.headers.get("referer") or "")
origin = f"{ref.scheme}://{ref.netloc}" if ref.netloc else ""
if origin not in allowed_origins:
raise Forbidden()
sent = request.headers.get("x-run-token") or ""
if not secrets.compare_digest(sent, app.state.run_token):
raise Forbidden()
```
Add the handler next to the other `exception_handler`s:
```python
@app.exception_handler(Forbidden)
def _forbidden(request: Request, exc: Forbidden):
return JSONResponse({"detail": "Not allowed from here"},
status_code=403)
```
In `page()`, add `"run_token": app.state.run_token,` to the template context dict.
Before `return app`:
```python
@app.post("/api/bot/start")
def bot_start(request: Request):
_check_writer(request)
try:
return run_status.summarize(runner.start())
except AlreadyRunning as exc:
return JSONResponse({"detail": str(exc)}, status_code=409)
@app.post("/api/bot/stop")
def bot_stop(request: Request):
_check_writer(request)
try:
return run_status.summarize(runner.stop())
except NotRunning as exc:
return JSONResponse({"detail": str(exc)}, status_code=409)
@app.get("/api/bot/status")
def bot_status():
return run_status.summarize(runner.status())
```
Module docstring: replace "Read-only." with "Read-only for the results; it can also start a VERIFY run of the bot and stop it (2026-10-14, `bot_runner.py`), each POST checked for this page's Origin and token."
- [ ] **Step 4: Implement the card in `web/templates/index.html`**
In `<head>`, after the viewport meta:
```html
<meta name="run-token" content="{{ run_token }}">
```
In `<style>`, before `</style>`:
```css
.botrun { background: var(--panel); border: 1px solid var(--line);
border-radius: 8px; padding: 12px 16px; margin-bottom: 16px; }
.botrun .row { display: flex; gap: 12px; align-items: center;
flex-wrap: wrap; }
.botrun button.stop { background: var(--fail); border-color: var(--fail); }
.botrun button[disabled] { opacity: .5; cursor: default; }
.botrun progress { width: 220px; }
.botrun pre { background: var(--head); padding: 8px; border-radius: 6px;
max-height: 260px; overflow: auto; font-size: 12px;
white-space: pre-wrap; }
.chip { padding: 2px 8px; border-radius: 10px; font-size: 12px;
font-weight: 600; }
```
As the first child of `<main>` (before `{% if not runs %}`):
```html
<section class="botrun" id="botrun">
<div class="row">
<strong>Bot run</strong>
<span id="br-message">…</span>
<button type="button" id="br-start">Start run</button>
<button type="button" id="br-stop" class="stop" hidden>Stop</button>
</div>
<p class="note">VERIFY mode — nothing is saved to eCW. CAPTCHA and 2FA
boxes still come up on this PC when the login needs a person.</p>
<div class="row" id="br-progress" hidden>
<progress id="br-bar" max="1" value="0"></progress>
<span id="br-count"></span>
<span id="br-current"></span>
<span class="chip cat Pass" id="br-pass"></span>
<span class="chip cat Fail" id="br-fail"></span>
<span class="chip cat Review" id="br-review"></span>
</div>
<details id="br-logbox" hidden><summary>Live log</summary>
<pre id="br-log"></pre></details>
</section>
```
Before `</body>`:
```html
<script>
(() => {
const token = document.querySelector('meta[name="run-token"]').content;
const $ = id => document.getElementById(id);
let wasRunning = null;
const mins = s => s == null ? '' :
` in ${Math.floor(s / 60)}m ${s % 60}s`;
async function post(path) {
const r = await fetch(path, {method: 'POST',
headers: {'X-Run-Token': token}});
let body = {};
try { body = await r.json(); } catch (e) {}
if (!r.ok) alert(body.detail || ('Failed: HTTP ' + r.status));
return r.ok;
}
// Page text from the log goes in with textContent only, never innerHTML.
function show(s) {
$('br-start').hidden = s.running;
$('br-stop').hidden = !s.running;
$('br-message').textContent = s.running ? s.message
: (s.started_at ? s.message + mins(s.elapsed_s) : 'No run yet');
const p = s.progress;
$('br-progress').hidden = !p;
if (p) {
$('br-bar').max = p[1]; $('br-bar').value = p[0];
$('br-count').textContent = `${p[0]} / ${p[1]}`;
$('br-current').textContent = s.running && s.current
? `claim ${s.current}` : '';
$('br-pass').textContent = `Pass ${s.counts.Pass || 0}`;
$('br-fail').textContent = `Fail ${s.counts.Fail || 0}`;
$('br-review').textContent = `Review ${s.counts.Review || 0}`;
}
$('br-logbox').hidden = !(s.tail && s.tail.length);
$('br-log').textContent = (s.tail || []).join('\n');
}
async function poll() {
let s = null;
try {
const r = await fetch('/api/bot/status');
s = await r.json();
show(s);
} catch (e) {
$('br-message').textContent = 'Status not available';
}
if (s && wasRunning && !s.running) { location.href = '/'; return; }
if (s) wasRunning = s.running;
setTimeout(poll, s && s.running ? 2000 : 10000);
}
$('br-start').onclick = async () => {
if (!confirm('Start a VERIFY run now? The bot logs in to eCW ' +
'(2FA approval mail) and checks the claims.')) return;
$('br-start').disabled = true;
await post('/api/bot/start');
$('br-start').disabled = false;
wasRunning = true;
};
$('br-stop').onclick = async () => {
if (!confirm('Stop the run? The claim it is on is left as it is ' +
'(nothing is saved in VERIFY mode).')) return;
await post('/api/bot/stop');
};
poll();
})();
</script>
```
- [ ] **Step 5: Run**
Run: `python tests\test_web_app.py` → all passed (old cases + 6 new).
- [ ] **Step 6: Stage**
`git add web/app.py web/templates/index.html tests/test_web_app.py`
---
### Task 7: Full suite, docs, live check, commit (with the operator)
**Files:**
- Modify: `Context.md` (CRLF — patch through a Python script, not an editor that changes line endings): §2 test count and "How to run" (Start from the page), §3 code layout (`core/run_lock.py`, `web/bot_runner.py`, `web/run_status.py`), §5 web page bullet (Start/Stop/status, VERIFY only, token), §6 rulings (2026-10-14: page runs VERIFY only; info boxes off for page runs; one live run at a time), §8 last commits, §9 EEWA-17 note.
- Modify: `.env.example` — document `QUIET_BOXES` (default 0; set by the page for its runs).
- [ ] **Step 1: Full suite**
Run every `tests\test_*.py`; total must be all passed. Confirm `Logs\bot_run.json` was not created by the suite (`dir Logs\bot_run.json` → not found, unless a real run happened).
- [ ] **Step 2: Docs** — update Context.md and `.env.example` as listed.
- [ ] **Step 3: Live check — only when the operator says to run** (4/hour login guard; mailed 2FA)
1. `.\run_web.bat`, open http://127.0.0.1:8000.
2. Start run → phase moves Logging in → Waiting for the 2FA approval mail → Checking claims; progress and Pass/Fail/Review chips move; no per-claim boxes on the desktop; the log has `[BOX]` and `[RESULT]` lines and `[MODE]` says VERIFY.
3. Start again while running → "already going".
4. Let it finish → card says "Finished in …", results table shows the new run.
5. A second short run: Stop mid-way → "Stopped by operator", Chrome gone.
- [ ] **Step 4: Ask the operator to commit** — then
`git add src/ecw_bot/core/run_lock.py src/ecw_bot/flows/export_medcor_ready.py src/ecw_bot/settings.py src/ecw_bot/core/msgbox.py src/ecw_bot/auth/login.py web/run_status.py web/bot_runner.py web/app.py web/templates/index.html tests/test_run_lock.py tests/test_run_status.py tests/test_bot_runner.py tests/test_msgbox.py tests/test_web_app.py tests/test_mode_banner.py tests/test_test_claim_ids_file.py docs/superpowers/specs/2026-10-14-start-bot-from-web-page-design.md docs/superpowers/plans/2026-10-14-start-bot-from-web-page.md .env.example`
and commit `feat: start a VERIFY bot run from the results page`; Context.md in its own `docs:` commit. Never `run.py`, `tests/test_guard.py`, `tools/sql/Create_CV_Table.rpt`.
---
## Rulings made while planning (spec refinements)
- **Record kept, not deleted:** the spec said the lock file is removed when `main()` returns. The plan keeps `Logs\bot_run.json` as the latest run's record and adds `ended_at` / `exit_code` / `stopped_by_operator`, so the page can say how the run ended even after a web restart. "Live" = no `ended_at` and the pid is a running python process. Cost if wrong: one JSON file left in `Logs\`.
- **Quieting in `info()`, not only `info_box()`:** the export-report box (`pages/claim_report.py`) calls `msgbox.info()` directly; quieting at `info()` covers it. `confirm()` is never quieted.
- **The page needs the results database to render:** if SQL Server is down the page shows its error and no Start button (unchanged behaviour). Start from the page then waits for the database; the terminal still works.
Reference in new issue View Git Blame Copy Permalink