diff --git a/Context.md b/Context.md index add0d49..40090c5 100644 --- a/Context.md +++ b/Context.md @@ -348,7 +348,7 @@ that box and on the per-claim save box means Cancel. | `flows/recovery.py` | A stuck screen inside a pass (§2): `recover()` (logout check, Escape + Patient Information close + clear, reopen search) and `run_claim()` (one retry per claim, 2 in a row → `PageStuck`, exit 9) | | `core/notify.py` | The agent's mail (§2): `incident()` / `summary()`; claim numbers masked, login / error PNGs only; never fatal | | `core/watchdog.py` | Claim watchdog (2026-10-22): `watching(limit)` around the claim loop, `beat(where)` at each claim step and every message box, `check()` raises `Stalled` → exit 8; on a stall it closes this process's Playwright driver (`node.exe`) so the frozen call returns, and exits 8 itself if the main thread never comes back | -| `core/run_events.py` | A Run's timed events → `Logs\events_.jsonl` (`start`, `emit`, `warning`, `claim_error`, `run_error` with traceback, `login_attempt_start/end`, `login_step`); never raises, never writes an OTP code / password / username | +| `core/run_events.py` | A Run's timed events → `Logs\events_.jsonl` (`start`, `emit`, `warning`, `claim_error`, `run_error` with traceback, `login_attempt_start/end`, `login_step`); never raises, never writes an OTP code / password / username. Since `c9a43ac` (2026-10-30) each line is also copied to `dbo.CV_RunEvent` (§5) — only for a Run in the real `Logs` folder with `RESULTS_DB=1`; the first database error turns the copy off for that Run | | `core/browser.py` | `launch()`: Playwright Chromium with the eCW proxy, or (`BROWSER_CDP_URL`) attach to the operator's Chrome — `clear_ecw_data()` at attach and close, `cdp_ready()` checked before login | | `core/msgbox.py` | Timed Windows message boxes; `default_cancel=True` makes Enter mean Cancel | | `core/text.py` | `first_line()` shared by core / conditions / flows | @@ -641,6 +641,24 @@ the Portal (restarted 2026-10-26, [portal] terminal tab). queue file — the agent would have to write the queue size to the DB). The mockup canvas: claude.ai artifact "Charge Validation Portal Mockups" (private to the operator). +- **Runs on Dokploy (2026-10-30, `c9a43ac`).** The Dokploy Portal cannot + read the bot PC's `Logs` folder, so its Runs page showed "No runs + match" for every date. Now every Run event is also a row in + `dbo.CV_RunEvent` (run_id, at, kind, body = the same JSON line; + created in SSMS by the operator that day, script in + `tools/sql/create_tables.sql`; `cv_portal` reads it through + `db_datareader`). `web/runs.py` reads Runs through an events source: + `FileEvents` (the PC's local Portal, as before) or `DbEvents` (a + `PORTAL_SERVER` Portal; one query per page). On Dokploy the Runs list, + filter, run detail and last-run banner come from the table, and the + Dashboard's Agent run card from `runs.live_status()` (the newest Run; + "running" = no end and an event in the last 10 minutes). Failure + screenshots stay on the PC (PHI): a server shows their names only. + `tools/import_run_events.py` loaded the 34 older Runs (2,043 events, + 22–29/10) once; a second run adds nothing. Checked live: 28–29/10 lists + 15 Runs; the 21:10 Run's detail shows the 7 m 7 s 2FA mail wait. ADR + 0001 (run events in files) still holds: the file is the record, the row + a copy. --- @@ -753,7 +771,9 @@ the Portal (restarted 2026-10-26, [portal] terminal tab). History: `git log --oneline`. -Last code commits: `78409f0` (2026-10-30, Portal sage redesign — §5; +Last code commits: `c9a43ac` (2026-10-30, run events in the database — +Runs page and Agent run card on Dokploy, §5; deployed from +`portal-20261030-runs.zip`), `78409f0` (2026-10-30, Portal sage redesign — §5; deployed to Dokploy from the working tree before this commit, same files), `24c9ce4` (2026-10-30, the self-healing agent: supervisor, recovery, security image Continue, Resend pickup, operator mail — §2, @@ -898,10 +918,19 @@ test claim ids — new posts must not (§7). `.env` (the operator's address) and one test mail; a first `python agent.py` to see the security image Continue click on the real page (its HTML is saved as `security_image_page` — tighten the selector from - it) and a late-mail Resend pickup. Not built: the agent's status in the - Portal (it runs on the bot PC; the Dokploy Portal sees only the - database — a status / queue-size row would carry it), Queue / Pending - tiles. + it) and a late-mail Resend pickup. The Dokploy Dashboard's Agent run + card shows the newest bot Run from `CV_RunEvent` (`c9a43ac`), but not the + agent's own passes or waits (the supervisor writes no events yet), and + there are no Queue / Pending tiles (no queue size in the database). +- **Dokploy wording (2026-10-30, asked, not done):** the Agent run card + says "see the log" and "CAPTCHA and 2FA boxes still come up on this PC" + — true for the PC's local Portal, misleading on Dokploy; reword for a + server Portal. +- **A public IP in the Dokploy logs (2026-10-30):** `/api/bot/status` + polls from `43.254.111.194` beside the internal `10.200.1.15`. The + Portal is meant to be Ecare network / VPN only — IT to confirm the + address is Ecare's own; if not, the domain is reachable from the + internet (PHI). - **29/10 live queue (690 claims).** 208 got a result in two runs (17:18 126, 19:29 82); the rest were not reached: Ctrl-C would not stop a run blocked in a Playwright call (the tab was closed instead), a stuck diff --git a/src/ecw_bot/core/run_events.py b/src/ecw_bot/core/run_events.py index cc1179d..85f6564 100644 --- a/src/ecw_bot/core/run_events.py +++ b/src/ecw_bot/core/run_events.py @@ -8,6 +8,11 @@ writes `Logs\\events_.jsonl` beside `run_.log`: one JSON object a line, each with its time to the millisecond, flushed at once so a killed Run keeps what it wrote. The Portal's Runs page reads these files. +Every event is also copied to `dbo.CV_RunEvent` (2026-10-30, with +`RESULTS_DB=1`): the Portal on Dokploy cannot read this PC's Logs folder. +The file stays the record; the row is the same JSON line. The first +database error switches the copy off for the rest of the Run. + An event never stops a Run: anything that goes wrong writing one drops that event and nothing else. Nothing secret goes in — no OTP code, no password, no eCW username: the callers pass fixed fields, never a whole log line. @@ -22,6 +27,9 @@ from pathlib import Path _handle = None _attempt = 0 +_run_id = "" +_db_on = True +_conn = None def close() -> None: @@ -31,13 +39,62 @@ def close() -> None: def _reset() -> None: """Close the file and forget the attempt count (tests; a new Run).""" - global _handle, _attempt + global _handle, _attempt, _run_id, _db_on, _conn try: if _handle: _handle.close() except Exception: pass - _handle, _attempt = None, 0 + try: + if _conn is not None: + _conn.close() + except Exception: + pass + _handle, _attempt, _run_id, _db_on, _conn = None, 0, "", True, None + + +def _sql_insert(run_id, at, kind, body) -> None: + """The real copy: one INSERT on a connection kept for the Run.""" + global _conn + from .. import settings as cfg + # A test that points LOGS at a temp folder never writes the real table. + if not cfg.RESULTS_DB or \ + Path(cfg.LOGS).resolve() != (Path(cfg.ROOT) / "Logs").resolve(): + return + if _conn is None: + import pyodbc + _conn = pyodbc.connect(cfg.RESULTS_DB_CONN, timeout=10, + autocommit=True) + _conn.cursor().execute( + "INSERT INTO dbo.CV_RunEvent (run_id, at, kind, body) " + "VALUES (?, ?, ?, ?)", [run_id, at, kind, body]) + + +_db_writer = _sql_insert + + +def _real_run(log_path) -> bool: + """A Run in the real Logs folder: a test's log in a temp folder never + reaches the database, whatever this PC's `.env` says.""" + try: + from .. import settings as cfg + return Path(log_path).resolve().parent == Path(cfg.LOGS).resolve() + except Exception: + return False + + +def _copy(run_id: str, kind: str, line: str) -> None: + """The line to the database too. Never raises.""" + global _db_on + if not _db_on or not run_id: + return + try: + _db_writer(run_id, datetime.now(), kind, line) + except Exception as exc: + _db_on = False + why = str(exc).splitlines()[0] if str(exc) else type(exc).__name__ + print(f"[EVENTS] Not copied to the database for the rest of this " + f"run: {why}") def run_id_of(log_path) -> str: @@ -77,8 +134,9 @@ def _line(kind: str, fields: dict) -> str: def start(log_path, mode: str, claim_limit=None, test_sheet: str = "") -> None: """Open this Run's events file and write `run_start`.""" - global _handle + global _handle, _run_id _reset() + _run_id = run_id_of(log_path) if _real_run(log_path) else "" try: _handle = open(path_for(log_path), "a", encoding="utf-8", buffering=1) @@ -98,20 +156,25 @@ def emit(kind: str, **fields) -> None: """Write one event. Never raises.""" if _handle is None: return + line = _line(kind, fields) try: - _handle.write(_line(kind, fields) + "\n") + _handle.write(line + "\n") _handle.flush() except Exception: pass + _copy(_run_id, kind, line) def append_to(log_path, kind: str, **fields) -> None: """Add one event to another Run's file — the Portal, after a Stop.""" + line = _line(kind, fields) try: with open(path_for(log_path), "a", encoding="utf-8") as fh: - fh.write(_line(kind, fields) + "\n") + fh.write(line + "\n") except Exception: pass + if _real_run(log_path): + _copy(run_id_of(log_path), kind, line) def warning(code: str, message: str, **fields) -> None: diff --git a/tests/test_run_events_db.py b/tests/test_run_events_db.py new file mode 100644 index 0000000..5f898db --- /dev/null +++ b/tests/test_run_events_db.py @@ -0,0 +1,129 @@ +r""" +core/run_events.py — every event also goes to `dbo.CV_RunEvent` (operator, +2026-10-30): the Portal on Dokploy cannot read the bot PC's Logs folder, so +its Runs page showed "No runs match" for every date. + +The file stays the record of a Run; the database row is a copy. A database +that is off, missing the table or down never stops a Run — after the first +failure the copy is switched off for the rest of the Run, said once in the +log. + + python tests\test_run_events_db.py + +No database: the writer is a stand-in. +""" + +import io +import sys +import json +import pathlib +import tempfile +import contextlib + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent / "src")) + +from ecw_bot import settings as cfg # noqa: E402 +from ecw_bot.core import run_events # noqa: E402 + +CASES = [] + + +def case(fn): + CASES.append(fn) + return fn + + +def _start(tmp, writer): + run_events._db_writer = writer + log = pathlib.Path(tmp) / "run_20261030_101500.log" + run_events.start(log, "VERIFY") + return log + + +@case +def every_event_is_copied_with_its_run_id_and_kind(tmp): + rows = [] + _start(tmp, lambda run_id, at, kind, body: rows.append( + (run_id, kind, json.loads(body)))) + run_events.emit("claim_start", n=1, total=2, claim="9103000") + run_events.close() + assert [k for _, k, _ in rows] == ["run_start", "claim_start"], rows + assert {r for r, _, _ in rows} == {"20261030_101500"} + assert rows[1][2]["claim"] == "9103000" and rows[1][2]["t"], rows + + +@case +def the_file_and_the_row_carry_the_same_line(tmp): + bodies = [] + log = _start(tmp, lambda run_id, at, kind, body: bodies.append(body)) + run_events.emit("warning", code="x", message="y") + run_events.close() + lines = run_events.path_for(log).read_text(encoding="utf-8").splitlines() + assert lines == bodies, (lines, bodies) + + +@case +def a_database_error_switches_the_copy_off_and_the_run_goes_on(tmp): + calls = [] + + def boom(*a): + calls.append(a) + raise OSError("no such table") + out = io.StringIO() + with contextlib.redirect_stdout(out): + log = _start(tmp, boom) + run_events.emit("claim_start", n=1) + run_events.emit("claim_start", n=2) + run_events.close() + assert len(calls) == 1, "tried once, then off" + assert out.getvalue().count("[EVENTS]") == 1, out.getvalue() + assert len(run_events.path_for(log).read_text().splitlines()) == 3 + + +@case +def an_event_added_after_a_stop_is_copied_too(tmp): + rows = [] + run_events._db_writer = lambda run_id, at, kind, body: rows.append(kind) + log = pathlib.Path(tmp) / "run_20261030_101500.log" + run_events.append_to(log, "stopped", by="sam") + assert rows == ["stopped"], rows + + +@case +def a_run_outside_the_logs_folder_is_never_copied(tmp): + rows = [] + run_events._db_writer = lambda *a: rows.append(a) + other = pathlib.Path(tmp) / "elsewhere" + other.mkdir() + run_events.start(other / "run_20261030_101500.log", "VERIFY") + run_events.emit("claim_start", n=1) + run_events.close() + run_events.append_to(other / "run_20261030_101500.log", "stopped") + assert rows == [], "a test's run never reaches the database" + + +def main() -> int: + failures = 0 + real = run_events._db_writer + for fn in CASES: + name = fn.__name__.replace("_", " ") + with tempfile.TemporaryDirectory() as tmp: + cfg.LOGS = pathlib.Path(tmp) # a real-looking Logs folder + try: + fn(tmp) + print(f" ok {name}") + except AssertionError as exc: + print(f" FAIL {name}: {exc}") + failures += 1 + except Exception as exc: + print(f" ERROR {name}: {type(exc).__name__}: {exc}") + failures += 1 + finally: + run_events._reset() + run_events._db_writer = real + print(f"\n{len(CASES) - failures}/{len(CASES)} passed") + return 1 if failures else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_runs_from_db.py b/tests/test_runs_from_db.py new file mode 100644 index 0000000..0b7d8b6 --- /dev/null +++ b/tests/test_runs_from_db.py @@ -0,0 +1,177 @@ +r""" +web/runs.py from the database (operator, 2026-10-30): the Portal on Dokploy +cannot read the bot PC's Logs folder, so its Runs page said "No runs match" +for every date. With `DbEvents` the same summaries come from +`dbo.CV_RunEvent` — the same JSON lines the files hold. + + python tests\test_runs_from_db.py + +No database: the repo is a stand-in. +""" + +import sys +import json +import pathlib + +ROOT = pathlib.Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / "src")) +sys.path.insert(0, str(ROOT / "tests")) +sys.path.insert(0, str(ROOT)) + +from web import runs # noqa: E402 + +CASES = [] + + +def case(fn): + CASES.append(fn) + return fn + + +def _line(day, t, kind, **f): + return json.dumps({**f, "t": f"2026-10-{day}T{t}.000", "kind": kind}) + + +def _run(day, hhmmss, end_code=0): + t = f"{hhmmss[:2]}:{hhmmss[2:4]}:{hhmmss[4:]}" + run_id = f"202610{day}_{hhmmss}" + lines = [_line(day, t, "run_start", source="terminal", started_by="ops", + mode="VERIFY", claim_limit=0, test_sheet="", pid=1, + log="x"), + _line(day, t, "claim_start", n=1, total=1, claim="9103000"), + _line(day, t, "run_end", exit_code=end_code)] + return run_id, lines + + +class FakeEventsRepo: + def __init__(self, *runs_): + self.rows = [(rid, line) for rid, lines in runs_ for line in lines] + self.calls = [] + + def run_ids(self): + self.calls.append(("run_ids",)) + return sorted({r for r, _ in self.rows}, reverse=True) + + def run_event_lines(self, run_ids): + self.calls.append(("run_event_lines", tuple(run_ids))) + return [(r, b) for r, b in self.rows if r in set(run_ids)] + + +@case +def runs_come_from_the_database_newest_first_and_filtered_by_day(): + repo = FakeEventsRepo(_run("28", "161611"), _run("29", "163000"), + _run("29", "170620", end_code=9)) + src = runs.DbEvents(repo) + got = runs.list_runs(src, day_from="2026-10-29", day_to="2026-10-29") + assert [r["run_id"] for r in got] == ["20261029_170620", + "20261029_163000"], got + assert got[0]["status"] == "Run Error" and got[1]["status"] == "Done" + assert repo.calls[-1] == ("run_event_lines", + ("20261029_170620", "20261029_163000")), \ + "one query for the day's runs, not one per run" + + +@case +def one_run_and_the_latest_run_come_from_the_database(): + repo = FakeEventsRepo(_run("28", "161611"), _run("29", "163000")) + src = runs.DbEvents(repo) + assert runs.load_run(src, "20261028_161611")["status"] == "Done" + assert runs.load_run(src, "20261030_000000") is None + assert runs.latest_run(src)["run_id"] == "20261029_163000" + + +@case +def a_screenshot_is_never_served_from_the_database(): + repo = FakeEventsRepo(("20261029_163000", [ + _line("29", "16:30:00", "screenshot", path="C:/x/101500_shot.png")])) + assert runs.screenshot_path(runs.DbEvents(repo), "20261029_163000", 0, + "C:/x") is None + + +@case +def the_portal_runs_page_reads_its_events_source(): + import test_web_app as t + repo = FakeEventsRepo(_run("29", "163000")) + client, _ = t.signed_in(t.FakeRepo(), t.FakeRunner(), + events=runs.DbEvents(repo)) + html = client.get("/runs", params={"day_from": "2026-10-29", + "day_to": "2026-10-29"}).text + assert "No runs match" not in html, html[-800:] + assert "/runs/20261029_163000" in html + + +def _live_lines(day, hhmm, ss_last, end=False): + """A run whose last event is at hh:mm:ss_last; ended or not.""" + run_id = f"202610{day}_{hhmm}00" + t0 = f"{hhmm[:2]}:{hhmm[2:]}:00" + lines = [_line(day, t0, "run_start", source="terminal", started_by="ops", + mode="VERIFY", claim_limit=0, test_sheet="", pid=1, + log="x"), + _line(day, t0, "queue", claims=["9103000", "9104000"]), + _line(day, t0, "claim_start", n=1, total=2, claim="9103000"), + _line(day, f"{hhmm[:2]}:{hhmm[2:]}:{ss_last}", "claim_result", + n=1, claim="9103000", category="Fail")] + if end: + lines.append(_line(day, f"{hhmm[:2]}:{hhmm[2:]}:{ss_last}", + "run_end", exit_code=0)) + return run_id, lines + + +@case +def the_server_dashboard_shows_a_run_still_writing_events_as_running(): + from datetime import datetime + repo = FakeEventsRepo(_live_lines("30", "1015", "40")) + st = runs.live_status(runs.DbEvents(repo), + now=datetime(2026, 10, 30, 10, 17, 0)) + assert st["running"] is True and st["progress"] == [1, 2], st + assert st["counts"] == {"Pass": 0, "Fail": 1, "Review": 0}, st + assert st["source"] == "terminal" and st["tail"] == [], st + + +@case +def the_server_dashboard_shows_an_ended_or_silent_run_as_ended(): + from datetime import datetime + repo = FakeEventsRepo(_live_lines("30", "1015", "40", end=True)) + st = runs.live_status(runs.DbEvents(repo), + now=datetime(2026, 10, 30, 10, 17, 0)) + assert st["running"] is False and st["exit_code"] == 0, st + assert "Done" in st["message"] or "exit 0" in st["message"].lower() \ + or st["message"], st + silent = FakeEventsRepo(_live_lines("30", "1015", "40")) + st = runs.live_status(runs.DbEvents(silent), + now=datetime(2026, 10, 30, 11, 30, 0)) + assert st["running"] is False, "no event for over 10 minutes" + assert runs.live_status(runs.DbEvents(FakeEventsRepo()))["running"] \ + is False + + +@case +def the_server_portals_bot_status_comes_from_the_database(): + import test_web_app as t + repo = FakeEventsRepo(_live_lines("30", "1015", "40", end=True)) + client, _ = t.signed_in(t.FakeRepo(), t.FakeRunner(), + events=runs.DbEvents(repo)) + st = client.get("/api/bot/status").json() + assert st["running"] is False and st["exit_code"] == 0, st + assert st["counts"]["Fail"] == 1 and st["progress"] == [1, 2], st + + +def main() -> int: + failures = 0 + for fn in CASES: + name = fn.__name__.replace("_", " ") + try: + fn() + print(f" ok {name}") + except AssertionError as exc: + print(f" FAIL {name}: {exc}") + failures += 1 + except Exception as exc: + print(f" ERROR {name}: {type(exc).__name__}: {exc}") + failures += 1 + print(f"\n{len(CASES) - failures}/{len(CASES)} passed") + return 1 if failures else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/import_run_events.py b/tools/import_run_events.py new file mode 100644 index 0000000..692e0d9 --- /dev/null +++ b/tools/import_run_events.py @@ -0,0 +1,71 @@ +r""" +Load this PC's older Run events into `dbo.CV_RunEvent` (2026-10-30), once, +so the Portal on Dokploy lists the Runs from before the bot copied its +events to the database itself. A Run already in the table is skipped, so a +second run of this tool adds nothing. + + python tools\import_run_events.py # import + python tools\import_run_events.py --dry-run # only say what it would do + +Run it from PowerShell on the bot PC (Windows login to the database). +""" + +import sys +import json +from datetime import datetime +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(ROOT / "src")) + +from ecw_bot import settings as cfg # noqa: E402 + +RUN_PREFIX = "events_" + + +def rows_of(run_id: str, text: str) -> list: + """(run_id, at, kind, line) per timed JSON line; a bad line skipped.""" + out = [] + for line in text.splitlines(): + try: + e = json.loads(line) + at = datetime.fromisoformat(e["t"]) + except (ValueError, KeyError, TypeError): + continue + out.append((run_id, at, str(e.get("kind", ""))[:40], line)) + return out + + +def main(argv) -> int: + dry = "--dry-run" in argv + import pyodbc + conn = pyodbc.connect(cfg.RESULTS_DB_CONN, timeout=15) + cur = conn.cursor() + have = {r[0] for r in cur.execute( + "SELECT DISTINCT run_id FROM dbo.CV_RunEvent").fetchall()} + added_runs = added_rows = 0 + for path in sorted(Path(cfg.LOGS).glob(f"{RUN_PREFIX}*.jsonl")): + run_id = path.stem[len(RUN_PREFIX):] + if run_id in have: + continue + rows = rows_of(run_id, path.read_text(encoding="utf-8", + errors="replace")) + if not rows: + continue + added_runs += 1 + added_rows += len(rows) + if not dry: + cur.executemany( + "INSERT INTO dbo.CV_RunEvent (run_id, at, kind, body) " + "VALUES (?, ?, ?, ?)", rows) + if not dry: + conn.commit() + conn.close() + verb = "Would add" if dry else "Added" + print(f"{verb} {added_runs} run(s), {added_rows} event(s); " + f"{len(have)} run(s) were already in the table.") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/tools/sql/create_tables.sql b/tools/sql/create_tables.sql index 9638c3d..6e02858 100644 --- a/tools/sql/create_tables.sql +++ b/tools/sql/create_tables.sql @@ -104,6 +104,23 @@ BEGIN END GO +-- Every Run's events (2026-10-30): the same JSON lines as the bot PC's +-- Logs\events_.jsonl, so the Portal on Dokploy has a Runs page and an +-- Agent run card. Written by the bot (db_datawriter), read by cv_portal +-- (db_datareader). tools/import_run_events.py loads the older files once. +IF OBJECT_ID('dbo.CV_RunEvent', 'U') IS NULL +BEGIN + CREATE TABLE dbo.CV_RunEvent ( + id BIGINT IDENTITY(1,1) PRIMARY KEY, + run_id VARCHAR(15) NOT NULL, -- 20261030_101500, the run log's stamp + at DATETIME2(3) NOT NULL, + kind NVARCHAR(40) NOT NULL, -- run_start, claim_start, run_end, ... + body NVARCHAR(MAX) NOT NULL -- the event's JSON line + ); + CREATE INDEX IX_CV_RunEvent_run ON dbo.CV_RunEvent (run_id, id); +END +GO + -- So the bot can connect with your Windows login (no password), give that -- login read/write here. Uncomment and run once, as an admin (e.g. sa): -- diff --git a/web/app.py b/web/app.py index 6896d27..b9d5d9f 100644 --- a/web/app.py +++ b/web/app.py @@ -325,12 +325,15 @@ def create_app(repo, allowed_hosts=ALLOWED_HOSTS, runner=None, allowed_origins=ALLOWED_ORIGINS, logs_dir=None, screenshots_root=None, users_store=None, sessions=None, can_start=True, - secure_cookie=False, ecare=None) -> FastAPI: + secure_cookie=False, ecare=None, events=None) -> FastAPI: app = FastAPI(title="eCW Charge Validation — Results", docs_url=None, redoc_url=None, openapi_url=None) if runner is None: runner = BotRunner() if can_start else NoRunner() logs_dir = logs_dir if logs_dir is not None else cfg.LOGS + # Where the Runs page and the last-run banner read Runs from: this PC's + # Logs folder, or (the Portal on Dokploy, 2026-10-30) the database. + events = events if events is not None else run_view.FileEvents(logs_dir) screenshots_root = (screenshots_root if screenshots_root is not None else cfg.ROOT / "Screenshots") # Made per server start and put in the page: a page on another site @@ -554,7 +557,7 @@ def create_app(repo, allowed_hosts=ALLOWED_HOSTS, runner=None, """The latest Run, if it ended badly — the Dashboard's red banner (EEWA-33). A Run Error, or a Run that ended without a record (killed); never a Stop, a finished Run or one still going.""" - last = run_view.latest_run(logs_dir, _live_run_id()) + last = run_view.latest_run(events, _live_run_id()) if last and last["status"] in ("Run Error", "Ended without a record"): return last return None @@ -580,7 +583,7 @@ def create_app(repo, allowed_hosts=ALLOWED_HOSTS, runner=None, raise BadFilter(f"Not a date: {value}") if status and status not in run_view.STATUSES: raise BadFilter(f"Not a result: {status}") - rows = run_view.list_runs(logs_dir, _live_run_id(), + rows = run_view.list_runs(events, _live_run_id(), day_from=day_from or None, day_to=day_to or None, status=status or None) @@ -592,7 +595,7 @@ def create_app(repo, allowed_hosts=ALLOWED_HOSTS, runner=None, @app.get("/runs/{run_id}", response_class=HTMLResponse) def run_page(request: Request, run_id: str): - run = run_view.load_run(logs_dir, run_id, _live_run_id()) + run = run_view.load_run(events, run_id, _live_run_id()) if run is None: return HTMLResponse("

No such run

", status_code=404) return _page(request, "run.html", @@ -602,7 +605,7 @@ def create_app(repo, allowed_hosts=ALLOWED_HOSTS, runner=None, @app.get("/runs/{run_id}/shot/{n}") def run_shot(run_id: str, n: int): - path = run_view.screenshot_path(logs_dir, run_id, n, screenshots_root) + path = run_view.screenshot_path(events, run_id, n, screenshots_root) if path is None: return HTMLResponse("

No such screenshot

", status_code=404) @@ -652,6 +655,10 @@ def create_app(repo, allowed_hosts=ALLOWED_HOSTS, runner=None, @app.get("/api/bot/status") def bot_status(): + # A server Portal cannot see the bot PC's process or log: the + # newest Run in the database instead (2026-10-30). + if isinstance(events, run_view.DbEvents): + return jsonable_encoder(run_view.live_status(events)) return run_status.summarize(runner.status()) # ── Sign-in (Stage 2, EEWA-32) ─────────────────────────────────────── @@ -873,8 +880,11 @@ def _safe_next(target) -> str: return target -app = create_app(SqlResultsRepo(cfg.RESULTS_DB_CONN), +_repo = SqlResultsRepo(cfg.RESULTS_DB_CONN) +app = create_app(_repo, users_store=SqlUserStore(cfg.RESULTS_DB_CONN), + # A server Portal has no Logs folder: Runs from the database. + events=run_view.DbEvents(_repo) if cfg.PORTAL_SERVER else None, ecare=(EcareSignIn(cfg.AUTH_TOKEN_API_URL) if cfg.AUTH_TOKEN_API_URL else None), **portal_settings(cfg.PORTAL_URLS, cfg.PORTAL_SERVER)) diff --git a/web/repo.py b/web/repo.py index 7d5619e..1c28178 100644 --- a/web/repo.py +++ b/web/repo.py @@ -136,3 +136,21 @@ class SqlResultsRepo: f"SELECT TOP 200 {_SELECT_COLUMNS} FROM {results_db.TABLE} " f"WHERE claim_no LIKE ? ORDER BY run_at DESC, s_no, id", [f"%{claim_no}%"]) + + # ── Run events (2026-10-30): the Runs page on a server Portal ───────── + + def run_ids(self) -> list: + """Every Run with events, newest first (at most 500).""" + return [r["run_id"] for r in self._query( + "SELECT TOP 500 run_id FROM dbo.CV_RunEvent " + "GROUP BY run_id ORDER BY run_id DESC")] + + def run_event_lines(self, run_ids) -> list: + """(run_id, JSON line) for those Runs, in the order written.""" + run_ids = list(run_ids)[:500] + if not run_ids: + return [] + marks = ", ".join("?" * len(run_ids)) + return [(r["run_id"], r["body"]) for r in self._query( + f"SELECT run_id, body FROM dbo.CV_RunEvent " + f"WHERE run_id IN ({marks}) ORDER BY run_id, id", run_ids)] diff --git a/web/runs.py b/web/runs.py index 6637b00..68512b4 100644 --- a/web/runs.py +++ b/web/runs.py @@ -64,12 +64,17 @@ def read_events(logs_dir, run_id) -> list: if not valid_run_id(run_id): return [] path = Path(logs_dir) / f"events_{run_id}.jsonl" - out = [] try: text = path.read_text(encoding="utf-8", errors="replace") except OSError: return [] - for line in text.splitlines(): + return _parse(text.splitlines()) + + +def _parse(lines) -> list: + """Event dicts from JSON lines, a bad or untimed line skipped.""" + out = [] + for line in lines: try: e = json.loads(line) except ValueError: @@ -224,34 +229,79 @@ def _time_of_id(run_id): return None +class FileEvents: + """Runs from this PC's `Logs/events_.jsonl` files.""" + + def __init__(self, logs_dir): + self.dir = Path(logs_dir) + + def run_ids(self) -> list: + try: + files = sorted(self.dir.glob("events_*.jsonl"), reverse=True) + except OSError: + return [] + return [p.stem[len("events_"):] for p in files + if valid_run_id(p.stem[len("events_"):])] + + def events(self, run_ids) -> dict: + return {r: read_events(self.dir, r) for r in run_ids + if (self.dir / f"events_{r}.jsonl").is_file()} + + +class DbEvents: + """Runs from `dbo.CV_RunEvent` (2026-10-30): the Portal on Dokploy, + which cannot read the bot PC's Logs folder. One query per page.""" + + def __init__(self, repo): + self.repo = repo + + def run_ids(self) -> list: + return [r for r in self.repo.run_ids() if valid_run_id(r)] + + def events(self, run_ids) -> dict: + run_ids = list(run_ids) + if not run_ids: + return {} + lines = {} + for run_id, body in self.repo.run_event_lines(run_ids): + lines.setdefault(run_id, []).append(body) + return {r: _parse(lines[r]) for r in run_ids if r in lines} + + +def _source(src): + """A path (the old way, and the tests) or an events source.""" + return src if hasattr(src, "run_ids") else FileEvents(src) + + def load_run(logs_dir, run_id, live_run_id=None, now=None): - """One Run's detail, or None (bad id, no events file).""" + """One Run's detail, or None (bad id, no events).""" if not valid_run_id(run_id): return None - if not (Path(logs_dir) / f"events_{run_id}.jsonl").is_file(): + found = _source(logs_dir).events([run_id]) + if run_id not in found: return None - return summarize(run_id, read_events(logs_dir, run_id), live_run_id, now) + return summarize(run_id, found[run_id], live_run_id, now) def list_runs(logs_dir, live_run_id=None, now=None, day_from=None, day_to=None, status=None) -> list: - """Every Run with an events file, newest first, filtered. `day_from` - / `day_to` are ISO dates (inclusive), `status` one of STATUSES.""" + """Every Run with events, newest first, filtered. `day_from` / + `day_to` are ISO dates (inclusive), `status` one of STATUSES.""" d_from = date.fromisoformat(day_from) if day_from else None d_to = date.fromisoformat(day_to) if day_to else None - out = [] - try: - files = sorted(Path(logs_dir).glob("events_*.jsonl"), reverse=True) - except OSError: - files = [] - for path in files: - run_id = path.stem[len("events_"):] - if not valid_run_id(run_id): - continue + src = _source(logs_dir) + ids = [] + for run_id in src.run_ids(): day = _time_of_id(run_id).date() if (d_from and day < d_from) or (d_to and day > d_to): continue - r = summarize(run_id, read_events(logs_dir, run_id), live_run_id, now) + ids.append(run_id) + found = src.events(ids) + out = [] + for run_id in ids: + if run_id not in found: + continue + r = summarize(run_id, found[run_id], live_run_id, now) if status and r["status"] != status: continue out.append(r) @@ -259,24 +309,22 @@ def list_runs(logs_dir, live_run_id=None, now=None, day_from=None, def latest_run(logs_dir, live_run_id=None, now=None): - """The newest Run's summary, or None — reads one file, not all.""" - try: - files = sorted(Path(logs_dir).glob("events_*.jsonl"), reverse=True) - except OSError: - return None - for path in files: - run_id = path.stem[len("events_"):] - if valid_run_id(run_id): - return summarize(run_id, read_events(logs_dir, run_id), - live_run_id, now) + """The newest Run's summary, or None — reads one Run, not all.""" + src = _source(logs_dir) + for run_id in src.run_ids(): + found = src.events([run_id]) + if run_id in found: + return summarize(run_id, found[run_id], live_run_id, now) return None def screenshot_path(logs_dir, run_id, n, screenshots_root): """The n-th screenshot event's .png, if it is inside `screenshots_root` and exists; else None. Only paths a Run itself recorded are served.""" - shots = [e for e in read_events(logs_dir, run_id) - if e["kind"] == "screenshot"] + if not valid_run_id(run_id): + return None + events = _source(logs_dir).events([run_id]).get(run_id, []) + shots = [e for e in events if e["kind"] == "screenshot"] if not 0 <= n < len(shots): return None try: @@ -290,3 +338,50 @@ def screenshot_path(logs_dir, run_id, n, screenshots_root): or not path.is_file()): return None return path + + +LIVE_STALE_S = 600 + + +def live_status(source, now=None) -> dict: + """The Dashboard's Agent run card on a server Portal (2026-10-30): the + newest Run in the events source, in `/api/bot/status`'s shape. Running + = no end yet and an event in the last LIVE_STALE_S seconds (the server + cannot see the bot PC's process). No log tail: that is the PC's log.""" + from .run_status import exit_text + now = now or datetime.now() + out = {"running": False, "starting": False, "started_at": None, + "ended_at": None, "exit_code": None, "stopped_by_operator": False, + "source": None, "started_by": None, "phase": "", "progress": None, + "current": None, "counts": {"Pass": 0, "Fail": 0, "Review": 0}, + "tail": [], "claims": [], "results_run_at": None, + "elapsed_s": None, "message": "No run yet"} + try: + ids = source.run_ids() + except Exception: + return out + for run_id in ids: + found = source.events([run_id]) + if run_id not in found or not found[run_id]: + continue + events = found[run_id] + r = summarize(run_id, events, None, now) + ended = any(e["kind"] in ("run_end", "stopped") for e in events) + fresh = (now - events[-1]["_t"]).total_seconds() < LIVE_STALE_S + running = not ended and fresh + stopped = r["status"] == "Stopped" + out.update( + running=running, source=r["source"], started_by=r["started_by"], + started_at=r["started"].isoformat() if r["started"] else None, + ended_at=r["ended"].isoformat() if r["ended"] else None, + exit_code=r["exit_code"], stopped_by_operator=stopped, + progress=([r["claims_done"], r["claims_total"]] + if r["claims_total"] else None), + counts={"Pass": r["pass"], "Fail": r["fail"], + "Review": r["review"]}, + elapsed_s=r["duration_s"], + message=(f"Running on the bot PC · {r['claims_done']} of " + f"{r['claims_total']} claims" if running + else exit_text(r["exit_code"], stopped))) + return out + return out