ecw-charge-validation/docs/superpowers/specs/2026-10-22-run-tracking-and-portal-sign-in-design.md

Run tracking and Portal Sign-in — design

Date: 2026-10-22. Agreed with the operator in chat, question by question (Q1–Q31, all "yes" to the recommended answers; Q3 changed to "who" = the Portal User). Terms: GLOSSARY.md. Decision record: docs/adr/0001-run-events-in-files.md. Jira: EEWA-31 (Stage 1), EEWA-32 (Stage 2), EEWA-33 (Stage 3), each blocking the next.

Goal

See every Run on the Portal: who started it, how long the eCW login took and where the time went, every Run Error, Claim Error and Warning with its time, and which claims a Run never reached. Only signed-in Portal Users see the Portal; every Run records who started (and stopped) it.

Decisions

#	Decision
Q1	Three kinds: Run Error (the Run stopped), Claim Error (one claim not checked), Warning (nothing stopped). Fail stays a Charge Summary category only.
Q2/Q28	Per eCW Login Attempt: start, end, duration, 2FA mail wait, result; steps timed: page open, password typed, captcha, 2FA mail requested (the Log In click), mail received, approval clicked, home page.
Q3/Q8	"Who" = the Portal User who signed in; Started By / Stopped By.
Q4	Events go to Logs\events_<run>.jsonl (ADR 0001).
Q5/Q25	Old run logs are not imported. Nothing is deleted.
Q6/Q18/Q19/Q22	A Runs page on the Portal: list + detail, detail refreshes every 5 s while its Run is going.
Q9	The Portal stays on 127.0.0.1.
Q10–Q17, Q20, Q21	Sign-in (Stage 2, below).
Q14	Terminal Runs are allowed; Started By = the Windows user.
Q15/Q23	Stage 3: Run Error banner; started_by column in CV_ChargeSummary; the Excel sheet unchanged.
Q24	Disk check at Run start: system drive under 2 GB = Warning, under 500 MB = Run Error, exit 2, before login.
Q26	Claim Error = search raised / form not read / never finished loading; Charge Summary still says Review. A claim not found in eCW = Warning.
Q27	Not Reached = queued claims a Run Error (or Stop) came before.

Stage 1 — Run events and the Runs page (EEWA-31)

core/run_events.py (new)

• start(log_path) — the events file is the log's name with run_ → events_ and .log → .jsonl (run_20261022_141530.log → events_20261022_141530.jsonl); the Run's id is 20261022_141530. Writes run_start.
• emit(kind, **fields) — one JSON line, {"t": local ISO time to the millisecond, "kind": kind, ...}, flushed at once. Never raises: an event that cannot be written is dropped, the Run carries on.
• Helpers: warning(code, message, **f), run_error(step, message, exit_code, exc=None) (with traceback.format_exc() when an exception is given), claim_error(claim, step, message), login_step(step, **f), login_attempt_start() / login_attempt_end(result, message="") (the module numbers attempts).
• No OTP code, password or eCW username is ever put in an event.

Event kinds

kind	fields
run_start	source (portal / terminal), started_by, mode (VERIFY / LIVE-WRITE), claim_limit, test_sheet (file name only), pid, log
disk	drive, free_mb
login_attempt_start	attempt
login_step	attempt, step: page_open, password, captcha_shown, captcha_solved, captcha_failed, twofa_requested, mail_received, approval_clicked, home
login_attempt_end	attempt, result: ok, twofa_timeout, captcha_failed, restricted, failed; message
queue	claims (list)
claim_start	n, total, claim
claim_result	n, claim, category (Pass / Fail / Review / Not found)
claim_error	claim, step (search / form / load), message
warning	code (not_found, fallback, dialog, cancel_retry, form_not_closed, disk_low, preflight), message, optional claim
screenshot	path (a failure shot), optional claim
run_error	step (config, disk, guard, login, restricted, twofa_timeout, preflight, claims, unexpected), exit_code, message, traceback
run_end	exit_code
stopped	stopped_by (written by the Portal after Stop; Stage 2 fills the name)

started_by: env ECW_STARTED_BY (the Portal sets it, Stage 2), else the Windows user. source: as bot_run.json has it (ECW_RUN_SOURCE).

Where the bot emits

• flows/export_medcor_ready.py: main() starts the events after runlog.start() and writes run_end in its finally; _main writes a run_error at every non-zero return and in every except, with the traceback; the disk check sits before login next to the CDP check; _login_with_retry wraps each login() in attempt start / end; _process_claims writes queue, claim_start, claim_result, claim_error (search raised, _error, _not_evaluated) and the not-found Warning.
• auth/login.py / captcha.py / twofactor.py / mailbox.py: the login steps.
• core/fallback.py report(), core/page_utils.py auto_dialog (the event only — its behaviour is unchanged, see Context.md §10) and the failure shot(), pages/claim_form.py Cancel retries and form-not-closed, pages/preflight.py FAIL: Warnings / screenshots.
• web/bot_runner.py stop(): appends stopped to the Run's events file.

Runs page

• web/runs.py (new): reads Logs\events_*.jsonl into Run summaries and a Run's detail. A Run with no run_end is Running when bot_run.json names it and its pid is alive, Stopped when a stopped event is there, else Ended without a record (killed). Result: Done (exit 0), Run Error (exit and step), Stopped, Running.
• GET /runs — newest first: start, Started By, Run Source, mode, duration, result, eCW Login Attempts + total login time, Pass / Fail / Review, Claim Errors, Warnings. Filters: from / to date, result.
• GET /runs/{run_id} — the login attempts table (each step's time, 2FA wait = mail received − 2FA requested), Run Errors (traceback folded), Claim Errors, Warnings, Not Reached claims, the timeline, screenshot links; refreshes every 5 s while Running.
• GET /runs/{run_id}/shot/{n} — the Run's n-th screenshot event, served only from inside Screenshots\ and only as .png: a saved page's HTML served from the Portal's origin would run eCW's scripts there, so the detail page names where the DOM file is instead of linking it.
• A "Runs" link on every page's header.

Disk check

Settings DISK_WARN_MB=2048, DISK_STOP_MB=500, on the system drive.

Stage 2 — Sign-in (EEWA-32)

• CV_PortalUser (username, display name, role, PBKDF2-SHA256 hash + salt
  • iterations, must_change, disabled, created / changed) and CV_PortalSignIn (time, username, event: sign_in / sign_out / wrong_password / locked / timeout) in Charge_Validation.
• Roles Admin / Operator / Viewer. Start / Stop: Admin, Operator. Users page and Sign-in history: Admin.
• tools\add_portal_user.py (password with getpass). Passwords ≥ 10 characters; change own; Admin reset → must change at next Sign-in; disable, never delete.
• 5 wrong passwords → 15 minutes locked. Session cookie (HttpOnly, SameSite=Strict), 30 minutes idle. Every page and /api behind it; 127.0.0.1 + Origin / token checks stay.
• The Portal passes the signed-in user to the child as ECW_STARTED_BY; stop() writes stopped_by. bot_run.json keeps started_by too, so the Dashboard's run card says "started by …".
• As built (web/users.py, web/app.py): sessions in the Portal's memory (a restart signs everyone out); disabling a user, resetting their password or changing their role ends / updates their open session at once; an unknown username reads and costs the same as a wrong password; the last active Admin cannot be demoted or disabled, nobody disables themselves; every Admin change is a CV_PortalSignIn row with by_user; every form POST carries the page token and is checked for Origin; next after Sign-in never leaves the site.

Stage 3 — Alerts and started_by (EEWA-33)

• Dashboard banner when the last Run ended in a Run Error.
• CV_ChargeSummary.started_by (nullable column, written from the Run's Started By).
• As built: the banner also shows for a Run that ended without a record (killed or crashed — no run_end), never for Done, Stopped or Running; it names when, Started By, the step, exit code and message, how many claims were Not Reached, and links the Run. create_tables.sql adds the column to an existing table (IF COL_LENGTH ... IS NULL ALTER TABLE). The bot checks the column before each insert (results_db.has_column) and saves the rows without it while it is missing — a table not yet updated never costs a Run its rows. The Portal selects it, shows it as "Started By" in a claim's side panel, and a table without it reads "Results table needs an update — run create_tables.sql" (SQLSTATE 42S22).

Out of scope

Network access to the Portal, email / Teams alerts, importing old logs, deleting old events, editing settings from the Portal.