ecw-charge-validation/docs/superpowers/specs/2026-10-22-run-tracking-and-portal-sign-in-design.md

# Run tracking and Portal Sign-in — design

Date: 2026-10-22. Agreed with the operator in chat, question by question
(Q1–Q31, all "yes" to the recommended answers; Q3 changed to "who" = the
Portal User). Terms: `GLOSSARY.md`. Decision record:
`docs/adr/0001-run-events-in-files.md`. Jira: EEWA-31 (Stage 1),
EEWA-32 (Stage 2), EEWA-33 (Stage 3), each blocking the next.

## Goal

See every Run on the Portal: who started it, how long the eCW login took and
where the time went, every Run Error, Claim Error and Warning with its time,
and which claims a Run never reached. Only signed-in Portal Users see the
Portal; every Run records who started (and stopped) it.

## Decisions

| # | Decision |
|---|---|
| Q1 | Three kinds: **Run Error** (the Run stopped), **Claim Error** (one claim not checked), **Warning** (nothing stopped). **Fail** stays a Charge Summary category only. |
| Q2/Q28 | Per **eCW Login Attempt**: start, end, duration, 2FA mail wait, result; steps timed: page open, password typed, captcha, 2FA mail requested (the Log In click), mail received, approval clicked, home page. |
| Q3/Q8 | "Who" = the **Portal User** who signed in; **Started By** / **Stopped By**. |
| Q4 | Events go to `Logs\events_<run>.jsonl` (ADR 0001). |
| Q5/Q25 | Old run logs are not imported. Nothing is deleted. |
| Q6/Q18/Q19/Q22 | A **Runs** page on the Portal: list + detail, detail refreshes every 5 s while its Run is going. |
| Q9 | The Portal stays on 127.0.0.1. |
| Q10–Q17, Q20, Q21 | Sign-in (Stage 2, below). |
| Q14 | Terminal Runs are allowed; Started By = the Windows user. |
| Q15/Q23 | Stage 3: Run Error banner; `started_by` column in `CV_ChargeSummary`; the Excel sheet unchanged. |
| Q24 | Disk check at Run start: system drive under 2 GB = Warning, under 500 MB = Run Error, exit 2, before login. |
| Q26 | Claim Error = search raised / form not read / never finished loading; Charge Summary still says Review. A claim not found in eCW = Warning. |
| Q27 | **Not Reached** = queued claims a Run Error (or Stop) came before. |

## Stage 1 — Run events and the Runs page (EEWA-31)

### `core/run_events.py` (new)

- `start(log_path)` — the events file is the log's name with `run_` →
  `events_` and `.log` → `.jsonl` (`run_20261022_141530.log` →
  `events_20261022_141530.jsonl`); the Run's id is `20261022_141530`.
  Writes `run_start`.
- `emit(kind, **fields)` — one JSON line, `{"t": local ISO time to the
  millisecond, "kind": kind, ...}`, flushed at once. **Never raises**: an
  event that cannot be written is dropped, the Run carries on.
- Helpers: `warning(code, message, **f)`, `run_error(step, message,
  exit_code, exc=None)` (with `traceback.format_exc()` when an exception is
  given), `claim_error(claim, step, message)`, `login_step(step, **f)`,
  `login_attempt_start()` / `login_attempt_end(result, message="")` (the
  module numbers attempts).
- No OTP code, password or eCW username is ever put in an event.

### Event kinds

| kind | fields |
|---|---|
| `run_start` | `source` (portal / terminal), `started_by`, `mode` (VERIFY / LIVE-WRITE), `claim_limit`, `test_sheet` (file name only), `pid`, `log` |
| `disk` | `drive`, `free_mb` |
| `login_attempt_start` | `attempt` |
| `login_step` | `attempt`, `step`: `page_open`, `password`, `captcha_shown`, `captcha_solved`, `captcha_failed`, `twofa_requested`, `mail_received`, `approval_clicked`, `home` |
| `login_attempt_end` | `attempt`, `result`: `ok`, `twofa_timeout`, `captcha_failed`, `restricted`, `failed`; `message` |
| `queue` | `claims` (list) |
| `claim_start` | `n`, `total`, `claim` |
| `claim_result` | `n`, `claim`, `category` (Pass / Fail / Review / Not found) |
| `claim_error` | `claim`, `step` (`search` / `form` / `load`), `message` |
| `warning` | `code` (`not_found`, `fallback`, `dialog`, `cancel_retry`, `form_not_closed`, `disk_low`, `preflight`), `message`, optional `claim` |
| `screenshot` | `path` (a failure shot), optional `claim` |
| `run_error` | `step` (`config`, `disk`, `guard`, `login`, `restricted`, `twofa_timeout`, `preflight`, `claims`, `unexpected`), `exit_code`, `message`, `traceback` |
| `run_end` | `exit_code` |
| `stopped` | `stopped_by` (written by the Portal after Stop; Stage 2 fills the name) |

`started_by`: env `ECW_STARTED_BY` (the Portal sets it, Stage 2), else the
Windows user. `source`: as `bot_run.json` has it (`ECW_RUN_SOURCE`).

### Where the bot emits

- `flows/export_medcor_ready.py`: `main()` starts the events after
  `runlog.start()` and writes `run_end` in its `finally`; `_main` writes a
  `run_error` at every non-zero return and in every `except`, with the
  traceback; the disk check sits before login next to the CDP check;
  `_login_with_retry` wraps each `login()` in attempt start / end;
  `_process_claims` writes `queue`, `claim_start`, `claim_result`,
  `claim_error` (search raised, `_error`, `_not_evaluated`) and the
  not-found Warning.
- `auth/login.py` / `captcha.py` / `twofactor.py` / `mailbox.py`: the login
  steps.
- `core/fallback.py` `report()`, `core/page_utils.py` `auto_dialog` (the
  event only — its behaviour is unchanged, see Context.md §10) and the
  failure `shot()`, `pages/claim_form.py` Cancel retries and form-not-closed,
  `pages/preflight.py` FAIL: Warnings / screenshots.
- `web/bot_runner.py` `stop()`: appends `stopped` to the Run's events file.

### Runs page

- `web/runs.py` (new): reads `Logs\events_*.jsonl` into Run summaries and a
  Run's detail. A Run with no `run_end` is **Running** when `bot_run.json`
  names it and its pid is alive, **Stopped** when a `stopped` event is
  there, else **Ended without a record** (killed). Result: Done (exit 0),
  Run Error (exit and step), Stopped, Running.
- `GET /runs` — newest first: start, Started By, Run Source, mode,
  duration, result, eCW Login Attempts + total login time, Pass / Fail /
  Review, Claim Errors, Warnings. Filters: from / to date, result.
- `GET /runs/{run_id}` — the login attempts table (each step's time, 2FA
  wait = mail received − 2FA requested), Run Errors (traceback folded),
  Claim Errors, Warnings, Not Reached claims, the timeline, screenshot
  links; refreshes every 5 s while Running.
- `GET /runs/{run_id}/shot/{n}` — the Run's n-th screenshot event, served
  only from inside `Screenshots\` and only as `.png`: a saved page's HTML
  served from the Portal's origin would run eCW's scripts there, so the
  detail page names where the DOM file is instead of linking it.
- A "Runs" link on every page's header.

### Disk check

Settings `DISK_WARN_MB=2048`, `DISK_STOP_MB=500`, on the system drive.

## Stage 2 — Sign-in (EEWA-32)

- `CV_PortalUser` (username, display name, role, PBKDF2-SHA256 hash + salt
  + iterations, must_change, disabled, created / changed) and
  `CV_PortalSignIn` (time, username, event: sign_in / sign_out /
  wrong_password / locked / timeout) in `Charge_Validation`.
- Roles Admin / Operator / Viewer. Start / Stop: Admin, Operator. Users
  page and Sign-in history: Admin.
- `tools\add_portal_user.py` (password with `getpass`). Passwords ≥ 10
  characters; change own; Admin reset → must change at next Sign-in;
  disable, never delete.
- 5 wrong passwords → 15 minutes locked. Session cookie (HttpOnly,
  SameSite=Strict), 30 minutes idle. Every page and `/api` behind it;
  127.0.0.1 + Origin / token checks stay.
- The Portal passes the signed-in user to the child as `ECW_STARTED_BY`;
  `stop()` writes `stopped_by`. `bot_run.json` keeps `started_by` too, so
  the Dashboard's run card says "started by …".
- As built (`web/users.py`, `web/app.py`): sessions in the Portal's memory
  (a restart signs everyone out); disabling a user, resetting their
  password or changing their role ends / updates their open session at
  once; an unknown username reads and costs the same as a wrong password;
  the last active Admin cannot be demoted or disabled, nobody disables
  themselves; every Admin change is a `CV_PortalSignIn` row with `by_user`;
  every form POST carries the page token and is checked for Origin;
  `next` after Sign-in never leaves the site.

## Stage 3 — Alerts and `started_by` (EEWA-33)

- Dashboard banner when the last Run ended in a Run Error.
- `CV_ChargeSummary.started_by` (nullable column, written from the Run's
  Started By).
- As built: the banner also shows for a Run that ended without a record
  (killed or crashed — no `run_end`), never for Done, Stopped or Running;
  it names when, Started By, the step, exit code and message, how many
  claims were Not Reached, and links the Run. `create_tables.sql` adds the
  column to an existing table (`IF COL_LENGTH ... IS NULL ALTER TABLE`).
  The bot checks the column before each insert (`results_db.has_column`)
  and saves the rows without it while it is missing — a table not yet
  updated never costs a Run its rows. The Portal selects it, shows it as
  "Started By" in a claim's side panel, and a table without it reads
  "Results table needs an update — run create_tables.sql" (SQLSTATE 42S22).

## Out of scope

Network access to the Portal, email / Teams alerts, importing old logs,
deleting old events, editing settings from the Portal.