Compare commits

..
3 Commits
Author SHA1 Message Date
VasanthandClaude Opus 5.5 3e7cc4afe9 Download assigned tasks; skip only "Facesheets Needed (GG)"
Tasks assigned to someone (BT, SM, ...) were skipped, so their facesheets
were never downloaded. Only GG tasks must be skipped now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 10:54:10 +05:30
VasanthandClaude Opus 5 bfd72825b4 Skip Not-Started tasks that are already assigned to someone
A task carrying an "Assigned to" avatar in the Planner grid has been picked
up by a person, so its attachments must not be downloaded and its status must
not be moved to In Progress. Until now the only filter was percentComplete
== 0, so an assigned task with an attachment was downloaded like any other —
on the Balboa Nursing and Rehab plan that was one of the two "Facesheets
Needed (GG)" tasks assigned to Dr. Mark Lukasik and Adina Smets.

- _not_started_tasks() now selects `assignments` as well.
- _task_assignee_ids()/_resolve_user_names(): the assignee check runs before
  the task-details call, so an assigned task costs no extra Graph request;
  display names are resolved once per user and cached for the run, falling
  back to the raw ID when the lookup fails.
- Summary workbook gains a "Tasks Skipped (Assigned)" column and a second
  "Tasks" sheet with one row per task — facility, task name, assigned to,
  attachment count, Downloaded Yes/No and the reason — so a skip can be
  explained without reading the log.
- New "All assigned — skipped" facility status, distinct from "No attachments".

scripts/probe_assignments.py is the read-only probe used to confirm the rule
against the live plan before changing behaviour.

Verified on Balboa Nursing and Rehab: 4 Not Started -> 8.24.26 and 8.21.26
downloaded (unassigned), both Facesheets Needed (GG) skipped. 110 tests pass
(tests/test_uipath_pilot.py and tests/test_config.py fail on this checkout for
pre-existing, unrelated reasons).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-24 18:46:00 +05:30
VasanthandClaude Opus 5 dd864e0b4c Fix Planner login: password field submitted empty since 2026-07-31
Microsoft changed the converged sign-in page to pre-render the password
view, so input#i0118 is present and visible while the email view is still
on screen. wait_for_selector() for the password field therefore returned
immediately, the fill landed mid view-switch and was wiped, and the Sign in
click submitted an empty box ("Please enter your password."). Planner never
loaded, no Graph request fired, and the run died with "Could not capture
Bearer token" on every machine from 2026-07-31 onwards.

Login flow:
- _fill_verified(): wait for the field to be interactive, fill, and read the
  value back before submitting; retry when it does not stick.
- _submit_password()/_submit_totp(): confirm the page actually advanced, and
  raise a specific error naming Microsoft's message when a credential is
  rejected instead of timing out silently.
- _skip_login_interrupt(): dismiss the post-sign-in interstitials, including
  "Let's keep your account secure", which was also never handled. The old
  "Stay signed in?" handler only matched input#idBtn_Back and Microsoft now
  renders that control as a <button>.

Token capture:
- Listen at context level so popups are observed.
- _TokenStore falls back to any graph.microsoft.com Bearer if no Planner-scoped
  call appears, and records every Graph URL seen for diagnosis.
- MSAL localStorage fallback when no Graph request is intercepted at all.
- On failure, dump a screenshot, the page HTML and the request summary next to
  the run log so the next regression does not need a code change to diagnose.
- Login timeout 120s -> 180s, with one retry of the whole login.

Also: force UTF-8 on stdout/stderr (the error banner crashed with
UnicodeEncodeError on a cp1252 console, hiding the real error), and only
prompt "Press Enter to close" when stdin is a TTY so scheduled runs do not
fail with EOFError.

Build: onedrive_planner_http.spec was gitignored and missing from the repo, so
build_exe.bat could not run; it is now tracked. requirements.txt was missing
httpx, playwright and python-dotenv.

Verified: scripts/verify_login.py completes login, token capture and read-only
Graph access (63 plans discovered). 103 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-03 12:46:59 +05:30
9 changed files with 1284 additions and 59 deletions

No files matched your search

+5
View File
@@ -4,6 +4,11 @@ dist/
_internal/ _internal/
*.exe *.exe
*.spec *.spec
# ...except the build recipe itself — build_exe.bat cannot build without it.
!onedrive_planner_http.spec
# Diagnostics written by scripts/diag_login.py
diag/
# Chrome binary # Chrome binary
chrome-win/ chrome-win/
+514 -59
View File
@@ -6,7 +6,8 @@ Architecture:
authenticated request to graph.microsoft.com, then closes immediately. authenticated request to graph.microsoft.com, then closes immediately.
2. All subsequent work uses httpx against the Microsoft Graph API: 2. All subsequent work uses httpx against the Microsoft Graph API:
- Discover plans via user's group memberships - Discover plans via user's group memberships
- List Not-Started tasks per facility plan - List Not-Started tasks per facility plan, skipping only
"Facesheets Needed (GG)" tasks (assigned or not, all others download)
- Resolve SharePoint file references via Graph shares endpoint - Resolve SharePoint file references via Graph shares endpoint
- Stream-download each PDF - Stream-download each PDF
3. Identical output structure to the Selenium bot: 3. Identical output structure to the Selenium bot:
@@ -160,7 +161,22 @@ def _ensure_playwright_browsers() -> None:
# Logging setup # Logging setup
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
def _force_utf8_console() -> None:
"""Make stdout/stderr tolerate non-ASCII.
Windows consoles default to cp1252, and both the log format and the error
banner contain characters outside it — printing them raises
UnicodeEncodeError and hides the real error from the operator.
"""
for stream in (sys.stdout, sys.stderr):
try:
stream.reconfigure(encoding="utf-8", errors="replace")
except Exception:
pass
def _setup_logging() -> Path: def _setup_logging() -> Path:
_force_utf8_console()
log_dir = PROJECT_ROOT / "Output" / TODAY_ISO_DATE / "logs" log_dir = PROJECT_ROOT / "Output" / TODAY_ISO_DATE / "logs"
log_dir.mkdir(parents=True, exist_ok=True) log_dir.mkdir(parents=True, exist_ok=True)
log_path = log_dir / "planner_http.log" log_path = log_dir / "planner_http.log"
@@ -191,12 +207,38 @@ def _setup_logging() -> Path:
# Auth — Playwright login, Bearer token capture # Auth — Playwright login, Bearer token capture
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
class _TokenStore: # Microsoft's converged sign-in page keeps the email, password and OTC inputs
"""Collects the latest Bearer token from intercepted Playwright requests. # in the same document, so a bare wait_for_selector() for a later step returns
# immediately while an earlier view is still on screen. Every fill therefore has
# to be verified before it is submitted — see _fill_verified().
EMAIL_SELECTOR = "input[name='loginfmt'], #i0116, input[type='email']"
PASSWORD_SELECTOR = "input[name='passwd'], #i0118, input[type='password']"
OTP_SELECTOR = (
"input[name='otc'], #idTxtBx_SAOTCC_OTC, input[autocomplete='one-time-code']"
)
Only accepts tokens from Planner/group-scoped Graph endpoints to avoid # Text Microsoft shows when it rejects a credential outright (as opposed to the
capturing narrow User.Read tokens that fire during the auth flow (e.g. # "you left the box empty" validation message, which we retry).
/me, /me/photo) before the Planner-scoped tokens arrive. _FATAL_LOGIN_MESSAGES = (
"your account or password is incorrect",
"we couldn't find an account",
"that microsoft account doesn't exist",
"your account has been locked",
"sign-in was blocked",
"you can't get there from here",
"code you entered isn't working",
"verification code is incorrect",
)
class _TokenStore:
"""Collects Bearer tokens from intercepted Playwright requests.
Two tiers. Planner/group-scoped Graph endpoints are preferred, because the
narrow tokens that fire early in the auth flow (``/me``, ``/me/photo``)
lack the Planner scopes we need. If no preferred token has shown up after
``fallback_after_s``, any graph.microsoft.com Bearer is accepted rather
than failing the whole run.
""" """
# Endpoints whose tokens are guaranteed to carry Planner/Group read scope # Endpoints whose tokens are guaranteed to carry Planner/Group read scope
@@ -207,42 +249,333 @@ class _TokenStore:
"/me/transitiveMemberOf", "/me/transitiveMemberOf",
) )
def __init__(self) -> None: def __init__(self, *, fallback_after_s: float = 45.0) -> None:
self.token: str | None = None self.token: str | None = None
self.preferred: bool = False
self.graph_urls: set[str] = set()
self.bearer_hosts: set[str] = set()
self._at: float = 0.0 self._at: float = 0.0
self._start = time.monotonic()
self._fallback_after_s = fallback_after_s
def on_request(self, request) -> None: def on_request(self, request) -> None:
url = request.url url = request.url
auth = request.headers.get("authorization", "")
if not auth.startswith("Bearer "):
return
try:
self.bearer_hosts.add(url.split("/")[2])
except IndexError:
pass
if "graph.microsoft.com" not in url: if "graph.microsoft.com" not in url:
return return
if not any(p in url for p in self._PLANNER_PATHS):
return # ignore narrow auth-flow tokens (e.g. /me profile) self.graph_urls.add(url.split("?")[0])
auth = request.headers.get("authorization", "")
if auth.startswith("Bearer "):
candidate = auth[7:] candidate = auth[7:]
is_preferred = any(p in url for p in self._PLANNER_PATHS)
if is_preferred:
if candidate != self.token or not self.preferred:
self.token = candidate
self.preferred = True
self._at = time.monotonic()
_log.debug("Planner-scoped token captured from: %.100s", url)
return
# Non-preferred Graph call. Only fall back to it once we have waited
# long enough that a Planner-scoped call clearly isn't coming.
if self.preferred:
return
if time.monotonic() - self._start < self._fallback_after_s:
_log.debug("Ignoring non-Planner Graph token (still waiting): %.100s", url)
return
if candidate != self.token: if candidate != self.token:
self.token = candidate self.token = candidate
self._at = time.monotonic() self._at = time.monotonic()
_log.debug("Planner-scoped token captured from: %.80s", url) _log.warning(
"No Planner-scoped Graph call seen in %.0fs — falling back to the "
"token from %.100s",
self._fallback_after_s, url,
)
@property @property
def age_s(self) -> float: def age_s(self) -> float:
return time.monotonic() - self._at if self.token else float("inf") return time.monotonic() - self._at if self.token else float("inf")
def _is_visible(page, selector: str) -> bool:
try:
return page.locator(selector).first.is_visible(timeout=1_000)
except Exception:
return False
def _page_text(page, limit: int = 3000) -> str:
try:
return page.inner_text("body")[:limit]
except Exception:
return ""
def _fatal_login_message(page) -> str | None:
"""Return Microsoft's error text when it has rejected a credential."""
text = _page_text(page).lower()
for phrase in _FATAL_LOGIN_MESSAGES:
if phrase in text:
return phrase
return None
def _fill_verified(page, selector: str, value: str, *, label: str, attempts: int = 3) -> None:
"""Fill a login field and confirm the value actually stuck.
The sign-in page pre-renders later steps, so a fill issued while the view is
still switching gets wiped and the form submits empty ("Please enter your
password."). Reading the value back is the only reliable guard.
"""
for attempt in range(1, attempts + 1):
try:
box = page.locator(selector).first
# The field can be in the DOM but not yet interactive while the
# sign-in page switches views, so wait for it to settle first.
box.wait_for(state="visible", timeout=20_000)
box.click(timeout=10_000)
box.fill("", timeout=10_000)
box.fill(value, timeout=10_000)
if box.input_value(timeout=5_000) == value:
return
except Exception as exc:
_log.debug("%s fill attempt %d raised: %s", label, attempt, exc)
_log.warning(
"%s field did not retain its value (attempt %d/%d) — retrying",
label, attempt, attempts,
)
time.sleep(1.5)
raise RuntimeError(
f"Could not enter the {label} on the Microsoft sign-in page — the field "
"kept clearing itself. The sign-in page layout may have changed again."
)
def _submit_password(page, password: str, *, attempts: int = 3) -> None:
"""Enter the password and confirm the page moved on to the next step."""
page.wait_for_selector(PASSWORD_SELECTOR, timeout=30_000)
for attempt in range(1, attempts + 1):
# Give the view switch time to finish; the box exists in the DOM while
# the email view is still showing, so wait_for_selector returns at once.
page.wait_for_timeout(1_500)
_fill_verified(page, PASSWORD_SELECTOR, password, label="password")
page.locator(PASSWORD_SELECTOR).first.press("Enter")
deadline = time.monotonic() + 25
while time.monotonic() < deadline:
if _is_visible(page, OTP_SELECTOR) or _login_interrupt_visible(page):
_log.info("Password accepted")
return
if "login.microsoftonline.com" not in page.url and "login.live.com" not in page.url:
_log.info("Password accepted (left the sign-in host)")
return
fatal = _fatal_login_message(page)
if fatal:
raise RuntimeError(f"Microsoft rejected the sign-in: {fatal!r}")
time.sleep(1)
_log.warning(
"Still on the password page after attempt %d/%d — retrying", attempt, attempts
)
raise RuntimeError(
"Could not get past the Microsoft password page. Page said: "
+ " | ".join(ln.strip() for ln in _page_text(page, 600).splitlines() if ln.strip())
)
def _submit_totp(page, totp_secret: str, *, attempts: int = 2) -> bool:
"""Enter the authenticator code if prompted. Returns True when submitted."""
import pyotp
if not _is_visible(page, OTP_SELECTOR):
_log.debug("No TOTP prompt detected")
return False
for attempt in range(1, attempts + 1):
code = pyotp.TOTP(totp_secret).now()
_fill_verified(page, OTP_SELECTOR, code, label="authenticator code")
page.locator(OTP_SELECTOR).first.press("Enter")
_log.info("TOTP submitted (attempt %d)", attempt)
deadline = time.monotonic() + 25
while time.monotonic() < deadline:
if _login_interrupt_visible(page):
return True
if not _is_visible(page, OTP_SELECTOR):
return True
fatal = _fatal_login_message(page)
if fatal:
raise RuntimeError(f"Microsoft rejected the authenticator code: {fatal!r}")
time.sleep(1)
_log.warning("Code page still showing — retrying with a fresh code")
time.sleep(5)
raise RuntimeError("Microsoft kept asking for the authenticator code.")
# Pages Microsoft inserts between a successful sign-in and the app. Each entry
# is (text identifying the page, selectors that skip it, label for the log).
# Order matters only in that the first matching entry wins.
#
# The selectors are deliberately narrow: #idBtn_Back is also the back arrow on
# the password view, so these are only ever clicked when the page text confirms
# which interrupt we are looking at.
_LOGIN_INTERRUPTS: tuple[tuple[str, tuple[str, ...], str], ...] = (
(
"keep your account secure",
(
"#iShowSkip",
"a:has-text('Not now')",
"button:has-text('Not now')",
"a:has-text('Skip setup')",
"#iCancel",
),
"MFA setup prompt",
),
(
"stay signed in",
(
"#idBtn_Back",
"input[type='submit'][value='No']",
"button:has-text('No')",
),
"Stay signed in?",
),
(
"help us protect your account",
("#iShowSkip", "a:has-text('Skip for now')", "a:has-text('Not now')"),
"account protection prompt",
),
)
def _login_interrupt_visible(page) -> bool:
text = _page_text(page, 1500).lower()
return any(needle in text for needle, _, _ in _LOGIN_INTERRUPTS)
def _skip_login_interrupt(page) -> bool:
"""Dismiss one post-sign-in interstitial. Returns True when one was clicked."""
text = _page_text(page, 1500).lower()
for needle, selectors, label in _LOGIN_INTERRUPTS:
if needle not in text:
continue
for selector in selectors:
try:
page.click(selector, timeout=2_000)
_log.info("Dismissed Microsoft %s via %s", label, selector)
page.wait_for_timeout(2_500)
return True
except Exception:
continue
_log.warning(
"Microsoft %s is on screen but no skip control matched — page says: %s",
label,
" | ".join(ln.strip() for ln in _page_text(page, 400).splitlines() if ln.strip()),
)
return False
return False
def _token_from_msal_storage(page) -> str | None:
"""Fallback: read a Graph access token out of the SPA's MSAL localStorage.
Used when the Planner app completes login but never fires a Graph request we
can intercept, so a future change to the SPA's network calls does not break
the whole download again.
"""
try:
entries = page.evaluate(
"""() => {
const out = [];
for (let i = 0; i < localStorage.length; i++) {
const k = localStorage.key(i);
if (!k || !k.toLowerCase().includes('accesstoken')) continue;
try { out.push(JSON.parse(localStorage.getItem(k))); } catch (e) {}
}
return out;
}"""
)
except Exception as exc:
_log.debug("Could not read localStorage: %s", exc)
return None
return _pick_graph_token(entries or [])
def _pick_graph_token(entries: list[dict]) -> str | None:
"""Choose the MSAL cache entry that carries Graph Planner/Group scopes."""
wanted = ("tasks.read", "tasks.readwrite", "group.read.all", "groupmember.read.all")
best: tuple[int, str] | None = None
for entry in entries:
if not isinstance(entry, dict):
continue
secret = entry.get("secret") or ""
if not secret:
continue
target = str(entry.get("target") or "").lower()
if "graph.microsoft.com" not in target and not any(w in target for w in wanted):
continue
score = sum(1 for w in wanted if w in target)
if best is None or score > best[0]:
best = (score, secret)
if best:
_log.warning("Using Graph token from the MSAL localStorage cache (no Graph request seen)")
return best[1]
return None
def _dump_login_failure(page, store: "_TokenStore") -> Path | None:
"""Save a screenshot, the page HTML and what we saw, next to the run log."""
try:
out_dir = PROJECT_ROOT / "Output" / TODAY_ISO_DATE / "logs"
out_dir.mkdir(parents=True, exist_ok=True)
stamp = datetime.now().strftime("%H%M%S")
stem = out_dir / f"login_failure_{stamp}"
try:
page.screenshot(path=str(stem.with_suffix(".png")), full_page=True)
except Exception:
pass
try:
stem.with_suffix(".html").write_text(page.content(), encoding="utf-8", errors="replace")
except Exception:
pass
graph_lines = [f" {u}" for u in sorted(store.graph_urls)] or [" none"]
summary = [
f"url : {page.url}",
f"text : {_page_text(page, 2000)}",
"",
f"hosts that received a Bearer: {sorted(store.bearer_hosts) or 'none'}",
"graph.microsoft.com URLs seen:",
*graph_lines,
]
stem.with_suffix(".txt").write_text("\n".join(summary), encoding="utf-8", errors="replace")
_log.error("Login failure diagnostics written to %s.*", stem)
return stem
except Exception as exc:
_log.debug("Could not write login diagnostics: %s", exc)
return None
def _playwright_login_and_capture_token( def _playwright_login_and_capture_token(
cfg: AppConfig, cfg: AppConfig,
*, *,
headless: bool = True, headless: bool = True,
timeout_s: float = 120, timeout_s: float = 180,
) -> tuple[str, dict[str, str]]: ) -> tuple[str, dict[str, str]]:
""" """
Open Playwright, complete Microsoft login, wait until the Planner app Open Playwright, complete Microsoft login, wait until the Planner app
fires its first Graph API call, capture the Bearer token, then close. fires its first Graph API call, capture the Bearer token, then close.
Returns (bearer_token, cookies_dict). Returns (bearer_token, cookies_dict).
""" """
import pyotp # available via existing totp_utils dependency
store = _TokenStore() store = _TokenStore()
proxy_cfg: dict | None = None proxy_cfg: dict | None = None
@@ -275,42 +608,30 @@ def _playwright_login_and_capture_token(
"Chrome/124.0.0.0 Safari/537.36" "Chrome/124.0.0.0 Safari/537.36"
) )
) )
# Context-level so popups and secondary pages are observed too.
ctx.on("request", store.on_request)
page = ctx.new_page() page = ctx.new_page()
page.on("request", store.on_request)
try:
planner_url = cfg.planner_url or "https://planner.cloud.microsoft/" planner_url = cfg.planner_url or "https://planner.cloud.microsoft/"
_log.info("Navigating to Planner for login (%s)", planner_url[:80]) _log.info("Navigating to Planner for login (%s)", planner_url[:80])
page.goto(planner_url, timeout=60_000) page.goto(planner_url, timeout=60_000)
# Username # Username
page.wait_for_selector("input[type='email'], input[name='loginfmt']", timeout=30_000) page.wait_for_selector(EMAIL_SELECTOR, timeout=30_000)
page.fill("input[type='email'], input[name='loginfmt']", cfg.onedrive_username) _fill_verified(page, EMAIL_SELECTOR, cfg.onedrive_username, label="username")
page.click("input[type='submit'], button[type='submit']") page.locator(EMAIL_SELECTOR).first.press("Enter")
# Password # Password
page.wait_for_selector("input[type='password'], input[name='passwd']", timeout=20_000) _submit_password(page, cfg.onedrive_password)
page.fill("input[type='password'], input[name='passwd']", cfg.onedrive_password)
page.click("input[type='submit'], button[type='submit']")
# TOTP (optional) # TOTP
try: _submit_totp(page, cfg.microsoft_totp_secret)
page.wait_for_selector(
"input[name='otc'], input[autocomplete='one-time-code']",
timeout=10_000,
)
otp = pyotp.TOTP(cfg.microsoft_totp_secret).now()
page.fill("input[name='otc'], input[autocomplete='one-time-code']", otp)
page.click("input[type='submit'], button[type='submit']")
_log.info("TOTP submitted")
except Exception:
_log.debug("No TOTP prompt detected")
# "Stay signed in?" — click No # "Let's keep your account secure", "Stay signed in?", etc.
try: for _ in range(4):
page.wait_for_selector("input[id='idBtn_Back']", timeout=8_000) if not _skip_login_interrupt(page):
page.click("input[id='idBtn_Back']") break
except Exception:
pass
# Wait until Planner fires a Graph API call so we capture the token # Wait until Planner fires a Graph API call so we capture the token
deadline = time.monotonic() + timeout_s deadline = time.monotonic() + timeout_s
@@ -323,20 +644,29 @@ def _playwright_login_and_capture_token(
except Exception: except Exception:
pass pass
if not store.token: if not store.token:
_skip_login_interrupt(page)
time.sleep(1) time.sleep(1)
if not store.token: if not store.token:
store.token = _token_from_msal_storage(page)
if not store.token:
_dump_login_failure(page, store)
raise RuntimeError( raise RuntimeError(
"Could not capture Bearer token — Microsoft login may have failed " "Could not capture Bearer token — Microsoft login may have failed "
"or Planner did not fire any Graph API requests within " f"or Planner did not fire any Graph API requests within {timeout_s}s. "
f"{timeout_s}s." f"Last URL: {page.url[:120]}. "
f"Graph URLs seen: {sorted(store.graph_urls) or 'none'}. "
"See the login_failure_*.png/.html/.txt files next to this log."
) )
cookies = {c["name"]: c["value"] for c in ctx.cookies()} cookies = {c["name"]: c["value"] for c in ctx.cookies()}
finally:
browser.close() browser.close()
_log.info( _log.info(
"Login complete — token captured (age %.1fs), %d cookies harvested", "Login complete — token captured (%s, age %.1fs), %d cookies harvested",
"Planner-scoped" if store.preferred else "fallback scope",
store.age_s, len(cookies), store.age_s, len(cookies),
) )
return store.token, cookies return store.token, cookies
@@ -367,10 +697,27 @@ def _build_client(cfg: AppConfig, token: str) -> httpx.Client:
) )
def _login_with_retry(
cfg: AppConfig, *, headless: bool = True, attempts: int = 2
) -> tuple[str, dict[str, str]]:
"""Run the Playwright login, retrying once on a transient sign-in hiccup."""
last: Exception | None = None
for attempt in range(1, attempts + 1):
try:
return _playwright_login_and_capture_token(cfg, headless=headless)
except Exception as exc:
last = exc
_log.error("Login attempt %d/%d failed: %s", attempt, attempts, exc)
if attempt < attempts:
time.sleep(10)
assert last is not None
raise last
def _refresh_token(client: httpx.Client, cfg: AppConfig) -> None: def _refresh_token(client: httpx.Client, cfg: AppConfig) -> None:
"""Re-login via Playwright and update the client's Authorization header in place.""" """Re-login via Playwright and update the client's Authorization header in place."""
_log.warning("Token expired (401) — re-harvesting via Playwright...") _log.warning("Token expired (401) — re-harvesting via Playwright...")
new_token, _ = _playwright_login_and_capture_token(cfg, headless=True) new_token, _ = _login_with_retry(cfg, headless=True)
client.headers["Authorization"] = f"Bearer {new_token}" client.headers["Authorization"] = f"Bearer {new_token}"
_log.info("Token refreshed.") _log.info("Token refreshed.")
@@ -534,13 +881,48 @@ def _not_started_tasks(client: httpx.Client, plan_id: str, cfg: AppConfig) -> li
client, client,
f"planner/plans/{plan_id}/tasks", f"planner/plans/{plan_id}/tasks",
cfg, cfg,
**{"$select": "id,title,percentComplete"}, **{"$select": "id,title,percentComplete,assignments"},
) )
ns = [t for t in tasks if (t.get("percentComplete") or 0) == 0] ns = [t for t in tasks if (t.get("percentComplete") or 0) == 0]
_log.info("Plan %s: %d tasks total, %d not started", plan_id, len(tasks), len(ns)) _log.info("Plan %s: %d tasks total, %d not started", plan_id, len(tasks), len(ns))
return ns return ns
def _task_assignee_ids(task: dict) -> list[str]:
"""User IDs a task is assigned to (the Planner grid's 'Assigned to' column)."""
return [uid for uid in (task.get("assignments") or {}) if uid]
def _is_gg_task(task: dict) -> bool:
"""'Facesheets Needed (GG)' tasks are the only ones the bot must not download."""
# ponytail: title match only; also check task details notes if GG ever appears there
return "facesheets needed (gg)" in " ".join((task.get("title") or "").lower().split())
def _resolve_user_names(
client: httpx.Client,
user_ids: list[str],
cfg: AppConfig,
cache: dict[str, str],
) -> list[str]:
"""Display names for assignee IDs, cached across the run.
A lookup failure is never fatal — the ID is shown instead so the summary
still records who the task was assigned to.
"""
names: list[str] = []
for uid in user_ids:
if uid not in cache:
try:
user = _get(client, f"users/{uid}", cfg, **{"$select": "displayName"})
cache[uid] = user.get("displayName") or uid
except Exception as exc:
_log.debug("user lookup %s failed: %s", uid, exc)
cache[uid] = uid
names.append(cache[uid])
return names
def _task_file_urls(client: httpx.Client, task_id: str, cfg: AppConfig) -> list[str]: def _task_file_urls(client: httpx.Client, task_id: str, cfg: AppConfig) -> list[str]:
"""Return SharePoint file URLs from a task's references (details endpoint).""" """Return SharePoint file URLs from a task's references (details endpoint)."""
try: try:
@@ -654,7 +1036,7 @@ def _build_workable_files(
output_root: Path, output_root: Path,
today_iso: str, today_iso: str,
) -> tuple[list[Path], Path]: ) -> tuple[list[Path], Path]:
ddmmyyyy = datetime.strptime(today_iso, "%Y-%m-%d").strftime("%d-%m-%Y") mmddyyyy = datetime.strptime(today_iso, "%Y-%m-%d").strftime("%m-%d-%Y")
fac_safe = _sanitize_folder_name(facility_name) fac_safe = _sanitize_folder_name(facility_name)
workable_root = output_root / today_iso / "Workable" workable_root = output_root / today_iso / "Workable"
medicare_root = output_root / today_iso / "Workable_Medicare" medicare_root = output_root / today_iso / "Workable_Medicare"
@@ -669,9 +1051,9 @@ def _build_workable_files(
for i, src in enumerate(downloaded, start=1): for i, src in enumerate(downloaded, start=1):
ext = src.suffix or ".pdf" ext = src.suffix or ".pdf"
base = ( base = (
f"{ddmmyyyy}_{fac_safe}" f"{mmddyyyy}_{fac_safe}"
if len(downloaded) == 1 if len(downloaded) == 1
else f"{ddmmyyyy}_{fac_safe}_{i}" else f"{mmddyyyy}_{fac_safe}_{i}"
) )
# Check if this workable file already exists (previous interrupted run) # Check if this workable file already exists (previous interrupted run)
dst_first = workable_root / f"{base}{ext}" dst_first = workable_root / f"{base}{ext}"
@@ -708,6 +1090,7 @@ def _write_summary(
) -> None: ) -> None:
failure_names = {f for f, _, _ in failures} failure_names = {f for f, _, _ in failures}
rows: list[dict] = [] rows: list[dict] = []
task_rows: list[dict] = []
for facility, payload in results.items(): for facility, payload in results.items():
if facility in failure_names: if facility in failure_names:
@@ -717,10 +1100,14 @@ def _write_summary(
tasks_att = int(payload.get("tasks_with_attachments") or 0) tasks_att = int(payload.get("tasks_with_attachments") or 0)
tasks_dl = int(payload.get("tasks_with_downloads") or 0) tasks_dl = int(payload.get("tasks_with_downloads") or 0)
tasks_skip = int(payload.get("tasks_skipped_no_attachment") or 0) tasks_skip = int(payload.get("tasks_skipped_no_attachment") or 0)
tasks_gg = int(payload.get("tasks_skipped_gg") or 0)
status_changed = int(payload.get("status_changed") or 0) status_changed = int(payload.get("status_changed") or 0)
task_rows.extend(payload.get("task_rows") or [])
if tasks_total <= 0: if tasks_total <= 0:
status = "No tasks" status = "No tasks"
elif tasks_att <= 0 and tasks_gg > 0:
status = "All GG — skipped"
elif tasks_att <= 0: elif tasks_att <= 0:
status = "No attachments" status = "No attachments"
elif tasks_dl <= 0 or file_count <= 0: elif tasks_dl <= 0 or file_count <= 0:
@@ -745,6 +1132,7 @@ def _write_summary(
"Files Downloaded": file_count, "Files Downloaded": file_count,
"Tasks Not Started": tasks_total, "Tasks Not Started": tasks_total,
"Tasks With Attachments": tasks_att, "Tasks With Attachments": tasks_att,
"Tasks Skipped (GG)": tasks_gg,
"Tasks Skipped (No File)": tasks_skip, "Tasks Skipped (No File)": tasks_skip,
"Status": status, "Status": status,
"Status Changed": status_changed_label, "Status Changed": status_changed_label,
@@ -757,14 +1145,26 @@ def _write_summary(
"Files Downloaded": 0, "Files Downloaded": 0,
"Tasks Not Started": 0, "Tasks Not Started": 0,
"Tasks With Attachments": 0, "Tasks With Attachments": 0,
"Tasks Skipped (GG)": 0,
"Tasks Skipped (No File)": 0, "Tasks Skipped (No File)": 0,
"Status": status, "Status": status,
"Status Changed": "N/A", "Status Changed": "N/A",
}) })
task_columns = [
"Facility Name", "Task Name", "Assigned To",
"Attachments", "Downloaded", "Reason",
]
tasks_df = pd.DataFrame(task_rows, columns=task_columns)
summary_path.parent.mkdir(parents=True, exist_ok=True) summary_path.parent.mkdir(parents=True, exist_ok=True)
pd.DataFrame(rows).to_excel(summary_path, index=False, sheet_name="Download") with pd.ExcelWriter(summary_path, engine="openpyxl") as writer:
_log.info("Summary -> %s", summary_path) pd.DataFrame(rows).to_excel(writer, index=False, sheet_name="Download")
tasks_df.to_excel(writer, index=False, sheet_name="Tasks")
_log.info(
"Summary -> %s (%d facility row(s), %d task row(s))",
summary_path, len(rows), len(tasks_df),
)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -797,26 +1197,59 @@ def _process_facility(
tasks_with_att = 0 tasks_with_att = 0
tasks_with_dl = 0 tasks_with_dl = 0
tasks_no_att = 0 # Not Started but no file attachment → skip tasks_no_att = 0 # Not Started but no file attachment → skip
tasks_gg = 0 # "Facesheets Needed (GG)" → skip
status_changed = 0 status_changed = 0
downloaded: list[Path] = [] downloaded: list[Path] = []
task_rows: list[dict] = []
name_cache: dict[str, str] = {}
_log.info( _log.info(
" Processing ONLY Not Started tasks with attachments (%d Not Started found)", " Processing Not Started tasks with attachments, assigned or not "
"(%d Not Started found)",
tasks_total, tasks_total,
) )
for task in tasks: for task in tasks:
tid = task["id"] tid = task["id"]
title = task.get("title", "") title = task.get("title", "")
assignee_ids = _task_assignee_ids(task)
assigned_to = (
", ".join(_resolve_user_names(client, assignee_ids, cfg, name_cache))
if assignee_ids else ""
)
# GG tasks are never downloaded and their status is never touched.
# Checked before the attachment lookup so they cost no extra Graph call.
if _is_gg_task(task):
_log.info(" SKIP '%s': Facesheets Needed (GG)", title[:60])
tasks_gg += 1
task_rows.append({
"Facility Name": facility_name,
"Task Name": title,
"Assigned To": assigned_to,
"Attachments": "",
"Downloaded": "No",
"Reason": "Skipped — Facesheets Needed (GG)",
})
continue
file_urls = _task_file_urls(client, tid, cfg) file_urls = _task_file_urls(client, tid, cfg)
if not file_urls: if not file_urls:
# Not Started but no attachment — skip entirely # Not Started but no attachment — skip entirely
_log.info(" SKIP '%s': Not Started but no attachment", title[:60]) _log.info(" SKIP '%s': Not Started but no attachment", title[:60])
tasks_no_att += 1 tasks_no_att += 1
task_rows.append({
"Facility Name": facility_name,
"Task Name": title,
"Assigned To": assigned_to,
"Attachments": 0,
"Downloaded": "No",
"Reason": "Skipped — no attachment",
})
continue continue
# Not Started WITH attachment(s) — download ALL of them # Not Started, WITH attachment(s) — download ALL of them
tasks_with_att += 1 tasks_with_att += 1
total_urls = len(file_urls) total_urls = len(file_urls)
task_dl = 0 task_dl = 0
@@ -836,6 +1269,21 @@ def _process_facility(
downloaded.append(path) downloaded.append(path)
task_dl += 1 task_dl += 1
if task_dl == total_urls:
reason = f"Downloaded {task_dl}/{total_urls} attachment(s)"
elif task_dl > 0:
reason = f"Partial — {task_dl}/{total_urls} attachment(s) downloaded"
else:
reason = f"Download failed — 0/{total_urls} attachment(s)"
task_rows.append({
"Facility Name": facility_name,
"Task Name": title,
"Assigned To": assigned_to,
"Attachments": total_urls,
"Downloaded": "Yes" if task_dl else "No",
"Reason": reason,
})
if task_dl == total_urls: if task_dl == total_urls:
# Every attachment downloaded — mark In Progress # Every attachment downloaded — mark In Progress
tasks_with_dl += 1 tasks_with_dl += 1
@@ -864,9 +1312,10 @@ def _process_facility(
) )
_log.info( _log.info(
" Summary: %d Not Started | %d with attachments (processed) | " " Summary: %d Not Started | %d GG (skipped) | "
"%d no attachment (skipped) | %d file(s) downloaded", "%d with attachments (processed) | %d no attachment (skipped) | "
tasks_total, tasks_with_att, tasks_no_att, len(downloaded), "%d file(s) downloaded",
tasks_total, tasks_gg, tasks_with_att, tasks_no_att, len(downloaded),
) )
return { return {
@@ -875,6 +1324,8 @@ def _process_facility(
"tasks_with_attachments": tasks_with_att, "tasks_with_attachments": tasks_with_att,
"tasks_with_downloads": tasks_with_dl, "tasks_with_downloads": tasks_with_dl,
"tasks_skipped_no_attachment": tasks_no_att, "tasks_skipped_no_attachment": tasks_no_att,
"tasks_skipped_gg": tasks_gg,
"task_rows": task_rows,
"status_changed": status_changed, "status_changed": status_changed,
"status_via_task_details": 0, "status_via_task_details": 0,
} }
@@ -953,7 +1404,7 @@ def run_planner_http_workflow(
# --- Login (Playwright, headless) --- # --- Login (Playwright, headless) ---
_log.info("Launching Playwright for Microsoft login...") _log.info("Launching Playwright for Microsoft login...")
token, _cookies = _playwright_login_and_capture_token(cfg, headless=True) token, _cookies = _login_with_retry(cfg, headless=True)
client = _build_client(cfg, token) client = _build_client(cfg, token)
# --- Discover all plans once --- # --- Discover all plans once ---
@@ -1016,6 +1467,7 @@ def run_planner_http_workflow(
"original": [], "workable": [], "comment": err_msg[:500], "original": [], "workable": [], "comment": err_msg[:500],
"tasks_total": 0, "tasks_with_attachments": 0, "tasks_total": 0, "tasks_with_attachments": 0,
"tasks_with_downloads": 0, "tasks_skipped_no_attachment": 0, "tasks_with_downloads": 0, "tasks_skipped_no_attachment": 0,
"tasks_skipped_gg": 0, "task_rows": [],
"status_changed": 0, "status_via_task_details": 0, "status_changed": 0, "status_via_task_details": 0,
} }
save_checkpoint( save_checkpoint(
@@ -1065,6 +1517,7 @@ def _build_arg_parser() -> argparse.ArgumentParser:
if __name__ == "__main__": if __name__ == "__main__":
_frozen = getattr(sys, "frozen", False) # True when running as .exe _frozen = getattr(sys, "frozen", False) # True when running as .exe
_force_utf8_console()
args = _build_arg_parser().parse_args() args = _build_arg_parser().parse_args()
exit_code = 0 exit_code = 0
@@ -1081,8 +1534,10 @@ if __name__ == "__main__":
traceback.print_exc() traceback.print_exc()
exit_code = 1 exit_code = 1
if _frozen and exit_code != 0: # On error only, and only for a real console: keep the window open so the
# On error only: keep the window open so the user can read what went wrong. # user can read what went wrong. Scheduled runs have no stdin and would
input("\n✗ An error occurred (see above). Press Enter to close...") # otherwise die with an EOFError that buries the actual failure.
if _frozen and exit_code != 0 and sys.stdin is not None and sys.stdin.isatty():
input("\n[X] An error occurred (see above). Press Enter to close...")
sys.exit(exit_code) sys.exit(exit_code)
+83
View File
@@ -0,0 +1,83 @@
# -*- mode: python ; coding: utf-8 -*-
"""PyInstaller spec for MISF_Planner_Download.exe (onedir build).
Layout produced (matches what is deployed to D:\\MISF_Planner_Download):
MISF_Planner_Download.exe
_internal/
src/*.py <- imported via sys.path at runtime
playwright/driver/... <- node.exe + cli.js, used to auto-install Chromium
<bundled dependencies>
"""
from PyInstaller.utils.hooks import collect_data_files, collect_submodules
block_cipher = None
# src/ is shipped as data, not as a package: onedrive_planner_http.py puts it on
# sys.path at import time (ROOT / "src"), and PROJECT_ROOT resolves next to the
# .exe so Input/ and Output/ stay outside the bundle.
datas = [("src", "src")]
# node.exe + package/cli.js — _ensure_playwright_browsers() shells out to these
# on a machine that has no Chromium yet.
datas += collect_data_files("playwright")
hiddenimports = [
"checkpoint",
"config",
"logging_setup",
"onedrive_planner",
"run_log",
"totp_utils",
"pyotp",
"openpyxl",
"openpyxl.cell._writer",
]
hiddenimports += collect_submodules("playwright")
a = Analysis(
["onedrive_planner_http.py"],
pathex=["src"],
binaries=[],
datas=datas,
hiddenimports=hiddenimports,
hookspath=[],
hooksconfig={},
runtime_hooks=[],
excludes=["tkinter", "matplotlib", "PyQt5", "PySide2", "IPython"],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False,
)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
exe = EXE(
pyz,
a.scripts,
[],
exclude_binaries=True,
name="MISF_Planner_Download",
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=False,
console=True,
disable_windowed_traceback=False,
argv_emulation=False,
target_arch=None,
codesign_identity=None,
entitlements_file=None,
)
coll = COLLECT(
exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=False,
upx_exclude=[],
name="MISF_Planner_Download",
)
+5
View File
@@ -4,3 +4,8 @@ selenium==4.27.1
webdriver-manager==4.0.2 webdriver-manager==4.0.2
pyotp==2.9.0 pyotp==2.9.0
pytest==8.3.4 pytest==8.3.4
# Planner HTTP download (onedrive_planner_http.py)
httpx>=0.28
playwright>=1.49
python-dotenv>=1.0
+280
View File
@@ -0,0 +1,280 @@
#!/usr/bin/env python3
"""
Diagnostic: instrumented Microsoft login for the Planner HTTP download.
Mirrors ``_playwright_login_and_capture_token`` in onedrive_planner_http.py but
records everything that function is blind to:
* every request the browser context makes (host, path, whether it carries a
Bearer token) -> diag/requests.txt
* page URL / title / visible text after each step -> diag/NN_<step>.txt
* screenshot + full HTML after each step -> diag/NN_<step>.png/.html
* the SPA's localStorage keys and any MSAL access
tokens sitting in it -> diag/msal.json
Run: .venv\\Scripts\\python scripts\\diag_login.py [--headless] [--wait 180]
"""
from __future__ import annotations
import argparse
import json
import sys
import time
from datetime import datetime
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
SRC = ROOT / "src"
for p in (str(ROOT), str(SRC)):
if p not in sys.path:
sys.path.insert(0, p)
import pyotp # noqa: E402
from playwright.sync_api import sync_playwright # noqa: E402
from config import read_app_config # noqa: E402
DIAG = ROOT / "diag" / datetime.now().strftime("%Y%m%d_%H%M%S")
class _Recorder:
"""Records every request; flags the ones carrying an Authorization header."""
def __init__(self) -> None:
self.rows: list[tuple[float, str, bool, str]] = []
self.t0 = time.monotonic()
self.bearer_hosts: dict[str, int] = {}
def on_request(self, request) -> None:
auth = request.headers.get("authorization", "")
has_bearer = auth.startswith("Bearer ")
url = request.url
self.rows.append((time.monotonic() - self.t0, url, has_bearer, request.method))
if has_bearer:
host = url.split("/")[2] if "//" in url else url
self.bearer_hosts[host] = self.bearer_hosts.get(host, 0) + 1
def dump(self, path: Path) -> None:
lines = [
f"{t:7.1f}s {'BEARER' if b else ' '} {m:6s} {u}"
for t, u, b, m in self.rows
]
path.write_text("\n".join(lines), encoding="utf-8", errors="replace")
_step_no = 0
def snapshot(page, label: str) -> None:
"""Dump URL, title, visible text, HTML and a screenshot for one step."""
global _step_no
_step_no += 1
stem = DIAG / f"{_step_no:02d}_{label}"
try:
url, title = page.url, page.title()
except Exception as exc: # page closed / navigating
url, title = f"<unavailable: {exc}>", ""
text = ""
try:
text = page.inner_text("body")[:4000]
except Exception as exc:
text = f"<no body text: {exc}>"
stem.with_suffix(".txt").write_text(
f"URL : {url}\nTITLE : {title}\n\n--- visible text ---\n{text}\n",
encoding="utf-8",
errors="replace",
)
try:
stem.with_suffix(".html").write_text(page.content(), encoding="utf-8", errors="replace")
except Exception:
pass
try:
page.screenshot(path=str(stem.with_suffix(".png")), full_page=True)
except Exception:
pass
print(f"\n[{_step_no:02d}] {label}")
print(f" url : {url}")
print(f" title : {title}")
first = " | ".join(ln.strip() for ln in text.splitlines() if ln.strip())[:300]
print(f" text : {first}")
def dump_storage(page) -> None:
"""Dump localStorage keys and any MSAL access-token entries."""
try:
data = page.evaluate(
"""() => {
const out = {};
for (let i = 0; i < localStorage.length; i++) {
const k = localStorage.key(i);
out[k] = localStorage.getItem(k);
}
return out;
}"""
)
except Exception as exc:
print(f" localStorage unreadable: {exc}")
return
summary = {}
for key, raw in data.items():
entry = {"len": len(raw or "")}
if "accesstoken" in key.lower():
try:
parsed = json.loads(raw)
entry["target"] = parsed.get("target")
entry["realm"] = parsed.get("realm")
entry["clientId"] = parsed.get("clientId")
entry["expiresOn"] = parsed.get("expiresOn")
entry["secret_len"] = len(parsed.get("secret", "") or "")
except Exception:
entry["parse"] = "failed"
summary[key] = entry
(DIAG / "msal.json").write_text(
json.dumps(summary, indent=2), encoding="utf-8", errors="replace"
)
tokens = [k for k in summary if "accesstoken" in k.lower()]
print(f" localStorage keys: {len(summary)}, MSAL access-token entries: {len(tokens)}")
for k in tokens:
print(f" - target={summary[k].get('target')!r} secret_len={summary[k].get('secret_len')}")
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--headless", action="store_true", help="run headless (default: visible)")
ap.add_argument("--wait", type=float, default=180, help="seconds to observe after login")
args = ap.parse_args()
DIAG.mkdir(parents=True, exist_ok=True)
print(f"Diagnostics -> {DIAG}")
cfg = read_app_config()
proxy_cfg = None
if cfg.proxy_ip and cfg.proxy_port:
proxy_cfg = {"server": f"http://{cfg.proxy_ip}:{cfg.proxy_port}"}
if cfg.proxy_username and cfg.proxy_password:
proxy_cfg["username"] = cfg.proxy_username
proxy_cfg["password"] = str(cfg.proxy_password)
print(f"Proxy: {proxy_cfg}")
rec = _Recorder()
with sync_playwright() as pw:
browser = pw.chromium.launch(headless=args.headless, proxy=proxy_cfg)
ctx = browser.new_context(
user_agent=(
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0.0.0 Safari/537.36"
)
)
ctx.on("request", rec.on_request)
page = ctx.new_page()
planner_url = cfg.planner_url or "https://planner.cloud.microsoft/"
print(f"Navigating: {planner_url}")
page.goto(planner_url, timeout=60_000)
snapshot(page, "after_goto")
# --- username ---
try:
page.wait_for_selector("input[type='email'], input[name='loginfmt']", timeout=30_000)
page.fill("input[type='email'], input[name='loginfmt']", cfg.onedrive_username)
page.click("input[type='submit'], button[type='submit']")
snapshot(page, "after_username")
except Exception as exc:
print(f" username step: {exc}")
snapshot(page, "username_failed")
# --- password ---
try:
page.wait_for_selector("input[type='password'], input[name='passwd']", timeout=20_000)
page.fill("input[type='password'], input[name='passwd']", cfg.onedrive_password)
page.click("input[type='submit'], button[type='submit']")
time.sleep(3)
snapshot(page, "after_password")
except Exception as exc:
print(f" password step: {exc}")
snapshot(page, "password_failed")
# --- TOTP ---
try:
page.wait_for_selector(
"input[name='otc'], input[autocomplete='one-time-code']", timeout=15_000
)
snapshot(page, "otp_prompt")
otp = pyotp.TOTP(cfg.microsoft_totp_secret).now()
page.fill("input[name='otc'], input[autocomplete='one-time-code']", otp)
page.click("input[type='submit'], button[type='submit']")
print(f" TOTP submitted ({otp})")
time.sleep(5)
snapshot(page, "after_totp")
except Exception as exc:
print(f" no TOTP prompt: {exc}")
snapshot(page, "no_otp_prompt")
# --- whatever comes next: DO NOT click anything, just observe ---
for wait_s in (5, 10, 20):
time.sleep(wait_s)
snapshot(page, f"observe_t{wait_s}")
# Report what the current page offers as clickable controls, so we can
# see exactly how the "Stay signed in?" prompt is rendered now.
try:
controls = page.evaluate(
"""() => Array.from(
document.querySelectorAll('button, input[type=submit], input[type=button], a[role=button]')
).slice(0, 40).map(e => ({
tag: e.tagName, id: e.id, type: e.type || '',
value: e.value || '', text: (e.innerText || '').trim().slice(0, 60),
visible: !!(e.offsetWidth || e.offsetHeight)
}))"""
)
(DIAG / "controls.json").write_text(
json.dumps(controls, indent=2), encoding="utf-8", errors="replace"
)
print("\n clickable controls on current page:")
for c in controls:
if c["visible"]:
print(f" {c['tag']}#{c['id'] or '-'} type={c['type']!r} "
f"value={c['value']!r} text={c['text']!r}")
except Exception as exc:
print(f" control scan failed: {exc}")
dump_storage(page)
# --- observe for the remaining budget, watching for a Graph bearer ---
deadline = time.monotonic() + args.wait
seen_graph = False
while time.monotonic() < deadline:
graph = [r for r in rec.rows if "graph.microsoft.com" in r[1] and r[2]]
if graph and not seen_graph:
seen_graph = True
print(f"\n *** first Graph bearer request at {graph[0][0]:.1f}s: {graph[0][1][:140]}")
time.sleep(2)
snapshot(page, "final")
dump_storage(page)
rec.dump(DIAG / "requests.txt")
print("\n" + "=" * 70)
print(f"Total requests recorded : {len(rec.rows)}")
print("Hosts that received a Bearer token:")
for host, n in sorted(rec.bearer_hosts.items(), key=lambda kv: -kv[1]):
print(f" {n:5d} {host}")
graph_urls = sorted({r[1].split("?")[0] for r in rec.rows if "graph.microsoft.com" in r[1]})
print(f"\nDistinct graph.microsoft.com paths seen: {len(graph_urls)}")
for u in graph_urls[:40]:
print(f" {u}")
print("=" * 70)
print(f"Artifacts: {DIAG}")
browser.close()
return 0
if __name__ == "__main__":
raise SystemExit(main())
+88
View File
@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""
Read-only probe: for one plan, list every Not-Started task with its assignee
count and attachment count, so the "skip assigned tasks" rule can be validated
against what the Planner grid shows before any behaviour is changed.
Downloads nothing and never PATCHes a task — safe against production Planner.
Run: .venv\\Scripts\\python scripts\\probe_assignments.py --facility "Balboa"
"""
from __future__ import annotations
import argparse
import logging
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
SRC = ROOT / "src"
for p in (str(ROOT), str(SRC)):
if p not in sys.path:
sys.path.insert(0, p)
from config import read_app_config # noqa: E402
from onedrive_planner_http import ( # noqa: E402
_build_client,
_discover_all_plans,
_get_paged,
_login_with_retry,
_match_plan,
_task_file_urls,
)
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--facility", required=True, help="facility / plan name fragment")
ap.add_argument("--headed", action="store_true")
args = ap.parse_args()
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)-7s %(message)s",
stream=sys.stdout,
)
cfg = read_app_config()
token, _ = _login_with_retry(cfg, headless=not args.headed)
client = _build_client(cfg, token)
plans = _discover_all_plans(client, cfg)
plan = _match_plan(plans, args.facility)
if not plan:
print(f"no plan matched {args.facility!r}")
return 1
print(f"\nplan: {plan.get('title')!r}\n")
tasks = _get_paged(
client,
f"planner/plans/{plan['id']}/tasks",
cfg,
**{"$select": "id,title,percentComplete,assignments,orderHint"},
)
print(f"{'%':>4} {'assignees':>9} {'files':>5} title")
print("-" * 78)
would_download = would_skip = 0
for t in tasks:
pct = t.get("percentComplete") or 0
if pct != 0:
continue
assignees = len(t.get("assignments") or {})
n_files = len(_task_file_urls(client, t["id"], cfg))
print(f"{pct:>4} {assignees:>9} {n_files:>5} {(t.get('title') or '')[:50]}")
if n_files:
if assignees:
would_skip += 1
else:
would_download += 1
print("-" * 78)
print(f"With the 'skip assigned' rule: {would_download} would download, "
f"{would_skip} would skip (attachment-bearing tasks only)")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+78
View File
@@ -0,0 +1,78 @@
#!/usr/bin/env python3
"""
Read-only end-to-end check of the login fix.
Runs the real Playwright login, then exercises the Graph API with the captured
token: plan discovery plus a Not-Started task listing for the first facility.
Downloads nothing and never PATCHes a task, so it is safe to run against
production Planner.
Run: .venv\\Scripts\\python scripts\\verify_login.py [--headed]
"""
from __future__ import annotations
import argparse
import logging
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
SRC = ROOT / "src"
for p in (str(ROOT), str(SRC)):
if p not in sys.path:
sys.path.insert(0, p)
from config import read_app_config, read_facility_list # noqa: E402
from onedrive_planner_http import ( # noqa: E402
_build_client,
_discover_all_plans,
_login_with_retry,
_match_plan,
_not_started_tasks,
)
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--headed", action="store_true", help="show the browser window")
args = ap.parse_args()
logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s %(levelname)-8s %(name)s — %(message)s",
stream=sys.stdout,
)
cfg = read_app_config()
facilities = read_facility_list()
print("\n--- login ---")
token, cookies = _login_with_retry(cfg, headless=not args.headed)
print(f"token length : {len(token)}")
print(f"cookies : {len(cookies)}")
client = _build_client(cfg, token)
print("\n--- plan discovery ---")
plans = _discover_all_plans(client, cfg)
print(f"plans visible: {len(plans)}")
for p in plans[:5]:
print(f" {p.get('title')!r} (group {p.get('_group_name')!r})")
facility = facilities[0]
print(f"\n--- read-only task listing for {facility!r} ---")
plan = _match_plan(plans, facility)
if not plan:
print(" no matching plan found for this facility")
return 1
tasks = _not_started_tasks(client, plan["id"], cfg)
print(f" plan {plan.get('title')!r}: {len(tasks)} Not-Started task(s)")
for t in tasks[:5]:
print(f" - {t.get('title')!r}")
print("\nOK — login, token capture and Graph API access all work.")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+135
View File
@@ -0,0 +1,135 @@
"""Assigned tasks download like any other; only "Facesheets Needed (GG)" is skipped.
A task assigned to someone (BT, SM, ...) with a facesheet attached must still be
downloaded. GG tasks are never downloaded and their status is never touched.
"""
from __future__ import annotations
import sys
from pathlib import Path
import pandas as pd
ROOT = Path(__file__).resolve().parent.parent
SRC = ROOT / "src"
for _p in (str(ROOT), str(SRC)):
if _p not in sys.path:
sys.path.insert(0, _p)
from onedrive_planner_http import _is_gg_task, _task_assignee_ids, _write_summary # noqa: E402
class TestTaskAssigneeIds:
def test_unassigned_task_has_no_assignees(self):
assert _task_assignee_ids({"title": "8.24.26"}) == []
assert _task_assignee_ids({"title": "8.21.26", "assignments": {}}) == []
assert _task_assignee_ids({"assignments": None}) == []
def test_assigned_task_returns_every_user_id(self):
task = {
"title": "Facesheets Needed (GG)",
"assignments": {
"aaaaaaaa-1111-2222-3333-444444444444": {"orderHint": "8"},
"bbbbbbbb-5555-6666-7777-888888888888": {"orderHint": "9"},
},
}
assert sorted(_task_assignee_ids(task)) == [
"aaaaaaaa-1111-2222-3333-444444444444",
"bbbbbbbb-5555-6666-7777-888888888888",
]
def test_blank_user_id_is_ignored(self):
assert _task_assignee_ids({"assignments": {"": {}}}) == []
class TestGgSkip:
def test_gg_title_is_skipped(self):
assert _is_gg_task({"title": "Facesheets Needed (GG)"})
assert _is_gg_task({"title": " facesheets needed (gg) "})
def test_assigned_non_gg_task_is_downloaded(self):
# Reo Vista / Balboa misses: assigned to BT / SM, must NOT be skipped.
assert not _is_gg_task({"title": "10.26.26", "assignments": {"u1": {}}})
assert not _is_gg_task(
{"title": "Melzer, Gilbert 8/23/63-Need Medi-Cal ID", "assignments": {"u2": {}}}
)
assert not _is_gg_task({"title": None})
class TestSummaryWorkbook:
def _write(self, tmp_path: Path, payload: dict) -> Path:
out = tmp_path / "Download_Summary_HTTP_test.xlsx"
_write_summary({"Balboa Nursing and Rehab": payload}, [], out, "2026-08-24 18:00")
return out
def _balboa_payload(self) -> dict:
"""The real Balboa plan: 2 downloaded (one assigned), 2 GG skipped."""
return {
"original": [Path("a.pdf"), Path("b.pdf")],
"tasks_total": 4,
"tasks_with_attachments": 2,
"tasks_with_downloads": 2,
"tasks_skipped_no_attachment": 0,
"tasks_skipped_gg": 2,
"status_changed": 2,
"task_rows": [
{"Facility Name": "Balboa Nursing and Rehab", "Task Name": "10.26.26",
"Assigned To": "B T", "Attachments": 1, "Downloaded": "Yes",
"Reason": "Downloaded 1/1 attachment(s)"},
{"Facility Name": "Balboa Nursing and Rehab", "Task Name": "8.21.26",
"Assigned To": "", "Attachments": 1, "Downloaded": "Yes",
"Reason": "Downloaded 1/1 attachment(s)"},
{"Facility Name": "Balboa Nursing and Rehab",
"Task Name": "Facesheets Needed (GG)", "Assigned To": "A S",
"Attachments": "", "Downloaded": "No",
"Reason": "Skipped — Facesheets Needed (GG)"},
{"Facility Name": "Balboa Nursing and Rehab",
"Task Name": "Facesheets Needed (GG)", "Assigned To": "D M",
"Attachments": "", "Downloaded": "No",
"Reason": "Skipped — Facesheets Needed (GG)"},
],
}
def test_download_sheet_reports_gg_skips(self, tmp_path):
out = self._write(tmp_path, self._balboa_payload())
df = pd.read_excel(out, sheet_name="Download")
assert df.loc[0, "Tasks Skipped (GG)"] == 2
assert df.loc[0, "Files Downloaded"] == 2
assert df.loc[0, "Status"] == "Success"
def test_tasks_sheet_has_a_yes_no_row_per_task(self, tmp_path):
out = self._write(tmp_path, self._balboa_payload())
df = pd.read_excel(out, sheet_name="Tasks")
assert list(df.columns) == [
"Facility Name", "Task Name", "Assigned To",
"Attachments", "Downloaded", "Reason",
]
assert list(df["Downloaded"]) == ["Yes", "Yes", "No", "No"]
facesheets = df[df["Task Name"] == "Facesheets Needed (GG)"]
assert len(facesheets) == 2
assert all(facesheets["Downloaded"] == "No")
assert all("GG" in r for r in facesheets["Reason"])
def test_facility_whose_tasks_are_all_gg(self, tmp_path):
out = self._write(tmp_path, {
"original": [],
"tasks_total": 3,
"tasks_with_attachments": 0,
"tasks_with_downloads": 0,
"tasks_skipped_no_attachment": 0,
"tasks_skipped_gg": 3,
"status_changed": 0,
"task_rows": [],
})
df = pd.read_excel(out, sheet_name="Download")
assert df.loc[0, "Status"] == "All GG — skipped"
def test_tasks_sheet_exists_even_with_no_task_rows(self, tmp_path):
out = self._write(tmp_path, {
"original": [], "tasks_total": 0, "tasks_with_attachments": 0,
"tasks_with_downloads": 0, "tasks_skipped_no_attachment": 0,
"tasks_skipped_gg": 0, "status_changed": 0, "task_rows": [],
})
df = pd.read_excel(out, sheet_name="Tasks")
assert df.empty
assert "Downloaded" in df.columns
+96
View File
@@ -0,0 +1,96 @@
"""Unit tests for the Planner HTTP login helpers (token capture + MSAL cache)."""
from __future__ import annotations
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
SRC = ROOT / "src"
for _p in (str(ROOT), str(SRC)):
if _p not in sys.path:
sys.path.insert(0, _p)
from onedrive_planner_http import _pick_graph_token, _TokenStore # noqa: E402
GRAPH_PLANNER = "https://graph.microsoft.com/beta/planner/plans/TestPlanId?$expand=details"
GRAPH_ME = "https://graph.microsoft.com/v1.0/me/photo/$value"
NON_GRAPH = "https://planner.cloud.microsoft/webui/api/something"
class _Request:
def __init__(self, url: str, token: str | None = None) -> None:
self.url = url
self.headers = {"authorization": f"Bearer {token}"} if token else {}
class TestTokenStore:
def test_captures_planner_scoped_token(self):
store = _TokenStore()
store.on_request(_Request(GRAPH_PLANNER, "planner-tok"))
assert store.token == "planner-tok"
assert store.preferred is True
def test_ignores_narrow_graph_token_inside_wait_window(self):
store = _TokenStore(fallback_after_s=45)
store.on_request(_Request(GRAPH_ME, "narrow-tok"))
assert store.token is None
# ...and records the URL so a failure is diagnosable from the log
assert GRAPH_ME.split("?")[0] in store.graph_urls
def test_falls_back_to_any_graph_token_after_window(self):
store = _TokenStore(fallback_after_s=0)
store.on_request(_Request(GRAPH_ME, "narrow-tok"))
assert store.token == "narrow-tok"
assert store.preferred is False
def test_planner_token_overrides_earlier_fallback(self):
store = _TokenStore(fallback_after_s=0)
store.on_request(_Request(GRAPH_ME, "narrow-tok"))
store.on_request(_Request(GRAPH_PLANNER, "planner-tok"))
assert store.token == "planner-tok"
assert store.preferred is True
def test_fallback_does_not_override_planner_token(self):
store = _TokenStore(fallback_after_s=0)
store.on_request(_Request(GRAPH_PLANNER, "planner-tok"))
store.on_request(_Request(GRAPH_ME, "narrow-tok"))
assert store.token == "planner-tok"
def test_non_graph_bearer_is_recorded_but_not_used(self):
store = _TokenStore(fallback_after_s=0)
store.on_request(_Request(NON_GRAPH, "spa-tok"))
assert store.token is None
assert "planner.cloud.microsoft" in store.bearer_hosts
def test_requests_without_authorization_are_ignored(self):
store = _TokenStore(fallback_after_s=0)
store.on_request(_Request(GRAPH_PLANNER))
assert store.token is None
assert store.graph_urls == set()
class TestPickGraphToken:
def test_picks_entry_with_graph_scopes(self):
entries = [
{"target": "openid profile", "secret": "id-tok"},
{"target": "Tasks.Read Group.Read.All", "secret": "graph-tok"},
]
assert _pick_graph_token(entries) == "graph-tok"
def test_prefers_the_entry_with_more_wanted_scopes(self):
entries = [
{"target": "https://graph.microsoft.com/Tasks.Read", "secret": "one"},
{"target": "Tasks.Read Group.Read.All GroupMember.Read.All", "secret": "many"},
]
assert _pick_graph_token(entries) == "many"
def test_returns_none_without_a_graph_entry(self):
entries = [{"target": "openid profile offline_access", "secret": "id-tok"}]
assert _pick_graph_token(entries) is None
def test_skips_entries_with_no_secret(self):
entries = [{"target": "Tasks.Read", "secret": ""}]
assert _pick_graph_token(entries) is None
def test_tolerates_junk_entries(self):
assert _pick_graph_token([None, "nonsense", {}, 42]) is None