Initial commit: MISF VOB download and upload automation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
commit
c2dec96b1f
43 files changed
+10944
No files matched your search
@@ -0,0 +1,23 @@
|
|||||||
|
# Copy to .env and fill in (do not commit .env)
|
||||||
|
ONEDRIVE_USERNAME=your.email@example.com
|
||||||
|
ONEDRIVE_PASSWORD=your_password
|
||||||
|
MICROSOFT_TOTP_SECRET=your_base32_authenticator_secret
|
||||||
|
|
||||||
|
# Browser: uses chrome-win\chrome-win\chrome.exe by default (bundled Chromium)
|
||||||
|
# MISF_USE_INSTALLED_CHROME=1 # skip chrome-win, use system Chromium
|
||||||
|
# CHROMIUM_PATH=C:\full\path\to\chrome.exe
|
||||||
|
# MISF_BROWSER_FOREGROUND=1 # show browser (default: minimized background)
|
||||||
|
# MISF_ALLOW_CHROME=1 # allow Google Chrome fallback
|
||||||
|
|
||||||
|
# Optional — Planner grid URL (tenant tid= in query string)
|
||||||
|
# PLANNER_URL=https://planner.cloud.microsoft/webui/mytasks/assignedtome/view/grid?tid=...
|
||||||
|
|
||||||
|
# Optional
|
||||||
|
# ONEDRIVE_URL=https://...
|
||||||
|
# PDF_FOLDER_PATH=C:\path\to\Input\PDFs
|
||||||
|
# MISF_DEBUG=1
|
||||||
|
# MISF_LOG_LEVEL=DEBUG
|
||||||
|
# PROXY_IP=
|
||||||
|
# PROXY_PORT=
|
||||||
|
# PROXY_USERNAME=
|
||||||
|
# PROXY_PASSWORD=
|
||||||
+41
@@ -0,0 +1,41 @@
|
|||||||
|
# Build artifacts
|
||||||
|
build/
|
||||||
|
dist/
|
||||||
|
_internal/
|
||||||
|
*.exe
|
||||||
|
*.spec
|
||||||
|
|
||||||
|
# Chrome binary
|
||||||
|
chrome-win/
|
||||||
|
|
||||||
|
# Input/Output data folders
|
||||||
|
Input/
|
||||||
|
Output/
|
||||||
|
|
||||||
|
# Python cache
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
*.pyo
|
||||||
|
*.pyd
|
||||||
|
.Python
|
||||||
|
*.egg-info/
|
||||||
|
.eggs/
|
||||||
|
|
||||||
|
# Virtual environments
|
||||||
|
venv/
|
||||||
|
env/
|
||||||
|
.env
|
||||||
|
.venv/
|
||||||
|
|
||||||
|
# IDE
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
|
||||||
|
# Logs
|
||||||
|
*.log
|
||||||
|
logs/
|
||||||
|
|
||||||
|
# OS
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
@@ -0,0 +1,228 @@
|
|||||||
|
# OneDrive / Planner download-only automation
|
||||||
|
|
||||||
|
Automates the MISF VOB **face-sheet download** workflow: sign in to OneDrive, open Planner, search each facility plan, download attachment PDFs, and mark tasks **In progress**.
|
||||||
|
|
||||||
|
Reference: [`docs/MISF-VOB-Planner-Workflow.docx`](docs/MISF-VOB-Planner-Workflow.docx)
|
||||||
|
|
||||||
|
This repository contains **only** OneDrive / Planner download automation. There is no Noridian Medicare, Medi-Cal, PDF extraction, or eligibility verification code.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Source requirements
|
||||||
|
|
||||||
|
| Doc step | Action |
|
||||||
|
|----------|--------|
|
||||||
|
| 1 | Open OneDrive URL, enter username / password, complete MFA (Google Authenticator 6-digit code) |
|
||||||
|
| 2–4 | Open **Planner** (direct URL in config; default: Assigned to me grid) |
|
||||||
|
| 5 | For each facility: search/select plan → open tasks → **download attachment PDFs** → set task status **In progress** (board card, or Task details → Status) |
|
||||||
|
|
||||||
|
**Clarifications:**
|
||||||
|
|
||||||
|
- Process **54 facilities** by typing each name into Planner’s **search box**, one at a time.
|
||||||
|
- MFA via **TOTP secret** in `Input/input.xlsx` (automated with `pyotp`).
|
||||||
|
|
||||||
|
**Not in this repo (upload path from the MISF guide):**
|
||||||
|
|
||||||
|
- Uploading completed work to OneDrive
|
||||||
|
- Marking tasks **Completed** after eligibility verification
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
start[run_download_workflow] --> config[Read input.xlsx]
|
||||||
|
config --> chrome[start_chrome_debug + attach_browser]
|
||||||
|
chrome --> login[Microsoft login + TOTP]
|
||||||
|
login --> planner[Open Planner / In my plans]
|
||||||
|
planner --> loop{For each facility}
|
||||||
|
loop --> search[Search facility in Planner]
|
||||||
|
search --> dl[Download task attachments]
|
||||||
|
dl --> status[Mark tasks In progress]
|
||||||
|
status --> loop
|
||||||
|
loop --> summary[Log summary + Excel report]
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Repository layout
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|------|------|
|
||||||
|
| [`run_download.py`](run_download.py) | Main entry point |
|
||||||
|
| [`src/planner_download.py`](src/planner_download.py) | Orchestrator: config → Chrome → login → facility loop → summary |
|
||||||
|
| [`src/onedrive_planner.py`](src/onedrive_planner.py) | Selenium: Microsoft login, Planner, downloads, In-Process |
|
||||||
|
| [`src/config.py`](src/config.py) | `read_app_config()`, `read_facility_list()` |
|
||||||
|
| [`src/totp_utils.py`](src/totp_utils.py) | TOTP secret sanitization / validation |
|
||||||
|
| [`scripts/validate_config.py`](scripts/validate_config.py) | Pre-flight config check (no browser) |
|
||||||
|
| [`src/chrome.py`](src/chrome.py) | Chrome debug startup, Selenium attach, proxy auth |
|
||||||
|
| [`scripts/bootstrap_input_xlsx.py`](scripts/bootstrap_input_xlsx.py) | Create config template |
|
||||||
|
| [`tests/test_download_config.py`](tests/test_download_config.py) | Config smoke tests |
|
||||||
|
|
||||||
|
```
|
||||||
|
.
|
||||||
|
├── README.md
|
||||||
|
├── requirements.txt
|
||||||
|
├── run_download.py
|
||||||
|
├── docs/
|
||||||
|
├── Input/
|
||||||
|
│ ├── input.xlsx.example
|
||||||
|
│ ├── input.xlsx # gitignored
|
||||||
|
│ └── PDFs/
|
||||||
|
├── Output/
|
||||||
|
├── scripts/
|
||||||
|
├── tests/
|
||||||
|
└── src/
|
||||||
|
├── planner_download.py
|
||||||
|
├── onedrive_planner.py
|
||||||
|
├── config.py
|
||||||
|
└── chrome.py
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Configuration (`Input/input.xlsx`)
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
copy Input\input.xlsx.example Input\input.xlsx
|
||||||
|
copy .env.example .env
|
||||||
|
```
|
||||||
|
|
||||||
|
Fill credentials in **either** place:
|
||||||
|
|
||||||
|
- **`Input/input.xlsx`** → Sheet1, column **B** (`OneDrive Username`, `OneDrive Password`, `Microsoft TOTP Secret`), **or**
|
||||||
|
- **`.env`** in the project root (`ONEDRIVE_USERNAME`, `ONEDRIVE_PASSWORD`, `MICROSOFT_TOTP_SECRET`)
|
||||||
|
|
||||||
|
Check config before running:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
python scripts\validate_config.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### Sheet `Sheet1`
|
||||||
|
|
||||||
|
| Key | Purpose |
|
||||||
|
|-----|---------|
|
||||||
|
| `OneDrive Username` | Microsoft login email |
|
||||||
|
| `OneDrive Password` | Password |
|
||||||
|
| `Microsoft TOTP Secret` | Base32 authenticator secret (`pyotp`) |
|
||||||
|
| `OneDrive URL` | OneDrive landing URL |
|
||||||
|
| `pdf_folder_path` | Download root (default: `Input/PDFs`) |
|
||||||
|
| `Proxy IP` / `Proxy Port` / `Proxy Username` / `Proxy Password` | Optional |
|
||||||
|
|
||||||
|
### Sheet `Facilities` — **put all 54 facility names here**
|
||||||
|
|
||||||
|
Open **`Input/input.xlsx`** (not the `.example` file) and edit the **`Facilities`** sheet:
|
||||||
|
|
||||||
|
| Column | Purpose |
|
||||||
|
|--------|---------|
|
||||||
|
| `Facility Name` | **One row per facility** (54 rows). Use the exact text you type in Planner search. |
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
| Facility Name |
|
||||||
|
|---------------|
|
||||||
|
| Amaya |
|
||||||
|
| Brighton Place San Diego |
|
||||||
|
| … (52 more rows) |
|
||||||
|
|
||||||
|
The bot reads this sheet via `read_facility_list()` in `src/config.py`. For a first test, keep only **2 rows** and run `python run_download.py --limit 2`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Production `.exe` (double-click)
|
||||||
|
|
||||||
|
Build a Windows folder you can copy to any PC:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
powershell -ExecutionPolicy Bypass -File scripts\build_production_exe.ps1
|
||||||
|
```
|
||||||
|
|
||||||
|
Output at **project root**: `MISF_VOB_Download.exe` and `_internal\` (same folder as `Input/`, `Output/`, `run_download.py`).
|
||||||
|
|
||||||
|
**Deploy:** zip the project folder (omit `.venv`, `build`, `src` if you only ship the exe build). On the target machine:
|
||||||
|
|
||||||
|
1. Unzip anywhere
|
||||||
|
2. Copy `Input\input.xlsx.example` → `Input\input.xlsx` and configure credentials + facilities
|
||||||
|
3. Double-click `MISF_VOB_Download.exe`
|
||||||
|
|
||||||
|
Config, `Input/`, and `Output/` are read from the **project root** (folder containing the `.exe`). See [`README_PRODUCTION.txt`](README_PRODUCTION.txt) after build.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
python -m venv .venv
|
||||||
|
.\.venv\Scripts\Activate.ps1
|
||||||
|
pip install -r requirements.txt
|
||||||
|
copy Input\input.xlsx.example Input\input.xlsx
|
||||||
|
# Edit Input\input.xlsx
|
||||||
|
|
||||||
|
python run_download.py --limit 2 # dry-run (first N facilities)
|
||||||
|
python run_download.py --facility "Amaya" # single facility
|
||||||
|
python run_download.py --resume # continue after last checkpoint
|
||||||
|
python run_download.py --keep-browser # leave browser open when done
|
||||||
|
python run_download.py # full run
|
||||||
|
```
|
||||||
|
|
||||||
|
**Outputs:**
|
||||||
|
|
||||||
|
- `Input/PDFs/<FacilityName>/*.pdf`
|
||||||
|
- `Output/<date>/Summary/Download_Summary_<date>.xlsx` (includes run timestamp and Planner URL)
|
||||||
|
- `Output/<date>/logs/download.log`
|
||||||
|
- `Output/<date>/checkpoint.json` (while a run is in progress; cleared on success)
|
||||||
|
|
||||||
|
Set `MISF_DEBUG=1` in `.env` to enable NDJSON debug traces (`debug-fc545e.log`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
python -m pytest
|
||||||
|
python scripts\validate_config.py
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
| Issue | What to check |
|
||||||
|
|-------|----------------|
|
||||||
|
| Login / MFA fails | TOTP secret; system clock (NTP) |
|
||||||
|
| Facility not found | `Facility Name` matches Planner search |
|
||||||
|
| No PDFs downloaded | Attachments exist; tune `src/onedrive_planner.py` selectors |
|
||||||
|
| Proxy errors | Proxy IP/Port in `input.xlsx`; profile `%USERPROFILE%\Automation\ChromiumProfile_MISF_Download` |
|
||||||
|
| Browser not found | Install [Chromium](https://www.chromium.org/getting-involved/download-chromium/) or set `CHROMIUM_PATH` in `.env` (Google Chrome only with `MISF_ALLOW_CHROME=1`) |
|
||||||
|
| `ChromeDriver only supports Chrome version 148` / attach fails | Install Chromium matching driver version; driver auto-downloads to `.drivers/` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Browser
|
||||||
|
|
||||||
|
The automation uses **Chromium only** (minimized in the background by default). It looks for, in order:
|
||||||
|
|
||||||
|
1. `CHROMIUM_PATH` or `BROWSER_PATH` environment variable
|
||||||
|
2. **`chrome-win/chrome-win/chrome.exe`** (portable bundle at project root — default for production)
|
||||||
|
3. Bundled **`chromedriver.exe`** in the same folder (no download needed)
|
||||||
|
4. System Chromium under `Program Files\Chromium`, `%LOCALAPPDATA%\Chromium`, Scoop (`MISF_USE_INSTALLED_CHROME=1` to prefer this over `chrome-win`)
|
||||||
|
5. Google Chrome only when `MISF_ALLOW_CHROME=1`
|
||||||
|
|
||||||
|
Set `MISF_BROWSER_FOREGROUND=1` to show/maximize the browser window. The production `.exe` runs without a console window; see `Output/<date>/logs/download.log`.
|
||||||
|
|
||||||
|
**Chromedriver:** `webdriver-manager` downloads a matching driver. If attach fails, install Chromium that matches your driver or set `CHROMIUM_PATH` explicitly.
|
||||||
|
|
||||||
|
User profile directory: `%USERPROFILE%\Automation\ChromiumProfile_MISF_Download`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## UiPath migration
|
||||||
|
|
||||||
|
An equivalent **UiPath Studio (Modern)** project lives in [`uipath/MISF_Planner_Download/`](uipath/MISF_Planner_Download/). Open `project.json` in UiPath Studio. Pre-flight check:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
powershell -File scripts\validate_uipath_pilot.ps1
|
||||||
|
python -m pytest tests\test_uipath_pilot.py -v
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Internal MISF / Access Psychology automation.
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
MISF VOB Download — Production (Windows)
|
||||||
|
=========================================
|
||||||
|
|
||||||
|
Double-click MISF_VOB_Download.exe to run the full facility download workflow.
|
||||||
|
|
||||||
|
FIRST-TIME SETUP
|
||||||
|
----------------
|
||||||
|
1. Copy Input\input.xlsx.example to Input\input.xlsx
|
||||||
|
2. Open Input\input.xlsx and fill:
|
||||||
|
- Sheet1: OneDrive Username, Password, Microsoft TOTP Secret
|
||||||
|
- Facilities: one facility name per row (Planner search text)
|
||||||
|
3. Keep chrome-win\chrome-win\ (chrome.exe + chromedriver.exe) in this folder.
|
||||||
|
Browser starts minimized in background. No separate Chromium install required.
|
||||||
|
4. Optional: copy .env.example to .env for credentials instead of Excel
|
||||||
|
|
||||||
|
RUN
|
||||||
|
---
|
||||||
|
- Double-click MISF_VOB_Download.exe
|
||||||
|
- A console window shows progress; press Enter when finished to close
|
||||||
|
- For scheduled tasks (no pause): set environment variable MISF_NO_PAUSE=1
|
||||||
|
|
||||||
|
OUTPUT
|
||||||
|
------
|
||||||
|
- Output\<date>\Original\<facility>\ — downloaded PDFs
|
||||||
|
- Output\<date>\Summary\Download_Summary_<date>.xlsx
|
||||||
|
- Output\<date>\logs\download.log
|
||||||
|
|
||||||
|
OPTIONAL COMMAND-LINE (run from cmd in this folder)
|
||||||
|
---------------------------------------------------
|
||||||
|
MISF_VOB_Download.exe --limit 2
|
||||||
|
MISF_VOB_Download.exe --facility "Amaya"
|
||||||
|
MISF_VOB_Download.exe --resume
|
||||||
|
|
||||||
|
REQUIREMENTS
|
||||||
|
------------
|
||||||
|
- Windows 10/11
|
||||||
|
- Network access to Microsoft 365 / Planner
|
||||||
|
- chrome-win\ folder with portable Chromium (or system Chromium / CHROMIUM_PATH)
|
||||||
|
- Progress log: Output\<date>\logs\download.log (no console window when using .exe)
|
||||||
|
|
||||||
|
Keep MISF_VOB_Download.exe, _internal, and chrome-win together in the project root
|
||||||
|
(same directory as Input and Output).
|
||||||
Binary file not shown.
@@ -0,0 +1,42 @@
|
|||||||
|
MISF VOB Download — Production (Windows)
|
||||||
|
=========================================
|
||||||
|
|
||||||
|
Double-click MISF_VOB_Download.exe to run the full facility download workflow.
|
||||||
|
|
||||||
|
FIRST-TIME SETUP
|
||||||
|
----------------
|
||||||
|
1. Copy Input\input.xlsx.example to Input\input.xlsx
|
||||||
|
2. Open Input\input.xlsx and fill:
|
||||||
|
- Sheet1: OneDrive Username, Password, Microsoft TOTP Secret
|
||||||
|
- Facilities: one facility name per row (Planner search text)
|
||||||
|
3. Keep chrome-win\chrome-win\ (chrome.exe + chromedriver.exe) in this folder.
|
||||||
|
Browser starts minimized in background. No separate Chromium install required.
|
||||||
|
4. Optional: copy .env.example to .env for credentials instead of Excel
|
||||||
|
|
||||||
|
RUN
|
||||||
|
---
|
||||||
|
- Double-click MISF_VOB_Download.exe
|
||||||
|
- A console window shows progress; press Enter when finished to close
|
||||||
|
- For scheduled tasks (no pause): set environment variable MISF_NO_PAUSE=1
|
||||||
|
|
||||||
|
OUTPUT
|
||||||
|
------
|
||||||
|
- Output\<date>\Original\<facility>\ — downloaded PDFs
|
||||||
|
- Output\<date>\Summary\Download_Summary_<date>.xlsx
|
||||||
|
- Output\<date>\logs\download.log
|
||||||
|
|
||||||
|
OPTIONAL COMMAND-LINE (run from cmd in this folder)
|
||||||
|
---------------------------------------------------
|
||||||
|
MISF_VOB_Download.exe --limit 2
|
||||||
|
MISF_VOB_Download.exe --facility "Amaya"
|
||||||
|
MISF_VOB_Download.exe --resume
|
||||||
|
|
||||||
|
REQUIREMENTS
|
||||||
|
------------
|
||||||
|
- Windows 10/11
|
||||||
|
- Network access to Microsoft 365 / Planner
|
||||||
|
- chrome-win\ folder with portable Chromium (or system Chromium / CHROMIUM_PATH)
|
||||||
|
- Progress log: Output\<date>\logs\download.log (no console window when using .exe)
|
||||||
|
|
||||||
|
Keep MISF_VOB_Download.exe, _internal, and chrome-win together in the project root
|
||||||
|
(same directory as Input and Output).
|
||||||
+53
@@ -0,0 +1,53 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
MISF VOB Download — production entry (PyInstaller / double-click).
|
||||||
|
|
||||||
|
Config, Input/, and Output/ live in the same folder as the .exe.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import traceback
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent
|
||||||
|
SRC = ROOT / "src"
|
||||||
|
if not getattr(sys, "frozen", False) and str(SRC) not in sys.path:
|
||||||
|
sys.path.insert(0, str(SRC))
|
||||||
|
|
||||||
|
from planner_download import build_arg_parser, run_download_workflow # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def _run_workflow() -> int:
|
||||||
|
exit_code = 0
|
||||||
|
try:
|
||||||
|
args = build_arg_parser().parse_args()
|
||||||
|
run_download_workflow(
|
||||||
|
facility_limit=max(0, args.limit),
|
||||||
|
facility_name=args.facility.strip() or None,
|
||||||
|
resume=args.resume,
|
||||||
|
keep_browser=args.keep_browser,
|
||||||
|
screenshot_on_error=args.screenshot and not args.no_screenshot,
|
||||||
|
)
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
print("\nCancelled by user.")
|
||||||
|
exit_code = 130
|
||||||
|
except SystemExit as exc:
|
||||||
|
exit_code = int(exc.code) if exc.code is not None else 0
|
||||||
|
except Exception:
|
||||||
|
traceback.print_exc()
|
||||||
|
exit_code = 1
|
||||||
|
return exit_code
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
if getattr(sys, "frozen", False):
|
||||||
|
from log_window import run_with_live_logs
|
||||||
|
|
||||||
|
return run_with_live_logs(_run_workflow)
|
||||||
|
return _run_workflow()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
[pytest]
|
||||||
|
testpaths = tests
|
||||||
|
pythonpath = src
|
||||||
|
addopts = -q
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
# Build-time only (production .exe)
|
||||||
|
-r requirements.txt
|
||||||
|
pyinstaller>=6.10
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
pandas==2.2.3
|
||||||
|
openpyxl==3.1.5
|
||||||
|
selenium==4.27.1
|
||||||
|
webdriver-manager==4.0.2
|
||||||
|
pyotp==2.9.0
|
||||||
|
pytest==8.3.4
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
MISF VOB — OneDrive / Planner face-sheet download.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python run_download.py
|
||||||
|
python run_download.py --limit 2
|
||||||
|
python run_download.py --facility "Amaya Healthcare"
|
||||||
|
python run_download.py --resume
|
||||||
|
python run_download.py --keep-browser
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import traceback
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent
|
||||||
|
SRC = ROOT / "src"
|
||||||
|
if str(SRC) not in sys.path:
|
||||||
|
sys.path.insert(0, str(SRC))
|
||||||
|
|
||||||
|
from planner_download import build_arg_parser, run_download_workflow # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
try:
|
||||||
|
args = build_arg_parser().parse_args()
|
||||||
|
run_download_workflow(
|
||||||
|
facility_limit=max(0, args.limit),
|
||||||
|
facility_name=args.facility.strip() or None,
|
||||||
|
resume=args.resume,
|
||||||
|
keep_browser=args.keep_browser,
|
||||||
|
screenshot_on_error=args.screenshot and not args.no_screenshot,
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
print("\nCancelled by user.")
|
||||||
|
return 130
|
||||||
|
except SystemExit as exc:
|
||||||
|
return int(exc.code) if exc.code is not None else 0
|
||||||
|
except Exception:
|
||||||
|
traceback.print_exc()
|
||||||
|
print("\nRun failed. See traceback above.", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
"""
|
||||||
|
Create Input/input.xlsx (if missing) and refresh Input/input.xlsx.example.
|
||||||
|
|
||||||
|
Run from repository root:
|
||||||
|
python scripts/bootstrap_input_xlsx.py
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
INPUT_DIR = PROJECT_ROOT / "Input"
|
||||||
|
CONFIG_PATH = INPUT_DIR / "input.xlsx"
|
||||||
|
EXAMPLE_PATH = INPUT_DIR / "input.xlsx.example"
|
||||||
|
|
||||||
|
FACILITY_KEYWORDS = [
|
||||||
|
"amaya", "brighton place san diego", "brighton place sd", "brighton place spring valley",
|
||||||
|
"brighton place-spring valley", "granite hill", "anberry", "arbor hills", "bayberry",
|
||||||
|
"bradley court", "castle manor", "friendship", "gramercy", "kearny", "lakeside",
|
||||||
|
"newport", "siena", "stanford court", "temecula", "walnut", "balboa", "cottonwood",
|
||||||
|
"country hills", "la mesa", "university care", "blue oak", "bridgewood", "capistrano",
|
||||||
|
"eden", "hillcrest heights", "la casa via", "torrey pines", "bradley gardens", "horizon",
|
||||||
|
"community care", "jacob", "the shores", "valley vista", "valle vista", "golden hill",
|
||||||
|
"grossmont post acute", "lemon grove", "mission hills", "parkside", "somerset",
|
||||||
|
"the springs", "victoria", "reo vista", "san diego post acute", "villa las palmas",
|
||||||
|
"ridgeview", "windsor gardens", "windsor rosewood",
|
||||||
|
]
|
||||||
|
FACILITIES = [k.title() for k in FACILITY_KEYWORDS]
|
||||||
|
FACILITIES.append("REPLACE_WITH_54TH_FACILITY_NAME")
|
||||||
|
|
||||||
|
DEFAULT_PLANNER_URL = (
|
||||||
|
"https://planner.cloud.microsoft/webui/mytasks/assignedtome/view/grid"
|
||||||
|
"?tid=0c15ad2c-19dd-4ded-9cd1-20d344386414"
|
||||||
|
)
|
||||||
|
|
||||||
|
SHEET1_ROWS = [
|
||||||
|
("Proxy IP", ""),
|
||||||
|
("Proxy Port", ""),
|
||||||
|
("Proxy Username", ""),
|
||||||
|
("Proxy Password", ""),
|
||||||
|
("OneDrive Username", ""),
|
||||||
|
("OneDrive Password", ""),
|
||||||
|
("Microsoft TOTP Secret", ""),
|
||||||
|
("Planner URL", DEFAULT_PLANNER_URL),
|
||||||
|
(
|
||||||
|
"OneDrive URL",
|
||||||
|
"https://netorg393890-my.sharepoint.com/personal/joy_accesspsychology_org"
|
||||||
|
"/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fjoy%5Faccesspsychology%5Forg%2FDocuments"
|
||||||
|
"%2FAccess%20Facility%20Spreadsheets3",
|
||||||
|
),
|
||||||
|
("pdf_folder_path", str(PROJECT_ROOT / "Input" / "PDFs")),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _build_workbook() -> tuple[pd.DataFrame, pd.DataFrame]:
|
||||||
|
sheet1 = pd.DataFrame(SHEET1_ROWS, columns=["Key", "Value"])
|
||||||
|
facilities = pd.DataFrame({"Facility Name": FACILITIES})
|
||||||
|
return sheet1, facilities
|
||||||
|
|
||||||
|
|
||||||
|
def _write(path: Path, sheet1: pd.DataFrame, facilities: pd.DataFrame) -> None:
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with pd.ExcelWriter(path, engine="openpyxl") as writer:
|
||||||
|
sheet1.to_excel(writer, sheet_name="Sheet1", index=False, header=False)
|
||||||
|
facilities.to_excel(writer, sheet_name="Facilities", index=False)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
sheet1, facilities = _build_workbook()
|
||||||
|
|
||||||
|
_write(EXAMPLE_PATH, sheet1, facilities)
|
||||||
|
print(f"Wrote template -> {EXAMPLE_PATH} ({len(FACILITIES)} facilities)")
|
||||||
|
|
||||||
|
if CONFIG_PATH.exists():
|
||||||
|
print(f"Skipped (already exists): {CONFIG_PATH}")
|
||||||
|
return
|
||||||
|
|
||||||
|
_write(CONFIG_PATH, sheet1, facilities)
|
||||||
|
print(f"Created local config -> {CONFIG_PATH}")
|
||||||
|
print("Edit input.xlsx with your credentials before running run_download.py")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# Build MISF_VOB_Download.exe and install at project root (double-click to run).
|
||||||
|
# Output: <project root>\MISF_VOB_Download.exe + _internal\
|
||||||
|
|
||||||
|
$ErrorActionPreference = "Stop"
|
||||||
|
$Root = Split-Path -Parent $PSScriptRoot
|
||||||
|
Set-Location $Root
|
||||||
|
|
||||||
|
Write-Host "=== MISF VOB Download - production build (project root) ===" -ForegroundColor Cyan
|
||||||
|
|
||||||
|
$PortableChrome = Join-Path $Root "chrome-win\chrome-win\chrome.exe"
|
||||||
|
$PortableDriver = Join-Path $Root "chrome-win\chrome-win\chromedriver.exe"
|
||||||
|
if (-not (Test-Path $PortableChrome)) {
|
||||||
|
throw "Portable Chromium not found: $PortableChrome (required for production build)"
|
||||||
|
}
|
||||||
|
if (-not (Test-Path $PortableDriver)) {
|
||||||
|
Write-Warning "Bundled chromedriver missing: $PortableDriver (will download at runtime)"
|
||||||
|
} else {
|
||||||
|
Write-Host "Using portable Chromium: $PortableChrome"
|
||||||
|
}
|
||||||
|
|
||||||
|
if (-not (Test-Path ".venv\Scripts\python.exe")) {
|
||||||
|
Write-Host "Creating virtual environment..."
|
||||||
|
python -m venv .venv
|
||||||
|
}
|
||||||
|
|
||||||
|
$Python = Join-Path $Root ".venv\Scripts\python.exe"
|
||||||
|
& $Python -m pip install --upgrade pip -q
|
||||||
|
& $Python -m pip install -r requirements.txt -q
|
||||||
|
& $Python -m pip install "pyinstaller>=6.10" -q
|
||||||
|
|
||||||
|
Write-Host "Running PyInstaller..."
|
||||||
|
& $Python -m PyInstaller --noconfirm MISF_VOB_Download.spec
|
||||||
|
if ($LASTEXITCODE -ne 0) {
|
||||||
|
throw "PyInstaller failed with exit code $LASTEXITCODE"
|
||||||
|
}
|
||||||
|
|
||||||
|
$BuiltExe = Join-Path $Root "dist\MISF_VOB_Download.exe"
|
||||||
|
$BuiltDir = Join-Path $Root "dist\MISF_VOB_Download"
|
||||||
|
$BuiltDirExe = Join-Path $BuiltDir "MISF_VOB_Download.exe"
|
||||||
|
|
||||||
|
if (Test-Path $BuiltExe) {
|
||||||
|
Write-Host "Installing one-file build to project root..."
|
||||||
|
$ExeDest = Join-Path $Root "MISF_VOB_Download.exe"
|
||||||
|
Get-Process -Name "MISF_VOB_Download" -ErrorAction SilentlyContinue | Stop-Process -Force
|
||||||
|
Start-Sleep -Seconds 1
|
||||||
|
Copy-Item $BuiltExe $ExeDest -Force
|
||||||
|
Remove-Item $BuiltExe -Force
|
||||||
|
} elseif (Test-Path $BuiltDirExe) {
|
||||||
|
Write-Host "Installing folder build to project root..."
|
||||||
|
$ExeDest = Join-Path $Root "MISF_VOB_Download.exe"
|
||||||
|
Get-Process -Name "MISF_VOB_Download" -ErrorAction SilentlyContinue | Stop-Process -Force
|
||||||
|
Start-Sleep -Seconds 1
|
||||||
|
$InternalDest = Join-Path $Root "_internal"
|
||||||
|
|
||||||
|
Copy-Item $BuiltDirExe $ExeDest -Force
|
||||||
|
if (Test-Path $InternalDest) {
|
||||||
|
Remove-Item $InternalDest -Recurse -Force
|
||||||
|
}
|
||||||
|
Move-Item (Join-Path $BuiltDir "_internal") $InternalDest -Force
|
||||||
|
Remove-Item $BuiltDir -Recurse -Force
|
||||||
|
} else {
|
||||||
|
throw "Expected build output not found: $BuiltExe or $BuiltDirExe"
|
||||||
|
}
|
||||||
|
$DistFolder = Join-Path $Root "dist"
|
||||||
|
if (Test-Path $DistFolder) {
|
||||||
|
$remaining = @(Get-ChildItem $DistFolder -Force -ErrorAction SilentlyContinue)
|
||||||
|
if ($remaining.Count -eq 0) {
|
||||||
|
Remove-Item $DistFolder -Force
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
$InputDir = Join-Path $Root "Input"
|
||||||
|
$OutputDir = Join-Path $Root "Output"
|
||||||
|
New-Item -ItemType Directory -Force -Path (Join-Path $InputDir "PDFs") | Out-Null
|
||||||
|
New-Item -ItemType Directory -Force -Path $OutputDir | Out-Null
|
||||||
|
Copy-Item (Join-Path $Root "docs\PRODUCTION_README.txt") (Join-Path $Root "README_PRODUCTION.txt") -Force
|
||||||
|
|
||||||
|
Write-Host ""
|
||||||
|
Write-Host "Build complete." -ForegroundColor Green
|
||||||
|
Write-Host " Run: $ExeDest"
|
||||||
|
Write-Host ""
|
||||||
|
Write-Host "First-time setup:" -ForegroundColor Yellow
|
||||||
|
Write-Host " 1. Copy Input\input.xlsx.example to Input\input.xlsx and fill credentials"
|
||||||
|
Write-Host " 2. Add facility names on the Facilities sheet"
|
||||||
|
Write-Host " 3. Keep chrome-win\ next to the .exe"
|
||||||
|
Write-Host " 4. Double-click MISF_VOB_Download.exe in this folder"
|
||||||
|
Write-Host ""
|
||||||
|
Write-Host 'See README_PRODUCTION.txt. Exe is at project root (no dist folder).'
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
"""Print config status (no browser). Run: python scripts/validate_config.py"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
sys.path.insert(0, str(ROOT / "src"))
|
||||||
|
|
||||||
|
from config import ( # noqa: E402
|
||||||
|
CONFIG_FILE,
|
||||||
|
DEFAULT_PLANNER_URL,
|
||||||
|
EXPECTED_FACILITY_COUNT,
|
||||||
|
credentials_setup_message,
|
||||||
|
missing_onedrive_credentials,
|
||||||
|
read_app_config,
|
||||||
|
read_facility_list,
|
||||||
|
)
|
||||||
|
from totp_utils import validate_totp_secret # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
path = CONFIG_FILE if CONFIG_FILE.exists() else ROOT / "Input" / "input.xlsx.example"
|
||||||
|
if not path.exists():
|
||||||
|
print(f"[FAIL] Missing {path}")
|
||||||
|
print(" copy Input\\input.xlsx.example Input\\input.xlsx")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
cfg = read_app_config(path)
|
||||||
|
missing = missing_onedrive_credentials(
|
||||||
|
cfg.onedrive_username,
|
||||||
|
cfg.onedrive_password,
|
||||||
|
cfg.microsoft_totp_secret,
|
||||||
|
)
|
||||||
|
if missing and path == CONFIG_FILE:
|
||||||
|
print("[FAIL] Credentials not configured:")
|
||||||
|
print(credentials_setup_message(missing))
|
||||||
|
return 1
|
||||||
|
|
||||||
|
try:
|
||||||
|
validate_totp_secret(cfg.microsoft_totp_secret or "JBSWY3DPEHPK3PXP")
|
||||||
|
except ValueError as exc:
|
||||||
|
if cfg.microsoft_totp_secret:
|
||||||
|
print(f"[FAIL] TOTP secret: {exc}")
|
||||||
|
return 1
|
||||||
|
print(f"[WARN] TOTP not set — format check skipped ({exc})")
|
||||||
|
|
||||||
|
facilities = read_facility_list(path, warn_if_not_expected_count=True)
|
||||||
|
planner = cfg.planner_url or DEFAULT_PLANNER_URL
|
||||||
|
if not planner.startswith("https://"):
|
||||||
|
print(f"[FAIL] Planner URL must be https — got: {planner[:80]}")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
print(f"[OK] Config file: {path.name}")
|
||||||
|
if cfg.onedrive_username:
|
||||||
|
print(f"[OK] Credentials set for {cfg.onedrive_username}")
|
||||||
|
else:
|
||||||
|
print("[OK] Template mode (no credentials in example file)")
|
||||||
|
print(f"[OK] Planner URL: {planner[:70]}...")
|
||||||
|
print(f"[OK] PDF folder: {cfg.pdf_folder_path or '(default Input/PDFs)'}")
|
||||||
|
print(f"[OK] {len(facilities)} facilities (expected {EXPECTED_FACILITY_COUNT})")
|
||||||
|
if len(facilities) != EXPECTED_FACILITY_COUNT:
|
||||||
|
print("[WARN] Facility count differs from expected 54 — verify Facilities sheet")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
#Requires -Version 5.1
|
||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Pre-flight validation for UiPath MISF Planner Download pilot (mirrors Python validate_config + output checks).
|
||||||
|
#>
|
||||||
|
param(
|
||||||
|
[string]$ProjectRoot = (Resolve-Path (Join-Path $PSScriptRoot "..")).Path,
|
||||||
|
[int]$ExpectedFacilities = 54,
|
||||||
|
[switch]$CheckOutput
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = "Stop"
|
||||||
|
$configPath = Join-Path $ProjectRoot "Input\input.xlsx"
|
||||||
|
$uipathProject = Join-Path $ProjectRoot "uipath\MISF_Planner_Download\project.json"
|
||||||
|
$today = Get-Date -Format "yyyy-MM-dd"
|
||||||
|
|
||||||
|
Write-Host "MISF UiPath pilot validation"
|
||||||
|
Write-Host "Project root: $ProjectRoot"
|
||||||
|
|
||||||
|
if (-not (Test-Path $configPath)) {
|
||||||
|
Write-Error "Missing config: $configPath (copy Input\input.xlsx.example)"
|
||||||
|
}
|
||||||
|
if (-not (Test-Path $uipathProject)) {
|
||||||
|
Write-Error "Missing UiPath project: $uipathProject"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Run Python config validator if available
|
||||||
|
$pyValidate = Join-Path $ProjectRoot "scripts\validate_config.py"
|
||||||
|
if (Test-Path $pyValidate) {
|
||||||
|
Write-Host "Running Python validate_config.py..."
|
||||||
|
python $pyValidate
|
||||||
|
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
|
||||||
|
}
|
||||||
|
|
||||||
|
# Verify UiPath workflow files exist
|
||||||
|
$requiredWorkflows = @(
|
||||||
|
"Main.xaml",
|
||||||
|
"01_ReadConfig.xaml",
|
||||||
|
"02_LoginPlanner.xaml",
|
||||||
|
"03_ProcessFacility.xaml",
|
||||||
|
"04_DownloadTaskAttachments.xaml",
|
||||||
|
"05_SetTaskInProgress.xaml",
|
||||||
|
"06_WriteSummary.xaml",
|
||||||
|
"Helpers\GenerateTotp.xaml",
|
||||||
|
"Helpers\SanitizeFolderName.xaml",
|
||||||
|
"Selectors\PlannerSelectors.json"
|
||||||
|
)
|
||||||
|
$uipathDir = Join-Path $ProjectRoot "uipath\MISF_Planner_Download"
|
||||||
|
foreach ($wf in $requiredWorkflows) {
|
||||||
|
$p = Join-Path $uipathDir $wf
|
||||||
|
if (-not (Test-Path $p)) {
|
||||||
|
Write-Error "Missing workflow file: $p"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Write-Host "All UiPath workflow files present."
|
||||||
|
|
||||||
|
if ($CheckOutput) {
|
||||||
|
$summary = Join-Path $ProjectRoot "Output\$today\Summary\Download_Summary_$today.xlsx"
|
||||||
|
$original = Join-Path $ProjectRoot "Output\$today\Original"
|
||||||
|
if (-not (Test-Path $summary)) {
|
||||||
|
Write-Warning "No summary yet: $summary (run UiPath Main with in_FacilityLimit=2)"
|
||||||
|
} else {
|
||||||
|
Write-Host "Summary found: $summary"
|
||||||
|
}
|
||||||
|
if (Test-Path $original) {
|
||||||
|
$pdfCount = (Get-ChildItem -Path $original -Recurse -Filter "*.pdf" -ErrorAction SilentlyContinue).Count
|
||||||
|
Write-Host "PDFs under Original: $pdfCount"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Write-Host "Pilot pre-flight OK."
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""MISF OneDrive / Planner download automation."""
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
"""Resume checkpoint for multi-facility download runs."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def load_checkpoint(path: Path) -> dict:
|
||||||
|
"""
|
||||||
|
Return checkpoint dict:
|
||||||
|
- last_completed_index: int
|
||||||
|
- completed_facilities: list[str]
|
||||||
|
|
||||||
|
Backward compatible with older files.
|
||||||
|
"""
|
||||||
|
if not path.is_file():
|
||||||
|
return {"last_completed_index": 0, "completed_facilities": []}
|
||||||
|
try:
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
return {
|
||||||
|
"last_completed_index": int(data.get("last_completed_index", 0) or 0),
|
||||||
|
"completed_facilities": list(data.get("completed_facilities", []) or []),
|
||||||
|
}
|
||||||
|
except (json.JSONDecodeError, TypeError, ValueError):
|
||||||
|
return {"last_completed_index": 0, "completed_facilities": []}
|
||||||
|
|
||||||
|
|
||||||
|
def save_checkpoint(
|
||||||
|
path: Path,
|
||||||
|
*,
|
||||||
|
last_completed_index: int,
|
||||||
|
facility_name: str,
|
||||||
|
completed_facilities: list[str] | None = None,
|
||||||
|
) -> None:
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"last_completed_index": last_completed_index,
|
||||||
|
"last_facility": facility_name,
|
||||||
|
"completed_facilities": completed_facilities or [],
|
||||||
|
},
|
||||||
|
indent=2,
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def clear_checkpoint(path: Path) -> None:
|
||||||
|
if path.is_file():
|
||||||
|
path.unlink()
|
||||||
+891
@@ -0,0 +1,891 @@
|
|||||||
|
"""Chromium (Chrome-compatible) remote-debug startup and Selenium attach."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import atexit
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import socket
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
import zipfile
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from urllib.error import URLError
|
||||||
|
from urllib.request import urlopen
|
||||||
|
|
||||||
|
from selenium import webdriver
|
||||||
|
from selenium.common.exceptions import WebDriverException
|
||||||
|
from selenium.webdriver.chrome.options import Options
|
||||||
|
from selenium.webdriver.chrome.service import Service
|
||||||
|
from selenium.webdriver.common.by import By
|
||||||
|
from selenium.webdriver.common.keys import Keys
|
||||||
|
|
||||||
|
from config import PROJECT_ROOT
|
||||||
|
from debug_log import write_debug as _agent_log
|
||||||
|
from run_log import log_step
|
||||||
|
|
||||||
|
_DEBUG_PORT = 9222
|
||||||
|
DEBUG_PORT = _DEBUG_PORT
|
||||||
|
_PROXY_CREDENTIALS: dict[str, str] = {}
|
||||||
|
_LAST_BROWSER_PATH: str | None = None
|
||||||
|
_BROWSER_PROCESS: subprocess.Popen | None = None
|
||||||
|
|
||||||
|
USER_DATA_DIR = os.path.join(
|
||||||
|
os.environ.get("USERPROFILE", str(Path.home())),
|
||||||
|
"Automation",
|
||||||
|
"ChromiumProfile_MISF_Download",
|
||||||
|
)
|
||||||
|
|
||||||
|
_DRIVERS_DIR = PROJECT_ROOT / ".drivers"
|
||||||
|
_CFT_DOWNLOAD_BASE = "https://storage.googleapis.com/chrome-for-testing-public"
|
||||||
|
DOWNLOADS_DIR = PROJECT_ROOT / "Input" / "PDFs"
|
||||||
|
_VERSION_RE = re.compile(r"(\d+\.\d+\.\d+\.\d+)")
|
||||||
|
|
||||||
|
# Microsoft 365 hosts bypass HTTP proxy when split-tunnel is configured.
|
||||||
|
CHROME_PROXY_BYPASS_OFFICE = (
|
||||||
|
"*.office.com;*.office365.com;*.microsoft.com;*.live.com;*.outlook.com;"
|
||||||
|
"outlook.office.com;outlook.office365.com;login.microsoftonline.com;login.live.com;"
|
||||||
|
"aadcdn.msauth.net;aadcdn.msftauth.net"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def set_default_download_directory(path: Path | str, *, create: bool = False) -> None:
|
||||||
|
"""Set browser download prefs root (used at attach time). Per-facility CDP still applies."""
|
||||||
|
global DOWNLOADS_DIR
|
||||||
|
DOWNLOADS_DIR = Path(path).resolve()
|
||||||
|
if create:
|
||||||
|
DOWNLOADS_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _allow_google_chrome_fallback() -> bool:
|
||||||
|
return os.environ.get("MISF_ALLOW_CHROME", "").strip().lower() in ("1", "true", "yes")
|
||||||
|
|
||||||
|
|
||||||
|
def _portable_chrome_win_roots() -> list[Path]:
|
||||||
|
return [
|
||||||
|
PROJECT_ROOT / "chrome-win" / "chrome-win",
|
||||||
|
PROJECT_ROOT / "chrome-win",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _portable_chromium_executable() -> str | None:
|
||||||
|
"""Bundled Chromium under project ``chrome-win/`` (preferred for production)."""
|
||||||
|
for root in _portable_chrome_win_roots():
|
||||||
|
for name in ("chrome.exe", "chromium.exe"):
|
||||||
|
path = root / name
|
||||||
|
if path.is_file():
|
||||||
|
return str(path)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _bundled_chromedriver_for(browser_path: str) -> str | None:
|
||||||
|
"""``chromedriver.exe`` shipped next to portable ``chrome.exe``."""
|
||||||
|
if not browser_path:
|
||||||
|
return None
|
||||||
|
driver = Path(browser_path).parent / "chromedriver.exe"
|
||||||
|
if driver.is_file():
|
||||||
|
return str(driver)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _installed_chromium_paths() -> list[str]:
|
||||||
|
"""System-installed Chromium (not Google Chrome)."""
|
||||||
|
local = os.environ.get("LOCALAPPDATA", "")
|
||||||
|
profile = os.environ.get("USERPROFILE", str(Path.home()))
|
||||||
|
return [
|
||||||
|
r"C:\Program Files\Chromium\Application\chrome.exe",
|
||||||
|
r"C:\Program Files\Chromium\Application\chromium.exe",
|
||||||
|
r"C:\Program Files (x86)\Chromium\Application\chrome.exe",
|
||||||
|
os.path.join(local, r"Chromium\Application\chrome.exe"),
|
||||||
|
os.path.join(profile, r"scoop\apps\chromium\current\chrome.exe"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _prefer_installed_chromium() -> bool:
|
||||||
|
"""Skip ``chrome-win/`` and use a system Chromium install instead."""
|
||||||
|
return os.environ.get("MISF_USE_INSTALLED_CHROME", "").strip().lower() in (
|
||||||
|
"1",
|
||||||
|
"true",
|
||||||
|
"yes",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _google_chrome_fallback_paths() -> list[str]:
|
||||||
|
local = os.environ.get("LOCALAPPDATA", "")
|
||||||
|
return [
|
||||||
|
r"C:\Program Files\Google\Chrome\Application\chrome.exe",
|
||||||
|
r"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe",
|
||||||
|
os.path.join(local, r"Google\Chrome\Application\chrome.exe"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def find_chromium_executable() -> str | None:
|
||||||
|
"""
|
||||||
|
Resolve Chromium only (Google Chrome is opt-in via ``MISF_ALLOW_CHROME=1``).
|
||||||
|
|
||||||
|
Default order: ``chrome-win/`` portable bundle, then system Chromium.
|
||||||
|
Override: ``CHROMIUM_PATH`` / ``BROWSER_PATH``, or ``MISF_USE_INSTALLED_CHROME=1``.
|
||||||
|
"""
|
||||||
|
for env_key in ("CHROMIUM_PATH", "BROWSER_PATH"):
|
||||||
|
raw = os.environ.get(env_key, "").strip().strip('"')
|
||||||
|
if raw and os.path.isfile(raw):
|
||||||
|
return raw
|
||||||
|
|
||||||
|
portable = _portable_chromium_executable()
|
||||||
|
installed = [p for p in _installed_chromium_paths() if os.path.isfile(p)]
|
||||||
|
|
||||||
|
if _prefer_installed_chromium():
|
||||||
|
candidates = installed + ([portable] if portable else [])
|
||||||
|
else:
|
||||||
|
candidates = ([portable] if portable else []) + installed
|
||||||
|
|
||||||
|
if _allow_google_chrome_fallback():
|
||||||
|
candidates = candidates + [
|
||||||
|
p for p in _google_chrome_fallback_paths() if os.path.isfile(p)
|
||||||
|
]
|
||||||
|
|
||||||
|
for path in candidates:
|
||||||
|
if path:
|
||||||
|
_agent_log(
|
||||||
|
"A",
|
||||||
|
"src/chrome.py:find_chromium_executable",
|
||||||
|
"Selected browser executable",
|
||||||
|
{"basename": os.path.basename(path), "from_portable": "chrome-win" in path.lower()},
|
||||||
|
)
|
||||||
|
return path
|
||||||
|
_agent_log(
|
||||||
|
"A",
|
||||||
|
"src/chrome.py:find_chromium_executable",
|
||||||
|
"No Chromium executable found",
|
||||||
|
{"chrome_win_checked": [str(r) for r in _portable_chrome_win_roots()]},
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _browser_runs_in_background() -> bool:
|
||||||
|
"""Default: foreground (visible). Set MISF_BROWSER_BACKGROUND=1 to start minimized."""
|
||||||
|
return os.environ.get("MISF_BROWSER_BACKGROUND", "").strip().lower() in (
|
||||||
|
"1",
|
||||||
|
"true",
|
||||||
|
"yes",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _launch_browser_process(cmd: list[str]) -> subprocess.Popen:
|
||||||
|
"""Start Chromium detached; minimized on Windows when running in background."""
|
||||||
|
global _BROWSER_PROCESS
|
||||||
|
kwargs: dict = {
|
||||||
|
"stdout": subprocess.DEVNULL,
|
||||||
|
"stderr": subprocess.DEVNULL,
|
||||||
|
}
|
||||||
|
if sys.platform == "win32" and _browser_runs_in_background():
|
||||||
|
startupinfo = subprocess.STARTUPINFO()
|
||||||
|
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
|
||||||
|
startupinfo.wShowWindow = 7 # SW_SHOWMINNOACTIVE — minimized, no focus steal
|
||||||
|
kwargs["startupinfo"] = startupinfo
|
||||||
|
_BROWSER_PROCESS = subprocess.Popen(cmd, **kwargs)
|
||||||
|
return _BROWSER_PROCESS
|
||||||
|
|
||||||
|
|
||||||
|
def stop_chrome_browser(driver: webdriver.Chrome | None = None) -> None:
|
||||||
|
"""Disconnect Selenium and terminate the Chromium process we launched."""
|
||||||
|
if driver is not None:
|
||||||
|
try:
|
||||||
|
driver.quit()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
global _BROWSER_PROCESS
|
||||||
|
proc = _BROWSER_PROCESS
|
||||||
|
_BROWSER_PROCESS = None
|
||||||
|
if proc is not None and proc.poll() is None:
|
||||||
|
try:
|
||||||
|
proc.terminate()
|
||||||
|
proc.wait(timeout=8)
|
||||||
|
except Exception:
|
||||||
|
try:
|
||||||
|
proc.kill()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if wait_for_debug_port(DEBUG_PORT, timeout=1):
|
||||||
|
_kill_processes_listening_on_port(DEBUG_PORT)
|
||||||
|
|
||||||
|
|
||||||
|
def _kill_processes_listening_on_port(port: int) -> None:
|
||||||
|
"""Best-effort cleanup when the stored Popen handle is unavailable."""
|
||||||
|
if sys.platform != "win32":
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
output = subprocess.check_output(
|
||||||
|
["netstat", "-ano", "-p", "tcp"],
|
||||||
|
text=True,
|
||||||
|
errors="replace",
|
||||||
|
**_hidden_subprocess_kwargs(),
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return
|
||||||
|
target_pids: set[int] = set()
|
||||||
|
needle = f":{port}"
|
||||||
|
for line in output.splitlines():
|
||||||
|
if needle not in line or "LISTENING" not in line.upper():
|
||||||
|
continue
|
||||||
|
parts = line.split()
|
||||||
|
if not parts:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
target_pids.add(int(parts[-1]))
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
for pid in target_pids:
|
||||||
|
if pid <= 0:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
subprocess.run(
|
||||||
|
["taskkill", "/PID", str(pid), "/T", "/F"],
|
||||||
|
check=False,
|
||||||
|
stdout=subprocess.DEVNULL,
|
||||||
|
stderr=subprocess.DEVNULL,
|
||||||
|
**_hidden_subprocess_kwargs(),
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _apply_background_window(driver: webdriver.Chrome) -> None:
|
||||||
|
try:
|
||||||
|
driver.maximize_window()
|
||||||
|
except WebDriverException:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _bring_browser_to_front(driver: webdriver.Chrome) -> None:
|
||||||
|
"""Best-effort focus for scheduled/server runs so captures are not blank."""
|
||||||
|
if sys.platform != "win32":
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
import ctypes
|
||||||
|
|
||||||
|
user32 = ctypes.windll.user32
|
||||||
|
hwnd = user32.GetForegroundWindow()
|
||||||
|
try:
|
||||||
|
rect = driver.get_window_rect()
|
||||||
|
target_x = int(rect.get("x", 0) + max(rect.get("width", 800) // 2, 1))
|
||||||
|
target_y = int(rect.get("y", 0) + max(rect.get("height", 600) // 2, 1))
|
||||||
|
user32.SetCursorPos(target_x, target_y)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
if hwnd:
|
||||||
|
user32.SetForegroundWindow(hwnd)
|
||||||
|
driver.switch_to.window(driver.current_window_handle)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def prepare_browser_for_screenshot(driver: webdriver.Chrome) -> None:
|
||||||
|
"""Maximize and focus the automation browser before board screenshots."""
|
||||||
|
try:
|
||||||
|
driver.switch_to.default_content()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
_apply_background_window(driver)
|
||||||
|
_bring_browser_to_front(driver)
|
||||||
|
time.sleep(0.6)
|
||||||
|
|
||||||
|
|
||||||
|
def _hidden_subprocess_kwargs() -> dict:
|
||||||
|
"""Run helper processes without flashing a console window on Windows."""
|
||||||
|
if sys.platform != "win32":
|
||||||
|
return {}
|
||||||
|
kwargs: dict = {"creationflags": subprocess.CREATE_NO_WINDOW}
|
||||||
|
startupinfo = subprocess.STARTUPINFO()
|
||||||
|
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
|
||||||
|
startupinfo.wShowWindow = 0
|
||||||
|
kwargs["startupinfo"] = startupinfo
|
||||||
|
return kwargs
|
||||||
|
|
||||||
|
|
||||||
|
def _capture_desktop_powershell(path: Path) -> bool:
|
||||||
|
"""Primary monitor capture on Windows (includes taskbar date/time)."""
|
||||||
|
if sys.platform != "win32":
|
||||||
|
return False
|
||||||
|
out = str(path.resolve())
|
||||||
|
ps_script = f"""
|
||||||
|
Add-Type -AssemblyName System.Windows.Forms,System.Drawing
|
||||||
|
$bounds = [System.Windows.Forms.Screen]::PrimaryScreen.Bounds
|
||||||
|
$bmp = New-Object System.Drawing.Bitmap $bounds.Width, $bounds.Height
|
||||||
|
$graphics = [System.Drawing.Graphics]::FromImage($bmp)
|
||||||
|
$graphics.CopyFromScreen($bounds.Location, [System.Drawing.Point]::Empty, $bounds.Size)
|
||||||
|
$bmp.Save('{out.replace("'", "''")}', [System.Drawing.Imaging.ImageFormat]::Png)
|
||||||
|
$graphics.Dispose()
|
||||||
|
$bmp.Dispose()
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
subprocess.run(
|
||||||
|
[
|
||||||
|
"powershell",
|
||||||
|
"-NoProfile",
|
||||||
|
"-WindowStyle",
|
||||||
|
"Hidden",
|
||||||
|
"-Command",
|
||||||
|
ps_script,
|
||||||
|
],
|
||||||
|
check=True,
|
||||||
|
timeout=20,
|
||||||
|
capture_output=True,
|
||||||
|
**_hidden_subprocess_kwargs(),
|
||||||
|
)
|
||||||
|
return path.is_file() and path.stat().st_size > 0
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def capture_desktop_screenshot(path: Path) -> bool:
|
||||||
|
"""Save a full-desktop PNG (taskbar clock visible). Falls back to False if unavailable."""
|
||||||
|
path = Path(path)
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
try:
|
||||||
|
from PIL import ImageGrab
|
||||||
|
|
||||||
|
ImageGrab.grab(all_screens=False).save(str(path), "PNG")
|
||||||
|
if path.is_file():
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
import mss
|
||||||
|
import mss.tools
|
||||||
|
|
||||||
|
with mss.mss() as sct:
|
||||||
|
monitor = sct.monitors[1]
|
||||||
|
shot = sct.grab(monitor)
|
||||||
|
mss.tools.to_png(shot.rgb, shot.size, output=str(path))
|
||||||
|
return path.is_file()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
if _capture_desktop_powershell(path):
|
||||||
|
return True
|
||||||
|
try:
|
||||||
|
import pyautogui
|
||||||
|
|
||||||
|
pyautogui.screenshot(str(path))
|
||||||
|
return path.is_file()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _is_image_mostly_black(path: Path, *, threshold: int = 18) -> bool:
|
||||||
|
"""True when a PNG is nearly all black (common on headless/disconnected server sessions)."""
|
||||||
|
try:
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
with Image.open(path) as img:
|
||||||
|
rgb = img.convert("RGB")
|
||||||
|
sample = rgb.resize((64, 64))
|
||||||
|
pixels = list(sample.getdata())
|
||||||
|
if not pixels:
|
||||||
|
return True
|
||||||
|
avg = sum(sum(p) for p in pixels) / (len(pixels) * 3)
|
||||||
|
return avg < threshold
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _capture_browser_with_timestamp_bar(driver: webdriver.Chrome, path: Path) -> bool:
|
||||||
|
"""
|
||||||
|
Fallback when desktop capture is black: maximized browser PNG plus a taskbar-style clock strip.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
import io
|
||||||
|
|
||||||
|
from PIL import Image, ImageDraw
|
||||||
|
|
||||||
|
prepare_browser_for_screenshot(driver)
|
||||||
|
png = driver.get_screenshot_as_png()
|
||||||
|
browser = Image.open(io.BytesIO(png)).convert("RGB")
|
||||||
|
bar_h = 48
|
||||||
|
canvas = Image.new("RGB", (browser.width, browser.height + bar_h), (30, 30, 30))
|
||||||
|
canvas.paste(browser, (0, 0))
|
||||||
|
draw = ImageDraw.Draw(canvas)
|
||||||
|
stamp = datetime.now().strftime("%I:%M %p %m/%d/%Y").lstrip("0")
|
||||||
|
draw.text((max(browser.width - 260, 8), browser.height + 14), stamp, fill=(255, 255, 255))
|
||||||
|
canvas.save(str(path), "PNG")
|
||||||
|
return path.is_file() and path.stat().st_size > 0
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def capture_fullscreen_screenshot(driver: webdriver.Chrome, path: Path) -> bool:
|
||||||
|
"""
|
||||||
|
Full-screen capture with system taskbar visible when an interactive desktop exists.
|
||||||
|
|
||||||
|
On scheduled/server sessions without a visible desktop, falls back to a maximized
|
||||||
|
browser screenshot plus a timestamp bar (instead of an all-black PNG).
|
||||||
|
"""
|
||||||
|
path = Path(path)
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
log_step("Fullscreen screenshot", path=str(path))
|
||||||
|
prepare_browser_for_screenshot(driver)
|
||||||
|
if capture_desktop_screenshot(path) and not _is_image_mostly_black(path):
|
||||||
|
log_step("Fullscreen screenshot saved", path=str(path))
|
||||||
|
return True
|
||||||
|
if _is_image_mostly_black(path):
|
||||||
|
try:
|
||||||
|
path.unlink(missing_ok=True)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
if _capture_browser_with_timestamp_bar(driver, path):
|
||||||
|
log_step("Browser screenshot with timestamp bar saved", path=str(path), mode="server_fallback")
|
||||||
|
return True
|
||||||
|
driver.save_screenshot(str(path))
|
||||||
|
log_step("Viewport screenshot saved (desktop capture unavailable)", path=str(path))
|
||||||
|
return path.is_file()
|
||||||
|
|
||||||
|
|
||||||
|
def wait_for_debug_port(port: int = DEBUG_PORT, timeout: int = 40) -> bool:
|
||||||
|
start = time.time()
|
||||||
|
while time.time() - start < timeout:
|
||||||
|
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
|
||||||
|
if s.connect_ex(("127.0.0.1", port)) == 0:
|
||||||
|
return True
|
||||||
|
time.sleep(1)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def is_debug_port_open(port: int = DEBUG_PORT) -> bool:
|
||||||
|
return wait_for_debug_port(port, timeout=1)
|
||||||
|
|
||||||
|
|
||||||
|
def _version_from_portable_dir(browser_path: str) -> str:
|
||||||
|
"""Read e.g. 150.0.7852.0 from 150.0.7852.0.manifest next to chrome.exe."""
|
||||||
|
folder = Path(browser_path).parent
|
||||||
|
for manifest in folder.glob("*.manifest"):
|
||||||
|
match = _VERSION_RE.search(manifest.name)
|
||||||
|
if match:
|
||||||
|
return match.group(1)
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_browser_version(browser_path: str) -> str:
|
||||||
|
"""Best-effort version for the browser we launched (no hanging --version)."""
|
||||||
|
version = _get_browser_version_from_debug_port(timeout_secs=5.0)
|
||||||
|
if version:
|
||||||
|
return version
|
||||||
|
if browser_path:
|
||||||
|
version = _version_from_portable_dir(browser_path)
|
||||||
|
if version:
|
||||||
|
return version
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_chromedriver_cft(browser_version: str) -> str:
|
||||||
|
"""
|
||||||
|
Download chromedriver matching ``browser_version`` from Chrome for Testing.
|
||||||
|
Cached under ``.drivers/<version>/chromedriver.exe``.
|
||||||
|
"""
|
||||||
|
dest = _DRIVERS_DIR / browser_version / "chromedriver.exe"
|
||||||
|
if dest.is_file():
|
||||||
|
return str(dest)
|
||||||
|
|
||||||
|
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
zip_url = f"{_CFT_DOWNLOAD_BASE}/{browser_version}/win64/chromedriver-win64.zip"
|
||||||
|
zip_path = dest.parent / "chromedriver-win64.zip"
|
||||||
|
print(f"[INFO] Downloading ChromeDriver {browser_version} (Chrome for Testing)...")
|
||||||
|
|
||||||
|
_download_timeout = int(
|
||||||
|
os.environ.get("MISF_CHROMEDRIVER_DOWNLOAD_TIMEOUT", "60")
|
||||||
|
)
|
||||||
|
last_exc: Exception | None = None
|
||||||
|
for attempt in range(1, 4):
|
||||||
|
try:
|
||||||
|
req = urlopen(zip_url, timeout=_download_timeout)
|
||||||
|
with open(zip_path, "wb") as fh:
|
||||||
|
shutil.copyfileobj(req, fh)
|
||||||
|
break
|
||||||
|
except Exception as exc:
|
||||||
|
last_exc = exc
|
||||||
|
print(f"[WARN] ChromeDriver download attempt {attempt}/3 failed: {exc}")
|
||||||
|
if zip_path.exists():
|
||||||
|
zip_path.unlink(missing_ok=True)
|
||||||
|
time.sleep(2 * attempt)
|
||||||
|
else:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"ChromeDriver {browser_version} download failed after 3 attempts: {last_exc}"
|
||||||
|
)
|
||||||
|
|
||||||
|
if not zipfile.is_zipfile(zip_path):
|
||||||
|
zip_path.unlink(missing_ok=True)
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Downloaded ChromeDriver archive is not a valid ZIP: {zip_url}"
|
||||||
|
)
|
||||||
|
|
||||||
|
with zipfile.ZipFile(zip_path) as archive:
|
||||||
|
member = next(
|
||||||
|
(name for name in archive.namelist() if name.endswith("chromedriver.exe")),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if not member:
|
||||||
|
raise RuntimeError(f"chromedriver.exe not found in {zip_url}")
|
||||||
|
archive.extract(member, dest.parent)
|
||||||
|
extracted = dest.parent / member
|
||||||
|
extracted.rename(dest)
|
||||||
|
|
||||||
|
try:
|
||||||
|
zip_path.unlink(missing_ok=True)
|
||||||
|
nested = dest.parent / "chromedriver-win64"
|
||||||
|
if nested.is_dir():
|
||||||
|
shutil.rmtree(nested, ignore_errors=True)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
print(f"[INFO] ChromeDriver installed -> {dest}")
|
||||||
|
return str(dest)
|
||||||
|
|
||||||
|
|
||||||
|
def _get_browser_version_from_debug_port(timeout_secs: float = 2.0) -> str:
|
||||||
|
"""
|
||||||
|
Read Chromium version from the running remote-debug instance.
|
||||||
|
|
||||||
|
Example payload: {"Browser":"Chrome/150.0.7852.0", ...}
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
with urlopen(f"http://127.0.0.1:{DEBUG_PORT}/json/version", timeout=timeout_secs) as resp:
|
||||||
|
raw = resp.read().decode("utf-8", errors="replace")
|
||||||
|
data = json.loads(raw)
|
||||||
|
browser = str(data.get("Browser", "")).strip()
|
||||||
|
# "Chrome/150.0.7852.0"
|
||||||
|
if "/" in browser:
|
||||||
|
return browser.split("/", 1)[1].strip()
|
||||||
|
return browser
|
||||||
|
except (URLError, TimeoutError, ValueError, OSError):
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def create_proxy_auth_extension(
|
||||||
|
proxy_ip: str, proxy_port: str, proxy_username: str, proxy_password: str
|
||||||
|
) -> str:
|
||||||
|
manifest_json = """{
|
||||||
|
"version": "1.0.0",
|
||||||
|
"manifest_version": 2,
|
||||||
|
"name": "Chrome Proxy Auth Extension",
|
||||||
|
"permissions": [
|
||||||
|
"proxy", "tabs", "unlimitedStorage", "storage",
|
||||||
|
"<all_urls>", "webRequest", "webRequestBlocking"
|
||||||
|
],
|
||||||
|
"background": { "scripts": ["background.js"] },
|
||||||
|
"minimum_chrome_version": "22.0.0"
|
||||||
|
}"""
|
||||||
|
_u = json.dumps(proxy_username)
|
||||||
|
_p = json.dumps(proxy_password)
|
||||||
|
background_js = f"""function callbackFn(details) {{
|
||||||
|
return {{ authCredentials: {{ username: {_u}, password: {_p} }} }};
|
||||||
|
}}
|
||||||
|
chrome.webRequest.onAuthRequired.addListener(
|
||||||
|
callbackFn, {{urls: ["<all_urls>"]}}, ["blocking"]
|
||||||
|
);"""
|
||||||
|
ext_dir = Path(tempfile.mkdtemp(prefix="proxy_auth_ext_"))
|
||||||
|
atexit.register(shutil.rmtree, str(ext_dir), True)
|
||||||
|
(ext_dir / "manifest.json").write_text(manifest_json, encoding="utf-8")
|
||||||
|
(ext_dir / "background.js").write_text(background_js, encoding="utf-8")
|
||||||
|
print(f"[INFO] Proxy auth extension folder -> {ext_dir}")
|
||||||
|
return str(ext_dir)
|
||||||
|
|
||||||
|
|
||||||
|
def start_chrome_debug(
|
||||||
|
proxy_ip=None,
|
||||||
|
proxy_port=None,
|
||||||
|
proxy_username=None,
|
||||||
|
proxy_password=None,
|
||||||
|
) -> bool:
|
||||||
|
global _PROXY_CREDENTIALS
|
||||||
|
|
||||||
|
browser_path = find_chromium_executable()
|
||||||
|
if not browser_path:
|
||||||
|
raise RuntimeError(
|
||||||
|
"Chromium not found. Place the portable bundle in chrome-win\\chrome-win\\ "
|
||||||
|
"(chrome.exe + chromedriver.exe), install Chromium, or set CHROMIUM_PATH. "
|
||||||
|
"(Google Chrome: MISF_ALLOW_CHROME=1.)"
|
||||||
|
)
|
||||||
|
|
||||||
|
log_step(
|
||||||
|
"Launch Chromium",
|
||||||
|
executable=browser_path,
|
||||||
|
background=str(_browser_runs_in_background()).lower(),
|
||||||
|
)
|
||||||
|
global _LAST_BROWSER_PATH
|
||||||
|
_LAST_BROWSER_PATH = browser_path
|
||||||
|
os.makedirs(USER_DATA_DIR, exist_ok=True)
|
||||||
|
|
||||||
|
cmd = [
|
||||||
|
browser_path,
|
||||||
|
f"--remote-debugging-port={DEBUG_PORT}",
|
||||||
|
"--remote-allow-origins=*",
|
||||||
|
f"--user-data-dir={USER_DATA_DIR}",
|
||||||
|
"--disable-blink-features=AutomationControlled",
|
||||||
|
"--disable-popup-blocking",
|
||||||
|
"--no-first-run",
|
||||||
|
"--no-default-browser-check",
|
||||||
|
"--password-store=basic",
|
||||||
|
"--disable-restore-session-state",
|
||||||
|
"--disable-session-crashed-bubble",
|
||||||
|
"--hide-crash-restore-bubble",
|
||||||
|
"about:blank",
|
||||||
|
]
|
||||||
|
if _browser_runs_in_background():
|
||||||
|
cmd.append("--start-minimized")
|
||||||
|
|
||||||
|
if proxy_ip and proxy_port:
|
||||||
|
port_str = str(proxy_port).strip()
|
||||||
|
if not port_str.isdigit():
|
||||||
|
print(f"[WARN] Invalid proxy port '{port_str}' — skipping proxy")
|
||||||
|
_PROXY_CREDENTIALS = {}
|
||||||
|
else:
|
||||||
|
cmd.append(f"--proxy-server=http://{proxy_ip.strip()}:{port_str}")
|
||||||
|
cmd.append(
|
||||||
|
"--proxy-bypass-list=127.0.0.1;localhost;<-loopback>;"
|
||||||
|
+ CHROME_PROXY_BYPASS_OFFICE
|
||||||
|
)
|
||||||
|
log_step("Proxy configured", host=proxy_ip.strip(), port=port_str)
|
||||||
|
if proxy_username and proxy_password:
|
||||||
|
ext_folder = create_proxy_auth_extension(
|
||||||
|
proxy_ip.strip(),
|
||||||
|
port_str,
|
||||||
|
proxy_username.strip(),
|
||||||
|
proxy_password,
|
||||||
|
)
|
||||||
|
cmd.append(f"--load-extension={ext_folder}")
|
||||||
|
_PROXY_CREDENTIALS = {
|
||||||
|
"username": proxy_username.strip(),
|
||||||
|
"password": str(proxy_password),
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
_PROXY_CREDENTIALS = {}
|
||||||
|
else:
|
||||||
|
print("[INFO] No proxy configured — direct connection")
|
||||||
|
_PROXY_CREDENTIALS = {}
|
||||||
|
|
||||||
|
if is_debug_port_open(DEBUG_PORT):
|
||||||
|
log_step("Debug port already in use — closing previous browser session")
|
||||||
|
stop_chrome_browser()
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
_launch_browser_process(cmd)
|
||||||
|
|
||||||
|
if not wait_for_debug_port(DEBUG_PORT, timeout=40):
|
||||||
|
print("[ERROR] Browser debug port did not open in time")
|
||||||
|
return False
|
||||||
|
|
||||||
|
print("[INFO] Browser debug port ready")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _find_chromedriver_path() -> str | None:
|
||||||
|
"""
|
||||||
|
Return a chromedriver path matching the browser we started.
|
||||||
|
|
||||||
|
Prefers ``chromedriver.exe`` bundled in ``chrome-win/``. Otherwise downloads
|
||||||
|
a matching Chrome-for-Testing driver, then Selenium Manager.
|
||||||
|
"""
|
||||||
|
for env_key in ("CHROMEDRIVER_PATH", "WEBDRIVER_CHROME_DRIVER"):
|
||||||
|
raw = os.environ.get(env_key, "").strip().strip('"')
|
||||||
|
if raw and os.path.isfile(raw):
|
||||||
|
_agent_log(
|
||||||
|
"A",
|
||||||
|
"src/chrome.py:_find_chromedriver_path",
|
||||||
|
"Using explicit chromedriver path",
|
||||||
|
{"basename": os.path.basename(raw)},
|
||||||
|
)
|
||||||
|
return raw
|
||||||
|
|
||||||
|
browser_path = _LAST_BROWSER_PATH or ""
|
||||||
|
bundled = _bundled_chromedriver_for(browser_path)
|
||||||
|
if bundled:
|
||||||
|
log_step("Using bundled chromedriver", path=bundled)
|
||||||
|
_agent_log(
|
||||||
|
"A",
|
||||||
|
"src/chrome.py:_find_chromedriver_path",
|
||||||
|
"Using chrome-win chromedriver",
|
||||||
|
{"basename": os.path.basename(bundled)},
|
||||||
|
)
|
||||||
|
return bundled
|
||||||
|
|
||||||
|
browser_version = _resolve_browser_version(browser_path)
|
||||||
|
_agent_log(
|
||||||
|
"A",
|
||||||
|
"src/chrome.py:_find_chromedriver_path",
|
||||||
|
"Detected browser version",
|
||||||
|
{
|
||||||
|
"browser_version": browser_version,
|
||||||
|
"from_portable": "chrome-win" in (browser_path or "").lower(),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
if browser_version:
|
||||||
|
try:
|
||||||
|
driver_path = _ensure_chromedriver_cft(browser_version)
|
||||||
|
_agent_log(
|
||||||
|
"A",
|
||||||
|
"src/chrome.py:_find_chromedriver_path",
|
||||||
|
"Chrome for Testing chromedriver ready",
|
||||||
|
{"browser_version": browser_version},
|
||||||
|
)
|
||||||
|
return driver_path
|
||||||
|
except Exception as exc:
|
||||||
|
print(
|
||||||
|
f"[WARN] Could not download ChromeDriver {browser_version}: {exc}"
|
||||||
|
)
|
||||||
|
_agent_log(
|
||||||
|
"A",
|
||||||
|
"src/chrome.py:_find_chromedriver_path",
|
||||||
|
"CfT chromedriver download failed",
|
||||||
|
{"error": str(exc)[:200]},
|
||||||
|
)
|
||||||
|
|
||||||
|
_agent_log(
|
||||||
|
"A",
|
||||||
|
"src/chrome.py:_find_chromedriver_path",
|
||||||
|
"Using Selenium Manager for chromedriver",
|
||||||
|
{},
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
"[WARN] Browser version unknown — Selenium Manager may pick the wrong driver. "
|
||||||
|
"Set CHROMEDRIVER_PATH or use installed Chrome (default)."
|
||||||
|
)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _fill_proxy_auth_js_alert_if_present(
|
||||||
|
driver, username: str, password: str, wait_secs: float
|
||||||
|
) -> bool:
|
||||||
|
deadline = time.monotonic() + wait_secs
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
try:
|
||||||
|
alert = driver.switch_to.alert
|
||||||
|
alert.send_keys(Keys.CONTROL + "a")
|
||||||
|
alert.send_keys(username)
|
||||||
|
alert.send_keys(Keys.TAB)
|
||||||
|
alert.send_keys(Keys.CONTROL + "a")
|
||||||
|
alert.send_keys(password)
|
||||||
|
alert.accept()
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
time.sleep(0.25)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _try_fill_proxy_auth_native_dialog(username: str, password: str) -> bool:
|
||||||
|
if sys.platform != "win32":
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
import pyautogui
|
||||||
|
except ImportError:
|
||||||
|
print("[INFO] pyautogui not installed — skip native proxy dialog fill")
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
time.sleep(0.6)
|
||||||
|
pyautogui.FAILSAFE = True
|
||||||
|
pyautogui.hotkey("ctrl", "a")
|
||||||
|
pyautogui.press("backspace")
|
||||||
|
pyautogui.write(username, interval=0.02)
|
||||||
|
pyautogui.press("tab")
|
||||||
|
pyautogui.hotkey("ctrl", "a")
|
||||||
|
pyautogui.press("backspace")
|
||||||
|
pyautogui.write(password, interval=0.02)
|
||||||
|
pyautogui.press("enter")
|
||||||
|
return True
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"[WARN] Native proxy keyboard fill failed: {exc}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _microsoft_login_visible(driver) -> bool:
|
||||||
|
try:
|
||||||
|
url = (driver.current_url or "").lower()
|
||||||
|
if "login.microsoftonline.com" in url or "login.live.com" in url:
|
||||||
|
return True
|
||||||
|
return bool(
|
||||||
|
driver.find_elements(By.CSS_SELECTOR, "input[type='email']")
|
||||||
|
or driver.find_elements(By.NAME, "loginfmt")
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def dismiss_proxy_auth_popup(
|
||||||
|
driver,
|
||||||
|
username: str,
|
||||||
|
password: str,
|
||||||
|
wait_secs: float = 5.0,
|
||||||
|
max_attempts: int = 3,
|
||||||
|
) -> bool:
|
||||||
|
for attempt in range(1, max_attempts + 1):
|
||||||
|
if _fill_proxy_auth_js_alert_if_present(driver, username, password, wait_secs):
|
||||||
|
print(f"[INFO] Proxy auth handled (JS dialog) attempt {attempt}/{max_attempts}")
|
||||||
|
return True
|
||||||
|
if attempt < max_attempts:
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
if _try_fill_proxy_auth_native_dialog(username, password):
|
||||||
|
time.sleep(2)
|
||||||
|
if _microsoft_login_visible(driver):
|
||||||
|
print("[INFO] Microsoft login visible after proxy auth — proceeding")
|
||||||
|
return True
|
||||||
|
|
||||||
|
if _microsoft_login_visible(driver):
|
||||||
|
print("[INFO] Microsoft login page present — proxy session usable")
|
||||||
|
return True
|
||||||
|
|
||||||
|
print("[WARN] Proxy auth popup could not be dismissed")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def attach_browser(
|
||||||
|
retries: int = 5,
|
||||||
|
download_dir: Path | str | None = None,
|
||||||
|
*,
|
||||||
|
create_download_dir: bool = False,
|
||||||
|
) -> webdriver.Chrome | None:
|
||||||
|
driver_path = _find_chromedriver_path()
|
||||||
|
service = Service(driver_path) if driver_path else Service()
|
||||||
|
dl_dir = Path(download_dir) if download_dir else DOWNLOADS_DIR
|
||||||
|
if create_download_dir:
|
||||||
|
dl_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
for attempt in range(1, retries + 1):
|
||||||
|
try:
|
||||||
|
log_step("Attach browser", attempt=f"{attempt}/{retries}", download_dir=str(dl_dir))
|
||||||
|
chrome_options = Options()
|
||||||
|
chrome_options.add_experimental_option(
|
||||||
|
"debuggerAddress", f"127.0.0.1:{DEBUG_PORT}"
|
||||||
|
)
|
||||||
|
chrome_options.add_experimental_option(
|
||||||
|
"prefs",
|
||||||
|
{
|
||||||
|
"download.default_directory": str(dl_dir),
|
||||||
|
"download.prompt_for_download": False,
|
||||||
|
"plugins.always_open_pdf_externally": True,
|
||||||
|
"download.directory_upgrade": True,
|
||||||
|
"credentials_enable_service": False,
|
||||||
|
"profile.password_manager_enabled": False,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
driver = webdriver.Chrome(service=service, options=chrome_options)
|
||||||
|
_ = driver.current_url
|
||||||
|
_apply_background_window(driver)
|
||||||
|
driver.set_page_load_timeout(90)
|
||||||
|
driver.set_script_timeout(30)
|
||||||
|
log_step("Browser attached successfully")
|
||||||
|
return driver
|
||||||
|
except WebDriverException as exc:
|
||||||
|
log_step("Attach failed", attempt=f"{attempt}/{retries}", error=str(exc)[:200])
|
||||||
|
if attempt < retries:
|
||||||
|
time.sleep(3)
|
||||||
|
|
||||||
|
print("[ERROR] Failed to attach browser after all retries")
|
||||||
|
return None
|
||||||
+273
@@ -0,0 +1,273 @@
|
|||||||
|
"""OneDrive / Planner download configuration from Input/input.xlsx."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from totp_utils import validate_totp_secret
|
||||||
|
|
||||||
|
if getattr(sys, "frozen", False):
|
||||||
|
PROJECT_ROOT = Path(sys.executable).parent
|
||||||
|
else:
|
||||||
|
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
|
||||||
|
CONFIG_FILE = PROJECT_ROOT / "Input" / "input.xlsx"
|
||||||
|
TODAY_ISO_DATE = datetime.now().strftime("%Y-%m-%d")
|
||||||
|
EXPECTED_FACILITY_COUNT = 54
|
||||||
|
|
||||||
|
DEFAULT_ONEDRIVE_URL = (
|
||||||
|
"https://netorg393890-my.sharepoint.com/personal/joy_accesspsychology_org"
|
||||||
|
"/_layouts/15/onedrive.aspx"
|
||||||
|
"?id=%2Fpersonal%2Fjoy%5Faccesspsychology%5Forg%2FDocuments"
|
||||||
|
"%2FAccess%20Facility%20Spreadsheets3"
|
||||||
|
)
|
||||||
|
|
||||||
|
DEFAULT_PLANNER_URL = (
|
||||||
|
"https://planner.cloud.microsoft/webui/mytasks/assignedtome/view/grid"
|
||||||
|
"?tid=0c15ad2c-19dd-4ded-9cd1-20d344386414"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Rows like REPLACE_WITH_54TH_FACILITY_NAME are skipped when loading facilities.
|
||||||
|
_FACILITY_PLACEHOLDER_RE = re.compile(
|
||||||
|
r"^REPLACE_WITH_.*",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
|
||||||
|
_log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class AppConfig:
|
||||||
|
proxy_ip: str
|
||||||
|
proxy_port: str
|
||||||
|
proxy_username: str
|
||||||
|
proxy_password: str
|
||||||
|
onedrive_username: str
|
||||||
|
onedrive_password: str
|
||||||
|
microsoft_totp_secret: str
|
||||||
|
onedrive_url: str
|
||||||
|
planner_url: str
|
||||||
|
pdf_folder_path: str
|
||||||
|
|
||||||
|
|
||||||
|
def _load_dotenv_file() -> None:
|
||||||
|
"""Load .env from project root into os.environ (no-override: existing vars take priority)."""
|
||||||
|
env_path = PROJECT_ROOT / ".env"
|
||||||
|
if not env_path.is_file():
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
load_dotenv(dotenv_path=env_path, override=False)
|
||||||
|
except ImportError:
|
||||||
|
# Fallback: basic KEY=VALUE parser (no multi-line or comment support)
|
||||||
|
for raw_line in env_path.read_text(encoding="utf-8").splitlines():
|
||||||
|
line = raw_line.strip()
|
||||||
|
if not line or line.startswith("#") or "=" not in line:
|
||||||
|
continue
|
||||||
|
key, _, value = line.partition("=")
|
||||||
|
key = key.strip()
|
||||||
|
value = value.strip().strip('"').strip("'")
|
||||||
|
if key and key not in os.environ:
|
||||||
|
os.environ[key] = value
|
||||||
|
|
||||||
|
|
||||||
|
def _env(name: str, *aliases: str) -> str:
|
||||||
|
for key in (name, *aliases):
|
||||||
|
val = os.environ.get(key, "").strip()
|
||||||
|
if val:
|
||||||
|
return val
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def _read_config_key_value_sheet(config_path: Path | None = None) -> dict[str, str]:
|
||||||
|
path = config_path or CONFIG_FILE
|
||||||
|
if not path.exists():
|
||||||
|
raise FileNotFoundError(f"input.xlsx not found at {path}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
df = pd.read_excel(str(path), sheet_name="Sheet1", header=None, usecols=[0, 1])
|
||||||
|
except Exception as exc:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Could not read config from {path} (Sheet1): {exc}"
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
def _clean(val) -> str:
|
||||||
|
s = str(val).strip()
|
||||||
|
return "" if s.lower() in ("nan", "none", "") else s
|
||||||
|
|
||||||
|
if len(df.columns) < 2:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Config sheet Sheet1 in {path} must have at least 2 columns (Key | Value)."
|
||||||
|
)
|
||||||
|
|
||||||
|
config: dict[str, str] = {}
|
||||||
|
for _, row in df.iterrows():
|
||||||
|
key = _clean(row[0])
|
||||||
|
value = _clean(row[1])
|
||||||
|
if key:
|
||||||
|
config[key] = value
|
||||||
|
return config
|
||||||
|
|
||||||
|
|
||||||
|
def missing_onedrive_credentials(
|
||||||
|
onedrive_username: str,
|
||||||
|
onedrive_password: str,
|
||||||
|
microsoft_totp_secret: str,
|
||||||
|
) -> list[str]:
|
||||||
|
missing = []
|
||||||
|
if not onedrive_username:
|
||||||
|
missing.append("OneDrive Username")
|
||||||
|
if not onedrive_password:
|
||||||
|
missing.append("OneDrive Password")
|
||||||
|
if not microsoft_totp_secret:
|
||||||
|
missing.append("Microsoft TOTP Secret")
|
||||||
|
return missing
|
||||||
|
|
||||||
|
|
||||||
|
def credentials_setup_message(missing: list[str]) -> str:
|
||||||
|
keys = ", ".join(missing)
|
||||||
|
return (
|
||||||
|
f"Missing: {keys}\n\n"
|
||||||
|
f"Option A — edit {CONFIG_FILE} Sheet1, column B next to each key.\n"
|
||||||
|
f"Option B — copy .env.example to .env in the project root and set:\n"
|
||||||
|
" ONEDRIVE_USERNAME, ONEDRIVE_PASSWORD, MICROSOFT_TOTP_SECRET\n"
|
||||||
|
" PLANNER_URL (optional)\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def read_app_config(config_path: Path | None = None) -> AppConfig:
|
||||||
|
"""Load full application config from input.xlsx with .env fallbacks."""
|
||||||
|
_load_dotenv_file()
|
||||||
|
config = _read_config_key_value_sheet(config_path)
|
||||||
|
|
||||||
|
proxy_ip = config.get("Proxy IP", "") or _env("PROXY_IP")
|
||||||
|
proxy_port = config.get("Proxy Port", "") or _env("PROXY_PORT")
|
||||||
|
proxy_username = config.get("Proxy Username", "") or _env("PROXY_USERNAME")
|
||||||
|
proxy_password = config.get("Proxy Password", "") or _env("PROXY_PASSWORD")
|
||||||
|
|
||||||
|
onedrive_username = config.get("OneDrive Username", "") or _env(
|
||||||
|
"ONEDRIVE_USERNAME", "OneDrive Username"
|
||||||
|
)
|
||||||
|
onedrive_password = config.get("OneDrive Password", "") or _env(
|
||||||
|
"ONEDRIVE_PASSWORD", "OneDrive Password"
|
||||||
|
)
|
||||||
|
microsoft_totp_secret = (
|
||||||
|
config.get("Microsoft TOTP Secret", "")
|
||||||
|
or config.get("OTP Secret", "")
|
||||||
|
or _env("MICROSOFT_TOTP_SECRET", "OTP_SECRET", "TOTP_SECRET")
|
||||||
|
)
|
||||||
|
onedrive_url = config.get("OneDrive URL", "") or _env("ONEDRIVE_URL") or DEFAULT_ONEDRIVE_URL
|
||||||
|
planner_url = (
|
||||||
|
config.get("Planner URL", "")
|
||||||
|
or config.get("Planner URL ", "")
|
||||||
|
or _env("PLANNER_URL")
|
||||||
|
or DEFAULT_PLANNER_URL
|
||||||
|
)
|
||||||
|
pdf_folder_path = config.get("pdf_folder_path", "") or _env("PDF_FOLDER_PATH")
|
||||||
|
|
||||||
|
_log.info("Proxy -> %s:%s", proxy_ip, proxy_port)
|
||||||
|
_log.info("OneDrive User -> %s", onedrive_username)
|
||||||
|
_log.info("Planner URL -> %s...", planner_url[:80])
|
||||||
|
_log.info("PDF folder path -> %s", pdf_folder_path or "(default Input/PDFs)")
|
||||||
|
|
||||||
|
return AppConfig(
|
||||||
|
proxy_ip=proxy_ip,
|
||||||
|
proxy_port=proxy_port,
|
||||||
|
proxy_username=proxy_username,
|
||||||
|
proxy_password=proxy_password,
|
||||||
|
onedrive_username=onedrive_username,
|
||||||
|
onedrive_password=onedrive_password,
|
||||||
|
microsoft_totp_secret=microsoft_totp_secret,
|
||||||
|
onedrive_url=onedrive_url,
|
||||||
|
planner_url=planner_url,
|
||||||
|
pdf_folder_path=pdf_folder_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def read_onedrive_config(config_path: Path | None = None) -> tuple:
|
||||||
|
"""
|
||||||
|
Legacy tuple API. Prefer :func:`read_app_config`.
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
(proxy_ip, proxy_port, proxy_username, proxy_password,
|
||||||
|
onedrive_username, onedrive_password, microsoft_totp_secret,
|
||||||
|
onedrive_url, pdf_folder_path, planner_url)
|
||||||
|
"""
|
||||||
|
cfg = read_app_config(config_path)
|
||||||
|
return (
|
||||||
|
cfg.proxy_ip,
|
||||||
|
cfg.proxy_port,
|
||||||
|
cfg.proxy_username,
|
||||||
|
cfg.proxy_password,
|
||||||
|
cfg.onedrive_username,
|
||||||
|
cfg.onedrive_password,
|
||||||
|
cfg.microsoft_totp_secret,
|
||||||
|
cfg.onedrive_url,
|
||||||
|
cfg.pdf_folder_path,
|
||||||
|
cfg.planner_url,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _is_placeholder_facility(name: str) -> bool:
|
||||||
|
return bool(_FACILITY_PLACEHOLDER_RE.match(name.strip()))
|
||||||
|
|
||||||
|
|
||||||
|
def read_facility_list(
|
||||||
|
config_path: Path | None = None,
|
||||||
|
*,
|
||||||
|
skip_placeholders: bool = True,
|
||||||
|
warn_if_not_expected_count: bool = True,
|
||||||
|
) -> list[str]:
|
||||||
|
path = config_path or CONFIG_FILE
|
||||||
|
if not path.exists():
|
||||||
|
raise FileNotFoundError(f"input.xlsx not found at {path}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
df = pd.read_excel(str(path), sheet_name="Facilities", usecols=[0])
|
||||||
|
except ValueError as exc:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Sheet 'Facilities' not found in {path}. "
|
||||||
|
"Add a sheet with column 'Facility Name' (one facility per row)."
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
# Always use the first column regardless of its header name.
|
||||||
|
col = df.columns[0]
|
||||||
|
|
||||||
|
names: list[str] = []
|
||||||
|
skipped: list[str] = []
|
||||||
|
for val in df[col]:
|
||||||
|
s = str(val).strip()
|
||||||
|
if not s or s.lower() in ("nan", "none", "facility name"):
|
||||||
|
continue
|
||||||
|
if skip_placeholders and _is_placeholder_facility(s):
|
||||||
|
skipped.append(s)
|
||||||
|
continue
|
||||||
|
names.append(s)
|
||||||
|
|
||||||
|
if skipped:
|
||||||
|
_log.warning(
|
||||||
|
"Skipped %d placeholder facility row(s): %s",
|
||||||
|
len(skipped),
|
||||||
|
", ".join(skipped[:3]),
|
||||||
|
)
|
||||||
|
|
||||||
|
if not names:
|
||||||
|
raise RuntimeError(f"No facility names found in 'Facilities' sheet of {path}.")
|
||||||
|
|
||||||
|
if warn_if_not_expected_count and len(names) != EXPECTED_FACILITY_COUNT:
|
||||||
|
_log.warning(
|
||||||
|
"Expected %d facilities, found %d (check Facilities sheet names).",
|
||||||
|
EXPECTED_FACILITY_COUNT,
|
||||||
|
len(names),
|
||||||
|
)
|
||||||
|
|
||||||
|
_log.info("Loaded %d facility name(s) from Facilities sheet", len(names))
|
||||||
|
return names
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
"""Shared NDJSON debug logging (MISF_DEBUG=1 only). No secrets/PII."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
from config import PROJECT_ROOT
|
||||||
|
|
||||||
|
_DEBUG_LOG = PROJECT_ROOT / "debug.log"
|
||||||
|
_AGENT_SESSION_LOG = PROJECT_ROOT / "debug-6d994d.log"
|
||||||
|
_AGENT_SESSION_ID = "6d994d"
|
||||||
|
|
||||||
|
|
||||||
|
def write_agent_session_debug(
|
||||||
|
hypothesis_id: str, location: str, message: str, data: dict
|
||||||
|
) -> None:
|
||||||
|
"""NDJSON debug log for Cursor agent debug sessions (MISF_DEBUG=1 only)."""
|
||||||
|
# region agent log
|
||||||
|
if os.environ.get("MISF_DEBUG", "").strip() not in ("1", "true", "yes"):
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
ts_ms = int(time.time() * 1000)
|
||||||
|
payload = {
|
||||||
|
"sessionId": _AGENT_SESSION_ID,
|
||||||
|
"runId": os.environ.get("MISF_RUN_ID", "run1"),
|
||||||
|
"hypothesisId": hypothesis_id,
|
||||||
|
"id": f"log_{ts_ms}_{uuid.uuid4().hex[:8]}",
|
||||||
|
"timestamp": ts_ms,
|
||||||
|
"location": location,
|
||||||
|
"message": message,
|
||||||
|
"data": data,
|
||||||
|
}
|
||||||
|
with _AGENT_SESSION_LOG.open("a", encoding="utf-8") as f:
|
||||||
|
f.write(json.dumps(payload, ensure_ascii=False) + "\n")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# endregion
|
||||||
|
|
||||||
|
|
||||||
|
def write_debug(hypothesis_id: str, location: str, message: str, data: dict) -> None:
|
||||||
|
if os.environ.get("MISF_DEBUG", "").strip() not in ("1", "true", "yes"):
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
ts_ms = int(time.time() * 1000)
|
||||||
|
payload = {
|
||||||
|
"sessionId": os.environ.get("MISF_SESSION_ID", "local"),
|
||||||
|
"runId": os.environ.get("MISF_RUN_ID", "run1"),
|
||||||
|
"hypothesisId": hypothesis_id,
|
||||||
|
"id": f"log_{ts_ms}_{uuid.uuid4().hex[:8]}",
|
||||||
|
"timestamp": ts_ms,
|
||||||
|
"location": location,
|
||||||
|
"message": message,
|
||||||
|
"data": data,
|
||||||
|
}
|
||||||
|
with _DEBUG_LOG.open("a", encoding="utf-8") as f:
|
||||||
|
f.write(json.dumps(payload, ensure_ascii=False) + "\n")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
@@ -0,0 +1,118 @@
|
|||||||
|
"""Live log viewer for windowed MISF_VOB_Download.exe (no console)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import queue
|
||||||
|
import threading
|
||||||
|
import tkinter as tk
|
||||||
|
from collections.abc import Callable
|
||||||
|
from tkinter import scrolledtext
|
||||||
|
|
||||||
|
|
||||||
|
class _TkLogHandler(logging.Handler):
|
||||||
|
"""Thread-safe logging handler that forwards records to a Tk text widget."""
|
||||||
|
|
||||||
|
def __init__(self, text_widget: scrolledtext.ScrolledText, root: tk.Tk) -> None:
|
||||||
|
super().__init__()
|
||||||
|
self._text = text_widget
|
||||||
|
self._root = root
|
||||||
|
self._queue: queue.Queue[str] = queue.Queue()
|
||||||
|
self._root.after(100, self._drain_queue)
|
||||||
|
|
||||||
|
def emit(self, record: logging.LogRecord) -> None:
|
||||||
|
try:
|
||||||
|
msg = self.format(record)
|
||||||
|
except Exception:
|
||||||
|
self.handleError(record)
|
||||||
|
return
|
||||||
|
self._queue.put(msg + "\n")
|
||||||
|
|
||||||
|
def _drain_queue(self) -> None:
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
line = self._queue.get_nowait()
|
||||||
|
except queue.Empty:
|
||||||
|
break
|
||||||
|
self._text.configure(state="normal")
|
||||||
|
self._text.insert(tk.END, line)
|
||||||
|
self._text.see(tk.END)
|
||||||
|
self._text.configure(state="disabled")
|
||||||
|
self._root.after(100, self._drain_queue)
|
||||||
|
|
||||||
|
|
||||||
|
def run_with_live_logs(workflow: Callable[[], int]) -> int:
|
||||||
|
"""
|
||||||
|
Show a scrollable log window and run *workflow* on a background thread.
|
||||||
|
|
||||||
|
Returns the workflow exit code after the user closes the window.
|
||||||
|
"""
|
||||||
|
root = tk.Tk()
|
||||||
|
root.title("MISF VOB Download — Live Log")
|
||||||
|
root.geometry("920x560")
|
||||||
|
root.minsize(640, 360)
|
||||||
|
|
||||||
|
header = tk.Label(
|
||||||
|
root,
|
||||||
|
text="Download in progress — logs update live below",
|
||||||
|
anchor="w",
|
||||||
|
font=("Segoe UI", 10, "bold"),
|
||||||
|
)
|
||||||
|
header.pack(fill="x", padx=10, pady=(10, 4))
|
||||||
|
|
||||||
|
text = scrolledtext.ScrolledText(
|
||||||
|
root,
|
||||||
|
wrap="word",
|
||||||
|
state="disabled",
|
||||||
|
font=("Consolas", 10),
|
||||||
|
)
|
||||||
|
text.pack(fill="both", expand=True, padx=10, pady=4)
|
||||||
|
|
||||||
|
status_var = tk.StringVar(value="Running...")
|
||||||
|
status_label = tk.Label(root, textvariable=status_var, anchor="w")
|
||||||
|
status_label.pack(fill="x", padx=10, pady=(0, 6))
|
||||||
|
|
||||||
|
close_btn = tk.Button(root, text="Close", state="disabled", width=12)
|
||||||
|
close_btn.pack(pady=(0, 10))
|
||||||
|
|
||||||
|
fmt = logging.Formatter(
|
||||||
|
"%(asctime)s [%(levelname)s] %(message)s",
|
||||||
|
datefmt="%Y-%m-%d %H:%M:%S",
|
||||||
|
)
|
||||||
|
handler = _TkLogHandler(text, root)
|
||||||
|
handler.setFormatter(fmt)
|
||||||
|
handler.setLevel(logging.INFO)
|
||||||
|
|
||||||
|
root_logger = logging.getLogger()
|
||||||
|
root_logger.addHandler(handler)
|
||||||
|
|
||||||
|
result: dict[str, int] = {"code": 0}
|
||||||
|
|
||||||
|
def _run_workflow() -> None:
|
||||||
|
try:
|
||||||
|
result["code"] = workflow()
|
||||||
|
except SystemExit as exc:
|
||||||
|
result["code"] = int(exc.code) if exc.code is not None else 0
|
||||||
|
except Exception:
|
||||||
|
result["code"] = 1
|
||||||
|
logging.getLogger(__name__).exception("Workflow failed")
|
||||||
|
finally:
|
||||||
|
def _on_done() -> None:
|
||||||
|
if result["code"] == 0:
|
||||||
|
status_var.set("Complete — you may close this window.")
|
||||||
|
else:
|
||||||
|
status_var.set(f"Finished with errors (exit {result['code']}). See log above.")
|
||||||
|
close_btn.configure(state="normal")
|
||||||
|
|
||||||
|
root.after(0, _on_done)
|
||||||
|
|
||||||
|
worker = threading.Thread(target=_run_workflow, daemon=True)
|
||||||
|
worker.start()
|
||||||
|
|
||||||
|
def _close() -> None:
|
||||||
|
root_logger.removeHandler(handler)
|
||||||
|
root.destroy()
|
||||||
|
|
||||||
|
close_btn.configure(command=_close)
|
||||||
|
root.protocol("WM_DELETE_WINDOW", _close)
|
||||||
|
root.mainloop()
|
||||||
|
return result["code"]
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
"""Structured logging for MISF download runs."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def setup_logging(project_root: Path, *, run_date: str | None = None) -> Path:
|
||||||
|
"""
|
||||||
|
Configure root logger and misf.run logger: console + download.log + moves.log.
|
||||||
|
Returns the main download log file path.
|
||||||
|
"""
|
||||||
|
date_str = run_date or datetime.now().strftime("%Y-%m-%d")
|
||||||
|
log_dir = project_root / "Output" / date_str / "logs"
|
||||||
|
log_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
log_path = log_dir / "download.log"
|
||||||
|
moves_path = log_dir / "moves.log"
|
||||||
|
|
||||||
|
level_name = os.environ.get("MISF_LOG_LEVEL", "INFO").upper()
|
||||||
|
level = getattr(logging, level_name, logging.INFO)
|
||||||
|
|
||||||
|
fmt = logging.Formatter(
|
||||||
|
"%(asctime)s [%(levelname)s] %(message)s",
|
||||||
|
datefmt="%Y-%m-%d %H:%M:%S",
|
||||||
|
)
|
||||||
|
|
||||||
|
root = logging.getLogger()
|
||||||
|
root.setLevel(level)
|
||||||
|
|
||||||
|
# Avoid duplicate handlers on repeated setup in same process.
|
||||||
|
for h in list(root.handlers):
|
||||||
|
root.removeHandler(h)
|
||||||
|
|
||||||
|
# Windowed .exe: log files only (no console / live log window).
|
||||||
|
if not getattr(sys, "frozen", False):
|
||||||
|
console = logging.StreamHandler()
|
||||||
|
console.setLevel(level)
|
||||||
|
console.setFormatter(fmt)
|
||||||
|
root.addHandler(console)
|
||||||
|
|
||||||
|
file_handler = logging.FileHandler(log_path, encoding="utf-8")
|
||||||
|
file_handler.setLevel(level)
|
||||||
|
file_handler.setFormatter(fmt)
|
||||||
|
root.addHandler(file_handler)
|
||||||
|
|
||||||
|
moves_handler = logging.FileHandler(moves_path, encoding="utf-8")
|
||||||
|
moves_handler.setLevel(level)
|
||||||
|
moves_handler.setFormatter(fmt)
|
||||||
|
|
||||||
|
run_logger = logging.getLogger("misf.run")
|
||||||
|
run_logger.setLevel(level)
|
||||||
|
run_logger.propagate = True
|
||||||
|
run_logger.addHandler(moves_handler)
|
||||||
|
|
||||||
|
# Module loggers used across the app
|
||||||
|
for name in ("misf", "src.planner_download", "src.onedrive_planner", "src.chrome", "src.config"):
|
||||||
|
logging.getLogger(name).setLevel(level)
|
||||||
|
|
||||||
|
logging.getLogger("misf.run").info(
|
||||||
|
"Logging enabled -> %s and %s (filter [STEP] in moves.log)",
|
||||||
|
log_path,
|
||||||
|
moves_path,
|
||||||
|
)
|
||||||
|
|
||||||
|
return log_path
|
||||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,747 @@
|
|||||||
|
# ================================================================
|
||||||
|
# MISF — OneDrive / Planner face-sheet download (orchestrator)
|
||||||
|
#
|
||||||
|
# Entry point: python run_download.py (from repository root)
|
||||||
|
# See README.md for the full business workflow and configuration.
|
||||||
|
# ================================================================
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import logging
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import traceback
|
||||||
|
import tkinter as tk
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from tkinter import messagebox
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from checkpoint import clear_checkpoint, load_checkpoint, save_checkpoint
|
||||||
|
from chrome import (
|
||||||
|
attach_browser,
|
||||||
|
dismiss_proxy_auth_popup,
|
||||||
|
set_default_download_directory,
|
||||||
|
start_chrome_debug,
|
||||||
|
stop_chrome_browser,
|
||||||
|
)
|
||||||
|
from config import (
|
||||||
|
CONFIG_FILE,
|
||||||
|
PROJECT_ROOT,
|
||||||
|
TODAY_ISO_DATE,
|
||||||
|
AppConfig,
|
||||||
|
credentials_setup_message,
|
||||||
|
missing_onedrive_credentials,
|
||||||
|
read_app_config,
|
||||||
|
read_facility_list,
|
||||||
|
)
|
||||||
|
from logging_setup import setup_logging
|
||||||
|
from run_log import log_step, log_url
|
||||||
|
from onedrive_planner import (
|
||||||
|
FacilityNotFoundError,
|
||||||
|
FacilityRunResult,
|
||||||
|
_remove_empty_directory,
|
||||||
|
_sanitize_folder_name,
|
||||||
|
configure_http_download_proxy,
|
||||||
|
consolidate_browser_to_single_tab,
|
||||||
|
login_and_open_planner,
|
||||||
|
process_facility,
|
||||||
|
reset_run_state,
|
||||||
|
save_failure_screenshot,
|
||||||
|
)
|
||||||
|
from totp_utils import validate_totp_secret
|
||||||
|
from debug_log import write_debug as _agent_log
|
||||||
|
|
||||||
|
_log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def _show_error_dialog(title: str, message: str) -> None:
|
||||||
|
try:
|
||||||
|
root = tk.Tk()
|
||||||
|
root.withdraw()
|
||||||
|
root.attributes("-topmost", True)
|
||||||
|
messagebox.showerror(title, message, parent=root)
|
||||||
|
root.destroy()
|
||||||
|
except Exception as dialog_err:
|
||||||
|
_log.error("%s: %s", title, message)
|
||||||
|
_log.warning("Could not show messagebox: %s", dialog_err)
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_deployment_layout() -> None:
|
||||||
|
"""Fail fast on a new machine when required folders/files are missing."""
|
||||||
|
issues: list[str] = []
|
||||||
|
example = PROJECT_ROOT / "Input" / "input.xlsx.example"
|
||||||
|
if not CONFIG_FILE.is_file():
|
||||||
|
hint = (
|
||||||
|
f"Copy and rename:\n {example}\n ->\n {CONFIG_FILE}"
|
||||||
|
if example.is_file()
|
||||||
|
else f"Create config file:\n {CONFIG_FILE}"
|
||||||
|
)
|
||||||
|
issues.append(f"Missing Input\\input.xlsx\n{hint}")
|
||||||
|
|
||||||
|
chrome_exe = PROJECT_ROOT / "chrome-win" / "chrome-win" / "chrome.exe"
|
||||||
|
chrome_driver = PROJECT_ROOT / "chrome-win" / "chrome-win" / "chromedriver.exe"
|
||||||
|
if not chrome_exe.is_file():
|
||||||
|
issues.append(
|
||||||
|
"Missing portable browser folder:\n"
|
||||||
|
f" {chrome_exe}\n"
|
||||||
|
"Copy the entire chrome-win\\ folder from the original machine "
|
||||||
|
"into the same folder as MISF_VOB_Download.exe."
|
||||||
|
)
|
||||||
|
elif not chrome_driver.is_file():
|
||||||
|
issues.append(
|
||||||
|
"Missing chromedriver next to portable chrome.exe:\n"
|
||||||
|
f" {chrome_driver}"
|
||||||
|
)
|
||||||
|
|
||||||
|
if issues:
|
||||||
|
message = (
|
||||||
|
"This PC is not set up to run MISF VOB Download yet.\n\n"
|
||||||
|
+ "\n\n".join(issues)
|
||||||
|
+ "\n\nSee README_PRODUCTION.txt in the app folder."
|
||||||
|
)
|
||||||
|
_show_error_dialog("MISF VOB Download — Setup Required", message)
|
||||||
|
raise FileNotFoundError(message.split("\n", 1)[0])
|
||||||
|
|
||||||
|
|
||||||
|
def _close_blank_tabs(driver) -> None:
|
||||||
|
"""Close any about:blank / empty tabs left by the Microsoft auth flow."""
|
||||||
|
try:
|
||||||
|
active = driver.current_window_handle
|
||||||
|
for handle in list(driver.window_handles):
|
||||||
|
if handle == active:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
driver.switch_to.window(handle)
|
||||||
|
url = (driver.current_url or "").strip().lower()
|
||||||
|
if url in ("about:blank", "", "data:,"):
|
||||||
|
driver.close()
|
||||||
|
_log.info("Closed blank tab (%s)", handle)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
driver.switch_to.window(active)
|
||||||
|
except Exception as exc:
|
||||||
|
_log.warning("_close_blank_tabs: %s", exc)
|
||||||
|
|
||||||
|
|
||||||
|
def _quit_browser(
|
||||||
|
driver, *, keep_browser: bool, is_proxy_error: bool = False
|
||||||
|
) -> None:
|
||||||
|
"""Close the browser unless keep_browser is set or a proxy error occurred."""
|
||||||
|
if keep_browser or is_proxy_error:
|
||||||
|
return
|
||||||
|
stop_chrome_browser(driver)
|
||||||
|
log_step("Browser closed")
|
||||||
|
|
||||||
|
|
||||||
|
def _failure_status(exc: Exception) -> str:
|
||||||
|
if isinstance(exc, FacilityNotFoundError):
|
||||||
|
return "Not appear"
|
||||||
|
err_msg = str(exc).lower()
|
||||||
|
if "no planner plan found" in err_msg or "plan not found" in err_msg:
|
||||||
|
return "Plan not found"
|
||||||
|
if "no files" in err_msg or "attachment" in err_msg:
|
||||||
|
return "No attachments"
|
||||||
|
return "FAILED"
|
||||||
|
|
||||||
|
|
||||||
|
def _success_status(
|
||||||
|
file_count: int,
|
||||||
|
tasks_total: int,
|
||||||
|
*,
|
||||||
|
tasks_with_downloads: int = 0,
|
||||||
|
tasks_with_attachments: int = 0,
|
||||||
|
tasks_skipped_no_attachment: int = 0,
|
||||||
|
) -> str:
|
||||||
|
if tasks_total <= 0:
|
||||||
|
return "No tasks"
|
||||||
|
if tasks_with_attachments <= 0:
|
||||||
|
if tasks_skipped_no_attachment > 0 or tasks_total > 0:
|
||||||
|
return "No attachments"
|
||||||
|
return "No tasks"
|
||||||
|
if tasks_with_downloads <= 0 or file_count <= 0:
|
||||||
|
return "Download failed"
|
||||||
|
if tasks_with_downloads >= tasks_with_attachments:
|
||||||
|
return "Success"
|
||||||
|
return "Partial"
|
||||||
|
|
||||||
|
|
||||||
|
def _format_status_changed(
|
||||||
|
tasks_with_downloads: int,
|
||||||
|
status_changed: int,
|
||||||
|
*,
|
||||||
|
via_task_details: int = 0,
|
||||||
|
) -> str:
|
||||||
|
if tasks_with_downloads <= 0:
|
||||||
|
return "N/A"
|
||||||
|
if status_changed >= tasks_with_downloads:
|
||||||
|
prefix = "Yes"
|
||||||
|
elif status_changed > 0:
|
||||||
|
prefix = "Partial"
|
||||||
|
else:
|
||||||
|
prefix = "No"
|
||||||
|
label = f"{prefix} - {status_changed}/{tasks_with_downloads} tasks"
|
||||||
|
if via_task_details > 0:
|
||||||
|
label += f" ({via_task_details} via task details)"
|
||||||
|
return label
|
||||||
|
|
||||||
|
|
||||||
|
def _write_download_summary(
|
||||||
|
results: dict[str, dict[str, object]],
|
||||||
|
failures: list[tuple[str, str, str]],
|
||||||
|
summary_path: Path,
|
||||||
|
*,
|
||||||
|
planner_url: str,
|
||||||
|
run_timestamp: str,
|
||||||
|
) -> None:
|
||||||
|
failure_names = {facility for facility, _, _ in failures}
|
||||||
|
rows = []
|
||||||
|
for facility, payload in results.items():
|
||||||
|
if facility in failure_names:
|
||||||
|
continue
|
||||||
|
original_paths = list(payload.get("original", []) or [])
|
||||||
|
file_count = len(original_paths)
|
||||||
|
tasks_total = int(payload.get("tasks_total", 0) or 0)
|
||||||
|
tasks_with_downloads = int(payload.get("tasks_with_downloads", 0) or 0)
|
||||||
|
tasks_with_attachments = int(payload.get("tasks_with_attachments", 0) or 0)
|
||||||
|
tasks_skipped_no_attachment = int(
|
||||||
|
payload.get("tasks_skipped_no_attachment", 0) or 0
|
||||||
|
)
|
||||||
|
status_changed = int(payload.get("status_changed", 0) or 0)
|
||||||
|
via_task_details = int(payload.get("status_via_task_details", 0) or 0)
|
||||||
|
rows.append({
|
||||||
|
"Run Timestamp": run_timestamp,
|
||||||
|
"Facility Name": facility,
|
||||||
|
"Files Downloaded": file_count,
|
||||||
|
"Status": _success_status(
|
||||||
|
file_count,
|
||||||
|
tasks_total,
|
||||||
|
tasks_with_downloads=tasks_with_downloads,
|
||||||
|
tasks_with_attachments=tasks_with_attachments,
|
||||||
|
tasks_skipped_no_attachment=tasks_skipped_no_attachment,
|
||||||
|
),
|
||||||
|
"Status Changed": _format_status_changed(
|
||||||
|
tasks_with_downloads,
|
||||||
|
status_changed,
|
||||||
|
via_task_details=via_task_details,
|
||||||
|
),
|
||||||
|
})
|
||||||
|
for facility, err, status in failures:
|
||||||
|
rows.append({
|
||||||
|
"Run Timestamp": run_timestamp,
|
||||||
|
"Facility Name": facility,
|
||||||
|
"Files Downloaded": 0,
|
||||||
|
"Status": status,
|
||||||
|
"Status Changed": "N/A",
|
||||||
|
})
|
||||||
|
|
||||||
|
summary_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
pd.DataFrame(rows).to_excel(summary_path, index=False, sheet_name="Download")
|
||||||
|
_log.info("Download summary -> %s", summary_path)
|
||||||
|
|
||||||
|
|
||||||
|
def _unique_workable_filename(workable_root: Path, base_name: str, ext: str) -> Path:
|
||||||
|
"""
|
||||||
|
Allocate a unique file path inside Workable/ using _N suffix when needed.
|
||||||
|
"""
|
||||||
|
ext = ext if ext.startswith(".") else f".{ext}"
|
||||||
|
first = workable_root / f"{base_name}{ext}"
|
||||||
|
if not first.exists():
|
||||||
|
return first
|
||||||
|
for n in range(2, 1000):
|
||||||
|
candidate = workable_root / f"{base_name}_{n}{ext}"
|
||||||
|
if not candidate.exists():
|
||||||
|
return candidate
|
||||||
|
raise RuntimeError(f"Could not allocate workable filename for {base_name}{ext}")
|
||||||
|
|
||||||
|
|
||||||
|
def _build_workable_files(
|
||||||
|
*,
|
||||||
|
facility_name: str,
|
||||||
|
downloaded: list[Path],
|
||||||
|
output_root: Path,
|
||||||
|
today_iso: str,
|
||||||
|
) -> tuple[list[Path], Path]:
|
||||||
|
"""
|
||||||
|
Copy downloaded files into Output/<date>/Workable with standardized names.
|
||||||
|
|
||||||
|
- No facility folders are created under Workable.
|
||||||
|
- Files are written directly under Workable/ with unique names.
|
||||||
|
"""
|
||||||
|
ddmmyyyy = datetime.strptime(today_iso, "%Y-%m-%d").strftime("%d-%m-%Y")
|
||||||
|
fac_safe = _sanitize_folder_name(facility_name)
|
||||||
|
workable_root = output_root / today_iso / "Workable"
|
||||||
|
medicare_root = output_root / today_iso / "Workable_Medicare"
|
||||||
|
workable_paths: list[Path] = []
|
||||||
|
if not downloaded:
|
||||||
|
return workable_paths, workable_root
|
||||||
|
|
||||||
|
workable_root.mkdir(parents=True, exist_ok=True)
|
||||||
|
medicare_root.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
for i, src in enumerate(downloaded, start=1):
|
||||||
|
ext = src.suffix or ".pdf"
|
||||||
|
base = f"{ddmmyyyy}_{fac_safe}" if len(downloaded) == 1 else f"{ddmmyyyy}_{fac_safe}_{i}"
|
||||||
|
dst = _unique_workable_filename(workable_root, base, ext)
|
||||||
|
try:
|
||||||
|
shutil.copy2(src, dst)
|
||||||
|
except OSError as exc:
|
||||||
|
_log.error("Failed to copy %s -> %s: %s", src.name, dst, exc)
|
||||||
|
raise
|
||||||
|
workable_paths.append(dst)
|
||||||
|
med_dst = medicare_root / dst.name
|
||||||
|
if med_dst.exists():
|
||||||
|
med_dst = _unique_workable_filename(medicare_root, dst.stem, ext)
|
||||||
|
try:
|
||||||
|
shutil.copy2(dst, med_dst)
|
||||||
|
except OSError as exc:
|
||||||
|
_log.error("Failed to mirror %s -> %s: %s", dst.name, med_dst, exc)
|
||||||
|
raise
|
||||||
|
|
||||||
|
_log.info(
|
||||||
|
"Workable files copied (%d) -> %s and %s",
|
||||||
|
len(workable_paths),
|
||||||
|
workable_root,
|
||||||
|
medicare_root,
|
||||||
|
)
|
||||||
|
return workable_paths, workable_root
|
||||||
|
|
||||||
|
|
||||||
|
def _prune_empty_date_output_dirs(output_root: Path, today_iso: str) -> None:
|
||||||
|
"""Remove empty Original/Workable day folders when no files were saved."""
|
||||||
|
for sub in ("Original", "Workable", "Workable_Medicare"):
|
||||||
|
root = output_root / today_iso / sub
|
||||||
|
if not root.is_dir():
|
||||||
|
continue
|
||||||
|
for child in list(root.iterdir()):
|
||||||
|
if child.is_dir() and not any(child.iterdir()):
|
||||||
|
try:
|
||||||
|
child.rmdir()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
if not any(root.iterdir()):
|
||||||
|
root.rmdir()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _remove_screenshots_under_output(output_root: Path) -> int:
|
||||||
|
"""Delete legacy _screenshots folders under Output (Original/Workable/etc.)."""
|
||||||
|
removed = 0
|
||||||
|
if not output_root.is_dir():
|
||||||
|
return removed
|
||||||
|
for shots_dir in output_root.rglob("_screenshots"):
|
||||||
|
if not shots_dir.is_dir():
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
shutil.rmtree(shots_dir)
|
||||||
|
removed += 1
|
||||||
|
_log.info("Removed screenshot folder: %s", shots_dir)
|
||||||
|
except Exception as exc:
|
||||||
|
_log.warning("Could not remove %s: %s", shots_dir, exc)
|
||||||
|
return removed
|
||||||
|
|
||||||
|
|
||||||
|
def _filter_facilities(
|
||||||
|
facilities: list[str],
|
||||||
|
*,
|
||||||
|
facility_limit: int,
|
||||||
|
facility_name: str | None,
|
||||||
|
) -> list[str]:
|
||||||
|
if facility_name:
|
||||||
|
name = facility_name.strip()
|
||||||
|
matches = [f for f in facilities if f.lower() == name.lower()]
|
||||||
|
if not matches:
|
||||||
|
raise ValueError(
|
||||||
|
f"Facility '{name}' not found on Facilities sheet. "
|
||||||
|
f"Loaded {len(facilities)} names."
|
||||||
|
)
|
||||||
|
return matches
|
||||||
|
if facility_limit > 0:
|
||||||
|
return facilities[:facility_limit]
|
||||||
|
return facilities
|
||||||
|
|
||||||
|
|
||||||
|
def run_download_workflow(
|
||||||
|
*,
|
||||||
|
facility_limit: int = 0,
|
||||||
|
facility_name: str | None = None,
|
||||||
|
resume: bool = False,
|
||||||
|
keep_browser: bool = False,
|
||||||
|
screenshot_on_error: bool = False,
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
Download face-sheet PDFs from Planner for configured facilities.
|
||||||
|
"""
|
||||||
|
reset_run_state()
|
||||||
|
run_timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
log_path = setup_logging(PROJECT_ROOT)
|
||||||
|
_agent_log(
|
||||||
|
"A",
|
||||||
|
"src/planner_download.py:run_download_workflow",
|
||||||
|
"Run started",
|
||||||
|
{
|
||||||
|
"limit": facility_limit,
|
||||||
|
"facility": facility_name or "",
|
||||||
|
"resume": bool(resume),
|
||||||
|
"keep_browser": bool(keep_browser),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
_log.info("=" * 66)
|
||||||
|
_log.info(" MISF OneDrive / Planner Download — Start")
|
||||||
|
_log.info(" Run date: %s", TODAY_ISO_DATE)
|
||||||
|
_log.info(" Log file: %s", log_path)
|
||||||
|
_log.info("=" * 66)
|
||||||
|
log_step("Run started", date=TODAY_ISO_DATE, log=str(log_path))
|
||||||
|
|
||||||
|
driver = None
|
||||||
|
checkpoint_path = PROJECT_ROOT / "Output" / TODAY_ISO_DATE / "checkpoint.json"
|
||||||
|
|
||||||
|
try:
|
||||||
|
_validate_deployment_layout()
|
||||||
|
cfg: AppConfig = read_app_config()
|
||||||
|
log_step("Config loaded", facilities="from input.xlsx", planner_url=(cfg.planner_url or "")[:120])
|
||||||
|
_agent_log(
|
||||||
|
"B",
|
||||||
|
"src/planner_download.py:run_download_workflow",
|
||||||
|
"Config loaded",
|
||||||
|
{
|
||||||
|
"planner_url_prefix": (cfg.planner_url or "")[:120],
|
||||||
|
"proxy_set": bool(cfg.proxy_ip and cfg.proxy_port),
|
||||||
|
"pdf_folder_set": bool(cfg.pdf_folder_path.strip()),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
missing = missing_onedrive_credentials(
|
||||||
|
cfg.onedrive_username,
|
||||||
|
cfg.onedrive_password,
|
||||||
|
cfg.microsoft_totp_secret,
|
||||||
|
)
|
||||||
|
if missing:
|
||||||
|
raise ValueError(credentials_setup_message(missing))
|
||||||
|
|
||||||
|
validate_totp_secret(cfg.microsoft_totp_secret)
|
||||||
|
|
||||||
|
configure_http_download_proxy(
|
||||||
|
cfg.proxy_ip,
|
||||||
|
cfg.proxy_port,
|
||||||
|
cfg.proxy_username,
|
||||||
|
str(cfg.proxy_password) if cfg.proxy_password else None,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Always download into Output/<date>/Original/<facility>/ to keep originals.
|
||||||
|
output_root = PROJECT_ROOT / "Output"
|
||||||
|
removed_shots = _remove_screenshots_under_output(output_root)
|
||||||
|
if removed_shots:
|
||||||
|
log_step("Removed legacy screenshot folders", count=removed_shots)
|
||||||
|
|
||||||
|
original_root = output_root / TODAY_ISO_DATE / "Original"
|
||||||
|
failure_log_dir = output_root / TODAY_ISO_DATE / "logs" / "failures"
|
||||||
|
set_default_download_directory(original_root, create=False)
|
||||||
|
|
||||||
|
facilities = read_facility_list()
|
||||||
|
facilities = _filter_facilities(
|
||||||
|
facilities,
|
||||||
|
facility_limit=facility_limit,
|
||||||
|
facility_name=facility_name,
|
||||||
|
)
|
||||||
|
|
||||||
|
start_index = 0
|
||||||
|
completed: set[str] = set()
|
||||||
|
if resume:
|
||||||
|
ckpt = load_checkpoint(checkpoint_path)
|
||||||
|
start_index = int(ckpt.get("last_completed_index", 0) or 0)
|
||||||
|
completed = {str(x) for x in (ckpt.get("completed_facilities", []) or [])}
|
||||||
|
if start_index > 0 or completed:
|
||||||
|
_log.info(
|
||||||
|
"Resuming after facility index %d (%d remaining); %d completed cached",
|
||||||
|
start_index,
|
||||||
|
max(0, len(facilities) - start_index),
|
||||||
|
len(completed),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
clear_checkpoint(checkpoint_path)
|
||||||
|
|
||||||
|
if start_index >= len(facilities):
|
||||||
|
_log.warning("Checkpoint past end of facility list — starting from beginning")
|
||||||
|
start_index = 0
|
||||||
|
clear_checkpoint(checkpoint_path)
|
||||||
|
|
||||||
|
log_step("Start browser (Chromium remote debug)")
|
||||||
|
_log.info("Starting Chromium (Chrome-compatible)...")
|
||||||
|
if not start_chrome_debug(
|
||||||
|
cfg.proxy_ip,
|
||||||
|
cfg.proxy_port,
|
||||||
|
cfg.proxy_username,
|
||||||
|
cfg.proxy_password,
|
||||||
|
):
|
||||||
|
_agent_log(
|
||||||
|
"C",
|
||||||
|
"src/planner_download.py:run_download_workflow",
|
||||||
|
"Browser failed to start",
|
||||||
|
{},
|
||||||
|
)
|
||||||
|
raise RuntimeError("Browser failed to start.")
|
||||||
|
_agent_log(
|
||||||
|
"C",
|
||||||
|
"src/planner_download.py:run_download_workflow",
|
||||||
|
"Browser started",
|
||||||
|
{},
|
||||||
|
)
|
||||||
|
|
||||||
|
driver = attach_browser(create_download_dir=False)
|
||||||
|
if driver is None:
|
||||||
|
_agent_log(
|
||||||
|
"C",
|
||||||
|
"src/planner_download.py:run_download_workflow",
|
||||||
|
"Browser attach failed",
|
||||||
|
{},
|
||||||
|
)
|
||||||
|
raise RuntimeError("Browser attach failed.")
|
||||||
|
log_step("Browser attached")
|
||||||
|
_log.info("Browser attached.")
|
||||||
|
consolidate_browser_to_single_tab(driver)
|
||||||
|
|
||||||
|
if cfg.proxy_username and cfg.proxy_password:
|
||||||
|
log_step("Dismiss proxy auth popup")
|
||||||
|
dismiss_proxy_auth_popup(
|
||||||
|
driver,
|
||||||
|
cfg.proxy_username.strip(),
|
||||||
|
str(cfg.proxy_password),
|
||||||
|
wait_secs=8.0,
|
||||||
|
max_attempts=3,
|
||||||
|
)
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
log_step("Login and open Planner")
|
||||||
|
login_and_open_planner(
|
||||||
|
driver,
|
||||||
|
cfg.onedrive_username,
|
||||||
|
cfg.onedrive_password,
|
||||||
|
cfg.microsoft_totp_secret,
|
||||||
|
planner_url=cfg.planner_url,
|
||||||
|
)
|
||||||
|
log_url(driver, "after_login")
|
||||||
|
_close_blank_tabs(driver)
|
||||||
|
_agent_log(
|
||||||
|
"D",
|
||||||
|
"src/planner_download.py:run_download_workflow",
|
||||||
|
"Planner login returned",
|
||||||
|
{"url": (driver.current_url or "")[:160]},
|
||||||
|
)
|
||||||
|
|
||||||
|
results: dict[str, dict[str, object]] = {}
|
||||||
|
failures: list[tuple[str, str, str]] = []
|
||||||
|
total_files = 0
|
||||||
|
|
||||||
|
for idx, fac in enumerate(facilities, start=1):
|
||||||
|
if idx <= start_index:
|
||||||
|
continue
|
||||||
|
if fac in completed:
|
||||||
|
_log.info("Skipping already completed facility: %s", fac)
|
||||||
|
continue
|
||||||
|
|
||||||
|
log_step("Facility loop", index=f"{idx}/{len(facilities)}", facility=fac)
|
||||||
|
_log.info("Facility %d/%d: %s", idx, len(facilities), fac)
|
||||||
|
dest = original_root / _sanitize_folder_name(fac)
|
||||||
|
_agent_log(
|
||||||
|
"E",
|
||||||
|
"src/planner_download.py:facility_loop",
|
||||||
|
"Facility start",
|
||||||
|
{"index": idx, "facility": fac[:80]},
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
facility_result: FacilityRunResult = process_facility(
|
||||||
|
driver,
|
||||||
|
fac,
|
||||||
|
dest,
|
||||||
|
screenshot_on_error=screenshot_on_error,
|
||||||
|
failure_log_dir=failure_log_dir if screenshot_on_error else None,
|
||||||
|
)
|
||||||
|
original_paths = facility_result.downloaded
|
||||||
|
file_count = len(original_paths)
|
||||||
|
comment = ""
|
||||||
|
if original_paths:
|
||||||
|
workable_paths, workable_root = _build_workable_files(
|
||||||
|
facility_name=fac,
|
||||||
|
downloaded=original_paths,
|
||||||
|
output_root=output_root,
|
||||||
|
today_iso=TODAY_ISO_DATE,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
workable_paths = []
|
||||||
|
workable_root = output_root / TODAY_ISO_DATE / "Workable"
|
||||||
|
comment = "0 files downloaded"
|
||||||
|
_remove_empty_directory(dest)
|
||||||
|
|
||||||
|
results[fac] = {
|
||||||
|
"original": original_paths,
|
||||||
|
"workable": workable_paths,
|
||||||
|
"comment": comment,
|
||||||
|
"tasks_total": facility_result.tasks_total,
|
||||||
|
"tasks_with_downloads": facility_result.tasks_with_downloads,
|
||||||
|
"tasks_with_attachments": facility_result.tasks_with_attachments,
|
||||||
|
"tasks_skipped_no_attachment": facility_result.tasks_skipped_no_attachment,
|
||||||
|
"status_changed": facility_result.status_changed,
|
||||||
|
"status_via_task_details": facility_result.status_via_task_details,
|
||||||
|
}
|
||||||
|
total_files += file_count
|
||||||
|
log_step(
|
||||||
|
"Facility complete",
|
||||||
|
facility=fac,
|
||||||
|
files=file_count,
|
||||||
|
tasks_with_downloads=str(facility_result.tasks_with_downloads),
|
||||||
|
status_changed=f"{facility_result.status_changed}/{facility_result.tasks_with_downloads}",
|
||||||
|
via_task_details=str(facility_result.status_via_task_details),
|
||||||
|
original=str(dest),
|
||||||
|
workable=str(workable_root),
|
||||||
|
)
|
||||||
|
_log.info(
|
||||||
|
"%s: %d original file(s), status %d/%d tasks (%d via task details) -> %s ; workable -> %s",
|
||||||
|
fac,
|
||||||
|
file_count,
|
||||||
|
facility_result.status_changed,
|
||||||
|
facility_result.tasks_with_downloads,
|
||||||
|
facility_result.status_via_task_details,
|
||||||
|
dest,
|
||||||
|
workable_root,
|
||||||
|
)
|
||||||
|
completed.add(fac)
|
||||||
|
log_step("Checkpoint saved", facility=fac, index=str(idx))
|
||||||
|
save_checkpoint(
|
||||||
|
checkpoint_path,
|
||||||
|
last_completed_index=idx,
|
||||||
|
facility_name=fac,
|
||||||
|
completed_facilities=sorted(completed),
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
err_msg = str(exc)
|
||||||
|
status = _failure_status(exc)
|
||||||
|
log_step("Facility failed", facility=fac, error=err_msg[:200], status=status)
|
||||||
|
_log.error("%s: %s", fac, err_msg)
|
||||||
|
if screenshot_on_error and driver is not None:
|
||||||
|
save_failure_screenshot(driver, fac, failure_log_dir=failure_log_dir)
|
||||||
|
failures.append((fac, err_msg, status))
|
||||||
|
results[fac] = {
|
||||||
|
"original": [],
|
||||||
|
"workable": [],
|
||||||
|
"comment": err_msg[:500],
|
||||||
|
"tasks_total": 0,
|
||||||
|
"tasks_with_downloads": 0,
|
||||||
|
"tasks_with_attachments": 0,
|
||||||
|
"tasks_skipped_no_attachment": 0,
|
||||||
|
"status_changed": 0,
|
||||||
|
"status_via_task_details": 0,
|
||||||
|
}
|
||||||
|
save_checkpoint(
|
||||||
|
checkpoint_path,
|
||||||
|
last_completed_index=idx,
|
||||||
|
facility_name=fac,
|
||||||
|
completed_facilities=sorted(completed),
|
||||||
|
)
|
||||||
|
|
||||||
|
summary_path = (
|
||||||
|
PROJECT_ROOT / "Output" / TODAY_ISO_DATE / "Summary"
|
||||||
|
/ f"Download_Summary_{TODAY_ISO_DATE}.xlsx"
|
||||||
|
)
|
||||||
|
_write_download_summary(
|
||||||
|
results,
|
||||||
|
failures,
|
||||||
|
summary_path,
|
||||||
|
planner_url=cfg.planner_url,
|
||||||
|
run_timestamp=run_timestamp,
|
||||||
|
)
|
||||||
|
|
||||||
|
_prune_empty_date_output_dirs(output_root, TODAY_ISO_DATE)
|
||||||
|
clear_checkpoint(checkpoint_path)
|
||||||
|
|
||||||
|
log_step("Run complete", files=total_files, failures=len(failures), summary=str(summary_path))
|
||||||
|
_log.info("=" * 66)
|
||||||
|
_log.info(
|
||||||
|
" DOWNLOAD COMPLETE — %d file(s), %d failure(s)",
|
||||||
|
total_files,
|
||||||
|
len(failures),
|
||||||
|
)
|
||||||
|
_log.info("=" * 66)
|
||||||
|
|
||||||
|
except Exception as exc:
|
||||||
|
_log.critical("Run failed: %s", exc)
|
||||||
|
traceback.print_exc()
|
||||||
|
|
||||||
|
err_lower = str(exc).lower()
|
||||||
|
is_proxy_error = any(
|
||||||
|
kw in err_lower
|
||||||
|
for kw in ("proxy", "407", "auth", "intercepted", "credential", "popup")
|
||||||
|
)
|
||||||
|
|
||||||
|
if is_proxy_error or keep_browser:
|
||||||
|
user_msg = (
|
||||||
|
"An error occurred during OneDrive/Planner download.\n\n"
|
||||||
|
"The browser has been left open for inspection.\n\n"
|
||||||
|
f"Error:\n{str(exc)[:300]}"
|
||||||
|
)
|
||||||
|
dialog_title = "Download Error — Browser Left Open"
|
||||||
|
else:
|
||||||
|
user_msg = (
|
||||||
|
"An error occurred during OneDrive/Planner download.\n\n"
|
||||||
|
f"Error:\n{str(exc)[:300]}\n\n"
|
||||||
|
f"Details: Output\\{TODAY_ISO_DATE}\\logs\\download.log"
|
||||||
|
)
|
||||||
|
dialog_title = "Download Error"
|
||||||
|
|
||||||
|
_show_error_dialog(dialog_title, user_msg)
|
||||||
|
_log.info("PROCESS ENDED WITH ERROR")
|
||||||
|
_quit_browser(driver, keep_browser=keep_browser, is_proxy_error=is_proxy_error)
|
||||||
|
return
|
||||||
|
|
||||||
|
if not keep_browser:
|
||||||
|
if driver is not None:
|
||||||
|
_log.info("Closing browser...")
|
||||||
|
_quit_browser(driver, keep_browser=False)
|
||||||
|
driver = None
|
||||||
|
else:
|
||||||
|
_log.info("Browser left open (--keep-browser)")
|
||||||
|
_log.info("PROCESS ENDED")
|
||||||
|
|
||||||
|
|
||||||
|
def build_arg_parser() -> argparse.ArgumentParser:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Download face-sheet PDFs from OneDrive / Planner.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--limit",
|
||||||
|
type=int,
|
||||||
|
default=0,
|
||||||
|
help="Process only the first N facilities (e.g. --limit 2 for a dry-run).",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--facility",
|
||||||
|
type=str,
|
||||||
|
default="",
|
||||||
|
help="Process a single facility by exact name (case-insensitive match).",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--resume",
|
||||||
|
action="store_true",
|
||||||
|
help="Resume from last completed facility (checkpoint file).",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--keep-browser",
|
||||||
|
action="store_true",
|
||||||
|
help="Do not close the browser when the run finishes.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--screenshot",
|
||||||
|
action="store_true",
|
||||||
|
help="Save failure screenshots to Output/<date>/logs/failures/ only.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--no-screenshot",
|
||||||
|
action="store_true",
|
||||||
|
help=argparse.SUPPRESS,
|
||||||
|
)
|
||||||
|
return parser
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
"""Step-by-step run logging for MISF automation (every major UI action)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
_RUN = logging.getLogger("misf.run")
|
||||||
|
|
||||||
|
|
||||||
|
def log_step(message: str, **fields: Any) -> None:
|
||||||
|
"""
|
||||||
|
Log one automation step. Use for every meaningful UI/browser action.
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
log_step("Open My plans -> Shared")
|
||||||
|
log_step("Filter facility", facility="Amaya", rows=2)
|
||||||
|
"""
|
||||||
|
extras = []
|
||||||
|
for key, value in fields.items():
|
||||||
|
if value is None:
|
||||||
|
continue
|
||||||
|
text = str(value).strip()
|
||||||
|
if not text:
|
||||||
|
continue
|
||||||
|
# Avoid huge URLs in logs
|
||||||
|
if len(text) > 240:
|
||||||
|
text = text[:237] + "..."
|
||||||
|
extras.append(f"{key}={text}")
|
||||||
|
suffix = f" | {' | '.join(extras)}" if extras else ""
|
||||||
|
_RUN.info("[STEP] %s%s", message, suffix)
|
||||||
|
|
||||||
|
|
||||||
|
def log_url(driver, label: str = "current") -> None:
|
||||||
|
"""Log current browser URL when driver is available."""
|
||||||
|
try:
|
||||||
|
url = (driver.current_url or "")[:240]
|
||||||
|
if url:
|
||||||
|
log_step(f"URL ({label})", url=url)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
"""Microsoft Authenticator TOTP secret sanitization and validation."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
|
||||||
|
_BASE32_CHARS = frozenset("ABCDEFGHIJKLMNOPQRSTUVWXYZ234567")
|
||||||
|
|
||||||
|
|
||||||
|
def sanitize_totp_secret(raw: str) -> str:
|
||||||
|
"""Normalize Base32 secret: strip separators, uppercase, map 0→O and 1→I."""
|
||||||
|
secret = re.sub(r"[\s\-]", "", (raw or "").strip()).upper().rstrip("=")
|
||||||
|
return secret.translate(str.maketrans({"0": "O", "1": "I"}))
|
||||||
|
|
||||||
|
|
||||||
|
def invalid_totp_characters(secret: str) -> list[str]:
|
||||||
|
return sorted({c for c in secret if c not in _BASE32_CHARS})
|
||||||
|
|
||||||
|
|
||||||
|
def validate_totp_secret(raw: str) -> str:
|
||||||
|
"""
|
||||||
|
Return sanitized secret or raise ValueError with a clear message.
|
||||||
|
"""
|
||||||
|
sanitized = sanitize_totp_secret(raw)
|
||||||
|
if not sanitized:
|
||||||
|
raise ValueError("Microsoft TOTP Secret is empty (check input.xlsx or .env).")
|
||||||
|
invalid = invalid_totp_characters(sanitized)
|
||||||
|
if invalid:
|
||||||
|
raise ValueError(
|
||||||
|
"Microsoft TOTP Secret is not valid Base32. "
|
||||||
|
f"Invalid character(s): {', '.join(invalid[:10])}. "
|
||||||
|
"Fix the secret (Base32 uses A-Z and 2-7 only)."
|
||||||
|
)
|
||||||
|
return sanitized
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
"""Tests for multi-attachment URL collection."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from onedrive_planner import (
|
||||||
|
_attachment_tab_count,
|
||||||
|
_merge_attachment_hrefs,
|
||||||
|
_should_mark_task_in_progress,
|
||||||
|
)
|
||||||
|
|
||||||
|
SAMPLE_A = (
|
||||||
|
"https://netorg393890.sharepoint.com/sites/ArborHills/Shared%20Documents/"
|
||||||
|
"RICHARDSON%20ARBRHLS%205.26.26.pdf"
|
||||||
|
)
|
||||||
|
SAMPLE_B = (
|
||||||
|
"https://netorg393890.sharepoint.com/sites/ArborHills/Shared%20Documents/"
|
||||||
|
"RICHARDSON%20ARBRHLS%206.4.26.pdf"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestAttachmentMulti:
|
||||||
|
def test_merge_keeps_distinct_pdf_urls(self):
|
||||||
|
merged = _merge_attachment_hrefs(
|
||||||
|
[SAMPLE_A],
|
||||||
|
[SAMPLE_B],
|
||||||
|
[SAMPLE_A],
|
||||||
|
)
|
||||||
|
assert len(merged) == 2
|
||||||
|
assert SAMPLE_A in merged
|
||||||
|
assert SAMPLE_B in merged
|
||||||
|
|
||||||
|
def test_merge_dedupes_same_filename(self):
|
||||||
|
merged = _merge_attachment_hrefs([SAMPLE_A], [SAMPLE_A + "?download=1"])
|
||||||
|
assert len(merged) == 1
|
||||||
|
|
||||||
|
def test_mark_status_only_after_all_files(self, tmp_path):
|
||||||
|
one = tmp_path / "a.pdf"
|
||||||
|
two = tmp_path / "b.pdf"
|
||||||
|
one.write_bytes(b"x")
|
||||||
|
two.write_bytes(b"y")
|
||||||
|
assert not _should_mark_task_in_progress([one], 2)
|
||||||
|
assert _should_mark_task_in_progress([one, two], 2)
|
||||||
|
assert not _should_mark_task_in_progress([], 2)
|
||||||
|
assert _should_mark_task_in_progress([one], 0)
|
||||||
|
|
||||||
|
def test_attachment_tab_count_parses_number(self):
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
tab = MagicMock()
|
||||||
|
tab.is_displayed.return_value = True
|
||||||
|
tab.text = "Attachments (2)"
|
||||||
|
tab.get_attribute.return_value = ""
|
||||||
|
driver = MagicMock()
|
||||||
|
driver.find_elements.return_value = [tab]
|
||||||
|
assert _attachment_tab_count(driver) == 2
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
"""Unit tests for SharePoint URL extraction from attachment rows."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
import base64
|
||||||
|
|
||||||
|
from onedrive_planner import (
|
||||||
|
_attachment_filename_key,
|
||||||
|
_direct_download_url,
|
||||||
|
_download_href_via_browser_fetch,
|
||||||
|
_extract_sharepoint_url,
|
||||||
|
_is_pdf_bytes,
|
||||||
|
_resolve_attachment_click_target,
|
||||||
|
_sharepoint_download_candidates,
|
||||||
|
_url_from_attachment_element,
|
||||||
|
_viewer_preview_url,
|
||||||
|
)
|
||||||
|
|
||||||
|
SAMPLE = (
|
||||||
|
"https://netorg393890.sharepoint.com/sites/AnberryNursingandRehabilitationCenter/"
|
||||||
|
"Shared%20Documents/ANBERRY%20FS%205.29.26.pdf"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestAttachmentUrl:
|
||||||
|
def test_extract_from_aria_label(self):
|
||||||
|
assert _extract_sharepoint_url(SAMPLE) == SAMPLE
|
||||||
|
|
||||||
|
def test_direct_download_transform(self):
|
||||||
|
dl = _direct_download_url(SAMPLE)
|
||||||
|
assert "download=1" in dl
|
||||||
|
|
||||||
|
def test_viewer_preview_url(self):
|
||||||
|
viewer = _viewer_preview_url(SAMPLE + "?web=1")
|
||||||
|
assert "web=1" in viewer
|
||||||
|
assert "download=1" not in viewer
|
||||||
|
viewer2 = _viewer_preview_url(SAMPLE)
|
||||||
|
assert "web=1" in viewer2
|
||||||
|
|
||||||
|
def test_url_from_span_element(self):
|
||||||
|
span = MagicMock()
|
||||||
|
span.get_attribute.side_effect = lambda name: (
|
||||||
|
SAMPLE if name == "aria-label" else None
|
||||||
|
)
|
||||||
|
span.text = ""
|
||||||
|
span.find_elements.return_value = []
|
||||||
|
assert _url_from_attachment_element(span) == SAMPLE
|
||||||
|
|
||||||
|
def test_sharepoint_download_candidates_include_layouts(self):
|
||||||
|
cands = _sharepoint_download_candidates(SAMPLE)
|
||||||
|
assert any("download.aspx" in c for c in cands)
|
||||||
|
assert any("download=1" in c for c in cands)
|
||||||
|
|
||||||
|
def test_is_pdf_bytes(self):
|
||||||
|
assert _is_pdf_bytes(b"%PDF-1.4 fake content " + b"x" * 6000)
|
||||||
|
assert not _is_pdf_bytes(b"%PDF-1.4 tiny " + b"x" * 100)
|
||||||
|
assert not _is_pdf_bytes(b"<html>viewer</html>")
|
||||||
|
|
||||||
|
def test_resolve_click_target_prefers_href_span(self):
|
||||||
|
span = MagicMock()
|
||||||
|
span.is_displayed.return_value = True
|
||||||
|
span.tag_name = "span"
|
||||||
|
span.get_attribute.side_effect = lambda name: (
|
||||||
|
f"JACKSON ARBRHLS 6.3.26.pdf {SAMPLE}" if name == "aria-label" else None
|
||||||
|
)
|
||||||
|
span.text = ""
|
||||||
|
span.find_elements.return_value = []
|
||||||
|
|
||||||
|
li = MagicMock()
|
||||||
|
li.tag_name = "li"
|
||||||
|
li.is_displayed.return_value = True
|
||||||
|
li.get_attribute.return_value = None
|
||||||
|
li.text = "JACKSON ARBRHLS 6.3.26.pdf"
|
||||||
|
li.find_elements.return_value = [span]
|
||||||
|
|
||||||
|
assert _resolve_attachment_click_target(li) is span
|
||||||
|
assert "6.3.26.pdf" in _attachment_filename_key(span)
|
||||||
|
|
||||||
|
def test_browser_fetch_saves_pdf(self, tmp_path):
|
||||||
|
driver = MagicMock()
|
||||||
|
pdf = b"%PDF-1.4 " + b"x" * 6000
|
||||||
|
driver.execute_async_script.return_value = {
|
||||||
|
"ok": True,
|
||||||
|
"b64": base64.b64encode(pdf).decode("ascii"),
|
||||||
|
"ct": "application/pdf",
|
||||||
|
"size": len(pdf),
|
||||||
|
}
|
||||||
|
driver.current_window_handle = "h1"
|
||||||
|
driver.window_handles = ["h1"]
|
||||||
|
|
||||||
|
def _exec_script(script, *args):
|
||||||
|
if "window.open" in str(script):
|
||||||
|
driver.window_handles = ["h1", "h2"]
|
||||||
|
|
||||||
|
driver.execute_script.side_effect = _exec_script
|
||||||
|
out = _download_href_via_browser_fetch(driver, tmp_path, SAMPLE, set())
|
||||||
|
assert len(out) == 1
|
||||||
|
assert out[0].exists()
|
||||||
|
assert _is_pdf_bytes(out[0].read_bytes())
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
"""Tests for board column detection helpers."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
from selenium.common.exceptions import InvalidSelectorException
|
||||||
|
|
||||||
|
from onedrive_planner import (
|
||||||
|
_FIRST_COLUMN_CLUSTER_WIDTH_PX,
|
||||||
|
_element_in_board_container,
|
||||||
|
_parent_element,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_first_column_cluster_width_reasonable():
|
||||||
|
assert _FIRST_COLUMN_CLUSTER_WIDTH_PX >= 150
|
||||||
|
|
||||||
|
|
||||||
|
def test_parent_element_stops_at_html_document_error():
|
||||||
|
"""ChromeDriver rejects .. on <html> with InvalidSelectorException."""
|
||||||
|
html = MagicMock()
|
||||||
|
html.tag_name = "html"
|
||||||
|
html.find_element.side_effect = InvalidSelectorException(
|
||||||
|
'The result of the xpath expression ".." is: [object HTMLDocument].'
|
||||||
|
)
|
||||||
|
assert _parent_element(html) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_parent_element_returns_parent_for_body():
|
||||||
|
body = MagicMock()
|
||||||
|
body.tag_name = "body"
|
||||||
|
parent = MagicMock()
|
||||||
|
parent.tag_name = "html"
|
||||||
|
body.find_element.return_value = parent
|
||||||
|
assert _parent_element(body) is parent
|
||||||
|
|
||||||
|
|
||||||
|
def test_element_in_board_container_does_not_crash_at_root():
|
||||||
|
"""Containment walk must not raise when ancestors hit document root."""
|
||||||
|
child = MagicMock()
|
||||||
|
child.tag_name = "div"
|
||||||
|
container = MagicMock()
|
||||||
|
container.tag_name = "div"
|
||||||
|
|
||||||
|
html = MagicMock()
|
||||||
|
html.tag_name = "html"
|
||||||
|
html.find_element.side_effect = InvalidSelectorException("document root")
|
||||||
|
|
||||||
|
body = MagicMock()
|
||||||
|
body.tag_name = "body"
|
||||||
|
body.find_element.return_value = html
|
||||||
|
|
||||||
|
child.find_element.return_value = body
|
||||||
|
|
||||||
|
assert _element_in_board_container(child, container) is False
|
||||||
@@ -0,0 +1,147 @@
|
|||||||
|
"""Unit tests for Chromium-only browser resolution."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
from chrome import (
|
||||||
|
PROJECT_ROOT,
|
||||||
|
_allow_google_chrome_fallback,
|
||||||
|
_browser_runs_in_background,
|
||||||
|
_bundled_chromedriver_for,
|
||||||
|
_google_chrome_fallback_paths,
|
||||||
|
_installed_chromium_paths,
|
||||||
|
_portable_chromium_executable,
|
||||||
|
_prefer_installed_chromium,
|
||||||
|
capture_fullscreen_screenshot,
|
||||||
|
find_chromium_executable,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_installed_paths_exclude_google():
|
||||||
|
installed = _installed_chromium_paths()
|
||||||
|
chrome_fb = _google_chrome_fallback_paths()
|
||||||
|
assert not any("Google" in p for p in installed)
|
||||||
|
assert any("Google" in p for p in chrome_fb)
|
||||||
|
|
||||||
|
|
||||||
|
def test_portable_chrome_win_preferred_over_installed():
|
||||||
|
portable = str(PROJECT_ROOT / "chrome-win" / "chrome-win" / "chrome.exe")
|
||||||
|
installed = r"C:\Program Files\Chromium\Application\chrome.exe"
|
||||||
|
|
||||||
|
def fake_isfile(path: str) -> bool:
|
||||||
|
return path in {portable, installed}
|
||||||
|
|
||||||
|
env = {"USERPROFILE": r"C:\Users\test", "LOCALAPPDATA": r"C:\Users\test\AppData\Local"}
|
||||||
|
with patch("chrome._portable_chromium_executable", return_value=portable):
|
||||||
|
with patch("chrome.os.path.isfile", side_effect=fake_isfile):
|
||||||
|
with patch.dict("os.environ", env, clear=True):
|
||||||
|
assert find_chromium_executable() == portable
|
||||||
|
|
||||||
|
|
||||||
|
def test_installed_used_when_portable_missing():
|
||||||
|
installed = r"C:\Program Files\Chromium\Application\chrome.exe"
|
||||||
|
|
||||||
|
def fake_isfile(path: str) -> bool:
|
||||||
|
return path == installed
|
||||||
|
|
||||||
|
env = {"USERPROFILE": r"C:\Users\test", "LOCALAPPDATA": r"C:\Users\test\AppData\Local"}
|
||||||
|
with patch("chrome._portable_chromium_executable", return_value=None):
|
||||||
|
with patch("chrome.os.path.isfile", side_effect=fake_isfile):
|
||||||
|
with patch.dict("os.environ", env, clear=True):
|
||||||
|
assert find_chromium_executable() == installed
|
||||||
|
|
||||||
|
|
||||||
|
def test_misf_use_installed_chrome_skips_portable():
|
||||||
|
portable = str(PROJECT_ROOT / "chrome-win" / "chrome-win" / "chrome.exe")
|
||||||
|
installed = r"C:\Program Files\Chromium\Application\chrome.exe"
|
||||||
|
|
||||||
|
def fake_isfile(path: str) -> bool:
|
||||||
|
return path in {portable, installed}
|
||||||
|
|
||||||
|
env = {
|
||||||
|
"MISF_USE_INSTALLED_CHROME": "1",
|
||||||
|
"USERPROFILE": r"C:\Users\test",
|
||||||
|
"LOCALAPPDATA": r"C:\Users\test\AppData\Local",
|
||||||
|
}
|
||||||
|
with patch("chrome._portable_chromium_executable", return_value=portable):
|
||||||
|
with patch("chrome.os.path.isfile", side_effect=fake_isfile):
|
||||||
|
with patch.dict("os.environ", env, clear=True):
|
||||||
|
assert find_chromium_executable() == installed
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_chromium_can_use_google_with_allow_flag():
|
||||||
|
google_path = r"C:\Program Files\Google\Chrome\Application\chrome.exe"
|
||||||
|
|
||||||
|
env = {
|
||||||
|
"MISF_ALLOW_CHROME": "1",
|
||||||
|
"USERPROFILE": r"C:\Users\test",
|
||||||
|
"LOCALAPPDATA": r"C:\Users\test\AppData\Local",
|
||||||
|
}
|
||||||
|
with patch("chrome._portable_chromium_executable", return_value=None):
|
||||||
|
with patch("chrome.os.path.isfile", side_effect=lambda p: p == google_path):
|
||||||
|
with patch.dict("os.environ", env, clear=True):
|
||||||
|
assert find_chromium_executable() == google_path
|
||||||
|
|
||||||
|
|
||||||
|
def test_bundled_chromedriver_next_to_portable_chrome():
|
||||||
|
browser = PROJECT_ROOT / "chrome-win" / "chrome-win" / "chrome.exe"
|
||||||
|
if not browser.is_file():
|
||||||
|
return
|
||||||
|
driver = _bundled_chromedriver_for(str(browser))
|
||||||
|
assert driver is not None
|
||||||
|
assert driver.endswith("chromedriver.exe")
|
||||||
|
|
||||||
|
|
||||||
|
def test_portable_chromium_executable_on_disk():
|
||||||
|
exe = _portable_chromium_executable()
|
||||||
|
expected = PROJECT_ROOT / "chrome-win" / "chrome-win" / "chrome.exe"
|
||||||
|
if expected.is_file():
|
||||||
|
assert exe == str(expected)
|
||||||
|
|
||||||
|
|
||||||
|
def test_foreground_default_background_opt_in():
|
||||||
|
with patch.dict("os.environ", {}, clear=True):
|
||||||
|
assert _browser_runs_in_background() is False
|
||||||
|
with patch.dict("os.environ", {"MISF_BROWSER_BACKGROUND": "1"}, clear=True):
|
||||||
|
assert _browser_runs_in_background() is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_allow_chrome_flag():
|
||||||
|
with patch.dict("os.environ", {}, clear=True):
|
||||||
|
assert _allow_google_chrome_fallback() is False
|
||||||
|
with patch.dict("os.environ", {"MISF_ALLOW_CHROME": "yes"}, clear=True):
|
||||||
|
assert _allow_google_chrome_fallback() is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_prefer_installed_flag():
|
||||||
|
with patch.dict("os.environ", {}, clear=True):
|
||||||
|
assert _prefer_installed_chromium() is False
|
||||||
|
with patch.dict("os.environ", {"MISF_USE_INSTALLED_CHROME": "1"}, clear=True):
|
||||||
|
assert _prefer_installed_chromium() is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_fullscreen_screenshot_maximizes_and_uses_desktop_when_not_black(tmp_path):
|
||||||
|
driver = MagicMock()
|
||||||
|
shot = tmp_path / "shot.png"
|
||||||
|
shot.write_bytes(b"\xff" * 100)
|
||||||
|
|
||||||
|
with patch.dict("os.environ", {}, clear=True):
|
||||||
|
with patch("chrome.capture_desktop_screenshot", return_value=True) as cap:
|
||||||
|
with patch("chrome._is_image_mostly_black", return_value=False):
|
||||||
|
assert capture_fullscreen_screenshot(driver, shot) is True
|
||||||
|
cap.assert_called_once_with(shot)
|
||||||
|
driver.maximize_window.assert_called()
|
||||||
|
|
||||||
|
|
||||||
|
def test_fullscreen_screenshot_falls_back_when_desktop_is_black(tmp_path):
|
||||||
|
driver = MagicMock()
|
||||||
|
shot = tmp_path / "shot.png"
|
||||||
|
shot.write_bytes(b"\x00" * 100)
|
||||||
|
|
||||||
|
with patch.dict("os.environ", {}, clear=True):
|
||||||
|
with patch("chrome.capture_desktop_screenshot", return_value=True):
|
||||||
|
with patch("chrome._is_image_mostly_black", return_value=True):
|
||||||
|
with patch("chrome._capture_browser_with_timestamp_bar", return_value=True) as fb:
|
||||||
|
assert capture_fullscreen_screenshot(driver, shot) is True
|
||||||
|
fb.assert_called_once()
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
"""Config and facility list tests (no browser)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
SRC = ROOT / "src"
|
||||||
|
sys.path.insert(0, str(SRC))
|
||||||
|
|
||||||
|
from config import ( # noqa: E402
|
||||||
|
read_app_config,
|
||||||
|
read_facility_list,
|
||||||
|
read_onedrive_config,
|
||||||
|
)
|
||||||
|
from onedrive_planner import _sanitize_folder_name, _xpath_literal # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def example_config() -> Path:
|
||||||
|
path = ROOT / "Input" / "input.xlsx.example"
|
||||||
|
assert path.exists(), f"Missing template: {path}"
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def test_helpers() -> None:
|
||||||
|
assert _sanitize_folder_name("Test/Facility") == "Test_Facility"
|
||||||
|
assert _xpath_literal("Plain Name") == "'Plain Name'"
|
||||||
|
assert "concat" in _xpath_literal('O\'Brien "East" Wing')
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_facility_list_skips_placeholder(example_config: Path) -> None:
|
||||||
|
facilities = read_facility_list(example_config, warn_if_not_expected_count=False)
|
||||||
|
assert facilities
|
||||||
|
assert not any("REPLACE_WITH" in f for f in facilities)
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_app_config_planner_url(example_config: Path) -> None:
|
||||||
|
cfg = read_app_config(example_config)
|
||||||
|
assert "planner.cloud.microsoft" in cfg.planner_url
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_onedrive_config_tuple_length(example_config: Path) -> None:
|
||||||
|
row = read_onedrive_config(example_config)
|
||||||
|
assert len(row) == 10
|
||||||
|
assert "planner.cloud.microsoft" in row[9]
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
"""Smoke tests for OneDrive download config (no browser)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
SRC = ROOT / "src"
|
||||||
|
sys.path.insert(0, str(SRC))
|
||||||
|
|
||||||
|
from config import CONFIG_FILE, read_app_config, read_facility_list # noqa: E402
|
||||||
|
from onedrive_planner import _sanitize_folder_name, _xpath_literal # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def test_helpers() -> None:
|
||||||
|
assert _sanitize_folder_name("Test/Facility") == "Test_Facility"
|
||||||
|
assert _xpath_literal("Plain Name") == "'Plain Name'"
|
||||||
|
assert _xpath_literal("O'Brien Care") == '"O\'Brien Care"'
|
||||||
|
assert "concat" in _xpath_literal('O\'Brien "East" Wing')
|
||||||
|
print("OK: helper functions")
|
||||||
|
|
||||||
|
|
||||||
|
def test_config_loads() -> None:
|
||||||
|
path = CONFIG_FILE if CONFIG_FILE.exists() else ROOT / "Input" / "input.xlsx.example"
|
||||||
|
assert path.exists(), f"Missing config: {path}"
|
||||||
|
facilities = read_facility_list(path)
|
||||||
|
assert len(facilities) >= 1
|
||||||
|
cfg = read_app_config(path)
|
||||||
|
assert cfg.planner_url
|
||||||
|
print(f"OK: {len(facilities)} facilities from {path.name}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
test_helpers()
|
||||||
|
test_config_loads()
|
||||||
|
print("All smoke tests passed.")
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
"""Tests for download summary Excel columns."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
from planner_download import (
|
||||||
|
_build_workable_files,
|
||||||
|
_format_status_changed,
|
||||||
|
_success_status,
|
||||||
|
_write_download_summary,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_success_status_labels():
|
||||||
|
assert _success_status(0, 0) == "No tasks"
|
||||||
|
assert _success_status(0, 3, tasks_skipped_no_attachment=3) == "No attachments"
|
||||||
|
assert _success_status(1, 3, tasks_with_downloads=1, tasks_with_attachments=1) == "Success"
|
||||||
|
assert _success_status(2, 4, tasks_with_downloads=2, tasks_with_attachments=2) == "Success"
|
||||||
|
assert _success_status(1, 4, tasks_with_downloads=1, tasks_with_attachments=2) == "Partial"
|
||||||
|
assert _success_status(0, 2, tasks_with_attachments=2) == "Download failed"
|
||||||
|
|
||||||
|
|
||||||
|
def test_format_status_changed():
|
||||||
|
assert _format_status_changed(0, 0) == "N/A"
|
||||||
|
assert _format_status_changed(2, 2, via_task_details=2) == "Yes - 2/2 tasks (2 via task details)"
|
||||||
|
assert _format_status_changed(3, 1, via_task_details=1) == "Partial - 1/3 tasks (1 via task details)"
|
||||||
|
assert _format_status_changed(1, 0) == "No - 0/1 tasks"
|
||||||
|
assert _format_status_changed(2, 1, via_task_details=1) == "Partial - 1/2 tasks (1 via task details)"
|
||||||
|
|
||||||
|
|
||||||
|
def test_write_download_summary_columns(tmp_path: Path, monkeypatch):
|
||||||
|
monkeypatch.setattr("planner_download.PROJECT_ROOT", tmp_path)
|
||||||
|
results = {
|
||||||
|
"Alpha": {
|
||||||
|
"original": [tmp_path / "a.pdf", tmp_path / "b.pdf"],
|
||||||
|
"tasks_total": 4,
|
||||||
|
"tasks_with_attachments": 2,
|
||||||
|
"tasks_skipped_no_attachment": 2,
|
||||||
|
"tasks_with_downloads": 2,
|
||||||
|
"status_changed": 1,
|
||||||
|
"status_via_task_details": 1,
|
||||||
|
},
|
||||||
|
"Beta": {
|
||||||
|
"original": [],
|
||||||
|
"tasks_total": 2,
|
||||||
|
"tasks_with_attachments": 0,
|
||||||
|
"tasks_skipped_no_attachment": 2,
|
||||||
|
"tasks_with_downloads": 0,
|
||||||
|
"status_changed": 0,
|
||||||
|
"status_via_task_details": 0,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
failures = [("Gamma", "not found", "Plan not found")]
|
||||||
|
summary_path = tmp_path / "Summary" / "Download_Summary_2026-06-05.xlsx"
|
||||||
|
_write_download_summary(
|
||||||
|
results,
|
||||||
|
failures,
|
||||||
|
summary_path,
|
||||||
|
planner_url="https://example.com",
|
||||||
|
run_timestamp="2026-06-05 12:00:00",
|
||||||
|
)
|
||||||
|
df = pd.read_excel(summary_path, keep_default_na=False)
|
||||||
|
assert list(df.columns) == [
|
||||||
|
"Run Timestamp",
|
||||||
|
"Facility Name",
|
||||||
|
"Files Downloaded",
|
||||||
|
"Status",
|
||||||
|
"Status Changed",
|
||||||
|
]
|
||||||
|
alpha = df[df["Facility Name"] == "Alpha"].iloc[0]
|
||||||
|
assert alpha["Status"] == "Success"
|
||||||
|
assert "1/2 tasks" in alpha["Status Changed"]
|
||||||
|
assert "via task details" in alpha["Status Changed"]
|
||||||
|
beta = df[df["Facility Name"] == "Beta"].iloc[0]
|
||||||
|
assert beta["Status"] == "No attachments"
|
||||||
|
assert beta["Status Changed"] == "N/A"
|
||||||
|
gamma = df[df["Facility Name"] == "Gamma"].iloc[0]
|
||||||
|
assert gamma["Status"] == "Plan not found"
|
||||||
|
assert gamma["Status Changed"] == "N/A"
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_workable_files_mirrors_to_medicare(tmp_path: Path):
|
||||||
|
src_dir = tmp_path / "original" / "FacA"
|
||||||
|
src_dir.mkdir(parents=True)
|
||||||
|
pdf = src_dir / "doc.pdf"
|
||||||
|
pdf.write_bytes(b"%PDF-1.4")
|
||||||
|
|
||||||
|
paths, workable_root = _build_workable_files(
|
||||||
|
facility_name="Fac A",
|
||||||
|
downloaded=[pdf],
|
||||||
|
output_root=tmp_path / "Output",
|
||||||
|
today_iso="2026-06-16",
|
||||||
|
)
|
||||||
|
medicare_root = tmp_path / "Output" / "2026-06-16" / "Workable_Medicare"
|
||||||
|
|
||||||
|
assert len(paths) == 1
|
||||||
|
assert paths[0].parent == workable_root
|
||||||
|
assert paths[0].is_file()
|
||||||
|
assert medicare_root.is_dir()
|
||||||
|
medicare_files = list(medicare_root.glob("*.pdf"))
|
||||||
|
assert len(medicare_files) == 1
|
||||||
|
assert medicare_files[0].name == paths[0].name
|
||||||
|
assert medicare_files[0].read_bytes() == pdf.read_bytes()
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
"""Unit tests for Shared plans facility search helpers."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from onedrive_planner import (
|
||||||
|
FACILITY_NOT_APPEAR_MSG,
|
||||||
|
_facility_search_terms,
|
||||||
|
_row_matches_facility_search,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_facility_search_terms_full_then_short():
|
||||||
|
terms = _facility_search_terms("Golden Hill Post Acute Care")
|
||||||
|
assert terms == [
|
||||||
|
"Golden Hill Post Acute Care",
|
||||||
|
"Golden Hill Post",
|
||||||
|
"Golden Hill",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_facility_search_terms_two_words_only():
|
||||||
|
terms = _facility_search_terms("Arbor Hills")
|
||||||
|
assert terms == ["Arbor Hills"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_facility_search_terms_dedupes():
|
||||||
|
terms = _facility_search_terms("Golden Hill")
|
||||||
|
assert terms == ["Golden Hill"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_row_matches_full_name_in_plan_title():
|
||||||
|
assert _row_matches_facility_search(
|
||||||
|
"Golden Hill Subacute and Rehab",
|
||||||
|
"Golden Hill Post Acute Care",
|
||||||
|
"Golden Hill",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_row_matches_exact():
|
||||||
|
assert _row_matches_facility_search(
|
||||||
|
"Arbor Hills",
|
||||||
|
"Arbor Hills",
|
||||||
|
"Arbor Hills",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_row_matches_rejects_unrelated():
|
||||||
|
assert not _row_matches_facility_search(
|
||||||
|
"Sunrise Manor",
|
||||||
|
"Golden Hill Post Acute Care",
|
||||||
|
"Golden Hill",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_not_appear_message_constant():
|
||||||
|
assert "shared plans" in FACILITY_NOT_APPEAR_MSG.lower()
|
||||||
@@ -0,0 +1,202 @@
|
|||||||
|
"""Unit tests for Planner Grid view helpers."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
from selenium.webdriver.common.by import By
|
||||||
|
|
||||||
|
from onedrive_planner import (
|
||||||
|
_clean_grid_task_title,
|
||||||
|
_grid_row_status,
|
||||||
|
_grid_row_task_name,
|
||||||
|
_is_attachments_empty,
|
||||||
|
_is_panel_placeholder_text,
|
||||||
|
_is_status_sorted_descending,
|
||||||
|
_parse_status_blob,
|
||||||
|
_sort_descending_menu_selected,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _grid_row(*, status_value: str = "", status_text: str = "", task_name: str = ""):
|
||||||
|
row = MagicMock()
|
||||||
|
row.is_displayed.return_value = True
|
||||||
|
row.text = task_name
|
||||||
|
|
||||||
|
status_btn = MagicMock()
|
||||||
|
status_btn.is_displayed.return_value = True
|
||||||
|
status_btn.get_attribute.side_effect = lambda name: (
|
||||||
|
status_value if name == "value" else ""
|
||||||
|
)
|
||||||
|
status_btn.text = status_text
|
||||||
|
|
||||||
|
name_cell = MagicMock()
|
||||||
|
name_cell.is_displayed.return_value = True
|
||||||
|
name_cell.get_attribute.return_value = f"Task Name {task_name}"
|
||||||
|
name_span = MagicMock()
|
||||||
|
name_span.is_displayed.return_value = True
|
||||||
|
name_span.text = task_name
|
||||||
|
|
||||||
|
def find_elements(by, xpath):
|
||||||
|
xp = xpath or ""
|
||||||
|
if "combobox" in xp:
|
||||||
|
return [status_btn]
|
||||||
|
if "gridcell" in xp and "span" not in xp:
|
||||||
|
return [name_cell]
|
||||||
|
if "span" in xp:
|
||||||
|
return [name_span]
|
||||||
|
return []
|
||||||
|
|
||||||
|
row.find_elements.side_effect = find_elements
|
||||||
|
name_cell.find_elements.side_effect = lambda by, xpath: [name_span]
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def test_grid_row_status_not_started():
|
||||||
|
row = _grid_row(status_value="notStarted", status_text="Not started")
|
||||||
|
assert _grid_row_status(row) == "not started"
|
||||||
|
|
||||||
|
|
||||||
|
def test_grid_row_status_reads_status_column_not_bucket():
|
||||||
|
row = MagicMock()
|
||||||
|
row.is_displayed.return_value = True
|
||||||
|
|
||||||
|
bucket_btn = MagicMock()
|
||||||
|
bucket_btn.is_displayed.return_value = True
|
||||||
|
bucket_btn.get_attribute.side_effect = lambda name: "todo" if name == "value" else ""
|
||||||
|
bucket_btn.text = "To do"
|
||||||
|
|
||||||
|
status_btn = MagicMock()
|
||||||
|
status_btn.is_displayed.return_value = True
|
||||||
|
status_btn.get_attribute.side_effect = lambda name: "notStarted" if name == "value" else ""
|
||||||
|
status_btn.text = "Not started"
|
||||||
|
|
||||||
|
status_cell = MagicMock()
|
||||||
|
status_cell.is_displayed.return_value = True
|
||||||
|
status_cell.get_attribute.return_value = "Status Not started"
|
||||||
|
status_cell.text = "Not started"
|
||||||
|
status_cell.find_elements.side_effect = lambda by, xpath: (
|
||||||
|
[status_btn] if "combobox" in xpath else []
|
||||||
|
)
|
||||||
|
|
||||||
|
bucket_cell = MagicMock()
|
||||||
|
bucket_cell.is_displayed.return_value = True
|
||||||
|
bucket_cell.get_attribute.return_value = "Bucket To do"
|
||||||
|
bucket_cell.text = "To do"
|
||||||
|
bucket_cell.find_elements.side_effect = lambda by, xpath: (
|
||||||
|
[bucket_btn] if "combobox" in xpath else []
|
||||||
|
)
|
||||||
|
|
||||||
|
def find_elements(by, xpath):
|
||||||
|
xp = xpath or ""
|
||||||
|
if "gridcell" in xp:
|
||||||
|
return [status_cell, bucket_cell]
|
||||||
|
if "combobox" in xp:
|
||||||
|
return [bucket_btn, status_btn]
|
||||||
|
return []
|
||||||
|
|
||||||
|
row.find_elements.side_effect = find_elements
|
||||||
|
assert _grid_row_status(row) == "not started"
|
||||||
|
|
||||||
|
|
||||||
|
def test_grid_row_status_in_progress():
|
||||||
|
row = _grid_row(status_value="inProgress", status_text="In progress")
|
||||||
|
assert _grid_row_status(row) == "in progress"
|
||||||
|
|
||||||
|
|
||||||
|
def test_grid_row_task_name_from_aria():
|
||||||
|
row = _grid_row(task_name="Missing Insurance Info")
|
||||||
|
assert _grid_row_task_name(row) == "Missing Insurance Info"
|
||||||
|
|
||||||
|
|
||||||
|
def test_clean_grid_task_title_strips_a11y_hint():
|
||||||
|
raw = "Missing Insurance Info. Press Alt + I to open task details, or pre"
|
||||||
|
assert _clean_grid_task_title(raw) == "Missing Insurance Info"
|
||||||
|
|
||||||
|
|
||||||
|
def test_grid_row_task_name_strips_a11y_from_span():
|
||||||
|
row = MagicMock()
|
||||||
|
row.is_displayed.return_value = True
|
||||||
|
row.text = ""
|
||||||
|
|
||||||
|
name_cell = MagicMock()
|
||||||
|
name_cell.is_displayed.return_value = True
|
||||||
|
name_cell.get_attribute.return_value = ""
|
||||||
|
name_span = MagicMock()
|
||||||
|
name_span.is_displayed.return_value = True
|
||||||
|
name_span.text = "6.10 priority :). Press Alt + I to open task details"
|
||||||
|
|
||||||
|
def find_elements(by, xpath):
|
||||||
|
xp = xpath or ""
|
||||||
|
if "combobox" in xp:
|
||||||
|
return []
|
||||||
|
if "gridcell" in xp and "span" not in xp:
|
||||||
|
return [name_cell]
|
||||||
|
if "span" in xp:
|
||||||
|
return [name_span]
|
||||||
|
return []
|
||||||
|
|
||||||
|
row.find_elements.side_effect = find_elements
|
||||||
|
name_cell.find_elements.side_effect = lambda by, xpath: [name_span]
|
||||||
|
assert _grid_row_task_name(row) == "6.10 priority :)"
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_panel_placeholder_text():
|
||||||
|
assert _is_panel_placeholder_text("Type a description or add notes here")
|
||||||
|
assert not _is_panel_placeholder_text("Missing Insurance Info")
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_status_sorted_descending():
|
||||||
|
header = MagicMock()
|
||||||
|
header.get_attribute.side_effect = lambda name: (
|
||||||
|
"descending" if name == "aria-sort" else ""
|
||||||
|
)
|
||||||
|
assert _is_status_sorted_descending(header) is True
|
||||||
|
|
||||||
|
header.get_attribute.side_effect = lambda name: (
|
||||||
|
"ascending" if name == "aria-sort" else ""
|
||||||
|
)
|
||||||
|
assert _is_status_sorted_descending(header) is False
|
||||||
|
|
||||||
|
header.get_attribute.side_effect = lambda name: "" if name == "aria-sort" else ""
|
||||||
|
assert _is_status_sorted_descending(header) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_sort_descending_menu_selected():
|
||||||
|
driver = MagicMock()
|
||||||
|
checked_btn = MagicMock()
|
||||||
|
checked_btn.is_displayed.return_value = True
|
||||||
|
|
||||||
|
driver.find_elements.side_effect = lambda by, xpath: (
|
||||||
|
[checked_btn]
|
||||||
|
if "aria-checked='true'" in xpath
|
||||||
|
else []
|
||||||
|
)
|
||||||
|
assert _sort_descending_menu_selected(driver) is True
|
||||||
|
|
||||||
|
driver.find_elements.side_effect = lambda by, xpath: []
|
||||||
|
assert _sort_descending_menu_selected(driver) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_status_blob():
|
||||||
|
assert _parse_status_blob("Not started") == "not started"
|
||||||
|
assert _parse_status_blob("Status Completed") == "completed"
|
||||||
|
assert _parse_status_blob("To do") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_attachments_empty_detects_empty_panel():
|
||||||
|
driver = MagicMock()
|
||||||
|
panel = MagicMock()
|
||||||
|
panel.is_displayed.return_value = True
|
||||||
|
panel.text = "Select, upload, or add a link to a file or website\nAdd attachment"
|
||||||
|
driver.find_elements.return_value = [panel]
|
||||||
|
|
||||||
|
with patch("onedrive_planner._attachment_tab_count", return_value=0), patch(
|
||||||
|
"onedrive_planner._discover_listed_attachment_names", return_value=[]
|
||||||
|
), patch("onedrive_planner._list_attachment_rows", return_value=[]):
|
||||||
|
assert _is_attachments_empty(driver) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_attachments_empty_false_when_tab_count():
|
||||||
|
driver = MagicMock()
|
||||||
|
with patch("onedrive_planner._attachment_tab_count", return_value=2):
|
||||||
|
assert _is_attachments_empty(driver) is False
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
"""Unit tests for Microsoft login page detection."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
from selenium.webdriver.common.by import By
|
||||||
|
|
||||||
|
from onedrive_planner import (
|
||||||
|
_email_form_visible,
|
||||||
|
_needs_microsoft_login,
|
||||||
|
_password_form_visible,
|
||||||
|
_planner_session_ready,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _driver(*, url: str = "", elements: dict | None = None):
|
||||||
|
drv = MagicMock()
|
||||||
|
drv.current_url = url
|
||||||
|
elements = elements or {}
|
||||||
|
|
||||||
|
def find_elements(by, sel):
|
||||||
|
key = (by, sel)
|
||||||
|
return elements.get(key, [])
|
||||||
|
|
||||||
|
drv.find_elements.side_effect = find_elements
|
||||||
|
return drv
|
||||||
|
|
||||||
|
|
||||||
|
def _input(displayed: bool = True):
|
||||||
|
el = MagicMock()
|
||||||
|
el.is_displayed.return_value = displayed
|
||||||
|
return el
|
||||||
|
|
||||||
|
|
||||||
|
class TestLoginHelpers:
|
||||||
|
def test_password_only_login_detected(self):
|
||||||
|
drv = _driver(
|
||||||
|
url="https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
|
||||||
|
elements={
|
||||||
|
(By.CSS_SELECTOR, "input[type='password']"): [_input()],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert _password_form_visible(drv)
|
||||||
|
assert not _email_form_visible(drv)
|
||||||
|
assert _needs_microsoft_login(drv)
|
||||||
|
assert not _planner_session_ready(drv)
|
||||||
|
|
||||||
|
def test_planner_ready_when_no_login_prompt(self):
|
||||||
|
drv = _driver(
|
||||||
|
url="https://planner.cloud.microsoft/webui/mytasks/assignedtome/view/grid",
|
||||||
|
elements={},
|
||||||
|
)
|
||||||
|
assert not _needs_microsoft_login(drv)
|
||||||
|
assert _planner_session_ready(drv)
|
||||||
|
|
||||||
|
def test_planner_url_with_password_still_needs_login(self):
|
||||||
|
drv = _driver(
|
||||||
|
url="https://planner.cloud.microsoft/webui/mytasks/assignedtome/view/grid",
|
||||||
|
elements={
|
||||||
|
(By.CSS_SELECTOR, "input[type='password']"): [_input()],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert _needs_microsoft_login(drv)
|
||||||
|
assert not _planner_session_ready(drv)
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
"""Unit tests for Planner left-nav helpers."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
from selenium.webdriver.common.by import By
|
||||||
|
|
||||||
|
from onedrive_planner import (
|
||||||
|
_left_nav_pane_collapsed,
|
||||||
|
_my_plans_nav_visible,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _nav_toggle(*, expanded: str = "false", displayed: bool = True):
|
||||||
|
btn = MagicMock()
|
||||||
|
btn.is_displayed.return_value = displayed
|
||||||
|
btn.get_attribute.side_effect = lambda name: expanded if name == "aria-expanded" else ""
|
||||||
|
return btn
|
||||||
|
|
||||||
|
|
||||||
|
def _nav_link(*, displayed: bool = True, enabled: bool = True):
|
||||||
|
el = MagicMock()
|
||||||
|
el.is_displayed.return_value = displayed
|
||||||
|
el.is_enabled.return_value = enabled
|
||||||
|
return el
|
||||||
|
|
||||||
|
|
||||||
|
def _mock_driver(*, elements: dict):
|
||||||
|
drv = MagicMock()
|
||||||
|
drv.find_elements.side_effect = lambda by, sel: elements.get((by, sel), [])
|
||||||
|
return drv
|
||||||
|
|
||||||
|
|
||||||
|
class TestNavHelpers:
|
||||||
|
def test_collapsed_when_aria_expanded_false(self):
|
||||||
|
drv = _mock_driver(
|
||||||
|
elements={
|
||||||
|
(By.XPATH, "//button[@aria-label='Left navigation pane']"): [
|
||||||
|
_nav_toggle(expanded="false")
|
||||||
|
],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
assert _left_nav_pane_collapsed(drv)
|
||||||
|
|
||||||
|
def test_not_collapsed_when_aria_expanded_true(self):
|
||||||
|
drv = _mock_driver(
|
||||||
|
elements={
|
||||||
|
(By.XPATH, "//button[@aria-label='Left navigation pane']"): [
|
||||||
|
_nav_toggle(expanded="true")
|
||||||
|
],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
assert not _left_nav_pane_collapsed(drv)
|
||||||
|
|
||||||
|
def test_my_plans_visible(self):
|
||||||
|
drv = _mock_driver(
|
||||||
|
elements={
|
||||||
|
(
|
||||||
|
By.XPATH,
|
||||||
|
"//*[self::span or self::div][normalize-space()='My plans']",
|
||||||
|
): [_nav_link()],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
assert _my_plans_nav_visible(drv)
|
||||||
|
|
||||||
|
def test_my_plans_hidden(self):
|
||||||
|
drv = _mock_driver(elements={})
|
||||||
|
assert not _my_plans_nav_visible(drv)
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
"""Tests for Planner tab URL classification."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from onedrive_planner import _is_facility_plan_url, _is_planner_hub_url
|
||||||
|
|
||||||
|
|
||||||
|
def test_facility_plan_url():
|
||||||
|
assert _is_facility_plan_url(
|
||||||
|
"https://planner.cloud.microsoft/webui/plan/PID_ABC/view/board?tid=x"
|
||||||
|
)
|
||||||
|
assert not _is_facility_plan_url(
|
||||||
|
"https://planner.cloud.microsoft/webui/mytasks/assignedtome/view/grid?tid=x"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_planner_hub_url():
|
||||||
|
assert _is_planner_hub_url(
|
||||||
|
"https://planner.cloud.microsoft/webui/mytasks/assignedtome/view/grid"
|
||||||
|
)
|
||||||
|
assert not _is_planner_hub_url(
|
||||||
|
"https://planner.cloud.microsoft/webui/plan/PID_ABC/view/board"
|
||||||
|
)
|
||||||
|
assert not _is_planner_hub_url("about:blank")
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
"""Tests for step logging helper."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from run_log import log_step
|
||||||
|
|
||||||
|
|
||||||
|
def test_log_step_formats_message(caplog):
|
||||||
|
caplog.set_level(logging.INFO, logger="misf.run")
|
||||||
|
log_step("Open facility", facility="Amaya", rows=2)
|
||||||
|
assert any("[STEP] Open facility" in r.message for r in caplog.records)
|
||||||
|
assert any("facility=Amaya" in r.message for r in caplog.records)
|
||||||
@@ -0,0 +1,306 @@
|
|||||||
|
"""Unit tests for board task filtering (not started + attachment)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from selenium.common.exceptions import NoSuchElementException
|
||||||
|
from selenium.webdriver.common.by import By
|
||||||
|
|
||||||
|
from onedrive_planner import (
|
||||||
|
_card_has_attachment,
|
||||||
|
_is_active_bucket_title,
|
||||||
|
_is_completed_tasks_header,
|
||||||
|
_is_in_active_planner_column_group,
|
||||||
|
_is_in_completed_tasks_section,
|
||||||
|
_is_probable_single_task_card,
|
||||||
|
_qualifies_not_started_with_attachment,
|
||||||
|
_task_card_matches_title,
|
||||||
|
_task_has_unchecked_complete_checkbox,
|
||||||
|
_task_is_in_progress,
|
||||||
|
_task_is_not_started,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _card(
|
||||||
|
*,
|
||||||
|
text: str = "",
|
||||||
|
aria: str = "",
|
||||||
|
attachment_child: bool = False,
|
||||||
|
checkbox_checked: bool = False,
|
||||||
|
progress_status: str | None = None,
|
||||||
|
):
|
||||||
|
el = MagicMock()
|
||||||
|
el.tag_name = "div"
|
||||||
|
el.text = text
|
||||||
|
el.is_displayed.return_value = True
|
||||||
|
el.get_attribute.return_value = aria
|
||||||
|
textbox = MagicMock()
|
||||||
|
textbox.is_displayed.return_value = True
|
||||||
|
textbox.text = text
|
||||||
|
textbox.get_attribute.return_value = ""
|
||||||
|
attach = MagicMock()
|
||||||
|
attach.is_displayed.return_value = True
|
||||||
|
attach.text = ""
|
||||||
|
attach.get_attribute.return_value = "1 attachment"
|
||||||
|
checkbox = MagicMock()
|
||||||
|
checkbox.is_displayed.return_value = True
|
||||||
|
checkbox.text = ""
|
||||||
|
checkbox.get_attribute.side_effect = lambda name: (
|
||||||
|
"true" if checkbox_checked else "false"
|
||||||
|
if name == "aria-checked"
|
||||||
|
else "Complete task"
|
||||||
|
if name == "aria-label"
|
||||||
|
else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
progress_btn = None
|
||||||
|
if progress_status == "in_progress":
|
||||||
|
progress_btn = MagicMock()
|
||||||
|
progress_btn.is_displayed.return_value = True
|
||||||
|
progress_btn.get_attribute.side_effect = lambda name: (
|
||||||
|
"In progress" if name == "aria-label" else ""
|
||||||
|
)
|
||||||
|
elif progress_status == "not_started":
|
||||||
|
progress_btn = MagicMock()
|
||||||
|
progress_btn.is_displayed.return_value = True
|
||||||
|
progress_btn.get_attribute.side_effect = lambda name: (
|
||||||
|
"Not started" if name == "aria-label" else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
def find_element(by, xpath):
|
||||||
|
if xpath == "./..":
|
||||||
|
raise NoSuchElementException
|
||||||
|
raise NoSuchElementException
|
||||||
|
|
||||||
|
el.find_element.side_effect = find_element
|
||||||
|
|
||||||
|
def _find_elements(by, xpath):
|
||||||
|
xp = xpath or ""
|
||||||
|
if xpath == "./..":
|
||||||
|
raise NoSuchElementException
|
||||||
|
if "textbox" in xp:
|
||||||
|
return [textbox] if text else []
|
||||||
|
if "attachment-indicator" in xp:
|
||||||
|
return [attach] if attachment_child else []
|
||||||
|
if "role='checkbox'" in xp or '@role="checkbox"' in xp:
|
||||||
|
if "@aria-checked='true'" in xp or 'aria-checked="true"' in xp:
|
||||||
|
return [checkbox] if checkbox_checked else []
|
||||||
|
if "@aria-checked='false'" in xp or 'aria-checked="false"' in xp:
|
||||||
|
return [] if checkbox_checked else [checkbox]
|
||||||
|
if "aria-checked='false'" in xp and "|" in xp:
|
||||||
|
return [] if checkbox_checked else [checkbox]
|
||||||
|
return [checkbox]
|
||||||
|
if progress_btn is not None and "button" in xp and "aria-label" in xp:
|
||||||
|
if progress_status == "in_progress" and "In progress" in xp:
|
||||||
|
return [progress_btn]
|
||||||
|
if progress_status == "not_started" and "Not started" in xp:
|
||||||
|
return [progress_btn]
|
||||||
|
return []
|
||||||
|
if "Attachment" in xp or "Attach" in xp or "attachment" in xp:
|
||||||
|
return [attach] if attachment_child else []
|
||||||
|
if "completed tasks" in xp.lower():
|
||||||
|
return []
|
||||||
|
return []
|
||||||
|
|
||||||
|
el.find_elements.side_effect = _find_elements
|
||||||
|
return el
|
||||||
|
|
||||||
|
|
||||||
|
class TestTaskFilter:
|
||||||
|
def test_not_started_with_paperclip_no_label_text(self):
|
||||||
|
card = _card(text="5.29.26", aria="Task card", attachment_child=True)
|
||||||
|
assert _card_has_attachment(card)
|
||||||
|
assert _task_is_not_started(card)
|
||||||
|
assert not _task_is_in_progress(card)
|
||||||
|
|
||||||
|
def test_skips_in_progress(self):
|
||||||
|
card = _card(text="Patient", aria="In progress, 1 attachment", attachment_child=True)
|
||||||
|
assert _task_is_in_progress(card)
|
||||||
|
assert not _task_is_not_started(card)
|
||||||
|
|
||||||
|
def test_skips_half_blue_in_progress_button(self):
|
||||||
|
"""Planner half-blue ring: button aria-label='In progress' (not the complete checkbox)."""
|
||||||
|
card = _card(
|
||||||
|
text="Abaite, Stephen 1/30/54-Need Insurance",
|
||||||
|
attachment_child=True,
|
||||||
|
progress_status="in_progress",
|
||||||
|
)
|
||||||
|
from onedrive_planner import _task_has_in_progress_status_button
|
||||||
|
|
||||||
|
assert _task_has_in_progress_status_button(card)
|
||||||
|
assert _task_is_in_progress(card)
|
||||||
|
assert not _task_is_not_started(card)
|
||||||
|
assert not _qualifies_not_started_with_attachment(card, from_indicator=True)
|
||||||
|
|
||||||
|
def test_processes_not_started_without_progress_button(self):
|
||||||
|
card = _card(
|
||||||
|
text="Richardson, Steve 1/21/56-Need Insurance",
|
||||||
|
attachment_child=True,
|
||||||
|
)
|
||||||
|
from onedrive_planner import _task_has_in_progress_status_button
|
||||||
|
|
||||||
|
assert not _task_has_in_progress_status_button(card)
|
||||||
|
assert _task_is_not_started(card)
|
||||||
|
assert _qualifies_not_started_with_attachment(card)
|
||||||
|
|
||||||
|
def test_skips_mixed_checkbox_in_progress(self):
|
||||||
|
card = _card(text="6.1.26", attachment_child=True)
|
||||||
|
checkbox = card.find_elements(By.XPATH, ".//label[@role='checkbox']")[0]
|
||||||
|
checkbox.get_attribute.side_effect = lambda name: (
|
||||||
|
"mixed"
|
||||||
|
if name == "aria-checked"
|
||||||
|
else "Complete task"
|
||||||
|
if name == "aria-label"
|
||||||
|
else ""
|
||||||
|
)
|
||||||
|
from onedrive_planner import _task_is_in_progress_scoped
|
||||||
|
|
||||||
|
assert _task_is_in_progress_scoped(card)
|
||||||
|
assert not _task_is_not_started(card)
|
||||||
|
|
||||||
|
def test_skips_completed(self):
|
||||||
|
card = _card(text="Done", aria="Completed", attachment_child=True, checkbox_checked=True)
|
||||||
|
assert not _task_is_not_started(card)
|
||||||
|
|
||||||
|
def test_checkbox_unchecked_is_not_started(self):
|
||||||
|
card = _card(text="5.29.26", attachment_child=True, checkbox_checked=False)
|
||||||
|
assert _task_has_unchecked_complete_checkbox(card)
|
||||||
|
assert _task_is_not_started(card)
|
||||||
|
|
||||||
|
def test_not_started_progress_ring_checkbox_qualifies(self):
|
||||||
|
"""Planner progress ring uses role=checkbox aria-label='Not started'."""
|
||||||
|
card = _card(text="6.3.26", attachment_child=True, progress_status="not_started")
|
||||||
|
checkbox = card.find_elements(By.XPATH, ".//label[@role='checkbox']")[0]
|
||||||
|
checkbox.get_attribute.side_effect = lambda name: (
|
||||||
|
"false"
|
||||||
|
if name == "aria-checked"
|
||||||
|
else "Not started"
|
||||||
|
if name == "aria-label"
|
||||||
|
else ""
|
||||||
|
)
|
||||||
|
from onedrive_planner import _filter_reject_reason, _task_is_strictly_not_started
|
||||||
|
|
||||||
|
assert _task_has_unchecked_complete_checkbox(card)
|
||||||
|
assert _task_is_strictly_not_started(card)
|
||||||
|
assert _filter_reject_reason(card, from_indicator=True) is None
|
||||||
|
assert _qualifies_not_started_with_attachment(card, from_indicator=True)
|
||||||
|
|
||||||
|
def test_planner_button_complete_checkbox_is_not_started(self):
|
||||||
|
"""Planner 2024+ uses button[role=checkbox] aria-label='Complete task'."""
|
||||||
|
card = _card(text="6.8.26", attachment_child=True, checkbox_checked=False)
|
||||||
|
checkbox = card.find_elements(
|
||||||
|
By.XPATH,
|
||||||
|
".//label[@role='checkbox'] | .//button[@role='checkbox']",
|
||||||
|
)[0]
|
||||||
|
checkbox.get_attribute.side_effect = lambda name: (
|
||||||
|
"false"
|
||||||
|
if name == "aria-checked"
|
||||||
|
else "Complete task"
|
||||||
|
if name == "aria-label"
|
||||||
|
else ""
|
||||||
|
)
|
||||||
|
from onedrive_planner import _filter_reject_reason, _task_has_unchecked_complete_checkbox
|
||||||
|
|
||||||
|
assert _task_has_unchecked_complete_checkbox(card)
|
||||||
|
assert _filter_reject_reason(card, from_indicator=True) is None
|
||||||
|
|
||||||
|
def test_in_progress_rejected_on_board_without_opening(self):
|
||||||
|
"""Half-blue In progress ring on the card — skip at board scan."""
|
||||||
|
card = _card(
|
||||||
|
text="Endersby, Mary",
|
||||||
|
attachment_child=True,
|
||||||
|
progress_status="in_progress",
|
||||||
|
)
|
||||||
|
from onedrive_planner import (
|
||||||
|
_filter_reject_reason,
|
||||||
|
_task_card_in_progress_on_board,
|
||||||
|
_task_is_strictly_not_started,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert _task_card_in_progress_on_board(card)
|
||||||
|
assert not _task_is_strictly_not_started(card)
|
||||||
|
assert _filter_reject_reason(card, from_indicator=True) == "in_progress"
|
||||||
|
assert _filter_reject_reason(card, from_indicator=False) == "in_progress"
|
||||||
|
assert not _qualifies_not_started_with_attachment(card, from_indicator=True)
|
||||||
|
|
||||||
|
def test_requires_attachment(self):
|
||||||
|
card = _card(text="5.29.26", aria="Not started")
|
||||||
|
assert not _card_has_attachment(card)
|
||||||
|
|
||||||
|
def test_completed_header_matcher(self):
|
||||||
|
assert _is_completed_tasks_header("Completed tasks")
|
||||||
|
assert _is_completed_tasks_header("Completed tasks 37")
|
||||||
|
assert not _is_completed_tasks_header("To do 5.29.26 Completed tasks section")
|
||||||
|
|
||||||
|
def test_indicator_qualifies_without_board_completed_noise(self):
|
||||||
|
ind = _card(text="6.3.26", aria="1 attachment", attachment_child=True)
|
||||||
|
assert _is_probable_single_task_card(ind)
|
||||||
|
assert _qualifies_not_started_with_attachment(ind, from_indicator=True)
|
||||||
|
|
||||||
|
def test_unbucketed_active_group_not_completed_section(self):
|
||||||
|
"""Tasks in taskBoardColumnGroup-unbucketed must not match completed subsection."""
|
||||||
|
from selenium.webdriver.common.by import By
|
||||||
|
|
||||||
|
indicator = MagicMock()
|
||||||
|
active_group = MagicMock()
|
||||||
|
active_group.get_attribute.side_effect = lambda name: (
|
||||||
|
"taskBoardColumnGroup-unbucketed" if name == "data-testid" else ""
|
||||||
|
)
|
||||||
|
indicator.get_attribute.return_value = ""
|
||||||
|
|
||||||
|
def _find_elements(by, xp):
|
||||||
|
if by == By.XPATH and "taskBoardColumnGroup-" in xp:
|
||||||
|
if "contains(@data-testid,'-completed')" in xp:
|
||||||
|
return []
|
||||||
|
return [active_group]
|
||||||
|
return []
|
||||||
|
|
||||||
|
indicator.find_elements.side_effect = _find_elements
|
||||||
|
|
||||||
|
assert _is_in_active_planner_column_group(indicator)
|
||||||
|
assert not _is_in_completed_tasks_section(indicator)
|
||||||
|
|
||||||
|
def test_active_bucket_titles(self):
|
||||||
|
assert _is_active_bucket_title("To do")
|
||||||
|
assert _is_active_bucket_title("TO DO")
|
||||||
|
assert _is_active_bucket_title("New Facesheets")
|
||||||
|
assert _is_active_bucket_title("New Facesheets 12")
|
||||||
|
assert _is_active_bucket_title("New Facesheet")
|
||||||
|
assert _is_active_bucket_title("Heather gets facesheets")
|
||||||
|
assert _is_active_bucket_title("No bucket")
|
||||||
|
assert not _is_active_bucket_title("Completed tasks")
|
||||||
|
assert not _is_active_bucket_title("Completed tasks 875")
|
||||||
|
|
||||||
|
def test_bucket_column_detected(self):
|
||||||
|
col = _card(
|
||||||
|
text="To do",
|
||||||
|
aria="To do, Use tab key to access a task card, left and right arrow",
|
||||||
|
attachment_child=True,
|
||||||
|
)
|
||||||
|
from onedrive_planner import _is_planner_bucket_column
|
||||||
|
|
||||||
|
assert _is_planner_bucket_column(col)
|
||||||
|
|
||||||
|
def test_weak_title_key_ignores_paperclip_count(self):
|
||||||
|
from onedrive_planner import _is_weak_task_title_key
|
||||||
|
|
||||||
|
assert _is_weak_task_title_key("3")
|
||||||
|
assert _is_weak_task_title_key("")
|
||||||
|
assert not _is_weak_task_title_key("6.3.26 Missing Insurance Info")
|
||||||
|
|
||||||
|
def test_board_wrapper_not_single_task_card(self):
|
||||||
|
board = _card(text="To do\n6.3.26\nCompleted tasks 37", attachment_child=True)
|
||||||
|
board.find_elements.side_effect = lambda by, xpath: (
|
||||||
|
[MagicMock(is_displayed=lambda: True)] * 3
|
||||||
|
if "attachment-indicator" in xpath
|
||||||
|
else [MagicMock(is_displayed=lambda: True)] * 3
|
||||||
|
if "checkbox" in xpath
|
||||||
|
else []
|
||||||
|
)
|
||||||
|
assert not _is_probable_single_task_card(board)
|
||||||
|
|
||||||
|
def test_task_card_matches_title(self):
|
||||||
|
card = _card(text="5.29.26\n1 attachment", attachment_child=True)
|
||||||
|
assert _task_card_matches_title(card, "5.29.26")
|
||||||
|
assert not _task_card_matches_title(card, "Alvarado, Shante")
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
"""Unit tests for task-details panel status detection (scoped to panel, not board)."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
from selenium.webdriver.common.by import By
|
||||||
|
|
||||||
|
from onedrive_planner import (
|
||||||
|
_panel_progress_status_label,
|
||||||
|
_task_details_shows_in_progress,
|
||||||
|
_task_details_shows_not_started,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _panel(*, status_aria: str):
|
||||||
|
panel = MagicMock()
|
||||||
|
panel.is_displayed.return_value = True
|
||||||
|
status_btn = MagicMock()
|
||||||
|
status_btn.is_displayed.return_value = True
|
||||||
|
status_btn.get_attribute.side_effect = lambda name: (
|
||||||
|
status_aria if name == "aria-label" else ""
|
||||||
|
)
|
||||||
|
title_box = MagicMock()
|
||||||
|
title_box.is_displayed.return_value = True
|
||||||
|
title_box.text = "6.5.26"
|
||||||
|
|
||||||
|
def find_elements(by, xpath):
|
||||||
|
xp = xpath or ""
|
||||||
|
if "Not started" in xp or "In progress" in xp or "Completed" in xp:
|
||||||
|
return [status_btn]
|
||||||
|
if "textbox" in xp:
|
||||||
|
return [title_box]
|
||||||
|
if "Status" in xp:
|
||||||
|
return []
|
||||||
|
return []
|
||||||
|
|
||||||
|
panel.find_elements.side_effect = find_elements
|
||||||
|
return panel
|
||||||
|
|
||||||
|
|
||||||
|
def _driver_with_panel(panel, *, board_in_progress: bool = False):
|
||||||
|
drv = MagicMock()
|
||||||
|
board_btn = MagicMock()
|
||||||
|
board_btn.is_displayed.return_value = board_in_progress
|
||||||
|
board_btn.get_attribute.side_effect = lambda name: (
|
||||||
|
"In progress" if name == "aria-label" else ""
|
||||||
|
)
|
||||||
|
|
||||||
|
def find_elements(by, xpath):
|
||||||
|
xp = xpath or ""
|
||||||
|
if "@role='dialog'" in xp or "TaskDetails" in xp or "Close task" in xp:
|
||||||
|
return [panel]
|
||||||
|
if "In progress" in xp and xp.startswith("//button"):
|
||||||
|
return [board_btn] if board_in_progress else []
|
||||||
|
if "Close" in xp:
|
||||||
|
return []
|
||||||
|
return []
|
||||||
|
|
||||||
|
drv.find_elements.side_effect = find_elements
|
||||||
|
return drv
|
||||||
|
|
||||||
|
|
||||||
|
class TestTaskPanelStatus:
|
||||||
|
def test_panel_not_started(self):
|
||||||
|
panel = _panel(status_aria="Not started")
|
||||||
|
assert _panel_progress_status_label(panel) == "not started"
|
||||||
|
|
||||||
|
def test_panel_in_progress(self):
|
||||||
|
panel = _panel(status_aria="In progress")
|
||||||
|
assert _panel_progress_status_label(panel) == "in progress"
|
||||||
|
|
||||||
|
def test_details_not_started_ignores_board_buttons(self):
|
||||||
|
panel = _panel(status_aria="Not started")
|
||||||
|
drv = _driver_with_panel(panel, board_in_progress=True)
|
||||||
|
assert _task_details_shows_not_started(drv)
|
||||||
|
assert not _task_details_shows_in_progress(drv)
|
||||||
|
|
||||||
|
def test_details_in_progress_from_panel_only(self):
|
||||||
|
panel = _panel(status_aria="In progress")
|
||||||
|
drv = _driver_with_panel(panel, board_in_progress=False)
|
||||||
|
assert _task_details_shows_in_progress(drv)
|
||||||
|
assert not _task_details_shows_not_started(drv)
|
||||||
|
|
||||||
|
def test_panel_in_progress_from_combobox_value(self):
|
||||||
|
panel = MagicMock()
|
||||||
|
panel.is_displayed.return_value = True
|
||||||
|
combo = MagicMock()
|
||||||
|
combo.is_displayed.return_value = True
|
||||||
|
combo.get_attribute.side_effect = lambda name: "inProgress" if name == "value" else ""
|
||||||
|
combo.text = ""
|
||||||
|
|
||||||
|
def find_elements(by, xpath):
|
||||||
|
xp = xpath or ""
|
||||||
|
if "combobox" in xp or "ComboBox" in xp:
|
||||||
|
return [combo]
|
||||||
|
return []
|
||||||
|
|
||||||
|
panel.find_elements.side_effect = find_elements
|
||||||
|
assert _panel_progress_status_label(panel) == "in progress"
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
"""Unit tests for TOTP secret sanitization."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
SRC = Path(__file__).resolve().parent.parent / "src"
|
||||||
|
sys.path.insert(0, str(SRC))
|
||||||
|
|
||||||
|
from totp_utils import sanitize_totp_secret, validate_totp_secret # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def test_sanitize_strips_separators() -> None:
|
||||||
|
assert sanitize_totp_secret("jbsw y3dp-ehpk3pxp") == "JBSWY3DPEHPK3PXP"
|
||||||
|
|
||||||
|
|
||||||
|
def test_sanitize_maps_zero_one() -> None:
|
||||||
|
assert sanitize_totp_secret("JBSW03DP") == "JBSWO3DP"
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_rejects_invalid_chars() -> None:
|
||||||
|
with pytest.raises(ValueError, match="not valid Base32"):
|
||||||
|
validate_totp_secret("not-valid-secret!")
|
||||||
|
|
||||||
|
|
||||||
|
def test_validate_accepts_standard_secret() -> None:
|
||||||
|
assert validate_totp_secret("JBSWY3DPEHPK3PXP") == "JBSWY3DPEHPK3PXP"
|
||||||
@@ -0,0 +1,149 @@
|
|||||||
|
"""UiPath migration helper tests and pilot output structure checks."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
UIPATH = ROOT / "uipath" / "MISF_Planner_Download"
|
||||||
|
HELPERS_CS = UIPATH / "Helpers"
|
||||||
|
|
||||||
|
|
||||||
|
def _read_cs(name: str) -> str:
|
||||||
|
return (HELPERS_CS / name).read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
class TestUiPathProjectStructure:
|
||||||
|
def test_project_json_exists(self):
|
||||||
|
assert (UIPATH / "project.json").is_file()
|
||||||
|
|
||||||
|
def test_main_workflows_exist(self):
|
||||||
|
for name in (
|
||||||
|
"Main.xaml",
|
||||||
|
"01_ReadConfig.xaml",
|
||||||
|
"02_LoginPlanner.xaml",
|
||||||
|
"03_ProcessFacility.xaml",
|
||||||
|
"04_DownloadTaskAttachments.xaml",
|
||||||
|
"05_SetTaskInProgress.xaml",
|
||||||
|
"06_WriteSummary.xaml",
|
||||||
|
):
|
||||||
|
assert (UIPATH / name).is_file(), f"missing {name}"
|
||||||
|
|
||||||
|
def test_helpers_exist(self):
|
||||||
|
assert (UIPATH / "Helpers" / "GenerateTotp.xaml").is_file()
|
||||||
|
assert (UIPATH / "Helpers" / "SanitizeFolderName.xaml").is_file()
|
||||||
|
assert (UIPATH / "Helpers" / "TotpGenerator.cs").is_file()
|
||||||
|
|
||||||
|
def test_selectors_documented(self):
|
||||||
|
assert (UIPATH / "Selectors" / "PlannerSelectors.json").is_file()
|
||||||
|
assert (UIPATH / "Selectors" / "IFRAME_SETUP.md").is_file()
|
||||||
|
data = json.loads((UIPATH / "Selectors" / "PlannerSelectors.json").read_text())
|
||||||
|
assert "planner_iframe" in data
|
||||||
|
assert "my_plans" in data["planner_iframe"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestHelperLogicMirrorsPython:
|
||||||
|
def test_sanitize_folder_name(self):
|
||||||
|
src = _read_cs("MisfHelpers.cs")
|
||||||
|
assert "SanitizeFolderName" in src
|
||||||
|
assert "InvalidFolderChars" in src
|
||||||
|
|
||||||
|
def test_totp_sanitize_matches_python(self):
|
||||||
|
src = _read_cs("MisfHelpers.cs")
|
||||||
|
assert 'Replace("0", "O")' in src
|
||||||
|
assert 'Replace("1", "I")' in src
|
||||||
|
|
||||||
|
def test_download_url_transform(self):
|
||||||
|
src = _read_cs("MisfHelpers.cs")
|
||||||
|
assert "web=1" in src
|
||||||
|
assert "download=1" in src
|
||||||
|
|
||||||
|
def test_default_planner_url(self):
|
||||||
|
src = _read_cs("MisfConstants.cs")
|
||||||
|
assert "planner.cloud.microsoft" in src
|
||||||
|
assert "0c15ad2c-19dd-4ded-9cd1-20d344386414" in src
|
||||||
|
|
||||||
|
|
||||||
|
class TestPilotOutputLayout:
|
||||||
|
"""Compare output folder layout with Python bot expectations."""
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def today_iso(self) -> str:
|
||||||
|
return datetime.now().strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
def test_output_directories_can_be_created(self, today_iso: str, tmp_path: Path):
|
||||||
|
original = tmp_path / "Output" / today_iso / "Original" / "Test_Facility"
|
||||||
|
workable = tmp_path / "Output" / today_iso / "Workable"
|
||||||
|
summary = tmp_path / "Output" / today_iso / "Summary"
|
||||||
|
for d in (original, workable, summary):
|
||||||
|
d.mkdir(parents=True, exist_ok=True)
|
||||||
|
assert original.is_dir()
|
||||||
|
assert workable.is_dir()
|
||||||
|
assert summary.is_dir()
|
||||||
|
|
||||||
|
def test_checkpoint_json_shape(self, tmp_path: Path):
|
||||||
|
ckpt = {
|
||||||
|
"last_completed_index": 2,
|
||||||
|
"last_facility": "Amaya",
|
||||||
|
"completed_facilities": ["Amaya", "Brighton Place San Diego"],
|
||||||
|
}
|
||||||
|
path = tmp_path / "checkpoint.json"
|
||||||
|
path.write_text(json.dumps(ckpt), encoding="utf-8")
|
||||||
|
loaded = json.loads(path.read_text())
|
||||||
|
assert loaded["last_completed_index"] == 2
|
||||||
|
assert isinstance(loaded["completed_facilities"], list)
|
||||||
|
|
||||||
|
|
||||||
|
class TestMainXamlExpressions:
|
||||||
|
def test_main_has_facility_limit_argument(self):
|
||||||
|
main = (UIPATH / "Main.xaml").read_text(encoding="utf-8")
|
||||||
|
assert "in_FacilityLimit" in main
|
||||||
|
assert "FilterFacilities" in main
|
||||||
|
|
||||||
|
def test_login_invokes_generate_totp(self):
|
||||||
|
login = (UIPATH / "02_LoginPlanner.xaml").read_text(encoding="utf-8")
|
||||||
|
assert "GenerateTotp.xaml" in login
|
||||||
|
assert "loginfmt" in login
|
||||||
|
|
||||||
|
def test_iframe_prefix_in_process_facility(self):
|
||||||
|
proc = (UIPATH / "03_ProcessFacility.xaml").read_text(encoding="utf-8")
|
||||||
|
assert "IFRAME" in proc
|
||||||
|
assert "My plans" in proc
|
||||||
|
|
||||||
|
def test_attachments_tab_download_flow(self):
|
||||||
|
py = (ROOT / "src" / "onedrive_planner.py").read_text(encoding="utf-8")
|
||||||
|
assert "_open_attachments_view" in py
|
||||||
|
assert "_ensure_board_scan_context" in py
|
||||||
|
assert "attachment-indicator" in py
|
||||||
|
assert "_card_has_attachment" in py
|
||||||
|
assert "_task_is_not_started" in py
|
||||||
|
assert "_open_task_via_attachment" in py
|
||||||
|
assert "_click_exact_attachment_name" in py
|
||||||
|
assert "_collect_attachment_hrefs" in py
|
||||||
|
assert "_collect_attachment_download_urls" in py
|
||||||
|
assert "_download_href_via_http" in py
|
||||||
|
assert "_download_attachment_row" in py
|
||||||
|
assert "_list_attachment_rows" in py
|
||||||
|
assert "_close_extra_download_tabs" in py
|
||||||
|
assert "_download_href_with_fallback" in py
|
||||||
|
assert "_download_one_href_browser_new_tab" in py
|
||||||
|
assert "HTTP only last" in py or "HTTP last" in py
|
||||||
|
assert "configure_http_download_proxy" in py
|
||||||
|
assert "_url_from_attachment_element" in py
|
||||||
|
assert "_scroll_attachments_panel" in py
|
||||||
|
dl = (UIPATH / "04_DownloadTaskAttachments.xaml").read_text(encoding="utf-8")
|
||||||
|
assert "Attachments" in dl
|
||||||
|
assert "sharepoint" in dl.lower()
|
||||||
|
|
||||||
|
def test_in_progress_task_details_only(self):
|
||||||
|
proc = (UIPATH / "05_SetTaskInProgress.xaml").read_text(encoding="utf-8")
|
||||||
|
assert "Task details" in proc
|
||||||
|
assert "In progress" in proc
|
||||||
|
py = (ROOT / "src" / "onedrive_planner.py").read_text(encoding="utf-8")
|
||||||
|
assert "_set_task_in_progress_from_task_details" in py
|
||||||
|
assert "_mark_task_in_progress" in py
|
||||||
|
assert "task details only" in py
|
||||||
|
assert "_set_task_in_progress_from_board(driver, card" not in py
|
||||||
Reference in new issue
Block a user