Initial commit: MISF VOB download and upload automation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
lubna-fatimaandClaude Sonnet 4.6 committed
commit
c2dec96b1f
43 files changed
+10944
No files matched your search
@@ -0,0 +1,228 @@
|
||||
# OneDrive / Planner download-only automation
|
||||
|
||||
Automates the MISF VOB **face-sheet download** workflow: sign in to OneDrive, open Planner, search each facility plan, download attachment PDFs, and mark tasks **In progress**.
|
||||
|
||||
Reference: [`docs/MISF-VOB-Planner-Workflow.docx`](docs/MISF-VOB-Planner-Workflow.docx)
|
||||
|
||||
This repository contains **only** OneDrive / Planner download automation. There is no Noridian Medicare, Medi-Cal, PDF extraction, or eligibility verification code.
|
||||
|
||||
---
|
||||
|
||||
## Source requirements
|
||||
|
||||
| Doc step | Action |
|
||||
|----------|--------|
|
||||
| 1 | Open OneDrive URL, enter username / password, complete MFA (Google Authenticator 6-digit code) |
|
||||
| 2–4 | Open **Planner** (direct URL in config; default: Assigned to me grid) |
|
||||
| 5 | For each facility: search/select plan → open tasks → **download attachment PDFs** → set task status **In progress** (board card, or Task details → Status) |
|
||||
|
||||
**Clarifications:**
|
||||
|
||||
- Process **54 facilities** by typing each name into Planner’s **search box**, one at a time.
|
||||
- MFA via **TOTP secret** in `Input/input.xlsx` (automated with `pyotp`).
|
||||
|
||||
**Not in this repo (upload path from the MISF guide):**
|
||||
|
||||
- Uploading completed work to OneDrive
|
||||
- Marking tasks **Completed** after eligibility verification
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
start[run_download_workflow] --> config[Read input.xlsx]
|
||||
config --> chrome[start_chrome_debug + attach_browser]
|
||||
chrome --> login[Microsoft login + TOTP]
|
||||
login --> planner[Open Planner / In my plans]
|
||||
planner --> loop{For each facility}
|
||||
loop --> search[Search facility in Planner]
|
||||
search --> dl[Download task attachments]
|
||||
dl --> status[Mark tasks In progress]
|
||||
status --> loop
|
||||
loop --> summary[Log summary + Excel report]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Repository layout
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| [`run_download.py`](run_download.py) | Main entry point |
|
||||
| [`src/planner_download.py`](src/planner_download.py) | Orchestrator: config → Chrome → login → facility loop → summary |
|
||||
| [`src/onedrive_planner.py`](src/onedrive_planner.py) | Selenium: Microsoft login, Planner, downloads, In-Process |
|
||||
| [`src/config.py`](src/config.py) | `read_app_config()`, `read_facility_list()` |
|
||||
| [`src/totp_utils.py`](src/totp_utils.py) | TOTP secret sanitization / validation |
|
||||
| [`scripts/validate_config.py`](scripts/validate_config.py) | Pre-flight config check (no browser) |
|
||||
| [`src/chrome.py`](src/chrome.py) | Chrome debug startup, Selenium attach, proxy auth |
|
||||
| [`scripts/bootstrap_input_xlsx.py`](scripts/bootstrap_input_xlsx.py) | Create config template |
|
||||
| [`tests/test_download_config.py`](tests/test_download_config.py) | Config smoke tests |
|
||||
|
||||
```
|
||||
.
|
||||
├── README.md
|
||||
├── requirements.txt
|
||||
├── run_download.py
|
||||
├── docs/
|
||||
├── Input/
|
||||
│ ├── input.xlsx.example
|
||||
│ ├── input.xlsx # gitignored
|
||||
│ └── PDFs/
|
||||
├── Output/
|
||||
├── scripts/
|
||||
├── tests/
|
||||
└── src/
|
||||
├── planner_download.py
|
||||
├── onedrive_planner.py
|
||||
├── config.py
|
||||
└── chrome.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration (`Input/input.xlsx`)
|
||||
|
||||
```powershell
|
||||
copy Input\input.xlsx.example Input\input.xlsx
|
||||
copy .env.example .env
|
||||
```
|
||||
|
||||
Fill credentials in **either** place:
|
||||
|
||||
- **`Input/input.xlsx`** → Sheet1, column **B** (`OneDrive Username`, `OneDrive Password`, `Microsoft TOTP Secret`), **or**
|
||||
- **`.env`** in the project root (`ONEDRIVE_USERNAME`, `ONEDRIVE_PASSWORD`, `MICROSOFT_TOTP_SECRET`)
|
||||
|
||||
Check config before running:
|
||||
|
||||
```powershell
|
||||
python scripts\validate_config.py
|
||||
```
|
||||
|
||||
### Sheet `Sheet1`
|
||||
|
||||
| Key | Purpose |
|
||||
|-----|---------|
|
||||
| `OneDrive Username` | Microsoft login email |
|
||||
| `OneDrive Password` | Password |
|
||||
| `Microsoft TOTP Secret` | Base32 authenticator secret (`pyotp`) |
|
||||
| `OneDrive URL` | OneDrive landing URL |
|
||||
| `pdf_folder_path` | Download root (default: `Input/PDFs`) |
|
||||
| `Proxy IP` / `Proxy Port` / `Proxy Username` / `Proxy Password` | Optional |
|
||||
|
||||
### Sheet `Facilities` — **put all 54 facility names here**
|
||||
|
||||
Open **`Input/input.xlsx`** (not the `.example` file) and edit the **`Facilities`** sheet:
|
||||
|
||||
| Column | Purpose |
|
||||
|--------|---------|
|
||||
| `Facility Name` | **One row per facility** (54 rows). Use the exact text you type in Planner search. |
|
||||
|
||||
Example:
|
||||
|
||||
| Facility Name |
|
||||
|---------------|
|
||||
| Amaya |
|
||||
| Brighton Place San Diego |
|
||||
| … (52 more rows) |
|
||||
|
||||
The bot reads this sheet via `read_facility_list()` in `src/config.py`. For a first test, keep only **2 rows** and run `python run_download.py --limit 2`.
|
||||
|
||||
---
|
||||
|
||||
## Production `.exe` (double-click)
|
||||
|
||||
Build a Windows folder you can copy to any PC:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File scripts\build_production_exe.ps1
|
||||
```
|
||||
|
||||
Output at **project root**: `MISF_VOB_Download.exe` and `_internal\` (same folder as `Input/`, `Output/`, `run_download.py`).
|
||||
|
||||
**Deploy:** zip the project folder (omit `.venv`, `build`, `src` if you only ship the exe build). On the target machine:
|
||||
|
||||
1. Unzip anywhere
|
||||
2. Copy `Input\input.xlsx.example` → `Input\input.xlsx` and configure credentials + facilities
|
||||
3. Double-click `MISF_VOB_Download.exe`
|
||||
|
||||
Config, `Input/`, and `Output/` are read from the **project root** (folder containing the `.exe`). See [`README_PRODUCTION.txt`](README_PRODUCTION.txt) after build.
|
||||
|
||||
---
|
||||
|
||||
## Quick start
|
||||
|
||||
```powershell
|
||||
python -m venv .venv
|
||||
.\.venv\Scripts\Activate.ps1
|
||||
pip install -r requirements.txt
|
||||
copy Input\input.xlsx.example Input\input.xlsx
|
||||
# Edit Input\input.xlsx
|
||||
|
||||
python run_download.py --limit 2 # dry-run (first N facilities)
|
||||
python run_download.py --facility "Amaya" # single facility
|
||||
python run_download.py --resume # continue after last checkpoint
|
||||
python run_download.py --keep-browser # leave browser open when done
|
||||
python run_download.py # full run
|
||||
```
|
||||
|
||||
**Outputs:**
|
||||
|
||||
- `Input/PDFs/<FacilityName>/*.pdf`
|
||||
- `Output/<date>/Summary/Download_Summary_<date>.xlsx` (includes run timestamp and Planner URL)
|
||||
- `Output/<date>/logs/download.log`
|
||||
- `Output/<date>/checkpoint.json` (while a run is in progress; cleared on success)
|
||||
|
||||
Set `MISF_DEBUG=1` in `.env` to enable NDJSON debug traces (`debug-fc545e.log`).
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
```powershell
|
||||
python -m pytest
|
||||
python scripts\validate_config.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Issue | What to check |
|
||||
|-------|----------------|
|
||||
| Login / MFA fails | TOTP secret; system clock (NTP) |
|
||||
| Facility not found | `Facility Name` matches Planner search |
|
||||
| No PDFs downloaded | Attachments exist; tune `src/onedrive_planner.py` selectors |
|
||||
| Proxy errors | Proxy IP/Port in `input.xlsx`; profile `%USERPROFILE%\Automation\ChromiumProfile_MISF_Download` |
|
||||
| Browser not found | Install [Chromium](https://www.chromium.org/getting-involved/download-chromium/) or set `CHROMIUM_PATH` in `.env` (Google Chrome only with `MISF_ALLOW_CHROME=1`) |
|
||||
| `ChromeDriver only supports Chrome version 148` / attach fails | Install Chromium matching driver version; driver auto-downloads to `.drivers/` |
|
||||
|
||||
---
|
||||
|
||||
## Browser
|
||||
|
||||
The automation uses **Chromium only** (minimized in the background by default). It looks for, in order:
|
||||
|
||||
1. `CHROMIUM_PATH` or `BROWSER_PATH` environment variable
|
||||
2. **`chrome-win/chrome-win/chrome.exe`** (portable bundle at project root — default for production)
|
||||
3. Bundled **`chromedriver.exe`** in the same folder (no download needed)
|
||||
4. System Chromium under `Program Files\Chromium`, `%LOCALAPPDATA%\Chromium`, Scoop (`MISF_USE_INSTALLED_CHROME=1` to prefer this over `chrome-win`)
|
||||
5. Google Chrome only when `MISF_ALLOW_CHROME=1`
|
||||
|
||||
Set `MISF_BROWSER_FOREGROUND=1` to show/maximize the browser window. The production `.exe` runs without a console window; see `Output/<date>/logs/download.log`.
|
||||
|
||||
**Chromedriver:** `webdriver-manager` downloads a matching driver. If attach fails, install Chromium that matches your driver or set `CHROMIUM_PATH` explicitly.
|
||||
|
||||
User profile directory: `%USERPROFILE%\Automation\ChromiumProfile_MISF_Download`
|
||||
|
||||
---
|
||||
|
||||
## UiPath migration
|
||||
|
||||
An equivalent **UiPath Studio (Modern)** project lives in [`uipath/MISF_Planner_Download/`](uipath/MISF_Planner_Download/). Open `project.json` in UiPath Studio. Pre-flight check:
|
||||
|
||||
```powershell
|
||||
powershell -File scripts\validate_uipath_pilot.ps1
|
||||
python -m pytest tests\test_uipath_pilot.py -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Internal MISF / Access Psychology automation.
|
||||
Reference in new issue
Block a user