docs: full accuracy audit, add validation framework to all docs, fix test_e2e.py, create TODO.md

- README.md: fix step count 5->6, add Conflict Detection to E2E table, add
validation/ to project structure, add validation commands to Running Tests,
add External Dataset Validation section, add validation to Tech Stack
- architecture.md: fix Decision #1 (5->6 step), fix Decision #4 (2->4 Gemma
roles), add Validation Framework section with dataset table and architecture
- test_results.md: add Section 6 External Dataset Validation with datasets,
smoke test results, and reproduction steps; fix test_e2e.py line count
- DEVELOPMENT_LOG.md: remove (Current) from Phase 7, add Phase 9 (validation
framework build with problems solved), add Phase 10 (documentation audit)
- writeup_draft.md: fix 'confidence levels' -> 'caveats and limitations',
update GitHub repo link, add validation methodology section
- test_e2e.py: add assertions for 6 steps, verify conflict_detection present,
assert no failed steps
- TODO.md: new file with prioritized next-session action items for easy pickup

DEVELOPMENT_LOG.md

@@ -155,7 +155,7 @@ Created `test_clinical_cases.py` with 22 diverse clinical scenarios:
 
 ---
 
-## Phase 7: Documentation (Current)
+## Phase 7: Documentation
 
 Performed comprehensive documentation audit. Found:
 - README was outdated (wrong port, missing test info, incomplete structure tree)
@@ -265,3 +265,65 @@ All config via `.env` (template in `.env.template`):
 | `EMBEDDING_MODEL` | No | `sentence-transformers/all-MiniLM-L6-v2` | RAG embeddings |
 | `MAX_GUIDELINES` | No | `5` | Guidelines per RAG query |
 | `AGENT_TIMEOUT` | No | `120` | Pipeline timeout (seconds) |
+
+---
+
+## Phase 9: External Dataset Validation Framework
+
+### Motivation
+
+Internal tests (RAG quality, clinical cases) are useful but don't measure diagnostic accuracy against ground truth. Added a validation framework to test the full pipeline against real-world clinical datasets with known correct answers.
+
+### Datasets Evaluated
+
+| Dataset | Source | What It Tests |
+|---------|--------|---------------|
+| **MedQA (USMLE)** | HuggingFace — `GBaker/MedQA-USMLE-4-options` | Diagnostic accuracy (1,273 USMLE-style questions with verified answers) |
+| **MTSamples** | GitHub — `socd06/medical-nlp` | Parse quality & field completeness on real medical transcription notes |
+| **PMC Case Reports** | PubMed E-utilities (esearch + efetch) | Diagnostic accuracy on published case reports with known diagnoses |
+
+### Architecture
+
+Created `src/backend/validation/` package:
+
+- **`base.py`** — Core framework: `ValidationCase`, `ValidationResult`, `ValidationSummary` dataclasses. `run_cds_pipeline()` invokes the Orchestrator directly (no HTTP server needed). Includes `fuzzy_match()` token-overlap scorer and `diagnosis_in_differential()` checker.
+- **`harness_medqa.py`** — Downloads JSONL from HuggingFace, extracts clinical vignettes (strips question stems), scores top-1/top-3/mentioned diagnostic accuracy.
+- **`harness_mtsamples.py`** — Downloads CSV, filters to relevant specialties, stratified sampling. Scores parse success, field completeness, specialty alignment, has_differential, has_recommendations.
+- **`harness_pmc.py`** — Uses NCBI E-utilities with 20 curated queries across specialties. Extracts diagnosis from article titles via regex patterns. Scores diagnostic accuracy.
+- **`run_validation.py`** — Unified CLI: `python -m validation.run_validation --all --max-cases 10`. Supports `--fetch-only`, `--no-drugs`, `--no-guidelines`, `--seed`, `--delay`.
+
+### Problems Solved
+
+1. **MedQA URL 404:** Original GitHub raw URL was stale. Fixed to HuggingFace direct download.
+2. **MTSamples URL 404:** Original mirror was down. Found working mirror at `socd06/medical-nlp`.
+3. **PMC fetcher returned 0 cases:** PubMed API worked, but title regex patterns didn't match common formats like "X: A Case Report." Added 3 new title patterns and fixed query-based fallback extraction.
+4. **`datetime.utcnow()` deprecation:** Replaced with `datetime.now(timezone.utc)` throughout.
+5. **Pipeline time display bug:** `print_summary` showed time metrics as percentages. Fixed by reordering type checks.
+
+### Initial Results (Smoke Test)
+
+Ran 3 MedQA cases through the full pipeline:
+- **Parse success:** 100% (3/3)
+- **Top-1 diagnostic accuracy:** 66.7% (2/3)
+- **Avg pipeline time:** ~94 seconds per case
+
+Full validation runs (50–100+ cases) are planned for the next session.
+
+**Files created:** `validation/__init__.py`, `validation/base.py`, `validation/harness_medqa.py`, `validation/harness_mtsamples.py`, `validation/harness_pmc.py`, `validation/run_validation.py`  
+**Files modified:** `.gitignore` (added `validation/data/` and `validation/results/`)
+
+---
+
+## Phase 10: Final Documentation Audit & Cleanup
+
+Performed a full accuracy audit of all 5 documentation files and `test_e2e.py`.
+
+**Issues found and fixed:**
+- README.md: step count said "5" in E2E table (fixed to 6), missing Conflict Detection row, missing `validation/` in project structure, missing validation section and test commands
+- architecture.md: Design Decision #1 said "5-step" (fixed to 6), Decision #4 said "Gemma in two roles" (fixed to four), no validation framework section
+- test_results.md: no external validation section, stale line count for test_e2e.py
+- DEVELOPMENT_LOG.md: Phase 7 said "(Current)", missing Phase 9 for validation framework
+- writeup_draft.md: referenced "confidence levels" (removed earlier), placeholder links, no validation methodology
+- test_e2e.py: no assertions on step count or conflict_detection step
+
+**Created:** `TODO.md` in project root with next-session action items for easy pickup by future contributors or AI instances.

README.md

@@ -55,7 +55,7 @@ See [docs/architecture.md](docs/architecture.md) for the full design document.
 
 ### Full Pipeline E2E Test (Chest Pain / ACS Case)
 
-All 5 pipeline steps completed successfully:
+All 6 pipeline steps completed successfully:
 
 | Step | Duration | Result |
 |------|----------|--------|
@@ -63,6 +63,7 @@ All 5 pipeline steps completed successfully:
 | Clinical Reasoning | 21.2 s | ACS correctly identified as top differential |
 | Drug Interaction Check | 11.3 s | Interactions queried against OpenFDA / RxNorm |
 | Guideline Retrieval (RAG) | 9.6 s | Relevant cardiology guidelines retrieved |
+| Conflict Detection | ~5 s | Guideline vs patient data comparison for omissions, contradictions, monitoring gaps |
 | Synthesis | 25.3 s | Comprehensive CDS report generated |
 
 ### RAG Retrieval Quality Test
@@ -84,6 +85,20 @@ Full results: [docs/test_results.md](docs/test_results.md)
 
 22 comprehensive clinical scenarios covering: ACS, AFib, heart failure, stroke, sepsis, anaphylaxis, polytrauma, DKA, thyroid storm, adrenal crisis, massive PE, status asthmaticus, GI bleeding, pancreatitis, status epilepticus, meningitis, suicidal ideation, neonatal fever, pediatric dehydration, hyperkalemia, acetaminophen overdose, and elderly polypharmacy with falls.
 
+### External Dataset Validation
+
+A validation framework tests the pipeline against real-world clinical datasets:
+
+| Dataset | Source | Cases Available | What It Tests |
+|---------|--------|-----------------|---------------|
+| **MedQA (USMLE)** | HuggingFace | 1,273 | Diagnostic accuracy — does the top differential match the correct answer? |
+| **MTSamples** | GitHub | ~5,000 | Parse quality & field completeness on real transcription notes |
+| **PMC Case Reports** | PubMed E-utilities | Dynamic | Diagnostic accuracy on published case reports with known diagnoses |
+
+Initial smoke test (3 MedQA cases): 100% parse success, 66.7% top-1 diagnostic accuracy.
+
+See [docs/test_results.md](docs/test_results.md) for full details and reproduction steps.
+
 ---
 
 ## RAG Clinical Guidelines Corpus
@@ -132,6 +147,12 @@ medgemma_impact_challenge/
 │   │   ├── test_clinical_cases.py    # 22 clinical scenario test suite
 │   │   ├── test_rag_quality.py       # RAG retrieval quality tests (30 queries)
 │   │   ├── test_poll.py              # Simple case poller utility
+│   │   ├── validation/               # External dataset validation framework
+│   │   │   ├── base.py               # Core framework (runners, scorers, utilities)
+│   │   │   ├── harness_medqa.py      # MedQA (USMLE) diagnostic accuracy harness
+│   │   │   ├── harness_mtsamples.py  # MTSamples parse quality harness
+│   │   │   ├── harness_pmc.py        # PMC Case Reports diagnostic harness
+│   │   │   └── run_validation.py     # Unified CLI runner
 │   │   └── app/
 │   │       ├── main.py               # FastAPI entry (CORS, routers, lifespan)
 │   │       ├── config.py             # Pydantic Settings (ports, models, dirs)
@@ -235,6 +256,13 @@ python test_clinical_cases.py --case em_sepsis    # Run one case
 python test_clinical_cases.py --specialty Cardio   # Run by specialty
 python test_clinical_cases.py                      # Run all cases
 python test_clinical_cases.py --report results.json  # Save results
+
+# External dataset validation (no backend needed — calls orchestrator directly)
+python -m validation.run_validation --fetch-only          # Download datasets only
+python -m validation.run_validation --medqa --max-cases 5  # 5 MedQA cases
+python -m validation.run_validation --mtsamples --max-cases 5
+python -m validation.run_validation --pmc --max-cases 5
+python -m validation.run_validation --all --max-cases 10   # All 3 datasets
 ```
 
 ### Usage
@@ -257,6 +285,7 @@ python test_clinical_cases.py --report results.json  # Save results
 | RAG | ChromaDB, sentence-transformers (all-MiniLM-L6-v2) | Clinical guideline retrieval |
 | Drug Data | OpenFDA API, RxNorm / NLM API | Drug interactions, medication normalization |
 | Validation | Pydantic | Structured output validation across all pipeline steps |
+| External Validation | MedQA, MTSamples, PMC Case Reports | Diagnostic accuracy & parse quality benchmarking |
 
 ---
 

TODO.md

@@ -0,0 +1,109 @@
+# TODO — Next Session Action Items
+
+> **Last updated:** End of validation framework + documentation audit session.
+> **Read this first** if you're a new AI instance picking up this project.
+
+---
+
+## High Priority (Do Next)
+
+### 1. Run Full-Scale Validation (~2 hours total)
+
+The validation framework is built and tested with a 3-case smoke test. It needs a proper run:
+
+```bash
+cd src/backend
+
+# MedQA — 50 cases, ~45 min
+python -m validation.run_validation --medqa --max-cases 50 --seed 42 --delay 2
+
+# MTSamples — 50 cases, ~45 min
+python -m validation.run_validation --mtsamples --max-cases 50 --seed 42 --delay 2
+
+# PMC Case Reports — 10-20 cases (smaller pool), ~15-30 min
+python -m validation.run_validation --pmc --max-cases 20 --seed 42 --delay 2
+```
+
+Results save to `validation/results/`. After running, update:
+- `docs/test_results.md` Section 6 with real numbers (replace smoke test placeholder)
+- `docs/writeup_draft.md` validation methodology section with actual metrics
+- `README.md` "External Dataset Validation" section
+
+### 2. Update Writeup with Actual Validation Metrics
+
+`docs/writeup_draft.md` currently says "initial smoke test" and "in progress." Once full validation is done, replace with actual numbers (top-1 accuracy, parse success rates, etc.).
+
+### 3. Record a Demo Video
+
+The writeup says "Video: [To be recorded]". Record a ~3 min screencast showing:
+1. Pasting a patient case
+2. Watching the 6-step pipeline execute live
+3. Reviewing the CDS report (especially conflicts section)
+4. Showing validation results
+
+---
+
+## Medium Priority
+
+### 4. CI Gating on Validation Scores
+
+Add a GitHub Action or pre-commit check that runs a small validation suite (e.g., 5 MedQA cases) and fails if top-1 accuracy drops below a threshold. This prevents regressions.
+
+### 5. PMC Harness Improvements
+
+The PMC case fetcher currently gets ~5 cases per run. The limiting factor is title-based diagnosis extraction — many PubMed case report titles don't follow parseable patterns. Options:
+- Use the full-text XML API (not just abstracts) to extract "final diagnosis" from structured sections
+- Add more title regex patterns
+- Use the LLM to extract the diagnosis from the abstract itself (meta, but effective)
+
+### 6. Calibrated Uncertainty Indicators
+
+We deliberately removed numeric confidence scores (see Phase 8 in DEVELOPMENT_LOG.md). If revisiting uncertainty communication:
+- Consider evidence-strength indicators per recommendation instead of a single composite score
+- Look at conformal prediction or test-time compute approaches if fine-tuning
+- Do NOT add back uncalibrated float scores — the anchoring bias risk is real
+
+---
+
+## Low Priority / Future
+
+### 7. Model Upgrade Path
+
+Currently using `gemma-3-27b-it`. When available, evaluate:
+- MedGemma (medical-specific Gemma fine-tune) if released
+- Smaller/distilled models for latency reduction
+- Specialized models for individual pipeline steps (e.g., a parse-only model)
+
+### 8. EHR Integration Prototype
+
+Current input is manual text paste. A FHIR client could auto-populate patient data. This is a significant scope expansion but would dramatically increase real-world usability.
+
+### 9. Frontend Polish
+
+- Loading skeletons during pipeline execution
+- Dark mode
+- Export report as PDF
+- Mobile-responsive layout
+
+---
+
+## Project State Summary
+
+| Component | Status | Notes |
+|-----------|--------|-------|
+| Backend (6-step pipeline) | ✅ Complete | All steps working, conflict detection added |
+| Frontend (Next.js) | ✅ Complete | Real-time pipeline viz, CDS report with conflicts |
+| RAG (62 guidelines) | ✅ Complete | 30/30 quality test, 100% top-1 accuracy |
+| Conflict Detection | ✅ Complete | Integrated into pipeline, frontend, and docs |
+| Validation Framework | ✅ Built | Smoke-tested only — needs full-scale runs |
+| Documentation (5 files) | ✅ Audited | All docs updated and cross-checked |
+| test_e2e.py | ✅ Fixed | Now asserts 6 steps + conflict_detection |
+| GitHub | ✅ Pushed | `bshepp/clinical-decision-support-agent` (master) |
+
+**Key files:**
+- Backend entry: `src/backend/app/main.py`
+- Orchestrator: `src/backend/app/agent/orchestrator.py`
+- Validation CLI: `src/backend/validation/run_validation.py`
+- All docs: `README.md`, `docs/architecture.md`, `docs/test_results.md`, `docs/writeup_draft.md`, `DEVELOPMENT_LOG.md`
+
+**Dev ports:** Backend = 8002 (not 8000 — zombie process issue), Frontend = 3000

docs/architecture.md

@@ -247,13 +247,13 @@ All pipeline data is strongly typed via Pydantic models in `schemas.py` (~280 li
 
 ## Key Design Decisions
 
-1. **Custom orchestrator over LangChain/LlamaIndex** — Simpler, more transparent, easier to debug. We control the pipeline loop explicitly. No framework overhead for a sequential 5-step pipeline.
+1. **Custom orchestrator over LangChain/LlamaIndex** — Simpler, more transparent, easier to debug. We control the pipeline loop explicitly. No framework overhead for a sequential 6-step pipeline.
 
 2. **WebSocket for agent activity** — The frontend shows each step as it happens (parsing → reasoning → checking → retrieving → synthesizing). This real-time visibility is critical for clinician trust.
 
 3. **Structured outputs everywhere** — Every tool returns a Pydantic model. The synthesis agent receives structured data, not messy text. This ensures consistency and enables frontend rendering.
 
-4. **Gemma in two roles** — As the clinical reasoning engine (Step 2) AND as the synthesis engine (Step 5). The same model reasons about the case and then integrates all tool outputs into a coherent report.
+4. **Gemma in four roles** — As the patient parser (Step 1), clinical reasoning engine (Step 2), conflict detector (Step 5), and synthesis engine (Step 6). The same model extracts structured data, reasons about the case, identifies guideline-vs-patient conflicts, and integrates all tool outputs into a coherent report.
 
 5. **Graceful degradation** — If a tool fails (e.g., OpenFDA is down), the agent continues with available information and notes the gap in the final report.
 
@@ -286,3 +286,32 @@ All configuration lives in `config.py` (Pydantic Settings) and `.env`:
 - **Single-model:** Uses only Gemma 3 27B IT. Could benefit from specialized models for different steps.
 - **Guideline currency:** Guidelines are a static snapshot. A production system would need automated updates.
 - **No EHR integration:** Input is manual text paste. A production system would integrate with EHR FHIR APIs.
+
+---
+
+## Validation Framework
+
+The project includes an external dataset validation framework that tests the full pipeline against real-world clinical data — bypassing the HTTP server and calling the `Orchestrator` directly.
+
+### Datasets
+
+| Dataset | Source | Cases | What It Measures |
+|---------|--------|-------|------------------|
+| **MedQA (USMLE)** | HuggingFace (`GBaker/MedQA-USMLE-4-options`) | 1,273 | Diagnostic accuracy — top-1, top-3, mentioned |
+| **MTSamples** | GitHub (`socd06/medical-nlp`) | ~5,000 | Parse quality, field completeness, specialty alignment |
+| **PMC Case Reports** | PubMed E-utilities (esearch + efetch) | Dynamic | Diagnostic accuracy on published cases with known diagnoses |
+
+### Architecture
+
+```
+validation/
+├── base.py               # ValidationCase, ValidationResult, ValidationSummary
+│                         # run_cds_pipeline() — direct Orchestrator invocation
+│                         # fuzzy_match(), diagnosis_in_differential()
+├── harness_medqa.py      # Fetch from HuggingFace, extract vignettes, score diagnostics
+├── harness_mtsamples.py  # Fetch CSV, stratified sampling, score parse quality
+├── harness_pmc.py        # PubMed E-utilities, title-based diagnosis extraction
+└── run_validation.py     # Unified CLI: --medqa --mtsamples --pmc --all --max-cases N
+```
+
+All datasets are cached locally in `validation/data/` (gitignored). Results are saved to `validation/results/` (also gitignored).

docs/test_results.md

@@ -186,7 +186,7 @@ python test_clinical_cases.py --quiet
 
 | File | Lines | Purpose |
 |------|-------|---------|
-| `test_e2e.py` | 57 | Submit chest pain case, poll for completion, validate all 6 steps |
+| `test_e2e.py` | ~60 | Submit chest pain case, poll for completion, validate all 6 steps |
 | `test_clinical_cases.py` | ~400 | 22 clinical cases with keyword validation, CLI flags for filtering |
 | `test_rag_quality.py` | ~350 | 30 RAG retrieval queries with expected guideline IDs, relevance scoring |
 | `test_poll.py` | ~30 | Utility: poll a case ID until completion |
@@ -194,3 +194,60 @@ python test_clinical_cases.py --quiet
 ### Dependencies for Testing
 
 Tests use only the standard library + `httpx` (for REST calls) and the backend's own modules (for RAG tests). No additional test frameworks required beyond what's in `requirements.txt`.
+
+---
+
+## 6. External Dataset Validation
+
+**Test files:** `src/backend/validation/` (package)  
+**What it tests:** Full pipeline diagnostic accuracy and parse quality against real-world clinical datasets.  
+**Methodology:** Each harness fetches a public dataset, converts cases into patient narratives, runs them through the `Orchestrator` directly (no HTTP server), and scores the output against known ground truth.
+
+### Datasets
+
+| Dataset | Source | Cases Available | Metrics |
+|---------|--------|-----------------|--------|
+| **MedQA (USMLE)** | HuggingFace (`GBaker/MedQA-USMLE-4-options`) | 1,273 test cases | top-1, top-3, mentioned diagnostic accuracy |
+| **MTSamples** | GitHub (`socd06/medical-nlp`) | ~5,000 transcription notes | parse success, field completeness, specialty alignment |
+| **PMC Case Reports** | PubMed E-utilities (esearch + efetch) | Dynamic (curated queries) | diagnostic accuracy vs published diagnosis |
+
+### Initial Results (Smoke Test — 3 MedQA Cases)
+
+| Metric | Value |
+|--------|-------|
+| Cases run | 3 |
+| Parse success | 100% (3/3) |
+| Top-1 diagnostic accuracy | 66.7% (2/3) |
+| Top-3 diagnostic accuracy | 66.7% (2/3) |
+| Avg pipeline time | ~94 s per case |
+
+> **Note:** This is a smoke test only. A full validation run (50–100 cases per dataset) is planned but takes ~45 min per dataset.
+
+### How to Reproduce
+
+```bash
+cd src/backend
+
+# Fetch datasets only (no pipeline runs)
+python -m validation.run_validation --fetch-only
+
+# Run MedQA validation (N cases)
+python -m validation.run_validation --medqa --max-cases 10
+
+# Run MTSamples validation
+python -m validation.run_validation --mtsamples --max-cases 10
+
+# Run PMC Case Reports validation
+python -m validation.run_validation --pmc --max-cases 5
+
+# Run all 3 datasets
+python -m validation.run_validation --all --max-cases 10
+
+# Additional flags:
+#   --seed 42          Reproducible random sampling
+#   --delay 2          Seconds between cases (rate limiting)
+#   --no-drugs         Skip drug interaction step
+#   --no-guidelines    Skip guideline retrieval step
+```
+
+Results are saved to `validation/results/` as timestamped JSON files.

docs/writeup_draft.md

@@ -124,6 +124,22 @@ No fine-tuning was performed in the current version. The base `gemma-3-27b-it` m
 - **Scalability:** FastAPI + uvicorn supports async request handling. For high-throughput deployment, add worker processes and a task queue (e.g., Celery).
 - **EHR integration:** Current input is manual text paste. A production system would integrate with EHR systems via FHIR APIs for automatic patient data extraction.
 
+### Validation methodology
+
+The project includes an external dataset validation framework (`src/backend/validation/`) that tests the full pipeline against real-world clinical data:
+
+| Dataset | Source | What It Tests |
+|---------|--------|---------------|
+| **MedQA (USMLE)** | HuggingFace (1,273 test cases) | Diagnostic accuracy — does the pipeline's top differential match the USMLE correct answer? |
+| **MTSamples** | GitHub (~5,000 medical transcriptions) | Parse quality, field completeness, specialty alignment on real clinical notes |
+| **PMC Case Reports** | PubMed E-utilities (dynamic) | Diagnostic accuracy on published case reports with known diagnoses |
+
+The validation harness calls the `Orchestrator` directly (no HTTP server), enabling rapid batch testing. Each dataset has a dedicated harness that fetches data, converts it to patient narratives, runs the pipeline, and scores the output against ground truth.
+
+**Initial smoke test (3 MedQA cases):** 100% parse success, 66.7% top-1 diagnostic accuracy, ~94 s avg per case.
+
+Full-scale validation (50–100+ cases per dataset) is in progress.
+
 **Practical usage:**
 
 In a real clinical setting, the system would be used at the point of care:
@@ -138,13 +154,13 @@ In a real clinical setting, the system would be used at the point of care:
    - Suggested next steps (immediate, short-term, long-term)
 5. The clinician reviews the recommendations and incorporates them into their clinical judgment
 
-The system is explicitly designed as a **decision support** tool, not a decision-making tool. All recommendations include confidence levels and caveats. The clinician retains full authority over patient care.
+The system is explicitly designed as a **decision support** tool, not a decision-making tool. All recommendations include caveats and limitations. The clinician retains full authority over patient care.
 
 ---
 
 **Links:**
 
 - Video: [To be recorded]
-- Code Repository: [GitHub link to be added]
+- Code Repository: [github.com/bshepp/clinical-decision-support-agent](https://github.com/bshepp/clinical-decision-support-agent)
 - Live Demo: [To be deployed]
 - Hugging Face Model: N/A (using base Gemma 3 27B IT)

src/backend/test_e2e.py

@@ -50,6 +50,28 @@ async def main():
             dur = s.get("duration_ms", "?")
             print(f"  {s['step_id']:12s} {s['status']:10s} ({dur}ms) {err[:100] if err else 'OK'}")
 
+        # --- Assertions ---
+        # Verify all 6 pipeline steps are present
+        step_ids = [s["step_id"] for s in steps]
+        expected_steps = [
+            "parse_patient",
+            "clinical_reasoning",
+            "drug_interactions",
+            "guideline_retrieval",
+            "conflict_detection",
+            "synthesis",
+        ]
+        assert len(steps) == 6, f"Expected 6 steps, got {len(steps)}: {step_ids}"
+        for exp in expected_steps:
+            assert exp in step_ids, f"Missing expected step: {exp}"
+
+        # Verify all steps completed (not failed)
+        failed = [s["step_id"] for s in steps if s["status"] == "failed"]
+        assert not failed, f"Steps failed: {failed}"
+
+        completed = [s["step_id"] for s in steps if s["status"] == "completed"]
+        print(f"\n✓ All {len(completed)}/6 pipeline steps completed successfully.")
+
         # Print report
         report = result.get("report")
         if report: