Medicaid Sherlock

Medicaid fraud detection and investigation system: a DuckDB-powered pipeline over HHS T-MSIS claims data and a multi-agent framework (Claude + Perplexity + NPPES) for provider-level intelligence.

---

What it does

• Scripts pipeline (batch): Ingest 227M+ Medicaid claims, compute procedure benchmarks and provider summaries, run five anomaly-detection algorithms, enrich top-risk providers with NPPES and web research, and produce an executive briefing plus cross-referenced leads.
• Multi-agent (conversational): Investigate any provider on demand. The Investigator coordinates an Analyst (DuckDB + NPI + HCPCS), Researcher (Perplexity), Network Mapper (billing/servicing relationships), and Report Generator to produce structured investigation reports.

See docs/ARCHITECTURE.md for data-flow diagrams and how scripts and agents connect.

---

Requirements

• Python 3.10+
• Data: Place medicaid-provider-spending.parquet (or your T-MSIS-style spending file) in data/. Expected columns include BILLING_PROVIDER_NPI_NUM, SERVICING_PROVIDER_NPI_NUM, HCPCS_CODE, CLAIM_FROM_MONTH, TOTAL_PAID, TOTAL_CLAIMS, TOTAL_UNIQUE_BENEFICIARIES.
• APIs (optional but recommended for full flow):
  • Anthropic API key — for 05 deep analysis narrative, 06 executive briefing, and multi-agent (Claude).
  • Perplexity API key — for web research in 05, 08, and the Researcher agent.
• NPPES: NPI lookups use the public NPPES Registry API (no key required).

---

Setup

1. Clone and virtualenv

git clone <repo-url>
cd medicaid-sherlock
python3 -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate

2. Install Python dependencies

pip install -r requirements.txt

3. Environment variables

Copy the example env file and set your keys (do not commit .env):

cp .env.example .env
# Edit .env and set ANTHROPIC_API_KEY and PERPLEXITY_API_KEY

Variable	Used by	Purpose
ANTHROPIC_API_KEY	05, 06, multi-agent	Claude (deep analysis, briefing, all agents)
PERPLEXITY_API_KEY	05, 08, multi-agent	Perplexity Sonar (web research)

4. Data

• Put your claims parquet in data/medicaid-provider-spending.parquet.
• HCPCS descriptions: the agent uses agent/hcpcs_codes.json (from CMS 2026 release). Populate or generate it if you need procedure lookups.

---

Scripts pipeline (run in order)

Run from the project root. Later steps depend on earlier outputs in data/.

Step	Script	Purpose
1	python scripts/01_explore.py	Schema and summary stats (row count, top providers, time range).
2	python scripts/02_enrich_npi.py	Extract unique NPIs to data/unique_npis.parquet (for bulk lookups if needed).
3	python scripts/03_benchmarks.py	Build procedure_benchmarks, provider_summaries, provider_monthly parquet files.
4	python scripts/04_anomaly_detection.py	Run 5 anomaly algorithms → data/all_anomalies.parquet.
5	python scripts/05_deep_analysis.py	Top-20 risk providers + NPPES + Perplexity → data/intelligence_report.json.
6	python scripts/06_generate_report.py	Claude turns intelligence_report.json into data/executive_briefing.md.
7	python scripts/07_cross_references.py	Cross-reference risk lists → data/cross_references.json.
8	python scripts/08_enrich_leads.py	Enrich cross-referenced leads with NPPES + Perplexity → data/enriched_leads.json.

Steps 5, 6, and 8 call external APIs; 1–4 and 7 are local DuckDB only.

---

Multi-agent (conversational)

From the project root, run the agent from the agent directory so imports resolve:

cd agent && python multi_agent.py

Then type natural-language requests, e.g. “Investigate NPI 1234567890” or “Who are the highest-risk providers for cost-per-claim?” The Investigator will delegate to the Analyst, Researcher, Network Mapper, and Report Generator and return a single answer. Type quit to exit.

Requires the same data/ outputs as the scripts (at least 03 and 04 run) so that all_anomalies, provider_summaries, procedure_benchmarks, and medicaid-provider-spending.parquet exist.

---

Project layout

medicaid-sherlock/
├── agent/                 # Multi-agent system
│   ├── multi_agent.py      # Investigator + Analyst, Researcher, Network Mapper, Report Generator
│   ├── tools.py            # DuckDB queries for Analyst
│   ├── hcpcs_loader.py     # HCPCS/CPT code lookup
│   └── hcpcs_codes.json    # Procedure code descriptions (e.g. CMS 2026)
├── data/                   # Input/output data (parquet, JSON, briefing)
│   ├── medicaid-provider-spending.parquet   # Input: T-MSIS-style claims
│   ├── procedure_benchmarks.parquet
│   ├── provider_summaries.parquet
│   ├── provider_monthly.parquet
│   ├── all_anomalies.parquet
│   ├── intelligence_report.json
│   ├── cross_references.json
│   ├── enriched_leads.json
│   └── executive_briefing.md
├── scripts/                # Batch pipeline
│   ├── 01_explore.py .. 08_enrich_leads.py
│   └── npi_lookup.py
├── docs/
│   ├── ARCHITECTURE.md     # Diagrams: scripts, agents, shared data
│   └── PROJECT_ANALYSIS.md # Suggestions and quality notes
├── dashboard/              # Optional Next.js UI (see dashboard/README if present)
├── .env.example
├── requirements.txt
└── README.md

---

Docs

• docs/ARCHITECTURE.md — How the scripts pipeline and multi-agent framework work and how they connect (Mermaid diagrams and file reference).
• docs/PROJECT_ANALYSIS.md — Analysis, improvement ideas, and known limitations.

---

License and disclaimer

This is an analytical tool to help investigators prioritize review. It does not determine fraud. Use outputs in accordance with your organization’s policies and applicable law.