Omni-Agent: Financial RAG

A production-grade, senior-level AI system designed for intelligent document ingestion, retrieval, and synthesis.

Features

• Multi-Agent Orchestration: Built with LangGraph, utilizing distinct Researcher and Reviewer personas for high-fidelity outputs.
• Advanced Retrieval: Implements Hybrid Search (BM25 Sparse + Dense Vectors) and Cross-Encoder Reranking (BAAI/bge-reranker-base) for extreme precision.
• Agentic Self-Reflection (Corrective RAG): Features a Relevance Grader node that intercepts poor retrieval and autonomously rewrites queries for optimization.
• MLOps & Observability: Traced end-to-end via Arize Phoenix, containerized with Docker, and evaluated via RAGAS.

Project Documentation

• Architecture Case Study: See ARCHITECTURE.md for a detailed breakdown of design tradeoffs and known retrieval failure modes.

Running the Application Locally

1. Prerequisites

Ensure you have Docker, Node.js (v18+), and Python 3.10+ installed. You must configure your .env file at the project root. (See .env.example).

2. Infrastructure Setup

Spin up the local Qdrant vector database:

docker-compose up -d

3. Backend (FastAPI + LangGraph)

Activate the virtual environment, install dependencies, and run the API:

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
uvicorn api.main:app --host 0.0.0.0 --port 8000

The API will be available at http://localhost:8000.

4. Frontend (Next.js)

Open a new terminal, install dependencies, and start the frontend:

cd frontend
npm install
npm run dev

The web interface will be available at http://localhost:3000.

Reproducibility & Evaluation

To run the automated RAGAS Evaluation Harness and view the quantitative metrics (Faithfulness, Answer Relevance, Context Precision):

1. Setup Environment: Ensure you have your OpenAI key exported (required for LLM-as-a-judge evaluation).

   export OPENAI_API_KEY="sk-..."
   pip install -r requirements-test.txt

2. Run Evaluation Harness:

   python tests/evaluate_rag.py

   This will output the final scores to the terminal and save a detailed rag_evaluation_results.csv file for CI/CD tracking.

MLOps: What Isn't Done Yet

While this project demonstrates advanced Retrieval-Augmented Generation (RAG) capabilities, the following MLOps practices are required to achieve full production readiness:

1. Model Registry & Versioning:
   • Currently, embedding models and LLM configurations are hardcoded via environment variables. We need a registry (e.g., MLflow, Weights & Biases) to version prompts, models, and retrieval hyperparameters systematically.
2. Automated Continuous Training (CT) / Data Pipeline:
   • The ingestion pipeline (ingestion/chunker.py) is run manually. A production system requires an orchestration tool (like Airflow or Prefect) to automatically fetch daily financial reports, chunk, embed, and upsert them to Qdrant without human intervention.
3. Data Drift & Concept Drift Detection:
   • We lack automated alerts for when user queries fundamentally change over time (concept drift) or when the financial corpus structure changes, which could break the unstructured chunking logic (data drift).
4. CI/CD for Machine Learning (CML):
   • While we have an evaluation harness (tests/evaluate_rag.py), it is not integrated into a CI/CD pipeline (e.g., GitHub Actions). Commits should automatically trigger RAGAS evaluations on a golden dataset, blocking merges if context precision or faithfulness drop below a defined threshold.
5. Infrastructure-as-Code (IaC) & Scalability:
   • The infra/ folder contains preliminary Helm charts and Terraform, but they are not fully wired to a GitOps operator like ArgoCD for automated multi-environment (Staging/Prod) deployments. Qdrant is currently running as a single local node, not a highly-available cluster.
6. A/B Testing Infrastructure:
   • There is no framework to route a percentage of live user traffic to different prompt templates or embedding models to measure real-world user satisfaction metrics (like implicit thumbs up/down).