PromptLens

A lightweight, open-source evaluation harness for prompts, LLMs, and agent workflows.

PromptLens runs golden test sets against multiple models, scores outputs using LLM-as-judge, tracks cost and latency, and generates beautiful visual reports—all locally, with no cloud dependencies.

The HTML report from a two-model run: summary cards, side-by-side model comparison with score distributions, and per-test-case responses with judge scores and explanations.

Features

Multi-Provider Support - Test Anthropic (Claude), OpenAI (GPT), Google (Gemini), You.com, and local models (Ollama, LM Studio)
Tool/Function Calling Evaluation - Test tool usage with automatic + LLM judge scoring across 5 criteria
LLM-as-Judge Scoring - Automated evaluation using another LLM with configurable criteria
Cost & Latency Tracking - Monitor per-query costs and response times across models
Beautiful Reports - Interactive HTML reports with charts, comparisons, and detailed results
Multiple Export Formats - HTML, JSON, CSV, Markdown, and JUnit XML outputs
CI-Native Quality Gates - JUnit XML reports plus a --fail-under score gate that fails the build on quality regressions
Cross-Run Comparison - Diff any two runs by test case and model with score, cost, and latency deltas, plus a --fail-on-regression CI gate
Parallel Execution - Async execution with configurable concurrency and retry logic
Portable & Local - No cloud backend, all data stays on your machine
Easy to Extend - Plugin architecture for custom providers, judges, and exporters

Installation

Install from source:

git clone https://github.com/brainsparker/PromptLens.git
cd PromptLens
pip install -e .

# Or with Poetry
poetry install

A PyPI release is planned. Until then, installing from source is the supported path.

Requirements

Python 3.9+ (3.10+ recommended)
An API key for at least one provider you want to evaluate (Anthropic, OpenAI, Google, You.com), or a local model via Ollama or LM Studio with no key at all

You can explore the CLI before setting any keys: promptlens validate examples/golden_sets/customer_support.yaml checks a golden set entirely offline.

Quick Start

1. Set up API keys

# Copy the example environment file
cp .env.example .env

# Edit .env and add your API keys
export ANTHROPIC_API_KEY=sk-ant-...
export OPENAI_API_KEY=sk-...
export GOOGLE_API_KEY=...
export YOU_API_KEY=...

2. Run an example evaluation

# Run the basic customer support evaluation
promptlens run examples/configs/basic_config.yaml

3. View the results

# Open the HTML report (path shown in CLI output)
open promptlens_results/latest/report.html

That's it! You've just evaluated an LLM against a golden test set.

→ Read the Complete Getting Started Guide for detailed workflows and use cases.

Usage

Creating a Golden Set

Golden sets are test cases in YAML or JSON format:

name: "My Test Set"
description: "Testing customer support responses"
version: "1.0"

test_cases:
  - id: "test-001"
    query: "How do I reset my password?"
    expected_behavior: "Provide clear step-by-step instructions"
    category: "account_management"
    tags: ["password", "account"]

  - id: "test-002"
    query: "What's your refund policy?"
    expected_behavior: "Explain the 30-day refund policy clearly"
    category: "policy"
    tags: ["refund", "billing"]

Save as my_tests.yaml.

Creating a Configuration File

golden_set: ./my_tests.yaml

models:
  - name: "Claude 3.5 Sonnet"
    provider: anthropic
    model: claude-3-5-sonnet-20241022
    temperature: 0.7
    max_tokens: 1024

  - name: "GPT-4 Turbo"
    provider: openai
    model: gpt-4-turbo-preview
    temperature: 0.7
    max_tokens: 1024

judge:
  provider: anthropic
  model: claude-3-5-sonnet-20241022
  temperature: 0.3

execution:
  parallel_requests: 3
  retry_attempts: 3

output:
  directory: ./promptlens_results
  formats: [html, json, csv, md]
  run_name: "My Evaluation"

Save as my_config.yaml.

Running the Evaluation

promptlens run my_config.yaml

CLI Commands

# Run evaluation
promptlens run <config.yaml>

# Validate a golden set
promptlens validate <golden_set.yaml>

# List past runs
promptlens list-runs

# Compare two runs and detect regressions
promptlens compare <baseline_run_id> <candidate_run_id>

# Export a run to different format
promptlens export <run_id> --format html

# Get help
promptlens --help

Supported Providers

Anthropic (Claude)

models:
  - name: "Claude 3.5 Sonnet"
    provider: anthropic
    model: claude-3-5-sonnet-20241022
    temperature: 0.7
    max_tokens: 1024

Supported Models:

claude-3-5-sonnet-20241022
claude-3-opus-20240229
claude-3-haiku-20240307

OpenAI (GPT)

models:
  - name: "GPT-4 Turbo"
    provider: openai
    model: gpt-4-turbo-preview
    temperature: 0.7
    max_tokens: 1024

Supported Models:

gpt-4-turbo-preview, gpt-4, gpt-3.5-turbo
o1-preview, o1-mini

Google (Gemini)

models:
  - name: "Gemini Pro"
    provider: google
    model: gemini-1.5-pro
    temperature: 0.7
    max_tokens: 1024

Supported Models:

gemini-1.5-pro, gemini-1.5-flash
gemini-pro

You.com AI

models:
  - name: "You.com GPT-4"
    provider: you
    model: gpt-4
    temperature: 0.7
    max_tokens: 1024

Supported Models:

gpt-4, claude-3-5-sonnet, llama-3-70b
Any model available through You.com's unified API

Setup:

Get API key from: https://api.you.com/
Set YOU_API_KEY environment variable
Use model names as specified in You.com docs

Local Models (Ollama, LM Studio)

models:
  - name: "Local Llama"
    provider: http
    model: llama3.1:8b
    endpoint: "http://localhost:11434/api/generate"
    temperature: 0.7
    max_tokens: 1024

Setup:

Install Ollama
Pull a model: ollama pull llama3.1:8b
Use the http provider with the endpoint URL

Configuration Reference

Models

models:
  - name: "Display Name"           # Human-readable name
    provider: anthropic             # anthropic, openai, google, http
    model: model-identifier         # Model ID
    endpoint: "http://..."          # Optional, for HTTP provider
    temperature: 0.7                # 0.0-1.0
    max_tokens: 1024                # Maximum output tokens
    additional_params:              # Provider-specific params
      # endpoint under additional_params is still accepted for backwards compatibility
      custom_option: "value"

Judge

judge:
  provider: anthropic               # Provider for judge model
  model: claude-3-5-sonnet-20241022 # Judge model (typically Claude or GPT-4)
  temperature: 0.3                  # Lower for consistent scoring
  criteria:                         # Evaluation criteria
    - accuracy
    - helpfulness
    - safety

Execution

execution:
  parallel_requests: 3              # Concurrent API calls
  retry_attempts: 3                 # Retries for failed requests
  timeout_seconds: 60               # Request timeout

Output

output:
  directory: ./promptlens_results   # Output directory
  formats:                          # Export formats
    - html                          # Interactive report
    - json                          # Raw JSON data
    - csv                           # Flattened spreadsheet
    - md                            # Markdown summary
    - junit                         # JUnit XML for CI test reporting
  run_name: "My Evaluation"         # Display name

CI/CD Integration

PromptLens speaks the language your CI system already understands: JUnit XML test reports and exit codes.

Add junit to your output formats, then gate the build on judge scores:

promptlens run config.yaml --fail-under 3.5

Each golden-set test case becomes a JUnit test case (one test suite per model).
A test case scoring below the threshold is reported as a failure, a model API error as an error, and an unjudged case as skipped.
If any model's average judge score falls below --fail-under, the command exits with code 2, failing the pipeline. Exit code 1 is reserved for run errors, so CI can tell quality regressions apart from infrastructure failures.

Example GitHub Actions step:

- name: Run prompt evals
  run: promptlens run config.yaml --fail-under 3.5

- name: Publish eval report
  uses: mikepenz/action-junit-report@v5
  if: always()
  with:
    report_paths: "promptlens_results/*/junit.xml"

The same junit.xml works with GitLab (artifacts:reports:junit), Jenkins, CircleCI, and any other JUnit-compatible report viewer.

Comparing Runs (Regression Detection)

An eval tells you how the system does today. What breaks products is the prompt tweak or model swap that quietly makes some answers worse while the average holds. promptlens compare diffs two stored runs, case by case:

# Establish a baseline
promptlens run config.yaml
# ... change your prompt or model ...
promptlens run config.yaml

# Compare the last known-good run against the newest one
promptlens compare <baseline_run_id> latest

Results are paired by test case and model, then classified:

Regressed - the judge score dropped by at least the threshold (default 1 point on the 1-5 scale), the case newly errored, or the case lost its judge score
Improved - the score rose by at least the threshold, or a baseline error was resolved
Unchanged - the delta stayed within the threshold
Incomparable - the baseline case had no judge score

Test cases and models present in only one run are reported as suite drift and excluded from the comparison, so editing your golden set never produces false regressions.

Useful options:

# Fail CI when anything regressed (exit code 2, matching --fail-under semantics)
promptlens compare baseline-run latest --fail-on-regression

# Only flag drops of 2+ points
promptlens compare baseline-run latest --threshold 2

# Write reports: markdown for PR comments, JSON for tooling
promptlens compare baseline-run latest --markdown comparison.md --json comparison.json

Example GitHub Actions step:

- name: Run prompt evals
  run: promptlens run config.yaml

- name: Compare against baseline
  run: |
    promptlens compare "$BASELINE_RUN_ID" latest \
      --fail-on-regression --markdown comparison.md

- name: Upload comparison report
  uses: actions/upload-artifact@v4
  if: always()
  with:
    name: eval-comparison
    path: comparison.md

Examples

Basic Single Model Evaluation

promptlens run examples/configs/basic_config.yaml

Multi-Model Comparison

promptlens run examples/configs/multi_model.yaml

Local vs Cloud Models

promptlens run examples/configs/local_model.yaml

See examples/README.md for more details.

Report Features

The HTML report includes:

Summary Dashboard - Total cost, time, test cases, and models
Model Comparison Cards - Side-by-side metrics for each model
Score Distribution Charts - Visual breakdown of scores (1-5)
Detailed Test Results - Expandable cards for each test case with:
- Original query and expected behavior
- Model responses
- Judge scores and explanations
- Cost and latency per response
Dark Theme - Easy on the eyes with accent colors for data
Responsive Design - Works on desktop and mobile

Use Cases

Prompt Iteration

Test different prompt versions to find the best performer:

Create test cases for your use case
Update your prompt
Run evaluation
Compare scores with previous run
Iterate

Model Selection

Compare models before committing to one:

Add multiple models to config
Run the same test set against all models
Compare costs, latency, and quality scores
Make data-driven decision

Regression Testing

Ensure prompt changes don't break existing behavior:

Maintain a golden set of important test cases
Run before and after making changes
Run promptlens compare <baseline> latest to catch regressions case by case
Gate CI/CD with --fail-on-regression

Agent Workflow Testing

Evaluate multi-step agent workflows:

Create test cases for agent tasks
Implement agent logic
Evaluate with PromptLens
Iterate on tools and prompting

Tool/Function Calling Evaluation

Test how well models use tools and functions:

Define tools with JSON schema
Specify expected tool calls
Evaluate parameter correctness, tool selection, and efficiency
Get multi-criteria scores with detailed feedback

Example test case:

- id: "tool-001"
  query: "What's the weather in San Francisco?"
  expected_behavior: "Call get_weather with location='San Francisco'"
  evaluation_mode: "tool_and_answer"

  tools:
    - name: "get_weather"
      description: "Get current weather"
      parameters:
        location:
          type: "string"
          required: true

  expected_tool_calls:
    - name: "get_weather"
      arguments:
        location: "San Francisco"

Evaluation includes:

Automatic comparison (expected vs actual tool calls)
Parameter correctness scoring (1-5)
Tool selection accuracy (1-5)
Tool usage efficiency (1-5)
Final answer quality (1-5)

Supported providers: Anthropic Claude, OpenAI GPT (other providers will warn gracefully)

Try it:

promptlens run examples/configs/tool_evaluation.yaml

See examples/golden_sets/tool_calling.yaml for complete examples.

Advanced Usage

Custom Judge Prompts

judge:
  provider: anthropic
  model: claude-3-5-sonnet-20241022
  custom_prompt: |
    You are evaluating a coding assistant's response.

    Query: {query}
    Expected: {expected_behavior}
    Response: {response}

    Rate 1-5 based on code correctness, efficiency, and style.

    SCORE: [1-5]
    EXPLANATION: [Your reasoning]

Provider-Specific Parameters

models:
  - name: "GPT-4 with JSON mode"
    provider: openai
    model: gpt-4-turbo-preview
    additional_params:
      response_format: {"type": "json_object"}

Parallel Execution Tuning

execution:
  parallel_requests: 10   # Higher for faster execution
  retry_attempts: 5       # More retries for flaky APIs
  timeout_seconds: 120    # Longer timeout for slow models

Architecture

promptlens/
├── models/          # Pydantic data models
├── providers/       # LLM provider implementations
├── loaders/         # Golden set loaders (JSON/YAML)
├── runners/         # Orchestration and execution
├── judges/          # LLM-as-judge scoring
├── exporters/       # Report generators
├── utils/           # Utilities (cost, retry, timing)
└── templates/       # HTML report templates

Key Design Principles:

Plugin Architecture - Easy to add new providers, judges, exporters
Async-First - Parallel execution for speed
Type-Safe - Pydantic models throughout
Modular - Each component is independent and testable

Extending PromptLens

Custom Provider

from promptlens.providers.base import BaseProvider
from promptlens.models.result import ModelResponse

class MyProvider(BaseProvider):
    async def generate(self, prompt: str, **kwargs) -> ModelResponse:
        # Your implementation
        pass

    def estimate_cost(self, prompt_tokens: int, completion_tokens: int) -> float:
        return 0.0

    @property
    def provider_name(self) -> str:
        return "my_provider"

# Register it
from promptlens.providers.factory import register_provider
register_provider("my_provider", MyProvider)

Custom Judge

from promptlens.judges.base import BaseJudge
from promptlens.models.result import JudgeScore

class RuleBasedJudge(BaseJudge):
    async def evaluate(self, test_case, model_response) -> JudgeScore:
        # Your scoring logic
        score = self.calculate_score(model_response.content)
        return JudgeScore(
            score=score,
            explanation="Rule-based evaluation",
            judge_model="rule-based",
            judge_provider="custom"
        )

Roadmap

Contributing

Contributions are welcome! Please:

Fork the repository
Create a feature branch
Make your changes
Add tests
Submit a pull request

See CONTRIBUTING.md for detailed guidelines.

License

MIT License - see LICENSE for details.

Acknowledgments

Inspired by the need for simple, local LLM evaluation tools
Built with Anthropic, OpenAI, and Google AI APIs
Uses Rich for beautiful CLI output
Charts powered by Chart.js

Support

Issues: https://github.com/brainsparker/PromptLens/issues
Discussions: https://github.com/brainsparker/PromptLens/discussions
Email: sparker@example.com

Made with ❤️ for the LLM developer community

Latest commit

History

Folders and files

Repository files navigation

PromptLens

Features

Installation

Requirements

Quick Start

1. Set up API keys

2. Run an example evaluation

3. View the results

Usage

Creating a Golden Set

Creating a Configuration File

Running the Evaluation

CLI Commands

Supported Providers

Anthropic (Claude)

OpenAI (GPT)

Google (Gemini)

You.com AI

Local Models (Ollama, LM Studio)

Configuration Reference

Models

Judge

Execution

Output

CI/CD Integration

Comparing Runs (Regression Detection)

Examples

Basic Single Model Evaluation

Multi-Model Comparison

Local vs Cloud Models

Report Features

Use Cases

Prompt Iteration

Model Selection

Regression Testing

Agent Workflow Testing

Tool/Function Calling Evaluation

Advanced Usage

Custom Judge Prompts

Provider-Specific Parameters

Parallel Execution Tuning

Architecture

Extending PromptLens

Custom Provider

Custom Judge

Roadmap

Contributing

License

Acknowledgments

Support

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages