LLMEval is a comprehensive evaluation system for assessing Large Language Models (LLMs) on mathematical reasoning, code generation, and general knowledge benchmarks. It supports both online (API-based) and offline (local inference) modes with built-in answer verification and multiple-choice evaluation.
- Multiple Inference Backends: Support for vLLM (GPU/NPU) and SGLang
- Flexible Evaluation Modes: Online (API), offline (local), and MC (loglikelihood/generate)
- Rich Benchmark Coverage: 14 benchmarks — AIME 2024/2025/2026, MATH-500, GSM8K, GPQA-Diamond, HMMT-25, HumanEval, MBPP, MMLU, MMLU-Pro, C-Eval
- Code Evaluation: HumanEval / MBPP with guarded subprocess execution and pass@k scoring
- MC Scoring: Answer-token or full-continuation loglikelihood accuracy, few-shot dedup
- One-Click Pipeline: Shell scripts for end-to-end inference → scoring
- Resume Capability: Automatically resume interrupted evaluations
- Verification Support: Built-in answer extraction and correctness verification
We have successfully reproduced various open-source model results on the AIME 2024 & AIME 2025 benchmarks.
For benchmarks like AIME24, which contains only 30 problems, it is crucial to sample multiple responses as this can introduce high variance across repeated runs. All results below use 64 samples per problem to ensure stability.
| Dataset | LLMEval | Official Report |
|---|---|---|
| AIME24 | 70.625 | 72.6 |
| AIME25 | 55.052 | 59.0 |
| MATH-500 | 93.2 | 94.3 |
| Dataset | LLMEval | Official Report |
|---|---|---|
| AIME24 | 78.65 | 79.5 |
| AIME25 | 67.22 | 69.5 |
| Dataset | LLMEval | Official Report |
|---|---|---|
| AIME24 | 81.25 | 82.2 |
| AIME25 | 72.66 | 73.3 |
| Dataset | LLMEval | Official Report |
|---|---|---|
| AIME24 | 70.41 | 69.0 |
| AIME25 | 59.16 | 53.3 |
| Software | Version |
|---|---|
| Python | >= 3.10 |
| torch | >= 2.0 |
For Huawei Ascend NPU users:
- CANN >= 8.1.RC1
- torch_npu >= 2.5.1
- vllm-ascend >= 0.7.3.post1
git clone https://github.com/jianzhnie/LLMEval.git
# Or use gitee mirror: git clone https://gitee.com/jianzhnie/LLMEval.git
cd LLMEval
pip install -e .For GPU users:
pip install vllm>=0.7.0For Huawei Ascend NPU users:
# Install vllm
git clone -b v0.7.3 --depth 1 https://github.com/vllm-project/vllm.git
cd vllm
pip install -r requirements-build.txt
VLLM_TARGET_DEVICE=empty pip install -e .
# Install vllm-ascend
git clone -b v0.7.3.post1 --depth 1 https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
export COMPILE_CUSTOM_KERNELS=1
python setup.py installFirst, start the vLLM server with the following command:
model_path="Qwen/QwQ-32B" # or the path where the model is located
model_name="Qwen/QwQ-32B"
num_gpus=8
max_model_len=32768 # ✅ supports 32k context
gpu_memory_utilization=0.9 # ✅ improves memory utilization
vllm serve \
--model $model_path \
--trust-remote-code \
--served-model-name $model_name \
--tensor-parallel-size $num_gpus \
--gpu-memory-utilization $gpu_memory_utilization \
--max-model-len $max_model_len \
--enforce-eager \
--port 8090Adjust the tensor_parallel_size parameter based on your available devices. Please refer to the script for more details.
Optionally, start an SGLang server/router. Since the evaluation can take days, we also suggest using SGLang with data parallelism to accelerate it. Refer to the SGLang documentation for more details.
# Use router to support better data parallelism
python -m sglang_router.launch_server --model-path Qwen/QwQ-32B --dp-size 4 --host=0.0.0.0 --port=30000Adjust the dp_size parameter based on your available devices. Also adjust the port in the following commands.
After starting the vLLM service, run the inference script to generate responses.
output_dir="./output/Qwen/QwQ-32B"
model_name="Qwen/QwQ-32B"
base_url="http://127.0.0.1:8090/v1"
n_samples=64 # Default sample size for aime24 and aime25
# Create output directory if it doesn't exist
mkdir -p "${output_dir}"
# --- Run Inference Tasks ---
# aime24 (repeated sample 64 times)
python -m llmeval.inference.online \
--input_file "./data/aime24.jsonl" \
--output_file "${output_dir}/aime24_bz${n_samples}.jsonl" \
--base_url "${base_url}" \
--model_name "${model_name}" \
--n_samples "${n_samples}" \
--system_prompt_type empty \
--max_workers 8
# aime25 (repeated sample 64 times)
python -m llmeval.inference.online \
--input_file "./data/aime25.jsonl" \
--output_file "${output_dir}/aime25_bz${n_samples}.jsonl" \
--base_url "${base_url}" \
--model_name "${model_name}" \
--n_samples "${n_samples}" \
--system_prompt_type empty \
--max_workers 8Please refer to the script for more details.
Note: We apply repeated sampling to reduce evaluation variance, but it may take a long time to complete (more than 8 hours depending on your device).
Math benchmarks — generation with math-verify scoring:
# Full pipeline (data → inference → scoring)
bash examples/longcat-flash/run_all.sh
# Quick test
BENCHMARKS=QUICK bash examples/longcat-flash/run_all.sh
# Hard problems
BENCHMARKS=HARD bash examples/longcat-flash/run_all.shMultiple-choice benchmarks — answer-token loglikelihood accuracy:
# 5-shot MMLU / C-Eval
N_SHOT=5 bash examples/longcat-flash/mc_infer.sh
bash examples/longcat-flash/mc_score.sh
# Complete-continuation scoring (requires a compatible Completions endpoint)
LOGLIKELIHOOD_MODE=continuation \
OUTPUT_DIR=./output/longcat-flash-continuation \
bash examples/longcat-flash/mc_infer.sh
# Generate mode (alternative)
MC_MODE=generate bash examples/longcat-flash/mc_infer.shUse a separate output directory when switching loglikelihood modes. See MC loglikelihood scoring for the algorithm and backend requirements.
Code benchmarks — generation with guarded subprocess execution + pass@k summary:
# HumanEval + MBPP
bash examples/longcat-flash/code_infer.sh
bash examples/longcat-flash/code_score.sh
# Pass@64 (64 samples, temperature 0.2)
N_SAMPLES=64 TEMPERATURE=0.2 bash examples/longcat-flash/code_infer.shOr run each math step separately:
bash examples/longcat-flash/math_infer.sh # inference
bash examples/longcat-flash/math_score.sh # scoringAlternatively, use the CLI directly:
# Math evaluation
python -m llmeval.evaluator \
--input_path "./output/longcat-flash/aime24_bz32.jsonl" \
--result_path "./output/longcat-flash/eval_score/aime24_bz32.json" \
--task_name "math_opensource/aime24" \
--max_workers 16
# Code evaluation (pass@k summary, 5s execution timeout)
python -m llmeval.evaluator \
--input_path "./output/longcat-flash/humaneval_bz1.jsonl" \
--result_path "./output/longcat-flash/eval_score/humaneval_bz1.json" \
--task_name "code_opensource/humaneval" \
--max_workers 32 \
--exec_timeout 5.0 \
--allow_unsafe_code--allow_unsafe_code executes model-generated Python. The built-in subprocess
guard limits time and common resources, blocks common network modules, and
clears environment variables, but it is not a security sandbox. Run code
benchmarks inside a locked-down container or sandbox with no network access, a
read-only host filesystem, a non-privileged user, and external resource limits.
Multi-sample math summaries include accuracy, problem_pass@k,
problem_majority@k, and per-problem correct_samples, observed_samples, and
sample_count. A problem with missing, failed, or timed-out samples is marked
incomplete and excluded from problem-level metrics, while its available samples
still contribute to sample-level accuracy.
Common parameters for both online and offline modes:
| Parameter | Default | Description |
|---|---|---|
--n_samples |
1 | Number of samples per problem |
--temperature |
0.6 | Sampling temperature |
--top_p |
0.95 | Nucleus sampling parameter |
--max_completion_tokens |
32768 | Maximum completion tokens to generate |
--seed |
0 | Generation seed |
--exec_timeout |
3.0 | Code execution timeout (seconds, code tasks only) |
Online mode specific:
| Parameter | Default | Description |
|---|---|---|
--base_url |
https://api.openai.com/v1 |
API server URL |
--model_name |
gpt-4o |
Model name for API |
--max_workers |
128 | Concurrent request threads |
--extra_body |
{} |
JSON object for provider-specific request extensions |
See OpenAI API parameters for parameter descriptions and Python, cURL, structured-output, tool-calling, and streaming examples.
Offline mode specific:
See the offline inference guide for a complete vLLM example.
| Parameter | Default | Description |
|---|---|---|
--model_name_or_path |
Required | Local model path or HuggingFace ID |
--tensor_parallel_size |
1 | GPU count for tensor parallelism |
--gpu_memory_utilization |
0.9 | GPU memory fraction |
--batch_size |
128 | Inference batch size |
--top_k |
40 | vLLM top-k sampling parameter |
--enable_thinking |
false | vLLM chat-template thinking option |
--skip_special_tokens |
true | Remove special tokens from generated text |
--repetition_penalty |
1.0 | vLLM repetition penalty |
| Category | Task Name | Scoring |
|---|---|---|
| Math | math_opensource/aime24 math_opensource/aime25 math_opensource/aime26 |
math-verify |
| Math | math_opensource/gsm8k math_opensource/math500 math_opensource/math math_opensource/hmmt25 |
math-verify |
| Science | math_opensource/gpqa_diamond math_opensource/hle_full |
math-verify |
| Code | code_opensource/humaneval code_opensource/mbpp |
guarded pass@k execution |
| Code | code_opensource/humaneval_plus code_opensource/mbpp_plus |
guarded pass@k execution |
| MC | mc_opensource/mmlu mc_opensource/mmlu_pro mc_opensource/ceval |
answer-token loglikelihood |
See docs/benchmark.md for dataset details and HuggingFace links.
If inference is interrupted, simply re-run the same command. The script automatically:
- Reads existing output file
- Counts completed samples per problem
- Continues from where it left off
New runs also write <output>.manifest.json. Before scoring, the evaluator uses
this sidecar to warn about missing or unexpected result rows. Evaluation continues
with the available samples, and failed inference samples remain absent from JSONL.
For contexts exceeding 32K tokens, use RoPE scaling:
vLLM:
vllm serve \
--model Qwen/Qwen3-8B \
--rope-scaling '{"rope_type":"yarn","factor":4.0}' \
--max-model-len 131072SGLang:
python -m sglang.launch_server \
--model-path Qwen/Qwen3-8B \
--json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0}}' \
--context-length 131072LLMEval/
├── llmeval/
│ ├── evaluator.py # Main orchestrator (math / mc / code)
│ ├── inference/ # Inference backends
│ │ ├── online.py # OpenAI-compatible API
│ │ ├── offline.py # vLLM local engine
│ │ └── mc.py # MC inference
│ ├── tasks/ # Task-specific scoring
│ │ ├── code_eval/ # Code (HumanEval / MBPP / pass@k)
│ │ │ ├── execute.py # Guarded subprocess execution
│ │ │ └── code_score.py # pass@k driver
│ │ ├── math_eval/ # Math (math-verify)
│ │ │ ├── math_score.py
│ │ │ └── utils_parser.py
│ │ └── mc_eval/ # MC (loglikelihood/generate)
│ │ └── mc_score.py
│ └── utils/ # Shared utilities
│ ├── config.py # Configuration dataclasses
│ ├── log.py # Logging
│ ├── retry.py # API retry logic
│ ├── prompts.py # System prompts
├── examples/ # Model-specific evaluation pipelines
│ ├── longcat-flash/ # One-click evaluation
│ │ ├── run_all.sh
│ │ ├── math_infer.sh # Math inference
│ │ ├── math_score.sh # Math scoring
│ │ ├── code_infer.sh # Code inference
│ │ ├── code_score.sh # Code scoring
│ │ ├── mc_infer.sh # MC inference
│ │ └── mc_score.sh # MC scoring
│ └── QwQ/ # QwQ-32B scripts (server, inference, scoring)
├── scripts/
│ ├── data_parallel_infer/ # Multi-node data parallel inference
│ └── data_process/ # Benchmark data preparation
│ ├── prepare_math_benchmarks.py
│ └── prepare_code_benchmarks.py
├── tests/ # Unit tests
├── data/ # Benchmark datasets
└── docs/ # Documentation
This project is licensed under the Apache License, Version 2.0.