Skip to content

overview.md

Latest commit

 

History

History
46 lines (33 loc) · 1.84 KB

overview.md

File metadata and controls

46 lines (33 loc) · 1.84 KB
Raw

Overview

Available Tasks

LLM Tasks

  • lm-arena-chat: Conversational prompts from LM Arena
  • gpqa: Graduate-level problem solving questions with reasoning
  • sourcegraph-fim: Fill-in-the-middle code completion

MLLM Tasks

  • image-chat: Conversational prompts that include images
  • video-chat: Conversational prompts that include videos
  • audio-chat: Conversational prompts that include audio clips

Diffusion Tasks

  • text-to-image: Image generation from text prompts
  • text-to-video: Video generation from text prompts

Request Datasets

We take model inputs from the following datasets for each task. Based on the input, we let the model generate their own outputs.

Task Dataset Auto-downloaded
lm-arena-chat LMArena Human Preference Yes
gpqa GPQA diamond Yes
sourcegraph-fim Sourcegraph FIM Yes
image-chat LMArena Vision Arena Yes
video-chat LLaVA-Video-178K No (see data-preparation.md)
audio-chat NVIDIA AudioSkills (FSD50K subset) No (see data-preparation.md)
text-to-image Open Image Preferences Yes
text-to-video EvalCrafter T2V Yes

Runtimes

The LLM/MLLM assumes that an underlying server provides the OpenAI Completions (for code completion) and Chat Completions (everything else) APIs. The server runs inside a container. For diffusion models, we use the xDiT runtime, and it does not run inside a container.

For the ML.ENERGY Benchmark 3.0 and the corresponding leaderboard, we used the following runtime versions:

Model Architecture Runtime Version
LLM/MLLM vLLM 0.11.1
Diffusion xDiT 0.4.5