Skip to content

Add GSM8KEnv and GSM8KAgent for Tunix agentic RL - #2022

Open
andytwigg wants to merge 2 commits into
mainfrom
atwigg/gsm8k-env
Open

Add GSM8KEnv and GSM8KAgent for Tunix agentic RL#2022
andytwigg wants to merge 2 commits into
mainfrom
atwigg/gsm8k-env

Conversation

@andytwigg

@andytwigg andytwigg commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR introduces a modular GSM8KEnv environment, GSM8KAgent agent, and an end-to-end training entrypoint script train_gsm8k_qwen3.py for Qwen3-1.7B for the Tunix Agentic RL framework, matching the architecture and conventions established in examples/frozenlake/.

Changes Included

  1. examples/math_gsm8k/env.py:

    • GSM8KEnv: Subclasses BaseTaskEnv to manage mathematical reasoning tasks.
    • Extracts ground-truth numerical answers from GSM8K solutions via extract_hash_answer (... #### <num>).
    • Parses model reasoning chains and LaTeX \boxed{...} answers using stack-based bracket matching with robust fallbacks.
    • Computes composite format and accuracy rewards (+0.1 format bonus, +1.0 exact accuracy, +0.5 partial credit).
    • Provides serialization and dynamic instantiation via from_dict().
  2. examples/math_gsm8k/agent.py:

    • GSM8KAgent: Subclasses ConversationAgentBase to maintain dialogue history and step-by-step reasoning trajectories for GRPO training.
  3. examples/math_gsm8k/data.py:

    • GSM8K dataset loaders supporting Hugging Face (openai/gsm8k), TFDS, and in-memory smoke-testing.
  4. examples/math_gsm8k/env_test.py:

    • 13 comprehensive unit tests validating answer extraction, tag checking, reward calculation, and agent-environment interaction.
  5. examples/math_gsm8k/train_gsm8k_qwen3.py:

    • Agentic GRPO training entrypoint script for GSM8K on TPU using Qwen3-1.7B.
    • Integrates GSM8KEnv and GSM8KAgent with GRPOLearner and RLEngine.
    • Configures ModelConfig.qwen3_1p7b() with bf16 reference and fp32 actor (supporting full fine-tuning and LoRA).
    • Supports vLLM and vanilla rollout engines on shared or distributed mesh.
    • Computes and logs step-by-step math reasoning metrics (solve ratio, format ratio, reward).

TAG=agy
CONV=2907f211-714a-4e9c-b009-de21347c291a

Introduce a modular, self-contained GSM8K environment and agent implementation
for the Tunix Agentic RL framework, following the pattern in FrozenLakeEnv:

- examples/math_gsm8k/env.py: Implements GSM8KEnv (subclassing BaseTaskEnv)
  with problem prompt formatting, bracket-matching answer extraction, XML
  reasoning/answer tag validation, and composite rewards (format + accuracy).
- examples/math_gsm8k/agent.py: Implements GSM8KAgent (subclassing
  ConversationAgentBase) managing dialogue history and step trajectories.
- examples/math_gsm8k/data.py: Provides GSM8K dataset loaders supporting
  Hugging Face ('openai/gsm8k'), TFDS, and smoke test fallback.
- examples/math_gsm8k/env_test.py: Comprehensive unit tests covering answer
  extraction, format scoring, environment lifecycle, and agent interaction.

TAG=agy
CONV=2907f211-714a-4e9c-b009-de21347c291a
Adds an agentic GRPO training entrypoint script for GSM8K on TPU using
Qwen3-1.7B, following the pattern of train_frozenlake_qwen3.py:
- Integrates GSM8KEnv and GSM8KAgent with GRPOLearner and RLEngine.
- Configures ModelConfig.qwen3_1p7b() with bf16 reference and fp32 actor.
- Supports both full-parameter and LoRA fine-tuning.
- Supports vLLM and vanilla rollout engines on shared or distributed mesh.
- Logs step-by-step math reasoning metrics (solve ratio, format ratio, reward).

TAG=agy
CONV=2907f211-714a-4e9c-b009-de21347c291a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants