fix: complete deterministic rollout samplingPyq/complete determinism concurrent rollout by pyq623 · Pull Request #1607 · areal-project/AReaL

pyq623 · 2026-08-14T12:27:45Z

Summary

Complete deterministic rollout sampling across concurrent rollout scheduling and the V1/V2 inference paths.

This PR makes rollout identity, request seed assignment, and result ordering reproducible while preserving concurrent execution by default. It also provides an explicit opt-in serial mode for strict V2 reproducibility when SGLang dynamic batching affects numerical identity.

Motivation

Deterministic inference requires more than enabling deterministic kernels on the SGLang server.

Previously, several gaps remained:

grouped samples could receive identities based on runtime completion order;
concurrent requests did not have a stable per-request seed assignment;
an OpenAI-compatible seed could reach ArealOpenAI without being carried through the complete request path;
V2 did not forward GenerationHyperparameters.seed as SGLang sampling_seed;
canonical result ordering did not fully cover concurrent rollout execution;
V2 samples within one group were always submitted with asyncio.gather, so their SGLang dynamic-batch composition could vary between runs.

Together, these gaps could make repeated runs diverge even when the global training seed and SGLang deterministic inference were enabled.

Changes

Stable concurrent rollout identity and ordering

assign stable sample identities to grouped rollouts;
preserve sample-index ordering when combining concurrently completed results;
stabilize dispatcher task selection and batch ordering;
reserve request indices before inference so concurrent requests receive distinct and reproducible indices;
separate logical sampling identity from physical session collision suffixes.

End-to-end sampling seed propagation

The request path is now:

  logical rollout identity + request index
  -> deterministic per-request seed
  -> ArealOpenAI.create(seed=...)
  -> GenerationHyperparameters.seed
  -> ModelRequest.gconfig.seed
  -> SGLang sampling_params["sampling_seed"]

This path is covered for:

Chat Completions;
Responses API seed acceptance;
V1 proxy and SGLangBackend;
V2 data proxy and SGLangBridgeBackend.

Explicit caller-provided seeds always take precedence over automatically derived seeds.

Shared seed derivation

Add a shared derive_deterministic_seed(identity, request_index) helper so V1 and V2 use the same stable derivation logic.

The generated seed is:

stable across processes and runs;
distinct across logical samples and request indices;
independent of Python hash randomization;
non-negative and suitable for SGLang sampling_seed.

Optional strict V2 group serialization

Add the explicit configuration:

InferenceEngineConfig.serialize_group_samples: bool = False

Behavior:

False:
samples within a V2 group continue to run concurrently with asyncio.gather

True:
samples run sequentially in stable member order
member 0 -> member 1 -> member 2 -> ...

This option is intentionally independent from deterministic_sampling.

deterministic_sampling stabilizes identities, seeds, scheduling, and result ordering while retaining concurrency. serialize_group_samples additionally stabilizes request arrival and SGLang batching conditions for strict reproducibility, at the expected cost of rollout throughput.

The default remains concurrent, so existing workloads are unaffected.

Observability

V2 workflow logs now include:

task ID;
group ID;
model version;
member index;
session ID;
serial or concurrent execution mode.

These fields make it possible to verify the effective rollout execution path from runtime logs.

Failure handling

Serial execution preserves the existing group-cleanup behavior:

a failed member does not prevent later members from running;
reward fallback is attempted for the failed member;
all sessions still reach group export and cleanup;
a group containing failed members is rejected after cleanup.

Documentation

Regenerate the English and Chinese CLI references with the new serialize_group_samples option.

Compatibility

All deterministic behavior remains opt-in:

deterministic_sampling defaults to False;
serialize_group_samples defaults to False;
concurrent V2 group execution remains the default;
requests without a seed retain their previous behavior unless deterministic sampling is enabled;
explicit request seeds are never overwritten;
SGLang per-request seeds still require sglang.enable_deterministic_inference=True to be honored by the server.

Testing

Focused tests were executed in the project Slurm training image:

tests/test_deterministic_sampling.py
tests/v2/inference_service/test_controller.py
tests/v2/inference_service/test_data_proxy_chat.py

118 passed, 4 skipped

The tests cover:

stable and distinct seed derivation;
concurrent request-index allocation;
explicit seed precedence;
ArealOpenAI forwarding seed into ModelRequest.gconfig;
V1 and V2 SGLang sampling_seed payloads;
canonical concurrent rollout ordering;
controller-to-workflow configuration propagation;
concurrent V2 group execution by default;
stable serial member ordering when explicitly enabled;
serial failure handling and export cleanup;
grouped V2 sessions receiving stable, distinct seeds.

Additional checks:

Ruff lint/format: passed
mdformat: passed
git diff --check: passed
Python compilation: passed
CLI documentation generation: passed

The remaining warning is an existing third-party torchao SyntaxWarning.

Derive stable sampling seeds for OpenAI proxy sessions and preserve canonical rollout group order while inference requests run concurrently. Consume completed work through a submission-order frontier so rollout completion timing cannot change training batch membership. Forward request seeds and deterministic-inference configuration to SGLang, and bind callbacks after task ID allocation.

sitabulaixizawaluduo · 2026-08-17T11:33:23Z

        if "top_p" not in kwargs:
            kwargs["top_p"] = 1.0

+        deterministic_sampling = session is not None and get_bool_env_var(


The issue is that the command to start Data Proxy did not pass this configuration or environment variable, so the V2 requests still will not automatically generate a seed, and the subsequent SGLang sampling_seed forwarding also cannot obtain the value.

Suggestion: Add deterministic_sampling to DataProxyConfig and startup parameters, or explicitly pass the environment variable when forking Data Proxy.

sitabulaixizawaluduo · 2026-08-17T11:47:51Z

The vllm-related part has not been fixed in the related part. I understand that deterministic_sampling is a backend-independent common configuration. If convenient, can you include the vllm part update and supplement the relevant UT?

Le8r0nJames · 2026-08-18T03:29:41Z

+                        task_id in self._pending_results for task_id in task_frontier
+                    )
+                else:
+                    results_ready = len(self._pending_results) >= count


In the synchronous, no-rejection case, the staleness manager allows only one
consumer batch to run per model version. Later task IDs cannot complete before
the current batch, so sorting completed results by task ID and disabling
shuffle appears sufficient; the frozen membership frontier seems redundant.
Is the frontier intended specifically for rejection/timeout or
submit-many/wait-few scenarios? If so, could that behavior be scoped and tested
separately?

Le8r0nJames · 2026-08-18T03:30:42Z

+    deterministic_sampling: bool = field(
+        default=False,
+        metadata={


Could you clarify the determinism contract for
max_head_offpolicyness > 0?

If asynchronous staleness is supported, how does this implementation guarantee
a stable task-to-weight-version mapping across runs? The inference version is
read when a generation request is actually sent, so the same logical task may
use different model versions depending on scheduling timing. Stable seeds,
result ordering, and a membership frontier do not appear to fix that.

If end-to-end determinism is only supported with
max_head_offpolicyness=0, should the configuration emit a warning or document
that requirement explicitly?

Le8r0nJames · 2026-08-18T03:33:33Z

+            logger.info(
+                "V2 rollout member start: task_id=%s group_id=%s member=%d "
+                "session_id=%s version=%s mode=%s",
+                task_id,
+                group_id,
+                member_index,
+                session_id,
+                version,
+                execution_mode,
+            )


Could the per-member start/finish messages be moved to DEBUG or gated by
enable_rollout_tracing? This emits two INFO lines per trajectory plus one line
per group. With batch_size=16 and n_samples=8 that is at least 272 additional
INFO lines per training step.

Le8r0nJames · 2026-08-18T03:36:53Z

@@ -2098,6 +2102,7 @@ class SGLangConfig:
    enable_memory_saver: bool = False
    allow_auto_truncate: bool = False
    attention_backend: str | None = "fa3"


SGLang documents deterministic inference support only for the flashinfer, fa3,
and triton attention backends. At the moment enable_deterministic_inference is
forwarded for any configured backend, which can give users a false
determinism guarantee.

Could we emit a warning when deterministic inference is enabled with an
explicit attention_backend outside {flashinfer, fa3, triton}? None can remain
allowed because it delegates to the SGLang default.

Le8r0nJames · 2026-08-18T03:43:49Z

+        if self.serialize_group_samples:
+            results = []
+            for member_index, (session_id, session_api_key) in enumerate(sessions):
+                results.append(
+                    await _run_one(member_index, session_id, session_api_key)
+                )
+        else:
+            results = await asyncio.gather(
+                *[
+                    _run_one(member_index, session_id, session_api_key)
+                    for member_index, (session_id, session_api_key) in enumerate(
+                        sessions
+                    )
+                ]
+            )


I don't think serializing members within one group is sufficient to stabilize
SGLang batch composition. Multiple prompt groups still execute concurrently,
so a serialized member from group A can be co-batched with requests from group
B in timing-dependent ways.

This guarantees per-group member order, but not “strict reproducibility” of
dynamic batching. Could we remove or weaken this claim and rely on SGLang
batch-invariant inference instead? A true serialization fallback would need a
global request scheduler, not a per-group loop.

Le8r0nJames and others added 2 commits August 14, 2026 16:56

fix: complete deterministic rollout sampling

1f5099b

pyq623 requested review from HwVanICI, TaoZex, fishcrap, garrett4wade, geshi001, guozhihao-224, nuzant, rchardx and sitabulaixizawaluduo as code owners August 14, 2026 12:27

sitabulaixizawaluduo reviewed Aug 17, 2026

View reviewed changes

sitabulaixizawaluduo mentioned this pull request Aug 17, 2026

feat(rollout): support deterministic concurrent rollouts #1611

Closed

15 tasks

Le8r0nJames reviewed Aug 18, 2026

View reviewed changes

Navigation Menu

Uh oh!

fix: complete deterministic rollout samplingPyq/complete determinism concurrent rollout - #1607

fix: complete deterministic rollout samplingPyq/complete determinism concurrent rollout#1607
pyq623 wants to merge 2 commits into
areal-project:mainfrom
pyq623:pyq/complete-determinism-concurrent-rollout

pyq623 commented Aug 14, 2026

Uh oh!

sitabulaixizawaluduo Aug 17, 2026

Uh oh!

sitabulaixizawaluduo commented Aug 17, 2026

Uh oh!

Le8r0nJames Aug 18, 2026

Uh oh!

Le8r0nJames Aug 18, 2026

Uh oh!

Le8r0nJames Aug 18, 2026

Uh oh!

Le8r0nJames Aug 18, 2026

Uh oh!

Le8r0nJames Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

pyq623 commented Aug 14, 2026

Summary

Motivation

Changes

Stable concurrent rollout identity and ordering

End-to-end sampling seed propagation

Shared seed derivation

Optional strict V2 group serialization

Observability

Failure handling

Documentation

Compatibility

Testing

Uh oh!

sitabulaixizawaluduo Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

sitabulaixizawaluduo commented Aug 17, 2026

Uh oh!

Le8r0nJames Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Le8r0nJames Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Le8r0nJames Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Le8r0nJames Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Le8r0nJames Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants