docs: add SkyPilot launch tutorial under platform_support

.github/workflows/pr-test.yml

@@ -595,7 +595,15 @@ jobs:
             },
             {
               "num_gpus": 0,
-              "test_file": "test_train_dump.py"
+              "test_file": "test_train_data_utils.py"
+            },
+            {
+              "num_gpus": 0,
+              "test_file": "test_rollout_data_utils.py"
+            },
+            {
+              "num_gpus": 0,
+              "test_file": "test_rollout_metrics.py"
             },
             {
               "num_gpus": 0,
@@ -617,10 +625,18 @@ jobs:
               "num_gpus": 0,
               "test_file": "test_logprob_response_spans.py"
             },
+            {
+              "num_gpus": 0,
+              "test_file": "observability/test_trace_utils.py"
+            },
             {
               "num_gpus": 0,
               "test_file": "test_value_temperature.py"
             },
+            {
+              "num_gpus": 0,
+              "test_file": "test_ppo_kl_metric.py"
+            },
             {
               "num_gpus": 0,
               "test_file": "test_cispo_loss.py"
@@ -697,6 +713,10 @@ jobs:
               "num_gpus": 0,
               "test_file": "test_qwen3_5_vl_native.py"
             },
+            {
+              "num_gpus": 0,
+              "test_file": "test_accelerator.py"
+            },
             {
               "num_gpus": 0,
               "test_file": "test_reloadable_process_group_world.py"
@@ -782,7 +802,7 @@ jobs:
           pip install torch --index-url https://download.pytorch.org/whl/cpu
           pip install pytest numpy packaging pyyaml omegaconf tqdm httpx requests ray pybase64 pylatexenc sympy aiohttp pillow safetensors psutil
 
-          pip install transformers
+          pip install transformers wandb
 
 
       - name: Install

.github/workflows/pr-test.yml.j2

@@ -66,21 +66,25 @@
       'label': 'run-ci-cpu-unittest',
       'always': True,
       'cpu': True,
-      'extra_pip_deps': 'transformers',
+      'extra_pip_deps': 'transformers wandb',
       'tests': [
         {'test_file': 'test_megatron_argument_validation.py', 'num_gpus': 0},
         {'test_file': 'test_deep_ep_tms_patch.py', 'num_gpus': 0},
         {'test_file': 'test_stateless_adam.py', 'num_gpus': 0},
         {'test_file': 'utils/test_megatron_server_arguments.py', 'num_gpus': 0},
         {'test_file': 'test_dp_schedule.py', 'num_gpus': 0},
         {'test_file': 'test_cp_utils.py', 'num_gpus': 0},
-        {'test_file': 'test_train_dump.py', 'num_gpus': 0},
+        {'test_file': 'test_train_data_utils.py', 'num_gpus': 0},
+        {'test_file': 'test_rollout_data_utils.py', 'num_gpus': 0},
+        {'test_file': 'test_rollout_metrics.py', 'num_gpus': 0},
         {'test_file': 'test_metric_report.py', 'num_gpus': 0},
         {'test_file': 'test_metric_report_dist.py', 'num_gpus': 0},
         {'test_file': 'test_loss_cp_invariance.py', 'num_gpus': 0},
         {'test_file': 'test_advantage_whiten_cp.py', 'num_gpus': 0},
         {'test_file': 'test_logprob_response_spans.py', 'num_gpus': 0},
+        {'test_file': 'observability/test_trace_utils.py', 'num_gpus': 0},
         {'test_file': 'test_value_temperature.py', 'num_gpus': 0},
+        {'test_file': 'test_ppo_kl_metric.py', 'num_gpus': 0},
         {'test_file': 'test_cispo_loss.py', 'num_gpus': 0},
         {'test_file': 'test_policy_loss.py', 'num_gpus': 0},
         {'test_file': 'test_ppo_logprob_entropy.py', 'num_gpus': 0},
@@ -100,6 +104,7 @@
         {'test_file': 'test_rollout_sample_hooks.py', 'num_gpus': 0},
         {'test_file': 'test_hf_to_megatron.py', 'num_gpus': 0},
         {'test_file': 'test_qwen3_5_vl_native.py', 'num_gpus': 0},
+        {'test_file': 'test_accelerator.py', 'num_gpus': 0},
         {'test_file': 'test_reloadable_process_group_world.py', 'num_gpus': 0},
         {'test_file': 'test_placement_group.py', 'num_gpus': 0},
         {'test_file': 'test_external_sglang_engines.py', 'num_gpus': 0},

docker/NOTES_GB10.md

@@ -52,7 +52,7 @@ https://catalog.ngc.nvidia.com/orgs/nvidia/containers/vllm?version=26.03-py3
 | 9 | CMake 3.31 rejects `CMAKE_CUDA_ARCHITECTURES=120f;121f` | `f` suffix for CUDA 13 Blackwell family is only supported in CMake ≥4.0 | Upgrade to `cmake==4.3.1` via pip (must override NGC `/etc/pip/constraint.txt` with `PIP_CONSTRAINT=`) and set `CMAKE_POLICY_VERSION_MINIMUM=3.5` for old bundled deps |
 | 10 | TE 2.10 build: `cuda_profiler_api.h: No such file` | CUDA 13 removed the public header for `cudaProfilerStart/Stop`; the symbols still exist in `libcudart.so.13`. TE's 3 `fused_softmax` TUs `#include` the header but don't call the APIs | Install a 20-line shim header at `/usr/local/cuda/include/cuda_profiler_api.h` declaring the two functions extern. Stored as `docker/patch/gb10/cuda_profiler_api.h` |
 | 11 | slime `train.py --help`: `'tuple' object has no attribute 'strip'` | Typo in `slime/utils/arguments.py:1073`: `help=("string",)` (trailing comma → tuple) instead of `help=("string")` | Remove trailing comma — simple one-line slime fix, upstream-able |
-| 12 | `sglang_router` x86_64-only wheel from `zhuzilin/sgl-router` fork | slime Dockerfile pins `zhuzilin/sgl-router` release (no arm64 builds); slime's `'slime' in version` assertion is only in `wandb_utils.py` (non-critical path) | Install upstream `sglang-router==0.3.2` from PyPI (has arm64 wheel). Accept wandb path fallback |
+| 12 | `sglang_router` x86_64-only wheel from `zhuzilin/sgl-router` fork | slime Dockerfile pins `zhuzilin/sgl-router` release (no arm64 builds); slime's `'slime' in version` assertion is only in `slime/observability/wandb_utils.py` (non-critical path) | Install upstream `sglang-router==0.3.2` from PyPI (has arm64 wheel). Accept wandb path fallback |
 | 13 | `antlr4-python3-runtime==4.13.2` → `Could not deserialize ATN with version 3` | Omegaconf's bundled grammar was generated with antlr 4.9 serialized format; runtime 4.13 only reads format v4 | Pin `antlr4-python3-runtime==4.9.3` |
 | 14 | `megatron.training` not importable after `pip install -e Megatron-LM` | Megatron-LM's setup.py only packages `megatron-core`; `megatron.training`, `megatron.rl`, `megatron.legacy` are sibling dirs meant to be on `PYTHONPATH` | `export PYTHONPATH=/root/src/Megatron-LM:$PYTHONPATH` (slime docs confirm this) |
 | 15 | `libz3.so` missing for tilelang | tilelang uses Z3 SMT solver for autoscheduling; NGC vllm base doesn't include libz3 | `apt-get update && apt-get install -y libz3-dev` (libz3-4 package alias needs update first) |

docker/npu_patch/slime.patch

@@ -84,11 +84,11 @@ index 7bc4f910..a00c83c51 100644
  from ray.actor import ActorHandle
  from torch_memory_saver import torch_memory_saver
 @@ -17,6 +21,7 @@ from slime.ray.train_actor import TrainRayActor
- from slime.utils import train_dump_utils
+ from slime.observability import train_data_utils, train_metric_utils
  from slime.utils.data import process_rollout_data
  from slime.utils.distributed_utils import get_gloo_group, init_process_group
 +from slime.utils.http_utils import _wrap_ipv6
- from slime.utils.logging_utils import init_tracking
+ from slime.observability.logging_utils import init_tracking
  from slime.utils.memory_utils import clear_memory, print_memory
  from slime.utils.misc import Box
 @@ -55,6 +60,8 @@ class MegatronTrainRayActor(TrainRayActor):
@@ -586,13 +586,13 @@ diff --git a/slime/ray/rollout.py b/slime/ray/rollout.py
 index 75cb053c..c54a3854 100644
 --- a/slime/ray/rollout.py
 +++ b/slime/ray/rollout.py
-@@ -28,6 +28,7 @@ from slime.utils.metric_utils import (
+@@ -28,6 +28,7 @@ from slime.observability.metric_utils import (
  from slime.utils.misc import Box, group_by, load_function
  from slime.utils.seqlen_balancing import get_seqlen_balanced_partitions
  from slime.utils.types import Sample
 +from slime.utils.common import is_npu
 
- from ..utils.metric_utils import has_repetition
+ from slime.observability.metric_utils import has_repetition
  from .utils import NOSET_VISIBLE_DEVICES_ENV_VARS_LIST, Lock
 @@ -76,7 +77,8 @@ class RolloutManager:
              self.all_rollout_engines = [None] * num_engines
@@ -631,7 +631,7 @@ index 2e900ca5..d0a25583 100644
 +++ b/slime/ray/train_actor.py
 @@ -13,16 +13,23 @@ from slime.ray.ray_actor import RayActor
  from slime.utils.distributed_utils import init_gloo_group
- from slime.utils.logging_utils import configure_logger
+ from slime.observability.logging_utils import configure_logger
  from slime.utils.memory_utils import clear_memory, print_memory
 +from slime.utils.common import is_npu
 
@@ -966,4 +966,4 @@ index 01883c47..18faa6d2 100644
 +    import mindspeed.megatron_adaptor
  from slime.ray.placement_group import create_placement_groups, create_rollout_manager, create_training_models
  from slime.utils.arguments import parse_args
- from slime.utils.logging_utils import configure_logger, init_tracking
+ from slime.observability.logging_utils import configure_logger, init_tracking

docs/en/developer_guide/trace.md

@@ -41,7 +41,7 @@ By default it also starts a local static server so you can open the generated HT
 
 ## Instrument custom code
 
-For custom rollout or reward code — including custom agent steps, tool calls, sandbox execution, and verifier calls in agentic workflows — reuse helpers from `slime.utils.trace_utils`:
+For custom rollout or reward code — including custom agent steps, tool calls, sandbox execution, and verifier calls in agentic workflows — reuse helpers from `slime.observability.trace_utils`:
 
 - `trace_span(target, name, attrs=...)`: record a duration span.
 - `trace_event(target, name, attrs=...)`: record an instant event.
@@ -57,7 +57,7 @@ Use `trace_function(...)` when the whole function should be represented as one s
 The decorator is what slime uses for the main rollout pipeline. For example, `generate_and_rm(...)` is traced per sample and `generate_and_rm_group(...)` is traced per sample group:
 
 ```python
-from slime.utils.trace_utils import trace_function
+from slime.observability.trace_utils import trace_function
 
 
 @trace_function("generate_and_rm", target="sample")
@@ -104,7 +104,7 @@ If you need to add attrs after part of the function has executed, use an inner `
 If you want to record SGLang generation metadata in a consistent way, reuse `build_sglang_meta_trace_attrs`:
 
 ```python
-from slime.utils.trace_utils import build_sglang_meta_trace_attrs, trace_span
+from slime.observability.trace_utils import build_sglang_meta_trace_attrs, trace_span
 
 with trace_span(sample, "sglang_generate") as span:
     output = await post(url, payload)
@@ -116,4 +116,3 @@ with trace_span(sample, "sglang_generate") as span:
 - Save a small number of rollouts first; the viewer is easiest to read when each dump contains a manageable number of samples.
 - The viewer is built from the saved `.pt` dump, so traces can be inspected offline on another machine.
 - For GPU/kernel-level SGLang profiling traces, see [Profiling](./profiling.md).
-

docs/en/get_started/customization.md

@@ -42,7 +42,7 @@ For most agentic use cases, **start with `--custom-generate-function-path` plus
 | Replace the entire rollout orchestration (only when per-sample customization is not enough) | [`--rollout-function-path`](#1-rollout-function---rollout-function-path) |
 | Control task sampling, buffering, requeueing, or custom prompt/task sources | [`--data-source-path`](#15-data-source---data-source-path) |
 | Attach custom loss masks, metadata, or convert agentic outputs into training data | [`--rollout-data-postprocess-path`](#8-rollout-data-postprocess---rollout-data-postprocess-path), [`--custom-convert-samples-to-train-data-path`](#13-samples-to-train-data-conversion---custom-convert-samples-to-train-data-path) |
-| Debug long-running custom generation, verifier calls, tool calls, or sandbox steps | trace utilities in [`slime.utils.trace_utils`](../developer_guide/trace.md) |
+| Debug long-running custom generation, verifier calls, tool calls, or sandbox steps | trace utilities in [`slime.observability.trace_utils`](../developer_guide/trace.md) |
 
 A native example of this pattern is [`examples/search-r1`](../../../examples/search-r1/), which adds search-augmented multi-turn generation via `--custom-generate-function-path` while keeping slime's default `sglang_rollout` outer loop. See also [`examples/multi_agent`](../../../examples/multi_agent/README.md) for a `--rollout-function-path`-based multi-agent pattern and [`examples/fully_async`](../../../examples/fully_async/README.md) for long-tail agentic generation.
 

docs/en/get_started/quick_start.md

@@ -584,6 +584,8 @@ export NCCL_SOCKET_IFNAME=$(ip -o -4 addr show | awk '$4 ~ /^10\\./ {print $2}')
 export NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME=$(ip -o -4 addr show | awk '$4 ~ /^10\./ {print $2}')
 ```
 
+For launching the same multi-node setup on Kubernetes or cloud instances with one command, see the [SkyPilot tutorial](../platform_support/skypilot_tutorial.md).
+
 slime has been deeply optimized for distributed training of large-scale Mixture of Experts (MoE) models. We provide some end-to-end training cases for reference:
 
 - [Example: 8xH100 Training GLM-4.7-Flash](../examples/glm4.7-30B-A3B.md)

docs/en/index.rst

@@ -110,9 +110,10 @@ Start by Use Case
 
 .. toctree::
    :maxdepth: 1
-   :caption: Hardware Platforms
+   :caption: Platforms
 
    platform_support/amd_tutorial.md
+   platform_support/skypilot_tutorial.md
 
 .. toctree::
    :maxdepth: 1