Skip to content

feat(ppo): support actor loss aggregation modes #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
EazyReal wants to merge 31 commits into from
Closed

feat(ppo): support actor loss aggregation modes #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
be8d7cf
feat(cli): add experimental cli scaffold for service-style subcommand…
sitabulaixizawaluduo Jun 29, 2026
ae66992
refactor: move 5 experimental modules into areal/v2 for 2.0 release (…
sitabulaixizawaluduo Jun 29, 2026
62512eb
feat: support v2 weight update disk mode for lora RL (#1450)
sitabulaixizawaluduo Jun 30, 2026
2824f72
feat(cli): add training service cli (#1446)
sitabulaixizawaluduo Jun 30, 2026
0ca30f1
feat(cli): add agent service cli (#1447)
sitabulaixizawaluduo Jun 30, 2026
b891741
feat(cli): add inference service cli (#1434)
sitabulaixizawaluduo Jun 30, 2026
bbc85c7
feat(agent_service): add agent service with OpenClaw and Hermes examp…
IF007 Jun 30, 2026
4ac8c36
feat(ppo): add reuse_train_logp proximal logp method (#1453)
Le8r0nJames Jul 1, 2026
9c6a703
fix(ppo): coerce ppo_n_minibatches to 1 for reuse_train_logp (#1457)
Le8r0nJames Jul 1, 2026
5ea8aae
Chore/add areal 2 report paper (#1461)
sitabulaixizawaluduo Jul 1, 2026
8979fa5
feat(launcher): add best-effort post-exit hook (#1459)
Le8r0nJames Jul 1, 2026
a315ea9
feat(megatron): add CP-safe vocab stats and MoE config support (#1460)
Le8r0nJames Jul 1, 2026
bbc10f0
fix(ppo): handle variable-size trajectory groups in reward normalizat…
Le8r0nJames Jul 1, 2026
87a034e
fix(openai): parse tool_call arguments from JSON string to dict befor…
Le8r0nJames Jul 1, 2026
d8ff8ce
feat(swe): add SWE-bench RL training workflow (#1462)
Le8r0nJames Jul 1, 2026
f2958b6
fix: fix safe-to-test CI workflow (#1456)
sitabulaixizawaluduo Jul 1, 2026
7f3b021
feat(swe): add SWE agent proxy, message preprocessors and tool-call s…
Le8r0nJames Jul 1, 2026
7c1d22d
test(ppo): update singleton leave-one-out group expectation (#1464)
Le8r0nJames Jul 1, 2026
77458d0
chore: update AReaL 2.0 report paper (#1465)
sitabulaixizawaluduo Jul 1, 2026
fee938e
chore: bump v2.0.0 (#1466)
sitabulaixizawaluduo Jul 1, 2026
90cb574
chore(swe): remove unused SWE SFT dataset loader (#1467)
Le8r0nJames Jul 2, 2026
9cf6852
fix(docker): rename zeroclaw installer flag to --skip-quickstart (#1469)
sitabulaixizawaluduo Jul 2, 2026
89e05ac
fix(megatron): use dp_reshardable optimizer sharding and load optimiz…
Le8r0nJames Jul 2, 2026
d99124e
fix(hermes): preserve singleton online reward signal (#1474)
jszzr Jul 3, 2026
4dc39d4
fix(mcore): TP-shard GroupRMSNorm gate-norm weight for DCP checkpoint…
Le8r0nJames Jul 6, 2026
ab87f06
fix(rollout): stop controller-managed workers from dp-scaling stalene…
Le8r0nJames Jul 6, 2026
aea57eb
fix(v2): export local workflow statistics (#1478)
jszzr Jul 7, 2026
9ac4ad6
docs: update NPU installation guide for ascend-v1.0.4 (#1502)
Adiactive Jul 13, 2026
a26a776
Docs: Update README and cli docs (#1521)
sitabulaixizawaluduo Jul 14, 2026
3cf0dfb
fix(engine): route text-only VLM microbatches through the padded forw…
Adiactive Jul 14, 2026
a8f0d5e
feat(ppo): support configurable loss aggregation
EazyReal Jul 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions .agents/skills/review-pr/MIGRATION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,8 @@ ______________________________________________________________________
1. `areal/experimental/models/archon/**` + `areal/experimental/engine/archon_engine.py`
\+ `areal/experimental/engine/archon_checkpoint.py` -> Distributed Runtime + Model
Compute & Attention (CRITICAL/HIGH)
1. `areal/experimental/agent_service/**` -> Service Orchestration (HIGH)
1. `areal/experimental/inference_service/**` -> Service Orchestration (HIGH)
1. `areal/v2/agent_service/**` -> Service Orchestration (HIGH)
1. `areal/v2/inference_service/**` -> Service Orchestration (HIGH)
1. `areal/experimental/engine/archon_weight_sync.py` -> Distributed Runtime (CRITICAL)
1. `areal/infra/rpc/**` -> Launcher & Infrastructure + Distributed Runtime (HIGH)
1. `areal/workflow/**`, `areal/trainer/**` -> Workflow & Trainer Contract (HIGH/MEDIUM)
Expand Down
32 changes: 16 additions & 16 deletions .agents/skills/review-pr/references/review-pr-domains-and-signals.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,19 +52,19 @@ ______________________________________________________________________

## Domain 4: Service Orchestration (HIGH)

| L2 Signal | File Path Pattern | Code Pattern |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| `service_routing_dataflow` | `areal/experimental/agent_service/gateway/`, `areal/experimental/agent_service/router/`, `areal/experimental/inference_service/data_proxy/`, `areal/experimental/inference_service/controller/` | `route`, `gateway`, `router`, `DataProxy`, `controller`, `batch` |
| `session_consistency` | `areal/experimental/agent_service/`, `areal/experimental/inference_service/` | `session`, `affinity`, `history`, `state` |
| L2 Signal | File Path Pattern | Code Pattern |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| `service_routing_dataflow` | `areal/v2/agent_service/gateway/`, `areal/v2/agent_service/router/`, `areal/v2/inference_service/data_proxy/`, `areal/v2/inference_service/controller/` | `route`, `gateway`, `router`, `DataProxy`, `controller`, `batch` |
| `session_consistency` | `areal/v2/agent_service/`, `areal/v2/inference_service/` | `session`, `affinity`, `history`, `state` |

## Domain 5: Workflow & Trainer Contract (HIGH/MEDIUM)

| L2 Signal | File Path Pattern | Code Pattern |
| -------------------------- | ----------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| `workflow_engine_boundary` | `areal/workflow/`, `areal/trainer/`, `areal/engine/` | `RolloutWorkflow`, `arun_episode`, `agenerate` |
| `dataset_surface` | `areal/dataset/` | `DataLoader`, `IterableDataset`, `get_*_dataset` |
| `async_contract` | `areal/workflow/`, `areal/experimental/agent_service/`, `areal/experimental/inference_service/` | `async def`, `await`, `aiofiles`, `asyncio` |
| `weight_version_contract` | `areal/api/engine_api.py`, `areal/workflow/`, `areal/trainer/` | `WeightUpdateMeta`, `set_version`, `weight version` |
| L2 Signal | File Path Pattern | Code Pattern |
| -------------------------- | --------------------------------------------------------------------------- | --------------------------------------------------- |
| `workflow_engine_boundary` | `areal/workflow/`, `areal/trainer/`, `areal/engine/` | `RolloutWorkflow`, `arun_episode`, `agenerate` |
| `dataset_surface` | `areal/dataset/` | `DataLoader`, `IterableDataset`, `get_*_dataset` |
| `async_contract` | `areal/workflow/`, `areal/v2/agent_service/`, `areal/v2/inference_service/` | `async def`, `await`, `aiofiles`, `asyncio` |
| `weight_version_contract` | `areal/api/engine_api.py`, `areal/workflow/`, `areal/trainer/` | `WeightUpdateMeta`, `set_version`, `weight version` |

## Domain 6: API & Config Compatibility (MEDIUM)

Expand Down Expand Up @@ -95,12 +95,12 @@ ______________________________________________________________________

## Domain 9: Launcher & Infrastructure (HIGH/MEDIUM)

| L2 Signal | File Path Pattern | Code Pattern |
| ------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------- |
| `launcher_resource_match` | `areal/infra/launcher/` | `LaunchConfig`, `RayLauncher`, `SlurmLauncher` |
| `scheduler_contract` | `areal/infra/scheduler/`, `areal/scheduler/` | `Scheduler`, `placement`, `resource` |
| `rpc_transport` | `areal/infra/rpc/`, `areal/experimental/inference_service/data_proxy/` | `RTensor`, `serialize`, `rpc`, `fetch` |
| `runtime_image_config` | `Dockerfile`, `.dockerignore` | `FROM`, `ARG`, `RUN`, `ENV`, `COPY`, `uv sync`, `VARIANT` |
| L2 Signal | File Path Pattern | Code Pattern |
| ------------------------- | ------------------------------------------------------------ | --------------------------------------------------------- |
| `launcher_resource_match` | `areal/infra/launcher/` | `LaunchConfig`, `RayLauncher`, `SlurmLauncher` |
| `scheduler_contract` | `areal/infra/scheduler/`, `areal/scheduler/` | `Scheduler`, `placement`, `resource` |
| `rpc_transport` | `areal/infra/rpc/`, `areal/v2/inference_service/data_proxy/` | `RTensor`, `serialize`, `rpc`, `fetch` |
| `runtime_image_config` | `Dockerfile`, `.dockerignore` | `FROM`, `ARG`, `RUN`, `ENV`, `COPY`, `uv sync`, `VARIANT` |

## Domain 10: Low-Risk Hygiene (LOW)

Expand Down
Loading
Loading