feat(engine): add Datum-based forward, backward, and optimizer APIs

[HuiyingLi]

What does this PR do ?

Introduce a Datum-first Engine boundary for forward, forward-backward, distributed batch routing, and optimizer updates so LLM/VLM recipes no longer own PP/CP/FSDP execution details.

Changelog

• Add Datum collation metadata with explicit per-token, per-Datum, and replicated loss-input layouts.
• Add structured Engine.forward and Engine.forward_backward results, including exact loss numerator and denominator statistics.
• Route padded and packed THD batches through PP/CP while preserving Datum order and restoring typed per-token outputs.
• Move gradient finalization, clipping, optimizer updates, MoE maintenance, LR scheduling, and FP8 FSDP scale refresh into Engine.optim_step.
• Add planned multi-call accumulation with whole-window normalization and lifecycle validation.
• Migrate LLM and VLM finetuning recipes to the Engine boundary and remove duplicated parallelism/update logic.
• Add unit, distributed, recipe-integration, and packed PP/CP functional coverage.

Validation

• 328 passed, 12 skipped across Engine, FP8, LLM recipe, and VLM recipe unit suites.
• Real H100 PP2 and PP2xCP2 packed raw/final THD validation, including Datum/layout routing and output restoration.
• Real 4xH100 PP2xDP2 FP8 run for two optimizer steps; every rank refreshed all 7 eligible local FP8 DTensor scales and the second step consumed the refreshed scales successfully.
• Ruff checks and git diff --check pass.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?

This is intentionally opened as a Draft while full CI and final reviewer feedback are pending.

Additional Information

• Related issue: N/A
• Multi-axis mRoPE with packed Transformer Engine THD context parallelism remains intentionally unsupported.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.