Frontier-Lab Training for Everyone
Blog post · Docs · Examples · Benchmarks
Halo is an open-source framework built by White Circle for training large language and multimodal models. It supports pre-training, SFT, preference optimization, fully asynchronous multi-turn RL, and much more.
The same codebase runs on one GPU or across multiple nodes, with EP, CP, TP, ETP, FSDP2, fused kernels, BF16 training, and distributed rollouts.
Halo trains Hugging Face models directly. Checkpoints still load with from_pretrained, and supporting a new model family typically takes under 140 lines of integration code.
On 8× B300, Halo trains gpt-oss-20b at up to 3.5× the throughput of stock TRL (3.2× at 25% less peak memory when both sides shard ZeRO-3).
Pre- & Post-Training · EP / CP / TP / ETP ·
Multi-Node · Verifiable & Multi-Turn RL · FA4 + Liger + Grouped GEMM · Full BF16
-
Halo trains Hugging Face models directly. Trainers subclass Hugging Face or TRL trainers, checkpoints remain standard SafeTensors, and
from_pretrainedcontinues to work. -
Parallelism is added to the existing model. EP wraps MoE blocks, CP wraps attention, and TP/ETP shard weights in place. There is no separate distributed implementation of the model.
-
The distributed stack is built on PyTorch. FSDP2, DTensor, and DeviceMesh provide the underlying primitives, while DeepEP handles the all-to-all communication for Expert Parallelism. EP, CP, TP, and ETP are set independently, and EP combines with CP, TP, or ETP across GPUs and nodes.
-
Training methods share the same infrastructure. Pre-training, SFT, preference optimization, distillation, and RL use the same parallelism and checkpointing code, so new methods don't need their own distributed implementation.
-
Performance optimizations are integrated. Halo includes DeepEP V2, FlashAttention 4, Liger, Grouped GEMM, padding-free packing, and AdamWBF16, with implementations and fallbacks for Blackwell, Hopper, and older GPUs.