DLRover makes the distributed training of large AI models easy, stable, fast and green. It can automatically train the Deep Learning model on the distributed cluster. It helps model developers to focus on model architecture, without taking care of any engineering stuff, say, hardware acceleration, distributed running, etc. Now, it provides automated operation and maintenance for deep learning training jobs on K8s/Ray. Major features as
- Full-Scene: Support deep learning full-scene distributed training computation implementation.
- Fault-Tolerance: The distributed training can continue running in the event of failures.
- Auto-Scaling: The distributed training can scale up/down resources to improve the stability, throughput and resource utilization.
- Others:
- XPU Timer Integration: With runtime xpu-timer integration, can possess stronger runtime diagnostics and fault tolerance capabilities.
- Flash Checkpoint: The distributed training can recover failures from the in-memory checkpoint in seconds.
Furthermore, DLRover offers extension libraries for PyTorch and TensorFlow to expedite training. These are also open-source projects available in our GitHub repositories.
- ATorch: an extension library of PyTorch to Speed Up Training of Large LLM.
- TFPlus(K8S platform only): an extension library of TensorFlow to Speed Up Training of Search, Recommendation and Advertisement.
- [2025/12] DLRover on Ray's new architecture achieves its first official release.
- [2025/08] Practice: Gang Scheduling with DLRover.
- [2025/03] Paper: How XPU-Timer working
- [2025/01] EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models, ICLR'25.
- [2024/06] DLRover-RM has been accepted by VLDB'24.
- [2024/04] Flash Checkpoint Supports HuggingFace transformers.Trainer to Asynchronously persist checkpoints.
- [2024/02] Flash Checkpoint Saves the Megatron-LM Checkpoint in Seconds.
- [2024/01] Flash Checkpoint to Recover Large Model Training From Failure in Seconds.