Roofline analysis reveals why ZeRO-3 fails for DeepSeek-V3 on Hopper
Speed of light modeling shows that FSDP creates a communication bottleneck on InfiniBand for DeepSeek-V3, making pipeline parallelism the necessary choice for efficient training.
Results for “distributed training”
Daily coverage of AI, developer tools and infrastructure. Each story explains what happened and why it matters, with a link to the original source.