PyTorch 2.15 brings FP8 and faster LLM decoding to Apple Silicon

The PyTorch team at Hugging Face has released version 2.15, marking a significant upgrade for machine learning engineers working on Apple Silicon. This update introduces native FP8 support, accelerates mixture-of-experts model decoding by up to 6.5 times, and drastically reduces memory usage for video generation tasks. The release targets macOS 15 and Python 3.11 or later, aiming to make Macs a first-class environment for training and running complex AI models.

What happened

For much of early 2026, developers using the MPS backend on Macs faced performance cliffs, unsupported operations, and unpredictable memory growth. The PyTorch team at Hugging Face addressed these issues by replacing the older MPSGraph execution path with dedicated Metal kernels for critical operations. This shift allows for more direct control over GPU resources and eliminates many of the bottlenecks that previously hindered local model development.

The most notable addition is support for float8_e4m3fn tensors, allowing developers to store values in half the space of BF16 while performing scaled matrix multiplication directly on the GPU. This feature aligns Mac capabilities with those already available on CUDA and CPU backends. Additionally, the update includes extensive optimizations for linear algebra, audio processing, and video decoding, ensuring that a wider range of models can run efficiently without falling back to slower execution paths.

Performance improvements are particularly striking for large language models. Mixture-of-experts architectures now benefit from a native grouped matmul implementation that keeps data on the GPU, reducing synchronization overhead. Dense models also see gains through optimized single-row linear layers and significantly faster softmax operations. These changes collectively transform the Mac from a device primarily used for inference into a viable tool for more demanding training and generation workloads.

How it works

The core technical shift in PyTorch 2.15 is the migration of operations from MPSGraph to custom Metal kernels. MPSGraph, while flexible, often introduced overhead due to graph compilation and CPU-GPU synchronization. By writing dedicated kernels for specific tasks like grouped matrix multiplication, constant padding, and softmax, the team ensures that data remains on the GPU and executes with minimal latency. For example, the new grouped matmul kernel reads group offsets directly on the GPU, avoiding the costly loop-over-groups approach used in previous versions.

Memory efficiency has been improved by reworking how padding operations handle driver memory. In previous versions, tiled video VAE decoding triggered thousands of padding calls that caused driver memory to balloon. The new Metal-based constant padding implementation manages these allocations more effectively, preventing the exponential growth in reserved memory. This mechanism allows complex video generation pipelines to run within standard memory limits rather than requiring excessive system resources.

Key details

• FP8 Support: Native support for torch.float8_e4m3fn tensors enables half-precision storage and scaled matrix multiplication on MPS.
• MoE Speedup: Mixture-of-experts models decode 5–6.5× faster, with Qwen3.6-35B-A3B reaching 61.3 tokens/s on an M3 Ultra.
• Memory Savings: Wan2.2 video VAE decode peak driver memory drops from 270 GiB to approximately 8 GiB.
• Linear Algebra: Complex LU factorization, triangular solves, and faster small-matrix inverses are now supported on MPS.
• Softmax Optimization: Softmax over large vocabularies is approximately 8× faster, dropping from 57 µs to 7 µs for typical sizes.
• System Requirements: The update requires macOS 15 or later and Python 3.11 or later.

Why it matters

For software engineers building AI products on Mac hardware, this release removes several major friction points. The ability to use FP8 weights means you can fit larger models into limited VRAM or reduce memory bandwidth pressure during inference. This is crucial for local development and testing of state-of-the-art architectures that were previously too resource-intensive for consumer-grade Apple Silicon machines.

The dramatic reduction in memory usage for video generation workflows also opens up new possibilities for creative applications. Developers who previously encountered out-of-memory errors or system instability when using tools like ComfyUI or Diffusers can now run these pipelines reliably. The fix for image editing corruption in Qwen-Image-2.1 further ensures that visual fidelity is maintained, making Macs a more trustworthy platform for production-grade image and video tasks.

What you can do

• Upgrade your development environment to macOS 15 and Python 3.11 to ensure compatibility with the new MPS features.
• Test FP8 quantization in your models using torch.float8_e4m3fn and F.scaled_mm to reduce memory footprint.
• Benchmark your mixture-of-experts models to measure the decoding speed improvements provided by the native grouped matmul.
• Update video generation pipelines to leverage the new constant padding kernels and monitor driver memory usage.
• Experiment with public MPS streams via torch.mps.Stream for finer-grained control over concurrent GPU operations.
• Report any remaining performance issues or missing operations to the PyTorch GitHub repository to help guide future updates.