Running trillion-parameter LLMs on Mac Studio clusters with MLX
Apple researchers demonstrate how to run the 816B Muse Spark Open model across four Mac Studios using MLX and RDMA over Thunderbolt.
Results for “llm inference”
Daily coverage of AI, developer tools and infrastructure. Each story explains what happened and why it matters, with a link to the original source.
Speculative decoding: how guessing tokens speeds up LLM inference
Speculative decoding lets large language models verify multiple token guesses in a single forward pass, reducing memory-bound latency without changing output distribution.
Secure multi-environment access patterns for Claude Platform on AWS
A technical guide details how to configure cross-account SigV4, workspace-scoped API keys, and OIDC federation for secure LLM inference across diverse environments.
Kubernetes introduces Gateway API extension for LLM inference routing
The Kubernetes project released the Gateway API Inference Extension to standardize routing for generative AI workloads, reducing latency and improving GPU utilization.
