Optimizing MoE inference on NVIDIA Hopper GPUs with kernel engineering
AWS engineers detail three kernel optimizations for Mixture-of-Experts models on Hopper GPUs, improving throughput by 10% and reducing latency by 9%.
RAG, evaluations, inference, fine-tuning and shipping LLM features to production.
AWS engineers detail three kernel optimizations for Mixture-of-Experts models on Hopper GPUs, improving throughput by 10% and reducing latency by 9%.
Speculative decoding lets large language models verify multiple token guesses in a single forward pass, reducing memory-bound latency without changing output distribution.
Meta engineers built a system that leverages fragmented, idle inference capacity to rank more ads and serve better models without impacting baseline latency.
NVIDIA and Princeton researchers introduce ReplaySSM, a technique that speeds up State Space Model decoding by reducing memory traffic and simplifying speculative rollback.
Mathematicians are using the Lean theorem prover to verify complex proofs. In 2026, AI-driven autoformalization turned manual transcription into a practical reality.
Researchers show that tokenizer-free byte models outperform subword models at scale by learning local text structures internally, enabling faster speculative decoding.
SpaceX’s Grok Bot now selects the best backend model for each task, reflecting a broader industry trend where companies use cheap models for volume and expensive ones for complex work.
Apple researchers introduce Normalizing Trajectory Models to achieve high-quality image generation in four steps while retaining exact likelihood training.
Scale AI cut 89 tasks from its public coding benchmark and locked the evaluation runtime after audits found reward hacking and solution leakage.
AI agents generate bursty, stateful traffic that breaks monolithic APIs. Engineers can fix this by decoupling compute from data using bulkheads and shared databases.
Anthropic launched Claude Haiku 5.5, matching OpenAI's GPT-6 Luna pricing for short contexts but introducing a less efficient tokenizer and higher costs for long inputs.
OpenAI released the Decisions API in public beta, offering typed answers like probabilities and choices from images or text using the gpt-6-luna model.
Burn 0.22 eliminates backend type parameters from user APIs, cutting rebuild times by up to 15x and adding LoRA support.
Google releases EmbeddingGemma 2, a lightweight open model that unifies text, code, audio, and video embeddings for efficient on-device retrieval.
Cloudflare has launched Clef, a 27B parameter model optimized for document extraction and routing. It processes text and images in a single pass via Ollama.