Nenya-2.5B brings efficient linear attention to local AI development

ArsalanZ has released Nenya-2.5B-A0.5B-Base, a new open-weight language model designed for efficient local execution. Published on Hugging Face in November 2026, this model targets developers who need capable AI assistants that run directly on consumer hardware without relying on cloud APIs. The release includes detailed benchmark data comparing its performance against established small models like OLMo-2 and Qwen3.5.

The model distinguishes itself by leveraging specific architectural choices that prioritize speed on modern GPUs, particularly Apple Silicon and NVIDIA cards. By utilizing Gated Delta Networks (GDN-2) and optimized Triton kernels, Nenya-2.5B achieves high token throughput while maintaining a relatively small memory footprint. This makes it an attractive option for engineers building privacy-focused applications or working in environments with limited connectivity.

What happened

The core event is the public release of the Nenya-2.5B-A0.5B-Base model weights and associated evaluation scripts. The model file is compact, weighing in at just 3.5 GB, which allows it to download quickly and fit easily into the memory of most modern laptops. Upon first use, the system fetches these weights automatically, streamlining the setup process for developers who want to test the model immediately.

Performance metrics highlight the model's efficiency on specific hardware. On an Apple M4 chip, the model runs directly on the GPU, achieving a generation speed of approximately 125 tokens per second. This level of performance enables real-time interaction for many use cases, such as code completion or interactive chatbots, without the latency typically associated with larger models running on central processing units. The release notes emphasize that this speed is contingent on using the correct backend configurations.

The author also provided comprehensive benchmark results using the lm-evaluation-harness version 0.4.13. These evaluations were conducted under strict conditions to ensure fairness, using bf16 weights and a 4,096-token context window. The benchmarks cover a wide range of tasks, including general knowledge, reasoning, and coding abilities. The results position Nenya-2.5B as a competitive alternative to other small models, despite being trained on significantly fewer tokens than some of its counterparts.

How it works

Nenya-2.5B relies on GDN-2 layers, a type of linear attention mechanism that differs from the standard softmax attention found in many transformer models. Linear attention scales more efficiently with sequence length, reducing the computational burden during inference. However, to realize these performance gains, the model requires specific software optimizations. On NVIDIA GPUs, users must install the flash-linear-attention library, specifically the CUDA extra, to enable Triton kernels. These kernels allow the GDN-2 layers to execute efficiently on the GPU hardware.

If the optimized kernels are not installed, or if the model is run on a CPU or through standard transformers on a Mac without proper configuration, it falls back to a pure-PyTorch recurrence implementation. While this fallback produces identical results, it is significantly slower, especially when processing long prompts. This distinction is crucial for developers deploying the model in production environments where latency matters. The model supports both bf16 and fp32 precision, though bf16 is recommended as it matches the training and evaluation settings.

For sampling, the model uses default parameters of temperature 0.9 and top-k 20, which encourage diverse and creative outputs. For tasks requiring deterministic answers, such as mathematics, code generation, or short factual queries, users can set the temperature to 0. This enables greedy decoding, which selects the most likely next token at each step, ensuring consistency and accuracy in structured outputs.

Key details

• Model Size: The weight file is 3.5 GB, making it easy to distribute and load on consumer devices.
• Hardware Performance: Achieves ~125 tokens per second on an Apple M4 GPU when configured correctly.
• Training Data: Trained on 0.06 trillion tokens, which is less than many competing models but still yields competitive results.
• Precision: Evaluated and recommended for use in bf16 precision, though fp32 is supported.
• Optimization: Requires flash-linear-attention with CUDA extras for optimal performance on NVIDIA GPUs; otherwise, it uses slower PyTorch recurrence.
• Context Window: Benchmarks were conducted using a 4,096-token window.

Why it matters

For software engineers and technical founders, the release of Nenya-2.5B highlights a trend toward more efficient local AI models. The ability to run a capable language model on a laptop GPU without internet access opens up new possibilities for privacy-sensitive applications. Developers can build tools that process sensitive data locally, ensuring that user information never leaves the device. This is particularly relevant for industries like healthcare, finance, and legal tech, where data sovereignty is a critical requirement.

Moreover, the model's performance relative to its training size suggests that architectural innovations like GDN-2 can compensate for smaller datasets. While models like SmolLM3 and Gemma 4 were trained on over 10 trillion tokens, Nenya-2.5B achieves comparable scores in several benchmarks with only 0.06 trillion tokens. This efficiency could lower the barrier to entry for teams wanting to fine-tune or customize models, as they may not need massive datasets to achieve useful performance levels.

The detailed benchmarking also provides a transparent view of where small models stand today. By comparing against established baselines like OLMo-2 and Qwen3.5, developers can make informed decisions about which model fits their specific needs. For instance, if code generation is a priority, the bits-per-byte metrics on GitHub data offer a clear comparison. This transparency helps the community avoid hype and focus on practical, measurable capabilities.

What you can do

• Download the 3.5 GB weight file from Hugging Face to test the model on your local machine.
• Install flash-linear-attention with the CUDA extra if you are using an NVIDIA GPU to ensure optimal speed.
• Use temperature 0 for tasks requiring precise outputs, such as code generation or mathematical problem-solving.
• Run the provided evaluation scripts to benchmark the model on your specific hardware and use case.
• Consider using bf16 precision for inference to match the model's training conditions and maximize performance.
• Explore fine-tuning the model on your own domain-specific data, leveraging its efficient architecture for faster iteration.