LookAtMyQuant compiler cuts model size with minimal accuracy loss
A new open-source quantization compiler uses trellis codes and Fisher information to compress large language models significantly while maintaining high benchmark scores.
GitHub user ShinMegamiBoson released LookAtMyQuant, an open-source quantization compiler that achieves substantially lower divergence from original models compared to standard methods. Published on November 21, 2026, the project demonstrates how targeted compression can reduce file sizes by more than half without sacrificing critical reasoning capabilities. The release includes detailed benchmarks for several recent models and a dedicated runtime for Apple Silicon hardware.
What happened
The core achievement of LookAtMyQuant is its ability to compress models like Gemma 4 E2B into GGUF formats with twelve times lower Kullback-Leibler divergence than the common Q4_K_M method at identical file sizes. The compiler does not apply uniform compression across the entire network. Instead, it measures the impact of compressing each specific component and allocates bits where they matter most. This approach allows developers to fit larger, more capable models into constrained memory environments without the typical degradation in output quality.
The project also introduces LookAtMyQuant 2, a refined recipe that compressed Qwen3.5-4B to just 1.23 GB. Despite this extreme reduction to 2.3 bits per weight, the model solved 330 out of 540 exact-answer math problems, nearly matching the performance of much larger counterparts. For the larger Qwen3.8-27B model, the compiler reduced the footprint from 53.7 GB to 7.2 GB. In this configuration, the compressed model matched the original’s score on complex math benchmarks and outperformed other quantized versions on code generation and general knowledge tests.
How it works
LookAtMyQuant operates as a post-training quantization tool that treats compression as an optimization problem. It calculates the Fisher information for each row of weights to determine its sensitivity to changes. Rows that contribute more to the model’s output precision receive more bits, while less critical rows are compressed more aggressively. The compiler uses trellis codes, which are efficient error-correcting codes, to represent the weights. This method ensures that the total file size stays within a strict budget while minimizing the overall distortion of the model’s behavior.
The process involves several distinct stages. First, the compiler measures the KL divergence caused by compressing individual parts of the model. It then creates a plan that allocates bits per row based on the calculated importance. Finally, it performs a distillation step where the model is fine-tuned for thousands of steps. During this phase, every layer is re-quantized once from the original weights, helping the model adapt to the new compressed representation. The entire pipeline is designed to be automated, with scripts available to run each job on cloud GPU infrastructure.
Key details
- The compiler achieved 12× lower KL divergence than Q4_K_M for Gemma 4 E2B at the same file size.
- Qwen3.5-4B was compressed to 1.23 GB, solving 330 of 540 math problems, close to the original’s 359.
- Qwen3.8-27B was reduced to 7.2 GB, matching the original model’s score on the GSM8K math benchmark.
- The Inkling-2-Mini model was compressed into four variants, with the 12.8 GB version matching 100% of the teacher’s benchmark scores.
- The Apple Silicon runtime supports speculative decoding, reaching over 100 tokens per second on median tasks for the 12.8 GB Inkling build.
- The full recipe for Qwen3.5-4B cost approximately $34 in GPU time to execute on cloud infrastructure.
Why it matters
For engineers building local AI applications, memory bandwidth and storage are often the primary bottlenecks. Traditional quantization methods like 4-bit or 8-bit integer conversion are easy to implement but often result in noticeable drops in reasoning ability, especially for complex tasks like coding or mathematics. LookAtMyQuant provides a path to retain high-level cognitive performance in models that are small enough to run on consumer hardware. This enables more sophisticated agents to operate on laptops and edge devices without relying on constant cloud connectivity.
The availability of the code under the MIT license also lowers the barrier for experimentation. Developers can now apply these advanced compression techniques to their own fine-tuned models. By using Fisher information to guide bit allocation, teams can create custom quantization plans that prioritize the specific skills their model needs, such as code generation or multi-language support. This level of control was previously reserved for large research labs with significant computational resources.
What you can do
- Install the Apple Silicon runtime using pip to test the pre-compressed Qwen and Inkling models locally.
- Use the
lamq generatecommand to run inference with speculative decoding for faster response times. - Run the compiler on smaller models like SmolLM2-360M to understand the measurement and allocation workflow.
- Examine the provided row plans to see how bits are distributed across different layers of the network.
- Adapt the
modal_app.pyscripts to run the full quantization recipe on your own models using cloud GPUs. - Compare the KL divergence metrics of your current quantization methods against the LookAtMyQuant baseline.



