Fine-tuning small browser models for reliable RAG answers

Bart de Goede recently published results from an experiment fine-tuning a small language model to improve retrieval-augmented generation (RAG) directly in the browser. By iterating through four distinct training datasets on a local Mac, he transformed a prone-to-hallucinate 0.6B-parameter model into a more reliable assistant that accurately refuses unanswerable questions. The final model, Qwen3-0.6B, now powers the search function on his blog with significantly fewer errors than the base version.

What happened

The initial attempt to use small language models for summarizing semantic search results failed because the models frequently hallucinated details not present in the source text. De Goede hypothesized that fine-tuning could teach the model to strictly adhere to provided passages or explicitly state when the information was missing. He set out to create a dataset using openly licensed technical writing from the Go, Rust, and Python communities, avoiding his own blog posts to prevent data leakage during evaluation.

The process involved four rounds of training, each revealing specific pitfalls in how small models learn from synthetic data. The first version worsened performance by mimicking the verbose style of the teacher model, while subsequent iterations adjusted the answer format to be shorter and more factual. The third version emerged as the most effective balance, reducing wrong answers by two-thirds compared to the base model, even if it occasionally refused to answer questions it technically could have addressed.

All code, datasets, and the final model weights were released publicly. This allows other developers to replicate the setup, test it on their own documentation, or retrain the model with custom system prompts. The entire training pipeline ran locally on an M4 Mac mini, demonstrating that high-quality fine-tuning for specific RAG tasks does not require massive cloud compute resources.

How it works

The system uses a retrieval mechanism that chunks documents and embeds them for semantic search. When a user asks a question, the system retrieves the top four relevant passages. These passages, along with a strict system prompt, are fed into the Qwen3-0.6B model running in the browser via WebGPU or WebAssembly. The prompt instructs the model to answer in at most two sentences using only the provided text or to decline answering if the passages are insufficient.

Training relied on distillation, where a larger model, Claude Opus 5.5, generated ideal responses for thousands of synthetic questions. De Goede used LoRA (Low-Rank Adaptation) with the mlx-lm library to fine-tune the small model. Each training run consisted of one epoch over 4,500 examples, taking approximately 80 minutes. The key insight was that the small model could not synthesize information across multiple passages; it could only extract specific facts from single passages or refuse.

Key details

• The final model is Qwen3-0.6B, fine-tuned specifically for extracting facts from four retrieved passages.
• Training used 2,193 documents from the Go, Rust, and Python blogs under permissive licenses.
• Version 3 achieved 16 faithful answers out of 30, matching the base model’s accuracy but reducing wrong answers from nine to three.
• The model correctly refused to answer all 15 questions that were not covered by the source material.
• Each training iteration took about 1 hour and 20 minutes on an M4 Mac mini with 24 GB of RAM.
• The dataset and code are public, allowing others to retrain the model for different domains like documentation sites.

Why it matters

For engineers building RAG systems, this experiment highlights that model size is less critical than data quality for specific extraction tasks. Small models running in the browser offer privacy and low latency benefits, but they lack the reasoning capacity to synthesize complex answers. By constraining the task to simple extraction and refusal, developers can deploy lightweight models that are robust against hallucinations without needing expensive server-side inference.

The failure of validation loss to predict real-world performance is another crucial lesson. Standard metrics showed improvement while the model’s actual utility degraded, proving that automated loss curves cannot replace manual evaluation on held-out test sets. Developers must evaluate checkpoints based on the specific behavior they want, such as correct refusal, rather than general linguistic fluency.

This approach also democratizes fine-tuning. By showing that effective models can be trained on consumer hardware in under two hours, it lowers the barrier for teams to customize open-source models for their specific knowledge bases. It shifts the focus from acquiring larger models to designing better training data that matches the cognitive limits of smaller architectures.

What you can do

• Download the bartdegoede/qwen3-0.6b-small-rag model from Hugging Face for immediate use in browser-based applications.
• Create a small eval set of 30 to 50 questions about your own documentation, including some that cannot be answered, to test baseline performance.
• Generate synthetic training data using a large language model, ensuring the answers are short, factual, and derived from single passages.
• Strip hedging phrases like "the text does not say" from positive training examples to prevent the model from learning unnecessary verbosity.
• Monitor checkpoint performance manually on your specific test set rather than relying solely on validation loss metrics during training.
• Consider retraining the model with custom system prompts if your application refers to "documentation" instead of a "blog" to improve contextual alignment.