Fine-tuning a 235B model on four GPUs: lessons from a failed deployment

Engineers at Vertexcover successfully fine-tuned the 235-billion-parameter Qwen3 model on a single machine with four H100 GPUs, achieving significant accuracy gains over paid API models. Despite the technical success and reduced per-token costs, the team decided against deploying the solution after newer open-source models rendered the complex infrastructure unnecessary.

What happened

The team aimed to replace expensive fine-tuned OpenAI and Gemini models used for entity classification in customer documents. They selected Qwen3-235B-A22B-Thinking, a Mixture of Experts (MoE) model, because it outperformed other open models in initial benchmarks. The goal was to maintain or improve accuracy while controlling costs through self-hosted hardware.

After one epoch of training on approximately 3,600 examples, the fine-tuned model improved the average F1 score by 7.6 points across all tenants. It also reduced the per-token cost to about one-third of the previous API-based solutions. The training process took 26 hours on four NVIDIA H100 GPUs, each with 80 GB of memory.

Despite these positive metrics, the project was shelved. The team found that serving the 235B thinking model with low latency and reasonable operational costs was a larger engineering challenge than the fine-tuning itself. Concurrently, new open-source models emerged that handled many classification tasks without any fine-tuning, eliminating the need for their custom infrastructure.

How it works

Qwen3-235B is an MoE model where only about 22 billion of its 235 billion parameters are active for any given token. Each layer contains 128 small expert networks, and a router directs tokens to the most relevant experts. This architecture keeps inference computationally cheaper than dense models of similar size but complicates memory management during loading and training.

The team used QLoRA (Quantized Low-Rank Adaptation) to fine-tune the model. This method compresses the base model to 4-bit precision, reducing its memory footprint from 470 GB to roughly 120 GB. Small adapter layers are then trained on top of the frozen base weights. The challenge lies in coordinating compression and sharding libraries, which have conflicting requirements for how model weights are stored and accessed in memory.

Successful training required DeepSpeed ZeRO-3 with CPU offload. This setup kept complete copies of the 120 GB compressed model in system RAM rather than GPU memory. During training, layers were moved from RAM to the GPUs as needed, allowing the four H100s to handle the workload despite individual memory limits. This approach traded speed for feasibility, resulting in the 26-hour training time.

Key details

• The model improved F1 scores for every tenant, with gains ranging from +4.81 to +11.96 points.
• Training used 3,600 examples including reasoning traces, as Qwen3 is a "thinking" model.
• Four H100 GPUs with 80 GB memory each and 960 GB of system RAM were required.
• Axolotl, Unsloth, and Answer.AI’s FSDP+QLoRA implementations failed due to memory or compatibility issues.
• LlamaFactory with standard FSDP failed because Transformers silently disabled memory-saving settings during quantization.
• The final working setup used LlamaFactory with DeepSpeed ZeRO-3 and CPU offload to manage memory.

Why it matters

This case highlights the hidden complexity of fine-tuning massive MoE models on limited hardware. Engineers often assume that quantization and sharding will seamlessly fit large models into available GPU memory, but library incompatibilities and memory ordering issues can prevent training from starting. Understanding these low-level constraints is critical for planning infrastructure for models exceeding 100 billion parameters.

It also serves as a reminder that technical capability does not always justify deployment. The effort to optimize inference for a 235B model can outweigh the benefits of fine-tuning, especially when the base model is already strong. Teams must evaluate the total cost of ownership, including serving latency and hardware requirements, not just training accuracy.

Finally, the rapid pace of open-source development means that today's hard engineering problem may be solved by a model release tomorrow. Investing weeks in custom fine-tuning pipelines carries the risk of sunk costs if a general-purpose model improves enough to handle the task out of the box. Strategic patience can sometimes yield better results than immediate customization.

What you can do

• Benchmark multiple open models before committing to fine-tuning, as base performance varies significantly.
• Verify library compatibility for MoE models, especially regarding quantization and sharding order.
• Ensure sufficient system RAM for CPU offloading when GPU memory is insufficient for the full compressed model.
• Monitor Transformers settings for silent changes to memory management during quantization processes.
• Evaluate inference costs and latency requirements early, not just training accuracy and cost.
• Reassess the need for fine-tuning regularly, as new base models may close performance gaps without customization.