Building with AI

Optimizing memory for long-context fine-tuning of open-weight models

A technical deep dive into reducing training time and memory usage when fine-tuning Qwen3.6-27B on 70K-token examples using FSDP, LoRA, and shared instruction sets.

Illustration of neural network layers connected to a document
Illustration generated for this article

Hardik Meisheri recently detailed a rigorous optimization process for fine-tuning Qwen3.6-27B, an open-weight vision-language model, on tasks requiring inputs up to 70,000 tokens. By addressing memory bottlenecks in Fully Sharded Data Parallel (FSDP) training and eliminating redundant computation for shared system prompts, he reduced experiment cycles from weeks to under 20 hours.

What happened

Meisheri aimed to fine-tune the model on a multimodal task where each training example included a lengthy system prompt containing instructions, rules, and worked examples. These prompts accounted for roughly 64,000 of the 70,000 tokens per example. Because the instructions evolved over time, keeping them in the prompt rather than baking them into weights allowed for easier updates without catastrophic forgetting. However, training on such long sequences consumed massive GPU memory, as the backward pass required retaining intermediate results for every token position.

Initial attempts to train single examples on 80 GiB H100 GPUs failed due to out-of-memory errors. The activation tensors alone for a 70,000-token sequence occupied approximately 43 GiB before accounting for recomputation overhead. Additionally, standard optimization techniques like Low-Rank Adaptation (LoRA) introduced unexpected memory spikes when dropout was applied to specific projection layers. Collective communication hangs under FSDP further stalled progress, caused by version mismatches and divergent execution paths across GPU ranks.

After resolving these infrastructure and memory issues, Meisheri implemented a method to share the computation of the system prompt across multiple examples. Since the prompt is identical for many items, its forward pass representations could be computed once and reused. This approach leveraged the causal nature of the transformer, where earlier tokens do not depend on later ones. The result was a significant speedup, moving from five-to-seven-day workflows to overnight runs, although the exact acceleration varied due to differences in data mixes.

How it works

The core challenge lies in how transformers handle long sequences during backpropagation. In a standard setup, gradients flow back through all 70,000 positions because the target output depends on the entire context. Activation checkpointing helps by storing only layer inputs and recomputing intermediates during the backward pass, but this still leaves large tensors in memory. For Qwen3.6-27B, which uses a mix of full attention and recurrent Gated DeltaNet layers, specific kernel autotuning was required. Triton’s autotuner initially failed on the H100 GPUs because it attempted to benchmark tile sizes for a sequence length bucket it had never encountered, consuming the remaining free memory. Pinning a default configuration resolved this.

Memory usage was further optimized by adjusting LoRA adapters. Applying dropout to the down_proj layer created large temporary tensors for the mask and dropped-out input, adding 3.4 GiB per layer. Removing the adapter from this specific projection eliminated the spike. To enable parallel processing of shared prompts, Meisheri avoided standard attention masks, which are ignored by the model’s recurrent layers. Instead, he structured the data so that multiple continuations branched off a single computed prompt buffer, ensuring that each example remained isolated while reusing the initial computation.

Key details

  • Model and Hardware: Qwen3.6-27B fine-tuned on NVIDIA H100 GPUs with 80 GiB memory using BF16 precision.
  • Sequence Length: Training examples reached 70,000 tokens, with 64,000 tokens dedicated to static system prompts.
  • Memory Optimization: Activation checkpointing reduced storage needs, but recomputation of feed-forward intermediates still required careful management of temporary buffers.
  • Kernel Tuning: Flash Linear Attention kernels required pre-computed default configurations to prevent Triton autotuner crashes during the first backward pass.
  • LoRA Configuration: Adapters were applied to query, key, value, output, and gate projections, but removed from down_proj to avoid excessive memory use from dropout masks.
  • Shared Computation: System prompts were processed once for groups of 75 to 90 examples, reducing total training time from days to under 20 hours for two epochs.

Why it matters

For engineers working with open-weight models, this case study highlights that theoretical memory savings from sharding techniques like FSDP do not always translate directly to practice when sequence lengths are extreme. Activations remain local to each GPU, meaning that doubling the number of GPUs halves weight storage but does not reduce the activation footprint per device. Understanding this distinction is critical for planning hardware requirements for long-context tasks.

The findings also demonstrate the fragility of distributed training setups. Minor version mismatches in dependencies like Transformers can cause silent hangs where GPUs appear busy but perform no useful work. Monitoring both utilization and power draw provides a more reliable diagnostic signal than logs alone. Furthermore, the interaction between dropout, LoRA ranks, and memory allocation shows that standard library defaults may not be optimal for very long sequences, requiring manual intervention to prevent out-of-memory errors.

What you can do

  • Pin kernel configurations: If using custom kernels like Flash Linear Attention, pre-run short benchmarks to generate default configs and disable runtime autotuning for long sequences.
  • Audit LoRA placements: Avoid applying dropout to high-dimensional projection layers like down_proj if memory is tight, as the mask and temporary copies consume significant space.
  • Verify collective consistency: Ensure all nodes import identical library versions and follow the same execution path to prevent FSDP hangs during all-gather operations.
  • Enable input gradients: Call model.enable_input_require_grads() before preparing the model to ensure gradients flow correctly through frozen embeddings.
  • Disable mixed precision autocast: When using activation checkpointing with BF16 weights, turn off automatic mixed precision to prevent dtype mismatches during recomputation.
  • Monitor power and utilization: Use tools like py-spy and nvidia-smi to distinguish between genuine computation and collective communication stalls.
  • #fine-tuning
  • #llm
  • #memory-optimization
  • #fsdp
  • #lora

Tools from the Bytechap store

Keep reading

All stories