Offline dictation uses screen text trees to boost accuracy without LLMs

A developer has released a technical breakdown of Handy Context, an offline dictation tool that improves accuracy by reading the text currently visible on a user’s screen. Published in November 2026, the project demonstrates how to bias a speech model using accessibility trees rather than screenshots or large language models. The approach runs entirely locally, offering a privacy-preserving alternative to cloud-based transcription services.

What happened

The creator of Handy Context detailed a pipeline that integrates screen context into speech recognition without retraining the underlying acoustic model. The system uses Parakeet-unified-en-0.6b, a frozen speech model, and enhances it by extracting text from the active window’s accessibility tree. This method allows the dictation software to prioritize words that are actually present in the user’s current workflow, such as code variables, slide titles, or document headers.

Initial experiments involved training additional layers on top of the frozen model to copy list words, similar to TCPGen techniques. These trained versions performed worse than the untrained pipeline, often introducing errors when the provided word list was unrelated to the spoken audio. The developer also tested small keyword spotters to filter screen words down to likely candidates. While these spotters reduced a 1,000-word list to about three terms while retaining 98% of correct matches, the untrained pipeline still achieved better overall scores, so the trained components were not included in the final release.

Several other common approaches failed to improve performance. Using a speech LLM like Qwen3-ASR-1.7B with a 1,000-word context did not reduce errors compared to having no list at all, while running ten times slower. Prompting Whisper large-v3-turbo with a vocabulary increased its word error rate from 3.53% to between 4.8% and 5.6%. Similarly, applying VoiceInk-style swapping raised the error rate on real slides from 8.25% to 8.79%, largely due to incorrect changes in word endings.

How it works

The system captures context by walking the accessibility tree of the front window when a shortcut is pressed. It collects text from elements like static text fields, headings, buttons, and table cells, stopping after 4,000 elements or 350 milliseconds. This data is combined with the window title and the last 8,000 characters of the focused field. The extracted text is split into terms, filtering out common words and short strings, while preserving capitalization for acronyms. The list is expanded with inflections found in the LibriSpeech vocabulary to cover plural forms and spelling variants.

Audio is encoded once and decoded up to nine times using different biasing strategies. The core mechanism uses NeMo’s prefix-tree booster to add scores to tokens that match terms from the screen context. The system generates multiple candidate transcripts, including plain decodes and two-pass decodes that filter terms based on similarity to the initial transcript. Similarity is calculated using Levenshtein distance and Metaphone codes. The final candidate is chosen by computing its exact log-likelihood under the unbiased model and adding a bonus for every word that appears in the expanded term list.

Post-processing steps further refine the output. The system proposes swaps from the winning text to terms on the expanded list if they meet a similarity threshold. It avoids swaps that only change word endings, a common source of errors in previous attempts. Capitalization is adjusted to match the on-screen spelling for proper nouns and acronyms. This entire process runs on a local server, posting audio and context to receive text back in under half a second for short dictations.

Key details

• The pipeline uses Parakeet-unified-en-0.6b at revision fe53cd8 without fine-tuning the model weights.
• Screen text is captured via accessibility trees from Chromium and Electron apps, limiting support to those frameworks for now.
• Memory usage for Handy Context is approximately 7.1 GiB, significantly higher than the base Handy model’s 0.84 GiB due to the PyTorch runtime.
• The system performs nine parallel decodes with varying bias strengths and two-pass filtering thresholds to find the best match.
• Evaluation on SlideSpeech and LibriSpeech datasets showed fewer list-word errors than both plain Parakeet and VoiceInk baselines.
• The tool currently supports English only and runs on a single desktop operating system, with no OCR capability for image-based text.

Why it matters

For engineers building voice interfaces, this work highlights the diminishing returns of simply scaling up models or adding LLMs for context awareness. The failure of Qwen3-ASR-1.7B to improve accuracy despite its size suggests that architectural fit and efficient biasing mechanisms matter more than raw parameter count for specific tasks. By relying on accessibility trees instead of OCR or screenshots, the system achieves low-latency context integration without the computational overhead of visual processing.

The memory footprint remains a significant barrier to widespread adoption. Running the full precision model in PyTorch consumes over 7 GiB of RAM, which is prohibitive for many consumer laptops. However, the author notes that porting the pipeline into a custom engine could reduce this to under 4 GiB. This trade-off between accuracy and resource usage is a critical consideration for developers aiming to deploy offline AI features on edge devices.

The success of the untrained pipeline over trained alternatives also challenges the assumption that fine-tuning is always necessary for domain adaptation. By leveraging robust decoding strategies and post-hoc scoring, developers can achieve high accuracy with frozen models. This approach simplifies maintenance and reduces the risk of catastrophic forgetting, making it a viable pattern for applications where context changes rapidly.

What you can do

• Extract text from application accessibility trees to build dynamic vocabularies for speech recognition tasks.
• Use prefix-tree boosting to bias existing speech models toward domain-specific terms without retraining.
• Implement two-pass decoding to filter context lists based on similarity to initial transcription hypotheses.
• Apply post-processing rules to prevent erroneous swaps that only alter word endings or suffixes.
• Monitor memory usage when integrating PyTorch models into desktop applications and consider custom engines for optimization.
• Evaluate context-aware systems using list-word error rates alongside standard word error rates to measure specific improvements.