Training-free decision routing for local MLX models on Apple Silicon

A new open-source Python library called MLX Direct Decider allows developers to implement training-free decision engines using existing machine learning models on Apple Silicon hardware. Released in mid-October 2026, the tool enables local large language models to perform action selection, classification, and semantic routing without requiring additional fine-tuning or separate decision weights.

The project targets engineers building local AI applications who need reliable structured outputs from generative models. By leveraging the native capabilities of the MLX framework, it provides a lightweight method to convert natural language requests into stable business identifiers or explicit rejection states.

What happened

MLX Direct Decider is an experimental library designed to extract decisions directly from the output logits of compatible MLX-LM and MLX-VLM checkpoints. Instead of generating text continuations that must be parsed later, the library reads candidate scores from the model’s unchanged original output head. This approach allows applications to select actions, answer boolean questions, or route semantics based on current application state and user utterances.

The system supports dynamic choices, meaning developers can update allowed actions as the application state changes while maintaining stable business IDs. It explicitly handles cases where no valid action exists by returning no_match or abstain outcomes. The library integrates via a Python API, command-line interface, or a local text HTTP service, requiring only native ARM Python 3.11 or higher and a local model checkpoint.

Version 0.1 remains experimental and has not been published to PyPI. It supports text decisions through MLX-LM and experimental text-plus-image decisions through MLX-VLM. The project includes verified checkpoints for models like Qwen and provides tools for version protection to reject stale decisions after state changes.

How it works

The core mechanism bypasses traditional text generation for decision tasks. When an application sends a request containing state, a user utterance, and a list of allowed choices, the model processes the prompt normally. However, instead of sampling tokens to form a sentence, the library intercepts the logits at the final position of the sequence. These logits represent the model’s confidence scores for each potential token or candidate.

The engine maps these raw scores to the provided candidates and selects the one with the highest stability. This method avoids the computational overhead of generating full text responses and the subsequent parsing errors often associated with JSON extraction from LLMs. For visual tasks, the MLX-VLM backend performs similar operations on combined text and image inputs, though it currently recomputes the full context for each request.

Prefix reuse is a key optimization feature. The system allows stable context to be reused while evaluating new utterances, reducing latency for subsequent decisions on the same page or state. However, final answers are never cached to ensure freshness. The output includes the candidate ID, raw logits, scores, margin, and timing details, giving developers visibility into the model’s confidence levels.

Key details

• No fine-tuning required: Uses existing MLX-LM or MLX-VLM checkpoints without additional training or decision-specific weights.
• Explicit rejection handling: Returns no_match or abstain when no action is appropriate, preventing hallucinated commands.
• Multi-modal support: Supports text-only decisions via MLX-LM and experimental text-plus-image decisions via MLX-VLM.
• Performance benchmarks: On an Apple M2 Max with 64 GiB RAM, Qwen3.5-9B achieved 83.3% accuracy on a 36-case evaluation with p50 latency of 194.8 ms.
• Integration options: Available via Python API, CLI, or localhost HTTP service with versioned execution authorization.
• Experimental status: Version 0.1 is not on PyPI and requires manual installation from the source repository.

Why it matters

For developers building local AI agents, reliability and latency are critical bottlenecks. Traditional methods often involve prompting the model to output JSON or specific code structures, which can fail due to formatting errors or verbose explanations. MLX Direct Decider offers a more robust alternative by treating decision-making as a classification task rather than a generation task. This reduces the risk of parsing failures and provides clearer signals when the model is uncertain.

The ability to run these decisions entirely on-device without cloud dependencies enhances privacy and reduces operational costs. By reusing model weights that developers may already have for chat or completion tasks, the library lowers the barrier to implementing complex agent behaviors. It allows for tighter integration between the AI model and the application’s internal state management, ensuring that actions are validated against current context before execution.

However, the experimental nature of the tool means it is not yet a drop-in solution for production systems. The accuracy varies significantly between models, as seen in the benchmarks where GLM-4.7-Flash achieved only 58.3% accuracy compared to Gemma’s 86.1%. Developers must carefully evaluate model performance for their specific use cases and implement guards against incorrect actions, as the scores are not calibrated probabilities of correctness.

What you can do

• Clone the repository from GitHub and install the library in a virtual environment using Python 3.11+ on Apple Silicon.
• Test the provided Qwen3.5-9B checkpoint with the included 36-case evaluation suite to benchmark baseline performance.
• Integrate the Python API into a local application to handle enum or boolean decisions using your existing MLX-LM models.
• Experiment with the MLX-VLM backend for simple image-based decision tasks, noting the current limitation of full context recomputation.
• Implement state version checks in your application to ensure that decisions are executed only if the application state has not changed since the request was made.
• Review the detailed testing methodology and failure cases in the project documentation to understand the limits of direct logit scoring.