Sibyl offers a hardened, self-hosted AI inference server for local GPUs

A new open-source project called Sibyl aims to simplify the deployment of private, local large language model inference servers. Released in pre-alpha status on GitHub by diffsec, the tool transforms a computer equipped with one or more graphics processing units into a secure, self-contained AI host. It supports hardware from AMD, NVIDIA, and Intel, offering a unified interface for managing models and serving requests without relying on external cloud providers.

What happened

Sibyl is designed as an all-in-one inference server that handles everything from operating system security to model management. The project provides a hardened host based on Fedora bootc, which ensures the operating system is an immutable, signed image. This approach allows for atomic updates and rollbacks, reducing the risk of configuration drift or corruption. The system enforces SELinux policies, enables Secure Boot, and encrypts the disk using LUKS2, with decryption keys managed by the TPM2 chip.

The software stack includes llama.cpp for serving GGUF models, utilizing llama-swap to load and switch models on demand. This architecture allows a single model to span multiple GPUs or permits each GPU to run a different model simultaneously. Sibyl also integrates Open WebUI, providing a chat interface with document search capabilities powered by a local embedding model. For developers, it exposes standard APIs, including OpenAI-compatible, Anthropic Messages, and Ollama-compatible endpoints, all secured behind TLS and per-program API keys.

Installation is designed to be straightforward, supporting deployment via USB stick or over the network using PXE. The installer erases the target disk, prints a recovery key, and sets up the administrator account. Once installed, users can manage the server through a web console or a terminal command-line interface. These consoles provide access to setup wizards, status monitoring, model downloads, router configuration, and system updates. The project also includes Arcane, a feature that allows users to run their own containers alongside the inference stack.

How it works

Sibyl operates by combining a secure, immutable operating system with a flexible model routing layer. At the core is a model router that can direct requests to local models or external cloud providers based on user-defined rules. This router supports an "auto" model setting, which selects the appropriate backend for each request. Users can set budgets for specific programs and create rules that ensure any request containing secrets remains on the local host, preventing data leakage to third-party services.

Model management is handled directly within the system, which searches Hugging Face for compatible files. It evaluates whether models fit the available GPU memory and handles downloads with resume capability and checksum verification. Each model can be configured with specific parameters for context size, cache, sampling, reasoning, and speculative decoding. This granular control allows engineers to optimize performance and resource usage for different workloads without manually editing configuration files outside the provided interfaces.

Key details

• Hardware support: Variants exist for AMD (Radeon and Ryzen AI), NVIDIA (GeForce and data-center GPUs from Turing onwards), and Intel (Arc and recent integrated graphics).
• Backend technologies: AMD uses HIP (ROCm) and Vulkan; NVIDIA uses CUDA and Vulkan; Intel uses Vulkan. The AMD variant has been tested on Radeon RX 7900 XTX cards.
• API compatibility: Serves OpenAI-compatible (/v1), Anthropic Messages, and Ollama-compatible (/api) endpoints behind TLS.
• Security features: Uses Fedora bootc for immutable OS updates, enforcing SELinux, Secure Boot, and LUKS2 disk encryption unlocked by TPM2.
• Management interfaces: Provides both a web console at port 8443 and a terminal console command sibyl for full system control.
• License: Released under the GNU Affero General Public License v3.0, with commercial licenses available from the maintainer.

Why it matters

For software teams building AI-enabled products, data privacy and infrastructure control are critical concerns. Cloud-based inference services often require sending sensitive data to external servers, which can violate compliance requirements or expose proprietary information. Sibyl addresses this by keeping all inference local, ensuring that data never leaves the premises unless explicitly routed by user-defined rules. The immutable nature of the host operating system further reduces the attack surface, making it suitable for environments where security audits and strict access controls are mandatory.

The standardization of APIs also lowers the barrier to entry for adopting local inference. Developers can use existing clients and libraries designed for OpenAI or Ollama without rewriting code. This compatibility means teams can switch between cloud and local providers seamlessly, or use a hybrid approach where sensitive tasks run locally and less critical ones use cloud resources. The ability to manage models, monitor usage, and configure routing through a unified console simplifies operations, reducing the DevOps overhead typically associated with self-hosted AI infrastructure.

What you can do

• Download the live ISO for your specific GPU vendor variant using the oras pull command provided in the documentation.
• Write the ISO to a USB stick and boot your target machine with Secure Boot enabled to begin installation.
• Review the hardware compatibility list, noting that only the AMD variant has been fully tested on physical hardware so far.
• Explore the model router settings to define rules that keep sensitive requests local while allowing others to use cloud providers.
• Contribute to the project by testing the NVIDIA or Intel variants if you have access to compatible hardware, and report issues on GitHub.
• Read the CONTRIBUTING.md file to understand the process for signing the Contributor License Agreement before submitting patches.