Jepsen exposes critical safety and stability flaws in ParticleDB 1.1.4

Independent testing firm Jepsen has released a comprehensive report on ParticleDB version 1.1.4, uncovering significant safety violations and stability issues in the database engine. The analysis, conducted between versions 1.1.4 and 1.1.18.11.rc-22, highlights discrepancies between marketing claims of formal verification and the actual behavior of the system under load. For engineers evaluating infrastructure for AI agents or high-throughput applications, these findings raise serious questions about the readiness of the platform for production use.

What happened

ParticleDB, developed by Orkes to support their Conductor workflow orchestration platform, aims to unify transactional SQL, key-value storage, vector search, and analytics into a single Rust-based engine. The company markets the database as offering full ACID compliance and performance orders of magnitude faster than PostgreSQL, with a focus on AI agent workloads. To validate these claims ahead of a broader release, Orkes engaged Jepsen to test the refactored codebase, starting with version 1.1.4. The tests focused on single-node behavior, as the provided builds did not support the replicated cluster features available only in enterprise cloud deployments.

The Jepsen team identified 96 distinct issues across various categories, including availability, broken features, transaction concurrency, and state corruption. While some problems were resolved in later release candidates, the report details severe anomalies such as spontaneous crashes without error messages, deadlocks after just two concurrent transactions, and silent ignoring of transaction isolation levels. The testing process involved simulating process crashes, disk corruption, and network faults to stress the system’s durability and consistency guarantees.

A major point of contention was the documentation surrounding transaction isolation and durability. ParticleDB’s homepage claimed "Full ACID" and "formally proven" safety via TLA+, yet no TLA+ specification was provided during the collaboration. Furthermore, documentation for transaction modes like occ (Optimistic Concurrency Control) and groupsync (WAL syncing) contained contradictory information about whether they offered strong durability or serializable isolation. In many cases, the database silently ignored standard SQL commands for setting isolation levels, defaulting instead to startup flags that were poorly explained in public docs.

How it works

Jepsen tests distributed systems by injecting faults and observing whether the system maintains its stated consistency models. For ParticleDB, the team used a combination of existing SQL workloads and new tests designed specifically for the database’s unique features. They employed the Elle checker to analyze transaction dependency graphs for cycles, which indicate violations of serializability or snapshot isolation. The tests ran against both the official Postgres JDBC client and the native ParticleDB Java SDK, though the latter was initially unavailable as a Maven dependency.

The testing suite included workloads that performed random reads and writes, appends to list structures, and internal consistency checks within single transactions. A notable method was the compare workload, which generated random SQL statements and executed them against both ParticleDB and PostgreSQL. If the results diverged, the tool minimized the test case to reproduce the bug. This approach revealed that ParticleDB often failed to match PostgreSQL’s behavior for basic queries, indicating incomplete or buggy implementation of the Postgres wire protocol.

Fault injection played a key role in exposing durability issues. By using LazyFS to simulate the loss of un-fsynced writes and killing processes abruptly, Jepsen verified whether the Write-Ahead Log (WAL) correctly preserved data. The tests covered three WAL sync modes: sync, groupsync, and nosync. Discrepancies between the documented behavior of these modes and their actual performance led to several findings where data could be lost or corrupted even when users believed they had chosen safe settings.

Key details

• Jepsen found 96 issues in ParticleDB, ranging from performance bottlenecks to critical data safety violations.
• Documentation claimed "formally proven" safety via TLA+, but no specification was provided to verify this claim.
• Transaction isolation levels set via SQL were often silently ignored, with behavior governed by opaque startup flags instead.
• Severe deadlocks occurred in versions up to 1.1.18.11.rc-6, sometimes freezing the node after just two concurrent transactions.
• Support for standard Postgres features was incomplete, with failures in prepared statements, aggregates in CASE expressions, and exponentiation operators.
• Spontaneous crashes without error logs were observed in version 1.1.18.6, rendering the system unreliable for long-running tasks.

Why it matters

For software engineers and technical leads, this report serves as a cautionary tale about relying on marketing claims for critical infrastructure. ParticleDB positions itself as a unified solution for AI agents, a domain where data consistency is paramount. The discovery that transaction isolation levels can be silently ignored means that applications assuming strong consistency may suffer from subtle, hard-to-debug race conditions. When a database claims "Full ACID" but behaves with snapshot isolation or weaker guarantees, developers may design systems that fail under concurrent load.

The instability issues, including deadlocks and unexplained crashes, directly impact system availability. In a production environment, a database that freezes after a few seconds of concurrent activity or crashes without logging errors is operationally unsustainable. These flaws force teams to invest heavily in monitoring and workaround strategies, negating the promised performance benefits. Moreover, the incomplete implementation of the Postgres wire protocol means that migrating existing applications to ParticleDB is not a drop-in replacement, requiring significant refactoring and testing.

The discrepancy between public documentation and internal operator guides also highlights a maturity gap. When essential safety properties are described differently in help text, homepages, and internal docs, users cannot make informed decisions about risk. For organizations considering ParticleDB for AI workflows, the lack of transparent, verifiable safety proofs and the presence of known data corruption bugs suggest that the technology is not yet ready for mission-critical deployments.

What you can do

• Avoid using ParticleDB versions prior to 1.1.18.11.rc-7 in production due to known deadlock and crash issues.
• Verify transaction isolation behavior empirically rather than relying on documentation; test for race conditions under concurrent load.
• Do not assume full Postgres compatibility; audit your SQL queries for unsupported features like aggregates in CASE statements or exponentiation.
• If using ParticleDB, prefer the table-2pl or serializable modes for critical data, despite potential performance costs, to ensure stronger consistency.
• Monitor for silent data corruption by implementing application-level checksums or comparison tests against a trusted database like PostgreSQL.
• Request access to internal operator guides or detailed safety specifications from the vendor before committing to the platform.