Running a full DuckDB database directly on S3 without extra services

Engineers have developed a method to run a complete DuckDB database directly on object storage like Amazon S3, eliminating the need for separate database servers or complex data lake catalogs. Published in October 2026, this approach uses a custom extension called duckdb_objfs to map DuckDB’s internal storage engine to SlateDB, an embedded key-value store designed for cloud objects.

What happened

The core innovation is a DuckDB extension that allows the database to treat an S3 bucket as its primary disk. Unlike existing solutions such as Iceberg or DuckLake, which focus on sharing Parquet tables across different engines, this method preserves DuckDB’s entire feature set. This includes advanced indexes, constraints, views, macros, and specific data types that are often lost or mapped poorly when converting to open table formats.

The system enables a single writer process to update the database while multiple other processes attach to the same data in read-only mode. These readers can execute complex SQL queries, including joins and aggregations, directly against the data in the bucket. The architecture ensures that readers always see a consistent state of the database, even if they are slightly behind the latest write, preventing partial or corrupted reads during updates.

How it works

The extension works by replacing the underlying disk layer rather than modifying DuckDB’s storage engine itself. It registers a custom file system handler for the duckdb_objfs:// scheme. When DuckDB attempts to read or write files, the request is forwarded through a C interface to a Rust core that manages SlateDB. SlateDB acts as an adapter, translating file operations into object storage calls using Apache OpenDAL, which supports various backends like S3, MinIO, and local directories.

To handle the mismatch between DuckDB’s in-place file updates and the immutable nature of object storage, the system aligns data chunks precisely. Each 256 KiB DuckDB block maps to a single 256 KiB chunk in SlateDB, starting after a 12 KiB header. This geometry ensures that writing a block requires only one put operation, and reading it requires one get operation, avoiding expensive read-modify-write cycles. Durability is maintained by batching dirty chunks and metadata into a single write batch that is flushed to the bucket before the file sync call returns.

Key details

• Full SQL support: The solution retains all DuckDB features, including ART indexes, spatial R-trees, and experimental HNSW vector indexes, which are not supported in standard Parquet-based table formats.
• Single writer model: Only one process can write to the database at a time. Writer fencing prevents conflicts by ensuring older writers receive I/O errors if a newer writer takes over.
• Consistent reads: Read-only attachments use non-fencing readers that update their view of the manifest every ten seconds, guaranteeing consistency without blocking the writer.
• Performance metrics: On an Apple M4, cold S3 reads are approximately 1.2 times slower than native HTTPFS, while local writes are about 0.19 times the speed of native local disk writes due to network latency.
• No extra services: The deployment requires only an S3 bucket. SlateDB handles coordination, compaction, and garbage collection within the writer process, removing the need for external catalog services.
• Integration with Duckherder: This storage layer serves as the foundation for Duckherder, a distributed execution engine that allows multiple workers to read partitioned data from the same bucket for parallel query processing.

Why it matters

For software engineers and data teams, this approach simplifies infrastructure by removing the operational burden of managing database servers, backups, and high-availability clusters. The bucket becomes the single source of truth, reducing the number of components that need provisioning, securing, and monitoring. This is particularly valuable for self-hosted applications or small teams that want the power of a relational database without the complexity of traditional database administration.

However, the trade-offs involve latency and write throughput. Because every commit requires a round trip to object storage, write-heavy workloads will be significantly slower than local disk operations. Engineers must design their applications to batch small writes and accept that read-only replicas may lag by up to ten seconds. This model is ideal for analytical workloads where data is appended or updated infrequently but queried frequently by multiple users or services.

What you can do

• Evaluate workload fit: Determine if your application can tolerate the latency of S3 writes and the ten-second read lag. This model suits analytical dashboards better than high-frequency transactional systems.
• Batch write operations: Group small updates into larger transactions to minimize the number of round trips to the object storage bucket, improving overall write throughput.
• Use read-only attachments: Scale out query capacity by attaching multiple read-only processes to the same database, allowing concurrent analysis without impacting the writer.
• Monitor IO stats: Use the provided duckdb_objfs_io_stats() and duckdb_objfs_cache_stats() functions to track request counts, latency, and cache performance for debugging and optimization.
• Plan for migration: Since the bucket contains SlateDB objects rather than a standard .duckdb file, use COPY FROM DATABASE to move data in and out, ensuring compatibility with other tools if needed.