Skip to main content

Documentation Index

Fetch the complete documentation index at: https://opendata.dev/docs/llms.txt

Use this file to discover all available pages before exploring further.

The OpenData family of databases share a common foundation to provide a unified experience and similar operational characteristics. Nearly all the databases share a similar read/write path. The write path accepts data and writes it both to a durable WAL as well as into in-memory data structures that are eventually flushed to object storage as SSTs in an LSM tree: Writes are assigned a canonical ordering before being acknowledged to ensure that data is written consistently. From there, small WAL (write-ahead log) files are created and flushed frequently to object storage to ensure durability within a configurable time delay. These files are used to recover the database in the event of a system failure. Simultaneously, data is written into an in-memory data structure we call a Delta. This is used to make new data immediately available for reads. Once the Delta is full, it is flushed to object storage as a SlateDB SST. While the Delta implementation is domain-specific (unique to each database), the management mechanism is common across all databases, ensuring that the ingest semantics are correct and well tested.
LSM Tree maintence tasks such as comapction and garbage collection are handled asynchronously and can run on a separate runtime (or even on a different machine). While each database can implement its own comapction strategy, we delegate the compaction execution to SlateDB.See Compaction in SlateDB.

​
Flexible Durability Guarantees

The common write coordination mechanism allows users to control the required durability before acknowledging the write. This allows users to trade off durability for write latency. This tradeoff is fundamental to the design of object-store native systems. At one end of the spectrum, writes can be acknowledged as soon as the data is buffered in the in-memory Delta. This offers the lowest latency but means that data written since the last WAL flush is at risk if the process crashes. At the other end, writes can wait until the WAL is flushed to object storage before being acknowledged, providing full durability at the cost of higher write latency.
Durability LevelAck ConditionLatencyData at Risk
BufferedData written to in-memory DeltaLowestWrites since last WAL flush
WAL-durableWAL flushed to object storageHigherNone (survives process crash)
Since WAL files are flushed frequently on a configurable interval, the buffered mode still provides durability within a bounded time window. Choosing the right level depends on the use case. A third option decouples durability from the writer entirely: clients can write into an object-storage queue, and the main writer drains that queue asynchronously. The write returns as soon as the batch is durable in object storage, so ingest keeps working across writer restarts or failovers. See Stateless Ingest for the Timeseries implementation.

​
Backpressure & Flow Control

When writes arrive faster than Deltas can be flushed to object storage, the system applies backpressure to prevent unbounded memory growth. Once the number of unflushed Deltas exceeds a configurable threshold, new writes are stalled until a flush completes and frees memory. This flow control mechanism ensures that the system degrades gracefully under load rather than running out of memory. The backpressure threshold is tuned alongside the Delta size and flush interval to balance write throughput against memory usage.