# Append records Source: https://opendata.dev/docs/api-reference/append-records /openapi/log.yaml post /api/v1/log/append Append one or more records to the log. Each record has a key (used for partitioning scans) and a value (arbitrary bytes, both base64-encoded in JSON). Set `awaitDurable` to `true` to block until the records are persisted to durable storage. When `false` (the default), the server acknowledges the write as soon as records are buffered in memory. # Count entries Source: https://opendata.dev/docs/api-reference/count-entries /openapi/log.yaml get /api/v1/log/count Count the number of log entries for a given key within an optional sequence range. # Delete vectors by ID Source: https://opendata.dev/docs/api-reference/delete-vectors-by-id /openapi/vector.yaml post /api/v1/vector/delete Delete one or more vectors from the collection by their user-provided string IDs. IDs that do not exist are silently ignored. Supports both binary protobuf (`application/protobuf`) and ProtoJSON (`application/protobuf+json`) request bodies. # Get a vector by ID Source: https://opendata.dev/docs/api-reference/get-a-vector-by-id /openapi/vector.yaml get /api/v1/vector/vectors/{id} Retrieve a single vector and its attributes by its user-provided ID. # Instant query Source: https://opendata.dev/docs/api-reference/instant-query /openapi/timeseries.yaml get /api/v1/query Evaluate a PromQL expression at a single point in time. The result can be a vector (set of instant samples), a scalar, a string, or even a matrix depending on the expression type. The `resultType` field in the response indicates the shape of the `result` array. If `time` is omitted the server uses the current server time. Also accepts POST with a `application/x-www-form-urlencoded` body containing the same parameters. # List keys Source: https://opendata.dev/docs/api-reference/list-keys /openapi/log.yaml get /api/v1/log/keys List distinct keys present in the log within an optional segment range. # List label names Source: https://opendata.dev/docs/api-reference/list-label-names /openapi/timeseries.yaml get /api/v1/labels Return the sorted list of all label names present in the database. Optionally filtered by series selectors and/or a time range. When `match[]` selectors are provided, only label names from series matching those selectors are returned. When `start` and/or `end` are provided, only series with samples in that time range are considered. Also accepts POST with a `application/x-www-form-urlencoded` body containing the same parameters. # List label values Source: https://opendata.dev/docs/api-reference/list-label-values /openapi/timeseries.yaml get /api/v1/label/{name}/values Return the sorted list of known values for a given label name. Optionally filtered by series selectors and/or a time range. For example, querying `/api/v1/label/job/values` returns all distinct values of the `job` label across all series. When `match[]` selectors are provided, only values from series matching those selectors are returned. When `start` and/or `end` are provided, only series with samples in that time range are considered. # List segments Source: https://opendata.dev/docs/api-reference/list-segments /openapi/log.yaml get /api/v1/log/segments List log segments whose start sequence falls within the given range. Segments are the physical storage units of the log. # List series Source: https://opendata.dev/docs/api-reference/list-series /openapi/timeseries.yaml get /api/v1/series Return the list of label sets (unique metric identities) that match a set of series selectors. This is useful for discovering which time series exist for a given metric name or label combination. At least one `match[]` argument must be provided. Each `match[]` value is a PromQL series selector (e.g. `up`, `http_requests_total{method="GET"}`). Multiple selectors are OR'd together — a series is returned if it matches any of the provided selectors. Also accepts POST with a `application/x-www-form-urlencoded` body containing the same parameters. # Range query Source: https://opendata.dev/docs/api-reference/range-query /openapi/timeseries.yaml get /api/v1/query_range Evaluate a PromQL expression over a range of time, returning a matrix of time series with regularly spaced samples. The expression is evaluated at each `step` interval between `start` and `end` (both inclusive). The result is always a matrix (`resultType: "matrix"`), where each series contains an array of `[timestamp, value]` pairs spaced by `step`. Also accepts POST with a `application/x-www-form-urlencoded` body containing the same parameters. # Remote write Source: https://opendata.dev/docs/api-reference/remote-write /openapi/timeseries.yaml post /api/v1/write Ingest time series data using the [Prometheus Remote Write 1.0](https://prometheus.io/docs/specs/prw/remote_write_spec/) protocol. The request body must be a **Snappy-compressed** protobuf `WriteRequest` message (as defined by the Prometheus remote write specification). Each `WriteRequest` contains one or more `TimeSeries` messages, each with a set of labels and one or more timestamp/value sample pairs. This endpoint is designed for integration with Prometheus remote-write compatible agents and collectors. It is not intended as a general-purpose high-throughput ingestion path — for bulk data loading, prefer dedicated import tooling. This endpoint is only available when the server is built with the `remote-write` feature. # Scan entries Source: https://opendata.dev/docs/api-reference/scan-entries /openapi/log.yaml get /api/v1/log/scan Scan log entries for a given key within an optional sequence range. Supports **long-polling**: set `follow=true` to hold the connection open until new data arrives or `timeout_ms` elapses (default 30 000 ms). This is useful for tailing a log stream in near-real-time. # Search for nearest neighbors Source: https://opendata.dev/docs/api-reference/search-for-nearest-neighbors /openapi/vector.yaml post /api/v1/vector/search Return the top `k` records for a query. Provide exactly one of `vector` (approximate nearest neighbor search) or `bm25` (full-text BM25 search over a text field); the two are mutually exclusive. Optionally filter results by metadata attributes using a filter expression. The `nprobe` parameter controls how many posting lists are searched for ANN queries. Higher values improve recall at the cost of latency. The `includeFields` parameter controls which attributes are returned for each result. When omitted, all attributes are returned. Supports both binary protobuf (`application/protobuf`) and ProtoJSON (`application/protobuf+json`) request bodies. # Upsert vectors Source: https://opendata.dev/docs/api-reference/upsert-vectors /openapi/vector.yaml post /api/v1/vector/write Upsert one or more vectors into the collection. Each vector has a user-provided string ID (up to 64 bytes), a dense embedding in the `vector` attribute, and optional metadata attributes. If a vector with the same ID already exists, it is replaced. Supports both binary protobuf (`application/protobuf`) and ProtoJSON (`application/protobuf+json`) request bodies. # Buffer Architecture Source: https://opendata.dev/docs/buffer/architecture How Buffer works Buffer is a library with two main components: **multiple producers** and **a single consumer**. Producers and the consumer communicate exclusively through a queue manifest in object storage — there is no direct network path between them and no stateful broker to operate. ## Producers Producers accept arbitrary byte entries from callers, buffer them in memory, and periodically flush them as binary data batches to object storage. Each producer exposes a minimal API: ```rust theme={null} pub async fn produce( &self, entries: Vec, metadata: Bytes, ) -> Result ``` Flushes are triggered either by elapsed time (default: 100 ms) or by batch size (default: 64 MiB). Data batches are binary and optionally compressed. Each batch is named with a [ULID](https://github.com/ulid/spec), which encodes a millisecond-precision timestamp. Once a batch is flushed, producers append its location and metadata to the queue. The queue is a single binary manifest in object storage that tracks the locations and metadata of data batches in append order. Producers interact with it through a queue producer, which uses compare-and-swap (CAS) writes to ensure that concurrent producers do not overwrite each other's entries: an append succeeds only if the manifest has not been modified since it was read. On conflict, the queue producer re-reads the manifest and retries. Additionally, the queue producer assigns consecutive sequence numbers to queue entries. The manifest format is designed for append efficiency. Existing entries are never deserialized during append, so appending a new entry is O(1) in the length of the queue. The `WriteHandle` returned by `produce()` contains a `DurabilityWatcher`. Callers can `await_durable()` to block until the entries have been persisted to object storage and enqueued in the manifest. This avoids data loss during failover, but it does not provide read-your-own-writes consistency — durability refers to the data batch, not to the database write. ## Consumer The consumer is the read-side counterpart to the producers. It iterates over data batches in object storage and delivers them to the database in the order recorded in the queue: ```rust theme={null} pub async fn next_batch(&mut self) -> Result> pub async fn ack(&mut self, sequence: u64) -> Result<()> ``` A call to `next_batch()` reads the next location in the queue manifest, fetches the data batch from object storage, and returns the data, metadata, and a unique sequence number to the caller. Sequence numbers are assigned by the queue producer at ingestion time in increasing order without gaps, and they are used both for acknowledgement and for tracking progress. Acknowledgements must be in order: acknowledging a sequence number that is not immediately after the last one is rejected, ensuring that no batch is silently skipped. While multiple producers can write to a queue manifest, only one consumer can read from it. This matches OpenData's single-writer paradigm and prevents zombie consumers — surviving after an infrastructure failure — from acknowledging batches and making them invisible to the active consumer. Single-consumer exclusivity is enforced by an epoch stored in the queue manifest: when a consumer starts, it reads the epoch, increments it, and writes it back. Reads with a stale epoch are rejected, so any zombie consumer is fenced as soon as a new one takes over. The consumer is also responsible for cleaning up processed data batches. At startup and every 100 acknowledged batches, all entries for acknowledged batches are dequeued from the manifest. A dedicated garbage collector task then deletes the corresponding data batches from object storage. The consumer API supports exactly-once delivery. To achieve it, the writer atomically persists the batch data and the sequence number from `next_batch()` to the database. If the atomic write succeeds but a failure occurs before `ack()` returns, a new writer can construct a new consumer with that last persisted sequence number so it resumes right after it. The new consumer also bumps the epoch, fencing the old one and preventing any duplicate writes. ## Read-ahead and batched acks The serial `next_batch` / `ack` API caps a consumer at one manifest read and one ack write per batch. Fetching a batch is an I/O wait; decoding it is CPU work; and writing it to another system is yet more I/O work. Doing everything on one task in a tight loop wastes both the object-store bandwidth and the machine's cores. For high-throughput consumers, `Consumer` exposes a parallel-fetch path: ```rust theme={null} pub async fn next_descriptors(&mut self, max: usize) -> Result>; pub async fn fetch_descriptor(&mut self, d: BatchDescriptor) -> Result; pub fn fetch_handle(&self) -> ConsumerFetchHandle; pub async fn ack_through(&mut self, sequence: u64) -> Result<()>; ``` The design batches the expensive low-volume operations (manifest read, ack) and parallelizes the high-volume operations (fetch, decode). The serial `next_batch` / `ack` API still works without changes: `next_batch` is a thin wrapper over `next_descriptors(1)` plus `fetch_descriptor`, and `ack` keeps its amortize-every-100-calls behavior. ### How the split works `next_descriptors(max)` reads the manifest once and returns up to `max` contiguous descriptors past an internal read-ahead cursor. It does no object-store fetch and does not mutate the durable ack frontier. `ConsumerFetchHandle` is a cheap, cloneable handle obtained from `fetch_handle()`. It holds an `Arc` to the object store and the manifest path. Its single method, `fetch(&self, descriptor) -> Result`, fetches and decodes one batch, touches no `Consumer` cursor, and is safe to call concurrently from many tasks. Cloning the handle is O(1). `ack_through(sequence)` advances the durable ack frontier through `sequence` in one `dequeue` write, no matter how far the frontier moves. The dequeue runs first; in-memory state and the `buffer.acks` counter advance only on success, so a fence or storage error leaves the consumer's state untouched and the call is safe to retry. ### Pipelined consumer pattern One task owns the `Consumer` and the manifest; N fetch workers each hold a clone of the handle and pull descriptors off a bounded channel: ```rust theme={null} let consumer = Consumer::new(config, last_acked).await?; let fetcher = consumer.fetch_handle(); // N fetch workers, each holding a clone of the handle: for _ in 0..fetch_concurrency { let f = fetcher.clone(); let rx = descriptor_rx.clone(); tokio::spawn(async move { while let Some(d) = rx.recv().await { let batch = f.fetch(d).await?; // ...hand off to a decode / sink stage... } Ok::<(), Error>(()) }); } // Owner task: pull a run of descriptors, fan them out, ack the // contiguous high watermark. loop { let descriptors = consumer.next_descriptors(K).await?; for d in descriptors { descriptor_tx.send(d).await?; } // ...wait for the contiguous-completed high watermark, then: consumer.ack_through(high).await?; } ``` Owner and workers run in parallel because their state is disjoint: two concurrent fetches against distinct descriptors do not contend on the manifest. ### Descriptor handout contract `next_descriptors` advances an internal `last_handed_out_sequence` *before* the caller has fetched the objects. Read-ahead is the point. That places one obligation on the caller: `ack_through(n)` does not verify that anything below `n` was processed. It unconditionally advances the durable frontier to `n`, rejecting only `n <= last_acked_sequence`. The caller must track which sequences have actually been processed and only call `ack_through` with the highest fully-processed *contiguous* sequence. Fetches may complete out of order; ack the contiguous-complete watermark, never past a still-pending sequence. If a descriptor is permanently lost (a worker panics, a channel send is dropped), the frontier stalls at that sequence. Recovery is process restart: `Consumer::new(config, last_acked_sequence)` re-emits everything the durable frontier has not acked. ### Fencing `next_descriptors` returns `Error::Fenced` on the next manifest read after fencing. A worker holding a stale handle keeps fetching successfully against object storage; the manifest owner learns of the fence and propagates it, typically by closing the descriptor channel. `ack_through` against a fenced manifest returns `Error::Fenced` and leaves state untouched, so a fenced consumer cannot advance the durable frontier. ### When to use which API The serial `next_batch` / `ack` API stays the right choice for low-throughput single-task consumers (the timeseries and vector buffer consumers use it) and for backfill tooling where a tight per-batch ack loop is simpler than the pipeline. Reach for the read-ahead path when consumer throughput is bounded by manifest round-trips (one `next_batch` cycle per batch caps throughput at `1 / manifest_RTT`), when fetch I/O and decode CPU need to overlap across batches, or when the workload is bursty enough that an in-flight bytes ceiling fits better than per-batch flow control. The opendata-contrib [generic ingest runtime](https://github.com/opendata-oss/opendata-contrib/blob/main/rfcs/0002-generic-ingest-runtime.md) is one pipelined consumer built on these primitives. Full design rationale, the descriptor handout contract in detail, and the failure-mode catalog live in [RFC 0003: Consumer Read-Ahead and `ack_through`](https://github.com/opendata-oss/opendata/blob/main/buffer/rfcs/0003-consumer-read-ahead.md). ## Usage example For an example of how Buffer can be integrated into an ingestion pipeline see [ingest into Timeseries](/docs/timeseries/ingest). # Buffer Benchmarks Source: https://opendata.dev/docs/buffer/benchmarks End-to-end latency, throughput, and cost for Buffer pipelines on object storage This page contains two benchmarks. The first measures throughput and latency of multiple OTel Collectors writing metric data to OpenData Timeseries. The second measures the throughput and latency of a single Otel Collector writing log data to Clickhouse. ## Writing Metrics to OpenData Timeseries We ran an 8-hour experiment running three Buffer producers, each producing 10 MB/s (30 MB/s total), with flush and poll intervals of 100 ms. Latency is measured from the producer receiving a data entry to the Buffer consumer reading it. | Metric | Result | | ---------------------- | ------------------------------- | | Throughput | 30 MB/s (3 producers × 10 MB/s) | | p50 end-to-end latency | \< 0.5 s | | p99 end-to-end latency | \~2 s | | Flush / poll interval | 100 ms | Latency and throughput holding steady across the 8-hour Buffer benchmark End-to-end latency is tunable: raising the flush and poll intervals lowers cost at the expense of latency, and lowering them reduces latency (to a floor of \~50–100 ms) at increased object-storage request cost. ### Cost S3 costs over the experiment, projected to a monthly bill: | Request type | Number | Rate | Monthly cost | | ------------ | ------------------------- | ---------------- | ------------- | | PUT | 16,241,670 | \$0.005 / 1,000 | \$81.21 | | GET | 23,908,140 | \$0.0004 / 1,000 | \$9.56 | | Storage | 27.54 GB (max during run) | \$0.023 / GB | \$0.63 | | **Total** | | | **\~\$91.40** | ## OTel → ClickHouse pipeline This benchmark used the pipelined producer and consumer runtimes to measure the maximum throughput of writing Logs from an Otel Collector to Clickhouse. A single loadgen drove 175,000 log records/sec into one OTel gateway pod over OTLP/gRPC. The gateway ran the [OpenData producer](https://github.com/opendata-oss/opendata-go#producer-architecture), which appended OTel payloads to Buffer on S3. A separate node running the [ClickHouse ingestor](https://github.com/opendata-oss/opendata-contrib/tree/main/connectors/clickhouse-ingestor) consumed the payloads and inserted them into ClickHouse. End-to-end latency was measured from two timestamps stamped onto each record (`_odb_gateway_received_at` at the gateway, `_odb_clickhouse_inserted_at` at INSERT time). Latency was the difference between these two values, computed via a query on Clickhouse. Both components, the OTel exporter and the ClickHouse ingestor, are documented on the [Integrations](/docs/buffer/integrations) page. Throughput holding at 1.1 Gbps with stable end-to-end latency over the run ### Cost | S3 cost component | Measured rate | List price (us-east-1) | Monthly | | ---------------------------------------------------------- | ------------- | ---------------------- | ----------- | | PUT — batch uploads (7.1/s) + manifest CAS commits (5.3/s) | \~12.5/s | \$0.005 / 1k | \~\$162 | | GET — consumer fetches + producer pre-CAS reads + polls | \~16/s | \$0.0004 / 1k | \~\$17 | | Storage — batches GC'd promptly after ack | tens of GB | \$0.023 / GB-mo | \< \$5 | | Network — in-region via S3 Gateway endpoint | — | \$0 | \$0 | | **Total** | | | **\~\$180** | ## Reproduce it * Setup for both components: [Buffer integrations](/docs/buffer/integrations) * Producer client and OTel exporter: [opendata-go](https://github.com/opendata-oss/opendata-go#producer-architecture) * ClickHouse ingestor: [opendata-contrib](https://github.com/opendata-oss/opendata-contrib/tree/main/connectors/clickhouse-ingestor) * 5-minute local OTel → ClickHouse pipeline: [tutorial](https://github.com/opendata-oss/opendata-contrib/tree/main/tutorial) * Design: [stateless buffer RFC](https://github.com/opendata-oss/opendata/blob/main/buffer/rfcs/0001-stateless-buffer.md) and the [pipelined consumer RFC](https://github.com/opendata-oss/opendata-contrib/blob/main/rfcs/0002-generic-ingest-runtime.md) # When to choose Buffer Source: https://opendata.dev/docs/buffer/comparisons How OpenData Buffer compares to Kafka, WarpStream, Kinesis, SQS, and DIY object-storage staging *We build OpenData and this is our best good-faith comparison. Corrections are [welcome](https://github.com/opendata-oss/opendata/issues).* The most common Kafka use case, shipping telemetry into a database or warehouse, doesn't need most of what a broker provides. Transactions and consumer-group coordination are the hard problems brokers exist to solve, and a pipeline with a fixed set of producers and one consumer needs neither. Buffer removes the broker entirely: producers flush batches straight to your object store bucket, and a fenced consumer reads them in order. ## Choose Buffer if you care about… **…moving data without operating a broker.** The data path is your bucket: the whole system is a library plus a manifest file on object storage. There is no service to provision or operate. **…data transport at S3 prices.** [Our benchmark](/docs/buffer/benchmarks) moved 30 MiB/s for \~\$91/month of object-storage charges. Public calculators price the same throughput at roughly \$1,300/month on [managed WarpStream](https://www.warpstream.com/pricing) and \$8,500/month [self-hosting Kafka](https://2minutestreaming.com/tools/apache-kafka-calculator/). **…highly available pipelines.** Producers are stateless: if one dies, another can take over, with no rebalancing protocol. If the downstream database is slow or down, batches accumulate safely in object storage instead of back-pressuring your apps or getting dropped. And zonal producers mean ingest never crosses AZs, resulting in zero cross-AZ transfer fees. **…deduplicating data on write.** Buffer maintains unique identities for each batch written to a sink that are stable across retries. This makes deduplication on write possible if you need those semantics. **…gigabit-scale throughput.** Pipelined producers and consumers keep many object-store requests in flight at once. We demonstrated [\~1 Gbps of log ingestion into ClickHouse](https://www.opendata.dev/blog/ingesting-1gbps-logs-to-clickhouse) with two nodes, each with 4 vCPUs and 8 GB of RAM. **…observability pipelines specifically.** An included OpenTelemetry exporter ships MELT data from any OTel Collector through Buffer. A bundled ClickHouse sink can read from Buffer and write to ClickHouse. OpenData Log and Vector can ingest from Buffer natively. ## How the alternatives stack up ✓ = yes · \~ = partially · ✗ = no | You care about… | **Buffer** | Kafka (self-hosted / MSK) | WarpStream / AutoMQ / Bufstream | Kinesis | SQS | DIY S3 staging | | --------------------------------------- | ---------- | ------------------------- | ------------------------------- | ----------- | ----------- | -------------- | | No broker or service in the data path | ✓ | ✗ | \~ ¹ | ✗ | ✗ | ✓ | | Cost ≈ S3 requests | ✓ ² | ✗ | \~ ² | ✗ | ✗ | ✓ | | Stateless producers, zero cross-AZ fees | ✓ | ✗ | ✓ | \~ | \~ | ✓ | | Deduplication on write | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ ³ | | Multi-Gbps pipeline throughput | ✓ | ✓ | ✓ | \~ ⁴ | ✗ ⁴ | \~ ³ | | License | MIT | Apache 2.0 | Proprietary / BSL | Proprietary | Proprietary | n/a | ¹ The Kafka-on-S3 systems remove the broker disks but keep the broker tier: you still run [agents](https://docs.warpstream.com/warpstream/overview/architecture), and WarpStream's depend on a proprietary hosted control plane. \ ² At 30 MiB/s: \~\$91/mo of S3 charges on Buffer vs. \~\$1,300/mo ([WarpStream's calculator](https://www.warpstream.com/pricing)) and \~\$8,500/mo ([2minutestreaming's Kafka calculator](https://2minutestreaming.com/tools/apache-kafka-calculator/)). \ ³ The closest competitor: producers writing files to a prefix and a consumer listing them. Same architecture and economics, but you'd have to solve the coordination between producer and consumer, write pipelined clients, etc.\ ⁴ [Kinesis](https://aws.amazon.com/kinesis/data-streams/pricing/) meters and throttles per shard, so Gbps means big shard fleets and bills; SQS is per-message with size limits. It's built for task queues rather than bulk ordered transport. The systems with Buffer's reliability semantics all put a broker tier in the path; the only alternative at similar cost is building the coordination yourself. If your pipeline has fixed producers and a fixed consumer, which describes many telemetry ingestion paths, the broker isn't doing necessary work. ## The tradeoffs * **Latency is sub-second to seconds:** p50 under 0.5s / p99 \~2s at 30 MiB/s, tunable toward a \~50–100ms floor at higher request cost. Millisecond delivery needs a broker. * **Participants are finite:** throughput contends on one compare-and-set manifest, so run one producer per zone and a small fixed consumer set. * **Rust only:** Buffer is a crate. The OTel exporter covers the most common producer without writing Rust, but there are no Java/Python/Go clients yet. This is the biggest practical limitation. * **At-least-once, per-producer ordering:** global total order and transactional produce are non-goals. Exactly-once is an effect of deterministic replay plus an idempotent sink. *** *Last reviewed July 2026. Please [tell us](https://github.com/opendata-oss/opendata/issues) if a claim has gone stale.* # Buffer Configuration Source: https://opendata.dev/docs/buffer/configuration Configure object storage, batching, and garbage collection for Buffer Buffer is a Rust library. Configuration is expressed as plain Rust structs that the host service constructs and hands to the `Producer` and `Consumer` APIs. Two top-level structs govern the write and read sides: `ProducerConfig` and `ConsumerConfig`. Both share the same `ObjectStoreConfig` from the `common` crate. The structs derive `Serialize`/`Deserialize` so host services *may* embed them in their own config formats, but the canonical surface documented here is the Rust API. ## `ProducerConfig` Controls where data batches and the queue manifest are written, how often batches are flushed, and when backpressure is applied. ```rust theme={null} pub struct ProducerConfig { /// Object store provider. Determines where data batches and the /// queue manifest are persisted. pub object_store: ObjectStoreConfig, /// Path prefix for data batch objects in object storage. /// Default: "ingest" pub data_path_prefix: String, /// Path to the queue manifest in object storage. /// Default: "ingest/manifest" pub manifest_path: String, /// Flush the in-memory batch when this much time has elapsed since /// the last flush. /// Default: 100 ms pub flush_interval: Duration, /// Flush the in-memory batch when its total size in bytes (entries /// plus metadata) exceeds this threshold. /// Default: 64 MiB (67_108_864) pub flush_size_bytes: usize, /// Maximum number of input entry vectors that may be buffered for /// the background batch writer before backpressure is applied to /// callers. /// Default: 1000 pub max_buffered_inputs: usize, /// Compression algorithm applied to the record block of each data /// batch. /// Default: CompressionType::None pub batch_compression: CompressionType, } ``` ## `ConsumerConfig` Controls where the queue manifest and data batches are read from, and how stale batches are garbage collected. ```rust theme={null} pub struct ConsumerConfig { /// Object store provider. Must point at the same bucket or directory /// as the paired producer. pub object_store: ObjectStoreConfig, /// Path to the queue manifest in object storage. Must match the /// producer's `manifest_path`. /// Default: "ingest/manifest" pub manifest_path: String, /// Path prefix for data batch objects. Must match the producer's /// `data_path_prefix`. /// Default: "ingest" pub data_path_prefix: String, /// How often the garbage collector runs. /// Default: 5 minutes pub gc_interval: Duration, /// Minimum age of an unreferenced batch file before it is eligible /// for deletion by the garbage collector. /// Default: 10 minutes pub gc_grace_period: Duration, } ``` The grace period prevents the consumer from deleting batches that a producer has just written but not yet enqueued in the manifest. ## `ObjectStoreConfig` A tagged enum re-exported from the `common` crate. Three variants are available: ```rust theme={null} ObjectStoreConfig::Local(LocalObjectStoreConfig { path: "./data".to_string(), }) ``` ```rust theme={null} ObjectStoreConfig::Aws(AwsObjectStoreConfig { region: "us-west-2".to_string(), bucket: "my-ingest-bucket".to_string(), }) ``` ```rust theme={null} ObjectStoreConfig::InMemory ``` ## `CompressionType` ```rust theme={null} pub enum CompressionType { /// No compression (default). None, /// Zstandard compression of the record block. Zstd, } ``` ## Pairing producers and consumers A producer and the consumer that drains it must agree on three values: * **`object_store`** must point at the same bucket or directory. * **`manifest_path`** must be identical, since it identifies the queue. * **`data_path_prefix`** must be identical, since the manifest stores batch locations relative to it. A mismatch on any of these silently routes traffic to a different queue or causes the consumer to fail to fetch batches it sees in the manifest. ## Examples ```rust theme={null} use std::time::Duration; use common::{ObjectStoreConfig, LocalObjectStoreConfig}; use buffer::{ProducerConfig, ConsumerConfig, CompressionType}; let object_store = ObjectStoreConfig::Local(LocalObjectStoreConfig { path: "./data".to_string(), }); let producer_config = ProducerConfig { object_store: object_store.clone(), data_path_prefix: "ingest".to_string(), manifest_path: "ingest/manifest".to_string(), flush_interval: Duration::from_millis(100), flush_size_bytes: 64 * 1024 * 1024, max_buffered_inputs: 1000, batch_compression: CompressionType::None, }; let consumer_config = ConsumerConfig { object_store, manifest_path: "ingest/manifest".to_string(), data_path_prefix: "ingest".to_string(), gc_interval: Duration::from_secs(5 * 60), gc_grace_period: Duration::from_secs(10 * 60), }; ``` ```rust theme={null} use std::time::Duration; use common::{ObjectStoreConfig, AwsObjectStoreConfig}; use buffer::{ProducerConfig, ConsumerConfig, CompressionType}; let object_store = ObjectStoreConfig::Aws(AwsObjectStoreConfig { region: "us-west-2".to_string(), bucket: "my-ingest-bucket".to_string(), }); let producer_config = ProducerConfig { object_store: object_store.clone(), data_path_prefix: "ingest".to_string(), manifest_path: "ingest/manifest".to_string(), flush_interval: Duration::from_millis(200), flush_size_bytes: 128 * 1024 * 1024, max_buffered_inputs: 1000, batch_compression: CompressionType::Zstd, }; let consumer_config = ConsumerConfig { object_store, manifest_path: "ingest/manifest".to_string(), data_path_prefix: "ingest".to_string(), gc_interval: Duration::from_secs(5 * 60), gc_grace_period: Duration::from_secs(10 * 60), }; ``` ```rust theme={null} use std::time::Duration; use common::ObjectStoreConfig; use buffer::{ProducerConfig, ConsumerConfig, CompressionType}; let producer_config = ProducerConfig { object_store: ObjectStoreConfig::InMemory, data_path_prefix: "ingest".to_string(), manifest_path: "ingest/manifest".to_string(), flush_interval: Duration::from_millis(100), flush_size_bytes: 64 * 1024 * 1024, max_buffered_inputs: 1000, batch_compression: CompressionType::None, }; let consumer_config = ConsumerConfig { object_store: ObjectStoreConfig::InMemory, manifest_path: "ingest/manifest".to_string(), data_path_prefix: "ingest".to_string(), gc_interval: Duration::from_secs(5 * 60), gc_grace_period: Duration::from_secs(10 * 60), }; ``` # Buffer Source: https://opendata.dev/docs/buffer/index An MIT-licensed, stateless buffer that turns object storage into a low-latency queue, replacing Kafka for many high throughput pipelines. Buffer is an MIT-licensed, stateless library that uses object storage as a durable queue between producers and consumers. Producers batch opaque byte entries and flush them to object storage as append-only data files. Consumers pull batches directly from object storage. A single manifest file on object storage coordinates producers and consumers through atomic compare-and-set operations. The producer and consumer frameworks are extensible and can accommodate a variety of data types and source and destination systems. And because both the data and the manifest live in object storage, Buffer is stateless and horizontally scalable, and it leaves durability and availability to S3. ## What it's good at Buffer was built for high throughput data pipelines which don't need strict ordering and where sink-side deduplication is acceptable. This describes a vast majority of observability and event pipelines, like those shipping metric, event, log, and trace data to observability and analytics stores. ## Why Buffer Buffer's only goal is to make object storage usable as a high volume and moderately low-latency queue. That focus is the reason it costs a small fraction of a managed or self-hosted Kafka cluster for the same throughput. The ingestors are stateless, so you scale and fail over by adding or replacing pods rather than rebalancing a stateful cluster. The manifest's compare-and-set gives you at-least-once delivery and fences zombie clients. The pipelined producer and consumer clients keep many object-storage requests in flight at once to maximize throughput. Each data batch carries a stable identity across retries so sinks can deduplicate correctly. See the [benchmarks](/docs/buffer/benchmarks) for end-to-end latency, throughput, and a cost comparison against Kafka. ## Tradeoffs * **Finite participants.** Throughput scales inversely with the number of participants, since metadata updates contend on a single object-storage compare-and-set. Run one writer per zone and a fixed, limited number of consumers. * **No partition multiplexing.** Partitioning is a user-space concern. If vertical scaling isn't enough, deploy multiple buffers and let Kubernetes handle scheduling and liveness. * **Higher latency than in-memory Kafka.** Batching to object storage and polling the manifest is a two-hop design. In practice that means a p99 of a couple of seconds, tunable down to a low floor by flushing and polling more often at the cost of more object-storage requests. ## Explore Buffer How Buffer works by leveraging object storage How to configure producers and consumers The OTel exporter and the ClickHouse ingest runtime End-to-end latency, throughput, and cost versus Kafka How Buffer compares to Kafka, WarpStream, Kinesis, SQS, and DIY staging View the source code, open issues, and contribute # Buffer Integrations Source: https://opendata.dev/docs/buffer/integrations First-party producer and consumer integrations for Buffer. The core Buffer library just provides APIs for reading and writing from S3. A working Buffer pipeline needs to build on these APIs to be useful. OpenData ships the following first-party integrations for both the production and consumption sides: * **Producer side:** A pipelined producer runtime with the [OpenData OTel exporter](#producer-side-the-opendata-otel-exporter) in [`opendata-go`](https://github.com/opendata-oss/opendata-go). This allows your OTel Collector to emit high volume logs, metrics, and traces to S3 via Buffer. * **Consumer side:** * An extensible [pipelined consumer runtime](#consumer-side-the-generic-ingest-runtime) in [`opendata-contrib`](https://github.com/opendata-oss/opendata-contrib) along with a high volume [ClickHouse sink](#clickhouse-sink-clickhouse-ingestor) built on it. * OpenData Timeseries has a native [Buffer Consumer](/docs/timeseries/ingest) that can ingests metrics from Buffer. Both are MIT-licensed. ## Producer Integrations ### OTel Collector Exporter The exporter is a normal OpenTelemetry Collector exporter. It serializes each OTLP request as protobuf, hands it to the [pipelined Buffer producer](https://github.com/opendata-oss/opendata-go#producer-architecture), and returns success to the collector pipeline only after the batch is durable (the data object is `PUT` and its metadata lands in the manifest via a CAS operation). You configure it like any other collector exporter, pointing it at an object store bucket and the manifest and data paths for the data type in question: ```yaml theme={null} exporters: opendata/logs: object_store: type: s3 bucket: my-otel-logs-bucket region: us-west-2 data_path_prefix: ingest/otel/logs/data manifest_path: ingest/otel/logs/manifest service: pipelines: logs: exporters: [opendata/logs] ``` The flush thresholds, compression, and pipelining knobs all have defaults. Checkout the [exporter README](https://github.com/opendata-oss/opendata-go/tree/main/exporter/opendataexporter) for the full config reference. Stock `otelcol-contrib` does not bundle the exporter. Use the pre-built collector image at `ghcr.io/opendata-oss/otel-collector`, or build a custom collector distribution with the OpenTelemetry Collector Builder. See the [otel-collector image docs](https://github.com/opendata-oss/opendata-go/tree/main/otel-collector). ## Consumer Integrations ### The generic consumer runtime The [consumer runtime](https://github.com/opendata-oss/opendata-contrib/tree/main/runtime/opendata-ingest-runtime) provides the framework to read from a Buffer queue and write to a configured sink. It provides the following machinery that can be reused to write to any sink. * **Pipelined stages.** Fetch, decode, and sink commit run as separate stages with bounded queues between them using Buffer's [read-ahead consumer API](/docs/buffer/architecture#read-ahead-and-batched-acks). * **Byte-budgeted backpressure.** In-flight bytes are bounded per source. * **Contiguous-completion ack.** The runtime ensures that only data batches that have been committed to the sink are dequeued and removed from object storage. It is designed to handle out of order acknowledgments. A concrete sink needs to implements the `Sink` trait along with the appropriate decoders. For more details, please read the design in [RFC 0002](https://github.com/opendata-oss/opendata-contrib/blob/main/rfcs/0002-generic-ingest-runtime.md). The `Sink` trait API is documented in the [runtime README](https://github.com/opendata-oss/opendata-contrib/tree/main/runtime/opendata-ingest-runtime). ### ClickHouse sink [`clickhouse-ingestor`](https://github.com/opendata-oss/opendata-contrib/tree/main/connectors/clickhouse-ingestor) is the first packaged connector. It wires the ingest runtime to the OTLP-logs decoder and a ClickHouse `Sink`, consuming OTLP logs from a Buffer queue and inserting them into a `ReplacingMergeTree` table. Each source range carries a deterministic idempotency token, so retries and replays collapse to effective exactly-once on merge. It ships as a pre-built binary and a container image. * The [local tutorial](https://github.com/opendata-oss/opendata-contrib/tree/main/tutorial) runs the whole OTel-to-ClickHouse pipeline on your laptop with `docker compose up` and a synthetic load generator. * The [clickhouse-ingestor README](https://github.com/opendata-oss/opendata-contrib/tree/main/connectors/clickhouse-ingestor) is the production operator guide: table DDL, the full config reference, deployment, dry-run validation, metrics, and troubleshooting. ## End-to-End Example A complete brokerless pipeline for ingesting OTel Data to ClickHouse using Buffer looks like this: There is no broker on the path. The collector and the ingestor coordinate only through one manifest object in object storage, with no direct network connection between them. To run the whole thing on your laptop in about two minutes, follow the [local tutorial](https://github.com/opendata-oss/opendata-contrib/tree/main/tutorial) (`docker compose up`, then a synthetic OTLP load generator). ## Build your own integration Both the producer and consumer runtimes are modular and are designed to be extended. * For A new sink (Iceberg, Postgres, a file format), implement the `Sink` trait from the ingest runtime in a fresh crate. * For a new OTel singnal into ClickHouse (OTLP traces, your own protobuf), implement a `Decoder` and an `Adapter` against the existing ClickHouse pipeline. Start from [RFC 0002](https://github.com/opendata-oss/opendata-contrib/blob/main/rfcs/0002-generic-ingest-runtime.md) and the [runtime README](https://github.com/opendata-oss/opendata-contrib/tree/main/runtime/opendata-ingest-runtime). # Introduction Source: https://opendata.dev/docs/index OpenData is a collection of operationaly simple, open source databases that share a unified storage foundation on object storage. Database software is often available for free as open source, but the real cost comes from the operational expertise required to keep them running. Operators must understand and configure behavior for replication, failover, backup, capacity planning, and disaster recovery. These concerns multiply with every additional database system since each ship with their own implementations. Is there a better way? We believe that the general avaialbility of Object Storage is the escape hatch. Services like S3 provide durability, replication, and availability *at commodity prices* with no operational overhead. When you push replication and durability down to the storage layer, the database itself becomes dramatically simpler. Entire categories of complexity disappear: * **Durability & Availability**: opendata systems can be deployed in production with a single replica and still maintain durability and high write availability. * **Failover & Scaling**: worker nodes are stateless and only use local disks for caching, making failovers and horizontal scaling trivial. * **Backups & Branches**: all data is stored immutably, so backups and branches are as easy as marking data immune to garbage collection. * **Cost to Serve**: object storage is not only less expensive than replicated disks, it also provides free cross-zone data transfer for ingestion and replication. By implementing these patterns once in [SlateDB](https://slatedb.io), we can share them across all databases in the OpenData family and provide a unified operational experience. ## Databases Prometheus-compatible metrics database Key-oriented event streaming SPANN-style approximate nearest neighbor search Highly-available ingestion buffer # Log Benchmarks Source: https://opendata.dev/docs/log/benchmarks Ingest throughput, poll latency, and cost for a single-node Log deployment ## Throughput A single `m5n.xlarge` node delivers either high write throughput or a large follower fan-out, depending on how you load it: | Scenario | Result | | ------------------------------------------- | ------------------------------------------------ | | Pure ingest (no concurrent reads) | \~**100 MB/s** | | Read-serving (ingest pushed to drive reads) | 20 MB/s ingest + **50,000 concurrent followers** | The read-serving configuration sustains 50,000 concurrent followers each tailing their own keyed log, with an 8 GB cache and 1M keys, at **\<50 ms p50** poll latency. Reader nodes scale linearly: add a reader to add read throughput. ## Poll latency ### Flat across key cardinality At a fixed 20 MB/s ingest, increasing the key count from 100K to 1M does not degrade p50 poll latency for reading a single key's log: Poll latency holding flat near 39 ms as key cardinality grows from 100K to 1M | Key cardinality | p50 poll latency | | --------------- | ---------------- | | 100K | 38.9 ms | | 250K | 39.0 ms | | 500K | 39.8 ms | | 1M | 39.7 ms | Per-key read latency is independent of how many keys live on the node. ### Scales with ingest rate Poll latency is sensitive to ingest rate. Below the cache-residency point (\~5 MB/s) the live tail stays cache-resident and latency is low and flat. Above it, ingest evicts hot data before compaction collocates keys, and the tail climbs: p50 and p99 poll latency rising with ingest rate, p99 climbing toward seconds past 5 MB/s | Ingest rate | p50 | p99 | | ------------------------------ | ----- | -------- | | 1.25 MB/s | 3 ms | 14 ms | | 5 MB/s (cache-residency point) | 4 ms | 22 ms | | 10 MB/s | 12 ms | 140 ms | | 20 MB/s | 28 ms | 900 ms | | 25 MB/s | 36 ms | 2,200 ms | In practice you can expect p50/p99 poll latencies of \~30 ms / \~300 ms at 25 MiB/s of throughput. ### Range vs. random sharding Scoping a reader to a contiguous key range improves cache locality and thus improves poll latencies. At a fixed 25 MB/s ingest over 1M keys, the difference between different sharding schemes is shown below: p50 latency staying flat for range-scoped readers while random sharding rises with parallelism | Reader sharding | GETs / op | p50 latency | | --------------- | --------- | ----------- | | Range-scoped | 0.07–0.15 | 2.7–3.5 ms | | Random | \~0.15 | 3.5–6.4 ms | Range scoping holds read amplification and latency flat as parallelism grows. ## Cost The cost breakdown of one fully-loaded node serving 20 MB/s ingest, 50,000 concurrent followers, and 1M keys, with 1 day of retention works out to: | Line item | Driver | Est. / mo | | ---------- | ------------------------------ | ---------------- | | Compute | 1× `m5n.xlarge`, on-demand | \$174 | | S3 writes | flush every \~2–3 s ≈ 1.0M PUT | \~\$10 | | S3 storage | 1.73 TB/day × 1 day | \~\$40 | | **Total** | compute + S3 | **\~\$224 / mo** | GET costs are not included since they are mostly workload dependent. However, if most of your reads are for recent data, they will be served from the local cache. You can add a reader for another \~50k followers at +\$174/mo. Log flushes to S3 approximately every 2–3 s in this configuration. You can configure write-ahead-logging for faster durability guarantees at the cost of additional S3 `PUT` requests. ## Learn more * Storage internals: [Log storage design](/docs/log/storage-design) and the [storage RFC](https://github.com/opendata-oss/opendata/blob/main/log/rfcs/0001-storage.md) * For the funneling use case, see [OpenData Buffer benchmarks](/docs/buffer/benchmarks) # When to choose Log Source: https://opendata.dev/docs/log/comparisons How OpenData Log compares to Kafka, WarpStream and the Kafka-on-S3 systems, S2, and NATS JetStream *We build OpenData and this is our best good-faith comparison. Corrections are [welcome](https://github.com/opendata-oss/opendata/issues).* Log is built to deliver ordered, replayable events for high cardinality entities: a stream per user, session, agent, or device. With Log, every key is its own ordered stream and there are no topics or partitions to manage. ## Choose Log if you care about… **…a stream per user, session, agent, or device, created by just writing to it.** There are no topics to declare and no partition counts to size, whether you have a thousand streams or a million. **…reading specific streams.** Per-key scans are range reads on an LSM tree, and they stay fast as key cardinality grows. In [our benchmarks](/docs/log/benchmarks), going from 100K to 1M keys left single-key poll latency flat. **…serving tens of thousands of concurrent followers.** One node sustained 50,000 concurrent followers across 1M keys at 20 MB/s ingest with sub-50ms p50, about \$224/month all-in. Each added reader node serves another \~50K followers for \~\$174/month. **…replay from any point in history with retention at S3 prices.** Any offset in any stream is a range read; retention is dropping entire time-based segments. **…never operating a broker cluster.** Log runs as one binary with an object store bucket. A single node is durable, read replicas pull straight from S3 (and can be scoped to key ranges), and splitting a node is a metadata operation: consumers resume at the same sequence numbers. ## How the alternatives stack up ✓ = yes · \~ = partially · ✗ = no | You care about… | **Log** | Kafka | WarpStream / AutoMQ | S2 (cloud) | s2-lite | NATS JetStream | | -------------------------------------- | ------- | ---------- | ------------------- | ----------- | ------- | -------------- | | A stream per entity, zero provisioning | ✓ | ✗ ¹ | ✗ ¹ | ✓ | ✓ | \~ ² | | Fast per-key reads at millions of keys | ✓ | ✗ ¹ | ✗ ¹ | ✓ | \~ ³ | \~ ² | | Follow thousands of keys cheaply | ✓ | \~ ⁴ | \~ ⁴ | managed | \~ ³ | ✓ | | Replayable history at S3 prices | ✓ | \~ ⁵ | ✓ | ✓ | ✓ | ✗ ⁶ | | No broker cluster; single node durable | ✓ | ✗ | \~ ⁷ | managed | ✓ | ✗ ⁶ | | License | MIT | Apache 2.0 | Proprietary / BSL ⁷ | Proprietary | MIT | Apache 2.0 | ¹ Kafka-protocol systems route by topic-partition: one key's history means scanning its partition, and stream-per-entity means an unworkable partition count, on disks or on S3. \ ² NATS handles millions of subjects at low latency, but replayable history is bounded by the size of your replicated disks. \ ³ [s2-lite](https://github.com/s2-streamstore/s2) is the nearest open-source neighbor to Log: MIT, stream-per-key, also built on SlateDB. It's a young single-node companion to [S2's cloud](https://s2.dev), without published high-cardinality serving numbers. \ ⁴ Each follower tailing its own "stream" multiplies the problem in ¹. \ ⁵ [Kafka tiered storage](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage) parks old segments on S3, but retention and serving remain broker concerns. \ ⁶ [JetStream](https://docs.nats.io/nats-concepts/jetstream) persists to RAFT-replicated local disks you manage. \ ⁷ [WarpStream's agents](https://docs.warpstream.com/warpstream/overview/architecture) are stateless but require its proprietary hosted control plane; [AutoMQ](https://github.com/AutoMQ/automq)'s core is BSL. The systems that can do per-entity streams are managed services or very young. The mature open-source systems are funnels with the wrong data model for routing. Log is the self-hosted system built for routing. ## The tradeoffs * **Latency:** durable end-to-end latency is \~2s by default, though tuning can bring it lower. Polls run p50 \~30ms / p99 \~300ms at 25 MiB/s, and degrade if ingestion rates thrash the cache. * **You track your own offsets:** consumer groups and efficient tail-following are [active roadmap items](https://github.com/opendata-oss/opendata/blob/main/log/README.md); today, store per-key sequence numbers yourself. * **One writer node:** write scaling is vertical for now; readers scale horizontally. *** *Last reviewed July 2026. Please [tell us](https://github.com/opendata-oss/opendata/issues) if a claim has gone stale.* # Log Source: https://opendata.dev/docs/log/index A key-oriented, objectstore-native log for routing events to addressable destinations Log is an MIT-licensed, key-oriented log built on [SlateDB](https://slatedb.io) and object storage. It maintains millions of individually keyed, ordered logs on a single node and scales to tens of thousands of active readers, each tailing its own key. It runs as a single Rust binary with object storage as its only durability requirement. ## What it's good at * **Routing workloads.** Messaging, feeds, agent traces, and microservice communication, where each destination wants a specific subset of keys rather than the whole stream. * **High key cardinality.** Hundreds of thousands to millions of keys on one node. Any key's log is a prefix scan on an LSM index, not a needle-in-a-haystack scan of an entire partition. * **Large read fan-out.** Tens of thousands of concurrent followers reading from a single instance, scaled further by adding read replicas that pull straight from object storage. * **Per-key isolation.** Position metadata is tracked per key, so a poison-pill message stays contained to its key instead of blocking an entire partition. ## Why Log Kafka was built to collect data from many high-cardinality sources and coalesce it onto one low-cardinality pipe. Log is built for the other half of logging, *routing*: delivering events from granular sources to granular destinations. If your consumers each care about a specific subset of keys, you want to be able to retrieve the events for a key as efficiently as possible. That's why Log addresses events by key rather than topic-partitions. You simply `scan` the Log by key and get ordered values back. Object storage as the only durable layer makes Log strongly consistent and cheap to operate. Read replicas scale linearly, and scoping a replica to a contiguous key range keeps its cache warm with data that nearby queries will reuse. See the [benchmarks](/docs/log/benchmarks) for throughput, poll latency, and cost on a single node. ## Tradeoffs * **Poll latency scales with ingest rate.** Higher ingest rates evicts hot data before compaction can collocate keys for efficient querying. The mitigation is scoping read replicas to smaller key ranges so the cache isn't thrashed by unrelated blocks. * **Object-storage latency floors apply.** Polls inherit object-store round-trip latency. Plan for tens of milliseconds at the median and hundreds at the tail. * **No built-in offset tracking.** Log separates data from consumption metadata. Track consumer offsets yourself in a key-value store such as SlateDB. ## Explore Log Install Log and append your first events in under five minutes Browse the full REST API for appending, scanning, and managing logs How Log maps streams to LSM keys with segment-based compaction Deploy, monitor, and secure Log for production workloads Ingest throughput, poll latency, and cost for a single node How Log compares to Kafka, the Kafka-on-S3 systems, S2, and NATS JetStream View the source code, open issues, and contribute # Log: Getting to Production Source: https://opendata.dev/docs/log/production Deploy, monitor, and secure Log on Kubernetes with S3 storage This guide covers everything you need to run Log in production on Kubernetes: a complete Helm chart, S3-backed storage with a local disk cache, health checks, monitoring, and security. ## Deployment ### Overview Since we haven't yet built partitioning into Log, a production Log deployment consists of a single replica only. The primary means of scaling would be scaling up, which can take you pretty far. Since all data is persisted on S3, data in a single node Log is highly durable. With that said, a production Log deployment consists of: * A **single-replica Deployment** running the `opendata-log` container * An **S3 bucket** for durable data storage * A **PersistentVolumeClaim** backed by a fast SSD for the SlateDB disk cache * A **ConfigMap** for SlateDB tuning parameters * A **ServiceAccount** with an IAM role for S3 access (IRSA on EKS) Log uses SlateDB's epoch-based fencing, which means only one writer can hold the epoch lock at a time. The Deployment uses the `Recreate` strategy so that the old pod is fully terminated before the new one starts — a `RollingUpdate` would cause the new pod to be fenced by the old one and never become ready. ### Helm chart Below is a complete Helm chart for deploying Log to production. Create these files under `charts/opendata-log/`. #### `values.yaml` ```yaml values.yaml theme={null} image: repository: ghcr.io/opendata-oss/log tag: "0.2.2" port: 8080 # S3 storage configuration s3: bucket: my-log-bucket region: us-west-2 # SlateDB disk cache — use a fast SSD-backed StorageClass cache: size: 100Gi storageClassName: gp3 maxCacheSizeBytes: 107374182400 # 100 GB # SlateDB tuning slatedb: defaultTtl: 604800000 # 7 days (ms) maxUnflushedBytes: 134217728 # 128 MB l0SstSizeBytes: 16777216 # 16 MB maxSstSize: 67108864 # 64 MB resources: requests: cpu: 100m memory: 512Mi limits: cpu: "1" memory: 8Gi # IRSA role ARN for S3 access serviceAccount: roleArn: "" ``` #### `templates/configmap.yaml` ```yaml templates/configmap.yaml theme={null} apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-slatedb-settings data: SlateDb.yaml: | default_ttl: {{ int .Values.slatedb.defaultTtl }} max_unflushed_bytes: {{ int .Values.slatedb.maxUnflushedBytes }} l0_sst_size_bytes: {{ int .Values.slatedb.l0SstSizeBytes }} compactor_options: max_concurrent_compactions: 2 max_sst_size: {{ int .Values.slatedb.maxSstSize }} garbage_collector_options: manifest_options: interval: '60s' min_age: '3600s' wal_options: interval: '60s' min_age: '60s' compacted_options: interval: '60s' min_age: '3600s' compactions_options: interval: '60s' min_age: '3600s' object_store_cache_options: root_folder: /cache max_cache_size_bytes: {{ int .Values.cache.maxCacheSizeBytes }} ``` #### `templates/serviceaccount.yaml` ```yaml templates/serviceaccount.yaml theme={null} apiVersion: v1 kind: ServiceAccount metadata: name: {{ .Release.Name }} annotations: eks.amazonaws.com/role-arn: {{ .Values.serviceAccount.roleArn }} ``` #### `templates/pvc.yaml` ```yaml templates/pvc.yaml theme={null} apiVersion: v1 kind: PersistentVolumeClaim metadata: name: {{ .Release.Name }}-cache spec: accessModes: - ReadWriteOnce storageClassName: {{ .Values.cache.storageClassName }} resources: requests: storage: {{ .Values.cache.size }} ``` #### `templates/deployment.yaml` ```yaml templates/deployment.yaml theme={null} apiVersion: apps/v1 kind: Deployment metadata: name: {{ .Release.Name }} spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: {{ .Release.Name }} template: metadata: labels: app: {{ .Release.Name }} spec: serviceAccountName: {{ .Release.Name }} terminationGracePeriodSeconds: 60 securityContext: runAsNonRoot: true runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 containers: - name: log image: {{ .Values.image.repository }}:{{ .Values.image.tag }} args: - "--port" - "{{ .Values.port }}" - "--s3-bucket" - "{{ .Values.s3.bucket }}" - "--s3-region" - "{{ .Values.s3.region }}" ports: - containerPort: {{ .Values.port }} name: http env: - name: RUST_LOG value: info resources: {{- toYaml .Values.resources | nindent 12 }} livenessProbe: httpGet: path: /-/healthy port: http initialDelaySeconds: 10 periodSeconds: 30 readinessProbe: httpGet: path: /-/ready port: http initialDelaySeconds: 5 periodSeconds: 10 volumeMounts: - name: slatedb-settings mountPath: /app/SlateDb.yaml subPath: SlateDb.yaml readOnly: true - name: cache mountPath: /cache volumes: - name: slatedb-settings configMap: name: {{ .Release.Name }}-slatedb-settings - name: cache persistentVolumeClaim: claimName: {{ .Release.Name }}-cache ``` #### `templates/service.yaml` ```yaml templates/service.yaml theme={null} apiVersion: v1 kind: Service metadata: name: {{ .Release.Name }} spec: selector: app: {{ .Release.Name }} ports: - port: {{ .Values.port }} targetPort: http name: http ``` #### Install the chart ```bash theme={null} helm install opendata-log ./charts/opendata-log \ --set s3.bucket=my-log-bucket \ --set s3.region=us-west-2 \ --set serviceAccount.roleArn=arn:aws:iam::123456789012:role/opendata-log ``` ### Disk cache SlateDB caches frequently accessed data on local disk to avoid repeated reads from S3. For production workloads, use an SSD-backed StorageClass: * **EKS**: Use `gp3` (General Purpose SSD) or `io2` for higher IOPS. For maximum performance, use instance-store NVMe volumes with a [local-static-provisioner](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner). * Size the cache based on your active working set. The default of **100 Gi** is a good starting point; increase if you see frequent cache evictions in the `slatedb_*` metrics. Avoid using HDD-backed volumes (e.g. `st1`, `sc1`) for the cache. SlateDB issues many small random reads, and spinning disks will bottleneck performance. ### Health checks Log exposes two health-check endpoints: | Endpoint | Type | Behavior | | ------------ | --------- | ------------------------------------------------------ | | `/-/healthy` | Liveness | Returns 200 if the process is running | | `/-/ready` | Readiness | Returns 200 if the storage check passes, 503 otherwise | Both probes are included in the Helm chart's Deployment template above. ### Graceful shutdown Log handles `SIGTERM` and `SIGINT` signals gracefully: 1. Stops accepting new connections 2. Drains in-flight requests 3. Flushes pending data to durable storage 4. Exits cleanly The Helm chart sets `terminationGracePeriodSeconds: 60` to give the server enough time to complete the flush before Kubernetes force-kills the pod. ## Monitoring All metrics are exposed at `/metrics` in Prometheus text format. ### Key metrics | Metric | Type | Labels | Description | | ------------------------------- | --------- | ------------------------------ | ---------------------------------------------- | | `log_append_records_total` | counter | — | Total records appended | | `log_append_bytes_total` | counter | — | Total bytes appended | | `log_records_scanned_total` | counter | — | Total records scanned | | `log_bytes_scanned_total` | counter | — | Total bytes scanned | | `http_requests_total` | counter | `method`, `endpoint`, `status` | Total HTTP requests handled | | `http_request_duration_seconds` | histogram | `method`, `endpoint` | Request latency distribution | | `http_requests_in_flight` | gauge | — | Number of HTTP requests currently being served | Log also exposes `slatedb_*` metrics from the underlying SlateDB storage engine. These are useful for debugging storage-level performance and compaction behavior. ### Example PromQL queries ```promql theme={null} # Request rate (requests per second over 5 minutes) rate(http_requests_total[5m]) # Error rate (5xx responses) rate(http_requests_total{status=~"5.."}[5m]) # p99 request latency histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) # In-flight requests http_requests_in_flight # Append throughput (records per second) rate(log_append_records_total[5m]) # Scan throughput (bytes per second) rate(log_bytes_scanned_total[5m]) ``` ## Security ### TLS and authentication Log does not include built-in TLS termination or authentication. Place a reverse proxy (nginx, Envoy, or a cloud load balancer) in front of Log to handle TLS and access control. ### Object storage security The Helm chart uses [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (IAM Roles for Service Accounts) so that the pod receives temporary AWS credentials automatically — no static access keys required. Create an IAM role with the following policy and attach it to the ServiceAccount via the `serviceAccount.roleArn` value: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::my-log-bucket", "arn:aws:s3:::my-log-bucket/*" ] } ] } ``` The IAM role's trust policy should scope access to your EKS cluster's OIDC provider and the specific service account: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/EXAMPLE" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.us-west-2.amazonaws.com/id/EXAMPLE:aud": "sts.amazonaws.com", "oidc.eks.us-west-2.amazonaws.com/id/EXAMPLE:sub": "system:serviceaccount:default:opendata-log" } } } ] } ``` Additional recommendations: * Enable **encryption at rest** on the S3 bucket (SSE-S3 or SSE-KMS). * Use a **VPC endpoint** for S3 to keep traffic off the public internet. * Block all public access on the bucket. * Add a **lifecycle rule** to transition old data to Intelligent-Tiering after 30 days and abort incomplete multipart uploads after 7 days. # Log Quickstart Source: https://opendata.dev/docs/log/quickstart Install Log locally and append your first events in minutes Get an append-only event log running on your machine in under five minutes. You will install Log, start the server, append events, and read them back using curl. ## Install OpenData Log Download and install the Log binary: ```bash theme={null} curl -fsSL https://www.opendata.dev/install.sh | sh -s -- log ``` This places the `opendata-log` binary in the current directory. ## Start the server Start Log on port 8081: ```bash theme={null} ./opendata-log --port 8081 ``` No configuration file is needed — all options are passed as CLI flags. Data is stored in `.data/` by default. ## Append records Append three order events to the `orders` key. Keys and values are base64-encoded in the JSON body: ```bash theme={null} curl -X POST http://localhost:8081/api/v1/log/append \ -H "Content-Type: application/json" \ -d '{ "records": [ { "key": "b3JkZXJz", "value": "eyJpdGVtIjoid2lkZ2V0IiwicXR5IjozLCJwcmljZSI6OS45OX0=" }, { "key": "b3JkZXJz", "value": "eyJpdGVtIjoiZ2FkZ2V0IiwicXR5IjoxLCJwcmljZSI6MjQuOTV9" }, { "key": "b3JkZXJz", "value": "eyJpdGVtIjoid2lkZ2V0IiwicXR5IjoxMCwicHJpY2UiOjkuOTl9" } ] }' ``` The key and values decode to: | Base64 | Plain text | | ------------------------------------------------------ | ----------------------------------------- | | `b3JkZXJz` | `orders` | | `eyJpdGVtIjoid2lkZ2V0IiwicXR5IjozLCJwcmljZSI6OS45OX0=` | `{"item":"widget","qty":3,"price":9.99}` | | `eyJpdGVtIjoiZ2FkZ2V0IiwicXR5IjoxLCJwcmljZSI6MjQuOTV9` | `{"item":"gadget","qty":1,"price":24.95}` | | `eyJpdGVtIjoid2lkZ2V0IiwicXR5IjoxMCwicHJpY2UiOjkuOTl9` | `{"item":"widget","qty":10,"price":9.99}` | The server responds with the starting sequence number and how many records were written: ```json theme={null} { "status": "success", "recordsAppended": 3, "startSequence": 0 } ``` ## Read records back Scan all entries for the `orders` key: ```bash theme={null} curl "http://localhost:8081/api/v1/log/scan?key=orders" ``` The response includes each entry's global sequence number and its base64-encoded value: ```json theme={null} { "status": "success", "key": "b3JkZXJz", "values": [ { "sequence": 0, "value": "eyJpdGVtIjoid2lkZ2V0IiwicXR5IjozLCJwcmljZSI6OS45OX0=" }, { "sequence": 1, "value": "eyJpdGVtIjoiZ2FkZ2V0IiwicXR5IjoxLCJwcmljZSI6MjQuOTV9" }, { "sequence": 2, "value": "eyJpdGVtIjoid2lkZ2V0IiwicXR5IjoxMCwicHJpY2UiOjkuOTl9" } ] } ``` Add `limit` to paginate through large result sets: ```bash theme={null} curl "http://localhost:8081/api/v1/log/scan?key=orders&limit=1" ``` ## Explore more endpoints List all keys in the log: ```bash theme={null} curl "http://localhost:8081/api/v1/log/keys" ``` List storage segments: ```bash theme={null} curl "http://localhost:8081/api/v1/log/segments" ``` ## Next steps * Explore the full API in the **API reference** section in the sidebar. * Learn how segments and compaction work in [Storage design](/docs/log/storage-design). # Log Storage Design Source: https://opendata.dev/docs/log/storage-design How Log maps streams to LSM keys with segment-based compaction Log stores all data as key-value records in SlateDB. Each user key is its own independent log stream (similar to a "topic" in Kafka). Writes are appended to the WAL and memtable, then flushed to sorted string tables (SSTs). LSM compaction naturally groups entries by key prefix over time, providing efficient sequential reads even for historical data. This page covers the conceptual storage model. For exact byte-level encoding schemas, see the [storage RFC on GitHub](https://github.com/opendata-oss/opendata/blob/main/log/rfcs/0001-storage.md). ## Key encoding SlateDB keys are a composite of the user key and a `u64` sequence number. A version prefix and record type discriminator provide forward compatibility. | Component | Description | | ------------ | ----------------------------------------------------------------------------------------------------- | | **Version** | A `u8` prefix (initially `1`) for forward compatibility | | **Type** | A `u8` discriminator identifying the record type (`0x01` for log entries, `0x02` for sequence blocks) | | **Key** | The user key, encoded as `Bytes` | | **Sequence** | A `u64` sequence number | This encoding preserves lexicographic key ordering, enabling key-range scans. Entries for the same key are ordered by sequence number. ## Record types | Record type | Description | | ---------------- | --------------------------------------------------------------------------------------------------- | | **LogEntry** | Stores the user's `(key, value)` pairs, ordered by segment and sequence number | | **SeqBlock** | Tracks sequence number block allocations for crash recovery (singleton record) | | **SegmentMeta** | Stores metadata for each segment including its start sequence and creation time | | **ListingEntry** | Tracks which keys are present in each segment, enabling key discovery without scanning the full log | ### Segments A segment is a logical boundary in the log's sequence space. Each segment represents a contiguous range of sequence numbers across the full keyspace. Segments are numbered starting from 0 and increment monotonically. The segment ID is encoded directly into every `LogEntry` key, which means SlateDB physically clusters records from the same segment together on disk. This provides two key benefits: * **Efficient seeking**: queries targeting a specific time range can skip segments outside that range without scanning the full log. * **Retention**: entire segments can be dropped when they age out, rather than tracking expiration per key. New segments are created automatically based on a configurable time-based trigger (e.g. every hour). Each segment's `SegmentMeta` record stores its `start_seq` and `start_time_ms`, with end boundaries derived from the next segment's start values. ### Listings The log entries provide no built-in way to discover which keys are present. Listing records solve this by tracking key presence per segment. When the writer encounters a key for the first time within a segment, it writes a `ListingEntry` record. Subsequent appends to the same key within that segment do not write additional listing records. When a new segment starts, tracking resets. This design ties key discovery to the segment lifecycle. When segments are deleted through retention, their listing records are removed as well, and keys that are no longer present in any remaining segment naturally fall out of scope. ## Sequence numbers Sequence numbers are assigned from a single monotonically increasing counter maintained by the SlateDB writer. Each key's log entries are ordered by sequence number, but numbers are not contiguous. The only guarantee is that within a key's log, sequence numbers are strictly increasing. ### Block-based allocation Rather than persisting the sequence number after every append, the writer pre-allocates blocks of sequence numbers and records the allocation in the LSM using a `SeqBlock` record. On crash recovery, the writer reads the last `SeqBlock` and allocates a fresh block starting after the previous range, skipping any unused numbers. This may create gaps in the sequence space but preserves monotonicity. ## SST enhancements Log proposes two enhancements to SlateDB's SST structure: | Enhancement | Purpose | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | **Block record counts** | Each block entry in the SST index includes a cumulative record count, enabling range counting at the index level without reading every entry | | **Bloom filter granularity** | Bloom filters are keyed on the log key alone (not the composite key with sequence number), so they indicate whether a given log is present in an SST | # OpenData: Architecture Source: https://opendata.dev/docs/overview/architecture OpenData databases share a common architecture built on [SlateDB](https://slatedb.io), an LSM tree engine on object storage. For details on the read and write paths, see [Writing Data](/docs/overview/writing-data) and [Reading Data](/docs/overview/reading-data). ## Single Writer OpenData databases are designed to operate as single-writer systems for each partition of the data. This is conceptually similar to a leader-follower architecture, but without the followers: object storage takes over the role that replicas traditionally play, providing durability and availability without requiring the writer to coordinate replication. Writer failover and fencing of old writers is handled by the object storage compare-and-set mechanism. The new writer fences the old one via a compare-and-set on the manifest and resumes from durable state in object storage. This ensures that data written to OpenData databases is strongly consistent as soon durability is acknowledged. ## Disaggregated Compaction & GC Because all data lives in object storage, compaction and garbage collection do not need to run on the same process or machine as the writer. This means that background maintenance never competes with ingest or queries for CPU, memory, or disk I/O. It also means compaction and GC workloads can be scheduled on cheaper, lower-priority compute (e.g. spot instances) since they are stateless and can be restarted at any time without data loss. The writer and readers continue to operate normally regardless of whether compaction is running. ## Stateless Zonal Ingestion Databases that don't require read-your-write consistency can ingest data from stateless services deployed in each availability zone. This allows for both high-availability ingest and avoids cross-zone data transfers. For example, ingesting metrics into `timeseries` without stateless zonal ingestion would require a remote write to the single-writer deployed in a particular zone. If the single-writer is down, the metric write request will fail. With stateless zonal ingestion, the metrics are ingested directly into S3 from within the region they are produced and then transferred to the single writer over S3. This both avoids the cross-zone data transfer costs and only depends on the availability of S3. Timeseries implements this pattern for OpenTelemetry metrics by leveraging the [`opendata` Buffer](/docs/buffer/index) library via the `opendata` OTel Collector exporter and an in-server buffer consumer. See [Stateless Ingest](/docs/timeseries/ingest) for the write path. ## Multiple Readers Because all state lives in object storage, you can deploy any number of read-only replicas independently of the writer. These replicas are stateless and can be added or removed at any time without coordination. Readers automatically watch object storage for new data written by the main writer, so they stay up to date without any direct coordination between processes. While performance depends on warm caches, data is always available for reads even without the cache by accessing it directly from object storage. This is a particularly useful property for scaling out read workloads: adding a new read partition does not require any coordination with the main writer or other readers. Other readers' caches will eventually drain as queries for that subset of data are no longer served while the new reader's cache will warm up as it serves queries. This architecture also enables zonal reads, which can help reduce latency and intra-zone data transfers. ## Checkpoints & Backups The nature of designing a system as an immutable LSM tree on top of object storage means that checkpointing data and/or creating backups is trivial. A single, O(1) metadata operation records the current state of the database and marks the files as immune from garbage collection. This checkpointed state can be retained for as long as desired and used to restore the database to a previous state, or to create a branch for testing or debugging. # OpenData: Deployment Source: https://opendata.dev/docs/overview/deployment OpenData is fundamentally designed to have a flexible architecture that scales to meet the use case. Smaller deployments can be deployed on a single machine while still providing (meaningfuly) high availability and object-storage backed durability. Larger deployments can scale to multiple zones and regions, providing high availability and disaster recovery. ## Embedded Option At their core each database can run as an embedded rust process that you can access via the rust API (FFI integrations for other languages are planned, though not currently prioritized). The embedded option is a fully viable option for production deployments. Since durability is managed fully by object storage, a crash/restart of an application embedding a database will not lose any data that was acknowledged as durable. While we recommend running OpenData services with persistent disks for cache durability, this is not required. In addition, if a pod running the embedded database crashes, new deployments of it will fence the old writer to ensure consistentcy in the event the old service comes back online and attempts to write new data. ## Single Binary Server OpenData systems are fully deployable with single binaries. We provide binaries for most common operating systems and architectures as well as docker images. This makes it easy to deploy an OpenData system both on your laptop as well as in production. The difference between the embedded and single binary server options is that the binaries package additional dependencies required to run an HTTP server to handle requests from a client over the network. ## Distributed System We are currently designing more native distributed deployment options that will support shuffling data across partitions for a smoother scaling experience. For larger deployments, we currently recommend partitioning your data into cells and deploying a single-writer database in each cell. This allows you to scale your deployment horizontally by adding more partitions. ## Get started Install and run a Prometheus-compatible metrics database on your machine in minutes. Install and run a key-oriented event streaming database on your machine in minutes. # OpenData: Reading Data Source: https://opendata.dev/docs/overview/reading-data Reading data from OpenData databases heavily leverages data caches to avoid unnecessary `GET` API calls to object storage. Reading data relies on a combination of domain-specific logic as well as shared storage access primitives. Each database implements its own indexes and data layouts for optimal read performance, but all use SlateDB queries to access the data through multiple levels of caches. ### Read Freshness Read freshness can be measured in terms of object storage round trips between a write being accepted and it being visible to a query: | Write Path | Read Target | S3 Round Trips | Description | | ---------------------------------------------------------------- | ------------------------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Direct write to writer | Writer | **0** | Data is written to the in-memory Delta and is immediately readable on the writer with no object storage interaction. | | [Zonal ingest](/docs/overview/architecture#stateless-zonal-ingestion) | Writer | **1** | Data is written to object storage by a stateless ingestor and must be picked up by the writer before it is readable. | | Any | [Read replica](/docs/overview/architecture#multiple-readers) | **+1** | Read replicas discover new data by watching object storage for manifest updates, adding one additional round trip on top of whatever the write path requires. | Each round trip typically adds around 100ms of latency plus the polling interval, which is configurable. In practice, the polling interval dominates the freshness delay. # OpenData: Storage Source: https://opendata.dev/docs/overview/storage All OpenData databases encode their durable storage as key-value records in SlateDB. The challenge is mapping rich domain models (time series, events, vectors) onto flat, ordered KV pairs. To accomplish this, we leverage a few design patterns to project concepts onto a key-value model. Not all databases use all of these patterns or techniques. This page serves as a general overview of the storage patterns used. ## Structured Key Model LSM trees provide efficient point lookups and range scans on key prefixes. This is due to the physical clustering of keys with based on their lexicographical sort ordering. To take advantage of this, OpenData systems carefully construct keys both to model the raw data but also to construct secondary indexes and embed them within the underlying key-value storage. In the example below, we have three different "types" of keys. The type of the key is encoded in the first byte `data|index|meta`, and then the value of the key is dependent on the type. The raw data key is just the user ID. For the inverted index, the key is `attribute_name:attribute_value` and the value are all the user ids that have the given attribute pair. The metadata keys are global keys that describe the database (e.g. the version or total number of rows). ## Segmenting Data Because keys are physically sorted in the LSM, encoding a segment identifier into the key naturally groups related records together on disk. A query that targets a single segment only scans the key range for that segment so everything outside it is never read. Each segment can also be managed independently: old segments can be compacted, rolled up, or garbage collected without affecting the rest of the keyspace. The segment field can represent whatever boundary makes sense for the data model — a time window, a partition ID, a record type prefix, or a monotonic sequence number. The key insight is that the LSM's sort order turns logical partitions into physical locality, giving you the benefits of partitioned storage without managing separate stores. ## Miscellaneous OpenData systems actively leverage some interesting features of SlateDB in order to optimize storage efficiency and performance: | Property | How it works | Benefit | | ---------------------- | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Prefix compression** | Common prefixes are only stored once the storage system. | OpenData keys rely heavily on prefixes, and these prefixes are often repeated. This reduces storage overhead without requiring explicit dictionary encoding. | | **Merge operators** | New entries are written as merge operands instead of read-modify-write. The LSM merges them lazily during compaction. | Turns N individual writes into a single merged result, avoiding read-modify-write cycles for posting lists and bitmaps. | For specific encoding details and storage layouts, see the per-database design pages: [Timeseries](/docs/timeseries/storage-design), [Log](/docs/log/storage-design), and [Vector](/docs/vector/storage-design). # OpenData: Writing Data Source: https://opendata.dev/docs/overview/writing-data The OpenData family of databases share a common foundation to provide a unified experience and similar operational characteristics. Nearly all the databases share a similar read/write path. The write path accepts data and writes it both to a durable WAL as well as into in-memory data structures that are eventually flushed to object storage as SSTs in an LSM tree: Writes are assigned a canonical ordering before being acknowledged to ensure that data is written consistently. From there, small WAL (write-ahead log) files are created and flushed frequently to object storage to ensure durability within a configurable time delay. These files are used to recover the database in the event of a system failure. Simultaneously, data is written into an in-memory data structure we call a `Delta`. This is used to make new data immediately available for reads. Once the `Delta` is full, it is flushed to object storage as a SlateDB SST. While the `Delta` implementation is domain-specific (unique to each database), the management mechanism is common across all databases, ensuring that the ingest semantics are correct and well tested. LSM Tree maintence tasks such as comapction and garbage collection are handled asynchronously and can run on a separate runtime (or even on a different machine). While each database can implement its own comapction strategy, we delegate the compaction execution to SlateDB. See [Compaction in SlateDB](https://slatedb.io/rfcs/0002-compaction/). ### Flexible Durability Guarantees The common write coordination mechanism allows users to control the required durability before acknowledging the write. This allows users to trade off durability for write latency. This tradeoff is fundamental to the design of object-store native systems. At one end of the spectrum, writes can be acknowledged as soon as the data is buffered in the in-memory Delta. This offers the lowest latency but means that data written since the last WAL flush is at risk if the process crashes. At the other end, writes can wait until the WAL is flushed to object storage before being acknowledged, providing full durability at the cost of higher write latency. | Durability Level | Ack Condition | Latency | Data at Risk | | ---------------- | ------------------------------- | ------- | ----------------------------- | | **Buffered** | Data written to in-memory Delta | Lowest | Writes since last WAL flush | | **WAL-durable** | WAL flushed to object storage | Higher | None (survives process crash) | Since WAL files are flushed frequently on a configurable interval, the buffered mode still provides durability within a bounded time window. Choosing the right level depends on the use case. A third option decouples durability from the writer entirely: clients can write into an object-storage queue, and the main writer drains that queue asynchronously. The write returns as soon as the batch is durable in object storage, so ingest keeps working across writer restarts or failovers. See [Stateless Ingest](/docs/timeseries/ingest) for the Timeseries implementation. ### Backpressure & Flow Control When writes arrive faster than Deltas can be flushed to object storage, the system applies backpressure to prevent unbounded memory growth. Once the number of unflushed Deltas exceeds a configurable threshold, new writes are stalled until a flush completes and frees memory. This flow control mechanism ensures that the system degrades gracefully under load rather than running out of memory. The backpressure threshold is tuned alongside the Delta size and flush interval to balance write throughput against memory usage. # Timeseries Benchmarks Source: https://opendata.dev/docs/timeseries/benchmarks Ingestion throughput, query latency, and cost for a single-node Timeseries deployment ## Ingestion Ingestion was measured with [p8s-bench](https://github.com/responsivedev/p8s-bench), a harness around VictoriaMetrics' [prometheus-benchmark](https://github.com/VictoriaMetrics/prometheus-benchmark) tool. The load generator produced samples for 5,100 targets every 60 seconds, for a total of \~3.3M unique active series. On a single `m5.xlarge` node (4 vCPU, 16 GB RAM) with SlateDB's WAL disabled, Timeseries sustained: | Metric | Result | | ------------------- | --------------------------- | | Sustained ingestion | **55k samples/sec** | | Daily volume | **4.7B samples/day** | | Active series | \~3.3M unique series | | Node | `m5.xlarge` (4 vCPU, 16 GB) | Soak chart showing ingestion holding at ~55k samples/sec over time Disabling the WAL is acceptable for many timeseries workloads, particularly those paired with a durable upstream log like [OpenData Buffer](/docs/buffer). ## Query latency Query latency depends mostly on how much data a query touches and whether that data is already in the SlateDB block cache. The chart below plots cold and warm query latency as a function of the number of series matched and scanned over a 6-hour time range. Bar chart comparing cold and warm query latency across increasing series counts Warm numbers are the ones that matter for day-to-day use: alerts and active dashboards keep their data warm, and once recent data is in the block cache, queries stop paying object-store round trips. On the benchmark `r5d.xlarge` node, roughly 8 GB of RAM plus \~140 GB of NVMe-backed disk cache keep several weeks of data warm (assuming 1–2 bytes per sample for Gorilla-compressed blocks). Cold reads pay the object-store round trip (10–100 ms). ## Cost The same workload (3.3M active series, 4.7B samples/day) costs roughly **\$560/month of compute**: | Component | Spec | Monthly | | ----------------- | ----------------------------------- | ------------- | | Writer | 1× `m5.xlarge` | \~\$140 | | Readers | 2× `r5d.xlarge` (140 GB local NVMe) | \~\$210 each | | **Compute total** | | **\~\$560** | | S3 PUT requests | WAL disabled | \~\$5–12 | | S3 storage | standard rates | a few dollars | ## Reproduce it * Ingestion harness: [p8s-bench](https://github.com/responsivedev/p8s-bench) * Underlying tool: [VictoriaMetrics prometheus-benchmark](https://github.com/VictoriaMetrics/prometheus-benchmark) * Storage design: [Timeseries storage design](/docs/timeseries/storage-design) and the [TSDB storage RFC](https://github.com/opendata-oss/opendata/blob/main/timeseries/rfcs/0001-tsdb-storage.md) # When to choose Timeseries Source: https://opendata.dev/docs/timeseries/comparisons How OpenData Timeseries compares to Prometheus, Thanos, Mimir, VictoriaMetrics, GreptimeDB, and managed observability *We build OpenData and this is our best good-faith comparison. Corrections [welcome](https://github.com/opendata-oss/opendata/issues).* Timeseries is the only Prometheus-compatible TSDB where object storage is the sole durable layer, so a production deployment is a stateless writer, stateless readers, and an object store bucket. Every alternative keeps durable state on disks somewhere, and the painful parts of self-hosting metrics at scale (managing replicated state with multiple services) follow from that. ## Choose Timeseries if you care about… **…keeping your Grafana dashboards and Prometheus tooling.** Remote write, scraping, PromQL, and OTLP ingest: you repoint your collectors, dashboards, and alerts at a new backend instead of adopting a new ecosystem. **…paying for compute and storage, not per-sample pricing.** [Our benchmark](/docs/timeseries/benchmarks) ran 3.3M active series and 4.7B samples/day on a single small writer, about \$560/month of compute for the full cluster. The same workload at [published rates](https://aws.amazon.com/prometheus/pricing/) is roughly \$12,800/month on Amazon Managed Prometheus, because they charge by the number of series. **…never losing metrics when a node dies.** Every write is durable in the object storage. **…a simple deployment.** A Timeseries deployment is one writer, N independent readers, and an object store bucket. The processes are decoupled, and scaling means adding or removing pods; no data rebalancing is involved. **…high cardinality without building a cluster.** Timeseries can handle millions of active series on one 4-vCPU writer, with reads scaled independently by adding reader pods. ## How the alternatives stack up ✓ = yes · \~ = partially · ✗ = no | You care about… | **Timeseries** | Prometheus | Thanos | Mimir | VictoriaMetrics | GreptimeDB | Managed (AMP, Grafana Cloud, …) | | ----------------------------------------------------------- | -------------- | ---------- | ---------- | ------ | --------------------- | --------------------- | ------------------------------- | | Drop-in for Prometheus dashboards & alerts | \~ ¹ | ✓ | ✓ | ✓ | \~ ² | ✓ | ✓ | | Cost = compute + S3, no per-sample pricing | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ ³ | | Durable with a single replica | ✓ | ✗ | \~ ⁴ | \~ ⁴ | ✗ ⁵ | ✓ | managed | | Simple deployment: independent stateless writer and readers | ✓ | ✓ ⁶ | ✗ | ✗ | \~ ⁵ | \~ ⁷ | managed | | Scale without migrating data | ✓ | ✗ | ✗ | ✗ | ✗ | \~ ⁷ | managed | | Millions of series without a cluster | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | n/a | | License | MIT | Apache 2.0 | Apache 2.0 | AGPLv3 | Apache 2.0, open core | Apache 2.0, open core | Proprietary | ¹ Full PromQL and Prometheus API compatibility are [roadmap items](https://github.com/opendata-oss/opendata/blob/main/timeseries/README.md). Test your real dashboards before committing. \ ² VictoriaMetrics implements [MetricsQL](https://docs.victoriametrics.com/metricsql/), which deliberately deviates from PromQL in places. \ ³ The benchmark workload above: \~\$12,830/mo on AMP at [published rates](https://aws.amazon.com/prometheus/pricing/) vs. \~\$560/mo of compute self-hosted. \ ⁴ Thanos and Mimir put historical blocks on object storage, but recent data lives on stateful sidecar/ingester disks with replication (see [Mimir's architecture](https://grafana.com/docs/mimir/latest/references/architecture/)), which is where the hash rings, quorums, and rebalancing come from. \ ⁵ Simple and durable until you outgrow one node; cluster mode is stateful, and replication is off unless you configure it. \ ⁶ Simple, but it doesn't scale past a single node. \ ⁷ [GreptimeDB](https://github.com/GreptimeTeam/greptimedb) is the nearest neighbor: object-storage-primary, Rust, native PromQL, as part of a broader metrics+logs+traces platform. Its standalone binary is simple; cluster mode adds a metadata service and datanode roles. Our bet is a narrower metrics-only primitive on the shared OpenData operating model. The fully Prometheus-compatible systems keep durable state on disks you manage. The simple systems stop being simple (or durable) at scale. The managed services trade the operational problem for per-sample pricing. Timeseries covers all six rows and is MIT-licensed. ## The tradeoffs * **Freshness:** batched object-storage writes add 1–5 seconds before samples are queryable. * **Cold reads** pay S3 round trips. NVMe caches keep weeks of data warm, covering most alerting and dashboard queries. But wide uncached historical scans will feel slower than disk-native systems. * **Compatibility gaps:** full PromQL/API coverage, retention policies, downsampling, and recording rules are roadmap items. * **One writer:** each Timeseries deployment can have only one writer which must be scaled vertically. Readers scale horizontally today. The [benchmarks](/docs/timeseries/benchmarks) show a single 4-vCPU writer sustaining 3.3M active series and 4.7B samples/day. *** *Last reviewed July 2026. Please [tell us](https://github.com/opendata-oss/opendata/issues) if a claim has gone stale.* # Timeseries Configuration Source: https://opendata.dev/docs/timeseries/configuration Configure scrape targets, storage backends, and retention for Timeseries Timeseries is configured through CLI arguments and a Prometheus-compatible YAML configuration file. The CLI arguments control server startup, while the YAML file defines scrape targets, storage backends, and operational parameters. ## CLI Arguments Pass these flags when starting the `opendata-timeseries` binary. | Flag | Environment Variable | Default | Description | | ---------------- | ------------------------ | ------- | ------------------------------------------------ | | `-c`, `--config` | `PROMETHEUS_CONFIG_FILE` | *none* | Path to the `prometheus.yaml` configuration file | | `-p`, `--port` | `OPEN_TSDB_PORT` | `9090` | Port to listen on | ```bash theme={null} ./opendata-timeseries --config prometheus.yaml --port 9090 ``` ## Configuration File The configuration file follows the Prometheus YAML format with additional sections for storage. The generic placeholders used in this reference are: * ``: a regular string * ``: an integer value * ``: a duration matching `[0-9]+(ms|s|m|h|d)` (e.g. `30s`, `1m`, `2h`) A value enclosed in `[ ]` is optional. The default value is shown after `| default =`. ```yaml theme={null} # Default settings applied to all scrape jobs. global: # How often to scrape targets. [ scrape_interval: | default = 15s ] # List of scrape job configurations. scrape_configs: [ - ... ] # Storage backend configuration (required). storage: # How often (in seconds) to flush data from memory to durable storage. [ flush_interval_secs: | default = 5 ] # Startup cache warmer (enabled by default). [ cache_warmer: ] # Durable buffer consumer. See the Stateless Ingest page for the full write path. [ buffer_consumer: ] ``` ### `` Each scrape config defines a set of targets and the parameters for scraping them. ```yaml theme={null} # Name for this scrape job. Added as the `job` label on all scraped metrics. job_name: # Override the global scrape interval for this job. [ scrape_interval: | default = ] # List of static target groups to scrape. static_configs: [ - ... ] ``` ### `` A static config defines a list of targets and a common set of labels to apply. ```yaml theme={null} # Endpoints to scrape in host:port format. targets: [ - ... ] # Labels to attach to all metrics scraped from these targets. labels: [ : ... ] ``` ### `` Storage is tagged by `type`. ```yaml theme={null} # The storage backend type. type: # One of: InMemory, SlateDb ``` Stores data in memory only. Data is lost on restart. Useful for testing and development. No additional fields are required. ```yaml theme={null} storage: type: InMemory ``` Persists data to an object store via [SlateDB](https://slatedb.io). ```yaml theme={null} storage: type: SlateDb # Path prefix for SlateDB data within the object store. [ path: | default = "data" ] # Path to a SlateDB settings file (TOML/YAML/JSON). # If omitted, SlateDB checks for SlateDb.toml, SlateDb.json, or SlateDb.yaml # in the working directory and merges any SLATEDB_-prefixed environment variables. [ settings_path: ] # Block cache (optional). See below. [ block_cache: ] # Object store provider configuration (required). object_store: ``` ### `` The block cache sits in front of SlateDB and holds decoded SST blocks. It is tagged by `type`. Only `FoyerHybrid` is currently available. ```yaml theme={null} block_cache: type: FoyerHybrid # In-memory tier capacity in bytes. memory_capacity: # On-disk tier capacity in bytes. disk_capacity: # Directory for the on-disk tier. disk_path: # When entries are promoted to the disk tier. # WriteOnInsertion persists every cached block; WriteOnEviction # only persists on memory eviction. [ write_policy: | default = WriteOnInsertion ] # Flush thread count for the disk engine. [ flushers: | default = 4 ] # Flush-pipeline buffer pool size in bytes. Actual allocation is ~2x # this value (flushers double-buffer). Defaults to memory_capacity / 32. [ buffer_pool_size: ] # Write-queue size threshold in bytes. Entries beyond this are dropped # rather than blocking the cache. [ submit_queue_size_threshold: | default = 1073741824 ] ``` See the [production guide](/docs/timeseries/production#block-cache) for recommended sizes. ### `` The object store is tagged by `type`. ```yaml theme={null} object_store: type: Local # Directory path for local storage. path: ``` ```yaml theme={null} object_store: type: Aws # AWS region (e.g. us-west-2). region: # S3 bucket name. bucket: ``` ```yaml theme={null} object_store: type: InMemory ``` ### `` On startup, the server scans recent time bucket key ranges through the storage reader. The block cache picks up those blocks as a side effect, so the first queries after a restart do not pay cold-cache latency. The warmer runs once and exits. Enabled by default. Set `cache_warmer: null` to disable. ```yaml theme={null} cache_warmer: # How far back to scan on startup. [ warm_range: | default = 24h ] # Whether to warm raw sample data in addition to index metadata. [ include_samples: | default = true ] ``` Disable the warmer when starting read-only replicas that only serve recent queries, where the up-front scan cost outweighs the benefit. ### `` Enables the durable-queue write path. See the [Stateless Ingest page](/docs/timeseries/ingest) for the full architecture, the OpenTelemetry Collector side, and operational guidance. ```yaml theme={null} buffer_consumer: # Object store holding the buffer queue. May differ from the TSDB's # storage bucket. object_store: # Must match the manifest path configured on the producer side. [ manifest_path: | default = "ingest/manifest" ] # Delay between polls when the queue is empty. [ poll_interval: | default = 1s ] ``` Requires read-write mode. When absent, no consumer starts. ## Examples ```yaml prometheus.yaml theme={null} global: scrape_interval: 15s scrape_configs: - job_name: "my-app" static_configs: - targets: ["localhost:9090"] storage: type: SlateDb path: data object_store: type: Local path: ./data flush_interval_secs: 5 ``` ```yaml prometheus.yaml theme={null} global: scrape_interval: 30s scrape_configs: - job_name: "node-exporter" scrape_interval: 15s static_configs: - targets: ["host-a:9100", "host-b:9100"] labels: env: production - job_name: "api-server" static_configs: - targets: ["api-1:8080", "api-2:8080"] labels: env: production storage: type: SlateDb path: data object_store: type: Aws region: us-west-2 bucket: my-metrics-bucket ``` ```yaml prometheus.yaml theme={null} global: scrape_interval: 5s scrape_configs: - job_name: "test-app" static_configs: - targets: ["localhost:9090"] storage: type: InMemory ``` # Timeseries Source: https://opendata.dev/docs/timeseries/index A Prometheus-compatible metrics database on object storage. OpenData Timeseries is an MIT-licensed, Prometheus-compatible metrics database built on [SlateDB](https://slatedb.io) and object storage. It speaks PromQL, scrapes Prometheus targets, accepts Prometheus remote write and OTLP metrics, and works out of the box with Grafana. It brings the object-store-native operating model, ie. stateless compute nodes with object storage as the only durable layer, to the Prometheus and Grafana ecosystem. ## What it's good at * **Drop-in Prometheus and Grafana.** PromQL, scraping, remote write, and OTLP ingest, so your existing dashboards and alerts keep working. * **High-cardinality ingestion.** Millions of active series on a single stateless writer, scaled by scaling the writer up. * **Operational simplicity.** All you need for high-scale metrics is one writer, zero or more readers, and an object store bucket. * **Cheap retention.** Local NVMe keeps weeks of recent data warm, while cold data lives on object storage at a large discount to disk-resident storage. ## Why Timeseries Running Prometheus-compatible systems at scale has meant running an unforgiving distributed storage system: quorum replication across stateful nodes, shard management, and rebalancing when nodes fail or scale. Thus, whether you self-host these systems or pay someone to manage them for you, they can get expensive very fast. However, once object storage is the only durable layer and local disks are demoted to caches, most of the complicated distributed database machinery disappears and the operational cost drops substantially. This is what Opendata Timeseries delivers. Because there is no replicated, partitioned state to babysit, you can realistically run Timeseries yourself and capture the cost gap against managed services. See the [benchmarks](/docs/timeseries/benchmarks) for ingestion throughput, query latency, and a cost comparison against managed Prometheus offerings. ## Tradeoffs * **Cold reads pay object-store round trips.** A query for data that isn't cached locally is slower than the same query against a disk-resident system. SlateDB's block cache keeps recent and frequently read data warm, which covers most alerting and dashboard queries. * **Higher end-to-end write latency.** Batching writes into immutable objects adds a few seconds before fresh data is queryable. Given scrape intervals are already on the order of a minute, that delay is rarely noticeable for observability. where the most optimization work is happening today. ## Explore Timeseries Install Timeseries and query your first metrics in under five minutes Browse the Prometheus-compatible query and write API How Timeseries organizes data in time buckets on SlateDB Deploy, monitor, and secure Timeseries for production workloads Ingestion throughput, query latency, and cost for a single node How Timeseries compares to Prometheus, Thanos, Mimir, and managed observability View the source code, open issues, and contribute # Timeseries Stateless Ingest Source: https://opendata.dev/docs/timeseries/ingest Durable OTLP ingest for Timeseries via an object-storage queue Timeseries can accept OpenTelemetry metrics through a durable queue in object storage instead of (or in addition to) the direct OTLP/HTTP endpoint. An [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) with the `opendata` exporter writes batches into the queue, and a background consumer inside the Timeseries server drains them into the TSDB. This is the [stateless zonal ingestion](/docs/overview/architecture#stateless-zonal-ingestion) pattern enabled by the [`opendata` Buffer](/docs/buffer/index) library applied to metrics. ## Why use it The direct OTLP/HTTP endpoint couples ingest availability to TSDB availability. If the server is down or crashes before it has flushed an accepted request, the metrics are lost. Routing writes through object storage changes that: * Producers keep writing even when the TSDB is unavailable. * A crashed consumer resumes from the last acked batch. * Traffic stays within the AZ, avoiding cross-zone transfer fees that add up at high metric volumes. The tradeoff is end-to-end latency: a metric is not queryable until the collector flushes its batch and the consumer reads, decodes, and writes it. For monitoring workloads this is acceptable; for sub-second freshness, use the direct OTLP/HTTP endpoint. ## Architecture Both paths converge on the same OTLP-to-Prometheus conversion and TSDB write, so semantics match exactly. You can run both at once if some pipelines need immediate writes and others tolerate queue latency. ## Collector side The `opendata` exporter lives in the [`opendata-go`](https://github.com/opendata-oss/opendata-go) repository and is distributed as a standalone OTel Collector component. Build a custom collector with the [OpenTelemetry Collector Builder](https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder). ### Builder config ```yaml builder-config.yaml theme={null} dist: name: opendata-otelcol description: OpenTelemetry Collector with OpenData exporter support output_path: ./_build otelcol_version: 0.149.0 receivers: - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.149.0 - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.149.0 processors: - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.149.0 exporters: - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.149.0 - gomod: github.com/opendata-oss/opendata-go/exporter/opendataexporter v0.3.0 ``` Build and run: ```bash theme={null} go install go.opentelemetry.io/collector/cmd/builder@v0.149.0 builder --config builder-config.yaml ./_build/opendata-otelcol --config collector-config.yaml ``` Or build a container image from the same builder config: ```dockerfile Dockerfile theme={null} FROM golang:1.26.1 AS builder ARG OCB_VERSION=0.149.0 WORKDIR /src RUN go install go.opentelemetry.io/collector/cmd/builder@v${OCB_VERSION} COPY builder-config.yaml /src/builder-config.yaml RUN builder --config /src/builder-config.yaml FROM gcr.io/distroless/base-debian12 COPY --from=builder /src/_build/opendata-otelcol /otelcol-custom ENTRYPOINT ["/otelcol-custom"] ``` ### Exporter config ```yaml collector-config.yaml theme={null} exporters: opendata: object_store: type: s3 bucket: my-ingest-bucket region: us-west-2 data_path_prefix: ingest/otel/metrics/data manifest_path: ingest/otel/metrics/manifest flush_interval: 10s flush_size_bytes: 1048576 compression: zstd service: pipelines: metrics: receivers: [otlp] processors: [batch] exporters: [opendata] ``` | Field | Description | | ------------------ | -------------------------------------------------------------------------------------------- | | `object_store` | Bucket where batches and the manifest are written. Must match the consumer's `object_store`. | | `data_path_prefix` | Path prefix for batch objects. | | `manifest_path` | Path to the queue manifest. The consumer reads the same path. | | `flush_interval` | Maximum time a batch is held before flushing. | | `flush_size_bytes` | Size threshold that triggers a flush. | | `compression` | `none` or `zstd`. Zstd uses level 3. | Each `ConsumeMetrics` call marshals the OTLP protobuf, writes it as one entry with a 4-byte metadata header identifying it as metrics, and awaits durable confirmation from object storage before returning to the pipeline. That means a batch processor upstream is the right place to trade off request rate against flush rate. ## Consumer side Turn on the consumer by adding `buffer_consumer` to `prometheus.yaml`: ```yaml prometheus.yaml theme={null} buffer_consumer: object_store: type: S3 region: us-west-2 bucket: my-ingest-bucket manifest_path: ingest/otel/metrics/manifest poll_interval: 1s ``` The consumer runs as a background task inside the Timeseries server. It requires read-write mode and only starts when this section is present. The object store does not have to be the same bucket the TSDB uses; the queue is a separate buffer. | Field | Default | Description | | --------------- | ----------------- | -------------------------------------------- | | `object_store` | required | Bucket holding the queue. | | `manifest_path` | `ingest/manifest` | Must match the exporter's `manifest_path`. | | `poll_interval` | `1s` | Delay between polls when the queue is empty. | On startup the consumer fences any previous consumer via the manifest's epoch-based compare-and-set, then begins polling. On shutdown it flushes pending acks before releasing the object store. ## Batch format Batches are self-describing. Each file contains a record block (optionally compressed) followed by a 7-byte footer that indicates the compression type, record count, and format version. The consumer reads the footer, decompresses the record block if needed, and parses the length-prefixed entries. See [RFC 0001](https://github.com/opendata-oss/opendata/blob/main/buffer/rfcs/0001-stateless-buffer.md) for the wire format and [RFC 0006](https://github.com/opendata-oss/opendata/blob/main/timeseries/rfcs/0006-buffer-consumer.md) for the 4-byte metadata header that tells the consumer a batch holds OTLP metrics. ## Delivery semantics At-least-once. If the consumer crashes between processing a batch and acking it, the batch is re-read on restart. Duplicate writes to the TSDB are idempotent: samples are keyed by `(series, timestamp)`, so a replay overwrites with the same value. ## Observability The consumer publishes metrics under the `buffer_` prefix on the Timeseries `/metrics` endpoint. | Metric | Type | Description | | ----------------------------------- | --------- | --------------------------------------------------- | | `buffer_batches_collected` | counter | Batches fetched from object store. | | `buffer_entries_collected` | counter | Entries across collected batches. | | `buffer_bytes_collected` | counter | Bytes read from object store. | | `buffer_acks` | counter | Batch acks processed. | | `buffer_consumer_lag_seconds` | gauge | Wall clock minus last batch ingestion time. | | `buffer_queue_length` | gauge | Entries currently in the manifest queue. | | `buffer_fetch_duration_seconds` | histogram | Per-batch fetch latency from object store. | | `buffer_gc_files_deleted` | counter | Batch files cleaned by GC. | | `buffer_gc_files_failed` | counter | Failed GC file deletions. | | `buffer_gc_duration_seconds` | histogram | GC cycle wall time. | | `buffer_manifest_writes` | counter | Manifest write attempts (label `role`). | | `buffer_manifest_conflicts` | counter | Manifest CAS conflicts (label `role`). | | `tsdb_ingest_entries_skipped_total` | counter | Entries skipped due to decode or conversion errors. | A growing `buffer_queue_length` without `buffer_acks` keeping up means the consumer is falling behind. Check TSDB write latency and consider reducing `flush_interval` on the exporter side (smaller, more frequent batches reduce per-batch variance) or raising the consumer's CPU budget. ## Running both paths `buffer_consumer` and the OTLP/HTTP endpoint coexist. A common setup is: * Route high-volume, lossy-tolerant OTel traffic through the queue. * Keep the direct endpoint for low-latency writes (scrapers, remote-write senders, local agents). Both paths use the same converter and write through `TimeSeriesDb::write`, so there is no semantic difference between a metric that arrived via the queue and one that arrived over HTTP. # Timeseries: Getting to Production Source: https://opendata.dev/docs/timeseries/production Deploy, monitor, and secure Timeseries on Kubernetes with S3 storage This guide covers everything you need to run Timeseries in production on Kubernetes: a complete Helm chart, S3-backed storage with a local disk cache, health checks, monitoring, and security. ## Deployment ### Overview Since we haven't yet built partitioning into Timeseries, a production Timeseries deployment consists of a single replica only. The primary means of scaling would be scaling up, which can take you pretty far. Since all data is persisted on S3, data in a single node Timeseries is highly durable. So, a production deployment of Timeseries consists of: * A **single-replica Deployment** running the `opendata-timeseries` container * An **S3 bucket** for durable data storage * A **PersistentVolumeClaim** backed by a fast SSD for the SlateDB disk cache * A **ConfigMap** for the Prometheus-compatible scrape configuration, S3 storage settings, and SlateDB tuning * A **ServiceAccount** with an IAM role for S3 access (IRSA on EKS) Timeseries uses SlateDB's epoch-based fencing, which means only one writer can hold the epoch lock at a time. The Deployment uses the `Recreate` strategy so that the old pod is fully terminated before the new one starts — a `RollingUpdate` creates the possibility for the new pod to be fenced by the old one and never become ready. ### Helm chart Below is a complete Helm chart for deploying Timeseries to production. Create these files under `charts/opendata-timeseries/`. #### `values.yaml` ```yaml values.yaml theme={null} image: repository: ghcr.io/opendata-oss/timeseries tag: "0.3.0" port: 9090 # S3 storage configuration s3: bucket: my-timeseries-bucket region: us-west-2 prefix: timeseries # SlateDB disk cache — use a fast SSD-backed StorageClass cache: size: 100Gi storageClassName: gp3 maxCacheSizeBytes: 107374182400 # 100 GB # SlateDB tuning slatedb: defaultTtl: 604800000 # 7 days data retention maxUnflushedBytes: 134217728 # 128 MB l0SstSizeBytes: 16777216 # 16 MB maxSstSize: 67108864 # 64 MB resources: requests: cpu: 500m memory: 512Mi limits: cpu: "2" memory: 2Gi # IRSA role ARN for S3 access serviceAccount: roleArn: "" # Scrape configuration (prometheus.yaml format) scrapeConfig: | global: scrape_interval: 30s scrape_configs: - job_name: "timeseries-self" scrape_interval: 15s static_configs: - targets: ["localhost:9090"] ``` #### `templates/configmap.yaml` ```yaml templates/configmap.yaml theme={null} apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-config data: prometheus.yaml: | {{ .Values.scrapeConfig | nindent 4 }} storage: type: SlateDb path: {{ .Values.s3.prefix }} object_store: type: Aws region: {{ .Values.s3.region }} bucket: {{ .Values.s3.bucket }} settings_path: /slatedb-settings/slatedb.yaml slatedb.yaml: | default_ttl: {{ int .Values.slatedb.defaultTtl }} max_unflushed_bytes: {{ int .Values.slatedb.maxUnflushedBytes }} l0_sst_size_bytes: {{ int .Values.slatedb.l0SstSizeBytes }} compactor_options: max_concurrent_compactions: 2 max_sst_size: {{ int .Values.slatedb.maxSstSize }} garbage_collector_options: manifest_options: interval: '60s' min_age: '3600s' wal_options: interval: '60s' min_age: '60s' compacted_options: interval: '60s' min_age: '3600s' compactions_options: interval: '60s' min_age: '3600s' object_store_cache_options: root_folder: /cache max_cache_size_bytes: {{ int .Values.cache.maxCacheSizeBytes }} ``` #### `templates/serviceaccount.yaml` ```yaml templates/serviceaccount.yaml theme={null} apiVersion: v1 kind: ServiceAccount metadata: name: {{ .Release.Name }} annotations: eks.amazonaws.com/role-arn: {{ .Values.serviceAccount.roleArn }} ``` #### `templates/pvc.yaml` ```yaml templates/pvc.yaml theme={null} apiVersion: v1 kind: PersistentVolumeClaim metadata: name: {{ .Release.Name }}-cache spec: accessModes: - ReadWriteOnce storageClassName: {{ .Values.cache.storageClassName }} resources: requests: storage: {{ .Values.cache.size }} ``` #### `templates/deployment.yaml` ```yaml templates/deployment.yaml theme={null} apiVersion: apps/v1 kind: Deployment metadata: name: {{ .Release.Name }} spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: {{ .Release.Name }} template: metadata: labels: app: {{ .Release.Name }} spec: serviceAccountName: {{ .Release.Name }} terminationGracePeriodSeconds: 60 securityContext: runAsNonRoot: true runAsUser: 1000 runAsGroup: 1000 fsGroup: 1000 containers: - name: timeseries image: {{ .Values.image.repository }}:{{ .Values.image.tag }} args: - "--config" - "/config/prometheus.yaml" - "--port" - "{{ .Values.port }}" ports: - containerPort: {{ .Values.port }} name: http env: - name: RUST_LOG value: info resources: {{- toYaml .Values.resources | nindent 12 }} livenessProbe: httpGet: path: /-/healthy port: http initialDelaySeconds: 10 periodSeconds: 30 readinessProbe: httpGet: path: /-/ready port: http initialDelaySeconds: 5 periodSeconds: 10 volumeMounts: - name: config mountPath: /config readOnly: true - name: slatedb-settings mountPath: /slatedb-settings readOnly: true - name: cache mountPath: /cache volumes: - name: config configMap: name: {{ .Release.Name }}-config items: - key: prometheus.yaml path: prometheus.yaml - name: slatedb-settings configMap: name: {{ .Release.Name }}-config items: - key: slatedb.yaml path: slatedb.yaml - name: cache persistentVolumeClaim: claimName: {{ .Release.Name }}-cache ``` #### `templates/service.yaml` ```yaml templates/service.yaml theme={null} apiVersion: v1 kind: Service metadata: name: {{ .Release.Name }} spec: selector: app: {{ .Release.Name }} ports: - port: {{ .Values.port }} targetPort: http name: http ``` #### Install the chart ```bash theme={null} helm install opendata-timeseries ./charts/opendata-timeseries \ --set s3.bucket=my-timeseries-bucket \ --set s3.region=us-west-2 \ --set serviceAccount.roleArn=arn:aws:iam::123456789012:role/opendata-timeseries ``` ### Disk cache SlateDB caches frequently accessed data on local disk to avoid repeated reads from S3. For production workloads, use an SSD-backed StorageClass: * **EKS**: Use `gp3` (General Purpose SSD) or `io2` for higher IOPS. For maximum performance, use instance-store NVMe volumes with a [local-static-provisioner](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner). * Size the cache based on your active working set. The default of **100 Gi** is a good starting point; increase if you see frequent cache evictions in the `slatedb_*` metrics. Avoid using HDD-backed volumes (e.g. `st1`, `sc1`) for the cache. SlateDB issues many small random reads, and spinning disks will bottleneck performance. ### Block cache On top of SlateDB's disk cache, Timeseries can keep decoded blocks in a hybrid memory-plus-disk block cache backed by [foyer](https://foyer.rs). Add it under `storage` in `prometheus.yaml`: ```yaml prometheus.yaml theme={null} storage: type: SlateDb path: timeseries object_store: type: Aws region: us-west-2 bucket: my-timeseries-bucket block_cache: type: FoyerHybrid memory_capacity: 8589934592 # 8 GiB disk_capacity: 107374182400 # 100 GiB disk_path: /cache/foyer write_policy: WriteOnInsertion flushers: 4 ``` Sizing guidance: * Set `memory_capacity` to roughly the recent working set (the last few hours of bucket index and sample data). * Point `disk_path` at the same SSD-backed PVC used by the disk cache; the two workloads coexist. * Keep `write_policy: WriteOnInsertion` so every cached block is also on disk. Restarts then hit the disk tier instead of re-reading from S3. * Raise `flushers` if the `foyer_*` write-queue metrics show backpressure. See [``](/docs/timeseries/configuration#block_cache_config) for the full field reference. ### Cache warmer On startup the server scans recent time bucket key ranges through the storage reader, which populates the block cache. Queries that hit in the first few seconds after a restart avoid a cold-cache penalty. This is on by default and covers the last 24 hours including sample data. To tune or disable it, see [``](/docs/timeseries/configuration#cache_warmer_config). ### Durable OTLP ingest For high-volume OTel metrics, run the [stateless ingest](/docs/timeseries/ingest) path instead of (or alongside) direct OTLP/HTTP writes. Producers keep writing during TSDB restarts, writes stay inside the AZ, and a crashed consumer resumes from the last acked batch on its own. ### Health checks Timeseries exposes two health-check endpoints: | Endpoint | Type | Behavior | | ------------ | --------- | ------------------------------------------------------------------- | | `/-/healthy` | Liveness | Returns 200 if the process is running | | `/-/ready` | Readiness | Returns 200 once the TSDB is initialized and ready to serve queries | Both probes are included in the Helm chart's Deployment template above. ### Graceful shutdown Timeseries handles `SIGTERM` and `SIGINT` signals gracefully: 1. Stops accepting new connections 2. Drains in-flight requests 3. Flushes TSDB data from memory to durable storage 4. Exits cleanly The Helm chart sets `terminationGracePeriodSeconds: 60` to give the server enough time to complete the flush before Kubernetes force-kills the pod. ## Monitoring All metrics are exposed at `/metrics` in Prometheus text format. Since Timeseries is itself a Prometheus-compatible data source, you can configure it to scrape its own metrics endpoint (included in the default `scrapeConfig` above). ### Key metrics | Metric | Type | Labels | Description | | ------------------------------------- | --------- | ------------------------------ | ------------------------------------------------ | | `scrape_samples_scraped` | counter | `job`, `instance` | Number of samples scraped per target | | `scrape_samples_failed` | counter | `job`, `instance` | Number of samples that failed validation | | `remote_write_samples_ingested_total` | counter | — | Total samples ingested via remote write | | `remote_write_samples_failed_total` | counter | — | Total samples that failed remote write ingestion | | `http_requests_total` | counter | `method`, `endpoint`, `status` | Total HTTP requests handled | | `http_request_duration_seconds` | histogram | `method`, `endpoint` | Request latency distribution | | `http_requests_in_flight` | gauge | — | Number of HTTP requests currently being served | Timeseries also exposes `slatedb_*` metrics from the underlying SlateDB storage engine. These are useful for debugging storage-level performance and compaction behavior. ### Example PromQL queries ```promql theme={null} # Request rate (requests per second over 5 minutes) rate(http_requests_total[5m]) # Error rate (5xx responses) rate(http_requests_total{status=~"5.."}[5m]) # p99 request latency histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) # In-flight requests http_requests_in_flight # Sample ingestion rate (remote write) rate(remote_write_samples_ingested_total[5m]) # Scrape sample throughput rate(scrape_samples_scraped[5m]) ``` ## Security ### TLS and authentication Timeseries does not include built-in TLS termination or authentication. Place a reverse proxy (nginx, Envoy, or a cloud load balancer) in front of Timeseries to handle TLS and access control. ### Object storage security The Helm chart uses [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (IAM Roles for Service Accounts) so that the pod receives temporary AWS credentials automatically — no static access keys required. Create an IAM role with the following policy and attach it to the ServiceAccount via the `serviceAccount.roleArn` value: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::my-timeseries-bucket", "arn:aws:s3:::my-timeseries-bucket/*" ] } ] } ``` The IAM role's trust policy should scope access to your EKS cluster's OIDC provider and the specific service account: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/EXAMPLE" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.us-west-2.amazonaws.com/id/EXAMPLE:aud": "sts.amazonaws.com", "oidc.eks.us-west-2.amazonaws.com/id/EXAMPLE:sub": "system:serviceaccount:default:opendata-timeseries" } } } ] } ``` Additional recommendations: * Enable **encryption at rest** on the S3 bucket (SSE-S3 or SSE-KMS). * Use a **VPC endpoint** for S3 to keep traffic off the public internet. * Block all public access on the bucket. * Add a **lifecycle rule** to transition old data to Intelligent-Tiering after 30 days and abort incomplete multipart uploads after 7 days. # Timeseries Quickstart Source: https://opendata.dev/docs/timeseries/quickstart Install Timeseries locally and query metrics in minutes Get a Prometheus-compatible metrics database running on your machine in under five minutes. You will install Timeseries, configure it to scrape its own metrics, query those metrics in the built-in UI, and then add an external target. ## Install OpenData Timeseries Download and install the Timeseries binary: ```bash theme={null} curl -fsSL https://www.opendata.dev/install.sh | sh -s -- timeseries ``` This places the `opendata-timeseries` binary in the current directory. ## Configure Timeseries Create a file called `prometheus.yaml` with the following contents: ```yaml prometheus.yaml theme={null} global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: "timeseries-server" scrape_interval: 5s scrape_timeout: 4s metrics_path: /metrics scheme: http static_configs: - targets: ["localhost:9090"] storage: type: SlateDb path: data object_store: type: Local path: ./data flush_interval_secs: 30 ``` This configuration has three sections: * **global** sets the default scrape and evaluation intervals for all jobs. * **scrape\_configs** tells Timeseries what to scrape. For now, it scrapes itself at `localhost:9090` every 5 seconds. * **storage** configures local [SlateDB](https://slatedb.io) as the storage backend. Metrics are flushed to disk every 30 seconds under `./data`. ## Start the server Run the binary with the configuration file you just created: ```bash theme={null} ./opendata-timeseries --config prometheus.yaml --port 9090 ``` Timeseries starts, begins scraping its own `/metrics` endpoint, and stores the results locally. ## Query metrics in the UI Open [http://localhost:9090](http://localhost:9090) in your browser. Timeseries ships with a built-in query UI where you can run PromQL expressions and visualize results as tables or graphs. ### Check that the target is up Type `up` into the expression bar and click **Execute**. Switch to the **Graph** tab to see the result over time. A steady line at `1` means the target is healthy and being scraped successfully. Query UI showing up metric graphed over time ### Explore more metrics Try these expressions in the query bar to explore the metrics Timeseries collects about itself: * `http_requests` — total HTTP requests broken down by method, endpoint, and status * `http_request_duration_seconds` — request latency histogram by endpoint * `scrape_samples_scraped` — how many samples were collected per scrape Use the **Table** tab for instant values or the **Graph** tab to see how metrics change over time. The **Duration** and **Step** controls above the graph let you adjust the time window and resolution. ## Scrape a target — Node Exporter Scraping its own metrics is useful for verification, but the real value comes from scraping external targets. [Node Exporter](https://prometheus.io/docs/guides/node-exporter/) exposes hardware and OS metrics and is a good first target. ### Download and run Node Exporter Download the latest release for your platform from the [Prometheus downloads page](https://prometheus.io/download/#node_exporter), extract it, and start it: ```bash theme={null} tar xvfz node_exporter-*.tar.gz && rm node_exporter-*.tar.gz cd node_exporter-* # on macOS you might need the next line to avoid macOS blocking the execution of the binary xattr -d com.apple.quarantine ./node_exporter ./node_exporter ``` Verify that metrics are being exposed: ```bash theme={null} curl localhost:9100/metrics ``` You should see a large set of metrics prefixed with `node_`. ### Add Node Exporter as a scrape target Update `prometheus.yaml` to include a new job under `scrape_configs`: ```yaml prometheus.yaml theme={null} scrape_configs: - job_name: "timeseries-server" scrape_interval: 5s scrape_timeout: 4s metrics_path: /metrics scheme: http static_configs: - targets: ["localhost:9090"] - job_name: "node-exporter" static_configs: - targets: ["localhost:9100"] ``` Restart the Timeseries server to pick up the new configuration: ```bash theme={null} ./opendata-timeseries --config prometheus.yaml --port 9090 ``` ### Query Node Exporter metrics After a few scrape intervals, head back to [http://localhost:9090](http://localhost:9090) and try these expressions: * `node_memory_free_bytes` — free system memory in bytes * `node_memory_total_bytes` — total system memory in bytes * `up` — now returns two results, one for each scrape target * `sum(up)` — total number of healthy targets ## Next steps * Learn about all available settings in [Configuration](/docs/timeseries/configuration). * Explore the full query and write API in the API Reference section in the sidebar. * Connect [Grafana](https://grafana.com) as a visualization layer — Timeseries is compatible with the Prometheus data source out of the box. # Timeseries Storage Design Source: https://opendata.dev/docs/timeseries/storage-design How Timeseries organizes data in time buckets on SlateDB Timeseries stores all data as key-value records in SlateDB. Data is organized into time buckets with inverted indexes for efficient label-based querying, a forward index for resolving series metadata, and Gorilla-compressed storage for time series samples. This page covers the conceptual storage model. For exact byte-level encoding schemas, see the [storage RFC on GitHub](https://github.com/opendata-oss/opendata/blob/main/timeseries/rfcs/0001-tsdb-storage.md). ## Storage components Timeseries has three storage components that work together to serve queries: | Component | Purpose | | ---------------------- | ----------------------------------------------------------------- | | **Raw series storage** | Stores `(timestamp, value)` pairs keyed by series ID | | **Inverted index** | Maps label/value pairs to series IDs for efficient label matching | | **Forward index** | Maps series IDs back to their full label sets | ### How a query uses the storage components To illustrate how these components interact, consider the query `sum by (instance) (metric{path="/query", method="GET", status="200"})`: 1. Parse the query into a query plan. 2. Look up each label selector in the inverted index (`__name__="metric"`, `path="/query"`, `method="GET"`, `status="200"`) to find matching series IDs. Intersect the results. 3. Resolve each matched series ID to its full label set in the forward index (including `instance`, needed for the `by (instance)` grouping). 4. Read the `(timestamp, value)` pairs for each matched series in the raw series storage matched series, aggregate, and return the result. ## Time buckets Data is divided into time buckets, where each bucket holds all data received within a specific window of time. The bucket boundary is encoded directly into every record key, which means SlateDB physically clusters records from the same time window together on disk. This design provides two key benefits: * **Efficient range scans**: queries that target recent data only scan the relevant buckets without reading historical data. * **Scoped cardinality**: series IDs are local to a time bucket, so cardinality is bounded by the number of series in each bucket rather than growing unboundedly over time. The initial implementation uses **1-hour buckets**. As buckets age, they will eventually be rolled up into coarser granularities (e.g. daily or weekly) through compaction, replacing the original fine-grained buckets. ## Record types All records share a common key prefix that encodes a version byte and a record tag. The record tag identifies the record type and, for bucket-scoped records, the time granularity. This allows different record types and bucket sizes to coexist in the same keyspace while maintaining clean sort ordering. | Record type | Scope | Description | | -------------------- | ------ | ------------------------------------------------------------------------------------------------------------ | | **BucketList** | Global | Enumerates the time buckets that contain data, including each bucket's granularity | | **SeriesDictionary** | Bucket | Maps label-set fingerprints to series IDs, used during ingestion to assign or look up IDs | | **ForwardIndex** | Bucket | Stores the full label set for each series ID, used during query execution to resolve labels | | **InvertedIndex** | Bucket | Maps each label/value pair to a [RoaringBitmap](https://roaringbitmap.org/) of series IDs (the posting list) | | **TimeSeries** | Bucket | Holds the Gorilla-compressed `(timestamp, value)` stream for each series | ### Inverted index The inverted index is the primary mechanism for label-based queries. For every label/value pair on a series, the index stores a posting list of all series IDs that share that pair. For example, if series 713 has labels `job="api"` and `status="500"`, then both posting lists include series 713. Posting lists are stored as [RoaringBitmaps](https://roaringbitmap.org/), which provide efficient compression and fast set operations (intersection, union) for combining multiple label selectors. The metric name (`__name__`) is treated as a regular label rather than a first-class key prefix. This gives the query planner flexibility to choose the most selective label to scan first. For example, sometimes filtering by `cluster="prod"` is more efficient than filtering by metric name. ### Forward index The forward index maps each series ID back to its canonical label set. After the inverted index identifies matching series, the forward index resolves each one to its full set of labels. This is necessary for operations like `by` and `without` grouping, which need labels that weren't part of the original selector. The forward index also stores metric metadata: the metric type (gauge, sum, histogram, exponential histogram, summary), temporality, and the monotonic flag. ### Time series storage and compression Raw time series data is stored as Gorilla-compressed streams of `(timestamp, value)` pairs. [Gorilla compression](https://www.vldb.org/pvldb/vol8/p1816-teller.pdf) exploits the temporal locality of time series data to achieve high compression ratios. This works by encoding timestamps using delta-of-delta encoding and values using XOR compression. ## Metric type handling Following the Prometheus approach, all OpenTelemetry metric types are normalized to `f64` values: * **Gauges and counters** are stored directly as `(timestamp, value)` pairs. * **Histograms** are decomposed into multiple series using Prometheus naming conventions: * `metric_bucket{le=""}` for each bucket boundary (cumulative counts, with a final `le="+Inf"` bucket) * `metric_sum` for the sum of all observations * `metric_count` for the total number of observations Delta histograms from OpenTelemetry are accumulated into cumulative form so the resulting series are monotonically increasing, matching Prometheus semantics. This mapping mirrors the [OTLP-to-Prometheus translation](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#otlp-metric-points-to-prometheus) used by the OpenTelemetry Collector, ensuring compatibility with Grafana, PromQL, and other Prometheus-native tooling. # Vector Benchmarks Source: https://opendata.dev/docs/vector/benchmarks Query latency, ingest throughput, and cost for OpenData Vector across standard ANN datasets These are the benchmark results for OpenData Vector were measured on a single `c6id.4xlarge` node against standard Approximate Nearest Neighbor (ANN) datasets. You can reproduce them, or run the same measurements against your own data, with [Vector Bench](#run-it-on-your-own-data). All latencies were recorded at a steady-state query throughput of **32 queries/s at 90% recall** (SIFT 1M was recorded at 97% recall). Reported ingestion throughput is the steady-state throughput for each dataset in isolation. ## Query latency Here is the query latencies for different standard vector data sets, split by cold and warm latency. Query latency and ingest throughput across ANN datasets | Dataset class | Warm latency | Cold latency (P90) | | --------------------------------- | ------------------- | ------------------ | | Small (Sift1M, Sift10M, Cohere1M) | low single-digit ms | ≤ \~1 s | | Large | low teens of ms | ≤ \~1 s | For cold queries, tail latency is dominated by the slowest time to fetch a posting list from S3. For warm queries against small datasets, all postings sit in the in-memory cache, so latency reflects how fast Vector loads postings and scores candidates. Larger datasets score more postings to reach high recall, and some of those reads come from local disk cache, which adds delay. ## Ingest throughput Vector sustains between **\~1K and \~12K vector writes/second**, depending on dataset size and vector dimensions. Re-ingesting the full SIFT 100M dataset took \~6 hours at a stable rate without significant stalls, demonstrating that the LIRE compaction protocol ingests incrementally. Ingest throughput holding stable over the Sift100M re-ingestion run Ingestion is bottlenecked on (1) traversing the centroid index to assign vectors to postings and (2) computing new clusters when centroids split during LIRE. More vectors means more centroids to search while more dimensions makes distance computation more expensive. ## Cost Vector serves 100M vectors for roughly **\$346/month of compute** (1× `c6id.4xlarge` on a 1-year reservation): | Component | Driver | Monthly | | --------------- | ----------------------------------------------------------------- | --------------- | | Compute | 1× `c6id.4xlarge` (1-yr reservation) | \$346 | | S3 PUT requests | \~1 SlateDB write/sec | tens of dollars | | S3 storage | SIFT 100M ≈ 200–500 GB after attributes + LSM space amplification | \~\$5–12 | S3 charges sit outside the compute figure but stay small: Vector batches writes into roughly one SlateDB write per second. ## Run it on your own data The benchmark harness is open source: [`vector/bench/`](https://github.com/opendata-oss/opendata/tree/main/vector/bench) in the OpenData repository. Vector Bench runs each dataset through three independent phases, since Vector's design fully decouples read and write workloads so you can capacity-plan them separately: | Phase | What it measures | | -------- | --------------------------------------------------------------------------------------------------------------------- | | `INGEST` | ingestion throughput as a 5-minute trailing window (sampled every minute) | | `WARM` | recall\@10, QPS, and p50/p90/p99 latency after a warmup pass, under a rate-limited concurrent query workload | | `COLD` | query performance with a freshly-allocated memory-only block cache, so every SST block read comes from object storage | ### Included datasets | Dataset | Vectors | Dimensions | Source | | ----------------------------------------- | ---------- | ---------- | ------------------------- | | SIFT/BIGANN (100K, 1M, 10M, 100M subsets) | up to 100M | 128 | image feature embeddings | | Cohere1M | 1M | 768 | Wikipedia text embeddings | | Cohere10M | 10M | 1024 | Wikipedia text embeddings | You can also point the benchmark at your own data in `fvecs`, `bvecs`, or parquet format. The simplest way to start is the 100K-vector SIFT subset that ships in the repo: ```bash theme={null} cargo run -p vector-bench --release -- --config vector/bench/sift100k.toml ``` That exercises all three phases against a SlateDB pointed at a local object store and takes \~20 seconds on a laptop. See the [README](https://github.com/opendata-oss/opendata/blob/main/vector/bench/README.md) for configuring larger runs against a real object store, and for the bundled Claude Skill that infers a config from your raw data. # When to choose Vector Source: https://opendata.dev/docs/vector/comparisons How OpenData Vector compares to turbopuffer, Pinecone, Qdrant, Milvus, Weaviate, Chroma, LanceDB, and pgvector *We build OpenData and this is our best good-faith comparison. Corrections are [welcome](https://github.com/opendata-oss/opendata/issues).* Vector's architecture is heavily inspired by [turbopuffer](https://www.turbopuffer.com/): object storage is the only source of truth, query serving nodes are stateless caches, and the index is designed around object storage's latency profile. Vector is that architecture as MIT-licensed software running in your own account. ## Choose Vector if you care about… **…paying proportional to your hot data, not your full dataset.** Cold vectors sit on S3 at [\~\$0.02/GB-month](https://aws.amazon.com/s3/pricing/), and compute is sized for what's actually being queried. In [our benchmarks](/docs/vector/benchmarks), one \$346/mo node serves 100M vectors. If most of your vectors are cold most of the time (per-tenant and per-user indexes are the classic case), the difference is hundreds versus thousands of dollars per month. **…a durable deployment that's one process and a bucket.** Every byte of data and metadata lives in an object store bucket, so there is no replication to manage and you can kill every compute node without losing anything. There are no supporting services to run: scaling reads means adding stateless replica pods that don't communicate with the writer or each other. **…an online database.** Concurrent clients upsert and query a single incrementally maintained index, rather than a serving layer you assemble yourself around a file format. **…owning your search infrastructure.** MIT licensed, running in your bucket and your VPC. There is no per-query pricing, data doesn't leave your account, and no vendor can reprice or discontinue the software. ## How the alternatives stack up ✓ = yes · \~ = partially · ✗ = no | You care about… | **Vector** | turbopuffer | Pinecone | Qdrant / Weaviate | Milvus | Chroma (distributed) | LanceDB (OSS) | pgvector | | ------------------------------------------ | ---------- | ----------- | ----------- | ------------------ | ------------ | -------------------- | ------------- | ------------ | | Cost proportional to hot data | ✓ | ✓ | \~ ¹ | ✗ | \~ ² | ✓ | ✓ | ✗ | | Durable deployment: one process + a bucket | ✓ | managed | managed | ✗ | \~ ² | \~ ³ | \~ ⁴ | ✗ | | Online database, consistent writes | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ ⁴ | ✓ | | License | MIT | Proprietary | Proprietary | Apache 2.0 / BSD ⁵ | Apache 2.0 ² | Apache 2.0 ³ | Apache 2.0 | PostgreSQL ⁵ | ¹ [Pinecone serverless](https://docs.pinecone.io/reference/architecture/serverless-architecture) tiers to object storage internally, but you buy it at managed-service prices rather than S3 prices in your own account. \ ² Milvus distributed uses object storage for durability but manages stateful shard assignment and etcd and a WAL/messaging layer. It's a cluster rather than a single process. \ ³ Distributed Chroma is the nearest open-source system: Apache 2.0 and genuinely object-storage-backed. Operationally it's a multi-service topology (frontends, query nodes, compactors, metadata) with collection-to-node routing, primarily offered as Chroma Cloud/BYOC. Single-node OSS Chroma is SQLite-backed rather than object-store-native. \ ⁴ [LanceDB OSS](https://github.com/lancedb/lancedb) is an embedded library over files. That works well on object storage for pipelines, but the online serving layer is its commercial cloud. \ ⁵ Open source, but durable state lives on replicated node disks: self-hosting means operating a stateful cluster, and cold data pays SSD prices. turbopuffer matches Vector on everything except the license. The open-source systems each lack either the stateless operations or the online database. Vector is the MIT-licensed system with both. ## The tradeoffs * **Latency:** warm queries run in low-single-digit to low-teens milliseconds (dataset-size dependent), cold queries up to \~1s at p90. This is fine for RAG and search, but not ideal for uniformly hot, sub-millisecond serving. * **Writes are batched:** durable acks can take \~1s, or \~100ms fronted by [Buffer](/docs/buffer) at the cost of read-your-writes. * **No single-query hybrid search:** Vector serves both ANN and BM25 full-text queries, but their scores aren't comparable, so a query scores by one or the other. To blend semantic and keyword relevance, run both and fuse the result lists client-side. *** *Last reviewed July 2026. Please [tell us](https://github.com/opendata-oss/opendata/issues) if a claim has gone stale.* # Vector Data Model Source: https://opendata.dev/docs/vector/data-model Collections, records, attributes, and vectors Each Vector database holds a single collection of records that share a common schema and vector properties. ## Collection A collection is defined by its vector properties and its attribute schema. The vector properties — **dimensions** and **distance metric** — are set at creation time and are immutable. | Property | Description | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | **Dimensions** | The number of dimensions for all vectors in the collection. All records must have a vector with exactly this many dimensions. | | **Distance metric** | The metric used to compute similarity between vectors. Either `l2` (Euclidean distance) or `dot_product`. | ## Records Each record in a collection has: * A **string ID** — a unique, user-provided identifier up to 64 bytes. * A set of **attributes** — typed key-value pairs defined by the collection schema. * A **vector** — a special attribute named `vector` that holds a dense vector of `f32` values with the collection's configured number of dimensions. ### Attributes Attributes are typed fields on a record. Each attribute in the schema has a name, a type, and a flag indicating whether it is **indexed**. | Type | Description | | ----------- | ----------------------------------------------------------- | | **String** | UTF-8 string | | **Int64** | 64-bit signed integer | | **Float64** | 64-bit floating point | | **Bool** | Boolean | | **Text** | UTF-8 text, tokenized and full-text indexed for BM25 search | #### Indexed vs. non-indexed attributes An attribute can be marked as **indexed** in the collection schema. Indexed attributes are maintained in an inverted index that maps attribute key-value pairs to the set of matching records. This enables efficient attribute-based filtering during queries — for example, filtering results where `category="shoes"`. Non-indexed attributes are stored with the record but cannot be used in filter predicates efficiently. Use non-indexed attributes for data that you want to retrieve but don't need to filter on. ### Text fields Attributes of type **Text** are always full-text indexed; the `indexed` flag does not apply to them. On write, Vector tokenizes the text and maintains a BM25 index over its terms. A query can then score records by BM25 relevance against a text field instead of by vector similarity, which makes Vector usable for keyword search as well as semantic search. The two scoring modes are separate per query. ### Vector The `vector` attribute is a reserved field that holds the record's dense embedding. It is a vector of `f32` values whose length must exactly match the collection's configured dimensions. This is the field used for similarity search — queries find the nearest records by comparing their vectors using the collection's distance metric. # Vector Source: https://opendata.dev/docs/vector/index ANN Vector and full-text search on object storage. Vector is an MIT-licensed, stateless search engine built on [SlateDB](https://slatedb.io) and object storage. It stores documents against a user-defined schema with dense vector and text fields, and serves both approximate nearest neighbor (ANN) search over the vectors and full-text (BM25) search over the text. Because all state lives in object storage, any node has full access to all data, a single replica is fully durable, and you can scale query throughput by adding stateless reader replicas. ## What it's good at * **Online ANN search from object storage.** Warm queries run in the single-digit to low-teens milliseconds while the index lives durably on object storage. * **Full-text search.** Text fields are tokenized and BM25-indexed on write, so the same engine serves keyword search alongside vector search, with the same filters and result format. Pair the two for hybrid relevance. * **Simple, durable deployment.** A single pod never loses data and fails over in seconds. Read replicas have no communication with the writer, so you scale reads without coordination. * **Stable incremental ingest.** A lazy adaptation of the LIRE protocol on top of SlateDB merges keeps ingest throughput steady as the index grows, with no expensive graph rebuilds. * **Flexible topologies.** Run embedded in your application, as a single node, as a writer with read replicas, or with buffered ingest through [OpenData Buffer](/docs/buffer). ## Why Vector Vector fills the gap between running `pgvector` and `pg_search` yourself and paying a vendor many multiples of hardware cost to operate a search database for you. A truly stateless architecture is what makes Vector cheaper and easier to operate than earlier vector databases. Any node can serve any data, so there are no shard assignments to manage and no rebalancing when nodes come and go. The index is an inverted file (IVF/SPANN) tuned for object storage: it batch-loads index data per round trip rather than making the sequential, hop-by-hop reads a graph index requires, so cold queries stay fast despite object-store latency. Reads and writes are fully decoupled, so you can capacity-plan ingest and query independently. See the [benchmarks](/docs/vector/benchmarks) for recall, warm and cold query latency, ingest throughput, and cost across standard datasets, along with a harness you can point at your own data. ## Tradeoffs * **Warm versus cold latency.** IVF scores more vectors than a graph index like HNSW, so warm queries are in the milliseconds rather than sub-millisecond. In exchange, cold queries stay sub-second where a cold HNSW graph can take dozens of seconds to load. * **Write latency.** Vector batches writes to amortize object-store PUTs and indexing work, so a write can take up to about a second to acknowledge. Putting Buffer in front cuts that to roughly a hundred milliseconds, without read-your-writes. * **Attribute filtering on vector queries.** Filters run after the ANN index yields candidates, so recall on heavily filtered vector queries can suffer. * **No single-query hybrid search.** ANN and BM25 scores are not comparable, so a query scores by one or the other. To blend semantic and keyword relevance, run both and fuse the result lists client-side. ## Explore Vector Install Vector and write your first documents in under five minutes Browse the full REST API for writing and searching vectors Understanding Vector's Data Model Vector configuration reference How Vector indexes documents to support efficient similarity search Deploy, monitor, and secure Vector for production workloads Query latency, ingest throughput, and cost across ANN datasets How Vector compares to turbopuffer, Pinecone, Qdrant, Milvus, and pgvector View the source code, open issues, and contribute # Vector: Getting to Production Source: https://opendata.dev/docs/vector/production Deploy, monitor, and secure Vector on Kubernetes with S3 storage This guide covers everything you need to run Vector in production on Kubernetes: a complete Helm chart, S3-backed storage with a local disk cache, health checks, monitoring, and security. ## Deployment ### Overview A basic production Vector deployment consists of a single replica that serves both writes and reads. Vector does not yet support partitioning, so you cannot scale the single writer/reader horizontally. The primary means of scaling is vertical scaling by allocating more cpu/memory/cache, which can take you pretty far. Since all data is persisted on S3, data in Vector is highly durable. So, a basic production deployment of Vector consists of: * A **single-replica Deployment** running the `opendata-vector` container * An **S3 bucket** for durable storage * A **PersistentVolumeClaim** for the SlateDB disk cache * A **ConfigMap** for Vector's configuration, S3 storage settings, and SlateDB settings. * A **ServiceAccount** with an IAM role for S3 access (IRSA on EKS) * A **Service** for exposing Vector to other applications Vector uses SlateDB's epoch-based fencing, which means only one writer can hold the epoch lock at a time. The Deployment uses the `Recreate` strategy so that the old pod is fully terminated before the new one starts — a `RollingUpdate` creates the possibility for the new pod to be fenced by the old one and never become ready. ### Helm Chart Below is a complete Helm chart for deploying Vector. Create these files under `charts/opendata-vector`. #### `values.yaml` ```yaml values.yaml theme={null} image: repository: ghcr.io/opendata-oss/vector tag: latest containerPort: 8080 command: [] args: [] resources: requests: cpu: "4" memory: 8Gi limits: cpu: "4" memory: 8Gi nodeSelector: {} tolerations: [] affinity: {} # --------------------------------------------------------------------------- # AWS / SlateDB object store configuration. # --------------------------------------------------------------------------- aws: region: us-west-2 bucket: my-vector-bucket prefix: vector # --------------------------------------------------------------------------- # Service account used by the vector pod. We rely on IRSA: the pod assumes the # IAM role specified in `serviceAccount.roleArn` via the EKS OIDC provider. # --------------------------------------------------------------------------- serviceAccount: create: true name: opendata-vector # ARN of the IAM role (REQUIRED) roleArn: my-iam-role-arn # --------------------------------------------------------------------------- # Disk cache for SlateDB (Foyer hybrid: memory + on-disk tier). # # The volume is mounted at `disk.mountPath` inside the container and Foyer is # pointed at it via the SlateDB `block_cache` config. # --------------------------------------------------------------------------- disk: size: 100Gi storageClass: gp3 mountPath: /var/cache/opendata-vector # Bytes used for the on-disk cache tier. Leave room for filesystem overhead. diskCapacityBytes: 96636764160 # 90 GiB # Bytes used for the in-memory cache tier. memoryCapacityBytes: 1073741824 # 4 GiB # --------------------------------------------------------------------------- # Service exposing the HTTP API. Defaults to ClusterIP — switch to LoadBalancer # if you want an external endpoint. # --------------------------------------------------------------------------- service: type: ClusterIP port: 8080 annotations: {} # --------------------------------------------------------------------------- # Vector configuration. These fields map onto the vector.yaml that is # rendered into the configmap and mounted at /config/vector.yaml. # --------------------------------------------------------------------------- vector: dimensions: 2 distanceMetric: L2 flushInterval: 60 splitThresholdVectors: 150 mergeThresholdVectors: 50 splitSearchNeighbourhood: 0 metadataFields: - name: label field_type: String indexed: true # Optional SlateDB toml settings. If non-empty the contents are written to the # configmap and SlateDB is pointed at it via `storage.settings_path`. slatedb: settings: | # SlateDB settings overrides go here. Leave empty to use SlateDB defaults. # Extra environment variables passed to the vector container. extraEnv: - name: RUST_LOG value: info ``` #### `templates/configmap.yaml` ```yaml templates/configmap.yaml theme={null} apiVersion: v1 kind: ConfigMap metadata: name: {{ include "vector.fullname" . }}-config labels: {{- include "vector.labels" . | nindent 4 }} data: vector.yaml: | storage: type: SlateDb path: {{ .Values.aws.prefix }} object_store: type: Aws region: {{ .Values.aws.region }} bucket: {{ .Values.aws.bucket }} {{- if .Values.slatedb.settings }} settings_path: /config/slatedb.toml {{- end }} block_cache: type: FoyerHybrid memory_capacity: {{ printf "%.0f" (.Values.disk.memoryCapacityBytes | float64) }} disk_capacity: {{ printf "%.0f" (.Values.disk.diskCapacityBytes | float64) }} disk_path: {{ .Values.disk.mountPath }}/block-cache dimensions: {{ .Values.vector.dimensions }} distance_metric: {{ .Values.vector.distanceMetric }} flush_interval: {{ .Values.vector.flushInterval }} split_threshold_vectors: {{ .Values.vector.splitThresholdVectors }} merge_threshold_vectors: {{ .Values.vector.mergeThresholdVectors }} split_search_neighbourhood: {{ .Values.vector.splitSearchNeighbourhood }} metadata_fields: {{- toYaml .Values.vector.metadataFields | nindent 6 }} {{- if .Values.slatedb.settings }} slatedb.toml: | {{ .Values.slatedb.settings | indent 4 }} {{- end }} ``` #### `templates/serviceaccount.yaml` ```yaml templates/serviceaccount.yaml theme={null} {{- if .Values.serviceAccount.create -}} {{- if not .Values.serviceAccount.roleArn -}} {{- fail "serviceAccount.roleArn must be set" -}} {{- end -}} apiVersion: v1 kind: ServiceAccount metadata: name: {{ include "vector.serviceAccountName" . }} labels: {{- include "vector.labels" . | nindent 4 }} annotations: eks.amazonaws.com/role-arn: {{ .Values.serviceAccount.roleArn | quote }} {{- end }} ``` #### `templates/pvc.yaml` ```yaml templates/pvc.yaml theme={null} apiVersion: v1 kind: PersistentVolumeClaim metadata: name: {{ include "vector.fullname" . }}-cache labels: {{- include "vector.labels" . | nindent 4 }} spec: accessModes: - ReadWriteOnce {{- if .Values.disk.storageClass }} storageClassName: {{ .Values.disk.storageClass }} {{- end }} resources: requests: storage: {{ .Values.disk.size }} ``` #### `templates/deployment.yaml` ```yaml templates/deployment.yaml theme={null} apiVersion: apps/v1 kind: Deployment metadata: name: {{ include "vector.fullname" . }} labels: {{- include "vector.labels" . | nindent 4 }} spec: replicas: 1 strategy: type: Recreate selector: matchLabels: {{- include "vector.selectorLabels" . | nindent 6 }} template: metadata: labels: {{- include "vector.selectorLabels" . | nindent 8 }} annotations: # Trigger a rolling restart when the rendered config changes. checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} spec: serviceAccountName: {{ include "vector.serviceAccountName" . }} securityContext: fsGroup: 1000 containers: - name: vector image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" imagePullPolicy: {{ .Values.image.pullPolicy }} {{- with .Values.command }} command: {{- toYaml . | nindent 12 }} {{- end }} {{- with .Values.args }} args: {{- toYaml . | nindent 12 }} {{- end }} ports: - name: http containerPort: {{ .Values.containerPort }} protocol: TCP env: - name: AWS_REGION value: {{ .Values.aws.region | quote }} {{- with .Values.extraEnv }} {{- toYaml . | nindent 12 }} {{- end }} readinessProbe: httpGet: path: /-/ready port: http initialDelaySeconds: 5 periodSeconds: 5 livenessProbe: httpGet: path: /-/ready port: http initialDelaySeconds: 30 periodSeconds: 15 resources: {{- toYaml .Values.resources | nindent 12 }} volumeMounts: - name: config mountPath: /config readOnly: true - name: cache mountPath: {{ .Values.disk.mountPath }} volumes: - name: config configMap: name: {{ include "vector.fullname" . }}-config - name: cache persistentVolumeClaim: claimName: {{ include "vector.fullname" . }}-cache {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} {{- end }} {{- with .Values.tolerations }} tolerations: {{- toYaml . | nindent 8 }} {{- end }} {{- with .Values.affinity }} affinity: {{- toYaml . | nindent 8 }} {{- end }} ``` #### `templates/service.yaml` ```yaml templates/service.yaml theme={null} apiVersion: v1 kind: Service metadata: name: {{ include "vector.fullname" . }} labels: {{- include "vector.labels" . | nindent 4 }} {{- with .Values.service.annotations }} annotations: {{- toYaml . | nindent 4 }} {{- end }} spec: type: {{ .Values.service.type }} ports: - name: http port: {{ .Values.service.port }} targetPort: http protocol: TCP selector: {{- include "vector.selectorLabels" . | nindent 4 }} ``` #### `templates/_helpers.tpl` ``` {{/* Expand the name of the chart. */}} {{- define "vector.name" -}} {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} {{- end -}} {{/* Fully qualified app name. */}} {{- define "vector.fullname" -}} {{- if .Values.fullnameOverride -}} {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} {{- else -}} {{- $name := default .Chart.Name .Values.nameOverride -}} {{- if contains $name .Release.Name -}} {{- .Release.Name | trunc 63 | trimSuffix "-" -}} {{- else -}} {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} {{- end -}} {{- end -}} {{- end -}} {{- define "vector.labels" -}} app.kubernetes.io/name: {{ include "vector.name" . }} app.kubernetes.io/instance: {{ .Release.Name }} app.kubernetes.io/managed-by: {{ .Release.Service }} app.kubernetes.io/component: vector helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} {{- end -}} {{- define "vector.selectorLabels" -}} app.kubernetes.io/name: {{ include "vector.name" . }} app.kubernetes.io/instance: {{ .Release.Name }} {{- end -}} {{- define "vector.serviceAccountName" -}} {{- if .Values.serviceAccount.create -}} {{- default (include "vector.fullname" .) .Values.serviceAccount.name -}} {{- else -}} {{- default "default" .Values.serviceAccount.name -}} {{- end -}} {{- end -}} ``` #### Install the chart ```bash theme={null} helm install opendata-vector ./charts/opendata-vector \ --set s3.bucket=my-vector-bucket \ --set s3.region=us-west-2 \ --set serviceAccount.roleArn=arn:aws:iam::123456789012:role/opendata-vector ``` ### Health checks Vector exposes two health-check endpoints: | Endpoint | Type | Behavior | | ------------ | --------- | ----------------------------------------------------------------- | | `/-/healthy` | Liveness | Returns 200 if the process is running | | `/-/ready` | Readiness | Returns 200 once Vector is initialized and ready to serve queries | Both probes are included in the Helm chart's Deployment template above. ## Monitoring All metrics are exposed at `/metrics` in Prometheus text format. ### Key Metrics | Metric | Type | Labels | Description | | -------------------------------------- | --------- | ------------------------------ | ------------------------------------------- | | `vector_upserts_total` | counter | - | Number of vectors upserted | | `vector_deletes_total` | counter | - | Number of vectors deleted | | `vector_indexer_writes_total` | counter | - | Number of writes processed by the indexer | | `vector_indexer_ann_splits_total` | counter | - | Number of centroids split by the indexer | | `vector_indexer_ann_merges_total` | counter | - | Number of centroids merged by the indexer | | `vector_indexer_ann_reassigns_total` | counter | - | Number of vectors reassigned by the indexer | | `vector_query_vectors_scored_total` | counter | - | Total number of vectors scored by queries | | `vector_query_search_duration_seconds` | histogram | Search latency distribution | | | `vector_api_requests_total` | counter | `method`, `endpoint`, `status` | Number of HTTP requests served | Vector also exposes `slatedb_*` metrics from the underlying SlateDB storage engine. These are useful for debugging storage-level performance and compaction behavior. ### TLS and authentication Vector does not include built-in TLS termination or authentication. Place a reverse proxy (nginx, Envoy, or a cloud load balancer) in front of Vector to handle TLS and access control. ### Object storage security The Helm chart uses [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (IAM Roles for Service Accounts) so that the pod receives temporary AWS credentials automatically — no static access keys required. Create an IAM role with the following policy and attach it to the ServiceAccount via the `serviceAccount.roleArn` value: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": [ "arn:aws:s3:::my-vector-bucket", "arn:aws:s3:::my-vector-bucket/*" ] } ] } ``` The IAM role's trust policy should scope access to your EKS cluster's OIDC provider and the specific service account: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/EXAMPLE" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks.us-west-2.amazonaws.com/id/EXAMPLE:aud": "sts.amazonaws.com", "oidc.eks.us-west-2.amazonaws.com/id/EXAMPLE:sub": "system:serviceaccount:default:opendata-vector" } } } ] } ``` Additional recommendations: * Enable **encryption at rest** on the S3 bucket (SSE-S3 or SSE-KMS). * Use a **VPC endpoint** for S3 to keep traffic off the public internet. * Block all public access on the bucket. # Vector Quickstart Source: https://opendata.dev/docs/vector/quickstart Install Vector locally and search your first documents in minutes Get a vector search database running on your machine in under five minutes. You will install Vector, configure a collection, upsert a few records, and run a similarity search using curl. ## Install OpenData Vector Download and install the Vector binary: ```bash theme={null} curl -fsSL https://www.opendata.dev/install.sh | sh -s -- vector ``` This places the `opendata-vector` binary in the current directory. ## Configure Vector Create a file called `vector.yaml` with the following contents: ```yaml vector.yaml theme={null} storage: type: InMemory flush_interval: 1 dimensions: 2 distance_metric: L2 metadata_fields: - name: label field_type: String indexed: true ``` This configures a collection with: * **2 dimensions** — each vector has two `f32` components. * **L2 distance** — Euclidean distance, where lower scores mean higher similarity. * **One indexed field** — a `label` string field you can filter on during search. ## Start the server Run the binary with the configuration file you just created: ```bash theme={null} ./opendata-vector --port 8080 vector --config vector.yaml ``` Vector starts and is ready to accept requests on port 8080. ## Upsert records Insert three records with 2-dimensional vectors. Requests use the `application/protobuf+json` content type: ```bash theme={null} curl -X POST http://localhost:8080/api/v1/vector/write \ -H "Content-Type: application/protobuf+json" \ -d '{ "upsertVectors": [ { "id": "north", "attributes": { "vector": [0.0, 1.0], "label": "north" } }, { "id": "east", "attributes": { "vector": [1.0, 0.0], "label": "east" } }, { "id": "northeast", "attributes": { "vector": [1.0, 1.0], "label": "northeast" } } ] }' ``` The server responds with a confirmation: ```json theme={null} { "status": "success", "vectorsUpserted": 3 } ``` ## Search for nearest neighbors Search for the 2 closest vectors to `[0.0, 0.9]` — a point near "north": ```bash theme={null} curl -X POST http://localhost:8080/api/v1/vector/search \ -H "Content-Type: application/protobuf+json" \ -d '{ "vector": [0.0, 0.9], "k": 2 }' ``` The response returns the nearest records ranked by L2 distance: ```json theme={null} { "status": "success", "results": [ { "score": 0.01, "vector": { "id": "north", "attributes": { "vector": [0.0, 1.0], "label": "north" } } }, { "score": 1.01, "vector": { "id": "northeast", "attributes": { "vector": [1.0, 1.0], "label": "northeast" } } } ] } ``` `north` is closest because `[0.0, 1.0]` is only 0.1 away from the query `[0.0, 0.9]` in Euclidean distance (score = 0.01 = 0.1²). ## Fetch a record by ID Retrieve a specific record using its ID: ```bash theme={null} curl http://localhost:8080/api/v1/vector/vectors/east ``` ```json theme={null} { "status": "success", "vector": { "id": "east", "attributes": { "vector": [1.0, 0.0], "label": "east" } } } ``` ## Next steps * To try a more realistic quickstart that indexes a large set of documents using a real embedding model, see the [example on GitHub](https://github.com/opendata-oss/opendata/tree/main/vector/quickstart). * Understand how records are structured in [Data Model](/docs/vector/data-model). * Learn how the vector index works in [Storage Design](/docs/vector/storage-design). * Browse the full REST API in the **API reference** section in the sidebar. # Vector Storage Design Source: https://opendata.dev/docs/vector/storage-design How Vector indexes records to support efficient similarity search Vector stores all records as key-value records in SlateDB. Fields are indexed in an inverted index for efficient attribute-based filtering. Vectors are indexed for efficient approximate nearest neighbour (ANN) on object storage in a durable structure based on [SPFresh](https://arxiv.org/abs/2410.14452). The vector index associates each vector with one or more of its closest "centroids" such that each centroid has a similar small number of associated vectors. The vectors assigned to each centroid are tracked in a posting list on object storage. The centroids are tracked in a separate structure that supports efficiently finding the nearest centroids to a query vector. Currently this structure is an in-memory HNSW graph. Search queries find centroids near the query vector, then load the associated postings from storage to yield a set of candidate results. The candidates are filtered using the attribute indexes and then sort-ranked to yield the closest records. This page covers the conceptual storage model. For lower level details ( compression, quantization, exact byte-level encoding schemas, etc see the [RFCs on GitHub](https://github.com/opendata-oss/opendata/tree/main/vector/rfcs) ## Stored Structures | Structure | Purpose | | ------------------ | ---------------------------------------------------------------------------------------------- | | **Dictionary** | Maps each record's id to an internal id | | **Forward Index** | Maps each record's internal id to its attributes | | **Inverted Index** | Maps indexed attribute key/value pairs to internal IDs for efficient attribute-based filtering | | **Centroids** | Stores the current set of centroid vectors | | **Postings** | Maps each centroid to the set of associated vectors | ### Dictionary The dictionary maps each record to its internal id. Records are written to Vector using a user-provided id that can be up to 64 bytes. Vector maps these to an internal 64-bit id for efficient storage and stores the mapping in this dictionary. ### Forward Index The forward index maps each record's internal id to the full set of record attributes, including the associated vector. ### Inverted Index The inverted index maps a given attribute key-value pair to the set of internal ids of records that match that key-value pair. The set is stored as a bitmap for efficient inclusion/exclusion tests and set operations when evaluating attribute-based filters. ### Centroids This is the current set of centroid vectors. When Vector is loaded, it reads the set of centroids to build the in-memory graph that enables it to efficiently find the closest centroids to a query vector. ### Postings The postings map each centroid to the set of associated nearby vectors. Each posting entry is keyed by the centroid id and stores both the record id and the record's vector. ### How a query uses the stored structures To illustrate how Vector uses the stored structures, lets consider a few high-level flows. First, consider a query that searches for 10 records near vector `[0.1, 0.2, 0.3]` with attribute filter `department="electronics"`: 1. If not yet loaded, load the centroids and build the in-memory centroid graph. 2. Search the centroid graph for centroids close to `[0.1, 0.2, 0.3]`. 3. Load all vectors from postings mapped to the nearby centroids. 4. Use the inverted index to filter the vectors that match `department="electronics"`. 5. Compute the distance of each vector to `[0.1, 0.2, 0.3]` and sort-rank these. 6. Return the 10 closest records ### How the database is maintained on an upsert Now let's take a look at how the database is maintained on an upsert. Consider an upsert of a record with id "calculator", vector `[0.1, 0.2, 0.3]` with an attribute `department="electronics"`: 1. Look up "calculator" in the dictionary. If it's present, then mark its existing id as deleted in the forward index, inverted index, and postings. 2. Assign a new internal id to the record. 3. Update the dictionary to associate "calculator" with the new internal id. 4. Write a mapping from the internal id to the record data in the forward index. 5. Find the nearest centroid(s) and add the new internal id and `[0.1, 0.2, 0.3]` to the postings. 6. Update the inverted index entry for `department="electronics"` to include the new internal id. ### How the database maintains the vector index Vector uses an algorithm similar to the one proposed in [SPFresh](https://arxiv.org/pdf/2410.14452) to maintain the vector index as records are inserted. First, consider the problem with a static vector index that maintains a fixed set of centroids. As vectors are upserted, the centroids' posting lists will grow larger and increasingly imbalanced. As illustrated above, serving a query requires loading multiple posting lists from storage and sort-ranking the associated vectors. Therefore query latencies are proportional to the size of the searched postings. To keep posting list sizes balanced, Vector tracks the size of each centroid's postings. When a centroid grows too large, the db triggers a "split" which computes a new set of centroids for the posting and reassigns the large centroid's vectors to the new centroids. When a centroid becomes too small, the db merges it with a nearby small centroid. This allows the db to maintain an efficient ANN index as vectors are upserted and the db grows and the shape of the indexed vector space changes.