Buffer

Buffer is an MIT-licensed, stateless library that uses object storage as a durable queue between producers and consumers. Producers batch opaque byte entries and flush them to object storage as append-only data files. Consumers pull batches directly from object storage. A single manifest file on object storage coordinates producers and consumers through atomic compare-and-set operations. The producer and consumer frameworks are extensible and can accommodate a variety of data types and source and destination systems. And because both the data and the manifest live in object storage, Buffer is stateless and horizontally scalable, and it leaves durability and availability to S3.

​

What it’s good at

Buffer was built for high throughput data pipelines which don’t need strict ordering and where sink-side deduplication is acceptable. This describes a vast majority of observability and event pipelines, like those shipping metric, event, log, and trace data to observability and analytics stores.

​

Why Buffer

Buffer’s only goal is to make object storage usable as a high volume and moderately low-latency queue. That focus is the reason it costs a small fraction of a managed or self-hosted Kafka cluster for the same throughput. The ingestors are stateless, so you scale and fail over by adding or replacing pods rather than rebalancing a stateful cluster. The manifest’s compare-and-set gives you at-least-once delivery and fences zombie clients. The pipelined producer and consumer clients keep many object-storage requests in flight at once to maximize throughput. Each data batch carries a stable identity across retries so sinks can deduplicate correctly. See the benchmarks for end-to-end latency, throughput, and a cost comparison against Kafka.

​

Tradeoffs

• Finite participants. Throughput scales inversely with the number of participants, since metadata updates contend on a single object-storage compare-and-set. Run one writer per zone and a fixed, limited number of consumers.
• No partition multiplexing. Partitioning is a user-space concern. If vertical scaling isn’t enough, deploy multiple buffers and let Kubernetes handle scheduling and liveness.
• Higher latency than in-memory Kafka. Batching to object storage and polling the manifest is a two-hop design. In practice that means a p99 of a couple of seconds, tunable down to a low floor by flushing and polling more often at the cost of more object-storage requests.

​

Explore Buffer