Log

Log is an MIT-licensed, key-oriented log built on SlateDB and object storage. It maintains millions of individually keyed, ordered logs on a single node and scales to tens of thousands of active readers, each tailing its own key. It runs as a single Rust binary with object storage as its only durability requirement.

​

What it’s good at

• Routing workloads. Messaging, feeds, agent traces, and microservice communication, where each destination wants a specific subset of keys rather than the whole stream.
• High key cardinality. Hundreds of thousands to millions of keys on one node. Any key’s log is a prefix scan on an LSM index, not a needle-in-a-haystack scan of an entire partition.
• Large read fan-out. Tens of thousands of concurrent followers reading from a single instance, scaled further by adding read replicas that pull straight from object storage.
• Per-key isolation. Position metadata is tracked per key, so a poison-pill message stays contained to its key instead of blocking an entire partition.

​

Why Log

Kafka was built to collect data from many high-cardinality sources and coalesce it onto one low-cardinality pipe. Log is built for the other half of logging, routing: delivering events from granular sources to granular destinations. If your consumers each care about a specific subset of keys, you want to be able to retrieve the events for a key as efficiently as possible. That’s why Log addresses events by key rather than topic-partitions. You simply scan the Log by key and get ordered values back. Object storage as the only durable layer makes Log strongly consistent and cheap to operate. Read replicas scale linearly, and scoping a replica to a contiguous key range keeps its cache warm with data that nearby queries will reuse. See the benchmarks for throughput, poll latency, and cost on a single node.

​

Tradeoffs

• Poll latency scales with ingest rate. Higher ingest rates evicts hot data before compaction can collocate keys for efficient querying. The mitigation is scoping read replicas to smaller key ranges so the cache isn’t thrashed by unrelated blocks.
• Object-storage latency floors apply. Polls inherit object-store round-trip latency. Plan for tens of milliseconds at the median and hundreds at the tail.
• No built-in offset tracking. Log separates data from consumption metadata. Track consumer offsets yourself in a key-value store such as SlateDB.

​

Explore Log