> ## Documentation Index
> Fetch the complete documentation index at: https://opendata.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Log

> A key-oriented, objectstore-native log for routing events to addressable destinations

Log is an MIT-licensed, key-oriented log built on [SlateDB](https://slatedb.io)
and object storage. It maintains millions of individually keyed, ordered logs on
a single node and scales to tens of thousands of active readers, each tailing
its own key. It runs as a single Rust binary with object storage as its only
durability requirement.

## What it's good at

* **Routing workloads.** Messaging, feeds, agent traces, and microservice
  communication, where each destination wants a specific subset of keys rather
  than the whole stream.
* **High key cardinality.** Hundreds of thousands to millions of keys on one
  node. Any key's log is a prefix scan on an LSM index, not a needle-in-a-haystack
  scan of an entire partition.
* **Large read fan-out.** Tens of thousands of concurrent followers reading from
  a single instance, scaled further by adding read replicas that pull straight
  from object storage.
* **Per-key isolation.** Position metadata is tracked per key, so a poison-pill
  message stays contained to its key instead of blocking an entire partition.

## Why Log

Kafka was built to collect data from many high-cardinality sources and
coalesce it onto one low-cardinality pipe. Log is built for the other half of
logging, *routing*: delivering events from granular  sources to granular
destinations.

If your consumers each care about a specific subset of keys, you want to be able to retrieve
the events for a key as efficiently as possible. That's why Log addresses events by key rather
than topic-partitions. You simply `scan` the Log by key and get ordered values back.

Object storage as the only durable layer makes Log strongly consistent and cheap
to operate. Read replicas scale linearly, and scoping a replica to a
contiguous key range keeps its cache warm with data that nearby queries will
reuse. See the [benchmarks](/log/benchmarks) for throughput, poll latency, and
cost on a single node.

## Tradeoffs

* **Poll latency scales with ingest rate.** Higher ingest rates evicts hot data before
  compaction can collocate keys for efficient querying. The mitigation is scoping
  read replicas to smaller key ranges so the cache isn't thrashed by unrelated
  blocks.
* **Object-storage latency floors apply.** Polls inherit object-store round-trip
  latency. Plan for tens of milliseconds at the median and hundreds at the tail.
* **No built-in offset tracking.** Log separates data from consumption metadata.
  Track consumer offsets yourself in a key-value store such as SlateDB.

## Explore Log

<CardGroup cols={2}>
  <Card title="Quickstart" icon="rocket" href="/log/quickstart">
    Install Log and append your first events in under five minutes
  </Card>

  <Card title="API Reference" icon="code" href="/api-reference/append-records">
    Browse the full REST API for appending, scanning, and managing logs
  </Card>

  <Card title="Storage Design" icon="hard-drive" href="/log/storage-design">
    How Log maps streams to LSM keys with segment-based compaction
  </Card>

  <Card title="Getting to Production" icon="server" href="/log/production">
    Deploy, monitor, and secure Log for production workloads
  </Card>

  <Card title="Benchmarks" icon="gauge-high" href="/log/benchmarks">
    Ingest throughput, poll latency, and cost for a single node
  </Card>

  <Card title="When to choose Log" icon="scale-balanced" href="/log/comparisons">
    How Log compares to Kafka, the Kafka-on-S3 systems, S2, and NATS JetStream
  </Card>

  <Card title="GitHub" icon="github" href="https://github.com/opendata-oss/opendata/tree/main/log">
    View the source code, open issues, and contribute
  </Card>
</CardGroup>
