Log Benchmarks

​

Throughput

A single m5n.xlarge node delivers either high write throughput or a large follower fan-out, depending on how you load it: The read-serving configuration sustains 50,000 concurrent followers each tailing their own keyed log, with an 8 GB cache and 1M keys, at <50 ms p50 poll latency. Reader nodes scale linearly: add a reader to add read throughput.

​

Poll latency

​

Flat across key cardinality

At a fixed 20 MB/s ingest, increasing the key count from 100K to 1M does not degrade p50 poll latency for reading a single key’s log: Per-key read latency is independent of how many keys live on the node.

​

Scales with ingest rate

Poll latency is sensitive to ingest rate. Below the cache-residency point (~5 MB/s) the live tail stays cache-resident and latency is low and flat. Above it, ingest evicts hot data before compaction collocates keys, and the tail climbs: In practice you can expect p50/p99 poll latencies of ~30 ms / ~300 ms at 25 MiB/s of throughput.

​

Range vs. random sharding

Scoping a reader to a contiguous key range improves cache locality and thus improves poll latencies. At a fixed 25 MB/s ingest over 1M keys, the difference between different sharding schemes is shown below: Range scoping holds read amplification and latency flat as parallelism grows.

​

Cost

The cost breakdown of one fully-loaded node serving 20 MB/s ingest, 50,000 concurrent followers, and 1M keys, with 1 day of retention works out to: GET costs are not included since they are mostly workload dependent. However, if most of your reads are for recent data, they will be served from the local cache. You can add a reader for another ~50k followers at +$174/mo. Log flushes to S3 approximately every 2–3 s in this configuration. You can configure write-ahead-logging for faster durability guarantees at the cost of additional S3 PUT requests.

​

Learn more

• Storage internals: Log storage design and the storage RFC
• For the funneling use case, see OpenData Buffer benchmarks