LazyLog: A New Log Abstraction for Low-Latency Applications

Streaming systems, at their core, are shared logs. These traditional shared logs enforce a strict, global order on all incoming data as it is written, which ensures strong consistency but adds significant latency during data ingestion.

In practice, we've observed that many modern applications -- such as analytics pipelines and event-driven systems -- don't need this strict order immediately when data is ingested. Instead, the order only matters later, when the data is consumed.

Based on this insight, we introduce LazyLog, a new approach to building shared logs. LazyLog delays the costly process of assigning a global order until just before the data is read, rather than at write time. This "lazy" approach significantly reduces write latency while still ensuring a consistent global view when needed.

We built two systems that implement LazyLog abstraction. These systems offer the same strong guarantees as traditional systems but with much lower write latencies. For teams building low-latency data pipelines or high-throughput distributed services, LazyLog offers a compelling alternative to conventional log-based systems.

LazyLog is the result of academic research at the University of Illinois. The paper about LazyLog has been published at SOSP, the flagship conference for systems research, winning a Best-Paper Award at the conference.