Real-Time Data Infrastructure at Scale: Lessons from Meta's Streaming Architecture

I'll deliver a concentrated dose of hard-won lessons from building and operating one of the world's largest real-time data processing systems. In just 15 minutes, this talk will share the most critical insights from processing hundreds of terabytes daily across billions of users, focusing on the breakthrough moments and hard-learned principles that transformed our streaming architecture.

This isn't a broad survey—it's a focused deep dive into the three most impactful challenges we've solved at Meta's scale, delivered with the intensity and practical focus that comes from years of production experience. Attendees will leave with immediately actionable strategies that can be applied regardless of their current scale.

We'll start by establishing the unique constraints of processing streaming data at Meta's scale - where traditional solutions break down and custom approaches become necessary. I'll share specific numbers around throughput, latency requirements, and the complexity of coordinating thousands of internal teams sharing the same infrastructure.

Lightning Architecture Overview

A rapid but comprehensive walkthrough of Meta's Scribe-Puma-Ptail streaming stack, focusing on the key architectural decisions that enabled massive scale rather than implementation details.

The Three Critical Breakthroughs

I'll focus on the three most transformative solutions we've implemented:

Schema Evolution Without Downtime: How we handle thousands of evolving event schemas across product teams while maintaining backward compatibility

Multi-Tenant Resource Isolation: Our approach to preventing noisy neighbor problems when thousands of teams share the same streaming infrastructure

Cross-DC Failure Recovery: Battle-tested strategies for maintaining consistency during regional outages

Each section will include a specific production war story, the technical solution we implemented, and the key principles other teams can apply.