The Architecture Twitter Uses to Deal with 150M Active Users, 300K QPS, a 22 MB/S Firehose, and Send Tweets in Under 5 Seconds

Some mouth-watering details from the VP of Eng. at Twitter on their backend infrastructure. Twitter’s mostly asymmetric structure and short head/long tail distribution of followers create an interesting challenge. 

Some tidbits:

– 50 times as much consumption than production of tweets
– 150M users, 400M tweets a day, 300K queries per second
– Largest generator of SMSs per active user of anyone in the world
– Push notification system accounts for 22MB/s
– The Redis cluster has a couple of terabytes of RAM, every active user is stored in RAM to keep latencies down (< 30 days)
– Tweet input statistics: 400m tweets per day; 5K/sec daily average; 7K/sec daily peak; >12K/sec during large events.
– Timeline delivery statistics: 30b deliveries / day (~21m / min); 3.5 seconds @ p50 (50th percentile) to deliver to 1m; 300k deliveries /sec; @ p99 it could take up to 5 minutes

 
Original presentation here: http://www.infoq.com/presentations/Twitter-Timeline-Scalability
 

High Scalability – High Scalability – The Architecture Twitter Uses to Deal with 150M Active Users, 300K QPS, a 22 MB/S Firehose, and Send Tweets in Under 5 Seconds

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s