HLD: Twitter / X

Twitter, now known as X, is a social media platform that allows users to post tweets, follow each other and view a timeline of tweets from accounts they follow. To design such a system, several functional and non-functional requirements must be considered.

Functional requirements include posting tweets which may contain text, images, or videos, following or unfollowing users, viewing a home timeline consisting of tweets from followed users in reverse chronological order, liking, retweeting and replying to tweets, viewing a user's profile with their tweet history, and searching for tweets.

Non-functional requirements include ensuring high availability since Twitter is a utility, using eventual consistency allowing the timeline to be slightly stale (seconds), low latency for quick timeline load (200ms), a read-heavy design with a 100:1 read-to-write ratio, and the ability to handle 300 million daily active users and 500 million tweets per day.

Estimating the scale of the platform, the average tweet is approximately 300 bytes with additional metadata. Therefore, 500 million tweets per day would require around 500 GB of storage per day. Media uploads could account for up to 100 TB per day. Timeline caching requires a sorted set for each user with a maximum of 800 tweets, and a TTL of 7 days.

API design includes endpoints for posting a tweet, getting a user's timeline, getting a user's tweets, liking or unliking a tweet, following or unfollowing a user. The core problem lies in generating the home timeline. Initially, a push approach was used where the user's followers would receive the tweet, but this proved problematic with celebrities having hundreds of thousands of followers.

A hybrid approach was adopted, combining fan-out on write for regular users with fan-out on read for celebrity users. Read requests are fulfilled by fetching pre-computed timelines from Redis and recent tweets from the celebrity accounts through fan-out reads.

The architecture consists of a CDN serving static assets and media, an API gateway and load balancer, tweet and timeline services, media service handling image uploads, resizing, and CDN storage, and a fan-out service using Apache Cassandra for write-heavy operations and Kafka for asynchronous fan-out. Redis is used for timeline caching with a sorted set of tweet IDs, and Elasticsearch is used for searching tweets.

The main challenge is the "celebrity" problem where hot accounts like Elon Musk, with millions of followers, pose a significant load on the system. To address this, the system avoids fan-out for these users, instead pre-calculating their timelines and caching recent tweets. This ensures the system remains scalable and performant, despite the popularity of certain accounts.