Urgent.News

What's breaking now, across hundreds of outlets.

Tech

How Instagram Split One Database Into Thousands

Shard keys, logical shards, and the 64-bit photo ID that says when a photo was made and which shard holds it Two different photos. Two different databases. Both stamped with the same ID: 3. That's what happens when you split one database into many pieces and every piece counts on its own. Instagram had to split its data early, so it had to solve exactly this. Its answer was hidden inside the…

As Instagram's user base surpassed 14 million in just a year, the app's data requirements grew exponentially. All information, including user details, photo specifics, and tags, resided within PostgreSQL databases. In a short period, over 25 photos and 90 likes were generated every second. While adding more database servers could alleviate the strain, Instagram needed a solution that would prevent the main server from becoming a bottleneck.

The answer lay in splitting the database into numerous pieces, each self-sufficient and requiring minimal coordination. This led to the introduction of shard keys, logical shards, and a unique 64-bit photo ID that determined the shard responsible for each photo. Splitting the database by upload time or using random IDs resulted in inefficiencies, as either hot shards or scattered user data would arise.

Instead, Instagram opted to group data around users, ensuring that each user's data remained on a single logical shard among thousands. This approach allowed Instagram to efficiently locate the appropriate server without searching through all available machines, resulting in a streamlined and scalable system.

Rewritten by urgent.news; may contain errors — check the original.

Read the original at dev.to →

Report a problem with this story

More in Tech

Your JWT Secret Is Probably Too Weak. Here Is the 30-Second Fix

"secret", "changeme" and "mysupersecretkey" are not keys. They are passwords, and passwords get cracked offline. Open the config of almost any tutorial project and you will find a line like this…

  • Weak JWT secrets like "secret" and "changeme" are easily cracked offline.
  • Use cryptographically secure random generators to create strong, unique secrets.

AutoSSL Quietly Dropped the Mail Name — a Cloudflare-Only Origin and the Move to DNS-01

A field story about a cPanel certificate that renewed on time and still broke every Outlook client — the port 80 rule almost nobody connects to mail — and the DNS-01 setup I now use for any origin…

  • Outlook client displays certificate warning due to missing mail name
  • AutoSSL issues certificate without mail name, causing warning
  • Resolving issue requires checking names served by mail ports

JWKS and `kid` Explained: How to Find the Key That Signed Your Token (and Spot a Leaked Private Key)

A practical guide to reading a jwks.json file, matching the right key, and checking it is safe to publish. Your API receives a token signed by an identity provider.

  • JSON Web Key (JWK) represents a single cryptographic key in JSON format per RFC 7517
  • JSON Web Key Set (JWKS) is a JSON object containing an array of JWKs published at public URLs
  • kid field in token header is used to find matching JWK in JWKS for signature verification

More from Saturday 21 November →