System Design: Database Sharding

This beginner-friendly course on database sharding focuses on understanding why sharding is needed, how it works, and when not to shard at all, rather than memorizing terms.

The whole course follows Chirp, a small social app where people post short messages, follow each other, and like posts. Chirp starts on a single database with a few hundred users and grows to 40 million users and 8 TB of data. Every lesson starts with something breaking: a profile that used to open in 30 milliseconds now takes 3 seconds, writes pile up during a big event, a pop star with 40 million followers overloads one machine, or adding a single machine forces 80% of the data to move. Then you fix it and look at the new problem the fix creates.

Across seven lessons, you learn about reads and writes, replicas, and why adding replicas can’t raise the write limit. You also learn what a shard is, how to choose a shard key, scatter-gather and co-location, hashing, and the four ways to split data. After that, you explore the routing layer, hotspots and the celebrity problem, resharding, consistent hashing and virtual nodes, fixed buckets, and joins across shards. Then you see how real databases such as Vitess, Citus, Cassandra, DynamoDB, and MongoDB handle sharding for you. The course ends with the three signals that tell you it's actually time to shard. Everything is explained in plain language, with a hand-drawn diagram for every step.

This course also includes four hands-on labs on a real Citus cluster. In them, you split a users table across machines; test different shard keys to see which ones spread the data evenly; overload one shard with a pop star's traffic and fix it; and finally, add a new machine, move data onto it, and join tables that live on different shards. Every lab comes with hints and full solutions.

If you know what a database table is and can use a terminal, you're ready to start.