One server, and then people showed up

Most system design explainers hand you a list of components and wish you luck. That's how I first learned it, and it left me able to define a load balancer without ever knowing when I'd need one.

So this is the same material in the order it actually happens: you ship one server, traffic arrives, something breaks, and you reach for the next tool. Every technique below exists because the previous one ran out.

I've marked what's textbook, what I've run myself, and what I'm still fuzzy on.

You have one server

One machine. It takes requests, talks to a database, sends responses. Your entire architecture diagram is a rectangle.

This is a good place to be, and people leave it far too early. A single modern server handles more traffic than almost any side project will ever see. The failure mode of premature architecture is worse than the failure mode of one box: you get all the operational cost of a distributed system and none of the load that would justify it.

So the honest first answer to "how do I scale this?" is usually don't yet.

But eventually the box gets hot. Now what?

Just buy a bigger machine

The cheapest real fix is to make the one machine bigger. More cores, more RAM, faster disk.

Vertical scaling — adding more resources to a single machine. Not to be confused with optimising your code, which is a separate lever that costs engineering time instead of money.

I want to be blunt about this because I had it wrong for a long time: vertical scaling is not the unsophisticated option. It's the option with the best effort-to-result ratio. Resizing an instance is a reboot. Sharding a database is a quarter.

Where it runs out — three ceilings, in the order you hit them:

1. Price. Cost climbs faster than capacity. Double the machine, more than double the bill.
2. Physics. The largest instances cloud providers rent top out in the tens of terabytes of RAM. Huge, but finite.
3. Amdahl's law. Cores only help the parallel part of your workload. If 20% of a request is inherently serial, no core count gets you past a 5x speedup.

And the ceiling that isn't about size at all: it's still one machine. One power supply, one kernel panic, one bad deploy.

Do the work before anyone asks

Before adding machines, take work out of the request path.

If a dashboard aggregates a million rows every time someone loads it, you're paying that cost per visitor. Compute it once at 3 AM and serve the result. Cron jobs, materialised views, precomputed feeds, warmed caches — same idea: move work from peak time to quiet time.

This is the highest-leverage thing on the list because it doesn't just spread load, it deletes it.

The trade: staleness. Precomputed data is by definition out of date, and now you own a question you didn't have before — how stale is too stale? For a leaderboard, minutes are fine. For a bank balance, nothing is.

The machine dies

It will. AWS's EC2 SLA for a single instance is 99.5% — that's about 44 hours a year of acceptable-to-them downtime, and single-instance means no credit until you're below it.

Here's the distinction my notes had fused into one bullet, and it matters:

Redundancy keeps you serving. A second machine, ready to take over. Backups keep your data. Point-in-time copies you can restore from.

They solve different disasters and neither covers the other. A hot standby will not save you from DROP TABLE users — it replicates the drop, faithfully, in milliseconds. A nightly backup will not save you from a Tuesday-afternoon outage.

You need both. Most people discover this the expensive way.

Add machines instead of a bigger one

You've hit the ceiling, so now there are several servers.

Horizontal scaling — adding more machines rather than growing one. In principle unbounded: you can always rent another box.

The thing nobody warns you about: horizontal scaling is easy for your code and hard for your state. If a request can land on any server, no server can keep anything important in memory. Sessions, uploads, in-process caches, that one counter — all of it has to move somewhere shared.

The trade: you've swapped a capacity problem for a consistency problem. That's usually the right swap. It is never a free one.

Now who decides which server?

Three servers and no traffic cop is just three servers. Something has to route.

Load balancer — sits in front of your servers and decides which one handles each request.

The interesting part is how it decides. Not randomly. Real strategies route on least connections, lowest observed latency, or weighted capacity — the goal is the shortest total turnaround time, queue plus processing, not just an even split. A round-robin that sends a request to a busy server is worse than a smart one that doesn't.

The trade, and this is the one my draft missed entirely: I spent a whole section warning about single points of failure, then introduced one. Everything now flows through the load balancer. If it dies, all of it dies.

The fix is that the load balancer itself gets redundancy — a pair sharing a floating IP, or a managed one where the provider hides the pair from you. Worth knowing what your cloud is doing on your behalf, because you're paying for it either way.

One codebase becomes many services

Same story, applied to your code instead of your machines. Auth, orders, returns — each becomes a service that scales and deploys on its own.

The real win isn't organisational tidiness, it's independent scaling. Checkout gets hammered on launch day; your settings page does not. Splitting them means you buy capacity only where the load is.

The trade, in one number: a function call is a memory reference, roughly 100 nanoseconds. A round trip inside the same datacenter is roughly 500 microseconds — about 5,000x slower (from Jeff Dean's latency numbers, still the best cheat sheet in the field).

You just converted a category of calls from the first kind to the second. Plus: partial failures, retries, distributed tracing, and transactions that no longer roll back cleanly. A monolith that fits in your head beats microservices that don't.

Same data, many places

Your users aren't in one country, and light isn't fast enough.

Mumbai to Virginia is about 12,500 km. Light in fibre moves at roughly 200,000 km/s, so that's a 125 ms round trip before your server has done a single thing. Real routes aren't straight lines — expect closer to 200 ms. You cannot optimise your way out of this. You can only move the data closer.

Replication — the same data, copied to multiple machines or regions.

So you put copies near your users. Reads get fast, and you survive losing an entire region.

The trade, unavoidable: the copies disagree. A write in Mumbai isn't instantly true in Virginia. That's CAP — Brewer's conjecture in 2000, proved by Gilbert and Lynch in 2002: when the network between your copies fails, you choose consistency or availability. You don't get to keep both.

Which means it's a product decision, not an infra one. Two people can't buy the same last ticket, so ticketing picks consistency. A like count can lag two seconds, so social picks availability. Answer this per feature, not per system — this is the part I'm least confident I'd get right under real load, and the part I'd most want a senior engineer to check.

Splitting the data itself

Replication copies all the data everywhere. That stops working when the dataset itself is too big for one machine.

Partitioning (sharding) — splitting one dataset across machines so each holds a subset. Replication duplicates; partitioning divides. My original notes had these as one concept, and they are not.

Users A–M here, N–Z there. Now storage and writes both scale.

The trade: queries that span shards get ugly, cross-shard transactions get very ugly, and choosing a bad shard key gives you one machine on fire while the rest idle. Changing that key later is one of the genuinely hard migrations.

Shard late. Shard reluctantly.

Parts that don't know about each other

The through-line under all of it.

Decoupling — components that interact through a contract, without knowing each other's internals.

The best framing I know is a delivery rider: they don't need to know whether the bag holds pizza or burgers. The kitchen can change its entire menu and the delivery system never notices, because the contract is "a bag goes to an address."

That's what buys you extensibility — and this is why it isn't a separate item on the list, it's the payoff. If you already run a PDF-to-Word service, adding CSV-to-XLSX means deploying a new service, not editing the old one. Nothing that works today gets touched.

Queues are the sharpest version of this: the producer doesn't know who consumes, or when, or whether they're even awake right now.

What I left out

Caching and CDNs, rate limiting, observability, consensus, message-queue delivery guarantees. Each is a post. Leaving them out was deliberate — they're the next layer, not this one.

The one thing I'd keep if you forget everything else: every item here is a trade, and the sequence matters more than the list. Reach for them in order, and stop as soon as the pain stops. Most systems never need the bottom half of this page.

Sources

• Jeff Dean, "Latency Numbers Every Programmer Should Know"
• Seth Gilbert & Nancy Lynch, "Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services" (2002)
• AWS, "Amazon Compute Service Level Agreement"
• Martin Kleppmann, Designing Data-Intensive Applications — chapters 5 and 6 on replication and partitioning