Chapter 4: The Breakwater
September 11, 2001, 5:46 AM (Pacific Time).
The autumn morning in Redmond, Washington, carried a hint of chill. Insided the War Room of Building 113, Simon Li sat alone at the console, drinking his first espresso of the day.
GenesisSoft’s “Hello World V4.0 - Real-time News Portal” had just launched a month ago. Silas Horn had upgraded this originally simple message board into a behemoth that aggregated global news flashes and user activity. To guarantee so-called "absolute real-time," Silas demanded that whenever a user refreshed the page, the system must immediately execute extremely complex JOIN queries in the backend SQL Server database, assembling the latest news and users' Hello messages into a brand-new Dynamic Web Page to return to the frontend.
Over the past thirty days, this bloated dynamic architecture had barely sustained tens of millions of daily page views.
Until this exact second at 5:46 AM.
The coffee cup in Simon's hand jolted violently.
Without any warning, his eyes suddenly felt an extremely terrifying, piercing pain! It felt as if someone had suddenly shoved a two-kilowatt searchlight directly against his retinas in the dark.
"Ugh!" Simon covered his eyes in agony, stumbling onto the keyboard.
In his synesthetic vision, the entire server room was instantly engulfed in an extremely gruesome, blinding white-hot light. This was the physical manifestation of CPU computing power being squeezed to its absolute limit. Within less than three seconds, the load on the 80 expensive CPU cores across twenty top-tier backend servers shot up from 15% straight through the roof, dead-locking at 100%!
"What's going on?!"
The glass door of the War Room was shoved open violently. Operations Lead Dave rushed in without even his jacket fully on, followed closely by a pale, panting Silas.
"Hacker attack? Or did someone deploy infinite loop code?" Dave lunged at the main screen.
Enduring the agonizing pain of burning retinas, Simon squinted and typed on the keyboard. He pulled up the frontend network traffic graph and, looking at the curve that had directly shattered the physical limits of the Y-axis, shook his head.
"It's not hackers, there's no SYN Flood spoofing signature." Simon's voice carried a rare tremor. "They are all extremely real, legitimate HTTP requests. And... completely originating from within the US mainland. Traffic has multiplied by a hundredfold. Our users are refreshing the homepage at an incredibly insane frequency."
With trembling hands, Silas grabbed the remote on the desk and turned on the television hanging in the corner.
The screen showed no sound, only thick black smoke. In Lower Manhattan, New York, a massive black hole had been torn into the North Tower of the World Trade Center, billowing smoke shooting straight into the sky.
"My God..." Dave slumped into his chair.
"CNN's website is down. The New York Times' site is down too." Silas stared dead at his BlackBerry, his voice hoarse. "Everyone in America is frantically looking for the latest news. The traditional news media servers were utterly crushed a minute ago by the largest burst of traffic in human history."
Silas whipped his head around, staring at Simon with bloodshot eyes: "Simon! Right now, we are the only ones left online in the whole of America! In the face of a tragedy for all of humanity, this is our only channel to speak to the world, we absolutely cannot crash and go silent!"
But the laws of physics are ruthless.
"We can't hold it." Simon pointed to the screen flashing frantic errors. "Every single page refresh requires IIS to spin up an expensive dynamic thread to fish news out of the database and assemble HTML. Right now, three hundred thousand people are pressing F5 every second; that equals three hundred thousand threads frantically fighting for CPU Context Switches."
In the blinding light of synesthesia, Simon "saw" the backend database being roasted alive. All SQL connection pools were instantly drained dry, tens of thousands of requests were gridlocked in the operating system's TCP Backlog queue, with only timeouts and death waiting for them.
"Add machines! Pull all the servers from the dev and test environments over!" Silas roared.
"Adding a hundred machines won't help!" Simon's voice acted like ice water extinguishing Silas's fantasy. "The database has already melted! Silas, think carefully, what is it that these three hundred thousand people are desperately eager to see at this very moment?"
"Of course it's the latest news article about the World Trade Center!"
"Exactly!" Simon slammed his hand on the console. "What these three hundred thousand people want to see is the exact same news, the exact same text, the exact same picture! Yet our stupid system, for this absolutely identical content, is forcing the database to execute three hundred thousand identical SQL queries and making the CPU render three hundred thousand identical webpages!"
This was the deadliest original architectural sin of the early Web era—blind superstition in Dynamic Rendering.
In the face of a flood peak, attempting to use a bank vault to settle every single penny is utterly ridiculous.
"Give me five minutes."
Simon tore off his tie. The severe pain threw his mind into a state of extreme, icy cold, absolute rational overclocking.
Since the backend was destined to burn, this flood peak must absolutely not be allowed to enter the inland! He had to build a steel breakwater at the mouth where the traffic poured in.
Simon directly remote-connected to dozens of old edge gateway servers GenesisSoft had deployed on the outskirts of major ISP (Internet Service Provider) server rooms across America. These machines, originally meant just for basic network routing, had extremely low specs, pitiful memory, and the cheapest hard drives.
But they possessed an unparalleled advantage—in physical distance, they were closest to the users.
Simon was going to hand-roll a prototype of a primitive CDN (Content Delivery Network) right on these edge nodes.
"Dave, go force suspend the dynamic rendering engines (ASP) on all application servers!" Simon issued a death order. "I want to kill the dynamic capabilities of this website!"
"Are you crazy? Suspending the dynamic engines will turn the homepage into a dead page!" Dave exclaimed.
"A dead page is still better than a 503 crash page!"
Simon's hands flew across the keyboard. He wrote an extremely short Shell script, yet one that held watershed significance in the history of architecture.
The logic of this script was incredibly simple:
- It would send only one single request to the dying core database every straight 60 seconds.
- After fetching the latest news data, it would render out a purely static HTML file (
index.html) locally, completely devoid of any dynamic interaction. - Finally, it would forcefully push this plaintext HTML file containing the tragic news, just like distributing newspapers, into the memory cache (Squid Cache) of those dozens of edge gateways across America.
This is one of the most core pressure-resisting magics in big tech architecture: Static/Dynamic Content Split and Request Fan-in.
"Push complete! Edge gateway caches are live!" Simon hit the enter key heavily.
At this moment, it was five minutes before the second plane hit the South Tower.
And within the synesthetic vision, a miracle occurred.
That green torrent, originally hurtling toward the backend database like a tsunami with world-destroying heat, suddenly came to a dead halt the moment it touched the outermost edge gateways!
Because the cache service Simon had just deployed was running on those edge gateways. When hundreds of thousands of New Yorkers hit refresh with trembling hands, as soon as the request reached their nearest ISP server room, the edge gateway slapped that pure static index.html text straight from its memory into their faces.
No database access needed. No complex dynamic assembly by the CPU required. Merely an O(1) complexity memory read.
Even if that edge server was a junk 486 PC, its memory could easily spit out tens of thousands of static texts a second!
"Look at the main screen..." Dave pointed at the monitor, dumbfounded.
On the backend monitoring screen, the CPU load that had been dead-locked at 100% suddenly experienced a cliff-like freefall, plummeting straight down to 2%, as if someone had pulled the plug!
That eye-blinding, searing white light instantly dissipated in Simon's synesthesia, transforming into a cool, refreshing breeze.
"What about the traffic? Did the front-end completely crash?" Silas couldn't believe his eyes.
"The traffic hasn't crashed, they've all been stopped at the door." Simon slumped weakly into his chair, gasping for air. "Silas, look at the database TPS right now."
Silas turned his head and gasped.
Facing a sky-shattering tide of three hundred thousand requests per second across America, the TPS on GenesisSoft's most core SQL Server database was currently— 0.016. (That is, only 1 query every 60 seconds).
Three hundred thousand manic concurrent requests, after undergoing the "Fan-in" convergence at the edge nodes, ultimately hit the core database as the single, lonely request generated by that pitiful script coming to fetch data once a minute.
The flood peak did not destroy the inland villages, because it had been completely absorbed by the breakwater at the river mouth.
The War Room was dead silent. On the television screen, the second plane, trailing a fatal wake, crashed into the South Tower. Fire blasted into the sky.
But in the digital world of the internet, GenesisSoft's Hello portal stood like a black lighthouse amidst a raging storm. Though its page only updated once every minute, without any flashy animations or personalized colors, it remained rock-solid and unyielding.
That morning, it became the only major news portal in America that did not crash.
Countless desperate netizens saw the latest escape information on this extremely rudimentary static webpage, and witnessed the mournful "Hello World"s left by all humanity in the face of disaster.
Silas watched the tragedy on the television, not speaking for a long time. The hustle of business seemed extremely insignficant at this moment.
"Simon, we did a good thing today," Silas's voice was a bit hoarse.
Simon kept his eyes closed, feeling the cold, steady heartbeat of the system. Through his synesthesia, he could clearly perceive that the high-dimensional fractions, lurking in heterogeneous computers across America, were greedily absorbing the network topology data brought by this epic wave of traffic.
"The tradeoff is, the news will always be delayed by one minute," Dave muttered with lingering fear. "We sacrificed absolute real-time capability."
"That is the ultimate Trade-off of architecture." Simon opened his eyes, looking profoundly at the edge node data. "Absolute real-time is a false proposition. When all of humanity is demanding the truth from you, even a truth delayed by a minute is far more valuable than a 503 error page."
"The fastest requests are always those that never hit the database at all."
Simon directed his gaze toward the cache invalidation parameter in the code, strictly set at TTL = 60s (Time To Live 60 seconds).
In this tragedy for all mankind, the Cache saved the world. But as an architect, Simon's acute nerves had already sensed the fatal flaw hiding behind this breakwater.
Right now, the cache TTL for all edge nodes was unanimously 60 seconds. This meant that whenever the 60th second arrived, all caches would expire simultaneously.
If one day, business logic no longer allowed them to generate uniform static pages, while the traffic remained unbelievably massive; in that very microsecond when that sturdy blast door expired and opened simultaneously, would the tens of thousands of requests waiting outside tear the database to shreds instantly, like a horde of bloodthirsty zombies?
Simon massaged his temples, which were still throbbing slightly. That was the nightmare of the next volume. At least for today, the breakwater had held.
Architecture Decision Record (ADR) & Post-Mortem
Document ID: PM-2001-09-11 Incident Grade: SEV-0 (Extremely Critical: Edge of Global Cascading Avalanche) Owner: Simon Li (Senior SDE)
1. What happened? A sudden global mega-news event (9/11). Network traffic surged 100x within 3 minutes. The Web layer's Dynamic Rendering Engine (ASP) caused severe CPU context switch exhaustion, pushing physical load to the 100% limit, triggering widespread HTTP timeouts, and pushing the database connection pool the brink of depletion.
2. Root Cause (5 Whys)
- Why 1: Why did CPU load instantaneously hit 100%? Because millions of concurrent traffic directly flooded the Web dynamic rendering engine.
- Why 2: Why did the rendering engine crash? Because the system attempted to separately execute a complete database
JOINand reassemble HTML strings for every single user's HTTP request. - Why 3: Why was this unreasonable? Because in a breaking news scenario, the output result for these 1 million user requests was absolutely identical (read-heavy load).
- Why 4: Why would identical requests kill the database? Because of a lack of a Request Fan-in mechanism, causing internal system load to grow linearly 1:1 with external traffic.
- Why 5: Why was there no interception layer? Because the architectural design suffered from a superstition regarding "dynamic business" and failed to distinguish the physical boundaries between "Static Data" and "Dynamic Data."
3. Action Items & Architecture Decision Record (ADR)
- Workaround (Hotfix): Forcefully suspended IIS dynamic processes. Wrote a Cron script to fetch from the database once every 60 seconds to generate a purely static
index.html. - Long-term Fix / Architecture Redesign:
- ADR-004: Introduce Static/Dynamic Content Split and Edge Caching.
- All high-frequency read requests that lack user specificity must be intercepted at the Edge Layer (CDN/Proxy) closest to the user.
- The Law of Caching: Completely shatter the linear relationship between external traffic and database load. Out of 1 million concurrent hits to the edge cache, only 1 request is permitted to originate an Origin-fetch to the core database.
4. Blast Radius & Trade-offs
- Trade-off: Bartered 60 seconds of Data Staleness (TTL) in exchange for absolute system Availability. Strong Consistency was formally abandoned in this scenario and downgraded to Eventual Consistency.
- Foreshadowing / Hidden Threat Warning: Currently, all edge caches utilize a fixed TTL invalidation mechanism. Should future business logic heavily rely on caching, at the exact instant the TTL expires concurrently (the Cache Invalidation Point), the system will face a terrifying Thundering Herd (Origin-fetch storm). Once the blast doors open, the database behind them will be entirely exposed.
Architect's Note: Connecting Past and Modern System Design
1. The Cruelest Law of Distributed Systems: First Encounter with the CAP Theorem In the "extremely cold" decision Simon made in this chapter (delaying news by a minute), he actually touched upon an inviolable iron law in modern distributed system architecture, akin to gravity itself—the CAP Theorem. The CAP Theorem dictates that in a distributed computing system, it is impossible to simultaneously and perfectly satisfy the following three guarantees:
- C (Consistency): Every read receives the most recent write (e.g., every refresh instantly shows the newest second of news).
- A (Availability): Every request receives a non-error response (the web page loads without a 503, even under immense system pressure).
- P (Partition Tolerance): The system continues to operate despite arbitrary delays or drops in network communication between nodes (the congestion caused by a traffic flood is essentially a type of P).
Under extreme traffic like 9/11 (a severe P, where networks are jammed full), an architect must make a bleeding choice between C and A. Simon chose to sacrifice Strong Consistency (C), enduring 60 seconds of stale data, to forcefully preserve the absolute Availability (A) of the system. Even today, when you face scaling systems or trending topics claiming to handle "hundreds of millions of concurrencies," the like counts you see are rarely absolute real-time, and different people might even see different numbers. That's not a Bug; that's the "staleness tradeoff" senior architects secretly hand over to the CAP Theorem so you don't face an error page.
2. The Magic of Modern CDNs: Edge Computing The script Simon hand-rolled back then is the most primitive model of a modern CDN (Content Delivery Network): Pushing data to the server room closest to the users. In modern system design, CDN methodologies are far more advanced and terrifying:
- Pull Origin Model: Today's CDNs don't require you to write scripts pushing static pages. All business APIs can sit behind a giant reverse proxy (like Nginx, Varnish) or Cloudflare. When the CDN detects that its local cache has expired (a Cache Miss), it actively "Pulls" data from the backend once, and then feeds it downstream to the subsequent army of millions.
- Edge Computing: This is something Simon couldn't have imagined in that era. Today's CDN nodes not only return images and static HTML but can even execute lightweight code (like Cloudflare Workers / AWS Lambda@Edge). It can directly handle complex A/B test routing, user authentication, or simple rate-limiting anti-fraud actions right in your neighborhood ISP server room fifty miles away, before the request has even crossed the ocean. True elite architecture isn't just about making the system "compute faster"; it's about giving requests absolutely no chance to even approach the headquarters' database. Intercepting and killing them at the extreme outer perimeter is the most gentle protection you can afford your core system.