November 29, 2024. Black Friday. The biggest sales day of the year. And my client's e-commerce site was dead.
Not "slow." Not "glitchy." Dead. 502 Bad Gateway. For fourteen hours. During the window where they expected to make 30% of their quarterly revenue.
This is the full postmortem. Not the sanitized version you'd put in a board presentation, but the real one — including every mistake I made, every assumption that turned out to be wrong, and exactly how I'd prevent it from happening again. Because if you're running any kind of web infrastructure, this will happen to you eventually. The question is whether you're ready.
The Setup: What We Were Running
My client, a mid-size fashion retailer, was running on what I considered a solid stack:
- 2x application servers on DigitalOcean (8GB RAM, 4 vCPUs each)
- 1x managed database (MySQL, 4GB RAM)
- Cloudflare for CDN and DDoS protection
- WooCommerce on WordPress (I know, I know)
- Redis for object caching
- Nginx as the web server
For their normal daily traffic of 3,000-5,000 visitors, this setup was overkill. I'd load tested it at 10x that volume and it handled it fine. What I didn't test for was what actually happened.
Timeline: How Everything Fell Apart
6:00 AM — The Calm Before
Everything looked perfect. Server metrics were green. Cloudflare was caching properly. I'd prewarmed the cache the night before. Had my coffee, checked monitoring dashboards, and felt smug about how prepared we were.
That smugness lasted about four hours.
10:14 AM — First Signs
Response times crept from 200ms to 800ms. Not alarming on its own — traffic was climbing as expected. Black Friday emails had gone out at 9 AM. But the curve was steeper than projected.
I checked the analytics: 12,000 concurrent users. More than double what I'd expected at this hour, but well within our load test parameters. So why was the site slowing down?
10:47 AM — The Database Problem
MySQL connections hit the ceiling. Not the server ceiling — the managed database's connection limit. DigitalOcean's 4GB managed database allows 150 concurrent connections. We were at 148 and climbing.
Here's what I didn't account for: WooCommerce with 15 active plugins generates a staggering number of database queries per page load. On a normal product page, I counted 127 queries. On the cart page, 203. Each with its own database connection that wasn't being pooled properly because our WordPress configuration wasn't using persistent connections.
10:52 AM — I Made It Worse
My first instinct was to restart PHP-FPM to clear hanging connections. Bad move. This killed all active sessions, which meant every customer with items in their cart got logged out. About 800 people lost their carts. Some of them had spent 30+ minutes building orders. We'd hear about this in customer support tickets for weeks.
11:15 AM — The Cascade
With the PHP restart, connections briefly dropped. Then they shot back up even faster because all those 800 logged-out users were now refreshing, logging back in, and re-adding items. Redis was also struggling now — the object cache hit its memory limit and started evicting keys, which meant more queries hitting MySQL directly.
At 11:23 AM, the first server went 502. At 11:31 AM, the second followed. Both Nginx instances were returning 502 Bad Gateway because PHP-FPM had exhausted its worker pool.
11:35 AM — The Panic Call
My client called. "The site is down." I'd been watching it happen in real-time but hearing a client say those words on Black Friday hits different. Real money. Real customers. Real consequences.
The 14-Hour Recovery: What I Actually Did
Hour 1-2: The Wrong Fix
I scaled the database to 8GB (300 connections). I doubled the PHP-FPM workers from 20 to 40 per server. I increased Redis memory from 512MB to 1GB.
The site came back briefly. For about 22 minutes. Then crashed again. Because I'd treated the symptoms, not the disease.
The real problem wasn't capacity — it was efficiency. No amount of hardware would fix 203 database queries per cart page load.
Hour 3-5: The Real Diagnosis
I enabled MySQL slow query logging and finally saw the culprit: a WooCommerce analytics plugin was running a full table scan on the orders table for every single page load. Not just admin pages — every. single. page. This plugin had been installed two weeks earlier and worked fine at normal traffic. Under Black Friday load, it was executing a query that took 3-8 seconds per request, blocking other connections.
I deactivated the plugin immediately via WP-CLI. Response times dropped by 60% within minutes.
Hour 5-8: Stabilization
The site was up but unstable. Traffic was still heavy (peaked at 18,000 concurrent), and we were hitting cache misses that shouldn't have been happening. Investigation revealed our Cloudflare page rules had a misconfiguration — the checkout path was being cached, which meant some customers were seeing other people's cart contents.
Yes, you read that right. Customer A could see Customer B's shipping address and order. This was a privacy incident on top of a performance incident. I immediately purged the Cloudflare cache and fixed the page rules to exclude /cart/*, /checkout/*, and /my-account/* from caching.
No customer reported seeing someone else's data, but we disclosed it to the client anyway. It was the right thing to do.
Hour 8-14: The Slow Climb Back
Even after fixing the core issues, rebuilding trust with the infrastructure took hours. MySQL connections were stable but I didn't trust the setup. I deployed connection pooling using ProxySQL as an intermediary between PHP and MySQL, limiting connections to 100 with queuing. This alone prevented any further connection storms.
By 1:30 AM Saturday morning — 14 hours after the first signs — everything was stable. The site stayed up through Cyber Monday without a hiccup.
The Damage Report
Let's talk numbers, because they hurt:
- Estimated lost revenue: $47,000 (based on previous year's Black Friday hourly sales during the outage window)
- Cart abandonment from forced logouts: ~800 customers
- Privacy incident: Unknown scope, no reports filed, but technically a GDPR-relevant event
- Emergency infrastructure costs: $340 (database upgrade, additional monitoring)
- Client trust: Severely damaged. They didn't leave, but it took months to rebuild.
The 7 Things I Do Differently Now
1. Load Test With Realistic Plugins
My original load test was on a clean WordPress install with WooCommerce. The production site had 15 plugins, each adding queries. Your load test must mirror your actual production environment, including every sketchy plugin the marketing team installed three months ago.
2. Database Connection Pooling Is Not Optional
ProxySQL or PgBouncer (for PostgreSQL) should be standard in any production setup expecting variable traffic. It costs almost nothing and prevents the most common database failure mode.
3. Cache Rules Need Security Review
We now audit Cloudflare/CDN caching rules monthly. Any path that serves user-specific content must be explicitly excluded. This includes not just checkout pages but API endpoints, AJAX handlers, and any path that sets cookies.
4. Kill Switch for Plugins
I maintain a ranked list of plugins that can be deactivated under emergency load. Analytics, social sharing, review popups — anything non-essential gets a kill switch I can trigger in 30 seconds via WP-CLI.
5. Pre-Scale, Don't React-Scale
For predictable traffic events (Black Friday, product launches, marketing campaigns), scale infrastructure 48 hours before, not when things break. I now double the database and add a third application server for every major sales event. The cost is minimal compared to the alternative.
6. Slow Query Monitoring Is Always On
Not just during incidents. Always. I use Percona Monitoring and Management (PMM) which catches query degradation before it becomes a crisis. The analytics plugin issue would have been caught in staging if we'd been monitoring query counts per page load.
7. Have a Communication Plan
I spent 20 minutes during the outage figuring out how to communicate with the client instead of fixing things. Now I have a template: first message within 5 minutes ("We're aware and working on it"), updates every 30 minutes, and a full postmortem within 48 hours.
Would This Have Happened on a Different Stack?
Partially. WooCommerce's query-heavy architecture made it worse, and the plugin ecosystem is a wild west of unoptimized code. On Shopify or BigCommerce, the database layer is abstracted away, so this specific failure mode wouldn't apply.
But the caching misconfiguration? That can happen on any stack. The connection pooling gap? Universal. The "didn't load test the actual production environment" mistake? I see it everywhere.
The stack matters less than the preparation. A well-prepared WooCommerce site can handle Black Friday. A poorly prepared Shopify site can have its own nightmare scenarios (I've seen apps cause infinite redirect loops during flash sales).
Final Thought
Every outage is a lesson you pay for in advance or in real-time. I paid in real-time, and it cost my client $47,000 and me several months of trust.
If you're reading this before your big traffic event: run the load test with your actual plugins. Set up connection pooling. Audit your cache rules. Have a kill switch list ready. Scale before you need to.
If you're reading this during an outage: check the slow query log first. It's almost always the database. And whatever you do, don't restart PHP-FPM during peak traffic. Learn from my mistake.
