“Deploying to 10 users and deploying to 10,000 users are two completely different engineering problems.”
When you have 10 users, you can restart your backend container in the middle of a Tuesday afternoon. If an API request drops, the user hits refresh. If a deployment introduces a 3-second latency bump, nobody files an incident report.
When you run an EdTech Learning Management System (LMS) serving 10,000+ daily active students preparing for cutthroat competitive exams like IIT JEE and NEET, deployment is no longer just a technical routine—it is high-stakes risk management.
Students in India taking a timed 3-hour computer-based mock exam have zero patience for technical hiccups. Many are connected via spotty 4G mobile data in tier-2 and tier-3 towns. If an API call that normally takes 300ms suddenly takes 4 seconds during a rolling deployment, their browser appears frozen for 15 seconds. Panic sets in, the student frenetically taps the submit button, takes a screenshot, and floods customer support with tickets: “Your portal hung and my exam timer expired!”
Here is the unfiltered production case study of how our 3-person engineering team (myself as Tech Lead alongside two developers/DevOps engineers) scaled and deployed a 14-microservice architecture, survived Sunday mock-test traffic surges and Shark Tank broadcast spikes, and built a zero-downtime canary deployment pipeline without resetting a single student's exam.
1. The Anatomy of Our 14-Microservice Platform
Our platform isn't a static brochure site; it's a complex distributed engine handling real-time study sessions, video streaming, payments, testing simulations, and heavy analytical rank calculations.
[ 10,000+ Active Students (Mobile & Web) ]
│
[ Cloudflare DNS & CDN ]
│
[ AWS ALB / NGINX Ingress ]
│
┌─────────────────────────────┼─────────────────────────────┐
▼ ▼ ▼
[ Gatekeeper Auth ] [ Course & Explore ] [ Payment Service ]
(JWT / OTP Engine) (GraphQL + Redis Cache) (Isolated / Razorpay)
│ │ │
└─────────────────────────────┼─────────────────────────────┘
│
┌─────────────────────┴─────────────────────┐
▼ ▼
[ Test Management Service ] [ Analytics & Results ]
- Timed CBT Exam Engine - Percentile Ranking
- Adaptive Question Pager - Top 10/20% Benchmarking
- In-Memory Redis Buffer - BullMQ Worker Engine
│ │
└─────────────────────┬─────────────────────┘
│
┌──────────────┴──────────────┐
▼ ▼
[ Redis Buffer & Cache ] [ PostgreSQL Primary ]
(Sub-millisecond writes) (Single Vertically Scaled)
│ │
└────────── Sync ─────────────┘Our daily student traffic follows three distinct biological study cycles:
Early-morning syllabus revision, chapter notes, and mock tests. Peak concurrency sits between 3,000 to 5,000 students.
Lecture video streaming, homework submission, and doubt resolution. Steady read-heavy load on GraphQL services.
Intensive late-night problem solving. Traffic remains surprisingly dense right up until 2:00 AM before quieting down.
2. The Two Monsters: Sunday 9:50 AM & The Shark Tank Broadcast
Standard day-to-day traffic is predictable. What pushes engineering architectures to the edge are sudden, violent surges. We had two major flashpoints to plan for:
The Sunday 10:00 AM JEE/NEET Mock Test
Every Sunday morning, thousands of students sit for a scheduled mock exam designed to mirror the exact National Testing Agency (NTA) computer-based test. We initially assumed our Payment Microservice would bear the brunt of this spike. But in reality, students had already purchased the test packages days in advance.
The real inferno struck our Test Management Microservice. At precisely 10:00:00 AM, thousands of concurrent requests hammered:
POST /api/v1/test/start • GET /api/v1/test/questions • POST /api/v1/test/submit-answerTo survive, we established our Sunday 9:50 AM Pre-Scaling Protocol. At 9:50 AM sharp, the engineering team logged onto AWS to warm up caches, scale connection pools, and monitor container CPU baselines 10 minutes prior to exam launch.
The Shark Tank Airing Emergency
One of our coaching partners was featured on television in an episode of Shark Tank. When an episode airs on national television, there is no gradual ramp-up—traffic spikes 15× within 60 seconds as curious viewers hit the landing pages, explore course catalogs, and test checkout funnels.
Because our Payment service was fully isolated from our core application logic, we pre-scaled public explore APIs, notified Razorpay in advance to ensure our merchant webhook and transaction limits were pre-warmed, and held our ground without dropping a single sale.
3. Pre-Launch Load Testing Without SDETs: How We Simulated 30,000 Users with k6
A common startup myth is that you need a dedicated team of Software Development Engineers in Test (SDETs) to properly stress-test a high-concurrency platform. We had three engineers. As Tech Lead, I decided we would engineer our own automated testing pipeline using k6, Prometheus, and Grafana.
Before allowing a single live student onto the new Test Management microservice, we designed end-to-end user journey simulation scripts in k6 that faithfully mirrored how real human students interact with our system:
Students don't all log in at 9:59 AM. Some log in at 8:00 AM, others at 9:15 AM. Our k6 scripts simulated staggered authentication, teaching us that our Auth service (Gatekeeper) had to be pre-scaled 2–3 hours before exams to prevent OTP verification bottlenecks.
Virtual users navigated from Dashboard → Explore Catalog → Enrolled Tests → Test Overview → Question Fetching → Answering → Submissions → Result Polling.
We ramped our k6 load tests progressively:
1,000 VUs ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░ [Baseline Latency: 42ms] 5,000 VUs ████████░░░░░░░░░░░░░░░░░░░░░░░░ [DB CPU: 18%, Redis: 12%] 10,000 VUs ██████████████░░░░░░░░░░░░░░░░░░ [Current Production Peak: 95ms P95] 15,000 VUs ████████████████████░░░░░░░░░░░░ [PostgreSQL connection pool stable] 20,000 VUs ██████████░░░░░░░░░░░░░░░░░░░░░░ [Single DB Instance Verified] 30,000 VUs ████████████████████████████████ [Absolute Stress Ceiling: Passed]
Even though our production peak was around 10,000–12,000 users, testing up to 30,000 virtual users gave us the safety margin needed to sleep peacefully before major exam releases.
4. Client-Side Resilience: Adaptive Pagination & IndexedDB Write-Ahead Logging
Many engineering teams mistakenly try to solve all scale issues solely on the backend. But in an EdTech application where students operate on unreliable networks, the frontend architecture is your first line of defense.
Adaptive Question Pagination
Downloading an entire 180-question JEE exam paper with complex SVG diagrams, chemical formulas, and mathematical notation at once caused major timeouts on 3G connections. We implemented an adaptive network-aware paginator:
IndexedDB Write-Ahead Log: Surviving Student Internet Drops
What happens when a student is in the middle of a mock test and their mobile carrier drops connectivity? In a naive React app, submitting an answer fails, an error toast appears, and a browser reload resets their progress.
We treated the student's browser as an offline-first distributed node using browser IndexedDB:
Every time a student selects an option (A, B, C, D), it is immediately committed to local IndexedDB storage before any network request is triggered. Even if the student loses power or loses connection for 40 minutes, when internet returns, the client silently flushes all pending answers to our server in a single batch.
30-Second Batching: Eliminating 80% of API Writes
Instead of firing an HTTP POST every time an option is clicked, our IndexedDB background syncer polls every 30 to 60 seconds. If a student answers 5 questions during that 60-second window, they are packed into one single HTTP payload:
{
"test_session_id": "sess_89f7a1c",
"client_timestamp": "2026-10-04T10:14:30Z",
"answers": [
{ "question_id": "q_101", "selected_option": "B", "time_spent_seconds": 45 },
{ "question_id": "q_102", "selected_option": "D", "time_spent_seconds": 78 },
{ "question_id": "q_103", "selected_option": "A", "time_spent_seconds": 32 }
]
}If no questions were attempted during the 30-second window, zero API requests are made. This alone reduced write pressure on our backend by over 80%.
5. Backend Architecture: The Redis Write Buffer That Saved PostgreSQL
Even with client-side batching, 5,000 concurrent students can generate hundreds of batch writes per second. In a standard relational database setup, writing each batch directly to PostgreSQL tables causes severe row-lock contention, index fragmentation, and connection pool exhaustion.
Instead of writing directly to PostgreSQL, we implemented an in-memory Redis Write-Back Buffer:
Student Batch Submission
│
▼
[ Test Management API ] ──(Immediate In-Memory Write < 2ms)──► [ Redis Write Buffer ]
│ │
(Return HTTP 200) │
(Buffered Queue every 30-60s)
▼
[ Sync Worker Process ]
│
(Bulk SQL INSERT/UPSERT)
▼
[ PostgreSQL Primary ]When an answer batch arrives at the backend:
By buffering writes in Redis and flushing in batches, we avoided complicated distributed database sharding or multi-master databases. Our entire test platform ran stably on a single vertically scaled PostgreSQL instance, comfortably supporting 20,000 tested users without a sweat.
6. Asynchronous Result Analytics: Decoupling Compute with BullMQ
The most dangerous moment in an online exam system is not when the exam starts—it is when the exam ends.
If 4,000 students submit their exams at exactly 12:00:00 PM, and your system attempts to immediately calculate each student's marks, negative marking deductions, percentile ranking against all other participants, subject-wise strengths, and generate a downloadable PDF report card, your servers will crash instantly.
The 60-Minute Grace Window
We made a crucial product decision: stop trying to show results immediately. Real NEET and JEE exams do not release percentiles the second the timer hits zero. We announced to students:
This gave our backend a 90-minute window to handle computation calmly via BullMQ (Redis-backed distributed worker queues):
BullMQ workers ingest answer keys, compute standard deviations, score distributions, and assign percentiles against top 10% and top 20% cohorts without blocking HTTP threads.
A dedicated BullMQ worker pool renders personalized scorecard PDFs in the background, pushes them to AWS S3, and dispatches email notifications with download links.
7. The Deployment Dilemma: Why Rolling Failed & Why Canary Won
As our daily active user count passed 10,000, our release cadence was steady: 1 to 2 deployments a day, ~5 deployments a week.
Our old deployment method was standard Rolling Updates (replacing container instances one by one behind the load balancer). But at scale, rolling deployments began exhibiting dangerous flaws:
- In-Flight Request Terminations: When containers were recycled, students with ongoing HTTP keep-alive connections received 502 Bad Gateway errors.
- Dual-Version Cache Inconsistencies: For a 5-minute rolling window, both V1 and V2 instances were serving traffic simultaneously. If V2 updated a Redis cache key structure, V1 instances threw runtime null pointer exceptions.
- Student Anxiety: If a deployment happened while a student was reading notes, the 2-second reload delay resulted in immediate panic.
Comparing the Strategic Options
| Evaluation Factor | Recreate | Blue-Green | Canary (Our Choice) |
|---|---|---|---|
| Customer Downtime | 1–3 minutes | Zero visible | Zero visible |
| Infrastructure Cost | 1x baseline | 2x baseline (Double infra) | Fractional (+10–15%) |
| Blast Radius | 100% of students | 100% upon switch | Capped at 10% |
| Rollback Speed | Slow (5–10 min rebuild) | Fast (DNS/LB swap) | Instant (< 60 seconds) |
| Suitability for 3 Engineers | Too primitive | Expensive for 14 services | Highly Pragmatic |
Why not Blue-Green?Running 14 microservices in a complete Blue-Green duplicate setup means running 28 service clusters concurrently during releases. For a lean startup, doubling the AWS bill for idle compute wasn't justifiable. Canary offered the perfect balance: zero downtime, low infrastructure overhead, and a tiny blast radius.
8. Our Production Canary Pipeline: 10% → 25% → 70% → 100%
We implemented weighted routing using NGINX and AWS Application Load Balancers (ALB) orchestrated through GitHub Actions.
Stage 1: 10% Traffic ──► [V2 Canary] ──► 10-minute Soak ──► Validate 5xx < 0.5% & P95 < 400ms
[V1 Stable: 90%]
│ Passed
Stage 2: 25% Traffic ──► [V2 Canary] ──► 15-minute Soak ──► Validate Redis memory & DB pool
[V1 Stable: 75%]
│ Passed
Stage 3: 70% Traffic ──► [V2 Canary] ──► 15-minute Soak ──► Full load verification
[V1 Stable: 30%]
│ Passed
Stage 4: 100% Traffic ─► [V2 Canary] ──► V1 instances gracefully drained & terminatedAutomated Health Scoring & The < 60s Rollback
During every soak window, our Prometheus and Grafana alerts monitor three mission-critical metrics:
Must remain strictly below 0.5%.
Must not exceed 400ms.
Heap memory must follow a clean sawtooth pattern.
If any health metric breaches threshold during the 10% canary stage, the GitHub Actions deployment pipeline triggers an automated rollback command. In NGINX, traffic weight to V2 drops to 0% immediately. Mean Time to Recovery (MTTR) is less than 60 seconds, and 90% of students never even saw the buggy version.
9. Zero-Downtime Database Migrations: The Expand / Contract Pattern
Application code deployment is relatively simple compared to database schema migrations. If you deploy a new version of code that queries a column that doesn't exist yet in PostgreSQL, or if an ALTER TABLE command locks the test_submissions table while 5,000 students are writing answers, the platform goes down.
We strictly follow the Expand / Contract Pattern across multiple releases:
NULLABLE or with safe defaults. Never rename a column directly in production.10. The Production Scorecard: Before vs. After
Here is how our deployment and operational metrics changed after implementing this architecture:
| Production Metric | Before (Traditional Rolling) | After (Canary + Buffer) |
|---|---|---|
| Customer-Visible Downtime | 10–45 seconds per deploy | 0.0 seconds |
| Rollback Duration (MTTR) | 8–15 minutes | < 60 seconds |
| Blast Radius of Buggy Release | 100% of active students | 10% isolated |
| DB Write Pressure During Exams | Thousands of direct SQL writes/sec | Buffered; 1 flush / 30s |
| Overall Platform Availability | 98.9% | 99.9% Uptime |
| Engineering Release Confidence | Deployment anxiety & dread | 5 safe deploys / week |
11. What We Learned as a 3-Person Team
1. You Don't Need Complex Distributed DBs to Scale
Before jumping to multi-master database clusters or exotic sharding solutions, ask if an in-memory buffer (Redis) and client-side batching (IndexedDB) can solve your write bottlenecks. A single, well-tuned PostgreSQL instance can take you much further than you think.
2. In EdTech, Client-Side Resilience is Paramount
Never rely on student internet connections being reliable. Treat the browser as a write-ahead log. If the server is unreachable or rebooting during a deployment, the user should never notice.
3. Rollback Speed Matters More Than Test Perfection
No matter how many k6 scripts you run, production traffic will eventually uncover an unforeseen edge case. Having an automated canary gate that detects anomalies and rolls back in under 60 seconds is the ultimate safety net.