The client is an online education platform with interactive lessons, homework, student and parent accounts, and online payments. The business is strongly seasonal: September brings the most new students, and platform downtime in the first weeks of the school year means losing customers for good. There was no in-house QA team; developers tested between other tasks. Pre-release regression took three days and still missed critical scenarios such as payment, lesson booking and joining a class. The client came to us after another incident in which a payment bug cost the school some of the leads it had collected over the summer.
Establish quality assurance from scratch:
a test strategy prioritized by business risk, automation of key scenarios, CI-integrated checks and load testing before the September peak. The constraint was to keep development running: releases continued every week, and September could not wait.
We started with risk analysis, not automated tests: which scenarios broke most often, and where did a bug cost the most? Automation followed that list strictly.
- Audited quality: reviewed bugs from the previous six months, interviewed teams and mapped the most painful scenarios—payment, lesson booking, joining a class and parent notifications.
- Wrote a test strategy with testing levels, ownership areas, a definition of done for features and bug-reporting procedures shared by all three development teams.
- Automated the critical path: end-to-end Playwright scenarios for the web platform, API tests for the backend and a smoke suite before every release.
- Integrated QA gates into CI: a release cannot proceed without a green smoke suite, full regression runs nightly, and results are visible to all teams in GitLab.
- Load-tested with k6 by modelling September peak traffic, finding bottlenecks and resolving them with the backend team before the season.
- Handed the process over to the client: documentation, procedures and training for the QA engineer in automated tests and the pipeline.
The key decision was to prioritize by business risk, not by “coverage elegance”. We deliberately did not pursue 100% automation: 75% coverage of critical scenarios delivered most of the effect because it protected the paths on which the platform earns revenue and loses customers. The rest is covered by manual exploratory testing before significant releases.
The main economic effect is shifting defect detection earlier: a bug caught by a CI automated test costs dozens of times less than one noticed by a student’s parent in production. The platform entered the season with a safety margin, and teams gained the ability to release changes frequently without fearing the “Friday effect”. The business also gained confidence: sales can now honestly promise partner schools a stable platform during peak enrolment.
| Metric | Result |
|---|---|
| Critical production bugs | 5–7 per month → 1–2 per quarter |
| Pre-release regression | 3 days by hand → 4 hours with automated tests |
| September traffic peak | Incidents every year → 0 incidents, no degradation |
| When defects are detected | Usually in production → Automated tests catch them before release |
Metrics are transcribed from the original case materials. A period is stated where it is recorded in the source.
- critical bugs a quarter (was 5–7 a month)
- 1–2
- regression with autotests (was 3 days by hand)
- 4 h
- of key scenarios covered by autotests
- 75%
- incidents in the September peak
- 0
Illustrative scale: 3 days are converted to 72 hours. For bar lengths, the monthly bug range is scaled to a quarter by multiplying by three; upper bounds are compared.
- 01Commit
- 02Smoke suiteno green, no release
- 03Releaseweekly
- 04Nightlyfull regression
Our recommended next steps for the platform:
- Visual regression testing to automatically check the layout of lessons, emails and personal accounts.
- A test-data factory to generate realistic students, groups and payments for each run.
- QA in production: feature flags and canary releases for gradually enabling new features.
We stopped being afraid of releases. Before, any Friday could turn into an investigation of complaints; now a production release is a routine fifteen-minute procedure.
— CTO of the Online School