Flaky Tests: How to Find, Quarantine, and Fix Them

A flaky test is one that both passes and fails on the same code and configuration without an intentional change to the product under test. Flakes erode trust faster than slow suites: teams learn to re-run, ignore red builds, and eventually stop believing automation.

Suite selection—what belongs in smoke vs extended regression—lives in regression test cases. This article owns diagnosis, quarantine, and root-cause fixes for unstable automated checks. Do not expand the regression inventory to “cover” flakes; stabilize or remove them.

What counts as flaky (and what does not)

Flaky: intermittent failure under unchanged revision, environment profile, and test code.

Not flaky (even if annoying):

• Failures that always reproduce on a specific branch or config
• Environment outages (auth provider down, shared staging broken)
• Real product races that users can hit—those are defects wearing a test costume

Label carefully. Calling a real race “flake” hides a bug. Calling an infra outage “product flake” wastes engineering time.

Why flakes matter beyond CI noise

• Merge queues stall or get force-green habits
• Signal-to-noise collapses; real regressions hide in the noise
• On-call and QA burn cycles on re-runs instead of investigation
• New automation is discounted (“tests are always red”)

Reliability is part of the automation strategy in the automation testing guide—not a cleanup sprint you schedule after the suite is “done.”

Detection: make intermittency visible

You cannot fix what you classify as “weird today.”

Signals in CI

Watch for:

• Same test failing on retry then passing
• Failures correlated with specific runners, shards, or times of day
• Pass rate below a threshold over a rolling window (for example, any test under ~98% on main over two weeks deserves a ticket)
• Clustered failures after dependency or browser upgrades

Collect history, not vibes. Export case IDs, job URLs, shard indexes, and retry counts into a simple dashboard or even a spreadsheet owned by QA/SDET.

Local reproduction tactics

• Run the suspect test in a loop (--repeat-each / pytest-repeat equivalents)
• Run it in parallel with neighbors that share data
• Pin browser/device versions used in CI
• Compare headed vs headless only as a diagnostic, not as a “fix”

If it only fails in CI, capture artifacts: trace, video, HAR, logs, screenshots, API correlation IDs.

Common root-cause families

Timing and synchronization

The classic UI flake: assert before the app is ready. Prefer deterministic waits—element state, network idle when appropriate, response predicates—over fixed sleeps. For Selenium, explicit waits beat implicit soup; see Selenium explicit waits. In Playwright, stick to auto-waiting locators and avoid manual timeouts as a first resort (Playwright testing tutorial).

Shared mutable state

Global users, shared carts, singleton feature flags, or “the” admin project mutated by every job. Parallelism makes this explode. Isolate with unique data, per-worker credentials, and cleanup hooks.

Test data and seeds

Ord-dependent seed scripts, non-idempotent migrations, or fixtures that assume empty tables. Prefer create/cleanup per test or deterministic factories with unique keys.

Environment drift

Different browser versions, timezones, locale, feature flags, or third-party sandbox rate limits between local and CI. Pin versions; document required flags; fail fast when config is incomplete.

Selectors and UI coupling

CSS that mirrors visual design, brittle XPath, or text that marketing changes weekly. Prefer role/label/test-id strategies and page objects with a single locator definition (page object model).

Async and eventual consistency

Jobs, search indexers, webhooks, and caches. Poll with capped timeouts and clear errors. Do not sleep(5) and hope. If consistency SLOs are multi-minute, that check may belong in nightly, not PR smoke—align packs with smoke vs sanity intent.

Cross-browser differences

A pass on Chromium and flake on WebKit often points to animation, focus, or download behavior—not “WebKit is flaky.” Track browser in failure metadata; see cross-browser testing.

Infrastructure and resource contention

Starved CI runners, saturated staging DB, contested ports. Separate product flakes from capacity problems; scale runners or reduce parallelism before rewriting tests.

Quarantine without lying to yourself

Quarantine means: known intermittent tests do not block the primary gate temporarily, while remaining visible and owned.

Healthy quarantine:

• Explicit list or tag (@flaky, quarantine job)
• Ticket with owner, hypothesis, and expiry date
• Still running (nightly or non-blocking) so you keep signal
• Metrics: count of quarantined tests over time should trend down

Unhealthy quarantine:

• Permanent @skip with comment “flaky”
• Silent retries until green with no ticket
• Moving every red test into quarantine to protect a release date

Quarantine is a burn-down list, not a parking lot.

Retry abuse

Retries are a diagnostic amplifier and a narrow shield for known infra blips—not a substitute for fixes.

Guidelines:

• Allow limited retries at the job level for install/network bootstrap if needed
• Do not blanket-retry all tests forever
• Log every retry; treat repeated retries as a flake candidate
• Never hide retries from Xray/import reports if you need honest execution history—pair with Xray best practices

If a test needs three retries to pass regularly, it is not green. It is red with paperwork.

Root-cause workflow (repeatable)

Use the same loop every time:

1. Capture — failing job URL, artifacts, shard, browser, case ID, retry count.
2. Classify — timing, data, env, selector, async, infra, or real race.
3. Reproduce — loop locally or on a dedicated CI replay job.
4. Hypothesize — one change at a time (wait condition, unique data, pinned version).
5. Fix or quarantine — fix preferred; quarantine only with owner + expiry.
6. Verify — loop again; watch the next N CI runs on main.
7. Prevent — add lint/rules (no hard sleeps, no shared user) if the class repeats.

Write the classification on the ticket. Future you will thank present you.

Fixes that usually work (by class)

For API suites, isolation and rate-limit credentials matter more than selectors—see API test automation and keep API packs out of UI flake buckets when possible.

Prevention: design for determinism

Prevention beats heroics.

• Own test data lifecycle from day one
• Ban unbounded sleeps in review checklists
• Keep page objects thin and locators unique
• Separate smoke (fast, stable) from deep packs
• Run new UI checks in PR with traces enabled until trustworthy
• Prefer API setup for UI preconditions when the product allows
• Document environment contracts next to the suite README
• Review quarantine weekly in the same ritual as flaky triage

When stories expand coverage, draft structured cases first (including negative and edge intent), then automate the stable slice. QA Workflow Assistant helps turn Jira acceptance packs into consistent cases so automation does not invent steps that were never stable in manual form.

Metrics worth tracking

Keep a small set:

• Flake rate — intermittent failures / executions per test and suite
• Quarantine count — open quarantined tests and average age
• Mean time to resolution — ticket open to fix merged
• Retry rate — how often jobs rely on retries to go green
• Escape defects — production issues in areas “covered” by flaky tests (trust debt)

Do not vanity-track raw test count. A smaller stable suite beats a museum of intermittents—selection still follows regression test cases; reliability follows this playbook.

CI design that helps diagnosis

Pipeline choices affect flake hunting:

• Upload traces/videos/logs on failure always
• Record shard index and browser in the job name
• Fail fast on smoke; allow fuller matrices on nightly
• Keep a “repro” workflow that re-runs a single test ID

Wire these into GitHub Actions for test automation so every red build leaves a trail.

Worked example: the “sometimes empty list” UI test

Symptom: Orders page shows seeded order fails ~1 in 10 on PR, always passes locally headed.

Investigation trail:

1. Trace shows the table rendered empty; network panel shows list API returned 200 with items: [].
2. Seed step used a shared buyer account; another shard deleted or completed the order mid-run.
3. Fix: create order via API with runId-scoped payload in beforeEach, assert on that ID, delete in afterEach.
4. Verify: --repeat-each=50 green locally; watch next twenty CI runs on main.
5. Prevention: add a review rule—“no shared mutable commerce fixtures in parallel jobs.”

Notice the failure looked like a selector flake. The root cause was shared data. Classification before rewriting waits saved a wasted day.

Worked example: API “401 mid-suite”

Symptom: late tests in a long file fail with 401; early tests pass; retry of the whole file sometimes passes.

Investigation trail:

1. Token minted once in beforeAll with a short TTL.
2. Suite duration exceeded TTL under CI load; local runs were faster.
3. Fix: refresh helper in the client on 401 once, or mint per worker with longer test-only TTL.
4. Prevention: assert token expiry margin at suite start; fail with “token TTL too short for suite” instead of cascading auth errors.

Same pattern appears when refresh cookies rotate and parallel workers invalidate each other—isolate sessions per worker.

Triage rituals that stick

Ad-hoc Slack threads do not reduce flake debt. Cadence does.

Suggested weekly ritual (30–45 minutes):

• Sort top intermittent failures by count on main
• Assign owners or confirm quarantine expiry
• Close tickets where CI history is clean for the agreed window
• Note new classes (for example, a third-party sandbox rate limit) for environment owners

Publish a short running log in the team wiki: date, case ID, class, fix link. Over a quarter you will see the same classes dominate—that is your prevention backlog.

When the product is racing

Sometimes the test is honest: two clicks in quick succession corrupt state, or two API writers disagree without locking. That is not quarantine fodder.

Response:

• File a product bug with reproduction from the automated check
• Keep a single automated repro in a non-gating or bug-linked job until fixed
• Do not “stabilize” by adding arbitrary sleeps that hide the race
• After the fix, promote the check into the normal pack

Calling a real race a flake is how production incidents inherit CI’s blind spots.

Onboarding new automation without importing flakes

New contributors often copy the nearest failing pattern. Protect them:

• Provide a template test with approved wait and data patterns
• Document “do not use” examples (hard sleeps, shared users, CSS-nth-child locators)
• Require traces on first merge of a new UI file
• Prefer API preconditioning samples next to UI specs

Good templates are cheaper than retrospective flake festivals. Pair this with structured case drafting so steps were never ambiguous before coding started.

FAQ

Should we delete flaky tests?

If the risk is already covered by a stable check, delete or merge. If the risk is real and the test is the only coverage, quarantine and fix—do not delete silently.

Are flakes mostly UI?

UI sees more timing/selector issues, but API suites flake from shared data, rate limits, and eventual consistency. Diagnose by class, not by layer prejudice.

How long can a test stay quarantined?

Set an expiry (for example, two sprints). Expired quarantine without progress becomes a skip—and a process smell.

Do page objects reduce flakes?

They reduce locator drift when used well. They do not fix shared data or missing waits. Pair POM with synchronization discipline (page object model).

Should Selenium implicit waits be the default?

No. Prefer explicit, condition-based waits (Selenium explicit waits). Implicit waits hide timing bugs and slow failure modes.

How do flakes interact with release gates?

Gates should consume stable packs. Quarantined tests must not silently count as passed coverage in Xray dashboards—align execution hygiene with Xray best practices.

Where do we document environment-only flakes?

In the suite runbook and the ticket, not only in Slack. Environment setup and integrations belong in Docs.

Final checklist

• Flake definition shared by eng + QA
• CI history identifies intermittent case IDs
• Artifacts (trace/log/video) always on failure
• Quarantine has owners and expiry dates
• Retries are limited and visible
• Root-cause class recorded on each ticket
• Shared state and hard sleeps are review blockers
• Smoke stays smaller and stricter than extended packs
• Metrics reviewed on a fixed cadence
• Fixes verified with looped and CI replay runs

CTA — stable cases before brittle automation

Flakes often start as vague steps automated too early. Draft clear, structured cases from Jira stories in QA Workflow Assistant, automate the deterministic slice, and keep environment wiring honest via Docs. Reliability is a feature of the suite—not an afterthought after the matrix turns red.