To find flaky tests reliably, you need history, not a single red build. A flaky test passes and fails on the same code, so the evidence is spread across many CI runs and re-runs that nobody looks at together. Almost every test framework can write JUnit XML, which makes it the one format you can analyse across a whole polyglot repository. This guide shows how to produce JUnit XML from the common runners, upload it with the commit and attempt number, collect weeks of reports with the GitHub CLI, score flakiness with a short Python script, publish a weekly report, and then reproduce and quarantine the worst tests.

Find flaky tests pipeline: CI uploads JUnit XML with commit metadata, a weekly job downloads run history, a Python script scores pass and fail flips and publishes a report
Flaky tests hide in history: the same commit, the same test, one pass and one failure.

From my own work: On the projects where I set this up, the team usually had a feeling about which tests were flaky, and the feeling was half right. The data found a few tests nobody suspected, because a retry setting had been quietly passing them on the second attempt for months.

What makes a test flaky

A test is flaky when it gives different results on the same code and environment. That definition gives three signals you can compute from reports:

Signal How it shows up Strength
Mixed results on one commit Fails on run attempt 1, passes on attempt 2 with the same SHA Strongest: same code, different outcome
Passed on retry A runner retry turned a failure into a pass, for example Surefire flakyFailure elements Strong, and invisible if you only read the final status
Flip rate on the main branch Pass, fail, pass, fail across consecutive runs of unrelated commits Good hint; confirm by reproducing

A test that fails consistently from one commit onwards is not flaky; it is broken by that commit. The script below ignores that pattern so the report stays focused.

JUnit XML in one minute

There is no single official schema, but runners agree on the core elements: a testsuites or testsuite root, one testcase per test with classname, name and time, and a failure, error or skipped child when the test did not pass.

<testsuites>
  <testsuite name="tests" tests="3" failures="1" errors="0" skipped="0" time="2.41">
    <testcase classname="tests.test_checkout" name="test_apply_coupon" time="0.12"/>
    <testcase classname="tests.test_checkout" name="test_payment_webhook" time="1.98">
      <failure message="TimeoutError: webhook not received within 1s">...</failure>
    </testcase>
    <testcase classname="tests.test_cart" name="test_totals" time="0.31"/>
  </testsuite>
</testsuites>

Maven Surefire adds something extra when you enable rerunFailingTestsCount: a test that failed and then passed on a rerun gets flakyFailure or flakyError children, while one that failed every time keeps failure plus rerunFailure elements (Maven Surefire: rerun failing tests). The script treats the first case as a flaky signal.

Step 1: write JUnit XML from every test runner

# pytest (built in)
pytest --junitxml=reports/pytest.xml

# Jest (npm install --save-dev jest-junit)
JEST_JUNIT_OUTPUT_DIR=reports npx jest --ci --reporters=default --reporters=jest-junit

# Playwright (built-in junit reporter)
PLAYWRIGHT_JUNIT_OUTPUT_NAME=reports/playwright.xml npx playwright test --reporter=junit,line

# Go (go install gotest.tools/gotestsum@latest)
gotestsum --junitfile reports/go.xml -- ./...

# Maven and Gradle write it by default
#   target/surefire-reports/TEST-*.xml
#   build/test-results/test/*.xml

Playwright can also set the output file in playwright.config.ts with reporter: [['line'], ['junit', { outputFile: 'reports/playwright.xml' }]] (Playwright: reporters). pytest documents its JUnit options, including how to record extra properties, in its output guide (pytest: JUnit XML).

Step 2: upload reports with run metadata

The analysis needs to know which commit and which attempt produced each report. Write a tiny meta.json next to the XML and upload both, even when tests fail. Including run_attempt in the artifact name keeps re-runs of the same workflow apart, and re-runs are the best flakiness evidence you have.

# .github/workflows/ci.yml (test job, after the test step)
      - name: Save run metadata
        if: always()
        run: |
          mkdir -p reports
          printf '{"sha":"%s","run_id":"%s","attempt":"%s","ref":"%s"}n' 
            "$GITHUB_SHA" "$GITHUB_RUN_ID" "$GITHUB_RUN_ATTEMPT" "$GITHUB_REF_NAME" > reports/meta.json

      - name: Upload JUnit reports
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: junit-${{ github.run_id }}-${{ github.run_attempt }}
          path: reports/
          retention-days: 30
          if-no-files-found: warn

If the test job uses a matrix, add the matrix values to the artifact name and to meta.json, for example junit-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.node }}, so a failure that only happens on one runtime is not averaged away.

Step 3: collect run history with the GitHub CLI

gh run download fetches the artifacts of a run, and --pattern limits it to matching names. When several artifacts are downloaded, each lands in its own folder named after the artifact (gh run download).

#!/usr/bin/env bash
# scripts/collect-junit.sh: download JUnit artifacts from recent CI runs
set -euo pipefail
WORKFLOW=${WORKFLOW:-ci.yml}
LIMIT=${LIMIT:-100}
mkdir -p history

gh run list --workflow "$WORKFLOW" --limit "$LIMIT" 
  --json databaseId,status --jq '.[] | select(.status=="completed") | .databaseId' |
while read -r id; do
  [ -d "history/$id" ] && continue                      # already downloaded
  gh run download "$id" --pattern 'junit-*' --dir "history/$id" 2>/dev/null 
    || echo "no JUnit artifacts for run $id" >&2
done
echo "reports collected: $(find history -name meta.json | wc -l)"

Including pull request runs adds useful data, because developers often re-run a failed PR check without changing anything. Filter with --branch main if you only want the default branch.

Step 4: score flaky tests with Python

The script reads every meta.json and the XML files beside it, records one outcome per test per run, then flags tests that passed and failed on the same commit, passed only after a retry, or flipped repeatedly. It uses only the standard library.

#!/usr/bin/env python3
"""flaky.py: find flaky tests in JUnit XML reports from many CI runs.

usage: python3 scripts/flaky.py history --min-runs 5 --out flaky.md
Each folder with a meta.json ({"sha", "run_id", "attempt"}) holds that run's XML files.
"""
import argparse, json, os, pathlib, sys
import xml.etree.ElementTree as ET
from collections import defaultdict


def outcome(tc):
    tags = {child.tag for child in tc}
    if tags & {'failure', 'error'}:
        return 'fail'
    if 'skipped' in tags:
        return 'skip'
    if tags & {'flakyFailure', 'flakyError'}:   # Surefire: failed, then passed on rerun
        return 'retried'
    return 'pass'


def load(root):
    rows = []
    for meta_file in pathlib.Path(root).rglob('meta.json'):
        meta = json.loads(meta_file.read_text())
        run = (int(meta['run_id']), int(meta.get('attempt', 1)))
        for xml_file in meta_file.parent.rglob('*.xml'):
            try:
                tree = ET.parse(xml_file)
            except ET.ParseError as exc:
                print(f'skipping {xml_file}: {exc}', file=sys.stderr)
                continue
            for tc in tree.iter('testcase'):
                key = f"{tc.get('classname', '')}::{tc.get('name', '')}"
                rows.append((key, meta['sha'], run, outcome(tc)))
    return rows


def score(rows, min_runs):
    by_test = defaultdict(list)
    for key, sha, run, result in rows:
        if result != 'skip':
            by_test[key].append((run, sha, result))
    report = []
    for key, obs in by_test.items():
        if len(obs) < min_runs:
            continue
        obs.sort()
        fails = sum(r == 'fail' for _, _, r in obs)
        retried = sum(r == 'retried' for _, _, r in obs)
        per_sha = defaultdict(set)
        for _, sha, r in obs:
            per_sha[sha].add('fail' if r == 'fail' else 'pass')
        mixed = sum(1 for s in per_sha.values() if len(s) == 2)
        seq = [r == 'fail' for _, _, r in obs]
        flips = sum(a != b for a, b in zip(seq, seq[1:]))
        if mixed or retried or (fails and flips >= 3):
            report.append({'test': key, 'runs': len(obs), 'fails': fails, 'retried': retried,
                           'mixed_commits': mixed, 'flip_rate': round(flips / (len(obs) - 1), 2)})
    return sorted(report, key=lambda r: (r['mixed_commits'] + r['retried'], r['flip_rate']),
                  reverse=True)


def markdown(report):
    lines = ['| Test | Runs | Fails | Passed on retry | Mixed commits | Flip rate |',
             '|---|---:|---:|---:|---:|---:|']
    for r in report:
        lines.append(f"| `{r['test']}` | {r['runs']} | {r['fails']} | {r['retried']} "
                     f"| {r['mixed_commits']} | {r['flip_rate']} |")
    return 'n'.join(lines) if report else 'No flaky tests found.'


if __name__ == '__main__':
    ap = argparse.ArgumentParser()
    ap.add_argument('root')
    ap.add_argument('--min-runs', type=int, default=5)
    ap.add_argument('--out', default='flaky.md')
    ap.add_argument('--fail-over', type=int, default=-1, help='exit 1 if more flaky tests than this')
    args = ap.parse_args()
    report = score(load(args.root), args.min_runs)
    text = '## Flaky testsnn' + markdown(report) + 'n'
    pathlib.Path(args.out).write_text(text)
    if os.environ.get('GITHUB_STEP_SUMMARY'):
        with open(os.environ['GITHUB_STEP_SUMMARY'], 'a') as fh:
            fh.write(text)
    print(text)
    sys.exit(1 if 0 <= args.fail_over < len(report) else 0)

The report reads like this, with the strongest evidence first:

Column What it tells you
Mixed commits Commits on which the test both passed and failed; any value above zero is near proof of flakiness
Passed on retry Runs where a runner retry hid a failure
Flip rate Share of consecutive runs where the outcome changed; a broken test has a flip rate near zero
Fails / Runs How often it hurts the team; use it to prioritise fixes

The reports come from your own CI, so the standard library parser is fine; if you ever parse XML from untrusted sources, use defusedxml instead.

Step 5: publish a weekly flaky test report

# .github/workflows/flaky-report.yml
name: Flaky test report
on:
  schedule:
    - cron: '0 6 * * 1'      # Mondays 06:00 UTC
  workflow_dispatch:
permissions:
  actions: read
  contents: read
jobs:
  report:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: '3.14'
      - name: Download JUnit history
        env:
          GH_TOKEN: ${{ github.token }}
        run: bash scripts/collect-junit.sh
      - name: Score flaky tests
        run: python3 scripts/flaky.py history --min-runs 5 --out flaky.md
      - uses: actions/upload-artifact@v7
        with:
          name: flaky-report
          path: flaky.md

The table appears in the job summary every Monday. Teams that want a hard rule can add --fail-over 10 so the job turns red when the flaky list grows beyond an agreed size.

This fits beside other report-driven checks: the same artifact-and-summary approach is used for performance budgets in the Lighthouse CI JSON report guide and for bundle size budgets in CI.

Step 6: reproduce, fix or quarantine

A flaky test is a bug in the test, the code or the environment, and the first job is to reproduce it by running it many times in isolation:

# pytest (pip install pytest-repeat); add -p randomly with pytest-randomly to shuffle order
pytest tests/test_checkout.py::test_payment_webhook --count=50 -x

# Playwright: repeat one test and keep the traces of failures
npx playwright test checkout.spec.ts --repeat-each=30 --workers=4 --trace=retain-on-failure

# Go
go test ./checkout -run TestPaymentWebhook -count=100 -race

# Jest: run one file repeatedly in a loop
for i in $(seq 1 30); do npx jest checkout.test.ts --silent || break; done
Common cause Typical symptom Fix
Fixed sleeps and timing Fails more on slow CI runners Wait for a condition or event, not a duration
Shared state and test order Fails only when run with other tests Isolate data per test; shuffle order locally to expose it
Dates, time zones and clocks Fails near midnight or month end Freeze or inject the clock
Real network calls Timeouts, rate limits Mock external services at the boundary
Concurrency and races Rare, random failures Run with race detectors and higher parallelism to reproduce
Unordered results Assertions on list order fail sometimes Sort before comparing or assert on sets

If you cannot fix a test this week, quarantine it openly: move it to a separate job that does not block merges, open an issue with an owner and a date, and keep it in the report. Retries are a tool for evidence, not a cure; Playwright already labels tests that pass only on retry as flaky in its reports, and those labels deserve the same attention. End-to-end suites are the usual hotspot, so the isolation advice in the Playwright and MSW testing guide pays off quickly, and the plugin matrix in the GitHub Actions CI guide for WordPress plugins is a good place to add JUnit uploads.

Checklist to find flaky tests

  • Every test runner writes JUnit XML into one reports folder.
  • Reports and meta.json uploaded with if: always(), named by run ID, attempt and matrix.
  • History collected weekly with gh run download.
  • Tests scored on mixed results per commit, passes on retry and flip rate.
  • The worst tests reproduced with repeat runs, then fixed or quarantined with an owner.

Want this done on your own server or site?

I am Saizul Amin, a web developer and technical SEO consultant. I set up and harden VPS servers, migrate WordPress sites, fix indexing problems and build automations. See my portfolio or get in touch.

FAQ

How do I find flaky tests in CI?

Save JUnit XML reports from every run together with the commit SHA and run attempt, collect them over a few weeks, and look for tests that both passed and failed on the same commit or passed only after a retry. Those are flaky by definition.

Why use JUnit XML for flaky test detection?

Nearly every test runner can write it, including pytest, Jest, Playwright, Go tools, Maven and Gradle, so one script can analyse a repository that mixes languages and frameworks.

Are test retries a good fix for flaky tests?

Retries keep the pipeline moving but hide the problem. Keep them if you need them, but record tests that pass only on retry and treat them as bugs to fix, because they still waste time and hide real failures.

How many runs do I need to detect a flaky test?

There is no fixed number. One commit with both a pass and a failure is already strong evidence. For flip-rate signals, a few dozen runs of the same test give a much clearer picture than a handful.

What should I do with a flaky test I cannot fix right away?

Quarantine it in a non-blocking job, open an issue with an owner and a deadline, and keep it in the weekly report. Do not delete it silently, because it may be revealing a real race condition in the application.

Sources

NEW BLOGS. REAL STRATEGIES. REAL RESULTS.

Join our community and receive powerful SEO tips, web optimization guides, and growth strategies as soon as they’re published.

Don’t just read blogs — stay ahead of the competition.

We don’t spam! Read our privacy policy for more info.

Share.
Leave A Reply