ci(github): add Mergify flaky test detection workflow by ashleyshaw · Pull Request #983 · lightspeedwp/.github

ashleyshaw · 2026-06-18T12:24:26Z

Build/CI Pull Request

This PR updates the build or CI configuration for Pipelines, linting, packaging, or release automation.
Please review the summary, baseline/target, and changelog below.

This PR Template enforces pipelines, linting, changelog, release, and label automation for all PRs and issues.
See the organisation-wide Automation Governance & Release Strategy for required rules.

Linked issues

Closes #979

Build/CI change

What: Adds a scheduled GitHub Actions workflow (.github/workflows/flaky-test-detection.yml) that runs Jest unit tests 5 times in parallel using a matrix strategy and uploads JUnit XML results to Mergify for flaky test analysis.
Why: Flaky tests are detected by comparing outcomes across multiple runs on the same commit SHA. Running tests repeatedly and uploading results to Mergify enables automatic flaky test identification without manual investigation.

Baseline & Target

Before: No flaky test detection; intermittent test failures required manual investigation.
After: Automated detection via Mergify CI — 5 parallel Jest runs every 12 hours (Mon–Fri), results uploaded for statistical comparison.

Rollback

Plan: Delete or disable .github/workflows/flaky-test-detection.yml. No other files are affected.

Notes

Requires MERGIFY_TOKEN secret (with ci scope) to be present in repository settings.
jest-junit is installed at runtime (npm install --no-save jest-junit) since it is not in devDependencies.
MERGIFY_JOB_NAME set to check to match the existing PR CI job name in testing.yml.

Changelog

Added

Mergify flaky test detection workflow running Jest unit tests in 5 parallel matrix jobs every 12 hours on weekdays, with JUnit XML results uploaded to Mergify CI. (Closes Add Mergify flaky test detection to CI #979)

Checklist (Global DoD / PR)

🤖 Generated with Claude Code

https://claude.ai/code/session_01EtPYPZotcNDvxig8bNXqUh

Generated by Claude Code

Adds a scheduled GitHub Actions workflow that runs Jest unit tests 5 times in parallel (matrix strategy) every 12 hours on weekdays, uploading JUnit XML results to Mergify for flaky test analysis. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EtPYPZotcNDvxig8bNXqUh

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EtPYPZotcNDvxig8bNXqUh

github-actions · 2026-06-18T12:25:09Z

🔍 Reviewer Summary for PR #983

CI Status: ❌ success
Files changed: 2
Risk Distribution: 1 critical, 0 high, 0 medium, 1 low

Recommendations

⚠️ 1 critical-risk file(s) modified (workflows, secrets)
⚠️ Security-sensitive files modified (review carefully)

coderabbitai · 2026-06-18T12:25:13Z

Warning

Review limit reached

@ashleyshaw, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 20 minutes and 32 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more credits in the billing tab to continue.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: ac1207cb-2be4-4430-a03c-253f30268444

📥 Commits

Reviewing files that changed from the base of the PR and between 37c2327 and 0e7389d.

📒 Files selected for processing (2)

.github/workflows/flaky-test-detection.yml
CHANGELOG.md

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch ci/github-mergify-flaky-detection

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

gemini-code-assist

Code Review

This pull request adds a new entry to the CHANGELOG.md documenting the addition of Mergify flaky test detection. The reviewer suggested a minor grammatical improvement to fix a comma splice in the description of the new workflow.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-06-18T12:25:22Z


 ### Added

+- **Mergify flaky test detection** — Added `.github/workflows/flaky-test-detection.yml` to detect flaky Jest unit tests via Mergify CI. Runs 5 parallel matrix jobs every 12 hours on weekdays, uploads JUnit XML results to Mergify using `mergifyio/gha-mergify-ci@v14` with `flaky_test_detection: true`. ([#979](https://github.com/lightspeedwp/.github/issues/979))


The sentence has a minor grammatical issue (comma splice) between 'weekdays' and 'uploads'. Replacing the comma with 'and' improves readability and sentence flow. For example:

- **Mergify flaky test detection** — Added ".github/workflows/flaky-test-detection.yml" to detect flaky Jest unit tests via Mergify CI. Runs 5 parallel matrix jobs every 12 hours on weekdays and uploads JUnit XML results to Mergify using "mergifyio/gha-mergify-ci@v14" with "flaky_test_detection: true". ([#979](https://github.com/lightspeedwp/.github/issues/979))

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EtPYPZotcNDvxig8bNXqUh

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bab98ebe52

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-18T12:28:23Z

+      - run: npm install --no-save jest-junit
+
+      - name: Run unit tests
+        run: npx jest --config .jest.config.cjs --reporters=jest-junit --outputFile=junit-unit-${{ matrix.run_number }}.xml || true


Write the JUnit XML where Mergify looks

In this workflow, the upload step later points Mergify at junit-unit-${{ matrix.run_number }}.xml, but Jest's --outputFile only writes Jest JSON when --json is also set (per the Jest CLI docs), and jest-junit uses JEST_JUNIT_OUTPUT_FILE/JEST_JUNIT_OUTPUT_NAME or reporter options for XML output. On every scheduled/manual matrix run this will leave the expected XML absent or still at junit.xml, so gha-mergify-ci has no JUnit report to ingest for flaky-test analysis; set the JEST_JUNIT_* env vars or reporter options to this filename.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-18T12:28:23Z

+      matrix:
+        run_number: [1, 2, 3, 4, 5]
+    env:
+      MERGIFY_JOB_NAME: check


Pass the job-name override to the action input

This job-level MERGIFY_JOB_NAME does not reach Mergify as intended because mergifyio/gha-mergify-ci@v14 sets MERGIFY_JOB_NAME inside its composite step from the job_name input, and this workflow does not pass that input. In the scheduled matrix context, uploads will therefore be associated with the matrix job rather than the existing PR CI check job that the workflow is trying to match; move this to with: job_name: check on the upload step.

Useful? React with 👍 / 👎.

- Replace --outputFile (Jest JSON flag) with JEST_JUNIT_OUTPUT_FILE env var so jest-junit writes JUnit XML to the expected path - Move MERGIFY_JOB_NAME from job env to job_name input on the upload step, which is how gha-mergify-ci@v14 reads the override Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EtPYPZotcNDvxig8bNXqUh

claude added 2 commits June 18, 2026 12:03

chore(changelog): add flaky test detection entry for #979

bab98eb

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EtPYPZotcNDvxig8bNXqUh

github-actions Bot added status:needs-review Awaiting code review type:build Build & CI area:ci Build and CI pipelines area:documentation Docs & guides labels Jun 18, 2026

github-actions Bot added lang:md Markdown content/docs type:ci CI/CD pipeline work priority:normal Default priority type:chore Chore / small hygiene change type:test Testing/coverage meta:needs-changelog Requires a changelog entry before merge labels Jun 18, 2026

gemini-code-assist Bot reviewed Jun 18, 2026

View reviewed changes

docs(changelog): fix comma splice in flaky test detection entry

10f0da4

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EtPYPZotcNDvxig8bNXqUh

github-actions Bot removed type:test Testing/coverage type:chore Chore / small hygiene change type:ci CI/CD pipeline work labels Jun 18, 2026

chatgpt-codex-connector Bot reviewed Jun 18, 2026

View reviewed changes

github-actions Bot added the type:ci CI/CD pipeline work label Jun 18, 2026

ashleyshaw enabled auto-merge (squash) June 18, 2026 12:33

ashleyshaw merged commit e24b51c into develop Jun 18, 2026
23 checks passed

ashleyshaw deleted the ci/github-mergify-flaky-detection branch June 18, 2026 12:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci(github): add Mergify flaky test detection workflow#983

ci(github): add Mergify flaky test detection workflow#983
ashleyshaw merged 4 commits into
developfrom
ci/github-mergify-flaky-detection

ashleyshaw commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jun 18, 2026 •

edited

Loading

Review limit reached

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jun 18, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 18, 2026

Uh oh!

chatgpt-codex-connector Bot Jun 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		### Added

		- Mergify flaky test detection — Added `.github/workflows/flaky-test-detection.yml` to detect flaky Jest unit tests via Mergify CI. Runs 5 parallel matrix jobs every 12 hours on weekdays, uploads JUnit XML results to Mergify using `mergifyio/gha-mergify-ci@v14` with `flaky_test_detection: true`. ([#979](https://github.com/lightspeedwp/.github/issues/979))

Conversation

ashleyshaw commented Jun 18, 2026

Build/CI Pull Request

Linked issues

Build/CI change

Baseline & Target

Rollback

Notes

Changelog

Added

Checklist (Global DoD / PR)

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Reviewer Summary for PR #983

Recommendations

Uh oh!

coderabbitai Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jun 18, 2026 •

edited

Loading

coderabbitai Bot commented Jun 18, 2026 •

edited

Loading