Every night, Playwright tests run against performantlabs.com. When a test breaks, a Claude agent diagnoses the failure, classifies it, and — when appropriate — opens a pull request with the fix. This page links to the code that decides what it's allowed to touch.
How we test this site.
The stack.
Nothing exotic. Every choice is justified by something concrete.
| Layer | Choice | Why |
|---|---|---|
| CMS | Drupal 11 (Canvas module v1.3.2) | Recent migration from D10; Canvas is our Layout Builder / theming stack. |
| Hosting | Pantheon (dev → test → live) | Multidev branches map 1:1 to PR branches. |
| E2E tests | Playwright (primary), Cypress (legacy support) | Playwright is where autonomous healing lives; Cypress suites still run. |
| Test toolkit | ATK (our open-source Drupal testing kit) | We maintain it; using it here is dogfooding. |
| CI | GitHub Actions, nightly cron | heal-tests-claude.yml orchestrates the healer; test-against-pantheon.yml orchestrates the run. |
| Reporting | CTRF JSON + Playwright HTML report | Structured failure data is what Claude reads. |
| AI | Claude (Sonnet) via Anthropic API | Operating under a CLAUDE.md ruleset committed to the repo. |
The nightly workflow.
It's two workflows, not one. The first runs the tests; the second heals them.
Workflow 1 — the tester (test-against-pantheon.yml). Every night at 23:57 UTC:
- Playwright runs the full suite against
https://live-performant-labs.pantheonsite.io, sharded across parallel runners for speed. - Results merge into
ctrf/ctrf-report.json— structured JSON describing every pass, fail, and flake. - If anything failed,
ai-ctrf claudeenriches the CTRF with per-failure summaries (model:claude-sonnet-4-20250514). - A GitHub issue is opened — or reused if one is already open — with the label
auto-heal.
Workflow 2 — the healer (heal-tests-claude.yml). On each new auto-heal issue:
- Claude checks out the repo, reads
CLAUDE.md, and identifies the failing test file(s) from the issue body. - It classifies each failure: test issue or website issue.
- Test issue → Claude fixes the test on a branch named
claude/issue-N; a subsequent step opens a PR automatically. - Website issue → Claude posts its analysis as an issue comment. No code changes.
A small detail that matters: the healer fires on issues: [opened], not [labeled]. Multiple labels would otherwise trigger multiple duplicate runs.
Where Claude is allowed to touch.
The guardrails live in a committed file: CLAUDE.md. It's not a prompt buried in a secret. It's in the repo, reviewable like any other code.
The load-bearing rule is the two-mode split:
CI mode (default in GitHub Actions) — conservative. Only fix test code. Never modify website source.
Local mode (when running
claudeon a developer's machine) — aggressive. Fix whatever is broken, test or website.
The reasoning is in the file itself: the CI runner cannot deploy code changes to Pantheon, so fixing website code from CI would be pointless. Test-only PRs are safe to review and merge.
A few other guardrails worth naming:
- Tool allowlist in the workflow: Claude can run
npx,npm,git, and a handful of read-only shell commands. It cannotcurl,wget, or touch arbitrary binaries. - 25-turn cap. No runaway sessions.
- 10-minute hard timeout. If a fix isn't obvious in 10 minutes, the run dies and a human picks it up.
- Classification framework in
CLAUDE.mdtells Claude how to decide "test issue vs website issue" — selector mismatches and stale assertions go in the test-fix bucket; missing elements, broken forms, and 500s go in the website-fix bucket and get a comment, not a PR.
None of this is clever. All of it is written down.
A real healing session.
Here's one from this February. Issue #231 landed overnight: Playwright had failed on the Contact Us smoke test. The error: timeout waiting for a link with accessible name "COTACT US".
Claude read CLAUDE.md, opened the failing test file, and made the classification:
TEST issue — The selector text is "COTACT US" (missing the second 'N'). The site's navigation correctly says "CONTACT US". The test's selector is a typo.
It fixed the one-line typo and committed to a claude/issue-231 branch:
- await page.getByRole('link', { name: 'COTACT US' }).first().click()
+ await page.getByRole('link', { name: 'CONTACT US' }).first().click()(From tests/atk_contact_us/atk_contact_us.spec.js.)
A human reviewed the PR, confirmed the diff matched the classification, and merged. Commit 1d0741170f. No calls. No flake dashboards. No "investigate this next sprint." The whole loop — failure → issue → classification → PR → merge — closed before Monday's standup.
If the typo had been on the website side instead of the test side (say, the navigation itself said "COTACT US"), the classification would have been website issue: Claude would have posted its analysis as a comment on the issue and declined to touch any code. That's not a limitation to work around. That's the contract.
Honest limits.
A few things to know:
- The workflow heals; it doesn't author. Claude doesn't write new tests. It fixes ones that already exist.
- It can't find what's not broken. A test that passes while checking the wrong thing will never trigger a heal.
- Truly flaky tests don't reproduce reliably. If a failure doesn't recur on re-run, Claude documents that and moves on.
- Every PR is reviewed by a human. The classification is Claude's. The merge is not.
This isn't magic. It's a workflow, a ruleset, and a human in the loop. The things we've gotten back are Monday mornings, and the calendar time of a senior engineer we no longer spend on flake triage.
Want this running on your site?
If you have a Drupal site with a Playwright suite and nightly CI — or want to get there — we run a 30-minute review. We'll look at your current workflow, tell you honestly where autonomous healing would (and wouldn't) help, and leave you with a one-page writeup.
No sales pitch. No obligation.
Or, if you'd rather start with the code: