ci: DH-23793: Automated agents to look at flaky E2E tests by cryolins · Pull Request #2782 · deephaven/web-client-ui

cryolins · 2026-10-26T17:25:06Z

Extensions to the e2e.yml pipeline to process flaky tests within runs and assign custom Copilot agents to look into fixes.

To setup:

Currently uses a personal access token with read and write access to actions, code, issues, and pull requests, and read access to metadata, to assign Copilot agents.
Store as COPILOT_ASSIGN_TOKEN as an actions repo secret
Could consider creating a github account specifically to handle providing the PAT. These will only be used in workflows, not by the agents.
- If the token is not set or expired, issues should get created and the summary will show but no agents get assigned.

Changes:

Added agent and instruction files under .github/ for the new agent, with a new workflow Copilot can run to setup with. Model is default Sonnet 5.5.
Increased logs and failure trace collection to provide more information for the agent and the triaging process.
- Agents are also told to add appropriate diagnostics to collect more info if they cannot find a fix.
Triage pipeline file at tests/ightly-triage.ts that deduplicates and classifies flakes, prints a summary to the actions summary, then opens up an issue and assigns an agent to it.
- Deduplicates by stripping variable info such as line numbers and timestamps from the first relevant part of the stack trace and hashing the result into a "fingerprint" that is 1-to-1 with an issue unless its data/cache gets reset.
- Classifies fingerprints/issues for human escalation/agent assignment by if it's a sensitive path or if previous agents attempted and could not fix.
- Builds issue body using read stack traces and links related attachments
- Assigns playwright-flake-investigator agents to the new issue if the token is set.
  - Max agents assigned per night at 8 just in case a wide failure doesn't trigger a large flow of agents. Can be adjusted.
- Reassigns a new agent when an issue (not its PR) is closed, either as completed or not implemented. Or: unassigned open issue from before that also recurred this night.
e2e.yml also pushed back later in the night by 4 hours

Notes on possible changes:

Currently this is set up with Copilot, if we want to switch to another provider ex. Claude, files and file contents may have to be moved to migrate it
Additional failure collection could help if no helpful changes are being made by the agent, but can increase pipeline time.
- ex. in playwright.config.ts , trace has been turned to retain-on-failure. Doing the same to video would provide some more info but as the comment notes it would slow down tests.
If agent behaviour is not fixing flakes in a manner that we consistently approve of, the agent instructions can be iterated on

…eatures

Copilot

🟡 Changes recommended

The exposed anonymous server endpoint and unresolved triage and reviewer-automation errors need correction before approval.

2 open issues

4 previously missed issues

In code that hasn't changed since last review

Skip draft PRs when requesting reviewers

.github/workflows/copilot-pr-reviewers.yml:15

Copilot initially opens its PRs as drafts, so the opened event runs this job before review can be requested. GitHub rejects reviewer requests on draft PRs, making the new workflow fail on that initial event. Exclude drafts from the job condition; the existing ready_for_review subscription will request the team once the PR is ready.

Use the supported Playwright trace viewer command

tests/nightly-flake-triage.ts:745

The repository pins Playwright 1.56.1, whose CLI has show-trace but no trace subcommand. All three commands emitted here therefore fail with an unknown-command error instead of inspecting the failure artifacts. Use the supported trace viewer command and remove the claim that it works without a GUI, or provide a compatible headless inspection tool.

Paginate timeline before selecting the latest merged PR

tests/nightly-flake-triage.ts:855

The timeline endpoint caps per_page at 100, so this reads only the first page. Once a diagnostics-only PR reference falls on a later page, triage misses it and counts the recurrence as a failed fix, potentially escalating prematurely. Paginate the timeline before selecting the latest merged PR.

Reconcile fresh issue states before recurrence accounting

tests/nightly-flake-triage.ts:1387

With more than 50 recurring tracked fingerprints, issues omitted from the state refresh can retain a cached open state even when the later label lookup reports them as closed. This expression prefers the stale state, leaving those issues closed while recording them as open and preventing the intended reopen and reassignment. Reconcile fresh issue states before recurrence accounting and use the reconciled state when reopening.

1 resolved since last review

Bind diagnostics exemptions to the current closing round

_{Review effort: Balanced}

…et existingByFingerprint earlier

Copilot

🟡 Changes recommended

Reproduction commands need the server-starting configuration, and timeline retrieval needs pagination to preserve correct recurrence accounting.

2 open issues

2 resolved since last review

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The PAT-backed automation requires a trusted repository-backed end-to-end run to validate agent assignment, recurring triggers, and team review requests before nightly enablement.

1 open issue

Paginate the timeline before collecting linked PRs

1 resolved since last review

Use the CI Playwright config to start preview servers

_{Review effort: Balanced}

mofojed

I think we can remove the auto-assign reviewer. Looking good though!

mofojed · 2026-10-28T17:49:12Z

+
+Use Playwright's locator/action auto-waiting.
+
+Do not use page.waitForTimeout().


I wonder if we should just say "Prefer not to use page.waitForTimeou()". That, coupled with the "Prefer deterministic application state" item below, should make it pretty clear.

The original idea was that especially for fixing tests to try to not make it pass just by waiting for things to clear, perhaps using more deterministic waits instead, or else to add diagnostics instead. Keeping it strict might be better to prevent the agent from falling back to it at all. If we do want to be a bit more lenient I'm thinking to make it specify proper documentation for why discouraged functions were used

mofojed · 2026-10-28T17:51:57Z

+
+## When you cannot find the root cause
+
+Do not guess, and do not quarantine. Make the _next_ occurrence diagnosable


I like this, this is interesting. We haven't really played around with adding more logging to the tests themselves.

mofojed · 2026-10-28T18:03:04Z

+          # Needs pull requests write and org members read; GITHUB_TOKEN cannot read org teams.
+          GH_TOKEN: ${{ secrets.REVIEW_REQUEST_TOKEN }}


If we're only creating a token so we can read the org team, I think we can just skip that. If we really want to automate assigning it, then we can add the tests folder to CODEOWNERS with web-reviewers. One less token to manage.

Okay I've just removed auto-assigning reviewers for now

mofojed · 2026-10-28T18:09:04Z

I wonder if this would be appropriate to use a decider for: https://strandsagents.com/blog/introducing-strands-decider/
We can see how this script holds up for now, a decider might have a bit more flexibility (but might be harder to configer? Haven't really played much with them yet)

Yea I haven't looked into decision models as much yet, but it looks interesting.

mofojed · 2026-10-28T18:16:19Z

+export const COPILOT_CUSTOM_AGENT =
+  process.env.COPILOT_CUSTOM_AGENT ?? 'playwright-flake-investigator';
+export const COPILOT_BASE_BRANCH = process.env.COPILOT_BASE_BRANCH ?? 'main';
+export const COPILOT_MODEL = process.env.COPILOT_MODEL ?? 'claude-sonnet-5.5';


Did you try any other models, like opus? Just curious.

I haven't really used Opus for this one. I had it with auto for a while, then I switched to Sonnet since I thought these may fall into the category of smaller tasks, and to switch to Claude models. The performance is better than auto of course, and I've left it in as the default. I could play around with manually assigning Opus on my fork though.

mofojed · 2026-10-28T18:23:29Z

+          # Assigning Copilot needs a user-to-server token; GITHUB_TOKEN is rejected.
+          # Unset this secret to keep triage label-only.
+          COPILOT_ASSIGN_TOKEN: ${{ secrets.COPILOT_ASSIGN_TOKEN }}


Not sure we need this - Copilot should automatically get assigned as a reviewer in our repo already.

Oh this token is for assigning the agent assignee to work on the task. That looked to require a token for.

https://docs.github.com/en/copilot/how-tos/use-copilot-agents/cloud-agent/use-cloud-agent-via-the-api#authentication

Copilot

🔵 Needs a closer look

Human assessment is needed because the triage logic can mark fixes as held without executing tracked tests and omit prior diagnostic investigation links from subsequent handoffs.

No open issues 🎉

1 resolved since last review

Paginate the timeline before collecting linked PRs

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Human assessment is needed because the new triage state machine lacks automated test coverage despite controlling issue reopening, failed-fix budgets, and agent launches.

No open issues 🎉

_{Review effort: Balanced}

mofojed · 2026-11-03T16:57:51Z

+jobs:
+  # Must be named exactly this -- Copilot won't pick up the job otherwise.
+  copilot-setup-steps:
+    runs-on: ubuntu-latest


ubuntu-latest now uses v26.04 of Ubuntu (recently migrated), which is causing issues: https://github.com/deephaven/web-client-ui/actions/runs/40941146661/job/123684540884?pr=2782

BEWARE: your OS is not officially supported by Playwright; installing dependencies for ubuntu26.04-x64 as a fallback. Cannot install dependencies for ubuntu26.04-x64 with Playwright 1.56.1! Failed to install browsers Error: ERROR: Playwright does not support chromium on ubuntu26.04-x64

Just pin it to ubuntu-24.04 for now like we do elsewhere.

Ah good point. I did see all the testing actually still runs on 22.04. I think we should have that instead unless the others are going to 24.04 as well right?

Copilot

🔵 Needs a closer look

Human assessment is needed because stackless timeouts from unrelated tests can share a fingerprint, merging separate investigations and their failed-fix budgets.

No open issues 🎉

_{Review effort: Balanced}

cryolins added 30 commits September 25, 2026 16:18

e2e triage test

09d2bd9

push test

c5583a1

try to trigger triage

debad59

try to trigger workflow with full matrix on push

2e735a3

new node version (try push again!)

35f04f3

try removing ansi codes

b0a48c6

testKey gets line number

815899f

sort by recency

b83ead5

run trigger 1

0bad8f7

run trigger 2

dd480a1

better fingerprinting and state tracking

36ef2e4

rename + rerun test

9e247dc

typescript rewrite

e83a13d

run nvmrc node version

a10941f

comments + remove history migration

61c8ab1

more cautious escalation from flake to failure

4d1e6a1

Convert applyTo formatting

f90cea9

agent instruction files

daf9046

assign agent when creating issue

34473fa

Merge branch 'dh-23793-test-cautious-escalation' into dh-23793-test

7a4769c

manual review changes: refactoring, clearer naming, comments, minor f…

e022cd9

…eatures

Fixes from review of review

d3ed4aa

smaller auto-assign test

c5ab462

add comment to rerun

6886de2

set assignments back to 6

a4f8255

improved information generation

92d9ca7

temp assign less agents

d3ad396

turn assign agents off

2b5e076

turn on lower to try

ac5c621

3 to test

6034a77

Copilot started reviewing on behalf of cryolins October 27, 2026 15:26

View session

Copilot AI reviewed Oct 27, 2026

View reviewed changes

Comment thread tests/docker-scripts/docker-compose.agent.yml Resolved

cryolins added 4 commits October 27, 2026 12:13

addressed localhost + reviewer assign + playwright trace comments + g…

fc5b593

…et existingByFingerprint earlier

move triage script to scripts

6c5f7ab

refactor: separate triage script into helper files

2b279fc

refactor: main loop separated into function steps

90772bc

cryolins requested a balanced review from Copilot October 27, 2026 18:43

Copilot started reviewing on behalf of cryolins October 27, 2026 18:44

View session

Copilot AI reviewed Oct 27, 2026

View reviewed changes

Comment thread scripts/nightly-ci/githubApi.ts Resolved

Comment thread scripts/nightly-ci/issues.ts Resolved

include ci config in the commands the agent reads

8ba7e9a

cryolins requested a balanced review from Copilot October 27, 2026 21:01

Copilot started reviewing on behalf of cryolins October 27, 2026 21:01

View session

Copilot AI reviewed Oct 27, 2026

View reviewed changes

cryolins marked this pull request as ready for review October 28, 2026 13:54

cryolins requested a review from mofojed October 28, 2026 13:54

mofojed reviewed Oct 28, 2026

View reviewed changes

Remove auto assign reviewers

d7c7b9c

Copilot AI balanced review requested due to automatic review settings October 28, 2026 20:18

Copilot started reviewing on behalf of cryolins October 28, 2026 20:18

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

cryolins requested a review from mofojed October 28, 2026 20:31

Prefer avoiding waitForTimeout()

ce4b9a5

Copilot AI balanced review requested due to automatic review settings October 29, 2026 19:33

Copilot started reviewing on behalf of cryolins October 29, 2026 19:33

View session

Copilot AI reviewed Oct 29, 2026

View reviewed changes

mofojed requested changes Nov 3, 2026

View reviewed changes

Pin ubuntu-22.04

2a8ac5e

Copilot AI balanced review requested due to automatic review settings November 3, 2026 17:50

Copilot started reviewing on behalf of cryolins November 3, 2026 17:51

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes


		Use Playwright's locator/action auto-waiting.

		Do not use page.waitForTimeout().


		## When you cannot find the root cause

		Do not guess, and do not quarantine. Make the _next_ occurrence diagnosable

		# Needs pull requests write and org members read; GITHUB_TOKEN cannot read org teams.
		GH_TOKEN: ${{ secrets.REVIEW_REQUEST_TOKEN }}

Conversation

cryolins commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

To setup:

Changes:

Notes on possible changes:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

mofojed left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cryolins Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cryolins Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cryolins commented Oct 26, 2026 •

edited

Loading

cryolins Oct 28, 2026 •

edited

Loading

cryolins Nov 3, 2026 •

edited

Loading