Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase test retries from 1 to 2 #73795

Merged
merged 1 commit into from
Dec 11, 2024
Merged

Conversation

unstubbable
Copy link
Contributor

Let's see how this affects the failure rate and duration of our tests in CI.

Test times for any given group might increase a bit, while hopefully at least increasing the chance that the whole test group succeeds. This would in theory reduce the number of times we have to re-run a whole failed test group, which is quite time consuming.

A possible downside of this change is that flaky tests which have a chance of getting fixed with a better implementation might be harder to detect.

@ijjk ijjk added the created-by: Next.js team PRs by the Next.js team. label Dec 11, 2024
@ijjk
Copy link
Member

ijjk commented Dec 11, 2024

Tests Passed

@ijjk
Copy link
Member

ijjk commented Dec 11, 2024

Stats from current PR

Default Build
General
vercel/next.js canary vercel/next.js hl/increase-retry-count Change
buildDuration 17.7s 15.9s N/A
buildDurationCached 14.9s 12.5s N/A
nodeModulesSize 409 MB 409 MB N/A
nextStartRea..uration (ms) 455ms 457ms N/A
Client Bundles (main, webpack)
vercel/next.js canary vercel/next.js hl/increase-retry-count Change
1187-HASH.js gzip 50.4 kB 50.4 kB N/A
8276.HASH.js gzip 169 B 168 B N/A
8377-HASH.js gzip 5.3 kB 5.3 kB N/A
bccd1874-HASH.js gzip 53 kB 53 kB N/A
framework-HASH.js gzip 57.5 kB 57.5 kB N/A
main-app-HASH.js gzip 233 B 235 B N/A
main-HASH.js gzip 33.8 kB 33.7 kB N/A
webpack-HASH.js gzip 1.71 kB 1.71 kB N/A
Overall change 0 B 0 B
Legacy Client Bundles (polyfills)
vercel/next.js canary vercel/next.js hl/increase-retry-count Change
polyfills-HASH.js gzip 39.4 kB 39.4 kB
Overall change 39.4 kB 39.4 kB
Client Pages
vercel/next.js canary vercel/next.js hl/increase-retry-count Change
_app-HASH.js gzip 193 B 193 B
_error-HASH.js gzip 193 B 193 B
amp-HASH.js gzip 512 B 510 B N/A
css-HASH.js gzip 343 B 342 B N/A
dynamic-HASH.js gzip 1.84 kB 1.84 kB
edge-ssr-HASH.js gzip 265 B 265 B
head-HASH.js gzip 363 B 362 B N/A
hooks-HASH.js gzip 393 B 392 B N/A
image-HASH.js gzip 4.44 kB 4.43 kB N/A
index-HASH.js gzip 268 B 268 B
link-HASH.js gzip 2.35 kB 2.34 kB N/A
routerDirect..HASH.js gzip 328 B 328 B
script-HASH.js gzip 397 B 397 B
withRouter-HASH.js gzip 323 B 326 B N/A
1afbb74e6ecf..834.css gzip 106 B 106 B
Overall change 3.59 kB 3.59 kB
Client Build Manifests
vercel/next.js canary vercel/next.js hl/increase-retry-count Change
_buildManifest.js gzip 747 B 745 B N/A
Overall change 0 B 0 B
Rendered Page Sizes
vercel/next.js canary vercel/next.js hl/increase-retry-count Change
index.html gzip 524 B 522 B N/A
link.html gzip 539 B 537 B N/A
withRouter.html gzip 520 B 519 B N/A
Overall change 0 B 0 B
Edge SSR bundle Size
vercel/next.js canary vercel/next.js hl/increase-retry-count Change
edge-ssr.js gzip 127 kB 127 kB N/A
page.js gzip 202 kB 202 kB N/A
Overall change 0 B 0 B
Middleware size
vercel/next.js canary vercel/next.js hl/increase-retry-count Change
middleware-b..fest.js gzip 670 B 667 B N/A
middleware-r..fest.js gzip 155 B 156 B N/A
middleware.js gzip 31 kB 31 kB N/A
edge-runtime..pack.js gzip 844 B 844 B
Overall change 844 B 844 B
Next Runtimes
vercel/next.js canary vercel/next.js hl/increase-retry-count Change
523-experime...dev.js gzip 322 B 322 B
523.runtime.dev.js gzip 314 B 314 B
app-page-exp...dev.js gzip 322 kB 322 kB
app-page-exp..prod.js gzip 127 kB 127 kB
app-page-tur..prod.js gzip 140 kB 140 kB
app-page-tur..prod.js gzip 135 kB 135 kB
app-page.run...dev.js gzip 312 kB 312 kB
app-page.run..prod.js gzip 122 kB 122 kB
app-route-ex...dev.js gzip 37.1 kB 37.1 kB
app-route-ex..prod.js gzip 25.1 kB 25.1 kB
app-route-tu..prod.js gzip 25.1 kB 25.1 kB
app-route-tu..prod.js gzip 24.9 kB 24.9 kB
app-route.ru...dev.js gzip 38.7 kB 38.7 kB
app-route.ru..prod.js gzip 24.9 kB 24.9 kB
pages-api-tu..prod.js gzip 9.56 kB 9.56 kB
pages-api.ru...dev.js gzip 11.4 kB 11.4 kB
pages-api.ru..prod.js gzip 9.56 kB 9.56 kB
pages-turbo...prod.js gzip 21.3 kB 21.3 kB
pages.runtim...dev.js gzip 27 kB 27 kB
pages.runtim..prod.js gzip 21.3 kB 21.3 kB
server.runti..prod.js gzip 915 kB 915 kB
Overall change 2.35 MB 2.35 MB
build cache
vercel/next.js canary vercel/next.js hl/increase-retry-count Change
0.pack gzip 2.04 MB 2.03 MB N/A
index.pack gzip 72.3 kB 72 kB N/A
Overall change 0 B 0 B
Diff details
Diff for main-HASH.js

Diff too large to display

Commit: 86664cd

Let's see how this affects the failure rate and duration of our tests in
CI.

Test times for any given group might increase a bit, while hopefully at
least increasing the chance that the whole test group succeeds. This
would in theory reduce the number of times we have to re-run a whole
failed test group, which is quite time consuming.

A possible downside of this change is that flaky tests which have a
chance of getting fixed with a better implementation might be harder to
detect.
@unstubbable unstubbable force-pushed the hl/increase-retry-count branch from f2a42ce to 86664cd Compare December 11, 2024 13:12
@unstubbable unstubbable marked this pull request as ready for review December 11, 2024 15:05
@unstubbable unstubbable merged commit 763ca5e into canary Dec 11, 2024
108 of 111 checks passed
@unstubbable unstubbable deleted the hl/increase-retry-count branch December 11, 2024 17:57
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 26, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
created-by: Next.js team PRs by the Next.js team. locked
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants