Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(examples): update examples tests to diff full output #5242

Merged

Conversation

mehulkar
Copy link
Contributor

@mehulkar mehulkar commented Jun 8, 2023

This will help debug examples tests when they fail.

(This is lifted from #5011)

Approve and merge #5240 first!

@vercel
Copy link

vercel bot commented Jun 8, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
examples-cra-web 🔄 Building (Inspect) Visit Preview 💬 Add feedback Jun 8, 2023 4:10pm
examples-svelte-web 🔄 Building (Inspect) Visit Preview 💬 Add feedback Jun 8, 2023 4:10pm
9 Ignored Deployments
Name Status Preview Comments Updated (UTC)
examples-basic-web ⬜️ Ignored (Inspect) Visit Preview Jun 8, 2023 4:10pm
examples-designsystem-docs ⬜️ Ignored (Inspect) Visit Preview Jun 8, 2023 4:10pm
examples-gatsby-web ⬜️ Ignored (Inspect) Visit Preview Jun 8, 2023 4:10pm
examples-kitchensink-blog ⬜️ Ignored (Inspect) Visit Preview Jun 8, 2023 4:10pm
examples-native-web ⬜️ Ignored (Inspect) Visit Preview Jun 8, 2023 4:10pm
examples-nonmonorepo ⬜️ Ignored (Inspect) Visit Preview Jun 8, 2023 4:10pm
examples-tailwind-web ⬜️ Ignored (Inspect) Visit Preview Jun 8, 2023 4:10pm
examples-vite-web ⬜️ Ignored (Inspect) Visit Preview Jun 8, 2023 4:10pm
turbo-site ⬜️ Ignored (Inspect) Visit Preview Jun 8, 2023 4:10pm

\xe2\x80\xa2 Remote caching disabled (esc)

Tasks: 5 successful, 5 total
Cached: 3 cached, 5 total
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not going to try to fix it here, but it's weird that the gatsby example doesn't get full turbo on the second run. might be expected

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(This test isn't even running in CI, because this combo isn't part of the matrix)

@mehulkar mehulkar force-pushed the mehulkar/turbo-1152-examples-tests-should-validate-output branch from 24da09a to c3f7031 Compare June 8, 2023 05:24
@mehulkar mehulkar force-pushed the mehulkar/turbo-1149-examples-tests-failing-on-main branch from 53c6644 to 4bfd3c4 Compare June 8, 2023 05:24
@mehulkar mehulkar force-pushed the mehulkar/turbo-1152-examples-tests-should-validate-output branch from c3f7031 to 99f5ce4 Compare June 8, 2023 05:24
@mehulkar mehulkar changed the title ci: update examples tests to diff full output ci(examples): update examples tests to diff full output Jun 8, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Jun 8, 2023

⚠️ CI failed ⚠️

The following steps have failed in CI:

  • Turbopack Rust tests (mac/win, non-blocking)

See workflow summary for details

@github-actions
Copy link
Contributor

github-actions bot commented Jun 8, 2023

✅ This change can build next-swc

@github-actions
Copy link
Contributor

github-actions bot commented Jun 8, 2023

Linux Benchmark for b5490c2

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 8286.64µs ± 29.72µs 8318.12µs ± 99.42µs +0.38%
bench_hmr_to_eval/Turbopack CSR/1000 modules 7314.79µs ± 42.98µs 7851.01µs ± 345.22µs +7.33%
bench_startup/Turbopack CSR/1000 modules 907.35ms ± 0.84ms 902.27ms ± 2.89ms -0.56%

@mehulkar mehulkar marked this pull request as ready for review June 8, 2023 07:08
@mehulkar mehulkar requested review from a team as code owners June 8, 2023 07:08
@mehulkar mehulkar requested review from tknickman and chris-olszewski and removed request for a team June 8, 2023 07:08
@github-actions
Copy link
Contributor

github-actions bot commented Jun 8, 2023

Linux Benchmark for 1c5696d

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 10.44ms ± 0.09ms 10.39ms ± 0.11ms -0.48%
bench_hmr_to_eval/Turbopack CSR/1000 modules 8625.54µs ± 65.61µs 8676.59µs ± 83.24µs +0.59%
bench_startup/Turbopack CSR/1000 modules 945.74ms ± 2.11ms 955.84ms ± 5.30ms +1.07%

@github-actions
Copy link
Contributor

github-actions bot commented Jun 8, 2023

Linux Benchmark for 2325bdb

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 4904.23µs ± 36.22µs 4919.28µs ± 25.74µs +0.31%
bench_hmr_to_eval/Turbopack CSR/1000 modules 4588.09µs ± 22.18µs 4600.41µs ± 37.66µs +0.27%
bench_startup/Turbopack CSR/1000 modules 744.23ms ± 0.79ms 742.64ms ± 1.32ms -0.21%

@github-actions
Copy link
Contributor

github-actions bot commented Jun 8, 2023

MacOS Benchmark for 2325bdb

Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 26.86ms ± 1.12ms 58.22ms ± 3.84ms +116.76% +73.73%
Click to view full benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 26.86ms ± 1.12ms 58.22ms ± 3.84ms +116.76% +73.73%
bench_hmr_to_eval/Turbopack CSR/1000 modules 26.22ms ± 0.30ms 26.07ms ± 0.13ms -0.59%
bench_startup/Turbopack CSR/1000 modules 3087.40ms ± 56.00ms 3203.09ms ± 85.76ms +3.75%

@github-actions
Copy link
Contributor

github-actions bot commented Jun 8, 2023

Linux Benchmark for 8afb85a

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 8790.09µs ± 88.59µs 8662.12µs ± 93.99µs -1.46%
bench_hmr_to_eval/Turbopack CSR/1000 modules 7763.17µs ± 49.40µs 7829.38µs ± 105.47µs +0.85%
bench_startup/Turbopack CSR/1000 modules 899.71ms ± 1.09ms 896.71ms ± 2.73ms -0.33%

Base automatically changed from mehulkar/turbo-1149-examples-tests-failing-on-main to main June 8, 2023 15:02
@github-actions
Copy link
Contributor

github-actions bot commented Jun 8, 2023

MacOS Benchmark for 8afb85a

Test Base PR % Significant %
bench_hmr_to_eval/Turbopack CSR/1000 modules 24.03ms ± 0.14ms 53.99ms ± 4.67ms +124.70% +83.68%
Click to view full benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 24.86ms ± 0.27ms 25.45ms ± 0.31ms +2.34%
bench_hmr_to_eval/Turbopack CSR/1000 modules 24.03ms ± 0.14ms 53.99ms ± 4.67ms +124.70% +83.68%
bench_startup/Turbopack CSR/1000 modules 3513.66ms ± 555.21ms 3111.47ms ± 67.56ms -11.45%

This will help debug examples tests when they fail.

- Switch env vars to CLI flags so they don't leak through
  to the turbo invocations in the actual test and change the output that
  we are not asserting against.
- Enable corepack inside the test and then disable it again at the ened
  of the test.
@mehulkar mehulkar force-pushed the mehulkar/turbo-1152-examples-tests-should-validate-output branch from 49a2dc8 to 2f1d036 Compare June 8, 2023 16:10
@github-actions
Copy link
Contributor

github-actions bot commented Jun 8, 2023

Linux Benchmark for d15000d

Click to view benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 9786.03µs ± 107.47µs 9687.00µs ± 22.69µs -1.01%
bench_hmr_to_eval/Turbopack CSR/1000 modules 8954.15µs ± 81.99µs 8946.14µs ± 200.21µs -0.09%
bench_startup/Turbopack CSR/1000 modules 947.84ms ± 2.18ms 951.65ms ± 4.97ms +0.40%

@mehulkar
Copy link
Contributor Author

mehulkar commented Jun 8, 2023

The JS packages workflow is failing because the --filters in that CI job are catching changes to the examples (and you can't trigger the examples tests without extra pass through arguments, so they fail). We need to figure out how to make the JS package tests be limited to only ./packages/ directory, but that won't happen in this PR

@mehulkar mehulkar merged commit 6e60a03 into main Jun 8, 2023
@mehulkar mehulkar deleted the mehulkar/turbo-1152-examples-tests-should-validate-output branch June 8, 2023 16:54
@github-actions
Copy link
Contributor

github-actions bot commented Jun 8, 2023

Windows Benchmark for d15000d

Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 15.53ms ± 0.01ms 15.64ms ± 0.04ms +0.74% +0.08%
Click to view full benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 15.53ms ± 0.01ms 15.64ms ± 0.04ms +0.74% +0.08%
bench_hmr_to_eval/Turbopack CSR/1000 modules 15.56ms ± 0.03ms 15.58ms ± 0.03ms +0.15%
bench_startup/Turbopack CSR/1000 modules 3322.49ms ± 25.39ms 3369.07ms ± 90.10ms +1.40%

@github-actions
Copy link
Contributor

github-actions bot commented Jun 8, 2023

MacOS Benchmark for d15000d

Test Base PR % Significant %
bench_hmr_to_eval/Turbopack CSR/1000 modules 25.93ms ± 0.23ms 51.52ms ± 7.02ms +98.69% +41.99%
Click to view full benchmark
Test Base PR % Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules 26.45ms ± 0.25ms 26.77ms ± 0.73ms +1.22%
bench_hmr_to_eval/Turbopack CSR/1000 modules 25.93ms ± 0.23ms 51.52ms ± 7.02ms +98.69% +41.99%
bench_startup/Turbopack CSR/1000 modules 4402.10ms ± 1081.84ms 3620.06ms ± 116.67ms -17.77%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants