Task Edges Set/List #8624

sokra · 2024-06-28T06:15:23Z

Description

Task children and dependencies are stored together in a single datastructure -> task edges
task edges is a map keyed by target task id and using a bitfield for common edges (child, output, cell 0)
removed lazy_remove_children feature flag -> this is always on now (Disabling it doesn't make sense as it negatively affects performance, we had the feature flag in case it it is broken, but seems fine)
When a task becomes Done, we use a Vec instead of a HashMap for better memory efficiency. It doesn't need to modified after that.

Testing Instructions

vercel · 2024-06-28T06:15:26Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
rust-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 1, 2024 4:37pm

8 Ignored Deployments

Name	Status	Preview	Updated (UTC)
examples-basic-web	⬜️ Ignored (Inspect)	Visit Preview	Jul 1, 2024 4:37pm
examples-designsystem-docs	⬜️ Ignored (Inspect)	Visit Preview	Jul 1, 2024 4:37pm
examples-gatsby-web	⬜️ Ignored (Inspect)	Visit Preview	Jul 1, 2024 4:37pm
examples-kitchensink-blog	⬜️ Ignored (Inspect)	Visit Preview	Jul 1, 2024 4:37pm
examples-native-web	⬜️ Ignored (Inspect)	Visit Preview	Jul 1, 2024 4:37pm
examples-svelte-web	⬜️ Ignored (Inspect)	Visit Preview	Jul 1, 2024 4:37pm
examples-tailwind-web	⬜️ Ignored (Inspect)	Visit Preview	Jul 1, 2024 4:37pm
examples-vite-web	⬜️ Ignored (Inspect)	Visit Preview	Jul 1, 2024 4:37pm

github-actions · 2024-06-28T06:17:28Z

🟢 Turbopack Benchmark CI successful 🟢

Thanks

github-actions · 2024-06-28T06:18:16Z

⚠️ This change may fail to build next-swc.

Logs

error: failed to select a version for `triomphe`.
    ... required by package `hstr v0.2.8`
    ... which satisfies dependency `hstr = "^0.2.8"` (locked to 0.2.8) of package `swc_atoms v0.6.7`
    ... which satisfies dependency `swc_atoms = "^0.6.5"` (locked to 0.6.7) of package `swc_core v0.95.4`
    ... which satisfies dependency `swc_core = "^0.95.4"` (locked to 0.95.4) of package `wasm v0.0.0 (/root/actions-runner/_work/turbo/turbo/packages/next-swc/crates/wasm)`
versions that meet the requirements `^0.1.11` (locked to 0.1.11) are: 0.1.11

all possible versions conflict with previously selected packages.

  previously selected package `triomphe v0.1.13`
    ... which satisfies dependency `triomphe = "^0.1.13"` of package `turbo-tasks v0.1.0 (https://github.com/vercel/turbo?rev=413cfbf596e054d65edb6870658103338e856cab#78c44bd1)`
    ... which satisfies git dependency `turbo-tasks` (locked to 0.1.0) of package `next-build-test v0.1.0 (/root/actions-runner/_work/turbo/turbo/packages/next-swc/crates/next-build-test)`

failed to select a version for `triomphe` which could resolve this conflict

See job summary for details

github-actions · 2024-06-28T06:28:00Z

⚠️ CI failed ⚠️

The following steps have failed in CI:

Turbopack Rust tests (mac/win, non-blocking)

See workflow summary for details

bgw

I read through the edges set implementation, but haven't gotten to reviewing how it's used yet.

crates/turbo-tasks-auto-hash-map/src/map.rs

crates/turbo-tasks-memory/src/edges_set.rs

bgw · 2024-06-29T03:39:45Z

crates/turbo-tasks-memory/src/edges_set.rs

+    pub fn into_set(self) -> TaskEdgesSet {
+        TaskEdgesSet {
+            edges: Box::new(self.edges.into_vec().into_iter().collect()),
+        }
+    }


How common is this operation? I bet if we were clever we could do an optimization for the case where we're converting into a SmallVec that avoids the iterator.

crates/turbo-tasks-auto-hash-map/src/map.rs

crates/turbo-tasks-memory/src/task.rs

bgw · 2024-07-02T17:42:32Z

Thanks for making the suggested changes! The edge_set implementation LGTM!

Tobias Koppers - fix typo (vercel/turborepo#8619) Benjamin Woodruff - Store aggregate read/execute count statistics (vercel/turborepo#8286) Tobias Koppers - box InProgress task state (vercel/turborepo#8644) Tobias Koppers - Task Edges Set/List (vercel/turborepo#8624) Benjamin Woodruff - Memory: Use `triomphe::Arc` for `SharedReference` (vercel/turborepo#8622) Will Binns-Smith - chore: release npm packages (vercel/turborepo#8614) Will Binns-Smith - devlow-bench: add git branch and sha to datapoints (vercel/turborepo#8602) --- Fixes a `triomphe` package version conflict between turbopack and swc by bumping it from 0.1.11 to 0.1.13.

### Description * Task children and dependencies are stored together in a single datastructure -> task edges * task edges is a map keyed by target task id and using a bitfield for common edges (child, output, cell 0) * removed lazy_remove_children feature flag -> this is always on now (Disabling it doesn't make sense as it negatively affects performance, we had the feature flag in case it it is broken, but seems fine) * When a task becomes Done, we use a Vec instead of a HashMap for better memory efficiency. It doesn't need to modified after that. ### Testing Instructions  --------- Co-authored-by: Benjamin Woodruff <benjamin.woodruff@vercel.com>

Tobias Koppers - fix typo (vercel/turborepo#8619) Benjamin Woodruff - Store aggregate read/execute count statistics (vercel/turborepo#8286) Tobias Koppers - box InProgress task state (vercel/turborepo#8644) Tobias Koppers - Task Edges Set/List (vercel/turborepo#8624) Benjamin Woodruff - Memory: Use `triomphe::Arc` for `SharedReference` (vercel/turborepo#8622) Will Binns-Smith - chore: release npm packages (vercel/turborepo#8614) Will Binns-Smith - devlow-bench: add git branch and sha to datapoints (vercel/turborepo#8602) --- Fixes a `triomphe` package version conflict between turbopack and swc by bumping it from 0.1.11 to 0.1.13.

sokra requested a review from a team as a code owner June 28, 2024 06:15

turbo-orchestrator bot added created-by: turbopack labels Jun 28, 2024

sokra requested a review from bgw June 28, 2024 06:17

sokra force-pushed the sokra/edges-set-list branch from 7c6e61c to 80088d7 Compare June 28, 2024 06:18

vercel bot deployed to Preview – rust-docs June 28, 2024 06:20 View deployment

sokra mentioned this pull request Jun 28, 2024

Turbopack: Task Edges Set/List vercel/next.js#67265

Closed

vercel bot deployed to Preview – rust-docs June 28, 2024 11:23 View deployment

bgw reviewed Jun 29, 2024

View reviewed changes

vercel bot deployed to Preview – rust-docs July 1, 2024 06:08 View deployment

kdy1 reviewed Jul 1, 2024

View reviewed changes

crates/turbo-tasks-auto-hash-map/src/map.rs Outdated Show resolved Hide resolved

crates/turbo-tasks-memory/src/task.rs Outdated Show resolved Hide resolved

vercel bot deployed to Preview – rust-docs July 1, 2024 06:20 View deployment

vercel bot deployed to Preview – rust-docs July 1, 2024 06:27 View deployment

sokra added 10 commits July 1, 2024 08:37

simplify

7a66b0d

store edges more efficiently

554f5f5

Use set while collecting, but store in list afterwards

af49750

allow dependencies set to include children

41ee0fd

add remove support

9e51d19

WIP

e714441

lazy remove dependencies

7f4a98b

fixups

e1ca24c

fixup

2e38b90

remove lazy_remove_children

d0f473d

sokra force-pushed the sokra/edges-set-list branch from ee4dbb7 to 50a72da Compare July 1, 2024 06:53

vercel bot deployed to Preview – rust-docs July 1, 2024 06:58 View deployment

sokra requested a review from bgw July 1, 2024 07:54

sokra added 5 commits July 1, 2024 14:38

fixup

27d032b

clippy

4222ea1

less memory memmoved

9e07d93

add comments

6034628

fix mark_as_finished

6640307

vercel bot deployed to Preview – rust-docs July 1, 2024 12:39 View deployment

perf

0520182

sokra force-pushed the sokra/edges-set-list branch from 4b8ad04 to 0520182 Compare July 1, 2024 13:07

vercel bot deployed to Preview – rust-docs July 1, 2024 13:09 View deployment

sokra added 4 commits July 1, 2024 15:42

keep event for force scheduling

034bb7d

remove event from Dirty state

026ce5c

bugfix

59dc9e2

perf

0361863

vercel bot deployed to Preview – rust-docs July 1, 2024 14:23 View deployment

sokra added 3 commits July 1, 2024 17:46

add missing cases

5c68917

add test case

1665ccf

keep counter of children

8e3352d

vercel bot deployed to Preview – rust-docs July 1, 2024 16:37 View deployment

sokra requested a review from kdy1 July 1, 2024 16:40

kdy1 approved these changes Jul 2, 2024

View reviewed changes

sokra merged commit 52ed46b into main Jul 2, 2024
57 of 60 checks passed

sokra deleted the sokra/edges-set-list branch July 2, 2024 06:39

bgw mentioned this pull request Jul 3, 2024

[chore] Update turbopack to turbopack-240703.1 vercel/next.js#67406

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Task Edges Set/List #8624

Task Edges Set/List #8624

sokra commented Jun 28, 2024

vercel bot commented Jun 28, 2024 •

edited

Loading

github-actions bot commented Jun 28, 2024 •

edited

Loading

github-actions bot commented Jun 28, 2024 •

edited

Loading

github-actions bot commented Jun 28, 2024 •

edited

Loading

bgw left a comment

bgw Jun 29, 2024

bgw commented Jul 2, 2024

Task Edges Set/List #8624

Task Edges Set/List #8624

Conversation

sokra commented Jun 28, 2024

Description

Testing Instructions

vercel bot commented Jun 28, 2024 • edited Loading

github-actions bot commented Jun 28, 2024 • edited Loading

🟢 Turbopack Benchmark CI successful 🟢

github-actions bot commented Jun 28, 2024 • edited Loading

github-actions bot commented Jun 28, 2024 • edited Loading

⚠️ CI failed ⚠️

bgw left a comment

Choose a reason for hiding this comment

bgw Jun 29, 2024

Choose a reason for hiding this comment

bgw commented Jul 2, 2024

vercel bot commented Jun 28, 2024 •

edited

Loading

github-actions bot commented Jun 28, 2024 •

edited

Loading

github-actions bot commented Jun 28, 2024 •

edited

Loading

github-actions bot commented Jun 28, 2024 •

edited

Loading