[experiment] Allocate DepGraph::read_index de-duplication hashset more lazily. #56724

michaelwoerister · 2018-12-11T20:02:31Z

Possibly an optimization for DepGraph::read_index(). Might need some tweaking. I also want to try a SIMD-based version.

rust-highfive · 2018-12-11T20:02:35Z

r? @eddyb

(rust_highfive has picked a reviewer for you, use r? to override)

michaelwoerister · 2018-12-11T20:02:41Z

@bors try

bors · 2018-12-11T20:02:54Z

⌛ Trying commit 661f6fc with merge c81a607...

…try> [experiment] Allocate DepGraph::read_index de-duplication hashset more lazily. Possibly an optimization for `DepGraph::read_index()`. Might need some tweaking. I also want to try a SIMD-based version.

bors · 2018-12-11T22:36:48Z

☀️ Test successful - status-travis
State: approved= try=True

michaelwoerister · 2018-12-12T07:50:19Z

@rust-timer build c81a607

rust-timer · 2018-12-12T07:50:20Z

Success: Queued c81a607 with parent 8375ab4, comparison URL.

rust-timer · 2018-12-12T09:35:41Z

Finished benchmarking try commit c81a607

michaelwoerister · 2018-12-12T10:12:21Z

Looks like the current version has no visible effect. I'll do a SIMD enabled version when I find the time.

michaelwoerister · 2018-12-12T15:25:24Z

@bors try

bors · 2018-12-12T15:25:36Z

⌛ Trying commit ed37dd0decd07fcc058cc5058418cfff8591962b with merge bcc6938d1d1ecf240b1fe74f41d00559e35bdba3...

rust-highfive · 2018-12-12T15:29:43Z

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:240831fc:start=1544628339672413278,finish=1544628340791953524,duration=1119540246
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
###########################################                               61.0%
######################################################################## 100.0%
[00:01:11] extracting /checkout/obj/build/cache/2018-10-30/cargo-beta-x86_64-unknown-linux-gnu.tar.gz
[00:01:11]     Updating crates.io index
[00:01:19] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:01:19] Build completed unsuccessfully in 0:00:36
[00:01:19] Makefile:81: recipe for target 'prepare' failed
[00:01:19] make: *** [prepare] Error 1
[00:01:20] Command failed. Attempt 2/5:
[00:01:20] Command failed. Attempt 2/5:
[00:01:21] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:01:21] Build completed unsuccessfully in 0:00:00
[00:01:21] Makefile:81: recipe for target 'prepare' failed
[00:01:21] make: *** [prepare] Error 1
[00:01:23] Command failed. Attempt 3/5:
[00:01:23] Command failed. Attempt 3/5:
[00:01:23] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:01:23] Build completed unsuccessfully in 0:00:00
[00:01:23] make: *** [prepare] Error 1
[00:01:23] Makefile:81: recipe for target 'prepare' failed
[00:01:26] Command failed. Attempt 4/5:
[00:01:26] Command failed. Attempt 4/5:
[00:01:26] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:01:26] Build completed unsuccessfully in 0:00:00
[00:01:26] Makefile:81: recipe for target 'prepare' failed
[00:01:26] make: *** [prepare] Error 1
[00:01:30] Command failed. Attempt 5/5:
[00:01:30] Command failed. Attempt 5/5:
[00:01:31] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:01:31] Build completed unsuccessfully in 0:00:00
[00:01:31] Makefile:81: recipe for target 'prepare' failed
[00:01:31] make: *** [prepare] Error 1
[00:01:31] The command has failed after 5 attempts.
---
travis_time:end:117b4ba9:start=1544628441684765542,finish=1544628441693001434,duration=8235892
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0401e2eb
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:06fb4ed6
travis_time:start:06fb4ed6
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:1403779a
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

bors · 2018-12-12T15:31:57Z

💔 Test failed - status-travis

rust-highfive · 2018-12-12T15:31:59Z

The job dist-x86_64-linux of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:0dd7bb8f
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
################################################################          89.9%
######################################################################## 100.0%
[00:04:05] extracting /checkout/obj/build/cache/2018-10-30/cargo-beta-x86_64-unknown-linux-gnu.tar.gz
[00:04:05]     Updating crates.io index
[00:04:12] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:04:12] Build completed unsuccessfully in 0:00:22
[00:04:12] make: *** [prepare] Error 1
[00:04:13] Command failed. Attempt 2/5:
[00:04:13] Command failed. Attempt 2/5:
[00:04:14] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:04:14] Build completed unsuccessfully in 0:00:00
[00:04:14] make: *** [prepare] Error 1
[00:04:16] Command failed. Attempt 3/5:
[00:04:16] Command failed. Attempt 3/5:
[00:04:16] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:04:16] Build completed unsuccessfully in 0:00:00
[00:04:16] make: *** [prepare] Error 1
[00:04:19] Command failed. Attempt 4/5:
[00:04:19] Command failed. Attempt 4/5:
[00:04:19] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:04:19] Build completed unsuccessfully in 0:00:00
[00:04:19] make: *** [prepare] Error 1
[00:04:23] Command failed. Attempt 5/5:
[00:04:23] Command failed. Attempt 5/5:
[00:04:24] error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
[00:04:24] Build completed unsuccessfully in 0:00:00
[00:04:24] make: *** [prepare] Error 1
[00:04:24] The command has failed after 5 attempts.
mpiler_builtins/modules/compiler-rt
---
10956 ./src/llvm/test/MC/Disassembler/AMDGPU
10820 ./src/tools/lldb/unittests
10508 ./src/llvm/test/MC/AMDGPU
10332 ./src/tools/clang/include
10140 ./src/tools/lldb/packages/Python/lldbsuite/test/functionalities/postmortem
10012 ./src/llvm-emscripten/test/MC/AMDGPU
9816 ./.git/modules/src/doc
9708 ./.git/modules/src/tools/rustfmt
9648 ./src/llvm-emscripten/test/MC/Disassembler/AMDGPU

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

michaelwoerister · 2018-12-12T16:02:38Z

@bors try

bors · 2018-12-12T16:02:43Z

🔒 Merge conflict

This pull request and the master branch diverged in a way that cannot be automatically merged. Please rebase on top of the latest master branch, and let the reviewer approve again.

How do I rebase?

Assuming self is your fork and upstream is this repository, you can resolve the conflict following these steps:

git checkout depgraph-readindex-tweak (switch to your branch)
git fetch upstream master (retrieve the latest master)
git rebase upstream/master -p (rebase on top of it)
Follow the on-screen instruction to resolve conflicts (check git status if you got lost).
git push self depgraph-readindex-tweak --force-with-lease (update this PR)

You may also read Git Rebasing to Resolve Conflicts by Drew Blessing for a short tutorial.

Please avoid the "Resolve conflicts" button on GitHub. It uses git merge instead of git rebase which makes the PR commit history more difficult to read.

Sometimes step 4 will complete without asking for resolution. This is usually due to difference between how Cargo.lock conflict is handled during merge and rebase. This is normal, and you should still perform step 5 to update this PR.

Error message

warning: Cannot merge binary files: Cargo.lock (HEAD vs. heads/homu-tmp)
Auto-merging Cargo.lock
CONFLICT (content): Merge conflict in Cargo.lock
Automatic merge failed; fix conflicts and then commit the result.

michaelwoerister · 2018-12-12T16:03:39Z

@bors try

bors · 2018-12-12T16:03:52Z

⌛ Trying commit 9ab4bb7cff6423c7d58cb2152aca73c8793ff86f with merge 79ff98b6c3799f0c9e748102f216ea4c909dd8cc...

bors · 2018-12-12T17:21:20Z

💔 Test failed - status-travis

rust-highfive · 2018-12-12T17:21:21Z

The job dist-x86_64-linux of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_fold:end:services

travis_fold:start:git.checkout
travis_time:start:095f8fe6
$ git clone --depth=2 --branch=try https://github.com/rust-lang/rust.git rust-lang/rust
---
[01:01:19]    Compiling arena v0.0.0 (/checkout/src/libarena)
[01:01:20]    Compiling syntax_pos v0.0.0 (/checkout/src/libsyntax_pos)
[01:01:26]    Compiling rustc_errors v0.0.0 (/checkout/src/librustc_errors)
[01:02:39]    Compiling syntax_ext v0.0.0 (/checkout/src/libsyntax_ext)
[01:02:43] error[E0433]: failed to resolve: use of undeclared type or module `mem`
[01:02:43]     --> src/librustc/dep_graph/graph.rs:1128:35
[01:02:43]      |
[01:02:43] 1128 |                     debug_assert!(mem::size_of::<DepNodeIndex>() == 4);
[01:02:43]      |                                   ^^^ use of undeclared type or module `mem`
[01:02:55] error[E0308]: mismatched types
[01:02:55] error[E0308]: mismatched types
[01:02:55]     --> src/librustc/dep_graph/graph.rs:1133:53
[01:02:55]      |
[01:02:55] 1133 |                         let data1 = _mm_loadu_si128(ptr);
[01:02:55]      |                                                     ^^^ expected struct `std::arch::x86_64::__m128i`, found i32
[01:02:55]      |
[01:02:55]      = note: expected type `*const std::arch::x86_64::__m128i`
[01:02:55]                 found type `*const i32`
[01:02:55] error[E0308]: mismatched types
[01:02:55] error[E0308]: mismatched types
[01:02:55]     --> src/librustc/dep_graph/graph.rs:1134:53
[01:02:55]      |
[01:02:55] 1134 |                         let data2 = _mm_loadu_si128(ptr.offset(4));
[01:02:55]      |                                                     ^^^^^^^^^^^^^ expected struct `std::arch::x86_64::__m128i`, found i32
[01:02:55]      |
[01:02:55]      = note: expected type `*const std::arch::x86_64::__m128i`
[01:02:55]                 found type `*const i32`
[01:03:13] error: aborting due to 3 previous errors
[01:03:13] 
[01:03:13] Some errors occurred: E0308, E0433.
[01:03:13] For more information about an error, try `rustc --explain E0308`.
---
travis_time:end:0adc0a96:start=1544635278227143545,finish=1544635278234094379,duration=6950834
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:039564c6
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:026802f0
travis_time:start:026802f0
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:038270b2
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

michaelwoerister · 2018-12-12T20:46:06Z

@bors try

bors · 2018-12-12T20:46:20Z

⌛ Trying commit 3d610b4 with merge b350a5644a4b98b0c16953bc075e40ec5cb2cbd0...

bors · 2018-12-12T23:09:13Z

☀️ Test successful - status-travis
State: approved= try=True

Zoxc · 2018-12-13T05:09:50Z

@rust-timer build b350a5644a4b98b0c16953bc075e40ec5cb2cbd0

rust-timer · 2018-12-13T05:09:51Z

Success: Queued b350a5644a4b98b0c16953bc075e40ec5cb2cbd0 with parent dd8fc7d, comparison URL.

Zoxc · 2018-12-13T05:16:14Z

Do you have any stats on how many reads a dep node typically gets and how many of these are duplicates?

Zoxc · 2018-12-13T05:23:07Z

I was considering just using an atomic Vec for read_index then do deduplication after when it can no longer be shared between threads.

michaelwoerister · 2018-12-13T07:14:26Z

Yes, @wesleywiser collected some data a while ago and even implemented delayed deduplication but it didn't improve performance and increased memory usage. Here's the relevant issue and PR:

Explore delayed read-edge deduplication or getting rid of it entirely - incr.comp.: Explore delayed read-edge deduplication or getting rid of it entirely. #45873
[incremental] Collect stats about duplicated edge reads from queries - [incremental] Collect stats about duplicated edge reads from queries #46068

I also did some experiments similar to this one but it didn't improve performance either:

incr.comp.: Use a set implementation optimized for small item counts for deduplicating read-edges - incr.comp.: Use a set implementation optimized for small item counts for deduplicating read-edges. #45577

I don't expect this to be much of a win, if at all. It's a spare-time project for me, more or less :) @nnethercote's optimization to use a SmallVec (#50565) was more important.

-Z incremental-info should tell you how many reads are duplicated, IIRC.

rust-timer · 2018-12-13T07:30:49Z

Finished benchmarking try commit b350a5644a4b98b0c16953bc075e40ec5cb2cbd0

michaelwoerister · 2018-12-13T10:40:25Z

Yeah, that doesn't seem to make a measurable difference. Bummer.

rust-highfive assigned eddyb Dec 11, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 11, 2018

michaelwoerister unassigned eddyb Dec 11, 2018

michaelwoerister closed this Dec 12, 2018

michaelwoerister reopened this Dec 12, 2018

michaelwoerister force-pushed the depgraph-readindex-tweak branch from 34c3d3c to ed37dd0 Compare December 12, 2018 15:24

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 12, 2018

michaelwoerister force-pushed the depgraph-readindex-tweak branch from ed37dd0 to 453141b Compare December 12, 2018 15:58

Allocate DepGraph::read_index de-duplication hashset more lazily.

a40d83e

michaelwoerister force-pushed the depgraph-readindex-tweak branch from 453141b to 9ab4bb7 Compare December 12, 2018 16:03

Implement DepGraph::read_index fast path via SSE2.

3d610b4

michaelwoerister force-pushed the depgraph-readindex-tweak branch from 9ab4bb7 to 3d610b4 Compare December 12, 2018 20:45

michaelwoerister closed this Dec 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[experiment] Allocate DepGraph::read_index de-duplication hashset more lazily. #56724

[experiment] Allocate DepGraph::read_index de-duplication hashset more lazily. #56724

michaelwoerister commented Dec 11, 2018

rust-highfive commented Dec 11, 2018

michaelwoerister commented Dec 11, 2018

bors commented Dec 11, 2018

bors commented Dec 11, 2018

michaelwoerister commented Dec 12, 2018

rust-timer commented Dec 12, 2018

rust-timer commented Dec 12, 2018

michaelwoerister commented Dec 12, 2018

michaelwoerister commented Dec 12, 2018

bors commented Dec 12, 2018

rust-highfive commented Dec 12, 2018

bors commented Dec 12, 2018

rust-highfive commented Dec 12, 2018

michaelwoerister commented Dec 12, 2018

bors commented Dec 12, 2018

michaelwoerister commented Dec 12, 2018

bors commented Dec 12, 2018

bors commented Dec 12, 2018

rust-highfive commented Dec 12, 2018

michaelwoerister commented Dec 12, 2018

bors commented Dec 12, 2018

bors commented Dec 12, 2018

Zoxc commented Dec 13, 2018

rust-timer commented Dec 13, 2018

Zoxc commented Dec 13, 2018

Zoxc commented Dec 13, 2018

michaelwoerister commented Dec 13, 2018

rust-timer commented Dec 13, 2018

michaelwoerister commented Dec 13, 2018

[experiment] Allocate DepGraph::read_index de-duplication hashset more lazily. #56724

[experiment] Allocate DepGraph::read_index de-duplication hashset more lazily. #56724

Conversation

michaelwoerister commented Dec 11, 2018

rust-highfive commented Dec 11, 2018

michaelwoerister commented Dec 11, 2018

bors commented Dec 11, 2018

bors commented Dec 11, 2018

michaelwoerister commented Dec 12, 2018

rust-timer commented Dec 12, 2018

rust-timer commented Dec 12, 2018

michaelwoerister commented Dec 12, 2018

michaelwoerister commented Dec 12, 2018

bors commented Dec 12, 2018

rust-highfive commented Dec 12, 2018

bors commented Dec 12, 2018

rust-highfive commented Dec 12, 2018

michaelwoerister commented Dec 12, 2018

bors commented Dec 12, 2018

michaelwoerister commented Dec 12, 2018

bors commented Dec 12, 2018

bors commented Dec 12, 2018

rust-highfive commented Dec 12, 2018

michaelwoerister commented Dec 12, 2018

bors commented Dec 12, 2018

bors commented Dec 12, 2018

Zoxc commented Dec 13, 2018

rust-timer commented Dec 13, 2018

Zoxc commented Dec 13, 2018

Zoxc commented Dec 13, 2018

michaelwoerister commented Dec 13, 2018

rust-timer commented Dec 13, 2018

michaelwoerister commented Dec 13, 2018