Eliminate excessive null-checks from slice iterators #21886

dotdash · 2015-02-03T07:55:43Z

The data pointer used in the slice is never null, using assume() to tell
LLVM about it gets rid of various unneeded null checks when iterating
over the slice.

Since the snapshot compiler is still using an older LLVM version, omit
the call in stage0, because compile times explode otherwise.

Benchmarks from #18193

running 5 tests
test _range    ... bench:     33329 ns/iter (+/- 417)
test assembly  ... bench:     33299 ns/iter (+/- 58)
test enumerate ... bench:     33318 ns/iter (+/- 83)
test iter      ... bench:     33311 ns/iter (+/- 130)
test position  ... bench:     33300 ns/iter (+/- 47)

test result: ok. 0 passed; 0 failed; 0 ignored; 5 measured

Fixes #18193

rust-highfive · 2015-02-03T07:55:52Z

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

huonw · 2015-02-03T08:37:45Z

From the description I take it this doesn't suffer from the compile time problem mentioned in #21418 (comment)? (Out of interest what changed in LLVM? Just general optimisations?)

Also, I found that the compiler aborted compiling libcore (can't remember if it was stage1 or stage2) when I did a similar change (#21448); this doesn't suffer from that?

dotdash · 2015-02-03T08:51:13Z

From the description I take it this doesn't suffer from the compile time problem mentioned in #21418 (comment)?

Right, I compared compile times for (IIRC) rustc_trans and rustc_typeck and there was no measureable difference in compile times with the assume in stage 2 (or maybe it was stage 1 that I tested, shouldn't matter).

I also just bootstrapped a new version of #21418 and that took ~24 minutes from finishing stage0 libcore to finishing stage2 librustc. Considering that just rustc_trans took 13 minutes when the slowdown was there, it looks good to me ;-)

(Out of interest what changed in LLVM? Just general optimisations?)

I don't know.

Also, I found that the compiler aborted compiling libcore (can't remember if it was stage1 or stage2) when I did a similar change (#21448); this doesn't suffer from that?

The benchmark was taken with the stage2 rustc, so no, it didn't abort.

huonw · 2015-02-03T08:56:58Z

Could you add the assume to next_back as well?

dotdash · 2015-02-03T09:10:24Z

Why next_back? Wouldn't as_mut_slice and into_iter be the places that correspond to as_slice?

I originally had it in next (which is why I forgot to add it to as_mut_slice, because next is generated in a macro that is used for both iter types), and that gave worse results, because the assumption got lost at some point when SROA (I think) replaced the load in next() with the one from as_slice.

huonw · 2015-02-03T09:53:38Z

Oh, huh. I totally misread this PR. I was thinking this was in <slice::Iter<T> as Iterator>::next.

huonw · 2015-02-03T09:56:36Z

I suspect that means that this doesn't solve a case like http://is.gd/pteFUM,

.LBB0_2:
    testq   %rdx, %rdx
    je  .LBB0_4
    addl    (%rdx), %eax
    addq    $4, %rdx
    addq    $-4, %rcx
    jne .LBB0_2

since there's no Vec involved. I wonder if that would be addressed by placing assumptions in the iter/iter_mut constructor?

dotdash · 2015-02-03T10:08:15Z

Will try to see if that helps

jdm · 2015-02-03T10:23:54Z

src/libcollections/vec.rs

@@ -427,8 +428,12 @@ impl<T> Vec<T> {
    #[stable(feature = "rust1", since = "1.0.0")]
    pub fn as_mut_slice<'a>(&'a mut self) -> &'a mut [T] {
        unsafe {
+            let ptr = *self.ptr;
+            if cfg!(not(stage0)) { // NOTE remove cfg after next snapshot
+                assume(ptr != 0 as *mut T);


ptr::null_mut() here and all other uses of 0 as *mut T

Actually, using is_null() works now, that didn't work the first time I tried, either because of an error on my side or because of the ptrtoint instruction it was generated with the old is_null() implementation.
Thanks!

dotdash · 2015-02-03T10:43:38Z

I wonder if that would be addressed by placing assumptions in the iter/iter_mut constructor?

At least in this case, the assume() gets canonicalized to !nonnull metadata on a load, and that gets lost in the SROA pass. I've added the call anyway, hoping for it to be useful in the future.

huonw · 2015-02-03T10:46:30Z

@bors r+ e49f

alexcrichton · 2015-02-03T19:40:37Z

@bors: rollup

alexcrichton · 2015-02-04T16:20:01Z

@bors: rollup-

bors · 2015-02-04T23:07:27Z

⌛ Testing commit e49f6d6 with merge cd0d956...

bors · 2015-02-04T23:27:57Z

💔 Test failed - auto-win-32-nopt-t

bluss · 2015-02-05T19:12:38Z

@huonw, that fold testcase seems to have been resolved now, it vectorizes! (rustc 1.0.0-nightly (ba2f13ef0 2015-02-04 20:03:55 +0000) Probably due to the other awesome nonnull changes?

huonw · 2015-02-07T12:05:34Z

The failure seems legitimate; although, possibly a LLVM bug?

dotdash · 2015-02-09T21:06:12Z

Yeah, I can reproduce that in my Windows VM.

dotdash · 2015-02-10T13:22:33Z

Yup, LLVM bug: http://reviews.llvm.org/D7533

Fixes the crash blocking rust-lang#21886.

nagisa · 2015-02-18T12:59:23Z

LLVM update landed. Needs rebase.

Casting the pointer to an integer requires a ptrtoint, while casting 0 to a pointer is directly folded to a `null` value.

The data pointer used in the slice is never null, using assume() to tell LLVM about it gets rid of various unneeded null checks when iterating over the slice. Since the snapshot compiler is still using an older LLVM version, omit the call in stage0, because compile times explode otherwise. Benchmarks from rust-lang#18193 ```` running 5 tests test _range ... bench: 33329 ns/iter (+/- 417) test assembly ... bench: 33299 ns/iter (+/- 58) test enumerate ... bench: 33318 ns/iter (+/- 83) test iter ... bench: 33311 ns/iter (+/- 130) test position ... bench: 33300 ns/iter (+/- 47) test result: ok. 0 passed; 0 failed; 0 ignored; 5 measured ```` Fixes rust-lang#18193

dotdash · 2015-02-18T13:06:24Z

@bors r=huonw 7412d1b

bors · 2015-02-18T13:06:27Z

⌛ Testing commit 7412d1b with merge c103604...

bors · 2015-02-18T14:51:02Z

💔 Test failed - auto-win-32-nopt-t

alexcrichton · 2015-02-18T15:48:10Z

@bors: retry

bors · 2015-02-18T17:54:44Z

⌛ Testing commit 7412d1b with merge a1cfc62...

The data pointer used in the slice is never null, using assume() to tell LLVM about it gets rid of various unneeded null checks when iterating over the slice. Since the snapshot compiler is still using an older LLVM version, omit the call in stage0, because compile times explode otherwise. Benchmarks from #18193 ```` running 5 tests test _range ... bench: 33329 ns/iter (+/- 417) test assembly ... bench: 33299 ns/iter (+/- 58) test enumerate ... bench: 33318 ns/iter (+/- 83) test iter ... bench: 33311 ns/iter (+/- 130) test position ... bench: 33300 ns/iter (+/- 47) test result: ok. 0 passed; 0 failed; 0 ignored; 5 measured ```` Fixes #18193

bors · 2015-02-18T19:17:07Z

💔 Test failed - auto-win-32-nopt-t

aturon · 2015-02-18T19:25:40Z

@bors: retry

The data pointer used in the slice is never null, using assume() to tell LLVM about it gets rid of various unneeded null checks when iterating over the slice. Since the snapshot compiler is still using an older LLVM version, omit the call in stage0, because compile times explode otherwise. Benchmarks from rust-lang#18193 ```` running 5 tests test _range ... bench: 33329 ns/iter (+/- 417) test assembly ... bench: 33299 ns/iter (+/- 58) test enumerate ... bench: 33318 ns/iter (+/- 83) test iter ... bench: 33311 ns/iter (+/- 130) test position ... bench: 33300 ns/iter (+/- 47) test result: ok. 0 passed; 0 failed; 0 ignored; 5 measured ```` Fixes rust-lang#18193

This adds the assume() calls back that got lost when rebasing rust-lang#21886.

This adds the assume() calls back that got lost when rebasing #21886.

frol · 2018-05-09T14:30:51Z

Could someone, please, point me to a tracking issue or a documentation about the excessive null-checks?

I want to learn more about the cause of extra testq %rsi, %rsi (test rsi, rsi) blocks when I compile the following snippets:

pub fn foo(s: &[u32]) -> u32 {
    s.iter().sum()
}

pub fn foo(s: &[u32]) -> u32 {
    s.iter().sum()
}

pub fn foo(s: Vec<i32>) -> usize {
    s.len()
}

and even

pub fn foo(_s: Vec<i32>) -> usize {
    0
}

produces assembly code which might have been avoided:

playground::foo:
	movq	8(%rdi), %rsi
	testq	%rsi, %rsi
	je	.LBB0_2
	pushq	%rax
	movq	(%rdi), %rdi
	shlq	$2, %rsi
	movl	$4, %edx
	callq	__rust_dealloc@PLT
	addq	$8, %rsp

.LBB0_2:
	xorl	%eax, %eax
	retq

nagisa · 2018-05-09T16:05:01Z

In both these cases the length, rather than pointer is being checked. For the slice case as early exit before a vectorised solution is used, for Vec drop case because Vec may actually not have backing storage (Vec::new does not allocate).

frol · 2018-05-09T16:22:32Z

for Vec drop case because Vec may actually not have backing storage (Vec::new does not allocate).

Indeed, I completely forgot that the owned object must be deallocated here.

For the slice case as early exit before a vectorised solution is used

Right...

@nagisa I am sorry for the trouble! Rust is awesome! :)

rust-highfive assigned nikomatsakis Feb 3, 2015

dotdash force-pushed the fast_slice_iter branch from fb2e27b to a2dcc1e Compare February 3, 2015 09:56

jdm reviewed Feb 3, 2015
View reviewed changes

dotdash force-pushed the fast_slice_iter branch from a2dcc1e to e49f6d6 Compare February 3, 2015 10:43

nikomatsakis assigned huonw and unassigned nikomatsakis Feb 3, 2015

This was referenced Feb 12, 2015

Update to release_36@228965 rust-lang/llvm#35

Closed

Update to release_36@229036 rust-lang/llvm#36

Closed

alexcrichton added a commit to alexcrichton/rust that referenced this pull request Feb 18, 2015

rollup merge of rust-lang#22332: dotdash/llvmup_20150213

02c2761

Fixes the crash blocking rust-lang#21886.

dotdash added 2 commits February 18, 2015 14:04

Avoid ptrtoint when checking if a pointer is null

52b5150

Casting the pointer to an integer requires a ptrtoint, while casting 0 to a pointer is directly folded to a `null` value.

dotdash force-pushed the fast_slice_iter branch from e49f6d6 to 7412d1b Compare February 18, 2015 13:05

alexcrichton mentioned this pull request Feb 18, 2015

Basic implementation of the string pattern API #22466

Merged

bors merged commit 7412d1b into rust-lang:master Feb 19, 2015

dotdash deleted the fast_slice_iter branch February 22, 2015 13:26

dotdash added a commit to dotdash/rust that referenced this pull request Feb 22, 2015

Eliminate more excessive null-checks from slice iterators

e457328

This adds the assume() calls back that got lost when rebasing rust-lang#21886.

dotdash mentioned this pull request Feb 22, 2015

Eliminate more excessive null-checks from slice iterators #22669

Merged

bors added a commit that referenced this pull request Feb 28, 2015

Auto merge of #22669 - dotdash:fast_slice_iter, r=huonw

1c93934

This adds the assume() calls back that got lost when rebasing #21886.

Gankra mentioned this pull request May 11, 2015

Handle overflow properly in core::slice #25300

Merged

pnkfelix mentioned this pull request Mar 21, 2023

Remove the assume(!is_null) from Vec::as_ptr #106967

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eliminate excessive null-checks from slice iterators #21886

Eliminate excessive null-checks from slice iterators #21886

dotdash commented Feb 3, 2015

rust-highfive commented Feb 3, 2015

huonw commented Feb 3, 2015

dotdash commented Feb 3, 2015

huonw commented Feb 3, 2015

dotdash commented Feb 3, 2015

huonw commented Feb 3, 2015

huonw commented Feb 3, 2015

dotdash commented Feb 3, 2015

jdm Feb 3, 2015

dotdash Feb 3, 2015

dotdash commented Feb 3, 2015

huonw commented Feb 3, 2015

alexcrichton commented Feb 3, 2015

alexcrichton commented Feb 4, 2015

bors commented Feb 4, 2015

bors commented Feb 4, 2015

bluss commented Feb 5, 2015

huonw commented Feb 7, 2015

dotdash commented Feb 9, 2015

dotdash commented Feb 10, 2015

nagisa commented Feb 18, 2015

dotdash commented Feb 18, 2015

bors commented Feb 18, 2015

bors commented Feb 18, 2015

alexcrichton commented Feb 18, 2015

bors commented Feb 18, 2015

bors commented Feb 18, 2015

aturon commented Feb 18, 2015

frol commented May 9, 2018

nagisa commented May 9, 2018

frol commented May 9, 2018

Eliminate excessive null-checks from slice iterators #21886

Eliminate excessive null-checks from slice iterators #21886

Conversation

dotdash commented Feb 3, 2015

rust-highfive commented Feb 3, 2015

huonw commented Feb 3, 2015

dotdash commented Feb 3, 2015

huonw commented Feb 3, 2015

dotdash commented Feb 3, 2015

huonw commented Feb 3, 2015

huonw commented Feb 3, 2015

dotdash commented Feb 3, 2015

jdm Feb 3, 2015

Choose a reason for hiding this comment

dotdash Feb 3, 2015

Choose a reason for hiding this comment

dotdash commented Feb 3, 2015

huonw commented Feb 3, 2015

alexcrichton commented Feb 3, 2015

alexcrichton commented Feb 4, 2015

bors commented Feb 4, 2015

bors commented Feb 4, 2015

bluss commented Feb 5, 2015

huonw commented Feb 7, 2015

dotdash commented Feb 9, 2015

dotdash commented Feb 10, 2015

nagisa commented Feb 18, 2015

dotdash commented Feb 18, 2015

bors commented Feb 18, 2015

bors commented Feb 18, 2015

alexcrichton commented Feb 18, 2015

bors commented Feb 18, 2015

bors commented Feb 18, 2015

aturon commented Feb 18, 2015

frol commented May 9, 2018

nagisa commented May 9, 2018

frol commented May 9, 2018