Memory corruption (?) from tower-grpc-build on nightly #47175

vorner · 2018-01-04T09:20:20Z

Hello

When playing with some development versions of tower-grpc-build (https://github.com/tower-rs/tower/grpc), I discovered a strange crash of the build script. I managed to shring it a bit, but I don't think this is minimal case (I just don't know how to continue pinning it down further).

Few observations:

The string it talks about in the backtrace seems like random junk (which isn't always the same). I'm not sure how it got there, but in general, Rust probably should prevent creation of strings with uninitialized content.
It doesn't happen on stable.
It happens on today's and yesterday's nightly (exact version below).
It doesn't happen on approx. a week old nightly, but I'm not at the computer that has the older version, I'll look exact version when I get to it in few hours.

As I needed to pin exact commits in Cargo.lock to make the branches and crates from git to compile together, I have the example as a whole repository: https://github.com/vorner/p-crash.

Steps to reproduce (I moved the failing code from build script to ordinary main.rs, therefore the OUT_DIR):

rustup default nightly
OUT_DIR="." cargo run

The backtrace:

thread 'main' panicked at 'byte index 1 is not a char boundary; it is inside '\u{0}' (bytes 0..1) of ` âU�âUâU� PâU�Cbyte index 1 is not a char boundary; it is inside ams_p._ping_re`[...]', /checkout/src/libcore/str/mod.rs:2234:5
stack backtrace:
   0:     0x564e394ee1fb - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::hb720ecb1cdd94d34
                               at /checkout/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1:     0x564e394f1c4e - std::sys_common::backtrace::print::h3b9052ae1243ac3c
                               at /checkout/src/libstd/sys_common/backtrace.rs:68
                               at /checkout/src/libstd/sys_common/backtrace.rs:57
   2:     0x564e394e8930 - std::panicking::default_hook::{{closure}}::h8386650ae58050b2
                               at /checkout/src/libstd/panicking.rs:381
   3:     0x564e394e8465 - std::panicking::default_hook::h939eef3f926b9ec4
                               at /checkout/src/libstd/panicking.rs:397
   4:     0x564e394e8cbb - std::panicking::rust_panic_with_hook::hd65467d489bb99a6
                               at /checkout/src/libstd/panicking.rs:577
   5:     0x564e394e8b0e - std::panicking::begin_panic::h06b05d85af0a01cf
                               at /checkout/src/libstd/panicking.rs:538
   6:     0x564e394e8a7a - std::panicking::begin_panic_fmt::hdcdf37dcdaa48dbb
                               at /checkout/src/libstd/panicking.rs:522
   7:     0x564e394e8a12 - rust_begin_unwind
                               at /checkout/src/libstd/panicking.rs:498
   8:     0x564e39540e40 - core::panicking::panic_fmt::h1bceab3183b318fb
                               at /checkout/src/libcore/panicking.rs:71
   9:     0x564e3953f4ed - core::str::slice_error_fail::h92f09d0a955fe52f
                               at /checkout/src/libcore/str/mod.rs:0
  10:     0x564e394a04de - core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::RangeFrom<usize>>::index::{{closure}}::h7353d5e4c348ff28
                               at /checkout/src/libcore/str/mod.rs:1987
  11:     0x564e3949e455 - <core::option::Option<T>>::unwrap_or_else::hde62f1acbfa5a0d6
                               at /checkout/src/libcore/option.rs:376
  12:     0x564e394a0ad9 - core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::RangeFrom<usize>>::index::hea07b70cbc76dfe2
                               at /checkout/src/libcore/str/mod.rs:1987
  13:     0x564e394a0494 - core::str::traits::<impl core::ops::index::Index<core::ops::range::RangeFrom<usize>> for str>::index::he1272c2136040a65
                               at /checkout/src/libcore/str/mod.rs:1734
  14:     0x564e3949d564 - unicode_segmentation::word::UWordBounds::get_next_cat::hd9df4c8a50c69cf3
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-segmentation-1.2.0/src/word.rs:611
  15:     0x564e3949cc48 - <unicode_segmentation::word::UWordBounds<'a> as core::iter::iterator::Iterator>::next::hfaaee7c167be77af
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-segmentation-1.2.0/src/word.rs:227
  16:     0x564e3949d7ee - <&'a mut I as core::iter::iterator::Iterator>::next::h1d6ba2952b78d957
                               at /checkout/src/libcore/iter/iterator.rs:2380
  17:     0x564e394a36b8 - <core::iter::Filter<I, P> as core::iter::iterator::Iterator>::next::h735c3caf6113fcd5
                               at /checkout/src/libcore/iter/mod.rs:1362
  18:     0x564e3949d44b - <unicode_segmentation::word::UnicodeWords<'a> as core::iter::iterator::Iterator>::next::h05b4feaa8a7b6004
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-segmentation-1.2.0/src/word.rs:30
  19:     0x564e394a12ea - heck::transform::h2bd9c6ea08dc34e6
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/heck-0.3.0/src/lib.rs:81
  20:     0x564e3949c503 - <str as heck::snake::SnakeCase>::to_snake_case::h13cce73744a245f8
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/heck-0.3.0/src/snake.rs:37
  21:     0x564e3942fef1 - prost_build::ident::to_snake::h6cb82fcbe7a31a26
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/ident.rs:8
  22:     0x564e3944b177 - core::ops::function::FnMut::call_mut::he9df2a59f21ce2ef
                               at /checkout/src/libcore/ops/function.rs:146
  23:     0x564e3942dd36 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &'a mut F>::call_once::hc412d0b7423e7dd9
                               at /checkout/src/libcore/ops/function.rs:271
  24:     0x564e3940dadf - <core::option::Option<T>>::map::ha68eff2bf856d1c0
                               at /checkout/src/libcore/option.rs:404
  25:     0x564e3944789a - <core::iter::Map<I, F> as core::iter::iterator::Iterator>::next::hf391941cd12c07f0
                               at /checkout/src/libcore/iter/mod.rs:1251
  26:     0x564e39447ce8 - <core::iter::Chain<A, B> as core::iter::iterator::Iterator>::next::hde73355ad686dd14
                               at /checkout/src/libcore/iter/mod.rs:758
  27:     0x564e39447aaa - <core::iter::Chain<A, B> as core::iter::iterator::Iterator>::next::h158ea3896fb6310f
                               at /checkout/src/libcore/iter/mod.rs:754
  28:     0x564e39449b76 - itertools::Itertools::join::h6e55fdae653e6cbd
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/itertools-0.6.5/src/lib.rs:1203
  29:     0x564e3943e209 - prost_build::code_generator::CodeGenerator::resolve_ident::hf5ad253193aac324
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/code_generator.rs:574
  30:     0x564e3943d192 - prost_build::code_generator::CodeGenerator::unpack_service::{{closure}}::h00de60f26a10433d
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/code_generator.rs:477
  31:     0x564e39433c77 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &'a mut F>::call_once::h4b79d43010f986cb
                               at /checkout/src/libcore/ops/function.rs:271
  32:     0x564e3940ce5e - <core::option::Option<T>>::map::h933162c45fb6401e
                               at /checkout/src/libcore/option.rs:404
  33:     0x564e394476c3 - <core::iter::Map<I, F> as core::iter::iterator::Iterator>::next::h0913e1a7621c5c9e
                               at /checkout/src/libcore/iter/mod.rs:1251
  34:     0x564e3941dc9c - <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::spec_extend::hb997b01e2b5e5058
                               at /checkout/src/liballoc/vec.rs:1844
  35:     0x564e3941ed6e - <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::from_iter::h777d37041a1c7591
                               at /checkout/src/liballoc/vec.rs:1827
  36:     0x564e3941f336 - <alloc::vec::Vec<T> as core::iter::traits::FromIterator<T>>::from_iter::h43714bb9d059e0ac
                               at /checkout/src/liballoc/vec.rs:1713
  37:     0x564e39444c87 - core::iter::iterator::Iterator::collect::h436cb81ed82afe86
                               at /checkout/src/libcore/iter/iterator.rs:1298
  38:     0x564e3943ca1b - prost_build::code_generator::CodeGenerator::unpack_service::hfea0f83504e71fec
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/code_generator.rs:465
  39:     0x564e39434d87 - prost_build::code_generator::CodeGenerator::generate::hab3bbb91478da3b9
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/code_generator.rs:111
  40:     0x564e3940a9c4 - prost_build::Config::generate::hd148016b47bb16b0
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/lib.rs:342
  41:     0x564e393d1aff - prost_build::Config::compile_protos::hc4c921ff79be61f5
                               at /home/vorner/.cargo/registry/src/github.com-1ecc6299db9ec823/prost-build-0.2.3/src/lib.rs:321
  42:     0x564e393e02e6 - tower_grpc_build::Config::build::h85b9824f7574fc87
                               at /home/vorner/.cargo/git/checkouts/tower-grpc-02dc54165a968a91/a844add/tower-grpc-build/src/lib.rs:92
  43:     0x564e393dde07 - sscli::main::h0f8da34d077bac22
                               at src/main.rs:4
  44:     0x564e393e0171 - std::rt::lang_start::{{closure}}::h115d56dea3cc829d
                               at /checkout/src/libstd/rt.rs:74
  45:     0x564e394e89d7 - std::panicking::try::do_call::h9688ac8626758e25
                               at /checkout/src/libstd/rt.rs:59
                               at /checkout/src/libstd/panicking.rs:480
  46:     0x564e395069fe - __rust_maybe_catch_panic
                               at /checkout/src/libpanic_unwind/lib.rs:101
  47:     0x564e394edfd5 - std::rt::lang_start_internal::ha33cbd0fd3958dd3
                               at /checkout/src/libstd/panicking.rs:459
                               at /checkout/src/libstd/panic.rs:365
                               at /checkout/src/libstd/rt.rs:58
  48:     0x564e393e0151 - std::rt::lang_start::h40dceadf8934dbfd
                               at /checkout/src/libstd/rt.rs:74
  49:     0x564e393dde5d - main
  50:     0x7f0656239040 - __libc_start_main
  51:     0x564e393934d9 - _start
  52:                0x0 - <unknown>

It is likely something fishy is happening in some of the development branches, but I still suspect something strange happening in the compiler, both because of the uninitialized memory (or, memory looking uninitialized) and because it passes on stable, but not nightly.

The text was updated successfully, but these errors were encountered:

nikomatsakis · 2018-01-04T15:36:33Z

cc @rust-lang/infra -- any chance somebody can bisect this a bit to see where the problem started?

nikomatsakis · 2018-01-04T15:36:45Z

triage: P-high

vorner · 2018-01-04T15:57:52Z

If it helps, this one goes through (rustc 1.24.0-nightly (9389e23a8 2017-12-31)):

OUT_DIR=. cargo +nightly-2018-01-01 run

And this one fails (rustc 1.24.0-nightly (b65f0bedd 2018-01-01)):

OUT_DIR=. cargo +nightly-2018-01-02 run

Anything more exact would need setting up of some non-trivial infrastructure on my own computer.

kennytm · 2018-01-04T16:15:58Z

Given the range 9389e23...b65f0be I'm pretty sure it is #46735, but let me bisect anyway...

Edit: bisect-rust failed due to can't find crate for `proc_macro` ಠ_ಠ needs to reduce it first.

alexcrichton · 2018-01-04T16:35:52Z

cc @Manishearth

kennytm · 2018-01-04T17:03:41Z

Reductions:

Reducing dependency to prost_build.

extern crate prost_build;

struct SG;
impl prost_build::ServiceGenerator for SG {
    fn generate(&self, _: prost_build::Service, _: &mut String) {
    }
}

fn main() {
    let mut cfg = prost_build::Config::new();
    cfg.service_generator(Box::new(SG));
    cfg.compile_protos(&["p.proto"], &["."]).unwrap();
}

Removed all dependencies. Repro on playground.

fn main() {
    let pb_ident = "p.q";
    let mut ident_path = pb_ident.split('.');
    ident_path.next_back();
    let result = ident_path.collect::<Vec<_>>();
    assert_eq!(result, vec!["p"]);
}

Edit: Hold on, this program affects beta (1.24.0-beta.1) as well 😱

kennytm · 2018-01-04T17:52:38Z

Bisecting in the range 9389e23...8e7a609

Test script:

#!/bin/sh
set -eu
$RUSTC_RELATIVE 1.rs
! ./1

(1.rs has content shown in #47175 (comment))

Bisect result:

$ target/release/bisect --preserve --test test.sh --start 9389e23a8a754097e233c7bf3ea1bb404ccf1075 --end 8e7a609e635b728eba65d471c985ab462dc4cfc7
INFO:rust_sysroot: Getting commits from the git checkout in 9389e23a8a754097e233c7bf3ea1bb404ccf1075...8e7a609e635b728eba65d471c985ab462dc4cfc7
INFO:rust_sysroot: Received 23 commits
Searching in 23 commits; about 5 steps
thread 'main' panicked at 'byte index 1747 is not a char boundary; it is inside '\u{0}' (bytes 1746..1747) of `qpassertion failed: `(left == right)`
  left: ``,
 right: ``  1.rscapacity overflowsrc/l`[...]', src/libcore/str/mod.rs:2234:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'main' panicked at 'byte index 1747 is not a char boundary; it is inside '\u{0}' (bytes 1746..1747) of `qpassertion failed: `(left == right)`
  left: ``,
 right: ``  1.rscapacity overflowsrc/l`[...]', src/libcore/str/mod.rs:2234:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
searched commits 9389e23a8a754097e233c7bf3ea1bb404ccf1075 through 8e7a609e635b728eba65d471c985ab462dc4cfc7
regression in 8; Some(Commit { sha: "b65f0bedd2f22d9661ecb7092f07746dc2ccfb0d", date: 2018-01-01T19:04:33Z, summary: "Auto merge of #46735 - Manishearth:memchr-find, r=burntsushi" })

So yeah the cause is definitely #46735, and since it slips into beta it affects 1.24 beta as well.

Manishearth · 2018-01-04T17:59:28Z

Huh. Wat. I'll fix this tomorrow.

Manishearth · 2018-01-05T04:57:40Z

Okay, so the bug seems to be a discrepancy in how I thought the searcher API is supposed to work.

    let pb_ident = "p.q";
    let mut searcher = '.'.into_searcher(pb_ident);
    println!("{:?} {:?}", searcher.next_match_back(), searcher.next_match());

in old Rust this returns Some(1,2) and None, in new Rust it returns the same thing twice.

I was under the impression that double ended searchers had independent fingers, but this seems to be wrong.

We really need to overhaul this API so that it actually makes sense.

Manishearth · 2018-01-05T05:42:45Z

Fixed in #47208 r?

…lang#47175)

@BurntSushi

Make double ended searchers use dependent fingers (fixes #47175) r? @BurntSushi @alexcrichton needs uplift to beta

SirVer · 2018-01-10T13:37:19Z

@Manishearth Our build is crashing through this bug. Nightly is green again after this PR was merged, however beta is still red.

Will beta still be rolled out to stable and break our build in a few weeks? Or will there be a beta hotfix with this PR included?

kennytm · 2018-01-10T13:41:34Z

@SirVer Yes we are going to backport #47208 to 1.24 beta.

Manishearth · 2018-01-10T14:01:27Z

It's been marked beta-nominated, so it will be included before we make a stable release.

I don't know if it will be included in a new beta before that. Either way, I'm the wrong person to ask, @alexcrichton would know

Mark-Simulacrum · 2018-01-10T14:05:48Z

Yes, we will (likely) publish a new beta with this backport. There's 5(?) weeks before the next release, so we'll need at least 2-3 more betas probably

…lang#47175)

SirVer · 2018-01-11T09:01:34Z

Awesome! Thanks for clarifying.

Mark-Simulacrum · 2018-01-13T01:17:32Z

@SirVer I believe the latest beta should have the fix (.2).

Upstream bug rust-lang/rust#47175 is fixed, so we enable beta again as required for passing.

kennytm added I-crash Issue: The compiler crashes (SIGSEGV, SIGABRT, etc). Use I-ICE instead when the compiler panics. regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. labels Jan 4, 2018

pnkfelix self-assigned this Jan 4, 2018

rust-highfive added the P-high High priority label Jan 4, 2018

vorner mentioned this issue Jan 4, 2018

Add ServiceGenerator::finalize tokio-rs/prost#65

Merged

kennytm removed I-unsound Issue: A soundness hole (worst kind of bug), see: https://en.wikipedia.org/wiki/Soundness regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. labels Jan 4, 2018

alexcrichton added the I-nominated label Jan 4, 2018

This was referenced Jan 4, 2018

panicked at 'byte index 1 is not a char boundary; it is inside '\u{1}' (bytes 0..1) tokio-rs/prost#66

Closed

Prost broken afer nightly-2017-12-31 (due to memchr change?) #47192

Closed

Manishearth mentioned this issue Jan 5, 2018

Make double ended searchers use dependent fingers #47208

Merged

kennytm mentioned this issue Jan 5, 2018

ICE after rust update #47218

Closed

Manishearth added a commit to Manishearth/rust that referenced this issue Jan 6, 2018

Make double ended string searchers use dependent fingers (fixes rust-…

ff55725

…lang#47175)

Manishearth added a commit to Manishearth/rust that referenced this issue Jan 6, 2018

Regression tests for rust-lang#47175

9066219

danburkert mentioned this issue Jan 6, 2018

Probst compile_protobuf crashes our CI tokio-rs/prost#67

Closed

bors added a commit that referenced this issue Jan 8, 2018

Auto merge of #47208 - Manishearth:double-ended-searcher, r=pnkfelix

b5392f5

Make double ended searchers use dependent fingers (fixes #47175) r? @BurntSushi @alexcrichton needs uplift to beta

bors closed this as completed in #47208 Jan 8, 2018

vorner mentioned this issue Jan 8, 2018

Preliminary support for .proto files with multiple services tower-rs/tower-grpc#10

Merged

MaloJaffre pushed a commit to MaloJaffre/rust that referenced this issue Jan 10, 2018

Make double ended string searchers use dependent fingers (fixes rust-…

61016c1

…lang#47175)

MaloJaffre pushed a commit to MaloJaffre/rust that referenced this issue Jan 10, 2018

Regression tests for rust-lang#47175

ccca24d

SirVer added a commit to SirVer/point_cloud_viewer that referenced this issue Jan 15, 2018

Make beta required to pass on travis again.

41833e8

Upstream bug rust-lang/rust#47175 is fixed, so we enable beta again as required for passing.

SirVer mentioned this issue Jan 15, 2018

Make beta required to pass on travis again. cartographer-project/point_cloud_viewer#67

Merged

wally-the-cartographer pushed a commit to cartographer-project/point_cloud_viewer that referenced this issue Jan 15, 2018

Make beta required to pass on travis again. (#67)

bb73289

Upstream bug rust-lang/rust#47175 is fixed, so we enable beta again as required for passing.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memory corruption (?) from tower-grpc-build on nightly #47175

Memory corruption (?) from tower-grpc-build on nightly #47175

vorner commented Jan 4, 2018

nikomatsakis commented Jan 4, 2018

nikomatsakis commented Jan 4, 2018

vorner commented Jan 4, 2018

kennytm commented Jan 4, 2018 •

edited

Loading

alexcrichton commented Jan 4, 2018

kennytm commented Jan 4, 2018 •

edited

Loading

kennytm commented Jan 4, 2018

Manishearth commented Jan 4, 2018

Manishearth commented Jan 5, 2018

Manishearth commented Jan 5, 2018

SirVer commented Jan 10, 2018

kennytm commented Jan 10, 2018 •

edited

Loading

Manishearth commented Jan 10, 2018

Mark-Simulacrum commented Jan 10, 2018

SirVer commented Jan 11, 2018

Mark-Simulacrum commented Jan 13, 2018

Memory corruption (?) from tower-grpc-build on nightly #47175

Memory corruption (?) from tower-grpc-build on nightly #47175

Comments

vorner commented Jan 4, 2018

nikomatsakis commented Jan 4, 2018

nikomatsakis commented Jan 4, 2018

vorner commented Jan 4, 2018

kennytm commented Jan 4, 2018 • edited Loading

alexcrichton commented Jan 4, 2018

kennytm commented Jan 4, 2018 • edited Loading

kennytm commented Jan 4, 2018

Manishearth commented Jan 4, 2018

Manishearth commented Jan 5, 2018

Manishearth commented Jan 5, 2018

SirVer commented Jan 10, 2018

kennytm commented Jan 10, 2018 • edited Loading

Manishearth commented Jan 10, 2018

Mark-Simulacrum commented Jan 10, 2018

SirVer commented Jan 11, 2018

Mark-Simulacrum commented Jan 13, 2018

kennytm commented Jan 4, 2018 •

edited

Loading

kennytm commented Jan 4, 2018 •

edited

Loading

kennytm commented Jan 10, 2018 •

edited

Loading