Remove the `alloc_jemalloc` crate #55238

alexcrichton · 2018-10-21T02:21:07Z

This commit removes the alloc_jemalloc crate from the standard library and all related configuration. We will no longer be shipping this unstable crate. Rationale for this is provided on #36963 and the many linked issues, but I can inline rationale here if desired!

We currently rely on jemalloc for increased perf in the Rust compiler, however. This perf run shows that if we switch to glibc 2.23's allocator that it's slower than jemalloc across many benchmarks. This perf run, however, shows that if we use jemalloc-sys from crates.io then rustc actually gets faster across all benchmarks! (presumably because it has a more recent version of jemalloc than our submodule).

As a result, it's expected that this doesn't regress any code (as it's just removing an unstable crate) and it should actually improve rustc performance because it updates jemalloc.

Closes #36963

rust-highfive · 2018-10-21T02:21:10Z

r? @estebank

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2018-10-21T02:21:27Z

cc @sfackler

sfackler · 2018-10-21T03:25:13Z

Woo!

There is one pretty significant difference between jemalloc 4 and 5. The new version depends on a background thread to clear out allocation caches, and the background thread defaults to off. Without it, we saw a very large increase in heap memory on a production service until we enabled background threads. See jemalloc/jemalloc#1128 for some background.

While this doesn't matter for rustc at all, it will probably have an impact on long running services. We may want to fold the switch to the system allocator as the default into this change as well?

lnicola · 2018-10-21T04:48:36Z

Ah, that might explain https://perf.rust-lang.org/compare.html?start=22cc2ae8057d14e980b7c784e1eb2eee26b59e7d&end=10c95ccfa7a7adc12f4e608621ca29f9b98eed29&stat=max-rss.

SimonSapin · 2018-10-21T10:38:41Z

We may want to fold the switch to the system allocator as the default into this change as well?

As far as I understand, that switch is already part of this PR.

alexcrichton · 2018-10-21T11:05:36Z

@sfackler ah yeah as @SimonSapin mentioned this PR is doing the switch as well as the update.

Despite it looking like we enable the background thread by default in jemalloc-sys the build script only actually disables the background thread if it's turned off. @sfackler do you know the right option necessary to enable the background thread so we can test in rustc?

sfackler · 2018-10-21T15:03:59Z

Those options compile in support for the background thread but it's still disabled at runtime by default. I don't think it would have much of an effect on rustc memory usage - my understanding is that it's used to clean up caches on idle threads and that active threads clean themselves up periodically when touching the allocator.

There are a bunch of settings you can poke at to control how aggressively jemalloc caches if you want: http://jemalloc.net/jemalloc.3.html

estebank · 2018-10-21T18:55:35Z

@bors r+

rust-highfive · 2018-10-21T23:32:27Z

The job dist-x86_64-musl of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:47:25] failures:
[00:47:25] 
[00:47:25] ---- [ui] ui/huge-enum.rs stdout ----
[00:47:25] 
[00:47:25] error: Error: expected failure status (Some(1)) but received status None.
[00:47:25] status: signal: 11
[00:47:25] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/huge-enum.rs" "--target=x86_64-unknown-linux-musl" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/huge-enum/a" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-musl/native/rust-test-helpers" "-Clinker=/musl-x86_64/bin/musl-gcc" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/huge-enum/auxiliary" "-A" "unused"
[00:47:25] ------------------------------------------
[00:47:25] 
[00:47:25] ------------------------------------------
[00:47:25] stderr:
[00:47:25] stderr:
[00:47:25] ------------------------------------------
[00:47:25] {"message":"the type `std::option::Option<[u32; 35184372088831]>` is too big for the current architecture","code":null,"level":"error","spans":[],"children":[],"rendered":"error: the type `std::option::Option<[u32; 35184372088831]>` is too big for the current architecture\n\n"}
[00:47:25] {"message":"aborting due to previous error","code":null,"level":"error","spans":[],"children":[],"rendered":"error: aborting due to previous error\n\n"}
[00:47:25] ------------------------------------------
[00:47:25] 
[00:47:25] thread '[ui] ui/huge-enum.rs' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:3284:9
[00:47:25] note: Run with `RUST_BACKTRACE=1` for a backtrace.
---
[00:47:25] 
[00:47:25] thread 'main' panicked at 'Some tests failed', tools/compiletest/src/main.rs:503:22
[00:47:25] 
[00:47:25] 
[00:47:25] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-musl" "--mode" "ui" "--target" "x86_64-unknown-linux-musl" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/bin/FileCheck" "--linker" "/musl-x86_64/bin/musl-gcc" "--host-rustcflags" "-Crpath -O -Zunstable-options " "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-musl/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--llvm-version" "8.0.0svn\n" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[00:47:25] 
[00:47:25] 
[00:47:25] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --target x86_64-unknown-linux-musl
[00:47:25] Build completed unsuccessfully in 0:44:38
---
travis_time:end:2b62d04c:start=1540164656425485779,finish=1540164656435475220,duration=9989441
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0faaa9c0
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:start:crashlog
obj/cores/core.2489.!checkout!obj!build!x86_64-unknown-linux-gnu!stage2!bin!rustc
[New LWP 2506]
[New LWP 2503]
[New LWP 2507]
[New LWP 2504]
[New LWP 2489]
[New LWP 2508]
warning: Could not load shared library symbols for 7 libraries, e.g. /lib/x86_64-linux-gnu/libc.so.6.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc /checkout/src/tes'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f53f9dbfeaa in llvm::TargetPassConfig::addMachinePasses() () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#0  0x00007f53f9dbfeaa in llvm::TargetPassConfig::addMachinePasses() () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#1  0x00007f53f9bbda0f in llvm::LLVMTargetMachine::addPassesToEmitFile(llvm::legacy::PassManagerBase&, llvm::raw_pwrite_stream&, llvm::raw_pwrite_stream*, llvm::TargetMachine::CodeGenFileType, bool, llvm::MachineModuleInfo*) () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#2  0x00007f53f8ac4354 in LLVMRustWriteOutputFile () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#3  0x00007f53f8978bbc in rustc_codegen_llvm::back::write::write_output_file::h43d626d1bbd593ab () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#4  0x00007f53f896a63d in rustc_codegen_llvm::back::write::codegen::_$u7b$$u7b$closure$u7d$$u7d$::h334d90084152b97a () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#5  0x00007f53f8961c39 in rustc::util::common::time_ext::h1093c0a1462f2cda () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#6  0x00007f53f897b389 in rustc_codegen_llvm::back::write::codegen::h7e2546fe2c85c981 () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#7  0x00007f53f897fbca in rustc_codegen_llvm::back::write::execute_work_item::h34a3780663354f1b () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#8  0x00007f53f893aba2 in std::sys_common::backtrace::__rust_begin_short_backtrace::h95caea62758ed294 () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#9  0x00007f53f8972e49 in std::panicking::try::do_call::hfbe7bba3f27cb8ef () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#10 0x00007f54034da3da in __rust_maybe_catch_panic () at libpanic_unwind/lib.rs:102
#11 0x00007f53f89779f8 in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::hf492e470c0c53396 () from ./checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_llvm-llvm.so
#12 0x00007f54034cad9e in call_once<(),()> () at liballoc/boxed.rs:682
#13 std::sys_common::thread::start_thread::he4cc3b686d083587 () at libstd/sys_common/thread.rs:24
#14 0x00007f54034a0756 in std::sys::unix::thread::Thread::new::thread_start::hff4201b83401a7d4 () at libstd/sys/unix/thread.rs:90
#15 0x00007f53fd50d6ba in ?? ()
#16 0x0000000000000000 in ?? ()
travis_time:end:0faaa9c0:start=1540164656445078288,finish=1540164657544458273,duration=1099379985
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:1eda26f9
travis_time:start:1eda26f9
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:13d3145f
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

gnzlbg · 2018-10-23T14:01:04Z

src/librustc_driver/lib.rs

+// enable this by default on other platforms, so other platforms aren't handled
+// here yet.
+#[cfg(feature = "jemalloc-sys")]
+extern crate jemalloc_sys;


I get that on these platforms global_allocator is not needed, but I still do wonder why this does not use it.

The #[global_allocator] attribute is actually incompatible with this due to the way linkage works and such. I can go into details if necessary, but the answer to your question at least is "it's required by the way we build rustc"

Ah damn, thanks for the explanation.

alexcrichton · 2018-10-24T09:06:35Z

@bors: retry

After many attempts locally I can't reproduce, so I'm curious if we can get another failure on bors...

jonhoo · 2018-11-03T18:03:53Z

Just for clarity, does this change just affect rustc, or does it also affect all crates? Since it closes #36963, does that mean that all crates will now be compiled with the system memory allocator, or are we just moving from alloc_jemalloc to jemalloc-sys?

gnzlbg · 2018-11-03T18:08:58Z

src/librustc_codegen_llvm/back/write.rs

@@ -2464,6 +2504,11 @@ impl OngoingCodegen {
    }
 }

+// impl Drop for OngoingCodegen {


@alexcrichton what's this?

Ah this is a mistake

SimonSapin · 2018-11-03T18:14:08Z

Yes, System is now the default on all platforms for all crate types (such as executable). It’s only rustc that this PR makes link to jemalloc-sys. I’ve written up an announcement:

alexcrichton · 2018-11-03T18:14:59Z

@jonhoo this affects all crates, no crate will be compiled against jemalloc by default now. The compilers we distribute for OSX/Linux will, however, still enable jemalloc (but only the compiler itself)

nnethercote · 2018-11-12T04:56:32Z

Here are the final perf results from the landing. Almost everything improved by at least 1%, and some were up to 15%. Pretty great!

SimonSapin · 2018-11-12T08:56:42Z

This is most likely from upgrading jemalloc from 4.5 to 5.1, isn’t it?

lnicola · 2018-11-12T09:02:11Z

Are there plans to do something about the memory usage regression?

gnzlbg · 2018-11-12T09:31:32Z

@lnicola the jemalloc-sys master branch has re-added support for enabling background threads unconditionally, which might help here - there is also an open issue about gathering statistics of rustc size-class usage which could help here too (https://github.com/alexcrichton/jemallocator/issues/79).

src/jemalloc is gone, remove its mention from COPYRIGHT The `src/jemalloc` submodule was removed in 61e8944 / rust-lang#55238.

Changelog: Version 1.32.0 (2019-01-17) ========================== Language -------- #### 2018 edition - [You can now use the `?` operator in macro definitions.][56245] The `?` operator allows you to specify zero or one repetitions similar to the `*` and `+` operators. - [Module paths with no leading keyword like `super`, `self`, or `crate`, will now always resolve to the item (`enum`, `struct`, etc.) available in the module if present, before resolving to a external crate or an item the prelude.][56759] E.g. ```rust enum Color { Red, Green, Blue } use Color::*; ``` #### All editions - [You can now match against `PhantomData<T>` types.][55837] - [You can now match against literals in macros with the `literal` specifier.][56072] This will match against a literal of any type. E.g. `1`, `'A'`, `"Hello World"` - [Self can now be used as a constructor and pattern for unit and tuple structs.][56365] E.g. ```rust struct Point(i32, i32); impl Point { pub fn new(x: i32, y: i32) -> Self { Self(x, y) } pub fn is_origin(&self) -> bool { match self { Self(0, 0) => true, _ => false, } } } ``` - [Self can also now be used in type definitions.][56366] E.g. ```rust enum List<T> where Self: PartialOrd<Self> // can write `Self` instead of `List<T>` { Nil, Cons(T, Box<Self>) // likewise here } ``` - [You can now mark traits with `#[must_use]`.][55663] This provides a warning if a `impl Trait` or `dyn Trait` is returned and unused in the program. Compiler -------- - [The default allocator has changed from jemalloc to the default allocator on your system.][55238] The compiler itself on Linux & macOS will still use jemalloc, but programs compiled with it will use the system allocator. - [Added the `aarch64-pc-windows-msvc` target.][55702] Libraries --------- - [`PathBuf` now implements `FromStr`.][55148] - [`Box<[T]>` now implements `FromIterator<T>`.][55843] - [The `dbg!` macro has been stabilized.][56395] This macro enables you to easily debug expressions in your rust program. E.g. ```rust let a = 2; let b = dbg!(a * 2) + 1; // ^-- prints: [src/main.rs:4] a * 2 = 4 assert_eq!(b, 5); ``` The following APIs are now `const` functions and can be used in a `const` context. - [`Cell::as_ptr`] - [`UnsafeCell::get`] - [`char::is_ascii`] - [`iter::empty`] - [`ManuallyDrop::new`] - [`ManuallyDrop::into_inner`] - [`RangeInclusive::start`] - [`RangeInclusive::end`] - [`NonNull::as_ptr`] - [`slice::as_ptr`] - [`str::as_ptr`] - [`Duration::as_secs`] - [`Duration::subsec_millis`] - [`Duration::subsec_micros`] - [`Duration::subsec_nanos`] - [`CStr::as_ptr`] - [`Ipv4Addr::is_unspecified`] - [`Ipv6Addr::new`] - [`Ipv6Addr::octets`] Stabilized APIs --------------- - [`i8::to_be_bytes`] - [`i8::to_le_bytes`] - [`i8::to_ne_bytes`] - [`i8::from_be_bytes`] - [`i8::from_le_bytes`] - [`i8::from_ne_bytes`] - [`i16::to_be_bytes`] - [`i16::to_le_bytes`] - [`i16::to_ne_bytes`] - [`i16::from_be_bytes`] - [`i16::from_le_bytes`] - [`i16::from_ne_bytes`] - [`i32::to_be_bytes`] - [`i32::to_le_bytes`] - [`i32::to_ne_bytes`] - [`i32::from_be_bytes`] - [`i32::from_le_bytes`] - [`i32::from_ne_bytes`] - [`i64::to_be_bytes`] - [`i64::to_le_bytes`] - [`i64::to_ne_bytes`] - [`i64::from_be_bytes`] - [`i64::from_le_bytes`] - [`i64::from_ne_bytes`] - [`i128::to_be_bytes`] - [`i128::to_le_bytes`] - [`i128::to_ne_bytes`] - [`i128::from_be_bytes`] - [`i128::from_le_bytes`] - [`i128::from_ne_bytes`] - [`isize::to_be_bytes`] - [`isize::to_le_bytes`] - [`isize::to_ne_bytes`] - [`isize::from_be_bytes`] - [`isize::from_le_bytes`] - [`isize::from_ne_bytes`] - [`u8::to_be_bytes`] - [`u8::to_le_bytes`] - [`u8::to_ne_bytes`] - [`u8::from_be_bytes`] - [`u8::from_le_bytes`] - [`u8::from_ne_bytes`] - [`u16::to_be_bytes`] - [`u16::to_le_bytes`] - [`u16::to_ne_bytes`] - [`u16::from_be_bytes`] - [`u16::from_le_bytes`] - [`u16::from_ne_bytes`] - [`u32::to_be_bytes`] - [`u32::to_le_bytes`] - [`u32::to_ne_bytes`] - [`u32::from_be_bytes`] - [`u32::from_le_bytes`] - [`u32::from_ne_bytes`] - [`u64::to_be_bytes`] - [`u64::to_le_bytes`] - [`u64::to_ne_bytes`] - [`u64::from_be_bytes`] - [`u64::from_le_bytes`] - [`u64::from_ne_bytes`] - [`u128::to_be_bytes`] - [`u128::to_le_bytes`] - [`u128::to_ne_bytes`] - [`u128::from_be_bytes`] - [`u128::from_le_bytes`] - [`u128::from_ne_bytes`] - [`usize::to_be_bytes`] - [`usize::to_le_bytes`] - [`usize::to_ne_bytes`] - [`usize::from_be_bytes`] - [`usize::from_le_bytes`] - [`usize::from_ne_bytes`] Cargo ----- - [You can now run `cargo c` as an alias for `cargo check`.][cargo/6218] - [Usernames are now allowed in alt registry URLs.][cargo/6242] Misc ---- - [`libproc_macro` has been added to the `rust-src` distribution.][55280] Compatibility Notes ------------------- - [The argument types for AVX's `_mm256_stream_si256`, `_mm256_stream_pd`, `_mm256_stream_ps`][55610] have been changed from `*const` to `*mut` as the previous implementation was unsound. [55148]: rust-lang/rust#55148 [55238]: rust-lang/rust#55238 [55280]: rust-lang/rust#55280 [55610]: rust-lang/rust#55610 [55663]: rust-lang/rust#55663 [55702]: https://github.com/rust-lag/rust/pull/55702/ [55837]: rust-lang/rust#55837 [55843]: rust-lang/rust#55843 [56072]: rust-lang/rust#56072 [56245]: rust-lang/rust#56245 [56365]: https:/ttps://github.com/rust-lang/rust/pull/56395/ [56759]: rust-lang/rust#56759 [cargo/6218]: rust-lang/cargo#6218 [cargo/6242]: rust-lang/cargo#6242 [`CStr::as_ptr`]: https://doc.rust-`Duration::as_secs`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.as_secs [`Duration::subsec_micros`]: https://doc.rust-lang.org/std/time/struct.Duration.html#method.subsec_micros [`Duration::subsec_millis`]: https://doc.rust-lang.org/sct.Duration.html#method.subsec_nanos [`Ipv4Addr::is_unspecified`]: https://doc.rust-lang.org/std/net/struct.Ipv4Addr.html#method.is_unspecified [`Ipv6Addr::new`]: https://doc.rust-lang.org/std/net/struct.Ipv6Addr.html#method.new [`Ipv6Addr::octets`]: httpw`]: https://doc.rust-lang.org/std/mem/struct.ManuallyDrop.html#method.new [`NonNull::as_ptr`]: https://doc.rust-lang.org/std/ptr/struct.NonNull.html#method.as_ptr [`RangeInclusive::end`]: https://doc.rust-lang.org/std/ops/struct.RangeInclusive.html#method.end [`RangeInclusive::start`]: https://doc.rust-lang.org/std/ops/struct.RangeInclusive.html#method.start [`UnsafeCell::get`]: https://doc.rust-lang.org/std/cell/struct.UnsafeCell.html#method.get [`slice::as_ptr`]: https://doc.rust-lang.org/std/primitive.slice.html#method.as_ptr [`char::is_ascii`]: https://doc.rust-lang.org/std/primitive.char.html#method.is_ascii [`i128::from_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i128.html#method.from_be_bytes [`i128::from_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i128.html#method.from_le_bytes [`i128::from_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i128.html#method.from_ne_bytes [`i128::to_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i128.html#method.to_be_bytes [`i128::to_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i128.html#method.to_le_bytes [`i128::to_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i128.html#method.to_ne_bytes [`i16::from_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i16.html#method.from_be_bytes [`i16::from_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i16.html#method.from_le_bytes [`i16::from_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i16.html#method.from_ne_bytes [`i16::to_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i16.html#method.to_be_bytes [`i16::to_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i16.html#method.to_le_bytes [`i16::to_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i16.html#method.to_ne_bytes [`i32::from_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i32.html#method.from_be_bytes [`i32::from_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i32.html#method.from_le_bytes [`i32::from_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i32.html#method.from_ne_bytes [`i32::to_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i32.html#method.to_be_bytes [`i32::to_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i32.html#method.to_le_bytes [`i32::to_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i32.html#method.to_ne_bytes [`i64::from_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i64.html#method.from_be_bytes [`i64::from_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i64.html#method.from_le_bytes [`i64::from_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i64.html#method.from_ne_bytes [`i64::to_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i64.html#method.to_be_bytes [`i64::to_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i64.html#method.to_le_bytes [`i64::to_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i64.html#method.to_ne_bytes [`i8::from_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i8.html#method.from_be_bytes [`i8::from_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i8.html#method.from_le_bytes [`i8::from_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i8.html#method.from_ne_bytes [`i8::to_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i8.html#method.to_be_bytes [`i8::to_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i8.html#method.to_le_bytes [`i8::to_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.i8.html#method.to_ne_bytes [`isize::from_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.isize.html#method.from_be_bytes [`isize::from_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.isize.html#method.from_le_bytes [`isize::from_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.isize.html#method.from_ne_bytes [`isize::to_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.isize.html#method.to_be_bytes [`isize::to_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.isize.html#method.to_le_bytes [`isize::to_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.isize.html#method.to_ne_bytes [`iter::empty`]: https://doc.rust-lang.org/std/iter/fn.empty.html [`str::as_ptr`]: https://doc.rust-lang.org/std/primitive.str.html#method.as_ptr [`u128::from_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u128.html#method.from_be_bytes [`u128::from_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u128.html#method.from_le_bytes [`u128::from_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u128.html#method.from_ne_bytes [`u128::to_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u128.html#method.to_be_bytes [`u128::to_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u128.html#method.to_le_bytes [`u128::to_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u128.html#method.to_ne_bytes [`u16::from_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u16.html#method.from_be_bytes [`u16::from_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u16.html#method.from_le_bytes [`u16::from_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u16.html#method.from_ne_bytes [`u16::to_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u16.html#method.to_be_bytes [`u16::to_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u16.html#method.to_le_bytes [`u16::to_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u16.html#method.to_ne_bytes [`u32::from_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u32.html#method.from_be_bytes [`u32::from_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u32.html#method.from_le_bytes [`u32::from_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u32.html#method.from_ne_bytes [`u32::to_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u32.html#method.to_be_bytes [`u32::to_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u32.html#method.to_le_bytes [`u32::to_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u32.html#method.to_ne_bytes [`u64::from_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u64.html#method.from_be_bytes [`u64::from_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u64.html#method.from_le_bytes [`u64::from_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u64.html#method.from_ne_bytes [`u64::to_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u64.html#method.to_be_bytes [`u64::to_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u64.html#method.to_le_bytes [`u64::to_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u64.html#method.to_ne_bytes [`u8::from_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u8.html#method.from_be_bytes [`u8::from_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u8.html#method.from_le_bytes [`u8::from_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u8.html#method.from_ne_bytes [`u8::to_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u8.html#method.to_be_bytes [`u8::to_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u8.html#method.to_le_bytes [`u8::to_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.u8.html#method.to_ne_bytes [`usize::from_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.usize.html#method.from_be_bytes [`usize::from_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.usize.html#method.from_le_bytes [`usize::from_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.usize.html#method.from_ne_bytes [`usize::to_be_bytes`]: https://doc.rust-lang.org/stable/std/primitive.usize.html#method.to_be_bytes [`usize::to_le_bytes`]: https://doc.rust-lang.org/stable/std/primitive.usize.html#method.to_le_bytes [`usize::to_ne_bytes`]: https://doc.rust-lang.org/stable/std/primitive.usize.html#method.to_ne_bytes

Tested with: rustc 1.32.0-nightly (25a42b2ce 2018-11-07) jemalloc was removed from the standard library recently: <rust-lang/rust#55238> As a result, we started seeing cases where the alignment requirements enforced on allocated regions are weaker than before. Some code ceased working because it erroneously relied on jemalloc's larger alignment values. This commit fixes this issue by introducing `AlignedVec`, a wrapper of `Vec`, which provides access to a portion of `Vec` that meets an alignment requirement required by SIMD operations, and by modifying code in question to use `AlignedVec` in place of `Vec`.

jemalloc was removed from the standard library recently: <rust-lang/rust#55238> Removed `extern crate alloc_system;` from `ngsloader` because this is the current default choice. `ngsengine` explicitly registers jemalloc (provided by the `jemallocator` crate) as the global allocator for optimal runtime performance.

switch `jemalloc-sys` back to `tikv-jemalloc-sys`, and update to 0.6.0 Some context: - we used to use jemalloc bindings from https://github.com/gnzlbg/jemallocator, since rust-lang#55238 - that crate was abandoned, picked up as a fork in https://github.com/tikv/jemallocator, so we switched to that in rust-lang#83152. - then they were able to publish to the original `jemalloc-sys` bindings crate, and `jemalloc-sys` and `tikv-jemalloc-sys` became the same thing -- so I switched back to the OG crate in rust-lang#96790 - they're now having publishing problems again: I've been waiting for tikv/jemallocator#96 for the `jemalloc-sys` 0.6.0 update for a few months, but `tikv-jemalloc-sys` is already updated to 0.6.0. A perf run showed some improvements, so this PR switches back to `tikv-jemalloc-sys` to update to 0.6.0. r? ghost

switch `jemalloc-sys` back to `tikv-jemalloc-sys`, and update to 0.6.0 Some context: - we used to use jemalloc bindings from https://github.com/gnzlbg/jemallocator, since rust-lang#55238 - that crate was abandoned, picked up as a fork in https://github.com/tikv/jemallocator, so we switched to that in rust-lang#83152. - then they were able to publish to the original `jemalloc-sys` bindings crate, and `jemalloc-sys` and `tikv-jemalloc-sys` became the same thing -- so I switched back to the OG crate in rust-lang#96790 - they're now having publishing problems again: I've been waiting for tikv/jemallocator#96 for the `jemalloc-sys` 0.6.0 update for a few months, but `tikv-jemalloc-sys` is already updated to 0.6.0. A perf run showed some improvements, so this PR switches back to `tikv-jemalloc-sys` to update to 0.6.0.

rust-highfive assigned estebank Oct 21, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 21, 2018

alexcrichton mentioned this pull request Oct 21, 2018

Switch the default global allocator to System, remove alloc_jemalloc, use jemallocator in rustc #36963

Closed

This comment has been minimized.

Sign in to view

SimonSapin mentioned this pull request Oct 21, 2018

liballoc_system without #[global_allocator] uses jemalloc #45966

Closed

alexcrichton force-pushed the rm-jemalloc branch from 7094bb6 to 94cb105 Compare October 21, 2018 11:04

This comment has been minimized.

Sign in to view

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 21, 2018

This comment has been minimized.

Sign in to view

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 21, 2018

kennytm added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 22, 2018

gnzlbg reviewed Oct 23, 2018

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 24, 2018

alexcrichton added the relnotes Marks issues that should be documented in the release notes of the next release. label Nov 3, 2018

gnzlbg reviewed Nov 3, 2018

View reviewed changes

ehuss mentioned this pull request Nov 4, 2018

Can't find "use-jemalloc" in config.toml. rust-lang/rustc-dev-guide#227

Closed

o01eg mentioned this pull request Nov 6, 2018

Use new rust.jemalloc option gentoo/gentoo-rust#380

Merged

symphorien mentioned this pull request Nov 7, 2018

rustc: build with system llvm and jemalloc NixOS/nixpkgs#49557

Merged

9 tasks

memoryruins mentioned this pull request Nov 28, 2018

Check for Memory Leaks firecracker-microvm/firecracker#119

Closed

pnkfelix mentioned this pull request Dec 3, 2018

rayon_1_0_0::sort perf regression #56283

Closed

This was referenced Dec 14, 2018

stable rustc --version hangs forever #56736

Closed

Why is jemalloc linked to rustc by default *only* via CI? #56812

Closed

ehuss mentioned this pull request Dec 19, 2018

RLS memory usage regression #56980

Closed

SimonSapin mentioned this pull request Jan 2, 2019

src/jemalloc is gone, remove its mention from COPYRIGHT #57267

Merged

ollie27 mentioned this pull request Jan 8, 2019

Vec<u8> clone in rustc 1.33.0 is 3 times slower than rustc 1.29.0 #57437

Closed

Centril added this to the 1.32 milestone Apr 26, 2019

lqd mentioned this pull request Dec 3, 2024

switch jemalloc-sys back to tikv-jemalloc-sys, and update to 0.6.0 #133792

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove the `alloc_jemalloc` crate #55238

Remove the `alloc_jemalloc` crate #55238

alexcrichton commented Oct 21, 2018

rust-highfive commented Oct 21, 2018

alexcrichton commented Oct 21, 2018

This comment has been minimized.

sfackler commented Oct 21, 2018 •

edited

Loading

lnicola commented Oct 21, 2018

This comment has been minimized.

SimonSapin commented Oct 21, 2018

alexcrichton commented Oct 21, 2018

sfackler commented Oct 21, 2018

estebank commented Oct 21, 2018

This comment has been minimized.

This comment has been minimized.

rust-highfive commented Oct 21, 2018

This comment has been minimized.

gnzlbg Oct 23, 2018

alexcrichton Oct 24, 2018

gnzlbg Oct 24, 2018

alexcrichton commented Oct 24, 2018

jonhoo commented Nov 3, 2018

gnzlbg Nov 3, 2018

alexcrichton Nov 3, 2018

SimonSapin commented Nov 3, 2018

alexcrichton commented Nov 3, 2018

nnethercote commented Nov 12, 2018

SimonSapin commented Nov 12, 2018

lnicola commented Nov 12, 2018

gnzlbg commented Nov 12, 2018

Remove the alloc_jemalloc crate #55238

Remove the alloc_jemalloc crate #55238

Conversation

alexcrichton commented Oct 21, 2018

rust-highfive commented Oct 21, 2018

alexcrichton commented Oct 21, 2018

This comment has been minimized.

sfackler commented Oct 21, 2018 • edited Loading

lnicola commented Oct 21, 2018

This comment has been minimized.

SimonSapin commented Oct 21, 2018

alexcrichton commented Oct 21, 2018

sfackler commented Oct 21, 2018

estebank commented Oct 21, 2018

This comment has been minimized.

This comment has been minimized.

rust-highfive commented Oct 21, 2018

This comment has been minimized.

gnzlbg Oct 23, 2018

Choose a reason for hiding this comment

alexcrichton Oct 24, 2018

Choose a reason for hiding this comment

gnzlbg Oct 24, 2018

Choose a reason for hiding this comment

alexcrichton commented Oct 24, 2018

jonhoo commented Nov 3, 2018

gnzlbg Nov 3, 2018

Choose a reason for hiding this comment

alexcrichton Nov 3, 2018

Choose a reason for hiding this comment

SimonSapin commented Nov 3, 2018

alexcrichton commented Nov 3, 2018

nnethercote commented Nov 12, 2018

SimonSapin commented Nov 12, 2018

lnicola commented Nov 12, 2018

gnzlbg commented Nov 12, 2018

Remove the `alloc_jemalloc` crate #55238

Remove the `alloc_jemalloc` crate #55238

sfackler commented Oct 21, 2018 •

edited

Loading