Skip to content

Bump FileEncoder buffer size to 64 kB #139640

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 13, 2025
Merged

Conversation

Zoxc
Copy link
Contributor

@Zoxc Zoxc commented Apr 10, 2025

This helps avoid file system overhead on Windows. The improvement are probably reduced a bit on other platforms. Making the buffer size even larger does further improve performance, but that increase memory use further.

BenchmarkBeforeAfterBeforeAfterBeforeAfter
TimeTime%Physical MemoryPhysical Memory%Committed MemoryCommitted Memory%
🟣 clap:check:unchanged0.3336s0.3242s💚 -2.82%96.97 MiB97.12 MiB 0.16%167.87 MiB167.97 MiB 0.06%
🟣 hyper:check:unchanged0.1331s0.1307s💚 -1.79%61.92 MiB62.05 MiB 0.20%124.66 MiB124.74 MiB 0.06%
🟣 regex:check:unchanged0.2485s0.2399s💚 -3.45%78.32 MiB78.55 MiB 0.29%145.22 MiB145.45 MiB 0.15%
🟣 syn:check:unchanged0.5321s0.5175s💚 -2.76%118.58 MiB118.77 MiB 0.16%192.99 MiB193.17 MiB 0.09%
Total1.2474s1.2123s💚 -2.81%355.78 MiB356.49 MiB 0.20%630.74 MiB631.32 MiB 0.09%
Summary1.0000s0.9729s💚 -2.71%1 byte1.00 bytes 0.20%1 byte1.00 bytes 0.09%

@rustbot
Copy link
Collaborator

rustbot commented Apr 10, 2025

r? @fee1-dead

rustbot has assigned @fee1-dead.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Apr 10, 2025
@saethlin
Copy link
Member

r? saethlin

@bors try @rust-timer queue

@rust-timer

This comment has been minimized.

@rustbot rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 10, 2025
@saethlin
Copy link
Member

r=me unless perf indicates this is a regression

bors added a commit to rust-lang-ci/rust that referenced this pull request Apr 10, 2025
Bump `FileEncoder` buffer size to 64 kB

This helps avoid file system overhead on Windows. The improvement are probably reduced a bit on other platforms. Making the buffer size even larger does further improve performance, but that increase memory use further.

<table><tr><td rowspan="2">Benchmark</td><td colspan="1"><b>Before</b></th><td colspan="2"><b>After</b></th><td colspan="1"><b>Before</b></th><td colspan="2"><b>After</b></th><td colspan="1"><b>Before</b></th><td colspan="2"><b>After</b></th></tr><tr><td align="right">Time</td><td align="right">Time</td><td align="right">%</th><td align="right">Physical Memory</td><td align="right">Physical Memory</td><td align="right">%</th><td align="right">Committed Memory</td><td align="right">Committed Memory</td><td align="right">%</th></tr><tr><td>🟣 <b>clap</b>:check:unchanged</td><td align="right">0.3336s</td><td align="right">0.3242s</td><td align="right">💚  -2.82%</td><td align="right">96.97 MiB</td><td align="right">97.12 MiB</td><td align="right"> 0.16%</td><td align="right">167.87 MiB</td><td align="right">167.97 MiB</td><td align="right"> 0.06%</td></tr><tr><td>🟣 <b>hyper</b>:check:unchanged</td><td align="right">0.1331s</td><td align="right">0.1307s</td><td align="right">💚  -1.79%</td><td align="right">61.92 MiB</td><td align="right">62.05 MiB</td><td align="right"> 0.20%</td><td align="right">124.66 MiB</td><td align="right">124.74 MiB</td><td align="right"> 0.06%</td></tr><tr><td>🟣 <b>regex</b>:check:unchanged</td><td align="right">0.2485s</td><td align="right">0.2399s</td><td align="right">💚  -3.45%</td><td align="right">78.32 MiB</td><td align="right">78.55 MiB</td><td align="right"> 0.29%</td><td align="right">145.22 MiB</td><td align="right">145.45 MiB</td><td align="right"> 0.15%</td></tr><tr><td>🟣 <b>syn</b>:check:unchanged</td><td align="right">0.5321s</td><td align="right">0.5175s</td><td align="right">💚  -2.76%</td><td align="right">118.58 MiB</td><td align="right">118.77 MiB</td><td align="right"> 0.16%</td><td align="right">192.99 MiB</td><td align="right">193.17 MiB</td><td align="right"> 0.09%</td></tr><tr><td>Total</td><td align="right">1.2474s</td><td align="right">1.2123s</td><td align="right">💚  -2.81%</td><td align="right">355.78 MiB</td><td align="right">356.49 MiB</td><td align="right"> 0.20%</td><td align="right">630.74 MiB</td><td align="right">631.32 MiB</td><td align="right"> 0.09%</td></tr><tr><td>Summary</td><td align="right">1.0000s</td><td align="right">0.9729s</td><td align="right">💚  -2.71%</td><td align="right">1 byte</td><td align="right">1.00 bytes</td><td align="right"> 0.20%</td><td align="right">1 byte</td><td align="right">1.00 bytes</td><td align="right"> 0.09%</td></tr></table>
@bors
Copy link
Collaborator

bors commented Apr 10, 2025

⌛ Trying commit 74ca129 with merge d6e561e...

@bors
Copy link
Collaborator

bors commented Apr 11, 2025

☀️ Try build successful - checks-actions
Build commit: d6e561e (d6e561e8c338f90cca31c4bb2b674e611584cbdb)

@rust-timer

This comment has been minimized.

@rust-timer
Copy link
Collaborator

Finished benchmarking commit (d6e561e): comparison URL.

Overall result: ❌✅ regressions and improvements - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
0.3% [0.3%, 0.3%] 2
Improvements ✅
(primary)
-0.3% [-0.3%, -0.3%] 1
Improvements ✅
(secondary)
-0.3% [-0.3%, -0.3%] 1
All ❌✅ (primary) -0.3% [-0.3%, -0.3%] 1

Max RSS (memory usage)

Results (primary -2.8%, secondary 2.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
3.9% [1.7%, 5.4%] 8
Improvements ✅
(primary)
-2.8% [-2.8%, -2.8%] 1
Improvements ✅
(secondary)
-2.4% [-2.7%, -2.2%] 3
All ❌✅ (primary) -2.8% [-2.8%, -2.8%] 1

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 783.243s -> 783.248s (0.00%)
Artifact size: 366.21 MiB -> 366.22 MiB (0.00%)

@rustbot rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 11, 2025
@saethlin
Copy link
Member

@bors r+

@bors
Copy link
Collaborator

bors commented Apr 11, 2025

📌 Commit 74ca129 has been approved by saethlin

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 11, 2025
@Zoxc
Copy link
Contributor Author

Zoxc commented Apr 11, 2025

Maybe MaybeUninit would help with helloworld?

@saethlin
Copy link
Member

I don't see a helloworld regression. What are you referring to?

@Zoxc
Copy link
Contributor Author

Zoxc commented Apr 11, 2025

There's a suspicious grouping of helloworld instruction regressions, but I don't think FileEncoder is used in the non-incremental builds, so that might be noise.

@saethlin
Copy link
Member

Those are very far below the significance threshold. They could as easily be PGO instability, and I don't think it's worth all the time to investigate such tiny changes.

@ChrisDenton
Copy link
Member

Is the maximum number of buffers tied to the number of threads or the number of files?

@Zoxc
Copy link
Contributor Author

Zoxc commented Apr 11, 2025

The number of files (up to 4), though I intend to add per thread buffers for one of those files.

@bors
Copy link
Collaborator

bors commented Apr 12, 2025

⌛ Testing commit 74ca129 with merge ae06b79...

@bors
Copy link
Collaborator

bors commented Apr 13, 2025

☀️ Test successful - checks-actions
Approved by: saethlin
Pushing ae06b79 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 13, 2025
@bors bors merged commit ae06b79 into rust-lang:master Apr 13, 2025
7 checks passed
@rustbot rustbot added this to the 1.88.0 milestone Apr 13, 2025
Copy link

What is this? This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 9ffde4b (parent) -> ae06b79 (this PR)

Test differences

Show 33004 test diffs

Stage 1

  • errors::verify_builtin_macros_format_positional_after_named_60: [missing] -> pass (J0)
  • errors::verify_const_eval_non_const_intrinsic_19: [missing] -> pass (J0)
  • errors::verify_metadata_unsupported_abi_i686_38: [missing] -> pass (J0)
  • errors::verify_parse_invalid_attr_unsafe_162: [missing] -> pass (J0)
  • errors::verify_parse_modifier_lifetime_136: [missing] -> pass (J0)
  • errors::verify_passes_repr_align_greater_than_target_max_74: [missing] -> pass (J0)
  • errors::verify_passes_rustc_layout_scalar_valid_range_arg_64: [missing] -> pass (J0)
  • lints::verify_lint_map_unit_fn_52: [missing] -> pass (J0)
  • owned_slice::tests::boxed: [missing] -> pass (J0)
  • parser::tests::minimum_depth: [missing] -> pass (J0)
  • spec::tests::x86_64_pc_solaris: [missing] -> pass (J0)
  • fs::tests::copy_file_does_not_exist: [missing] -> pass (J1)
  • hash::sip::bench_long_str: [missing] -> pass (J1)
  • iter::bench_peekable_chain_sum: [missing] -> pass (J1)
  • os::unix::net::tests::long_path: [missing] -> pass (J1)
  • sort::tests::unstable::stability_cell_i32_random_d2: [missing] -> pass (J1)
  • str::contains_bang_str::long_lorem_ipsum: [missing] -> pass (J1)
  • tests::exact_filter_match: [missing] -> pass (J1)
  • alloc::layout_round_up_to_align_edge_cases: [missing] -> pass (J2)
  • collections::hash::map::tests::const_with_hasher: [missing] -> pass (J2)
  • collections::vec_deque::tests::make_contiguous_big_head: [missing] -> pass (J2)
  • fmt::test_once: [missing] -> pass (J2)
  • iter::adapters::flatten::test_flatten_one_shot_arrays: [missing] -> pass (J2)
  • iter::adapters::step_by::test_iterator_step_by_nth_try_fold: [missing] -> pass (J2)
  • lazy::partialeq_impl: [missing] -> pass (J2)
  • nonzero::test_match_option_string: [missing] -> pass (J2)
  • num::dec2flt::parse::many_digits: [missing] -> pass (J2)
  • num::test_try_i32i16: [missing] -> pass (J2)
  • num::test_u16f32: [missing] -> pass (J2)
  • rc::test_from_owned: [missing] -> pass (J2)
  • slice::test_chunks_nth: [missing] -> pass (J2)
  • slice::test_slice_from_ptr_range: [missing] -> pass (J2)
  • slice::test_swap_remove: [missing] -> pass (J2)
  • str::contains_weird_cases: [missing] -> pass (J2)
  • str::test_contains: [missing] -> pass (J2)
  • string::test_replace_range_unbounded: [missing] -> pass (J2)
  • thin_box::align64small: [missing] -> pass (J2)
  • time::mul: [missing] -> pass (J2)
  • vec::test_splice_inclusive_out_of_bounds: [missing] -> pass (J2)
  • sort::tests::stable::correct_string_random: [missing] -> ignore (J3)
  • sort::tests::stable::panic_observable_is_less_saw_mixed: [missing] -> ignore (J3)
  • sort::tests::unstable::panic_retain_orig_set_cell_i32_saw_mixed: [missing] -> ignore (J3)
  • switch_stdout: [missing] -> ignore (J3)
  • f16::test_round: [missing] -> pass (J4)

Stage 2

  • analyze_source_file::tests::non_narrow_long: pass -> [missing] (J0)
  • coverage::counters::union_find::tests::transitive: pass -> [missing] (J0)
  • errors::verify_builtin_macros_cfg_accessible_has_args_15: pass -> [missing] (J0)
  • errors::verify_codegen_ssa_extract_bundled_libs_write_file_74: pass -> [missing] (J0)
  • errors::verify_mir_build_literal_in_range_out_of_bounds_6: pass -> [missing] (J0)
  • errors::verify_mir_build_nan_pattern_21: pass -> [missing] (J0)
  • errors::verify_parse_expect_label_found_ident_138: pass -> [missing] (J0)
  • errors::verify_parse_unexpected_token_after_struct_name_found_keyword_81: pass -> [missing] (J0)
  • errors::verify_passes_should_be_applied_to_struct_enum_15: pass -> [missing] (J0)
  • errors::verify_session_unsupported_crate_type_for_target_42: pass -> [missing] (J0)
  • html::tests::href_relative_parts_child_module: pass -> [missing] (J0)
  • html::url_parts_builder::tests::push_front_empty: pass -> [missing] (J0)
  • lints::verify_lint_invalid_asm_label_format_arg_102: pass -> [missing] (J0)
  • lints::verify_lint_invalid_null_arguments_46: pass -> [missing] (J0)
  • spec::tests::x86_64_pc_cygwin: pass -> [missing] (J0)
  • ascii::long::case08_fake_simd_u64: pass -> [missing] (J1)
  • btree::set::clone_100_and_drain_all: pass -> [missing] (J1)
  • btree::set::clone_100_and_remove_all: pass -> [missing] (J1)
  • hash::sip::bench_bytes_4: pass -> [missing] (J1)
  • slice::brute_force_rotate_test_0: pass -> [missing] (J1)
  • sort::tests::stable::self_cmp_string_ascending: pass -> [missing] (J1)
  • sort::tests::unstable::correct_u64_random: pass -> [missing] (J1)
  • sort::tests::unstable::violate_ord_retain_orig_set_string_random_s95: pass -> [missing] (J1)
  • stats::tests::test_norm10medium: pass -> [missing] (J1)
  • tests::run_include_ignored_option: pass -> [missing] (J1)
  • vec::bench_next_chunk: pass -> [missing] (J1)
  • collections::btree::map::tests::test_into_iter_drop_leak_height_1: pass -> [missing] (J2)
  • collections::hash::map::tests::test_index_nonexistent: pass -> [missing] (J2)
  • f128::test_num_f128: pass -> [missing] (J2)
  • f64::test_nan: pass -> [missing] (J2)
  • iter::adapters::array_chunks::test_iterator_array_chunks_infer: pass -> [missing] (J2)
  • iter::adapters::step_by::test_iterator_step_by_nth_try_rfold: pass -> [missing] (J2)
  • iter::adapters::take::test_iterator_take: pass -> [missing] (J2)
  • net::ip_addr::ipv6_from_octets: pass -> [missing] (J2)
  • net::socket_addr::is_v6: pass -> [missing] (J2)
  • num::i8::test_signum: pass -> [missing] (J2)
  • num::test_try_i64i16: pass -> [missing] (J2)
  • num::test_try_isizeu8: pass -> [missing] (J2)
  • num::u128::test_parse_bytes: pass -> [missing] (J2)
  • num::wrapping::test_wrapping_u64: pass -> [missing] (J2)
  • rc::test_unique_rc_drops_contents: pass -> [missing] (J2)
  • reentrant_lock::trylock_works: pass -> [missing] (J2)
  • single_line_formatting: pass -> [missing] (J2)
  • slice::memchr::no_match_empty: pass -> [missing] (J2)
  • slice::test_chunks_exact_iterator_0: pass -> [missing] (J2)
  • slice::test_copy_from_slice_dst_shorter: pass -> [missing] (J2)
  • slice::test_from_fn: pass -> [missing] (J2)
  • slice::test_len_divzero: pass -> [missing] (J2)
  • slice::test_rchunks_count: pass -> [missing] (J2)
  • sort::tests::stable::correct_i32_random_s95: pass -> [missing] (J2)
  • thin_box::align2big: pass -> [missing] (J2)
  • vec::test_drain_range: pass -> [missing] (J2)
  • sort::tests::stable::deterministic_i32_descending: ignore -> [missing] (J3)
  • sort::tests::unstable::deterministic_string_descending: ignore -> [missing] (J3)
  • sort::tests::unstable::self_cmp_cell_i32_random: ignore -> [missing] (J3)
  • sort::tests::unstable::self_cmp_i32_random: ignore -> [missing] (J3)

(and 16404 additional test diffs)

Additionally, 16500 doctest diffs were found. These are ignored, as they are noisy.

Job group index

Job duration changes

  1. x86_64-apple-2: 1271.5s -> 4189.9s (229.5%)
  2. test-various: 2272.4s -> 4162.4s (83.2%)
  3. x86_64-gnu-aux: 4289.5s -> 6079.2s (41.7%)
  4. aarch64-apple: 2877.3s -> 3553.3s (23.5%)
  5. dist-x86_64-apple: 10760.2s -> 8345.0s (-22.4%)
  6. x86_64-apple-1: 7095.9s -> 7841.3s (10.5%)
  7. dist-apple-various: 7312.4s -> 6670.8s (-8.8%)
  8. aarch64-gnu-debug: 4310.8s -> 4079.7s (-5.4%)
  9. i686-gnu-nopt-1: 8587.7s -> 8167.8s (-4.9%)
  10. dist-aarch64-linux: 5233.4s -> 5488.1s (4.9%)
How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

@Zoxc Zoxc deleted the big-file-encoder branch April 13, 2025 00:16
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (ae06b79): comparison URL.

Overall result: ❌✅ regressions and improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
0.4% [0.4%, 0.4%] 1
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
-0.3% [-0.3%, -0.3%] 1
All ❌✅ (primary) - - 0

Max RSS (memory usage)

Results (primary -1.0%, secondary 1.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
3.5% [2.2%, 5.6%] 7
Improvements ✅
(primary)
-1.0% [-1.0%, -1.0%] 1
Improvements ✅
(secondary)
-4.0% [-8.4%, -1.3%] 3
All ❌✅ (primary) -1.0% [-1.0%, -1.0%] 1

Cycles

Results (secondary -2.6%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
- - 0
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
-2.6% [-2.6%, -2.6%] 1
All ❌✅ (primary) - - 0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 777.191s -> 777.2s (0.00%)
Artifact size: 365.50 MiB -> 365.49 MiB (-0.00%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants