rustc: Fail fast when compiling a source file larger than 4 GiB #132791

tyilo · 2024-11-08T21:55:44Z

Currently if you try to compile a file that is larger than 4 GiB, rustc will first read the whole into memory before failing.

If we can read the metadata of the file, we can fail before reading the file.

rustbot · 2024-11-08T21:55:52Z

r? @compiler-errors

rustbot has assigned @compiler-errors.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

compiler-errors · 2024-11-08T23:06:18Z

compiler/rustc_span/src/source_map.rs

@@ -115,6 +115,10 @@ impl FileLoader for RealFileLoader {
    }

    fn read_file(&self, path: &Path) -> io::Result<String> {
+        if path.metadata().is_ok_and(|metadata| metadata.len() > u32::MAX.into()) {


Could we report this as an io::Error::other rather than fataling here?

Yes, this works and produces:

error: couldn't read /home/tyilo/tmp/big_source.rs: rustc does not support files larger than 4 GiB error: aborting due to 1 previous error

beetrees · 2024-11-09T01:02:24Z

compiler/rustc_span/src/source_map.rs

@@ -115,6 +115,10 @@ impl FileLoader for RealFileLoader {
    }

    fn read_file(&self, path: &Path) -> io::Result<String> {
+        if path.metadata().is_ok_and(|metadata| metadata.len() > u32::MAX.into()) {
+            eprintln!("fatal error: rustc does not support files larger than 4GB");


Suggested change

eprintln!("fatal error: rustc does not support files larger than 4GB");

eprintln!("fatal error: rustc does not support text files larger than 4GiB");

rustc supports binary files (via include_bytes!) with no hard-coded upper limit beyond available memory, so the error message should specify only text files are affected. Also I think the right unit here is GiB (1024³): I believe GB is usually 1000³ (although it's sometimes used for 1024³ making GB somewhat ambiguous).

rustc supports binary files (via include_bytes!) with no hard-coded upper limit beyond available memory, so the error message should specify only text files are affected.

Just testing this, it doesn't seem to be true.

Also I think the right unit here is GiB (1024^3): I believe GB is usually 1000^3 (although it's sometimes used for 1024^3 making GB somewhat ambiguous).

Changed.

Just testing this, it doesn't seem to be true.

It is true only if the file is not valid UTF-8 (which is the case for most binary files). Due to this line of code, if the file is valid UTF-8 the full file is loaded into the source map and is subject to the 4 GiB limit.

Ah, ok that's weird. Would it make sense to change that to use try_new_source_file instead and also fallback to a span for the empty string, if the included files is greater than 4 GiB, but valid UTF-8?

Unfortunately that wouldn't fully fix the problem, as it would make the behaviour of rustc inconsistent depending on the order source files are loaded, since the 4 GiB limit is cumulative. If two files, a.txt and b.txt, both exist and are both 3 GiB of valid UTF-8, and one is loaded with include_bytes!("a.txt") and the other with include_str!("b.txt"), then at the moment both compilation will fail regardless of which order they are loaded in as the total loaded source file size exceeds 4 GiB. If load_binary_file is changed to use try_new_source_file and fallback if it fails, then compilation will succeed only if include_bytes!("a.txt") is processed after include_str!("b.txt"): if the include_bytes! is processed first, then try_new_source_file will succeed and fill up 3 GiB of the source map, meaning that the later include_str! will fail as there isn't enough space in the source map remaining.

The ultimate solution for consistent behaviour here (the 4 GiB limit only applying to files loaded as text files) would be for binary files to be tracked for dep-info separately from regular source files, meaning that binary files are not loaded into the source map at all. This would require some extra handling for expand_include_str, however, as that is currently the only user of the span returned by load_binary_file (and would therefore want the file to be loaded into the source map, unlike expand_include_bytes). As this is a pre-existing issue, it's fine to leave this to a future PR.

clubby789 · 2024-11-10T17:08:28Z

cc #132862, which could be fixed by making the check >= u32::MAX and adjusting the error accordingly

compiler-errors

plz also squash commit history

compiler-errors · 2024-11-21T04:12:10Z

compiler/rustc_span/src/source_map.rs

@@ -115,6 +115,12 @@ impl FileLoader for RealFileLoader {
    }

    fn read_file(&self, path: &Path) -> io::Result<String> {
+        if path.metadata().is_ok_and(|metadata| metadata.len() > SourceFile::MAX_FILE_SIZE.into()) {
+            return Err(io::Error::other(format!(
+                "rustc doest not support text files larger than {} bytes",


also the wording is inconsistent between here and the other errors.

compiler-errors · 2024-11-21T04:13:35Z

@rustbot author

Fixes rust-lang#132862

compiler-errors · 2024-11-24T04:40:18Z

@bors r+

bors · 2024-11-24T04:40:21Z

📌 Commit 5caf516 has been approved by compiler-errors

It is now in the queue for this repository.

bors · 2024-11-24T07:37:35Z

⌛ Testing commit 5caf516 with merge ab3cf26...

bors · 2024-11-24T10:20:00Z

☀️ Test successful - checks-actions
Approved by: compiler-errors
Pushing ab3cf26 to master...

rust-timer · 2024-11-24T11:37:54Z

Finished benchmarking commit (ab3cf26): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (primary 2.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.3%	[1.5%, 3.7%]	4
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.3%	[1.5%, 3.7%]	4

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 796.131s -> 798.172s (0.26%)
Artifact size: 336.32 MiB -> 336.30 MiB (-0.01%)

hkBst · 2024-11-28T13:46:02Z

compiler/rustc_span/src/lib.rs

@@ -1843,6 +1843,8 @@ impl StableSourceFileId {
 }

 impl SourceFile {
+    const MAX_FILE_SIZE: u32 = u32::MAX - 1;


why the -1 here?

See #132862

This PR is fail fast for files LARGER than 4 GiB. 4GiB is 2^32B is (u32::MAX + 1) bytes, so U32::MAX is 4Gi - 1, and U32::MAX - 1 is 4Gi - 2. What is the purpose of a constant for 4GiB - 2B when preventing use of files larger than 4GiB? Is that not off by 2?

Originally this PR was for enforcing that files of size 4 GiB and larger would not work, meaning the maximum file size would be 4 GiB - 1 B = u32::MAX bytes.

However, I then discovered #132862, so files of size 4 GiB - 1 B and larger would not work, so the maximum file size is actually 4 GiB - 2 B = u32::MAX - 1. So no the current value is correct.

I've just tested this again to make sure. Using the current stable and nightly version of rustc, I get the following results:

File size (bytes) stable result nightly result

4,294,967,294 success success

4,294,967,295 ICE (#132862) instant failure

4,294,967,296 failure after reading the file to memory instant failure

rustbot assigned compiler-errors Nov 8, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Nov 8, 2024

compiler-errors requested changes Nov 8, 2024

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 8, 2024

beetrees reviewed Nov 9, 2024

View reviewed changes

tyilo requested review from beetrees and compiler-errors November 9, 2024 08:08

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 9, 2024

tgross35 mentioned this pull request Nov 13, 2024

change PR to "waiting on author" after "changes requested" review #1373 rust-lang/triagebot#1857

Closed

compiler-errors reviewed Nov 21, 2024

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 21, 2024

rustc: Fail fast when compiling a source file larger than 4 GiB - 1 B

5caf516

Fixes rust-lang#132862

tyilo force-pushed the big-file-fail-fast branch from c996de3 to 5caf516 Compare November 22, 2024 23:25

tyilo requested a review from compiler-errors November 22, 2024 23:26

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 22, 2024

compiler-errors approved these changes Nov 24, 2024

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 24, 2024

bors added the merged-by-bors This PR was explicitly merged by bors. label Nov 24, 2024

bors merged commit ab3cf26 into rust-lang:master Nov 24, 2024
7 checks passed

rustbot added this to the 1.85.0 milestone Nov 24, 2024

hkBst reviewed Nov 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustc: Fail fast when compiling a source file larger than 4 GiB #132791

rustc: Fail fast when compiling a source file larger than 4 GiB #132791

tyilo commented Nov 8, 2024

rustbot commented Nov 8, 2024

compiler-errors Nov 8, 2024

tyilo Nov 9, 2024

beetrees Nov 9, 2024

tyilo Nov 9, 2024

beetrees Nov 9, 2024

tyilo Nov 10, 2024

beetrees Nov 10, 2024

clubby789 commented Nov 10, 2024

compiler-errors left a comment

compiler-errors Nov 21, 2024

compiler-errors Nov 21, 2024

compiler-errors commented Nov 21, 2024

compiler-errors commented Nov 24, 2024

bors commented Nov 24, 2024

bors commented Nov 24, 2024

bors commented Nov 24, 2024

rust-timer commented Nov 24, 2024

hkBst Nov 28, 2024

tyilo Nov 28, 2024

hkBst Dec 1, 2024

tyilo Dec 1, 2024

	eprintln!("fatal error: rustc does not support files larger than 4GB");
	eprintln!("fatal error: rustc does not support text files larger than 4GiB");

File size (bytes)	stable result	nightly result
4,294,967,294	success	success
4,294,967,295	ICE (#132862)	instant failure
4,294,967,296	failure after reading the file to memory	instant failure

rustc: Fail fast when compiling a source file larger than 4 GiB #132791

rustc: Fail fast when compiling a source file larger than 4 GiB #132791

Conversation

tyilo commented Nov 8, 2024

rustbot commented Nov 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clubby789 commented Nov 10, 2024

compiler-errors left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

compiler-errors commented Nov 21, 2024

compiler-errors commented Nov 24, 2024

bors commented Nov 24, 2024

bors commented Nov 24, 2024

bors commented Nov 24, 2024

rust-timer commented Nov 24, 2024

Overall result: no relevant changes - no action needed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment