Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reorganize/refactor bootstrap codebase #116196

Merged
merged 6 commits into from
Oct 17, 2023

Conversation

onur-ozkan
Copy link
Member

@onur-ozkan onur-ozkan commented Sep 27, 2023

Currently, bootstrap stores everything on the root path, including very large modules, which makes things very hard to scale and adds too much complexity.

This PR has the following objectives:

  • Improving scalability.
  • Making bootstrap source more understandable for the new contributors(or for everyone).
  • Improving the development experience and making maintenance easier for the bootstrap team.

The new source structure:

.
├── defaults
│   ├── README.md
│   ├── config.codegen.toml
│   ├── config.compiler.toml
│   ├── config.dist.toml
│   ├── config.library.toml
│   └── config.tools.toml
├── mk
│   └── Makefile.in
├── src
│   ├── bin
│   │   ├── main.rs
│   │   ├── rustc.rs
│   │   ├── rustdoc.rs
│   │   └── sccache-plus-cl.rs
│   ├── core
│   │   ├── build_steps
│   │   │   ├── check.rs
│   │   │   ├── clean.rs
│   │   │   ├── compile.rs
│   │   │   ├── dist.rs
│   │   │   ├── doc.rs
│   │   │   ├── format.rs
│   │   │   ├── install.rs
│   │   │   ├── llvm.rs
│   │   │   ├── mod.rs
│   │   │   ├── run.rs
│   │   │   ├── setup.rs
│   │   │   ├── suggest.rs
│   │   │   ├── synthetic_targets.rs
│   │   │   ├── test.rs
│   │   │   ├── tool.rs
│   │   │   └── toolstate.rs
│   │   ├── config
│   │   │   ├── config.rs
│   │   │   ├── flags.rs
│   │   │   └── mod.rs
│   │   ├── builder.rs
│   │   ├── download.rs
│   │   ├── metadata.rs
│   │   ├── mod.rs
│   │   └── sanity.rs
│   ├── tests
│   │   ├── builder.rs
│   │   ├── config.rs
│   │   └── setup.rs
│   ├── utils
│   │   ├── bin_helpers.rs
│   │   ├── cache.rs
│   │   ├── cc_detect.rs
│   │   ├── channel.rs
│   │   ├── dylib.rs
│   │   ├── helpers.rs
│   │   ├── job.rs
│   │   ├── metrics.rs
│   │   ├── mod.rs
│   │   ├── render_tests.rs
│   │   └── tarball.rs
│   └── lib.rs
├── Cargo.lock
├── Cargo.toml
├── README.md
├── bootstrap.py
├── bootstrap_test.py
├── build.rs
├── configure.py
└── download-ci-llvm-stamp

The next step involves:

  • Adding more doc-comments to the bootstrap internals (although we already have a decent amount, there is space for improvement).
  • Breaking large modules into smaller, more manageable modules.
  • Significantly increasing our unit test coverage (which is currently lacking).

This PR should serve as an initial step to make the tasks above much more easier.

r? Mark-Simulacrum

@rustbot
Copy link
Collaborator

rustbot commented Sep 27, 2023

r? @albertlarsan68

(rustbot has picked a reviewer for you, use r? to override)

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Sep 27, 2023
@onur-ozkan onur-ozkan added S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) and removed T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 27, 2023
@onur-ozkan onur-ozkan marked this pull request as draft September 27, 2023 13:12
@onur-ozkan onur-ozkan changed the title [WIP] re-organize/refactor bootstrap codebase [WIP] re-organize/refactor bootstrap Sep 30, 2023
@onur-ozkan onur-ozkan changed the title [WIP] re-organize/refactor bootstrap [WIP] re-organize/refactor bootstrap codebase Sep 30, 2023
@onur-ozkan onur-ozkan force-pushed the reorganize-bootstrap-sources branch 5 times, most recently from fc4fb8c to 3a7cf2c Compare October 7, 2023 20:38
@onur-ozkan onur-ozkan marked this pull request as ready for review October 7, 2023 20:53
@onur-ozkan onur-ozkan changed the title [WIP] re-organize/refactor bootstrap codebase reorganize/refactor bootstrap codebase Oct 7, 2023
@onur-ozkan
Copy link
Member Author

I am open to making improvements on this work if anyone has suggestions.

cc @rust-lang/bootstrap

@onur-ozkan onur-ozkan added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-blocked Status: Marked as blocked ❌ on something else such as an RFC or other implementation work. labels Oct 7, 2023
@bors
Copy link
Contributor

bors commented Oct 8, 2023

☔ The latest upstream changes (presumably #116514) made this pull request unmergeable. Please resolve the merge conflicts.

@bors
Copy link
Contributor

bors commented Oct 12, 2023

☔ The latest upstream changes (presumably #116448) made this pull request unmergeable. Please resolve the merge conflicts.

@rust-log-analyzer

This comment has been minimized.

@bors
Copy link
Contributor

bors commented Oct 14, 2023

☔ The latest upstream changes (presumably #116407) made this pull request unmergeable. Please resolve the merge conflicts.

Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
@onur-ozkan
Copy link
Member Author

@bors r=Mark-Simulacrum

@bors
Copy link
Contributor

bors commented Oct 17, 2023

📌 Commit 3ecff1b has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 17, 2023
@bors
Copy link
Contributor

bors commented Oct 17, 2023

⌛ Testing commit 3ecff1b with merge 347452e...

@bors
Copy link
Contributor

bors commented Oct 17, 2023

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 347452e to master...

1 similar comment
@bors
Copy link
Contributor

bors commented Oct 17, 2023

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 347452e to master...

@bors bors added merged-by-bors This PR was explicitly merged by bors. labels Oct 17, 2023
@bors bors merged commit 347452e into rust-lang:master Oct 17, 2023
12 checks passed
@rustbot rustbot added this to the 1.75.0 milestone Oct 17, 2023
bors-ferrocene bot added a commit to ferrocene/ferrocene that referenced this pull request Oct 17, 2023
54: Pull upstream master 2023 10 17 r=pietroalbini a=Veykril

* rust-lang/rust#116196
* rust-lang/rust#116824
* rust-lang/rust#116822
* rust-lang/rust#116477
* rust-lang/rust#116826
* rust-lang/rust#116820
  * rust-lang/rust#116811
  * rust-lang/rust#116808
  * rust-lang/rust#116805
  * rust-lang/rust#116800
  * rust-lang/rust#116798
  * rust-lang/rust#116754
* rust-lang/rust#114370
* rust-lang/rust#116804
  * rust-lang/rust#116802
  * rust-lang/rust#116790
  * rust-lang/rust#116786
  * rust-lang/rust#116709
  * rust-lang/rust#116430
  * rust-lang/rust#116257
  * rust-lang/rust#114157
* rust-lang/rust#116731
* rust-lang/rust#116550
* rust-lang/rust#114330
* rust-lang/rust#116724
* rust-lang/rust#116782
  * rust-lang/rust#116776
  * rust-lang/rust#115955
  * rust-lang/rust#115196
* rust-lang/rust#116775
* rust-lang/rust#114589
* rust-lang/rust#113747
* rust-lang/rust#116772
  * rust-lang/rust#116771
  * rust-lang/rust#116760
  * rust-lang/rust#116755
  * rust-lang/rust#116732
  * rust-lang/rust#116522
  * rust-lang/rust#116341
  * rust-lang/rust#116172
* rust-lang/rust#110604
* rust-lang/rust#110729
* rust-lang/rust#116527
* rust-lang/rust#116688
* rust-lang/rust#116757
  * rust-lang/rust#116753
  * rust-lang/rust#116748
  * rust-lang/rust#116741
  * rust-lang/rust#116594
* rust-lang/rust#116691
* rust-lang/rust#116643
* rust-lang/rust#116683
* rust-lang/rust#116635
* rust-lang/rust#115515
* rust-lang/rust#116742
  * rust-lang/rust#116661
  * rust-lang/rust#116576
  * rust-lang/rust#116540
* rust-lang/rust#116352
* rust-lang/rust#116737
  * rust-lang/rust#116730
  * rust-lang/rust#116723
  * rust-lang/rust#116715
  * rust-lang/rust#116603
  * rust-lang/rust#116591
  * rust-lang/rust#115439
* rust-lang/rust#116264
* rust-lang/rust#116727
  * rust-lang/rust#116704
  * rust-lang/rust#116696
  * rust-lang/rust#116695
  * rust-lang/rust#116644
  * rust-lang/rust#116630
* rust-lang/rust#116728
  * rust-lang/rust#116689
  * rust-lang/rust#116679
  * rust-lang/rust#116618
  * rust-lang/rust#116577
  * rust-lang/rust#115653
* rust-lang/rust#116702
* rust-lang/rust#116015
* rust-lang/rust#115822
* rust-lang/rust#116407
* rust-lang/rust#115719
* rust-lang/rust#115524
* rust-lang/rust#116705
* rust-lang/rust#116645
* rust-lang/rust#116233
* rust-lang/rust#115108
* rust-lang/rust#116670
* rust-lang/rust#116676
* rust-lang/rust#116666



Co-authored-by: Benoît du Garreau <bdgdlm@outlook.com>
Co-authored-by: Colin Finck <colin@reactos.org>
Co-authored-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Co-authored-by: Joshua Liebow-Feeser <joshlf@users.noreply.github.com>
Co-authored-by: León Orell Valerian Liehr <me@fmease.dev>
Co-authored-by: Trevor Gross <tmgross@umich.edu>
Co-authored-by: Evan Merlock <evan@merlock.dev>
Co-authored-by: joboet <jonasboettiger@icloud.com>
Co-authored-by: Ralf Jung <post@ralfj.de>
Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>
Co-authored-by: Mark Rousskov <mark.simulacrum@gmail.com>
Co-authored-by: onur-ozkan <work@onurozkan.dev>
Co-authored-by: Nicholas Nethercote <n.nethercote@gmail.com>
Co-authored-by: The 8472 <git@infinite-source.de>
Co-authored-by: Samuel Thibault <samuel.thibault@ens-lyon.org>
Co-authored-by: reez12g <reez12g@gmail.com>
Co-authored-by: Jakub Beránek <berykubik@gmail.com>
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (347452e): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
0.8% [0.8%, 0.8%] 1
Regressions ❌
(secondary)
- - 0
Improvements ✅
(primary)
-1.3% [-1.3%, -1.3%] 1
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) -0.2% [-1.3%, 0.8%] 2

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 626.612s -> 625.857s (-0.12%)
Artifact size: 305.59 MiB -> 305.58 MiB (-0.00%)

aDotInTheVoid added a commit to aDotInTheVoid/simpleinfra that referenced this pull request Jul 3, 2024
Closes rust-lang#246

This became even more broken with rust-lang/rust#116196, which renamed `src/bootstrap/lib.rs`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants