Rustc Driver Chapter #76

Michael-F-Bryan · 2018-03-07T14:25:10Z

I've started working on a rustc-driver chapter.

It's still very early days and I've written it as someone with experience instrumenting rustc from the outside so I don't know if it'll flow with the other chapters, so feedback, comments, and criticism are most welcome! 😁

The basic points covered are:

What is it? glue code/the compiler's main() function
What phases does it go through?
At which points in the compilation process can I inspect the compiler's state, and what can I see

I also want to add something like @nrc's https://github.com/nrc/stupid-stats (which is awesome btw, thanks @nrc!) as an appendix for future explorers. If it's fine with him, I might end up copying most of it across and update it to take into account any changes to the compiler since the tutorial was written.

(fixes #74)

nikomatsakis · 2018-03-10T12:10:55Z

Thanks for doing this!

mark-i-m

Thanks! A few minor nits...

mark-i-m · 2018-03-08T01:02:41Z

src/rustc-driver.md

+stuff). The `rustc_driver` crate also provides external users with a method
+for running code at particular times during the compilation process, allowing
+third parties to effectively use `rustc`'s internals as a library for
+analysing a crate.


It should probably be noted somewhere that doing so is completely unstable.

But also that compiler devs should avoid making breaking changes where possible, since it is an API.

"analysing a crate" - or emulating the compiler - that is if you want to compile some code and do so in-process, then you use the driver API (e.g., the RLS)

It should probably be noted somewhere that doing so is completely unstable.

That's a good point. I'll add a big warning saying we're using compiler internals and, similar to nightly, the internal APIs are always going to be unstable.

mark-i-m · 2018-03-08T01:04:23Z

src/appendix-code-index.md

 `ParseSess` | struct | This struct contains information about a parsing session | [The parser](the-parser.html) | [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs)
+`Session` | struct | The data associated with a compilation session | [the Parser](the-parser.html), [The Rustc Driver](rustc-driver.html) | [src/librustc/session/mod.html](https://github.com/rust-lang/rust/blob/master/src/librustc/session/mod.rs)


Nit: s/the Parser/The parser/

(for consistency elsewhere)

I think that's actually a typo from when I started the original The parser chapter. It's a title, so both words should be capitalized.

mark-i-m · 2018-03-08T01:06:33Z

src/rustc-driver.md

+2. *Configure and Expand:* Resolve `#[cfg]` attributes and expand macros
+3. *Run Analysis Passes:* Run the resolution, typechecking, region checking
+   and other miscellaneous analysis passes on the crate
+4. *Translate to LLVM:* Turn the analysed program into executable code


Hmm... Should it be

Turn the analysed program into LLVM IR

Run LLVM

or are these actually 1 step in the code?

I'd imagine they are two discrete steps, but from rustc-driver's perspective they all happen at the same time. From what I can tell there are no callbacks between the after_analysis phase and the end of compilation, so for all intents and purposes they both happen in the same "LLVM codegen" phase.

I should mention that explicitly though.

Yeah, it is one phase - we never actually generate LLVM IR (unless you specify that as the output) we use LLVM as a library where the IR is implicit (at least the last time I hacked on the back end we did).

mark-i-m · 2018-03-08T01:11:40Z

src/appendix-stupid-stats.md

@@ -0,0 +1,396 @@
+# Appendix A: A tutorial on creating a drop-in replacement for rustc
+


I haven't read @nrc's post, but if this is a copy of it, then we should probably add a citation with a link to the original.

Good point! It's just a copy-paste at the moment so we should mention @nrc is the original author. It also means I'll be able to find the link to the original document when I want to refer to it later on.

mark-i-m · 2018-03-10T19:06:00Z

src/appendix-stupid-stats.md

+                     s: &Session,
+                     i: &Input,
+                     odir: &Option<Path>,
+                     ofile: &Option<Path>)


I think this function needs to be updated to work on the most recent compiler:

fn late_callback( &mut self, + t: &TransCrate, m: &getopts::Matches, s: &Session, + c: &CrateStore, i: &Input, odir: &Option<PathBuf>, ofile: &Option<PathBuf>, ) -> Compilation { + self.default_calls.late_callback(t, m, s, c, i, odir, ofile); Compilation::Continue }

Made a PR nrc/stupid-stats#8

mark-i-m · 2018-03-10T19:07:04Z

Also, it looks like a rebase is needed?

nrc · 2018-03-11T21:57:59Z

If it's fine with him

It is fine with me :-) You should check it still compiles and work - the driver is unstable, so it may have changed since I last checked stupid-stats.

nrc · 2018-03-11T22:00:11Z

src/appendix-code-index.md

@@ -7,6 +7,10 @@ compiler.
 Item            |  Kind    | Short description           | Chapter            | Declaration
 ----------------|----------|-----------------------------|--------------------|-------------------
 `CodeMap` | struct | The CodeMap maps the AST nodes to their source code | [The parser](the-parser.html) | [src/libsyntax/codemap.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/codemap.rs)
+`CompileState` | struct | State that is passed to a callback at each compiler pass | [The Rustc Driver](rustc-driver.html) | [src/librustc_driver/driver.rs](https://github.com/rust-lang/rust/blob/master/src/librustc_driver/driver.rs)
+`ast::Crate` | struct | Syntax-level representation of a parsed crate | | [src/librustc/hir/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/ast.rs)
+`hir::Crate` | struct | Top-level data structure representing the crate being compiled | | [src/librustc/hir/mod.rs](https://github.com/rust-lang/rust/blob/master/src/librustc/hir/mod.rs)


"Top-level" isn't very helpful here - the HIR is basically a compiler-internal version of the AST, i.e., tools and other users should not use it (but can use the AST). It does not match the source text as closely as the AST, being more designed for compiler use, but still fits the 'textbook' definition of an AST.

Would it make sense to say "more compiler friendly" form of the AST? I believe the HIR is a desugared, more useful form of AST where we've broken things out into their categories (items, trait impls, etc).

nrc · 2018-03-11T22:01:26Z

src/rustc-driver.md

+stuff). The `rustc_driver` crate also provides external users with a method
+for running code at particular times during the compilation process, allowing
+third parties to effectively use `rustc`'s internals as a library for
+analysing a crate.


But also that compiler devs should avoid making breaking changes where possible, since it is an API.

nrc · 2018-03-11T22:02:22Z

src/rustc-driver.md

+stuff). The `rustc_driver` crate also provides external users with a method
+for running code at particular times during the compilation process, allowing
+third parties to effectively use `rustc`'s internals as a library for
+analysing a crate.


"analysing a crate" - or emulating the compiler - that is if you want to compile some code and do so in-process, then you use the driver API (e.g., the RLS)

nrc · 2018-03-11T22:02:50Z

src/rustc-driver.md

+From `rustc_driver`'s perspective, the main phases of the compiler are:
+
+1. *Parse Input:* Initial crate parsing
+2. *Configure and Expand:* Resolve `#[cfg]` attributes and expand macros


Name resolution is part of this phase

Ooh, I didn't know that! I though name resolution happened some time after the after_hir_lowering and before after_analysis gets called.

My dyna-bindgen tool was doing analysis at the after_hir_lowering stage and I couldn't figure out how to get the fully qualified name of a function arguments type. I thought I needed to wait until later and use the TyCtxt to make queries.

nrc · 2018-03-11T22:03:36Z

src/rustc-driver.md

+
+1. *Parse Input:* Initial crate parsing
+2. *Configure and Expand:* Resolve `#[cfg]` attributes and expand macros
+3. *Run Analysis Passes:* Run the resolution, typechecking, region checking


Not sure what 'resolution' means here, but name resolution now happens earlier. I think 'borrow-checking' is more common than 'region-checking'

I think trait resolution happens here though, I think...

yes, correct

Haha, I'm not sure either! The rustdocs just say "resolution" so I thought I'd fudge it 😛

nrc · 2018-03-11T22:05:04Z

src/rustc-driver.md

+2. *Configure and Expand:* Resolve `#[cfg]` attributes and expand macros
+3. *Run Analysis Passes:* Run the resolution, typechecking, region checking
+   and other miscellaneous analysis passes on the crate
+4. *Translate to LLVM:* Turn the analysed program into executable code


Yeah, it is one phase - we never actually generate LLVM IR (unless you specify that as the output) we use LLVM as a library where the IR is implicit (at least the last time I hacked on the back end we did).

mark-i-m · 2018-03-12T01:41:41Z

@nrc

It is fine with me :-) You should check it still compiles and work - the driver is unstable, so it may have changed since I last checked stupid-stats.

Thanks! I made a PR to make it compile and run on a recent compiler: nrc/stupid-stats#8

@Michael-F-Bryan Could you update the contents of the PR accordingly please :)

cc: nrc/stupid-stats#8

Michael-F-Bryan · 2018-03-12T11:18:26Z

@mark-i-m I've rebased and updated the stupid stats appendix. Would you be able to have another look?

If you know anyone who's hacked on the driver recently it may be useful to CC them in. There's probably a lot of stuff I've missed.

mark-i-m · 2018-03-12T22:40:54Z

@Michael-F-Bryan Thanks!

I personally don't know of anyone else to CC... The git blame of librustc_driver/lib.rs seems to show quite a lot of people :)

I think we can probably merge this and let others raise issues later for additions.

mark-i-m mentioned this pull request Mar 8, 2018

Code index #50

Closed

15 tasks

mark-i-m suggested changes Mar 10, 2018

View reviewed changes

nrc reviewed Mar 11, 2018

View reviewed changes

Michael-F-Bryan added 5 commits March 12, 2018 18:54

Added a very rough rustc-driver chapter

83d5f64

Renamed appendices and added @nrc's guide

3bb1b1a

Thank you link checker!

1d80615

Added a couple definitions to the code index

42e4eee

Addressed some of @nrc and @mark-i-m's comments

e3d0056

Michael-F-Bryan force-pushed the rustc-driver branch from a89f2ba to e3d0056 Compare March 12, 2018 11:01

Michael-F-Bryan added 3 commits March 12, 2018 19:04

Updated stupid-stats

aecac1c

cc: nrc/stupid-stats#8

Fixed a broken link

5686d8c

Added links back to nrc's stupid-stats

e6946bc

mark-i-m approved these changes Mar 12, 2018

View reviewed changes

mark-i-m merged commit 9874533 into rust-lang:master Mar 12, 2018

Michael-F-Bryan deleted the rustc-driver branch March 13, 2018 00:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rustc Driver Chapter #76

Rustc Driver Chapter #76

Michael-F-Bryan commented Mar 7, 2018

nikomatsakis commented Mar 10, 2018

mark-i-m left a comment

mark-i-m Mar 8, 2018

nrc Mar 11, 2018

nrc Mar 11, 2018

Michael-F-Bryan Mar 12, 2018

mark-i-m Mar 8, 2018

Michael-F-Bryan Mar 12, 2018

mark-i-m Mar 8, 2018

Michael-F-Bryan Mar 10, 2018

nrc Mar 11, 2018

mark-i-m Mar 8, 2018

Michael-F-Bryan Mar 10, 2018 •

edited

Loading

mark-i-m Mar 10, 2018

mark-i-m Mar 12, 2018

mark-i-m commented Mar 10, 2018

nrc commented Mar 11, 2018

nrc Mar 11, 2018

Michael-F-Bryan Mar 12, 2018

nrc Mar 11, 2018

nrc Mar 11, 2018

nrc Mar 11, 2018

Michael-F-Bryan Mar 12, 2018

nrc Mar 11, 2018

mark-i-m Mar 12, 2018

nrc Mar 12, 2018

Michael-F-Bryan Mar 12, 2018

nrc Mar 11, 2018

mark-i-m commented Mar 12, 2018

Michael-F-Bryan commented Mar 12, 2018

mark-i-m commented Mar 12, 2018

		`ParseSess` \| struct \| This struct contains information about a parsing session \| [The parser](the-parser.html) \| [src/libsyntax/parse/mod.rs](https://github.com/rust-lang/rust/blob/master/src/libsyntax/parse/mod.rs)
		`Session` \| struct \| The data associated with a compilation session \| [the Parser](the-parser.html), [The Rustc Driver](rustc-driver.html) \| [src/librustc/session/mod.html](https://github.com/rust-lang/rust/blob/master/src/librustc/session/mod.rs)

		@@ -0,0 +1,396 @@
		# Appendix A: A tutorial on creating a drop-in replacement for rustc

Rustc Driver Chapter #76

Rustc Driver Chapter #76

Conversation

Michael-F-Bryan commented Mar 7, 2018

nikomatsakis commented Mar 10, 2018

mark-i-m left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Michael-F-Bryan Mar 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mark-i-m commented Mar 10, 2018

nrc commented Mar 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mark-i-m commented Mar 12, 2018

Michael-F-Bryan commented Mar 12, 2018

mark-i-m commented Mar 12, 2018

Michael-F-Bryan Mar 10, 2018 •

edited

Loading