Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable reuse of .o files if nothing has changed #34956

Merged
merged 9 commits into from
Jul 29, 2016

Conversation

nikomatsakis
Copy link
Contributor

@nikomatsakis nikomatsakis commented Jul 21, 2016

This PR completes a first "spike" for incremental compilation by enabling us to reuse .o files when nothing has changed. When in incr. mode, we will save .o files into the temporary directory, then copy them back out again if they are still valid. The code is still a bit rough but it does seem to work. =)

r? @michaelwoerister

Fixes #34036
Fixes #34037
Fixes #34038

@michaelwoerister
Copy link
Member

Excellent :) I'll take a look at it first thing tomorrow.

@jonas-schievink
Copy link
Contributor

I just tried to build glium with this and I'm hitting multiple ICEs about local/foreign items:

https://gist.github.com/jonas-schievink/536a1ad7aa065193e4954242c9885cbd

(I've set RUSTFLAGS=-Zincremental=target/INCREMENTAL before running cargo)

/// them even in the absence of a tcx.)
#[derive(Clone, Debug, PartialEq, Eq, Hash, RustcEncodable, RustcDecodable)]
pub enum WorkProductId {
PartitionObjectFile(String), // see (*TransPartition) below
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Our nomenclature around "partitions", "codegen units", "translation units", etc is a bit inconsistent. We are using these names more or less interchangeably and it would probably be good to decide on the canonical term.

@michaelwoerister
Copy link
Member

OK, I've done a first pass over this. It's so nice to see all the work from the last 10 months finally coming together! (I mean, not that we are anywhere near being done yet, of course :))

The PR looks very good. The only thing I'm not so sure about is the factoring of the WorkProduct functionality. It seems a bit strange to have the partitioning-hashes in there, something which will only be used for object files (and maybe .bc files), but not for something like MIR. At the same time the WorkProductId does not scale to things with a DefId. It seems to me that there is room for improvement. However, I won't block the PR on this.

Another thing that isn't quite clear to me is the relationship between the "temp" directory and the "incr. comp.". Are they the same when incr. comp. is enabled? Or do we copy back and forth between them? I guess, we'll clarify these questions when we tackle #34957, #34955, and #32754.

@nikomatsakis
Copy link
Contributor Author

The only thing I'm not so sure about is the factoring of the WorkProduct functionality. It seems a bit strange to have the partitioning-hashes in there, something which will only be used for object files (and maybe .bc files), but not for something like MIR. At the same time the WorkProductId does not scale to things with a DefId. It seems to me that there is room for improvement.

As I said on IRC, I also felt like the WorkProduct thing wasn't as general as I originally thought it would be. I suspect it might be better to just rip it out and change all that stuff to be specific to object files -- though I guess what is there could work just as well for bc files as it does for o files, so it's at least general enough for that.

(OTOH, it won't work for MIR, since we'd want something with a def-id as key for that; they also probably don't need the auxiliary hash info.)

I'm inclined to leave it now but just expect it to evolve, but that's mostly because I'm lazy. I'd also be happy to rip it out.

@michaelwoerister
Copy link
Member

I'm inclined to leave it now but just expect it to evolve, but that's mostly because I'm lazy. I'd also be happy to rip it out.

I'm fine with leaving it as it is in the PR for now. If we had a clear vision of what it ought to look like, that would be different.

@bors
Copy link
Contributor

bors commented Jul 22, 2016

☔ The latest upstream changes (presumably #34917) made this pull request unmergeable. Please resolve the merge conflicts.

@nikomatsakis
Copy link
Contributor Author

@bors r=mw

@bors
Copy link
Contributor

bors commented Jul 22, 2016

📌 Commit 5c56fca has been approved by mw

@bors
Copy link
Contributor

bors commented Jul 23, 2016

⌛ Testing commit 5c56fca with merge f62c12b...

@bors
Copy link
Contributor

bors commented Jul 24, 2016

💔 Test failed - auto-linux-musl-64-opt

@nikomatsakis
Copy link
Contributor Author

@michaelwoerister care to review the final commit?

@bors
Copy link
Contributor

bors commented Jul 26, 2016

💔 Test failed - auto-win-msvc-64-opt-rustbuild

@alexcrichton
Copy link
Member

@bors: retry

On Tue, Jul 26, 2016 at 5:23 AM, bors notifications@github.com wrote:

💔 Test failed - auto-win-msvc-64-opt-rustbuild
https://buildbot.rust-lang.org/builders/auto-win-msvc-64-opt-rustbuild/builds/1979


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#34956 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAD95JZkQsxuMpO50qTI2ocLXsY7R55Nks5qZfw2gaJpZM4JSAbm
.

@bors
Copy link
Contributor

bors commented Jul 28, 2016

☔ The latest upstream changes (presumably #34907) made this pull request unmergeable. Please resolve the merge conflicts.

A work product right now is just a `.o` file. In the future it probably
includes other kinds of files, such as `.bc` files saving the
unoptimized LLVM IR.

However, because WorkProductIds must be independent of DefIds, so that
they don't need translation, this system may not be suitable *as is* for
storing fine-grained information (such as the MIR for individual defs),
as it was originally intended. We will want to refactor some for that.
We used to use `Name`, but the session outlives the tokenizer, which
means that attempts to read this field after trans has complete
otherwise panic. All reads want an `InternedString` anyhow.
Work products are deleted if any of their inputs are dirty.
This checks the `previous_work_products` data from the dep-graph and
tries to simply copy a `.o` file if possible.  We also add new
work-products into the dep-graph, and create edges to/from the dep-node
for a work-product.
In the older version, a `.o` and ` .bc` file were separate
work-products.  This newer version keeps, for each codegen-unit, a set
of files of different kinds. We assume that if any kinds are available
then all the kinds we need are available, since the precise set of
switches will depend on attributes and command-line switches.

Should probably test this: the effect of changing attributes in
particular might not be successfully tracked?
@nikomatsakis
Copy link
Contributor Author

@bors r=mw

@bors
Copy link
Contributor

bors commented Jul 28, 2016

📌 Commit 2f9fff2 has been approved by mw

@nikomatsakis
Copy link
Contributor Author

@bors r-

seeing make check errors locally :(

actually we shouldn't even hash nested items at all, but that is
addressed in a followup PR
@nikomatsakis
Copy link
Contributor Author

@bors r=mw

@bors
Copy link
Contributor

bors commented Jul 28, 2016

📌 Commit 42cd5d4 has been approved by mw

@bors
Copy link
Contributor

bors commented Jul 28, 2016

⌛ Testing commit 42cd5d4 with merge 54c0dcf...

bors added a commit that referenced this pull request Jul 28, 2016
Enable reuse of `.o` files if nothing has changed

This PR completes a first "spike" for incremental compilation by enabling us to reuse `.o` files when nothing has changed. When in incr. mode, we will save `.o` files into the temporary directory, then copy them back out again if they are still valid. The code is still a bit rough but it does seem to work. =)

r? @michaelwoerister

Fixes #34036
Fixes #34037
Fixes #34038
@WiSaGaN
Copy link
Contributor

WiSaGaN commented Aug 5, 2016

Will this reduce the time of cargo check if only one of several files changes?

@nikomatsakis
Copy link
Contributor Author

@WiSaGaN it should, yes, but we are still fixing some known shortcomings there.

@nikomatsakis nikomatsakis deleted the incr-comp-o-files branch October 3, 2016 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants