Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a bootsnap command to allow to precompile gems without booting the application #326

Merged
merged 1 commit into from
Oct 30, 2020

Conversation

casperisfine
Copy link
Contributor

In normal usage, bootsnap's iseq cache is computed on the fly as the ruby files are required.

Which means that if you want to precompute the cache, the only way is to boot the application.

In my case I'm building Docker image for ruby applications, and I'm caching all the gems in a content addressed layer. So I would like to be able to precompile the gems iseq cache without having to have a bootable application, so that the bootsnap cache is part of the bundler layer.

This is exactly what this new executable allows, it setup bundler to get all the relevant gems in $LOAD_PATH, and then precompile all the requirable.

Copy link
Member

@DazWorrall DazWorrall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice idea 👍

Copy link
Member

@burke burke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great idea 👍

@casperisfine casperisfine force-pushed the executable branch 11 times, most recently from cb62720 to e26dd90 Compare October 29, 2020 17:20
class CLI
unless Regexp.method_defined?(:match?)
module RegexpMatchBackport
refine Regepx do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it's just a CLI and the process is thrown away, it might be enough to define the method directly? Or is the code actually loaded? I thought it was just parsed and the iseq cached.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather avoid doing it. I'll likely add some tests for that class, so it's better if it doesn't monkey patch Regexp.


source.map { |d| File.expand_path(d) }.each do |path|
Dir[File.join(path, '**/*.rb')].each do |ruby_file|
if !exclude || !exclude.match?(ruby_file)
Copy link
Member

@etiennebarrie etiennebarrie Oct 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or even just use =~ which does not allocate a MatchData if that's what we're going for.

def invalid_usage!(message)
STDERR.puts message
STDERR.puts
STDERR.puts parser
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe warn would be simpler?

sources += $LOAD_PATH
end

CompileCache::ISeq.compile_option_updated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really sure it's worth changing the API or adding an alias, but this name could be clearer, now that it's called from outside of CompileCache::ISeq.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. I think I'll refactor this so that I don't need this to be used.

@casperisfine
Copy link
Contributor Author

So after trying this on one of our mid sized apps, I made a few changes.

I added --exclude PATTERN option because I noticed some large gems with lots of never loaded files. e.g. google-api or aws- gems. They contains tons of autogenerated code that most people never require. Without this option the bootsnap cache grew from ~100MB to ~280MB. After excluding the couple of outlier gems it is down to a more reasonable ~180MB.

I also realized that rake environment doesn't eager load the app even if config.eager_load = true is set, it needs a distinct config.rake_eager_load = true option. Until then I was relying on rake assets:precompile to eager load the app and warm the bootsnap cache, but now I think it would be preferable to more explicitly precompile the app code. So now the command has a --gemfile flag to automatically compile all the gems, and take a list of paths to be precompiled.

I'll add some tests and some documentation tomorrow and likely will cut a new release.

@casperisfine casperisfine force-pushed the executable branch 7 times, most recently from 46b10f5 to f9e74de Compare October 30, 2020 11:51
@casperisfine
Copy link
Contributor Author

Hum, it's been hours and the travis builds still didn't start. We should migrate to GitHub Actions soon.

@casperisfine casperisfine merged commit b54a367 into master Oct 30, 2020
@casperisfine casperisfine deleted the executable branch October 30, 2020 13:36
@casperisfine casperisfine temporarily deployed to rubygems November 2, 2020 09:57 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants