Bundler::Resolver.resolve time increases a lot as gems from git sources are added.

[mathieuravaux]

During the sort, before resolving, the number of sources that contain a dependency is calculated, for each dependency.

For projects with a lot of gems specified as a git dependency, this is very slow.
This also seems unnecessary, since these 'git sources' will contain only one gem version, which is resolved de facto. Maybe they could be activated first, and then their 'git sources' ignored when calculating #gems_size ?

Here is a profile that shows 60.76% of an app boot time spent in Bundler::Resolver#gems_size, which asks each one of the git source if they contain each one of the known dependencies: http://f.cl.ly/items/0l0m0e0v312l28371E3j/boot_time_profile.html

The Gemfile contains 122 gems, 14 of them being specified from a particular git repository.

$ time BUNDLE_FROZEN=true ./bundle_bin/thin start 
     7,01s user 1,60s system 87% cpu 9,895 total
$ time ./bundle_bin/thin start 
    13,04s user 1,55s system 91% cpu 15,872 total

Since the resolve is by default done each time our projects boot, this considerably drags down the development.

[indirect]

Are you using the 1.1 prerelease? We started caching the results of #gem_size/#search, it seems to have helped.

[mathieuravaux]

The tests were done with the 1.1.rc

Mathieu Ravaux

On mardi 1 novembre 2011 at 06:42, Andre Arko wrote:

Are you using the 1.1 prerelease? We started caching the results of #gem_size/#search, it seems to have helped.

Reply to this email directly or view it on GitHub:
#1511 (comment)

[indirect]

Great. I'd love to see a patch or a benchmark that ignores git sources as you suggest, is that something you'd be up for?

[mathieuravaux]

I've given it a try and it worked OK for the most part, but broke some specs for special cases, like with git submodules.

I'll dig a bit deeper to understand them, but It seems to me that to solve certain dependencies, we have to dig through the git dependencies anyway.

However, since the Path and Git sources usually contain only a few entries, we could use only one Index for all of them, instead of an Index for each of them. This should still significantly speed up the gems_size calls by iterating through only 2 indexes instead of N+1.

What do you think ?

Mathieu

[indirect]

That sounds similar to the way we have one Rubygems index for local specs, one for remote specs, etc. Seems good to me.

[spastorino]

@mathieuravaux are you still working on a fix for that?

[mathieuravaux]

@spastorino No, I'm currently not and won't be able to in the next weeks.

I let the bundle frozen during normal work, so the dependency resolution doesn't run every time the app boots, which was the issue for me in the first place.

[spastorino]

@mathieuravaux ae592a9 let me know how that goes for you. There's still more room to improve but I don't really know the Bundler code base :P. We should probably take another look after trying my patch. It has improved 3x in speed for me but still a bit slower than 1.0.x