-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ModuleCache initializing takes a minute #53
Comments
This should have broke
This indicates either a really huge app (with 26000+ classes/modules/ For example here the an output for an app I have (running ruby 2.0):
As can be seen the app has 4235 modules/classes, and iterating over them 10 times (this should be equivalent to iterating over 42350 modules/classes) takes 0.8s.
Check how fast "pure"
This way we can rule out that native Then, check whether
|
Yeah the Rails app I'm working on is pretty ginormous. Here's the benchmarks:
and then calling
but take note that I this is just ONE call. Iterating over 10x would take 5min+. Lastly this is the output of the unbind:
|
Mine current hunch is that you are running into an issue with This is because: if a module/class name is unknown (which it is for "anonymous" modules/classes) the whole Ruby constant namespace is traversed by MRI trying to find the name of the You can check how many of the modules/classes of your app are "anonymous" like this:
Does this yield thousands modules/classes for you (I'm expecting it to be something like 20000+)? |
I think your hunch is right, here:
So this is interesting. I did read about anonymous module/classes and their side effects. I'll have to read up on it. |
Is keeping a persistent set/hash of anon class to skip the
And here's the benchmark after:
|
Don't know what to do about that abysmal anon module/class Not sure if I'll ever be able to fix this for you (short of fixing MRI). You could alleviate the problem by giving your anonymous modules names, not sure have feasible that is given your app's constraints: def deanonymize_module_or_class(mod)
Object.const_set(:XYZ, mod)
mod.name # ensure the name is computed and cached by MRI
Object.send(:remove_const, :XYZ) # unset the const, `mod` will now forever retain XYZ const name
end Note that Ruby will not care that 19224 of your classes/modules now have the same |
I can't, they wouldn't get garbage collected by Ruby VM then... I also can't set a an i-var on them like this: if mod.instance_variable_get(:@anon_module)
# skip
elsif Util.anonymous_const_name?(mod._mod_name)
mod.instance_variable_set(:@anon_module, true)
else
# work with mod
end Because next time around a previously anonymous module might no longer be anonymous and |
This is what I was planning. In config/application.rb I've added the deanonymizing code in an after_initialize block. Obviously this adds about an extra minute to the application boot time, but much better than the 40s per reload time. I guess unless we monkey patch MRI, this is good enough for me. What I'm wondering is how does languages like Python handles anon name look ups. |
This is purely an MRI implementation problem, other Ruby VMs (JRuby and Rubinius) don't suffer from this issue, they always keep track of whether a module/class has a name or not, if it doesn't This is because MRI optimizes for boot-up performance (when a module acquires a name, MRI chooses not to bother storing it on a module's object, thinking "if somebody asks for a module name, we'll get around to it and figure it out") whereas JRuby and Rubinius always keep module object's name up to date (an anonymous module has a null pointer, if it ever acquires a name a pointer is updated with the proper name). PS: personally I believe MRI behavior is a legacy artifact/quirk and switching it to JRuby/Rubinius "mode" won't result in any boot-up slow down. |
@pxue not sure if |
@thedarkone Nice! we should back contribute to the discussion. we have some nice real world benches here of the problem. |
I'm having problem where every reload is taking about a minute. With the debug mode on I was able to trace through the code to dependencies_patch Line: 115
Apparently iterating over ObjectSpace and mapping _mod_name takes a really long time.
For example, if I open up rails console and run
ObjectSpace.each_object(Module){|mod| mod._mod_name }
take about 100 seconds. The count returned is 26000+.
I've monkey patched Line 231 of same file
module_cache = nil
so that it doesn't reload the cache on every file reload. This improved the reload time significantly to around 1s, but I'm not sure what side effects I've introduced (I'm a ruby nub).Any insights or point me to the right direction? Thanks!
The text was updated successfully, but these errors were encountered: