Huge performance hit when upgrading from Mongoid 7.5.4 to 8.1.4

[braindeaf]

It's a little difficult to demonstrate but we've recently been having a number of issues with our application, performance was not great on a number of pages. We found in some cases there were multiple millions of allocations just for one report page and it took 60s - 90s to load. Time to really address bottlenecks in performance. While we found some quick gains we thought it's worth bringing Mongoid up to the latest version.

It wasn't easy, we used many after_commit callbacks which checked changes in the models, so there was alot using previous_changes when we used to use changes and all non-optional belongs_to associations now validates the associated object. It was tricky in some places since callbacks on embedded objects had some unexpected behaviour. Indeed the .any_of produced different query syntax. Anyway...

After seeing some performance gains in development we tested the same pages in staging and then went for a production deploy about a month ago. Since then we've been plagued with much slower pages and timeouts that weren't there previously. Pages that took 5 seconds (yes that is bad and needs improvement anyway) now take 90 seconds and timeout. Has anyone else reported performance hits since upgrading? Does Mongoid maybe change the way we use our indexes? We use a cluster in production and not sure if this would / should make a difference.

For now we're reverting our upgrade changes, pulling in some of the performance gains we found along the way. But we can't sit on an old version forever and need to a way to move forward. Does anyone have any suggestions on how to debug this?

Many thanks

RobL

[alexbevi]

Hey @braindeaf,

Indeed the .any_of produced different query syntax. Anyway...

This sounds like MONGOID-5172, however we'd need to see a before/after of what's being produced to confirm/deny.

Has anyone else reported performance hits since upgrading?

Note that there are some regressions we've identified in 8.x of Mongoid such as MONGOID-5635 and MONGOID-5704, however without more detail about your specific configuration it would be challenging to determine if these are affecting you.

Does Mongoid maybe change the way we use our indexes?

Index utilization is a function of the query planner - which executes server side. As a result Mongoid would have no impact here unless you're hinting an index or if a default scope is defined that includes sort criteria that isn't indexed.

But we can't sit on an old version forever and need to a way to move forward. Does anyone have any suggestions on how to debug this?

First I would start with the slow pages you've identified. How many Mongoid operations are being sent to the server? How long are they taking? This info should be available in the Rails logs and should be pretty easy to extract. Are these operations faster/slower/same when testing the exact same page in 7.x?

The goal here is to identify if database operations are impacted directly. If they're not, then Mongoid internals may have changed that are effecting performance. The easiest way to troubleshoot this is to isolate the logic that is Mongoid-specific and benchmark it against 7.x and 8.x.

If you can isolate and benchmark specific logic, this can also be investigated/validated by our team to help identify opportunities for improvement.

Note that improving the performance of Mongoid is a priority for the product team so we're grateful for any information you can share.

[braindeaf]

Sorry for the delay on following up on this. I am not quite sure how we can offer further insight as we've now moved back to Mongoid 7.5.4. What I can demonstrate is that (from NewRelic) you can see that there is a upward bump in the graph from the 7th January when we deployed, which demonstrably reduced when we reverted. Obviously this is an average.

If there is an effective way we can inject some better instrumentation to trace the differences between our two branches performance we can go some way to helping with this issue.

[alexbevi]

@braindeaf thanks for sharing. We believe that MONGOID-5704 is likely a large contributor to the issues you're facing, though likely not the only issue. I also believe you're already participating on the community thread at https://www.mongodb.com/community/forums/t/mongoid-8-slow-insertions/225228 where this is being discussed.

Our team is actively working to address these regressions and hope to have more information to share via the tickets we've mentioned in this discussion.

We also appreciate the offer for help in testing/instrumenting your application. I will reach out directly in the event we can coordinate with your team on this.

[braindeaf]

Excellent, absolutely. We are happy to test our branch against a particular branch / commit as and when and report back.

[alexbevi]

@braindeaf we've addressed a major performance regression and released 8.0.8 and 8.1.5 of Mongoid with this fix. Would love to hear how this affects the performance of your workload(s) once you've had a chance to test.