Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a memoizing ttl library to reduce solr calls #212

Merged
merged 3 commits into from
Oct 21, 2024

Conversation

billdueber
Copy link
Collaborator

The implementation of `solr_cloud-connection` is purposefully chatty,
not caching anything because during the admin cycle, you want to see
your changes reflected immediatley and the number of operations is generally small.

My use of uncached values (checking the name of the collection underlying an alias) on what
turned out to be basically every call was disastrous.

This PR creates use `ttl_memoizeable` to memoize the two calls for data that can
change after a new upload: the hyp_to_bib data, and the date of indexing (for the
footer).

In theory we should just have some sort of a listener that tells us when
the underlying collection has changed and have that trigger things,
but we don't, so here we are.

 Notable changes are:
   * Two methods in load_local_config now memoize values for the
     hyperbib mapping and the name of the current underlying
     connection.
   * The formerly-recursive method of determining the underlying collection
     name (because it's legal, in general, to have aliass that point to other
     aliases) has been removed, since we just don't need it.
   * The footer now shows, ridiculously, the time down to the second.
     This is purely to make testing easier, because seeing a change in
     the footer date is an easy way to know the change has been made.

The implementation of `solr_cloud-connection` is purposefully chatty,
not caching anything because during the admin cycle, you want to see your
changes reflected immediatley and the number of operations is generally small.

My use of uncached values (checking the name of the collection underlying an
alias) on what turned out to be basically every call was disastrous.

This PR creates an instance of [Concurrent::TimerTask](https://ruby-concurrency.github.io/concurrent-ruby/master/Concurrent/TimerTask.html) that runs the check,
and updates if needed, every 20 seconds.

 Notable changes are:
   * All the logic about updates is moved into `config/initializers/solr_admin_cache.rb`
   * The formerly-recursive method of determining the underlying collection name (because it's legal,
     in general, to have aliass that point to other aliases) has been removed, since we just don't need it.
   * The cached values are stored in `Concurrent::Atom` instances in the Services module with everything else
   * `load_local_config.rb` basically just calls `#value` on the Services values.
   * The footer now shows, ridiculously, the time down to the second. This is purely to make testing easier,
     because seeing a change in the footer date is an easy way to know the change has been made.
The implementation of `solr_cloud-connection` is purposefully chatty,
not caching anything because during the admin cycle, you want to see
your changes reflected immediatley and the number of operations is generally small.

My use of uncached values (checking the name of the collection underlying an alias) on what
turned out to be basically every call was disastrous.

This PR creates use `ttl_memoizeable` to memoize the two calls for data that can
change after a new upload: the hyp_to_bib data, and the date of indexing (for the
footer).

In theory we should just have some sort of a listener that tells us when
the underlying collection has changed and have that trigger things,
but we don't, so here we are.

 Notable changes are:
   * Two methods in load_local_config now memoize values for the
     hyperbib mapping and the name of the current underlying
     connection.
   * The formerly-recursive method of determining the underlying collection
     name (because it's legal, in general, to have aliass that point to other
     aliases) has been removed, since we just don't need it.
   * The footer now shows, ridiculously, the time down to the second.
     This is purely to make testing easier, because seeing a change in
     the footer date is an easy way to know the change has been made.
Copy link
Member

@botimer botimer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should do it!

@billdueber billdueber merged commit 04724d3 into main Oct 21, 2024
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants