Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor code #501

Open
tino097 opened this issue May 9, 2022 · 0 comments
Open

Refactor code #501

tino097 opened this issue May 9, 2022 · 0 comments

Comments

@tino097
Copy link
Member

tino097 commented May 9, 2022

With some of the current changes in core and moving to SOLR 8, we should reconsider some rework on this extension. There are few issues that I would point as concerns:

  1. Creating of the HarvestObject for each dataset. - This is executing at the end of gather stage and if there is significant number of datasets e.g 100k+ it could lead performance issues. My suggestion is to create internal method _create_harvest_object which could be called on every package_search itteration.
  2. Deleting the deleted packages from source. - As it was mention in the Ian's comment, we could use the recently_changed_packages_activity_list API to get the packages for re-harvesting.
  3. Adding the harvesters tab to ckan admin page.
    Add the harvesters tab to ckan admin dashboard #500

@ckan/core

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant