You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With some of the current changes in core and moving to SOLR 8, we should reconsider some rework on this extension. There are few issues that I would point as concerns:
Creating of the HarvestObject for each dataset. - This is executing at the end of gather stage and if there is significant number of datasets e.g 100k+ it could lead performance issues. My suggestion is to create internal method _create_harvest_object which could be called on every package_search itteration.
Deleting the deleted packages from source. - As it was mention in the Ian's comment, we could use the recently_changed_packages_activity_list API to get the packages for re-harvesting.
With some of the current changes in core and moving to SOLR 8, we should reconsider some rework on this extension. There are few issues that I would point as concerns:
HarvestObject
for each dataset. - This is executing at the end of gather stage and if there is significant number of datasets e.g 100k+ it could lead performance issues. My suggestion is to create internal method_create_harvest_object
which could be called on everypackage_search
itteration.deleted
packages from source. - As it was mention in the Ian's comment, we could use therecently_changed_packages_activity_list
API to get the packages for re-harvesting.harvesters
tab to ckan admin page.Add the harvesters tab to ckan admin dashboard #500
@ckan/core
The text was updated successfully, but these errors were encountered: