Use member_ids_ssim instead of file_set_ids_ssim #6513
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
file_set_ids_ssim has always been a verbatim copy of member_ids_ssim, and in works with many members it can become an awkwardly large piece of duplicated data in solr. PcdmObjectIndexer does not duplicate this field, but various systems expect it to be available. This change modifies those systems to use member_ids_ssim. Filtering by has_model_ssim is added where needed to filter collections from results.
dlpierce
force-pushed
the
use-member_ids_ssim
branch
from
9ae466f
to
7108409
Compare
bwatson78
approved these changes
Dec 7, 2023
jeremyf
added a commit
to scientist-softserv/iiif_print
that referenced
this pull request
Feb 23, 2024
For several years Hyrax has index `file_set_ids_ssim` as a verbatim copy of `member_ids_ssim`. With Hyrax 5, we're removing the `file_set_ids_ssim` from indexing; And given that it's been a verbatim copy since 2017 or so, it's relatively safe to assume that we can favor, without application impact, the `member_ids_ssim` over the `file_set_ids_ssim` value. It would be nice to have `file_set_ids_ssim` but not as a verbatim copy. Someday, we might have nice things. Related to: - samvera/hyrax#6513 - samvera/hyrax@7108409
jeremyf
added a commit
to samvera/bulkrax
that referenced
this pull request
Feb 23, 2024
For several years Hyrax has index `file_set_ids_ssim` as a verbatim copy of `member_ids_ssim`. With Hyrax 5, we're removing the `file_set_ids_ssim` from indexing; And given that it's been a verbatim copy since 2017 or so, it's relatively safe to assume that we can favor, without application impact, the `member_ids_ssim` over the `file_set_ids_ssim` value. It would be nice to have `file_set_ids_ssim` but not as a verbatim copy. Someday, we might have nice things. In the case of Bulkrax, given that we don't require Hyrax, we need to have a configurable option as well as one that is a consistent fallback to what came before. Related to: - samvera/hyrax#6513 - samvera/hyrax@7108409
jeremyf
added a commit
to samvera/hyku
that referenced
this pull request
Feb 23, 2024
For several years Hyrax has index `file_set_ids_ssim` as a verbatim copy of `member_ids_ssim`. With Hyrax 5, we're removing the `file_set_ids_ssim` from indexing; And given that it's been a verbatim copy since 2017 or so, it's relatively safe to assume that we can favor, without application impact, the `member_ids_ssim` over the `file_set_ids_ssim` value. It would be nice to have `file_set_ids_ssim` but not as a verbatim copy. Someday, we might have nice things. Related to: - samvera/hyrax#6513 - samvera/hyrax@7108409
jeremyf
added a commit
to samvera/bulkrax
that referenced
this pull request
Feb 23, 2024
* ♻️ # ♻️ Favor member_ids_ssim over file_set_ids_ssim For several years Hyrax has index `file_set_ids_ssim` as a verbatim copy of `member_ids_ssim`. With Hyrax 5, we're removing the `file_set_ids_ssim` from indexing; And given that it's been a verbatim copy since 2017 or so, it's relatively safe to assume that we can favor, without application impact, the `member_ids_ssim` over the `file_set_ids_ssim` value. It would be nice to have `file_set_ids_ssim` but not as a verbatim copy. Someday, we might have nice things. In the case of Bulkrax, given that we don't require Hyrax, we need to have a configurable option as well as one that is a consistent fallback to what came before. Related to: - samvera/hyrax#6513 - samvera/hyrax@7108409 * Update app/parsers/bulkrax/parser_export_record_set.rb
jeremyf
added a commit
to scientist-softserv/iiif_print
that referenced
this pull request
Feb 23, 2024
For several years Hyrax has index `file_set_ids_ssim` as a verbatim copy of `member_ids_ssim`. With Hyrax 5, we're removing the `file_set_ids_ssim` from indexing; And given that it's been a verbatim copy since 2017 or so, it's relatively safe to assume that we can favor, without application impact, the `member_ids_ssim` over the `file_set_ids_ssim` value. It would be nice to have `file_set_ids_ssim` but not as a verbatim copy. Someday, we might have nice things. Related to: - samvera/hyrax#6513 - samvera/hyrax@7108409
laritakr
added a commit
to samvera/hyku
that referenced
this pull request
Apr 2, 2024
* Delete _repository_content.html.erb Having this file was causing double rendering of bulkrax links in the sidebar. Issues: - #1850 - scientist-softserv/hykuup_knapsack#111 * use hyrax-4-valkyrie-support bulkrax branch Update bulkrax to point to hyrax-4-valkyrie-support * 🐛 App was not loading after updating Hyrax This commit will adjust for the changes introduced in this commit: samvera/hyrax@8fc0894 * ⚙️ Don't remove backtrace in dev * remove bulkrax_identifier from Hyku bulrax_identifier is not a required property. Each client should set their desired source_identifier in their application. Issue: - scientist-softserv/hykuup_knapsack#136 * 🧹 Make the appropriate link generate This commit will add a decorator to check the `human_readable_type` of the given object is a Valkyrie migration object (which by convention ends with 'resource') and adjust the generated link accordingly. For example, a GenericWorkResource should generate links like `/concern/generic_work_resources/...` instead of `/concern/generic_works/...`. The models were updated to include the Hyrax::NestedWorks so the `Attach Child` button would populate the appropriate work types. Lastly, the `Gemfile.lock` was updated to include the latest version of Hyrax `double_combo` branch. * 🧹 Add Valkyrie test adapter For us to leverage the useful shared specs of Hyrax, we need a test adapter. * 🧹 Always with the coppers * Updating Hyrax to latest version * 🧹 Rework logic for #hydra_model Using 'human_readable_type' was going to be flemsy. Instead we are opting for adding the `valkyrie_bsi` to the index and asking that. * 📚 Add docs regarding knapsack * 🧹 Favor not using valkyrie for the spec * ♻️ Only enable auto-redirect when Valkyrie enabled. * 🧹 Configure Bulkrax We configure Bulkrax to use the `ValkyrieObjectFactory` along with the `ValkyrieMigrationCoercer` so a `GenericWork` will create a `GenericWorkResource`. Also, updated the revision of Bulkrax. * Revert "🧹 Remove files declared in Knapsack" This reverts commit d9fa1a2. * Revert "remove bulkrax_identifier from Hyku" This reverts commit 887fbf2. * ♻️ Bump Hyrax ref * ♻️ Favor Hyrax::SolrService Ideally the code would reference the SolrService in one manner. This is an effort to create an insulating layer around that. * make find_by_bulkrax_identifier.rb more dynamic Not all applications will use bulkrax_identifier as their source_identifier. This code makes it more dynamic. TODO: Make Wings::CustomQueries more dynamic * Leverage updated IIIF Print * 🧹 Appeasing rubocop * find_by_source_identifier files moved to bulkrax ref: - samvera/bulkrax@29f2264 * Revert "find_by_source_identifier files moved to bulkrax" This reverts commit 78d26c5. * update bulkrax and iiif_print versions find_by_source_identifier files were moved into bulkrax. * 🧹 apply conditional to hyku indexing specs refer to object as @object, and thus caused a lot of spec failures without this. * 💄 rubocop fix * 🧹 Fix set child flag This commit will make sure that the set child flag transaction fires when ingesting with Bulkrax. Also fixed a weird bug where the extra space in the query was causing the query to fail. * add guard for ActiveFedora's member_ids AF's member_ids returns an array of ids so mapping (:id) will not work. * Update Bulkrax and IiifPrint Pull in fixes to make bulkrax relationships work for valkyrie. It also fixed the setting of is_child and the linking of relationships Related: - samvera/bulkrax#908 - scientist-softserv/iiif_print#328 * remove HYKU_IIIF_PRINT conditional Per Rob, remove HYKU_IIIF_PRINT conditional. Iiif print will be enabled by default. * update hyrax and iiif print * Valkyrize reindex rake tasks This commit adds reindexing for Valkyrie objects. Issue: - scientist-softserv/hykuup_knapsack#146 Co-Authored-By: Kirk Wang <k3wang@gmail.com> * 🐛 Ensure underlying change to permissions * Fix for xray-rails not working Issue: - scientist-softserv/hykuup_knapsack#71 Co-Authored-By: C Barton <43180845+CB987@users.noreply.github.com> * Bumping Hyrax version * 💄rubocop fixes Co-Authored-By: C Barton <43180845+CB987@users.noreply.github.com> * update hyrax, bulkrax, iiif_print * update hyrax * Pin Hyrax to functioning build * Delete find_by_source_identifier_spec.rb This spec is no longer needed as its relevant code moved into bulkrax. * Update iiif_print ref: add a reindex after the child work has been saved. - scientist-softserv/iiif_print#332 * 🧹 Remove `form` key from with_pdf_viewer The `show_pdf_viewer` and `show_pdf_download_button` fields were showing up on the form even though in the yaml they were set to false. It looks like just removing the `form` key from the yaml file will prevent them from showing up. Also updated the `Gemfile.lock` file. * 🧹 Fix specs to reflect usage of Double Combo * 🐛 Use registered instead of restricted Restricted is the visibility, registered is the concept. This fixes that misnomer. * 🧹 Amend spec to highlight double combo nuance * 🧹 Rework lease expiry tests See https://github.com/samvera/hyku/wiki/Updating-Hyku-6-with-Hyrax-5-Developer-Notes#access-control-list-acl-considerations * 💄 endless and ever appeasing of the coppers * 🧹 Fix broken spec * 💄 endless and ever appeasing of the cops * Adding documentation * 🧹 Adjust AdminSet spec count to reflect query service * 🧹 Address assumptive redirect for /concern/generic_work/ * 🧹 Fixing redirect assumptions for insitutional visibility * 🧹 Favor up to date double_combo * Fixing a few specs to reflection Valkyriziation * 🧹 Updating Hyrax version Grab the latest sweet sweet double combo * include Bulkrax helpers and update Bulkrax * Rework spec to not rely on reload * 🧹 Fixing tests to use generic_resource_factory The factory is directly copied from Hyrax; and is not fully functional, but it's better than what we've got. * Fixing a spec that needs a #to_a * ♻️ Add Hyku.bulkrax_enabled? Also refactor to set the default work type for Bulkrax later in the application boot cycle. When it was where it was at, I encountered issues with wings model registration. * ♻️ Favor re-using the controller and hopefully routes * Demonstrate how we're using lazy migrations and model naming * ♻️ Remove generated files Favor the Lazy migration strategy * 🧹 endless and ever appeasing of the coppers * ☑️ Tidying up tests * ☑️ Fixing a few broken tests By extending the ability class to include all of the collection models, we open up some additional considerations. * Move ability declarations out of loop, where loop not needed * 💄 endless and ever appeasing of the coppers * ☑️ Fill out require attributes * ☑️ Working through failing tests This does involve fixing a bug that is masking another underlying bug. * Fixing broken specs * 🐛 Refactor collection ability class * Add elsif for FileSet and configure IIIF Print This commit will add an elsif statement to the wings adapter to handle FileSet objects. It will also add the configuration for IIIF Print to use the IiifPrint::PersistenceLayer::ValkyrieAdapter. * Bumping double_combo version * Favor helper method over instance variable * ☑️ We need to create a method to stub * ☑️ Ever working towards tests * 💄 endless and ever appeasing of the coppers * 🎁 Add some FileSet interface friendliness * ☑️ Favor factory over process through actor stack * ☑️ Adding tests for GenericWorkResrouce * 🎁 Favor inheriting from admin_set * ☑️ Removing spec/factories defined Hyrax * 💄 endless and ever appeasing of the coppers * ♻️ Fixing some tests to rely on factories instead of actor * 🚧 Update Hyrax and IIIF Print This commit will update the Hyrax and IIIF Print revisions which should allow us to ingest images and have it render in the UV. Uploading a PDF should also work but PDF splitting current does not work. * ☑️ Fixing a few tests by re-purposing some factories * WIP * Removing unneeded decorator and associated spec * 🧹 Ignore vendor/gems directory * ☑️ Adding documentation regarding Factories * ☑️ Fixing tests and factories for roles_service * ♻️ Favor site configuration for allow downloads Hyrax 5 has a display_media_download_link configuration option that we can repurpose for the allow_downloads functionality that we are looking to back-port from Hyku. * ☑️ Fixing a broken test * Fixing a few broken specs, still some broken * 💄 endless and ever appeasing of the coppers * Removing long-ago removable code * Modify Collection & Admin Set Valkyrization Addresses: ☄️ EPIC: Valkyrize Hyku scientist-softserv/hykuup_knapsack#35 valkyrize collection and admin set resources scientist-softserv/hykuup_knapsack#94 Moves away from directly using Hyrax::PcdmCollection and Hyrax::AdministrativeSet instead favoring inheriting from those (with CollectionResource and AdminSetResource). This allows for inheriting from Hyrax factories and extending them; thus bringing Hyku’s testing ecosystem closer to Hyrax. Note: There are a few of the new specs that still fail. - * Coppers again * ☑️ Favor explicit load of FactoryBot factories * ☑️ Working on getting more tests passing * Fixing ever more specs that keep breaking underneath me * 💄 endless and ever appeasing of the coppers * ♻️ Verify AdminSetResource factory build * Adding specs to demonstrate factories * 🧹 Avoid assuming factory-bot is available. * 🧹 We want to create admin set resources * 🧹 Fix broken spec due to not assuming admin group * ☑️ Fixing specs by re-using factories * ☑️ Adjust spec to use proper collection factory * ☑️ Favor factory over instance double * ☑️ Favor factory over explicit class * ☑️ Favor factory over stub, and correct form class * ☑️ Ensuring we start from a clean slate * Fixing specs * ☑️ Fixing tests squashing bugs that might show up * Fixing specs to use resource classes * Bumping version of IIIF Print * 🐛 Add .each to find_all call Without this, we pass a block to a method that does not anticipate receiving a block. * Fixing feature specs by ensuring admin_group * 🐛 Fixing a couple of different bugs Ever more to do. * 💄 endless and ever appeasing of the coppers * WIP * ♻️ Adding further collection ability testing * ♻️ Put long list of objects into a loop * ♻️ Adding more valkyrie native collection specs * ♻️ Re-arranging collection declarations * ♻️ Working in the spec mines regarding permission * Adding specs for disable * ♻️ Reworking specs and addressing significant typo * 🧹 Fix specs concerning collection management * Ever hacking on the specs * 💄 endless and ever appeasing of the coppers * 🧹 Fix tests regarding collection management * 🧹 Fixing Hyrax::CollectionType#collections query * 🧹 Fixing a few broken specs * 🧹 Reworking factory process * 💄 endless and ever appeasing of the coppers * ♻️ Favor member_ids_ssim over file_set_ids_ssim For several years Hyrax has index `file_set_ids_ssim` as a verbatim copy of `member_ids_ssim`. With Hyrax 5, we're removing the `file_set_ids_ssim` from indexing; And given that it's been a verbatim copy since 2017 or so, it's relatively safe to assume that we can favor, without application impact, the `member_ids_ssim` over the `file_set_ids_ssim` value. It would be nice to have `file_set_ids_ssim` but not as a verbatim copy. Someday, we might have nice things. Related to: - samvera/hyrax#6513 - samvera/hyrax@7108409 * ☑️ Add role trait to reflect tests * 🧹 Set various presenter's with correct file_presenter_class Related to: - samvera/hyrax#6717 * Bumping the double_combo * Bumping version of IIIF Print and Bulkrax * 💄 endless and ever appeasing of the coppers * Adding tests to snare failing specs that work locally * ♻️ Working on factory bot creation of permission templates * ♻️ Fix spec harness for permissions * 📚 Add spec for factories to explain what happens * Update double_combo and add bulkrax migrations * Fixing some underlying specs * ♻️ Fix selector issue * ☑️ Let These Specs Pass! * ☑️ Typo * 💄 endless and ever appeasing of the coppers * Remove redundant declaration * ☑️ Remove flakey tests These tests relied on existing data in solr but did not seek to have clean data, thus received leaking data from other tests. Meaning they would fail. Ugh, tests should (by default) favor starting from a "clean state" instead of inheriting state from other tests. * ☑️ Favor FactoryBot.valkyrie_create To do otherwise, invites a series of weird interactions. * ☑️ Add explicit setting of models before discovery factories * 🧹 Bump to new double_combo * Remove transient as the underlying problem fixed * ♻️ Remove auto-creation of file * ☑️ Favor Hyrax::SolrService.connection to match interface * Bumping to new version of Hyrax double combo * Update Hyrax pulling in Jeremy's changes: custom queries and assigned strategies. * Removing mocked tests that don't prove much * Add embargo and lease to wings We started seeing problems with the embargo (and presumably lease) now allowing us to save the work. This commit adds the necessary fields to the big else statement in the wings initializer. Ref: - scientist-softserv/hykuup_knapsack#157 * Update riiif config to replace deprecated method The `cache_duration_in_days` is deprecated and locally it was causing the web service to freeze. * Adding tenant for request spec * Updating factory spec to catch a failure * Update Hyrax to move binaries This commit will update Hyrax with the ability to move the binaries from Fedora to Disk. There are also some changes to the database schema to make persisting work. Wings initializer was updated to map the Hydra::PCDM::File to the Hyrax::FileMetadata class as well. * start spec to test file set migration * 🐛 Fix Account switch bug * ♻️ Remove method as duplicate of Hyrax The Hyrax::Group.new override handles the "find_by" logic we introduced. * 🎁 Auto-magically migrate file sets and binaries * 💄 endless and ever appeasing of the coppers * ♻️ Remove stray puts statement * Attempting to squash a flakey set of specs * Update #parent_path to account for various case statements The previous assumption was that we'd always have a SolrDocument, which is not true. This method has been updated to be more flexible. * 💄 endless and ever appeasing of the coppers * I had to bail and copy the factories from Hyrax * 💄 endless and ever appeasing of the coppers * ♻️ Disable some consistently failing specs Note, this likely means I can revert a previous commit about adjusting the copies of factories from Hyrax. * Update hyrax version pulls in thumbnail loading fix * Revert "I had to bail and copy the factories from Hyrax" This reverts commit b0cbbac. * Update Hyrax pulls in WIP for af to valkyrie thumbnails + fix for serving thumbnails * Bumping Hyrax version * Updating Hyrax to latest build with FS fixes * Merge branch 'i35-valkyrize-hyku' of https://github.com/samvera/hyku into i35-valkyrize-hyku * ♻️ Update Hyku to new Double Combo and config file_set_model * 💄 endless and ever appeasing of the coppers * Addressing potential routing errors Co-authored-by: Shana Moore <shana@scientist.com> * Remove indexing of collection membership Looking at the Hyrax code, it appears that it should already be done. * ♻️ Adding Hyku::Application.work_types (see docs) * ♻️ Configure `Bulkrax.*_model_class` methods * copy Rob's reprocessor script from GBH into Hyku Co-authored-by: Rob Kaufman <rob@notch8.com> * resolve most rubocop warnings * update rubocop_todo Since we're mostly bringing over the reprocessor script as-is, refactoring the method length is out of scope. * Load HykuKnapsack decorators This commit will allow the HykuKnapsack decorators to be loaded by the application rb as well. * temp: debug build issues * Revert "temp: debug build issues" This reverts commit 06350fc. * Update Bulkrax * Update Bulkrax * Update Bulkrax and Hyrax * 🧹 Update bulkrax gem * Update Bulkrax * 🐛 Fix undefined local variable iiif_print There was a change in this version of IIIF Print that needed an update to routes.rb. Ref: - scientist-softserv/iiif_print#302 * 🐛 Fix collection membership bug (#2169) * 🐛 Fix collection membership bug A hyrax bug made it into Hyku via the collections controller decorator. This removes the unnecessary `collection_params method`. Refs - scientist-softserv/hykuup_knapsack#183 - samvera/hyrax@7add213 * Ignore rubocop error * update Bulkrax * Show PDF.js viewer when PDF.js FlipFlop is true This is a workaround to unblock Mobius ingests. If the feature flipper to display PDF.js is true, we will display that viewer regardless of the property values of :show_pdf_viewer and :show_pdf_download_button. We will revisit valkyrizing this feature fully in the future, when we start the iiif x valkyrie epic. Issue: - scientist-softserv/hykuup_knapsack#185 Co-Authored-By: Kirk Wang <k3wang@gmail.com> * revert previous commit and update logic of #show_pdf_viewer instead This fixes a failing spec while still rendering PDF.js in the UI Co-Authored-By: Kirk Wang <k3wang@gmail.com> * Update valkyrie gem * update valkyrie gem * Update bulkrax to fix bug Refs scientist-softserv/hykuup_knapsack#182 * 🧹 Clean up featured works and collections The featured works and collections had some funky css problems where it made sorting them impossible. This commit will override some Hyrax scss and fix some css classes on partials to make the featured works/collections a better user experience. * Update IiifPrint (#2172) Fixes bug deleting works & file sets. Refs scientist-softserv/hykuup_knapsack#187 * ♻️ Favor method_missing over delegation In the `double_combo` branch, we replaced the need for delegation with method_missing. Thus we can remove the delegate declarations and instead rely on method missing behavior. * ♻️ Remove config that was there for flakey tests * Update app/forms/hyrax/forms/admin/appearance_decorator.rb * Favor published valkyrie gem version * 🎁 Valkyrize Reindex jobs (#2173) Addresses all reindex jobs to ensure that they will work for either valkyrie or active fedora objects. Ensures that works and collections are appropriately reindexed as default work and collection images are added OR removed (previously, reindexing didn't occur with removal of default images, resulting in works displaying with a missing image.) * update gemfile to add support for sentry * 🗡 Stab in the dark at fixing an intermittent bug * add conditional to support HykuUp Knapsack specs * add conditional to support HykuUp Knapsack specs * 💄 rubocop fix * 💄 rubocop fix * 🧹 Add core to hash in spec Since we changed the SolrEndpoint, this spec was failing. This change should fix it. * 🧹 Add storage/files as a persisted volumn This commit will add storage/files as a persisted volume for rancher set ups. Hyrax uses this directory to store the binaries. * update bulkrax, hyrax, iiif_print * bump to rc3 --------- Co-authored-by: Kirk Wang <kirk.wang@scientist.com> Co-authored-by: Jeremy Friesen <jeremy.n.friesen@gmail.com> Co-authored-by: Kirk Wang <k3wang@gmail.com> Co-authored-by: C Barton <43180845+CB987@users.noreply.github.com> Co-authored-by: LaRita Robinson <larita@scientist.com> Co-authored-by: Benjamin Kiah Stroud <32469930+bkiahstroud@users.noreply.github.com> Co-authored-by: Rob Kaufman <rob@notch8.com> Co-authored-by: LaRita Robinson <laritakr@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
file_set_ids_ssim has always been a verbatim copy of member_ids_ssim, and in works with many members it can become an awkwardly large piece of duplicated data in solr. PcdmObjectIndexer does not duplicate this field, but various systems expect it to be available. This change modifies those systems to use member_ids_ssim. Filtering by has_model_ssim is added where needed to filter collections from results.
This should not be breaking for existing applications unless they have custom logic that depends on file_set_ids_ssim which is still indexed for ActiveFedora objects.
Changes proposed in this pull request:
@samvera/hyrax-code-reviewers