Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Valkyrie Indexer Setup and Specs #4221

Closed
wants to merge 59 commits into from
Closed

Valkyrie Indexer Setup and Specs #4221

wants to merge 59 commits into from

Commits on Jan 25, 2020

  1. Add indexer to work_resource generator

    Julie Allinson committed Jan 25, 2020
    Configuration menu
    Copy the full SHA
    6868bb4 View commit details
    Browse the repository at this point in the history
  2. rename index_hash to index_document for consistency

    Julie Allinson committed Jan 25, 2020
    Configuration menu
    Copy the full SHA
    69aba72 View commit details
    Browse the repository at this point in the history
  3. Use the configured storage adapter

    We weren't respecting storage adapter configuration for valkyrie! This fixes
    that.
    Tom Johnson committed Jan 25, 2020
    Configuration menu
    Copy the full SHA
    961aeb8 View commit details
    Browse the repository at this point in the history
  4. indexer setup, including fixes / tests for activefedoraconverter and …

    …modeltransformer
    
    monograph indexer spec
    Julie Allinson committed Jan 25, 2020
    3 Configuration menu
    Copy the full SHA
    0b21d22 View commit details
    Browse the repository at this point in the history
  5. active_fedora_converter etc. changes

    Julie Allinson committed Jan 25, 2020
    Configuration menu
    Copy the full SHA
    e4acc51 View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2020

  1. add registration of indexers into an initializer

    Julie Allinson committed Jan 26, 2020
    Configuration menu
    Copy the full SHA
    b9d3689 View commit details
    Browse the repository at this point in the history
  2. remove additional test from valkyrie_indexer_spec.rb - this has moved…

    … to generator PR
    Julie Allinson committed Jan 26, 2020
    Configuration menu
    Copy the full SHA
    fc31e18 View commit details
    Browse the repository at this point in the history
  3. subscribe the MetadataIndexListener in config/initializers/listeners.…

    …rb; call the event after_create_concern
    Julie Allinson authored and Tom Johnson committed Jan 26, 2020
    Configuration menu
    Copy the full SHA
    ed0ab01 View commit details
    Browse the repository at this point in the history

Commits on Jan 27, 2020

  1. Refactor CollectionMemberSearchBuilder to better use FilterByType

    `Hyrax::FilterByType` already includes infrastructure for filtering by
    collection and work types. Rather than implement a new switch, use its existing
    `only_works?` and `only_collections?` to filter by "generic type".
    Tom Johnson committed Jan 27, 2020
    Configuration menu
    Copy the full SHA
    48e7525 View commit details
    Browse the repository at this point in the history
  2. Test CollectionMemberSearchBuilder with a Valkyrie collection

    Adds tests for collection member search using Valkyrie `Hyrax::PcdmCollection`.
    Tom Johnson committed Jan 27, 2020
    Configuration menu
    Copy the full SHA
    f11054b View commit details
    Browse the repository at this point in the history
  3. Merge pull request #4217 from samvera/storage-adapter-conf

    Use the configured storage adapter
    jeremyf authored Jan 27, 2020
    Configuration menu
    Copy the full SHA
    bccd86e View commit details
    Browse the repository at this point in the history
  4. handle custom use types for files

    * adjust approach to converting files
    * add custom query find_file_metadata_by_use which is a valkyrie replacement for af object filter_files_by_type
    * add helper methods original_file, extracted_text, and thumbnail in FileSet resource
    
    Co-authored-by: cjcolvar <cjcolvard@indiana.edu>
    elrayle and cjcolvar committed Jan 27, 2020
    Configuration menu
    Copy the full SHA
    fc1f7dc View commit details
    Browse the repository at this point in the history
  5. add tests for custom query methods; refactor file use

    Refactor file use:
    * added methods to Hyrax::FileSet defining which Valkyrie::Vocab::PCDMUse URIs should be used in all cases for the primary uses of original_file, extracted_text, and thumbnail.
    * updated all uses of Valkyrie::Vocab::PCDMUse URIs for the primary uses to use the methods defined in FileSet
    
    Added tests for new methods
    elrayle committed Jan 27, 2020
    Configuration menu
    Copy the full SHA
    6c51756 View commit details
    Browse the repository at this point in the history
  6. refactor FileMetadata#use to #type

    Instead of putting the use in a use attributes, conversion from AF File copies over all types as they exist in AF into the types attribute of FileMetadata.  To find the use, the :include? method is used to check for a specific use in the types.  This avoids data duplication by not having both type and use.  And it avoids data integrity issues when we aren’t able to correctly identify the use out of the array of types.
    
    Co-authored-by: cjcolvar <cjcolvard@indiana.edu>
    elrayle and cjcolvar committed Jan 27, 2020
    Configuration menu
    Copy the full SHA
    fd2d89d View commit details
    Browse the repository at this point in the history
  7. Still spicy?

    Supersedes #4208.
    Tom Johnson authored and Tom Johnson committed Jan 27, 2020
    Configuration menu
    Copy the full SHA
    ba4d8f7 View commit details
    Browse the repository at this point in the history
  8. Add a vegan option

    cjcolvar authored Jan 27, 2020
    Configuration menu
    Copy the full SHA
    6e3d83a View commit details
    Browse the repository at this point in the history
  9. use guard clause; bump regen

    Julie Allinson committed Jan 27, 2020
    Configuration menu
    Copy the full SHA
    930b5fe View commit details
    Browse the repository at this point in the history
  10. mark versioning pending; return first for original_file, extracted_te…

    …xt, thumbnail
    
    2 changes here…
    
    ### Parts of FileSet
    
    AF behavior #original_file, #extracted_text, and #thumbnail each return a single PCDM::File,  So this now has the methods that get these relatiionships from FileMetadata to return `.first`
    
    ### Versioning
    
    Versioning was looking through to ActiveFedora through the original_file method defined in wings/hydra/pcdm/pcdm_valkyrization_behavior.  Since that was method was removed with the addition of Hyrax::FileSet#original_file, it was no longer getting the lookthrough behavior.  The tests related to versioning have been marked pending.  Issue #3923 addresses FileMetadata and versioning.
    elrayle committed Jan 27, 2020
    Configuration menu
    Copy the full SHA
    3cbcc99 View commit details
    Browse the repository at this point in the history
  11. Merge pull request #4219 from samvera/generate-indexer

    Add indexer to work_resource generator
    jeremyf authored Jan 27, 2020
    Configuration menu
    Copy the full SHA
    c7e8353 View commit details
    Browse the repository at this point in the history
  12. Merge pull request #4224 from samvera/cjcolvar-patch-1

    Plant-based spicy
    straleyb authored Jan 27, 2020
    Configuration menu
    Copy the full SHA
    4252be9 View commit details
    Browse the repository at this point in the history
  13. bump regen

    Julie Allinson committed Jan 27, 2020
    Configuration menu
    Copy the full SHA
    bdd75c1 View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2020

  1. prefer use of each_with_object

    Requested code change to use each_with_object was expanded to include all places where the pattern applied in the effected file.
    This also includes requested change to case statement.
    elrayle committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    408ae2e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c8e91a5 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    fc402da View commit details
    Browse the repository at this point in the history
  4. Merge pull request #4225 from samvera/wings/custom_file_assoc

    handle conversion of custom use types for files
    straleyb authored Jan 28, 2020
    Configuration menu
    Copy the full SHA
    093ac4a View commit details
    Browse the repository at this point in the history
  5. Setup RSpec metadata for selecting a Valkyrie adapter

    Use the test adapter in specs by doing:
    
    `context 'some context', valkyrie_adapter: :test_adapter do`
    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    eb5004b View commit details
    Browse the repository at this point in the history
  6. Create FileSetDescription to find characterization for FileSets

    Valkyrie-based FileSet objects don't have direct access to characterization
    information. In ActiveFedora, we used a `#characterization_proxy` to provide
    that information. Here, we provide a small service that can find the "primary"
    file (`FileMetadata`) for a given FileSet, and ask it about its description.
    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    f93029a View commit details
    Browse the repository at this point in the history
  7. Move PCDM Use URI constants to FileMetadata::Use

    The concept of "use" is more properly localized to `FileMetadata` than
    `FileSet`, since `FileMetadata` holds the actual use (`#type`) attribute.
    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    6edec00 View commit details
    Browse the repository at this point in the history
  8. Set a default FileMetadata#type

    Rather than using some branching logic in `FileMetadata.for`, take advantage of
    Valkyrie attributes' default values. When initializing a `FileMetadata`, assume
    it's an `ORIGINAL_FILE` unless a different use is passed.
    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    9989e38 View commit details
    Browse the repository at this point in the history
  9. Register the new :find_many_file_metadata_by_use query

    This query was added, but never registered. It needs to be added to the
    `.queries` method for the query handler in order to be picked up by most
    adapters.
    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    a65e59a View commit details
    Browse the repository at this point in the history
  10. Drop Hyrax::FileMetadata#used_for?

    This method is supplanted by a new custom query. Use that query instead
    whereever `used_for?` was called.
    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    bb79e44 View commit details
    Browse the repository at this point in the history
  11. Remove FileSet type convienence methods

    Since `#type` is located in `FileMetadata` and there's a convenient custom
    query, these methods mostly function to provide `.first`. My feeling is that
    this is a bit obfuscating; why not encourage queriers to handle the edge case
    where there are multiple matches in context appropriate ways instead? In some
    cases that might be `.first`, but not always.
    
    This also has the benefit of removing some business logic from the model.
    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    52d8338 View commit details
    Browse the repository at this point in the history
  12. Implement Valkyrie indexing for Collections

    Add an indexer for `Hyrax::PcdmCollection` and index `generic_type_sim`.
    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    e8e1069 View commit details
    Browse the repository at this point in the history
  13. Implement indexing for collection thumbnails

    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    e50f1de View commit details
    Browse the repository at this point in the history
  14. Update app/services/hyrax/thumbnail_path_service.rb

    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    e588855 View commit details
    Browse the repository at this point in the history
  15. Refactor FileActor handling of PCDM Use vocabulary

    For valkyrie normalization, take advantage for the new
    `Hyrax::FileMetadata::Use.uri_for` module method to avoid reproducing
    logic. When the argument is already a URI, just return it to avoid casting
    unnecessarily.
    
    In the ActiveFedora normalizer, `RDF::URI` already implements equality
    appropriately, so `#casecmp` isn't needed and we can flatten to a simple switch
    statement.
    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    2a9270a View commit details
    Browse the repository at this point in the history
  16. Correct the ThumbnailImage URI in the PCDM Use vocab

    The constant previously used an incorrect URI. `Thumbnail` should be
    `ThumbnailImage`.
    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    98298fe View commit details
    Browse the repository at this point in the history
  17. Introduce Hyrax.custom_queries

    Avoid typing `Hyrax.query_service.custom_queries` everywhere. Provide a method
    on the Hyrax level instead.
    Tom Johnson committed Jan 28, 2020
    Configuration menu
    Copy the full SHA
    a749f23 View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2020

  1. Merge pull request #4223 from samvera/valkyrie_collections_controller

    Implement Valkyrie indexing for Collections
    straleyb authored Jan 29, 2020
    Configuration menu
    Copy the full SHA
    23e3a25 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8c7cb5d View commit details
    Browse the repository at this point in the history
  3. updated lint regarding no params on method call

    straleyb authored and Tom Johnson committed Jan 29, 2020
    Configuration menu
    Copy the full SHA
    2d97b68 View commit details
    Browse the repository at this point in the history
  4. update solr indexing adapter to use valkyrie indexer

    straleyb authored and Tom Johnson committed Jan 29, 2020
    Configuration menu
    Copy the full SHA
    2214537 View commit details
    Browse the repository at this point in the history
  5. Remove attribute reader for resource_indexer

    straleyb authored and Tom Johnson committed Jan 29, 2020
    Configuration menu
    Copy the full SHA
    74428d6 View commit details
    Browse the repository at this point in the history
  6. Use resource_indexer instance variable

    straleyb authored and Tom Johnson committed Jan 29, 2020
    Configuration menu
    Copy the full SHA
    ab432d4 View commit details
    Browse the repository at this point in the history
  7. remove unnecessary instance variable

    Transitions from using the instance variable to using the already included method that returns the resource indexer.
    straleyb authored and Tom Johnson committed Jan 29, 2020
    Configuration menu
    Copy the full SHA
    fe7bd63 View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2020

  1. Removing Rails 6.0 deprecation warning

    Prior to this change, specs reported the following warning:
    
    ```console
    DEPRECATION WARNING: Delegating constraints to arel is deprecated and
    will be removed in Rails 6.0.
    ```
    
    I removed the deprecation warnings with this change (e.g. adding an
    explicit call to #arel).
    jeremyf committed Jan 30, 2020
    Configuration menu
    Copy the full SHA
    fa429e7 View commit details
    Browse the repository at this point in the history
  2. Merge pull request #4231 from samvera/removing-constraints-deprecatio…

    …n-warning
    
    Removing Rails 6.0 deprecation warning
    straleyb authored Jan 30, 2020
    Configuration menu
    Copy the full SHA
    226e010 View commit details
    Browse the repository at this point in the history
  3. Merge pull request #4215 from samvera/valk-char

    Create `FileSetDescription` to find characterization for FileSets
    elrayle authored Jan 30, 2020
    Configuration menu
    Copy the full SHA
    1d5c2f2 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c90215c View commit details
    Browse the repository at this point in the history
  5. Support custom URI use/type in FileSetDescription

    When constructing a description of a Valkyrie description, prefer to use URIs.
    Tom Johnson committed Jan 30, 2020
    Configuration menu
    Copy the full SHA
    8a597a2 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2020

  1. Merge pull request #4235 from samvera/use-uri-use

    Support custom URI use/type in `FileSetDescription`
    jeremyf authored Jan 31, 2020
    Configuration menu
    Copy the full SHA
    abe5383 View commit details
    Browse the repository at this point in the history
  2. Ensuring CSV usage has same namespace consideration

    This is a bit of a stab in the dark. However, we had a failed Ruby
    build with the following message:
    
    ```console
    1) Hyrax::FileSetCSVService when specifying terms and separator csv
         Failure/Error:
           ::CSV.generate do |csv|
             csv << terms.map do |term|
               values = file_set.send(term)
               values = values.respond_to?(:to_a) ? values.to_a : [values] # make sure we have an array
               values.join(multi_value_separator)
             end
           end
    
         NameError:
           uninitialized constant CSV
    ```
    
    What I suspect to be in play is that the corresponding spec does not
    similarly namespace the `CSV` constant. The production ruby file uses
    `::CSV` whereas the spec ruby file uses `CSV`. This commit seeks to use
    the same namespacing, hopefully reducing an intermittent error.
    
    See the following  CircleCI build for the failures:
    
    -  https://circleci.com/gh/samvera/hyrax/17063#tests/containers/3
    jeremyf committed Jan 31, 2020
    Configuration menu
    Copy the full SHA
    279ccb8 View commit details
    Browse the repository at this point in the history
  3. Merge pull request #4238 from samvera/using-same-namespace-convention…

    …s-for-csv
    
    Ensuring CSV usage has same namespace consideration
    jeremyf authored Jan 31, 2020
    Configuration menu
    Copy the full SHA
    da3b463 View commit details
    Browse the repository at this point in the history
  4. Merge pull request #4227 from samvera/inherit_permissions_job

    Valkyrize inherit_permissions_job
    jeremyf authored Jan 31, 2020
    Configuration menu
    Copy the full SHA
    6fc0b59 View commit details
    Browse the repository at this point in the history
  5. indexer setup, including fixes / tests for activefedoraconverter and …

    …modeltransformer
    
    monograph indexer spec
    Julie Allinson authored and straleyb committed Jan 31, 2020
    Configuration menu
    Copy the full SHA
    0d098b6 View commit details
    Browse the repository at this point in the history
  6. active_fedora_converter etc. changes

    Julie Allinson authored and straleyb committed Jan 31, 2020
    Configuration menu
    Copy the full SHA
    e122dd0 View commit details
    Browse the repository at this point in the history
  7. remove additional test from valkyrie_indexer_spec.rb - this has moved…

    … to generator PR
    Julie Allinson authored and straleyb committed Jan 31, 2020
    Configuration menu
    Copy the full SHA
    6565c88 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    12fa4d0 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    09fe003 View commit details
    Browse the repository at this point in the history