Merge to master: Release 2.4.0 - Multi-backend stores integrations, R…

…DF 3.0 and SOLR API (#58)

* Feature: Add  Virtuso, Allegrograph and Graphdb integration to GOO (#48)

* simplify the test configuration init

* add docker based tests rake task to run test against 4s, ag, gb, vo

* remove faraday gem usage

* update test CI to test against all the supported backends with diffirent slice sizes

* add high level helper to to know which backend we are currently using

* extract sparql processor module from where module

* handle language_match? value to upcase by default

* add support for virtuoso and graphdb sparql client

* replace delete sparql query by delete graph in the model complex test

* add some new edge cases tests t o test_where.rb and test_schemaless

* make test_chunks_write.rb tests support multiple backends

* replace native insert_data with execute_append_request in model save

* remove add_rules as it seems to no more be used

* move expand_equivalent_predicates from loader to builder module

* build two diffirent queries depending on which backend used

* update mapper to handle the two different queries depending on the backend used

* simplify the loader code, by removing inferable variables

* refactor and simplify map_attributes method

* fix test chunks write concenrency issues

* Refactor: clean model settings module code (#52)

* remove old file no more used

* extract attribute settings module from the model settings module

* remove the inmutable feature as deprecated and not used

* rename callbacks method names

* Feature: Add after_save and after_destroy hooks to models  (#53)

* remove old file no more used

* extract attribute settings module from the model settings module

* remove the inmutable feature as deprecated and not used

* rename callbacks method names

* add hooks module

* Feature: update rdf gem to latest version  (#56)

* un pin rdf version, to use the latest and add rdf vocab and xml

* update URI class monkey patch because Addressable does no more exist

* RDF::SKOS is replaced with RDF::Vocab::SKOS in the latest version of RDF

* pin rdf version to 3.2.11 the latest version that support ruby 2.7

* monkey path Literal::DateTime format to be supported by 4store

* remove  addressable dependency

* Fix: saving a model removing unmodified attributes after consecutive save

* Fix: enforce to use str() when doing a filter with a string value  (#57)

* enforce to use str() when doing a filter with a string

* update agraph version to 8.1.0

* Fix: monkey path RDF to not remove xsd:string by default

* Feature: Enhance SOLR integration and add a Schema API (#54)

* add an abstraction to SOLR integeration and add Schema API

* add SOLR  Schema API tests

* update SOLR backend configuration and init

* use the new Solr connector in the model search interface

* update search test to cover the new automatic indexing and unindexing

* handle the solr container initialization when running docker for tests

* add  omit_norms options for SolrSchemaGenerator

* fix solr schema initial dynamic fields declaration and replace the usage of   mapping-ISOLatin1Accent

* delay the schema generation to after model declarations or in demand

* add solr edismax fitlers tests

* fix indexBatch and unindexBatch tests

* add security checks to the index and unindex functions

* change dynamic fields names to have less code migration

* update clear_all_schema to remove all copy and normal fields

* add an option to force solr initialization if wanted

* handle indexing embed objects of a model

* add index update option

* fix clear all schema to just remove all the fields and recreate them

* add index_enabled?  helper for models

* perform a status test  when initializing the solr connector

* extract init_search_connection function from init_search_connections

* fix typo in indexOptimize call

* add solr search using  HTTP post instead of GET for large queries

* make indexed resource_id case insensitive (#59)

* Fix: Invalidating cache after insertion of a new  element (#60)

* create a test to reproduce the cache invalidate on insert bug

* use again insert_data instead of execute_append_request because the first invalidate the cache

* update sparql client to  version 3.2.0

* handle the case virtuoso insert data  bug

* use development branch of sparql-client

* fix search resource_id case insensitive by using string_ci instead

.github/workflows/ruby-unit-test.yml

@@ -12,21 +12,26 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        ruby-version: ['2.7']
+        goo-slice: [ '20', '100', '500' ]
+        ruby-version: [ '2.7', '3.0' ]
+        triplestore: [ 'fs', 'ag', 'vo', 'gb' ]
 
     steps:
     - uses: actions/checkout@v3
     - name: Install Dependencies
-      run: sudo apt-get -y install raptor2-utils
+      run: sudo apt-get update && sudo apt-get -y install raptor2-utils
     - name: Set up Ruby
       uses: ruby/setup-ruby@v1
       with:
         ruby-version: ${{ matrix.ruby-version }}
         bundler-cache: true # runs 'bundle install' and caches installed gems automatically
-    - name: Start backend services via docker-compose
-      run: docker compose up -d
+    - name: Add config file
+      # tempoaray workaround for the config.rb file requirement
+      run:  echo 'Goo.config do |config| end' > config/config.rb
+    - name: List directory contents
+      run: ls -R ./test/data
     - name: Run tests
-      run: bundle exec rake test
+      run: GOO_SLICES=${{ matrix.goo-slice }} bundle exec rake test:docker:${{ matrix.triplestore }} TESTOPTS="-v"
     - name: Upload coverage to Codecov
       uses: codecov/codecov-action@v3
 

.gitignore

@@ -25,3 +25,5 @@ doc/
 
 .idea/*
 projectFilesBackup/*
+
+config/config.rb

.ruby-version

@@ -0,0 +1 @@
+2.7.8

Gemfile

@@ -4,7 +4,6 @@ gemspec
 
 gem "activesupport"
 gem "cube-ruby", require: "cube"
-gem "faraday", '~> 1.9'
 gem "rake"
 gem "uuid"
 gem "request_store"
@@ -23,4 +22,5 @@ group :profiling do
   gem "thin"
 end
 
-gem 'sparql-client', github: 'ontoportal-lirmm/sparql-client', branch: 'master'
+gem 'sparql-client', github: 'ontoportal-lirmm/sparql-client', branch: 'development'
+gem 'faraday', '2.7.11' #unpin if we no more support ruby 2.7

Gemfile.lock

@@ -1,20 +1,22 @@
 GIT
   remote: https://github.com/ontoportal-lirmm/sparql-client.git
-  revision: aed51baf4106fd0f3d0e3f9238f0aad9406aa3f0
-  branch: master
+  revision: c96da3ad479724a31ccd6217ab9939dddfaca40e
+  branch: development
   specs:
-    sparql-client (1.0.1)
-      json_pure (>= 1.4)
-      net-http-persistent (= 2.9.4)
-      rdf (>= 1.0)
+    sparql-client (3.2.2)
+      net-http-persistent (~> 4.0, >= 4.0.2)
+      rdf (~> 3.2, >= 3.2.11)
 
 PATH
   remote: .
   specs:
     goo (0.0.2)
       addressable (~> 2.8)
       pry
-      rdf (= 1.0.8)
+      rdf (= 3.2.11)
+      rdf-raptor
+      rdf-rdfxml
+      rdf-vocab
       redis
       rest-client
       rsolr
@@ -30,87 +32,85 @@ GEM
       multi_json (~> 1.3)
       thread_safe (~> 0.1)
       tzinfo (~> 0.3.37)
-    addressable (2.8.1)
+    addressable (2.8.6)
       public_suffix (>= 2.0.2, < 6.0)
+    base64 (0.2.0)
     builder (3.2.4)
     coderay (1.1.3)
-    concurrent-ruby (1.2.2)
-    connection_pool (2.3.0)
+    concurrent-ruby (1.2.3)
+    connection_pool (2.4.1)
     cube-ruby (0.0.3)
     daemons (1.4.1)
     docile (1.4.0)
-    domain_name (0.5.20190701)
-      unf (>= 0.0.5, < 1.0.0)
+    domain_name (0.6.20240107)
     eventmachine (1.2.7)
-    faraday (1.10.3)
-      faraday-em_http (~> 1.0)
-      faraday-em_synchrony (~> 1.0)
-      faraday-excon (~> 1.1)
-      faraday-httpclient (~> 1.0)
-      faraday-multipart (~> 1.0)
-      faraday-net_http (~> 1.0)
-      faraday-net_http_persistent (~> 1.0)
-      faraday-patron (~> 1.0)
-      faraday-rack (~> 1.0)
-      faraday-retry (~> 1.0)
+    faraday (2.7.11)
+      base64
+      faraday-net_http (>= 2.0, < 3.1)
       ruby2_keywords (>= 0.0.4)
-    faraday-em_http (1.0.0)
-    faraday-em_synchrony (1.0.0)
-    faraday-excon (1.1.0)
-    faraday-httpclient (1.0.1)
-    faraday-multipart (1.0.4)
-      multipart-post (~> 2)
-    faraday-net_http (1.0.1)
-    faraday-net_http_persistent (1.2.0)
-    faraday-patron (1.0.0)
-    faraday-rack (1.0.0)
-    faraday-retry (1.0.3)
+    faraday-net_http (3.0.2)
+    ffi (1.16.3)
+    htmlentities (4.3.4)
     http-accept (1.7.0)
     http-cookie (1.0.5)
       domain_name (~> 0.5)
     i18n (0.9.5)
       concurrent-ruby (~> 1.0)
-    json_pure (2.6.3)
+    link_header (0.0.8)
     macaddr (1.7.2)
       systemu (~> 2.6.5)
-    method_source (1.0.0)
-    mime-types (3.4.1)
+    method_source (1.1.0)
+    mime-types (3.5.2)
       mime-types-data (~> 3.2015)
-    mime-types-data (3.2023.0218.1)
+    mime-types-data (3.2024.0305)
     minitest (4.7.5)
     multi_json (1.15.0)
-    multipart-post (2.3.0)
     mustermann (3.0.0)
       ruby2_keywords (~> 0.0.1)
-    net-http-persistent (2.9.4)
+    net-http-persistent (4.0.2)
+      connection_pool (~> 2.2)
     netrc (0.11.0)
     pry (0.14.2)
       coderay (~> 1.1)
       method_source (~> 1.0)
-    public_suffix (5.0.1)
-    rack (2.2.6.3)
+    public_suffix (5.0.5)
+    rack (2.2.9)
     rack-accept (0.4.5)
       rack (>= 0.4)
     rack-post-body-to-params (0.1.8)
       activesupport (>= 2.3)
-    rack-protection (3.0.5)
-      rack
-    rake (13.0.6)
-    rdf (1.0.8)
-      addressable (>= 2.2)
-    redis (5.0.6)
-      redis-client (>= 0.9.0)
-    redis-client (0.13.0)
+    rack-protection (3.2.0)
+      base64 (>= 0.1.0)
+      rack (~> 2.2, >= 2.2.4)
+    rake (13.2.1)
+    rdf (3.2.11)
+      link_header (~> 0.0, >= 0.0.8)
+    rdf-raptor (3.2.0)
+      ffi (~> 1.15)
+      rdf (~> 3.2)
+    rdf-rdfxml (3.2.2)
+      builder (~> 3.2)
+      htmlentities (~> 4.3)
+      rdf (~> 3.2)
+      rdf-xsd (~> 3.2)
+    rdf-vocab (3.2.7)
+      rdf (~> 3.2, >= 3.2.4)
+    rdf-xsd (3.2.1)
+      rdf (~> 3.2)
+      rexml (~> 3.2)
+    redis (5.2.0)
+      redis-client (>= 0.22.0)
+    redis-client (0.22.1)
       connection_pool
-    request_store (1.5.1)
+    request_store (1.6.0)
       rack (>= 1.4)
     rest-client (2.1.0)
       http-accept (>= 1.7.0, < 2.0)
       http-cookie (>= 1.0.2, < 2.0)
       mime-types (>= 1.16, < 4.0)
       netrc (~> 0.8)
-    rexml (3.2.5)
-    rsolr (2.5.0)
+    rexml (3.2.6)
+    rsolr (2.6.0)
       builder (>= 2.1.2)
       faraday (>= 0.9, < 3, != 2.0.0)
     ruby2_keywords (0.0.5)
@@ -123,34 +123,30 @@ GEM
       simplecov (~> 0.19)
     simplecov-html (0.12.3)
     simplecov_json_formatter (0.1.4)
-    sinatra (3.0.5)
+    sinatra (3.2.0)
       mustermann (~> 3.0)
       rack (~> 2.2, >= 2.2.4)
-      rack-protection (= 3.0.5)
+      rack-protection (= 3.2.0)
       tilt (~> 2.0)
     systemu (2.6.5)
-    thin (1.8.1)
+    thin (1.8.2)
       daemons (~> 1.0, >= 1.0.9)
       eventmachine (~> 1.0, >= 1.0.4)
       rack (>= 1, < 3)
     thread_safe (0.3.6)
-    tilt (2.1.0)
-    tzinfo (0.3.61)
-    unf (0.1.4)
-      unf_ext
-    unf_ext (0.0.8.2)
+    tilt (2.3.0)
+    tzinfo (0.3.62)
     uuid (2.3.9)
       macaddr (~> 1.0)
 
 PLATFORMS
-  ruby
-  x86_64-darwin-16
+  x86_64-darwin-23
   x86_64-linux
 
 DEPENDENCIES
   activesupport
   cube-ruby
-  faraday (~> 1.9)
+  faraday (= 2.7.11)
   goo!
   minitest (< 5.0)
   pry
@@ -166,4 +162,4 @@ DEPENDENCIES
   uuid
 
 BUNDLED WITH
-   2.3.22
+   2.4.22

Rakefile

@@ -50,12 +50,6 @@ Rake::TestTask.new do |t|
   t.warning = false
 end
 
-Rake::TestTask.new do |t|
-  t.name = "test:inmutable"
-  t.test_files = FileList['test/test_inmutable.rb']
-  t.warning = false
-end
-
 Rake::TestTask.new do |t|
   t.name = "test:inverse"
   t.test_files = FileList['test/test_inverse.rb']
@@ -107,7 +101,6 @@ end
 desc "Console for working with data"
 task :console do
   require_relative "test/test_case"
-  GooTest.configure_goo
   binding.pry
 end
 

config/config.rb.sample

@@ -0,0 +1,23 @@
+Goo.config do |config|
+  # 4store
+  config.goo_backend_name    = '4store'
+  config.goo_port            = 8080
+  config.goo_host            = 'localhost'
+  config.goo_path_query      = '/sparql/'
+  config.goo_path_data       = '/data/'
+  config.goo_path_update     = '/update/'
+
+  # AllegroGraph
+  # config.goo_backend_name  = 'AG'
+  # config.goo_port          = 10035
+  # config.goo_host          = 'localhost'
+  # config.goo_path_query    = "/repositories/ontoportal"
+  # config.goo_path_data     = "/repositories/ontoportal/statements/"
+  # config.goo_path_update   = "/repositories/ontoportal/statements/"
+
+  config.search_server_url   = 'http://localhost:8983/solr/term_search_core1'
+  config.redis_host          = 'localhost'
+  config.redis_port          = 6379
+  config.bioportal_namespace = 'http://data.bioontology.org/'
+  config.queries_debug       = false
+end