Skip to content

Commit

Permalink
Merge to master - Release 2.3.5 - Optimize order by and filters queri…
Browse files Browse the repository at this point in the history 
…es (#35)

* add tests for the new dsl to write property data types

* append the property :type values to the :enforce array

* update solution mapper to support multilingual

* update solution mapper to support multilingual

* fix typo ( name )

* add validators tests file

* add validator interface module

* implement data_type validator

* migrate existence validator to the new DSL

* migrate uniqueness validator to the new DSL

* implement object_type validator with the new DSL

* migrate range validator to the new DSL

* refactor the enforce module to use the new validators implementation

* force to regenerate the id when we update related attribute (named_with)

* require the validators implementation

* update existence validator to not accept empty to_s  objects

* update exist? test

* add symmetric validator tests for no_list and list cases

* implement symmetric validator

* move re used methods to the parent class

* update symmetric code and error message

* add distinct of validator tests

* implement distinct_of validator

* add superior_equal_to validator tests

* extract property method to ValidatorBase class

* implement superior_equal_to validator

* add inverse of validator tests

* implement inverse_of validator

* use the class method property in distinct of

* add proc validator tests

* add instance proc validators

* fix call_proc validator to test if the returned values are correct

* add model_with_yaml_scheme test

* implement YAMLScheme module

* use YAMLScheme module in Settings module

* use platform lang and code refacto

* filter by lang in properties

* do some refactoring

* add unmapped_get to goo resources

* update lang filter module to support requested_lang and portal_lang

* use the new lang filter module in the solution_mapper

* remove the usage of the old lang filter module in map_attributes

* add request language global variable

* fix datatype check for list values

* remove old unused test if clause

* for no unmapped values cast them to object before sending

* for resource unmapped_set merge new value if an array

* prevent add_object_to_model if no_lang and previous value exist

* move from the mapper  lang_filter related code to  lang_filter module

* move internal lang filter module methods to private section

* add request_store gem to save request language globally

* save requested  language in model_load options

* force requested_lang and portal_langs to be upcase and symbol

* change methodes/vars names

* get the last item in objects instead of passing the current object

* Revert "get the last item in objects instead of passing the current object"

This reverts commit 996922a.

* handle this case where values is nil in save_model_values

* handle the casf of nil values for  the SuperiorEqualTo validator

* add onUpdate callback tests

* implement enforce_callback to run an attribute callback

* move the attribute default callback to the save method

* implement onUpdate DSL in the ressource settings

* call to the attributes onUpdate callback in the save method

* in validators bring attribute if needed

* make superior_equal_to works for list attributes

* add email validator test

* implement email validator

* add filters patterns to select variables

* make make regex filter no-case sensitive

* if requested_lang = 'all' return all

* support select multilanguage

* show the values with their corresponding language

* use @attributes_to_translate

* change methode name

* remove platform languages

* add complex_order_by unit test

* refactor query_builder to extract internal_variables as instance variable

* update order_by to work for joined patterns (object attributes)

* downcase lang key

* Fix the issue of undefined 'id'  of the language filter module

* Show literal attribute if we requested all the languages

* Use portal language by default in the language filter module

* group unmapped properties by lang

* Feature: group unmapped properties by language  (#38)

* group unmapped properties by lang

* downcase language keys of unmapped properties

---------

Co-authored-by: Syphax bouazzouni <gs_bouazzouni@esi.dz>

* assert that pre in an array in get_value_object

* add label to attributes_to_translate

* update define_method

* update solution mapper

* update get_preload_value

* Feature: Support multi lingual -  add show_language argument to the attributes getters (#39)

* update define_method

* update solution mapper

* update get_preload_value

* fix save_model_values if unmmaped condition

* fix getters for list attributes  to not take only the first value

* remove the languages hash for the unmapped if not a mutli langual asked

* move some language helper from the mapper to the lang_filter module

* move @requested_lang variable from the mapper to the lang_filter module

* remove no more used @attributes_to_translate variable in lang_filter

* fix save_model_values method to not save RDF:Literal object but a string

* remove  not used method in lang filter module

* refecator and rename some methods of the lang_filter module

* use the new name of the lang filter methods in the solution mapper

* replace the getters argument to show languages from :show_all_languages to :show_languages: true

* catch transform_values of unmapped if it is nil

* change the getters show_all_languages argument from to include_languages

* make the map_attributes handle the option showing all the languages

* fix order by an attribute that is already filtered

* don't add the filtered variables to the select clause of the query

* add filters patterns to select variables

* fix pagination with order_by with filter that returns empty pages for 4store

* include the in the select variables filtered variables

* optimize pagination query by not re-doing the filters and order in the include query

---------

Co-authored-by: HADDAD Zineddine <hz_haddad@esi.dz>
  • Loading branch information
@syphax-bouazzouni @haddadzineddine
syphax-bouazzouni and haddadzineddine authored Dec 5, 2023
1 parent d4d9417 commit 74ea47d
Show file tree
Hide file tree
Showing 12 changed files with 569 additions and 488 deletions.
2 changes: 1 addition & 1 deletion lib/goo/base/resource.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ def unmmaped_to_array
end

def unmapped(*args)
@unmapped.transform_values do |language_values|
@unmapped&.transform_values do |language_values|
self.class.not_show_all_languages?(language_values, args) ? language_values.values.flatten: language_values
end
end
Expand Down
5 changes: 5 additions & 0 deletions lib/goo/base/where.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,11 @@ def process_query_intl(count=false)

options_load[:ids] = ids if ids
models_by_id = {}
if @page_i && (options_load[:models].length > 0)
options_load.delete(:filters)
options_load.delete(:order_by)
end

if (@page_i && options_load[:models].length > 0) ||
(!@page_i && (@count.nil? || @count > 0))
models_by_id = Goo::SPARQL::Queries.model_load(options_load)
Expand Down
4 changes: 2 additions & 2 deletions lib/goo/sparql/mixins/solution_lang_filter.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ def fill_models_with_all_languages(models_by_id)
end


def set_model_value(model, predicate, values)
set_value(model, predicate, values) do
def set_model_value(model, predicate, values, value)
set_value(model, predicate, value) do
model.send("#{predicate}=", values, on_load: true)
end
end
Expand Down
87 changes: 55 additions & 32 deletions lib/goo/sparql/query_builder.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,21 +25,18 @@ def build_select_query(ids, variables, graphs, patterns,
query_options, properties_to_include)

patterns = graph_match(@collection, @graph_match, graphs, @klass, patterns, query_options, @unions)

aggregate_projections, aggregate_vars,
variables, optional_patterns = get_aggregate_vars(@aggregate, @collection, graphs,
@klass, @unions, variables)

@order_by, variables, optional_patterns = init_order_by(@count, @klass, @order_by, optional_patterns, variables,patterns, query_options, graphs)
variables, patterns = add_some_type_to_id(patterns, query_options, variables)

aggregate_projections, aggregate_vars, variables, optional_patterns = get_aggregate_vars(@aggregate, @collection, graphs, @klass, @unions, variables)
query_filter_str, patterns, optional_patterns, filter_variables =
filter_query_strings(@collection, graphs, @klass, optional_patterns, patterns, @query_filters)
@order_by, variables, optional_patterns = init_order_by(@count, @klass, @order_by, optional_patterns, variables,patterns, query_options, graphs)
order_by_str, order_variables = order_by_string


variables = [] if @count
variables.delete :some_type

select_distinct(variables, aggregate_projections, filter_variables)
select_distinct(variables, aggregate_projections, filter_variables, order_variables)
.from(graphs)
.where(patterns)
.union_bind_in_where(properties_to_include)
Expand All @@ -55,7 +52,10 @@ def build_select_query(ids, variables, graphs, patterns,
@query.union(*@unions) unless @unions.empty?

ids_filter(ids) if ids
order_by if @order_by


@query.order_by(*order_by_str) if @order_by


put_query_aggregate_vars(aggregate_vars) if aggregate_vars
count if @count
Expand Down Expand Up @@ -117,16 +117,17 @@ def put_query_aggregate_vars(aggregate_vars)
self
end

def order_by
order_by_str = @order_by.map do |attr, order|
def order_by_string
order_variables = []
order_str = @order_by&.map do |attr, order|
if order.is_a?(Hash)
sub_attr, order = order.first
attr = @internal_variables_map[sub_attr]
attr = @internal_variables_map.select{ |internal_var, attr_var| attr_var.eql?({attr => sub_attr}) || attr_var.eql?(sub_attr)}.keys.last
end
order_variables << attr
"#{order.to_s.upcase}(?#{attr})"
end
@query.order_by(*order_by_str)
self
[order_str,order_variables]
end

def from(graphs)
Expand All @@ -141,10 +142,11 @@ def from(graphs)
self
end

def select_distinct(variables, aggregate_projections, filter_variables)
def select_distinct(variables, aggregate_variables, filter_variables, order_variables)
select_vars = variables.dup
reject_aggregations_from_vars(select_vars, aggregate_projections) if aggregate_projections
select_vars = (select_vars + filter_variables).uniq if @page # Fix for 4store pagination with a filter
reject_aggregations_from_vars(select_vars, aggregate_variables) if aggregate_variables
# Fix for 4store pagination with a filter https://github.com/ontoportal-lirmm/ontologies_api/issues/25
select_vars = (select_vars + filter_variables + order_variables).uniq if @page
@query = @query.select(*select_vars).distinct(true)
self
end
Expand All @@ -165,23 +167,24 @@ def ids_filter(ids)
def patterns_for_match(klass, attr, value, graphs, patterns, unions,
internal_variables, subject = :id, in_union = false,
in_aggregate = false, query_options = {}, collection = nil)
new_internal_var = value
if value.respond_to?(:each) || value.instance_of?(Symbol)
next_pattern = value.instance_of?(Array) ? value.first : value

#for filters
next_pattern = { next_pattern => [] } if next_pattern.instance_of?(Symbol)

value = "internal_join_var_#{internal_variables.length}".to_sym
new_internal_var = "internal_join_var_#{internal_variables.length}".to_sym
if in_aggregate
value = "#{attr}_agg_#{in_aggregate}".to_sym
new_internal_var = "#{attr}_agg_#{in_aggregate}".to_sym
end
internal_variables << value
@internal_variables_map[attr] = value
internal_variables << new_internal_var
@internal_variables_map[new_internal_var] = value.empty? ? attr : {attr => value}
end

add_rules(attr, klass, query_options)
graph, pattern =
query_pattern(klass, attr, value: value, subject: subject, collection: collection)
query_pattern(klass, attr, value: new_internal_var, subject: subject, collection: collection)
if pattern
if !in_union
patterns << pattern
Expand All @@ -194,7 +197,7 @@ def patterns_for_match(klass, attr, value, graphs, patterns, unions,
range = klass.range(attr)
next_pattern.each do |next_attr, next_value|
patterns_for_match(range, next_attr, next_value, graphs,
patterns, unions, internal_variables, subject = value,
patterns, unions, internal_variables, subject = new_internal_var,
in_union, in_aggregate, collection = collection)
end
end
Expand Down Expand Up @@ -270,15 +273,35 @@ def init_order_by(count, klass, order_by, optional_patterns, variables, patterns
order_by.each do |attr, direction|

if direction.is_a?(Hash)
# TODO this part can be improved/refactored, the complexity was added because order by don't work
# if the pattern is in the mandatory ones (variable `patterns`)
# and optional (variable `optional_patterns`) at the same time
sub_attr, direction = direction.first
graph_match_iteration = Goo::Base::PatternIteration.new(Goo::Base::Pattern.new({attr => [sub_attr]}))
old_internal = internal_variables.dup
old_patterns = optional_patterns.dup

walk_pattern(klass, graph_match_iteration, graphs, optional_patterns, @unions, internal_variables, in_aggregate = false, query_options, @collection)
variables << (internal_variables - old_internal).last
new_variables = (internal_variables - old_internal)
internal_variables.delete(new_variables)
new_patterns = optional_patterns - old_patterns
already_existent_pattern = patterns.select{|x| x[1].eql?(new_patterns.last[1])}.first

if already_existent_pattern
already_existent_variable = already_existent_pattern[2]
optional_patterns = old_patterns
key = @internal_variables_map.select{|key, value| key.eql?(new_variables.last)}.keys.first
@internal_variables_map[key] = (already_existent_variable || new_variables.last) if key

#variables << already_existent_variable
else
#variables << new_variables.last
end

else
quad = query_pattern(klass, attr)
optional_patterns << quad[1]
variables << attr
#variables << attr
end

#patterns << quad[1]
Expand Down Expand Up @@ -325,7 +348,12 @@ def query_filter_sparql(klass, filter, filter_patterns, filter_graphs,
end
filter_var = inspected_patterns[filter_pattern_match]

unless filter_operation.value.instance_of?(Goo::Filter)
if filter_operation.value.instance_of?(Goo::Filter)
filter_operations << "#{sparql_op_string(filter_operation.operator)}"
query_filter_sparql(klass, filter_operation.value, filter_patterns,
filter_graphs, filter_operations,
internal_variables, inspected_patterns, collection)
else
case filter_operation.operator
when :unbound
filter_operations << "!BOUND(?#{filter_var.to_s})"
Expand All @@ -349,11 +377,6 @@ def query_filter_sparql(klass, filter, filter_patterns, filter_graphs,
" #{value.to_ntriples}")
end

else
filter_operations << "#{sparql_op_string(filter_operation.operator)}"
query_filter_sparql(klass, filter_operation.value, filter_patterns,
filter_graphs, filter_operations,
internal_variables, inspected_patterns, collection)
end
end
end
Expand Down Expand Up @@ -399,7 +422,7 @@ def add_some_type_to_id(patterns, query_options, variables)
end

def internal_variables
@internal_variables_map.values
@internal_variables_map.keys
end
end
end
Expand Down
Loading

0 comments on commit 74ea47d

Please sign in to comment.