Skip to content

Feature/custom translation of scripts #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Feature/custom translation of scripts #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fba61e8
working on script custom translation
Startouf Oct 23, 2018
117df0a
Ability to translate custom scripts and add precision to scores
Startouf Oct 23, 2018
b14895d
Add the YML example for named scripts ^^"
Startouf Oct 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 50 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,30 +29,74 @@ $ gem install elasticsearch-explain-response

## Usage

### Summarize the explanation in one line
### Initialize the custom explainer

```ruby
require 'elasticsearch'
client = Elasticsearch::Client.new
result = client.explain index: "megacorp", type: "employee", id: "1", q: "last_name:Smith"
puts Elasticsearch::API::Response::ExplainResponse.new(result["explanation"]).render_in_line
custom_explainer = Elasticsearch::API::Response::ExplainResponse.new(result["explanation"])
```

### Summarize the explanation in one line

```ruby
custom_explainer.render_in_line
#=>
1.0 = (1.0(termFreq=1.0)) x 1.0(idf(2/3)) x 1.0(fieldNorm)
```

### Summarize the explanation in lines

```ruby
require 'elasticsearch'
client = Elasticsearch::Client.new
result = client.explain index: "megacorp", type: "employee", id: "1", q: "last_name:Smith"
puts Elasticsearch::API::Response::ExplainResponse.new(result["explanation"]).render
custom_explainer.render
#=>
1.0 = 1.0(fieldWeight)
1.0 = 1.0(tf(1.0)) x 1.0(idf(2/3)) x 1.0(fieldNorm)
1.0 = 1.0(termFreq=1.0)
```

### Customize your rendering

#### Translate your custom scripts

```ruby
custom_painless_script = <<~PAINLESS.delete("\n")
(
doc['response_rate'].value > 0.5 &&
doc['unavailable_until'].empty
) ? 1 : 0)
PAINLESS

custom_translator => (lambda do |value|
if value == 1
'1/1 Is available and good chance of reply'
else
'0/1 Not available or low chance of reply'
end
end)

script_translation_map = {
custom_painless_script => custom_translator
}

custom_explainer = Elasticsearch::API::Response::ExplainResponse.new(
result["explanation"],
script_translation_map: script_translation_map
)
custom_explainer.render
#=>
0.1 = 0.1(script(Custom script:0/1 Not available or low chance of reply))

```

#### Change basic formatting

```ruby
custom_explainer.render(
precision: 6 # 6 decimals
)

## Contributing

1. Fork it ( https://github.com/tomoya55/elasticsearch-explain-response/fork )
Expand Down
3 changes: 2 additions & 1 deletion lib/elasticsearch/api/response/explain_node.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
require "elasticsearch/api/response/renderable"
require 'forwardable'

module Elasticsearch
module API
module Response
class ExplainNode
include Renderable
extend Forwardable
extend ::Forwardable
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note in Ruby 2.5 omitting the :: will perform a same folder constant lookup. + we need to require 'forwardable' explicitely. Should be retro-compatible


attr_reader :score, :description, :details, :level
attr_accessor :children
Expand Down
241 changes: 134 additions & 107 deletions lib/elasticsearch/api/response/explain_parser.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,17 @@ module Response
class ExplainParser
include Helpers::StringHelper

# @param [Hash<String, Proc<Float>>] script_translation_map: {}
# @example {
# "doc['has_custom_boost'].value":
# ->(value) { value == 1 ? 'Has a custom boost' : 'Does not have a custom boost'
# "doc['response_rate'].value >= 0.5 ? 1 : 0":
# ->(value) { value == 1 ? 'Has a good response rate' : 'Has a bad response rate <50%'
# }
def initialize(script_translation_map: {})
@script_translation_map = script_translation_map
end

def parse(explain_tree)
root = create_node(explain_tree, level: 0)
parse_details(root)
Expand All @@ -14,122 +25,138 @@ def parse(explain_tree)

private

def create_node(detail, level:)
ExplainNode.new(
score: detail["value"] || 0.0,
description: parse_description(detail["description"]),
details: detail["details"] || [],
level: level
)
end
def create_node(detail, level:)
ExplainNode.new(
score: detail["value"] || 0.0,
description: parse_description(detail["description"], node_value: detail["value"] || 0.0),
details: detail["details"] || [],
level: level
)
end

def parse_details(node)
node.details.each do |detail|
child = create_node(detail, level: node.level.succ)
node.children << child
parse_details(child)
end
def parse_details(node)
node.details.each do |detail|
child = create_node(detail, level: node.level.succ)
node.children << child
parse_details(child)
end
end

def parse_description(description)
case description
when /\Aweight\((\w+)\:(\w+)\s+in\s+\d+\)\s+\[\w+\]\, result of\:\z/
type = "weight"
operation = "weight"
operator = "x"
field = $1
value = $2
when /\Aidf\(docFreq\=(\d+)\, maxDocs\=(\d+)\)\z/
type = "idf"
operation = "idf(#{$1}/#{$2})"
when /\Atf\(freq\=([\d.]+)\)\, with freq of\:\z/
type = "tf"
operation = "tf(#{$1})"
when /\Ascore\(doc\=\d+\,freq=[\d\.]+\)\, product of\:\z/
type = "score"
operation = "score"
operator = "x"
when /\Amatch filter\: (?:cache\()?(?:(?<op>[\w]+)\()*(?<c>.+)\)*\z/
type = "match"
operation = "match"
operation += ".#{$~[:op]}" if $~[:op] && !%w[QueryWrapperFilter].include?($~[:op])
content = $~[:c]
content = content[0..-2] if content.end_with?(')')
hash = tokenize_contents(content)
field = hash.keys.join(", ")
value = hash.values.join(", ")
when /\AFunction for field ([\w\_]+)\:\z/
type = "func"
operation = "func"
field = $1
when /\AqueryWeight\, product of\:\z/
type = "queryWeight"
operation = "queryWeight"
operator = "x"
when /\AfieldWeight in \d+\, product of\:\z/
type = "fieldWeight"
operation = "fieldWeight"
operator = "x"
when /\AqueryNorm/
type = "queryNorm"
operation = "queryNorm"
when /\Afunction score\, product of\:\z/,
/\Afunction score\, score mode \[multiply\]\z/
type = "func score"
operator = "x"
when /\Afunction score\, score mode \[sum\]\z/
type = "func score"
operator = "+"
when /\Ascript score function\, computed with script:\"(?<s>.+)\"\s*(?:and parameters:\s*(?<p>.+))?/m
def parse_description(description, node_value:)
case description
when /\Aweight\((\w+)\:(\w+)\s+in\s+\d+\)\s+\[\w+\]\, result of\:\z/
type = "weight"
operation = "weight"
operator = "x"
field = $1
value = $2
when /\Aidf\(docFreq\=(\d+)\, maxDocs\=(\d+)\)\z/
type = "idf"
operation = "idf(#{$1}/#{$2})"
when /\Atf\(freq\=([\d.]+)\)\, with freq of\:\z/
type = "tf"
operation = "tf(#{$1})"
when /\Ascore\(doc\=\d+\,freq=[\d\.]+\)\, product of\:\z/
type = "score"
operation = "score"
operator = "x"
when /\Amatch filter\: (?:cache\()?(?:(?<op>[\w]+)\()*(?<c>.+)\)*\z/
type = "match"
operation = "match"
operation += ".#{$~[:op]}" if $~[:op] && !%w[QueryWrapperFilter].include?($~[:op])
content = $~[:c]
content = content[0..-2] if content.end_with?(')')
hash = tokenize_contents(content)
field = hash.keys.join(", ")
value = hash.values.join(", ")
when /\AFunction for field ([\w\_]+)\:\z/
type = "func"
operation = "func"
field = $1
when /\AqueryWeight\, product of\:\z/
type = "queryWeight"
operation = "queryWeight"
operator = "x"
when /\AfieldWeight in \d+\, product of\:\z/
type = "fieldWeight"
operation = "fieldWeight"
operator = "x"
when /\AqueryNorm/
type = "queryNorm"
operation = "queryNorm"
when /\Afunction score\, product of\:\z/,
/\Afunction score\, score mode \[multiply\]\z/
type = "func score"
operator = "x"
when /\Afunction score\, score mode \[sum\]\z/
type = "func score"
operator = "+"
when /\Ascript score function\, computed with script:\"(?<s>.+)\"\s*(?:and parameters:\s*(?<p>.+))?/m
operation = "script"
script, param = $~[:s], $~[:p]
param.gsub!("\n", '') if param
script = script.gsub("\n", '')
if (script_translator = translator_of_custom_script_function(script))
type = "translated_script"
field = 'Custom script'
value = script_translator.call(node_value)
else
type = "script"
operation = "script"
script, param = $~[:s], $~[:p]
script = script.gsub("\n", '')
script = "\"#{script}\""
param.gsub!("\n", '') if param
field = script.scan(/doc\[\'([\w\.]+)\'\]/).flatten.uniq.compact.join(" ")
value = [script, param].join(" ")
when /\AConstantScore\(.+\), product of\:\z/
type = "constant"
operation = "constant"
when /\Aconstant score/
type = "constant"
operation = "constant"
when "static boost factor", "boostFactor"
type = "boost"
operation = "boost"
when /product\sof\:?/, "[multiply]"
type = "product"
operation = "product"
operator = "x"
when "Math.min of"
type = "min"
operator = "min"
when "Math.max of"
type = "max"
operator = "max"
when /sum of\:?/
type = "sum"
operator = "+"
when "maxBoost"
type = "maxBoost"
when /_score\:\s*/
type = "score"
operation = "score"
else
type = description
operation = description
end

Description.new(
raw: description,
type: type,
operator: operator,
operation: operation,
field: field,
value: value,
)
when /\AConstantScore\(.+\), product of\:\z/
type = "constant"
operation = "constant"
when /\Aconstant score/
type = "constant"
operation = "constant"
when "static boost factor", "boostFactor"
type = "boost"
operation = "boost"
when /product\sof\:?/, "[multiply]"
type = "product"
operation = "product"
operator = "x"
when "Math.min of"
type = "min"
operator = "min"
when "Math.max of"
type = "max"
operator = "max"
when /sum of\:?/
type = "sum"
operator = "+"
when "maxBoost"
type = "maxBoost"
when /_score\:\s*/
type = "score"
operation = "score"
else
type = description
operation = description
end

Description.new(
raw: description,
type: type,
operator: operator,
operation: operation,
field: field,
value: value,
)
end

# @param [String] ES script
#
# @return [Lambda]
# @yieldparam [Float] Associated ES score
#
def translator_of_custom_script_function(script)
code = script[/.*Code\='([^\,]*)\'/,1]
@script_translation_map[code]
end
end
end
end
Expand Down
5 changes: 3 additions & 2 deletions lib/elasticsearch/api/response/explain_response.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ def result_as_hash(result, options = {})
def initialize(explain, options = {})
@explain = explain || {}
@indent = 0
@trim = options.has_key?(:trim) ? options.delete(:trim) : true
@trim = options.key?(:trim) ? options.delete(:trim) : true
@script_translation_map = options.delete(:script_translation_map) || {}
@rendering_options = options

parse_details
Expand All @@ -69,7 +70,7 @@ def render_as_hash(&block)

def parse_details
@root ||= begin
tree = ExplainParser.new.parse(explain)
tree = ExplainParser.new(script_translation_map: @script_translation_map).parse(explain)
tree = ExplainTrimmer.new.trim(tree) if trim
tree
end
Expand Down
Loading