Filter Expression support #440
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is v6 of the perf_SUITE tests. The test adds a complex index entry to every object, and then adds a new test phase to test regex queries. There are three profiles added so the full, mini and profiling versions of perf_SUITE can be run without having to edit the file itself: e.g. ./rebar3 as perf_mini do ct --suite=test/end_to_end/perf_SUITE When testing as `perf_prof` summarised versions of the eprof results are now printed to screen. The volume of keys within the full test suite has been dropped ... just to make life easier so that test run times are not excessively increase by the new features.
The re2 library, though not offering the complete functionality of pcre, is known to be much faster in many scenarios. Switch to re2 in preparation for the OTP switch to re2 in OTP28
Assume to make the job of gs easier - name makes a massive difference to load time in OTP 24.
alos try and improve test stability by increasing pause
Identify and log out memory usage by test phase
Allow for a capturing regular expression to be passed with a filter function that will filter based on the captured output. Also allows a specific captured term to be returned as the returned term (rather than the whole index term)
First draft of allowing an externally provided comparison expression to be validated, parsed and evaluated
Required as re2 can only output binary not string in captures.
Allow for capture with either delimited index terms, or with regex capture, and then filtering of captured attributes using a logical expression. The xref check `locals_not_used` is now ignored due to issues with auto-generated module.
Allow for the generation of projected attributes from an index term
Allow for additional functions in eval. Change the leveled query API to generalise application of eval/filter expressions.
Remove `capture` option - as all can be achieved via eval/filter.
Closed
The Filter should expect to compare BETWEEN low AND high ... supporting high AND low may be confusing. The filter should allow for identifier to be in any location in a comparator i.e. string BETWEEN identifier AND identifier integer > identifier This make check some validity constraints harder at parse time (i.e. that High >= Low), but makes tse result more compatible. For ease, of implementation IN can only be used for a comparison between a string and a list of strings (with the identifier representing either part).
now passes 10k tests
Differentiate between positive/negative integers and non-negative integers. Stop parsing of unusable split/index values.
Works on index terms of type unicode_binary()
ThomasArts
reviewed
May 23, 2024
ThomasArts
reviewed
May 23, 2024
ThomasArts
reviewed
May 23, 2024
ThomasArts
reviewed
May 23, 2024
| " | map($birthday, <=, ((\"0430\", 2024)), 2023, $yoc)" | ||
| " | subtract($yoc, $yob, $age)" | ||
| " | add($age, 1, $age_next)" | ||
| " | to_string($age, $age)" |
There was a problem hiding this comment.
Didn't realize one can overload variables to have different types... probably not the nicest thing to use in an example.
ThomasArts
reviewed
May 23, 2024
| %% Lexer for eval expressions | ||
|
|
||
| Definitions. | ||
| WhiteSpace = ([\t\f\v\r\n\s]+) |
There was a problem hiding this comment.
When reading the documentation, I would say \s subsumes the others.
ThomasArts
reviewed
May 23, 2024
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
Co-authored-by: Thomas Arts <thomas.arts@quviq.com>
…n' into mas-d31-mas.i1433-filterexpression
This reverts commit 1b320c6.
Update tot he lexers, to require strings to be non-empty, which rules out a raft of potential failure scenarios during evaluation. Handle unexpected types in eval functions by not creating the attribute rather than throwing an exception. Add a new ends_with function, that behaves as begins_with in reverse.
Allow for return_keys of logical combinations of queries
Avoid confusion over meaning of NOT
Don't rely on ordering of inputs, use a numeric identifier as a key
ThomasArts
approved these changes
May 28, 2024
ThomasArts
approved these changes
Nov 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Allow indexes to be evaluated into a Key/Value map using an eval pipeline, and then filtered using a filter expression