-
Notifications
You must be signed in to change notification settings - Fork 139
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] JSON Search Support #2652
Comments
it would help a lot to support |
Json Functions ProposalI have created a working prototype for the This is currently implemented in the sql sub-project to make the json functions available not only as PPL command. In other words: The function can be used (like any other built in function) in sql and ppl. The proposed (and so far implemented syntax) is:
The function returns the result as string (scalar value or full json) An error is thrown when:
No error is thrown when:
Examples:
Open questions:
Not yet covered
|
@anasalkouz @YANG-DB @rupal-bq any comments on the proposal so far? |
Can we perhaps specify a document ID to use as the json for the query? I've got a lot of json blobs that I'd love to search through instead of breaking them up before ingest. |
can you post an example how this can look like? |
source input {
"Statement": [
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::example-bucket/*"
},
{
"Effect": "Deny",
"Action": "s3:DeleteObject",
"Resource": "arn:aws:s3:::example-bucket/*"
}
]
} query SELECT ...
FROM ...
WHERE json_extract(p.policy_std, '$.Statement[*].Effect') = 'Allow'
AND f.name = 'hellopython';
|
Will add this ... |
@salyh I've added a separate |
Is your feature request related to a problem?
Community members have asked for easier JSON parsing and analysis capabilities which allow them to not only search JSON logs and extract fields without writing complex parse expressions, but perform computations on JSON array values, such as finding the sum of all values in the array, where the number of elements in the array is not known.
What solution would you like?
Allow users to extract and transform data from JSON-formatted events and fields. Users should be able to extract all values in an array by specifying a wildcard for the individual element position and doing an aggregation operation on them. Users should be able to extract: 1/single or multiple top level fields 2/nested fields 3/keys in arrays and perform operations on the values.
** Examples ***
What alternatives have you considered?
No other solutions are available in PPL
Do you have any additional context?
No
The text was updated successfully, but these errors were encountered: