Starter implementation for `spath` command #4120

Swiddis · 2025-08-24T04:42:00Z

Description

For now, implementing spath (#4119) as a lightweight conversion directly into a json_extract eval. We'll add more specific semantics later, but this covers the most common usage for the command as-is.

For more path behavior, see:

sql/integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLJsonBuiltinFunctionIT.java

Line 107 in b220be4

public void testJsonExtract() throws IOException {

Related Issues

Related to #4119

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

core/src/main/java/org/opensearch/sql/ast/tree/SPath.java

core/src/main/java/org/opensearch/sql/expression/function/jsonUDF/JsonExtractFunctionImpl.java

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

ppl/src/main/antlr/OpenSearchPPLParser.g4

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

Swiddis

Notes for reviewers

Swiddis · 2025-08-25T22:05:05Z

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

+    String outField = null;
+    String path = null;
+
+    for (OpenSearchPPLParser.SpathParameterContext param : ctx.spathParameter()) {


If an argument appears multiple times, we overwrite it with the new value.

This is consistent with how the other commands behave under the same scenario.

Even input parameter can be multiple? I think it may be worth adding this comment in code.

Swiddis · 2025-08-25T22:05:30Z

core/src/main/java/org/opensearch/sql/expression/function/jsonUDF/JsonExtractFunctionImpl.java

  private static String doJsonize(Object candidate) {
-    if (isScalarObject(candidate)) {
+    if (candidate == null) {
+      return "null"; // Matches isScalarObject, but not toString-able.


Usability patch. Without this, JSON_EXTRACT raises exceptions on any missing fields (which kills extracting from flexibly-typed inputs, one of the major reasons to rely on string types).

The user-facing error without this on a missing value is java.lang.NullPointerException: Cannot invoke "Object.toString()" because "candidate" is null.

Swiddis · 2025-08-25T22:06:03Z

core/src/main/java/org/opensearch/sql/ast/tree/SPath.java

+    return AstDSL.eval(
+        this.child,
+        AstDSL.let(
+            AstDSL.field(outField),
+            AstDSL.function(
+                "json_extract", AstDSL.field(inField), AstDSL.stringLiteral(this.path))));
+  }


This is the core behavior, the rest is plumbing/parsing boilerplate.

It spath is translate to eval, does spath node still required?

We're going to be adding more functionality to it later, so I think it's worthwhile to have the full node here. We could probably be more efficient by parsing it directly into an eval at the tree level if it stays as just this step, though.

penghuo · 2025-08-25T22:32:46Z

docs/user/ppl/cmd/spath.rst

+    +----------+---+
+    | doc      | n |
+    |----------+---|
+    | {"n": 1} | 1 |


results is string? "1"

Yep, returns a string -- we can't really have it dynamically choose the type until we have the ability to dynamically make schemas at planning time

penghuo · 2025-08-25T22:34:02Z

docs/user/ppl/cmd/spath.rst

+
+PPL query::
+
+    PPL> source=test_spath | spath input=doc output=first_element list{0} | spath input=doc output=all_elements list{} | spath input=doc output=nested nest_out.nest_in;


What is the expectation of conflict output?

### POST {{baseUrl}}/tttt/_bulk Content-Type: application/x-ndjson { "index": { "_id": 1 } } { "@timestamp": "2025-08-25 00:00:01", "id": 1, "nested_out": {"nested_in": "success"} } { "index": { "_id": 2 } } { "@timestamp": "2025-08-25 00:00:02", "id": 2, "nested_out": {"nested_in": "failed"} } ### POST {{baseUrl}}/_plugins/_ppl/ Content-Type: application/x-ndjson { "query": "source=tttt | eval doc='{\"list\": [1, 2, 3, 4], \"nest_out\": {\"nest_in\": \"a\"}}' | spath input=doc output=nested_out nest_out.nest_in | fields nested_out.nested_in" } ### POST {{baseUrl}}/_plugins/_ppl/ Content-Type: application/x-ndjson { "query": "source=tttt | eval doc='{\"list\": [1, 2, 3, 4], \"nest_out\": {\"nest_in\": \"a\"}}' | spath input=doc output=nested_out nest_out.nest_in | fields nested_out" }

Same as with eval, it overwrites the existing field. This seems correct to me

penghuo · 2025-08-25T22:36:50Z

docs/user/ppl/cmd/spath.rst

+    +------------------------------------------------------+---------------+--------------+--------+
+    | doc                                                  | first_element | all_elements | nested |
+    |------------------------------------------------------+---------------+--------------+--------|
+    | {"list": [1, 2, 3, 4], "nest_out": {"nest_in": "a"}} | 1             | [1,2,3,4]    | a      |


result is string? "[1,2,3,4]"

"null", and "[]"

We could also return null instead of "null", but it seems consistent with the whole "stringify into json" rule that the rest of the json_extract method has going on

penghuo · 2025-08-25T22:38:58Z

core/src/main/java/org/opensearch/sql/ast/tree/SPath.java

+    return AstDSL.eval(
+        this.child,
+        AstDSL.let(
+            AstDSL.field(outField),
+            AstDSL.function(
+                "json_extract", AstDSL.field(inField), AstDSL.stringLiteral(this.path))));
+  }


It spath is translate to eval, does spath node still required?

ppl/src/test/java/org/opensearch/sql/ppl/parser/AstBuilderTest.java

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

RyanL1997

Hi @Swiddis , thanks for taking this on. Generally lgtm, just need to fix the license header for some classes.

RyanL1997 · 2025-08-28T16:44:45Z

ppl/src/test/java/org/opensearch/sql/ppl/utils/SPathRewriteTest.java

@@ -0,0 +1,62 @@
+package org.opensearch.sql.ppl.utils;


Missing license header.

RyanL1997 · 2025-08-28T16:46:01Z

core/src/main/java/org/opensearch/sql/ast/tree/SPath.java

@@ -0,0 +1,55 @@
+package org.opensearch.sql.ast.tree;


Missing license header

RyanL1997 · 2025-08-28T16:49:14Z

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLSpathCommandIT.java

+        executeQuery("source=test_spath | spath input=doc output=result path=n | fields result");
+    verifySchema(result, schema("result", "string"));
+    verifyDataRows(result, rows("1"), rows("2"), rows("3"));
+  }


nit: maybe add another error case of malformed JSON / invalid paths

dai-chen

Please fix all license header as Ryan commented.

opensearch-trigger-bot · 2025-08-28T17:56:14Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4120-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 4dc4ba99f3bcabeebc41685fea2684fe5c6db9b1
# Push it to GitHub
git push --set-upstream origin backport/backport-4120-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4120-to-2.19-dev.

* Stub spath entry in grammar Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add basic grammar parsing for SPath Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Implement simple rewrite logic (WIP) Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Fix all the breakage Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Style fixes + repair build Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add some docs Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Anchor to json_extract Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Allow arbitrary argument ordering Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add SQL translation tests Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Empty commit (CI retry) Signed-off-by: Simeon Widdis <sawiddis@amazon.com> --------- Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

dai-chen · 2025-08-28T18:39:18Z

Please fix all license header as Ryan commented.

@Swiddis Are all license header fixed?

Swiddis · 2025-08-28T18:46:38Z

It was auto-merged

LantaoJin · 2025-09-10T19:32:58Z

@Swiddis need to fix the backporting manually. And add the backport-manually label when backporting PR is submitted.

* Starter implementation for `spath` command (#4120) * Stub spath entry in grammar Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add basic grammar parsing for SPath Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Implement simple rewrite logic (WIP) Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Fix all the breakage Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Style fixes + repair build Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add some docs Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Anchor to json_extract Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Allow arbitrary argument ordering Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add SQL translation tests Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Empty commit (CI retry) Signed-off-by: Simeon Widdis <sawiddis@amazon.com> --------- Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add missing license headers from 4120 Signed-off-by: Simeon Widdis <sawiddis@amazon.com> --------- Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis added 5 commits August 24, 2025 01:14

Stub spath entry in grammar

9e3a4c0

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add basic grammar parsing for SPath

944e160

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Implement simple rewrite logic (WIP)

97d48d8

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Fix all the breakage

3f515f3

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Style fixes + repair build

26cb9bb

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis commented Aug 25, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/ast/tree/SPath.java Outdated Show resolved Hide resolved

Swiddis added the enhancement New feature or request label Aug 25, 2025

Swiddis commented Aug 25, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/expression/function/jsonUDF/JsonExtractFunctionImpl.java Outdated Show resolved Hide resolved

Add some docs

7d978af

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis force-pushed the feature/json-extraction branch from 3ff1cbd to 7d978af Compare August 25, 2025 21:14

Anchor to json_extract

1a02210

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis commented Aug 25, 2025

View reviewed changes

ppl/src/main/antlr/OpenSearchPPLParser.g4 Outdated Show resolved Hide resolved

Swiddis marked this pull request as ready for review August 25, 2025 21:20

Swiddis requested review from GumpacG, MaxKsyunz, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, penghuo, ps48, seankao-az, vamsimanohar and ykmr1224 as code owners August 25, 2025 21:20

Swiddis requested a review from yuancu as a code owner August 25, 2025 21:20

Swiddis changed the title ~~Simple stub for spath command~~ Starter implementation for spath command Aug 25, 2025

Swiddis added PPL Piped processing language backport 2.19-dev labels Aug 25, 2025

Allow arbitrary argument ordering

1bb05ce

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis commented Aug 25, 2025

View reviewed changes

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java Show resolved Hide resolved

Swiddis commented Aug 25, 2025

View reviewed changes

penghuo reviewed Aug 25, 2025

View reviewed changes

Swiddis added 2 commits August 26, 2025 16:52

Add SQL translation tests

d65fc03

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Empty commit (CI retry)

d401382

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis enabled auto-merge (squash) August 27, 2025 21:26

penghuo approved these changes Aug 28, 2025

View reviewed changes

RyanL1997 approved these changes Aug 28, 2025

View reviewed changes

This comment was marked as duplicate.

Sign in to view

dai-chen approved these changes Aug 28, 2025

View reviewed changes

Swiddis merged commit 4dc4ba9 into opensearch-project:main Aug 28, 2025
23 checks passed

opensearch-trigger-bot bot added the backport-failed label Aug 28, 2025

Swiddis mentioned this pull request Aug 28, 2025

License headers not checked in Spotless #4159

Closed

Swiddis mentioned this pull request Sep 11, 2025

Audit field availability for new keywords added with the 3.3 improvements #4280

Open

LantaoJin added the backport-manually Filed a PR to backport manually. label Sep 24, 2025

Swiddis mentioned this pull request Oct 3, 2025

[RFC] Improved structured data extraction with spath #4119

Closed

This was referenced Jan 11, 2026

docs: add PPL spath command report for v3.3.0 tkykenmt/opensearch-feature-explorer#1579

Merged

[feature] SQL/PPL Spath Command tkykenmt/opensearch-feature-explorer#1331

Closed

Swiddis mentioned this pull request Feb 3, 2026

[RFC] Add nomv command in PPL #5082

Open


		PPL query::

		PPL> source=test_spath \| spath input=doc output=first_element list{0} \| spath input=doc output=all_elements list{} \| spath input=doc output=nested nest_out.nest_in;

Starter implementation for spath command #4120

Starter implementation for spath command #4120

Uh oh!

Conversation

Swiddis commented Aug 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Swiddis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swiddis Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swiddis Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RyanL1997 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as duplicate.

Uh oh!

dai-chen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Aug 28, 2025

Uh oh!

dai-chen commented Aug 28, 2025

Uh oh!

Swiddis commented Aug 28, 2025

Uh oh!

LantaoJin commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Starter implementation for `spath` command #4120

Starter implementation for `spath` command #4120

Swiddis commented Aug 24, 2025 •

edited

Loading

Swiddis Aug 26, 2025 •

edited

Loading

Swiddis Aug 26, 2025 •

edited

Loading

LantaoJin commented Sep 10, 2025 •

edited

Loading