Skip to content

Support multisearch command in calcite#4332

Merged
RyanL1997 merged 38 commits intoopensearch-project:mainfrom
ahkcs:feat/multisearch_cmd
Oct 2, 2025
Merged

Support multisearch command in calcite#4332
RyanL1997 merged 38 commits intoopensearch-project:mainfrom
ahkcs:feat/multisearch_cmd

Conversation

@ahkcs
Copy link
Contributor

@ahkcs ahkcs commented Sep 18, 2025

Description

This PR introduces the multisearch command, a new PPL command that allows combining results from multiple search subsearches into a single unified result set.

Resolves: #4348

Key Features:

  • Combines results from 2+ search subsearches with different filtering criteria
  • Supports ALL PPL commands within subsearches (where, eval, fields, stats, sort, dedup, etc.)
  • Enables timestamp-based result interleaving for time-series data analysis
  • Uses efficient UNION ALL + ORDER BY architecture for optimal performance

Example Usage:

-- Basic age group analysis
source=accounts | multisearch
  [search source=accounts | where age < 30 | eval age_group = "young"]
  [search source=accounts | where age >= 30 | eval age_group = "adult"]
| stats count by age_group

-- Success rate monitoring pattern
source=logs | multisearch
  [search source=logs | where status="success" | eval result="success"]
  [search source=logs | where status!="success" | eval result="total"]
| stats count(eval(result="success")) as success_count, count() as total_count

@ahkcs ahkcs force-pushed the feat/multisearch_cmd branch from 92ab992 to 5c43a2b Compare September 23, 2025 23:57
Signed-off-by: Kai Huang <ahkcs@amazon.com>
dai-chen
dai-chen previously approved these changes Oct 1, 2025
@@ -0,0 +1,200 @@
{"index":{"_id":"1"}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

np: do we really need so many test data to demonstrate the command usage in documentation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking we can reuse this dataset for future doctest for timestamp

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated to trim the dataset to 20 records

Signed-off-by: Kai Huang <ahkcs@amazon.com>
ykmr1224
ykmr1224 previously approved these changes Oct 1, 2025
Signed-off-by: Kai Huang <ahkcs@amazon.com>
@dai-chen
Copy link
Collaborator

dai-chen commented Oct 1, 2025

Do we add and benchmark Big5 query for each new command like this?

@ahkcs
Copy link
Contributor Author

ahkcs commented Oct 1, 2025

Do we add and benchmark Big5 query for each new command like this?

I think we can add IT tests to benchmark Big5 queries like this, I'll open a separate PR for this:
#4163

@ahkcs
Copy link
Contributor Author

ahkcs commented Oct 1, 2025

Do we add and benchmark Big5 query for each new command like this?

For Multisearch command, I think the result will be exactly the same as Append command as they share the same implementation

@RyanL1997 RyanL1997 merged commit ca5a5bd into opensearch-project:main Oct 2, 2025
51 of 52 checks passed
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4332-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 ca5a5bd6ee36ecdbbb2d3dc7a4b73ba28ecafa95
# Push it to GitHub
git push --set-upstream origin backport/backport-4332-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4332-to-2.19-dev.

ahkcs added a commit to ahkcs/sql that referenced this pull request Oct 2, 2025
---------

Signed-off-by: Kai Huang <ahkcs@amazon.com>
(cherry picked from commit ca5a5bd)
penghuo pushed a commit that referenced this pull request Oct 3, 2025
…4431)

* Support `multisearch` command in calcite (#4332)

---------

Signed-off-by: Kai Huang <ahkcs@amazon.com>
(cherry picked from commit ca5a5bd)

* fix ExplainIT

Signed-off-by: Kai Huang <ahkcs@amazon.com>

* put missing file back

Signed-off-by: Kai Huang <ahkcs@amazon.com>

---------

Signed-off-by: Kai Huang <ahkcs@amazon.com>
@LantaoJin LantaoJin added the backport-manually Filed a PR to backport manually. label Oct 9, 2025
asifabashar added a commit to asifabashar/sql that referenced this pull request Oct 10, 2025
* main-apple: (218 commits)
  Add ignorePrometheus Flag for integTest and docTest (opensearch-project#4442)
  Create fab-radar.yml
  PPL `fillnull` command enhancement (opensearch-project#4421)
  reverting to _doc + _id (opensearch-project#4435)
  Support `multisearch` command in calcite (opensearch-project#4332)
  Add 3.3 release notes (opensearch-project#4422) (opensearch-project#4423)
  [SQL/PPL] Fix the `count(*)` and `dc(field)` to be capped at MAX_INTEGER opensearch-project#4416 (opensearch-project#4418)
  Change the default search sort tiebreaker to `_shard_doc` for PIT search (opensearch-project#4378)
  [Enhancement] Add error handling for known limitation of sql `JOIN` (opensearch-project#4344)
  Bugfix: SQL type mapping for legacy JDBC output (opensearch-project#3613)
  Version bump: 3.3 (opensearch-project#4417)
  Add max/min eval functions (opensearch-project#4333)
  Support time modifiers in search command  (opensearch-project#4224)
  Fix numbered token bug and make it optional output in patterns command (opensearch-project#4402)
  refactor span (opensearch-project#4334)
  Move release notes categories (opensearch-project#3818)
  [Doc] Enable doctest with Calcite (opensearch-project#4379)
  Mod function should return decimal instead of float when handle the operands are decimal literal (opensearch-project#4407)
  Scale of decimal literal should always be positive in Calcite (opensearch-project#4401)
  Enable Calcite by default and implicit fallback the unsupported commands (opensearch-project#4372)
  ...
@github-project-automation github-project-automation bot moved this from Todo to Done in PPL 2026 Roadmap Jan 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.19-dev backport-failed backport-manually Filed a PR to backport manually. calcite calcite migration releated feature PPL Piped processing language

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Support multisearch command for PPL

7 participants