ES|QL: add support for include_execution_metadata parameter #134446

luigidellaquila · 2025-09-10T12:36:27Z

Add support for parameter in ES|QL request.
Add support for ~~include_cps_metadata~~ include_execution_metadata parameter in ES|QL request.

It will be supported both in Stateful and in Serverless, as a synonym for include_ccs_metadata (that will remain for bwc)

elasticsearchmachine · 2025-09-10T12:37:16Z

Hi @luigidellaquila, I've created a changelog YAML for you.

…ta' into esql/include_cps_metadata

elasticsearchmachine · 2025-09-10T15:18:50Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

smalyshev · 2025-09-10T23:09:43Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlQueryRequest.java

    private boolean columnar;
    private boolean profile;
    private boolean includeCCSMetadata;
+    private Boolean includeCPSMetadata;


Is there a reason why we need both of these and can't just have include_cps_metadata option turn on includeCCSMetadata?

The only practical reason is that I'm doing the validation at a later stage, in TransportEsqlQueryAction.java (just because I already have the cluster state there, and I can tell if we are in serverless), so at that point I need to know which of the flags was set.

I don't know if that's a strong enough reason, maybe we can try to inject the cluster state in RequestXContent.objectParserCommon() and do the validation at parse time, but it seems a bit inconvenient

I see. I just wonder how important it is that include_cps_metadata would only work in CPS instead of just being a synonym for the other option and make the whole thing simple.

I don't have a strong preference on this.
In S2D38 it was decided that it will be available only on Serverless, so I'm just sticking to this decision

I just wonder how important it is that include_cps_metadata would only work in CPS instead of just being a synonym for the other option and make the whole thing simple.

As Luigi said, we don't want to expose it in stateful. And ideally I think we would put in the API spec that

include_ccs_metadata is only present in stateful

include_cps_metadata is only present in serverless

so that clients/users generally use the correct parameter.

However, practically we would accept include_ccs_metadata in serverless if the user puts that (a hidden option), but we wouldn't accept include_cps_metadata in stateful. I think this is consistent with Luigi's current implementation.

Thanks @quux00, I like the idea of having a more generic name that we can also apply to Stateful, but the concept of "execution" in ES|QL is pretty broad, and includes compute details (that today you can retrieve with profile=true, and I'm pretty sure we don't want in this context).

I wonder if include_execution_info could be confusing for this reason, and if we could find a more specific name. include_execution_metadata could be an option

I agree with @luigidellaquila, execution sounds too broad and could mean a lot of things. It sounds like we are trying to find a common name for a remote project and remote cluster. I wonder if we can use include_remote_metadata? Possibly we need to come up with a name for operations happening across multiple clusters/projects without explicitly mentioning cluster/projects.

@quux00 @luigidellaquila @idegtiarenko @naj-h
I like the idea of using a generic name, but I also agree that include_execution_info is a very, very broad name. In the context of ESQL, "execution" can mean a lot of things, and is far too broad about query execution in linked projects.

If we need a new name, include_execution_info is not my preference. Let's find something else.

I do not mind include_execution_metadata, even though that is also pretty generic.
include_project_metadata may be an option too.

QQ: do we still use _clusters in the response? This doesn't feel super compatible with project model in CPS.

include_remote_metadata

Would be problematic because this section also includes the local cluster info and a potential future goal of this additional info is to include it even when it is a local only search, as I mentioned above, as not doing it is a violation of flat world principles in serverless.

include_project_metadata

Would be problematic as it has the same problem of include_cps_metadata, namely the "project" concept doesn't exist in stateful and we want a generic name that can work in both.

include_execution_metadata

This is fine with me.

QQ: do we still use _clusters in the response?

We asked this of Product and the decision was to leave it, since modifying it would result in different response bodies between serverless and stateful.

I am OK with include_execution_metadata

quux00 · 2025-09-16T14:49:56Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlQueryRequest.java

+
+    public Boolean includeCPSMetadata() {
+        return includeCPSMetadata;
+    }


Should we add a validation check either in this class or in the RestEsqlQueryAction that the user hasn't set both include_ccs_metadata and include_cps_metadata? It's a minor edge case, but we probably don't want users in serverless to be able to set them both and potentially to different values, so we should throw an exception if both are set (in serverless) with an error message like "Both include_cps_metadata and include_ccs_metadata query parameters are set. Use only include_cps_metadata."

It reduces chances of user errors, so ++, let me add it

++ both should not be allowed at the same time

quux00 · 2025-09-16T14:52:00Z

Let's also do an API spec ticket for this in https://github.com/elastic/elasticsearch-specification

I think we should docuement there that:

include_ccs_metadata is only present in stateful
include_cps_metadata is only present in serverless

so that clients/users generally use the correct parameter.
(even though we will actually accept include_ccs_metadata in serverless, we just won't document that).

luigidellaquila · 2025-09-16T15:49:09Z

Thanks @quux00, I created a ticket for the spec and I'm adding the validation, as you suggested.

… same request

idegtiarenko · 2025-09-17T09:10:29Z

server/src/main/java/org/elasticsearch/transport/RemoteClusterService.java

         *  the functionality to do it the right way is not yet ready -- replace this code when it's ready.
         */
-        this.canUseSkipUnavailable = settings.getAsBoolean("serverless.cross_project.enabled", false) == false;
+        this.crossProjectEnabled = settings.getAsBoolean("serverless.cross_project.enabled", false);


I'm no longer using this for this PR, but I'll need it for include_execution_metadata, so I'm keeping it here.

idegtiarenko · 2025-09-17T09:11:46Z

...ql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/CrossProjectQueryIT.java

+        return CollectionUtils.appendToCopy(super.nodePlugins(clusterAlias), CpsPlugin.class);
+    }
+
+    protected EsqlQueryResponse runQuery(String query, Boolean cpsMetadataInResponse) {


NIT:

Suggested change

protected EsqlQueryResponse runQuery(String query, Boolean cpsMetadataInResponse) {

protected EsqlQueryResponse runQuery(String query, Boolean includeCpsMetadata) {

idegtiarenko · 2025-09-19T07:28:27Z

...ql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/CrossProjectQueryIT.java

+        }
+    }
+
+    private static final Setting<String> CpsEnableSetting = Setting.simpleString(


NIT: should it be named CPS_ENABLED_SETTING? I am a bit surprised spotless did not complain about it.

I borrowed that code from other tests, but I agree with you that CPS_ENABLED_SETTING would be better

As we discussed off-line, I'm moving this test to Serverless

## Summary ES is adding a new parameter for retrieving ccs metadata. This works the same as the `include_ccs_metadata` but they want us to use the new one instead elastic/elasticsearch#134446. The reasons they did this change are: - they are trying to avoid the "ccs" term in serverless as much as possible - to be consistent with the flat world model of "all projects should behave the same in a search", in the future we might be changing include_execution_metadata to also return info for local-only searches, which it doesn't do right now. Both params will be available in serverless and stateful

ES|QL: add support for include_cps_metadata parameter

a2f8e24

luigidellaquila added >enhancement :Analytics/ES|QL AKA ESQL labels Sep 10, 2025

elasticsearchmachine added the v9.2.0 label Sep 10, 2025

luigidellaquila added 4 commits September 10, 2025 14:37

Update docs/changelog/134446.yaml

f31b2ae

Media type tests

fc798aa

Merge remote-tracking branch 'luigidellaquila/esql/include_cps_metada…

eb9769b

…ta' into esql/include_cps_metadata

Merge branch 'main' into esql/include_cps_metadata

b49d8e4

luigidellaquila requested review from idegtiarenko, quux00 and smalyshev September 10, 2025 14:01

luigidellaquila marked this pull request as ready for review September 10, 2025 15:18

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Sep 10, 2025

smalyshev reviewed Sep 10, 2025

View reviewed changes

quux00 reviewed Sep 16, 2025

View reviewed changes

luigidellaquila mentioned this pull request Sep 16, 2025

Add include_execution_metadata to ES|QL query spec elastic/elasticsearch-specification#5299

Closed

luigidellaquila added 3 commits September 16, 2025 18:07

Don't allow both include_cps_metadata and include_ccs_metadata in the…

5065bbe

… same request

Merge branch 'main' into esql/include_cps_metadata

b42c726

Merge branch 'main' into esql/include_cps_metadata

7c81329

idegtiarenko reviewed Sep 17, 2025

View reviewed changes

luigidellaquila added 2 commits September 17, 2025 12:18

Better names

285eb09

Merge branch 'main' into esql/include_cps_metadata

6b40a52

idegtiarenko approved these changes Sep 18, 2025

View reviewed changes

Merge branch 'main' into esql/include_cps_metadata

8631236

luigidellaquila changed the title ~~ES|QL: add support for include_cps_metadata parameter~~ ES|QL: add support for include_execution_metadata parameter Sep 18, 2025

include_ccs_metadata -> include_execution_metadata

84de275

idegtiarenko reviewed Sep 19, 2025

View reviewed changes

luigidellaquila added 2 commits September 19, 2025 09:34

Merge branch 'main' into esql/include_cps_metadata

acd19c4

Remove redundant test

cfa4d97

luigidellaquila enabled auto-merge (squash) September 19, 2025 07:37

luigidellaquila merged commit 5e1bae4 into elastic:main Sep 19, 2025
34 checks passed

This was referenced Sep 22, 2025

ES|QL: add include_execution_metadata to the query spec elastic/elasticsearch-specification#5327

Closed

ES|QL: add include_execution_metadata to the query spec elastic/elasticsearch-specification#5328

Merged

stratoula mentioned this pull request Sep 23, 2025

[ES|QL] Changes the ccs flag elastic/kibana#236030

Merged

shainaraskas mentioned this pull request Oct 10, 2025

Update release notes for Elastic Cloud Serverless elastic/docs-content#3431

Open

	protected EsqlQueryResponse runQuery(String query, Boolean cpsMetadataInResponse) {
	protected EsqlQueryResponse runQuery(String query, Boolean includeCpsMetadata) {

ES|QL: add support for include_execution_metadata parameter #134446

ES|QL: add support for include_execution_metadata parameter #134446

Conversation

luigidellaquila commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Sep 10, 2025

Uh oh!

elasticsearchmachine commented Sep 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luigidellaquila Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quux00 Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quackaplop Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quackaplop Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quux00 commented Sep 16, 2025

Uh oh!

luigidellaquila commented Sep 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

luigidellaquila commented Sep 10, 2025 •

edited

Loading

luigidellaquila Sep 11, 2025 •

edited

Loading

quux00 Sep 16, 2025 •

edited

Loading

quackaplop Sep 18, 2025 •

edited

Loading

quackaplop Sep 18, 2025 •

edited

Loading