.fleet-action-results cannot be accessed by cloud

[stefnestor]

Elasticsearch Version

8.15.1

Installed Plugins

No response

Java Version

bundled

OS Version

ESS

Problem Description

👋 howdy, team!

Expanding #89261 (from #71667) for impact and potential resolution variation, Support+users using ECE+ESS UI+API rather than ES API directly cannot view information for .fleet-actions-results due to the Fleet header restriction.

# GET .ds-.fleet-actions-results-2024.XX.XX-0000XX/_ilm/explain
{ "error": {
  "reason": "Data stream(s) [.fleet-actions-results] may not be accessed by product [cloud]",
  "root_cause": [{
-   "reason": "Data stream(s) [.fleet-actions-results] may not be accessed by product [cloud]",
    "type": "illegal_argument_exception"
  }], "type": "illegal_argument_exception"
}, "status": 400 }

If Support CURL overrides to recommended header x-elastic-product-origin: fleet we only update error to

{ "error": {
    "reason": "multiple values for single-valued header [X-elastic-product-origin].",
    "root_cause": [{
-        "reason": "multiple values for single-valued header [X-elastic-product-origin].",
        "type": "illegal_argument_exception"
    }], "type": "illegal_argument_exception"
}, "status": 400 }

This is true for at least Index Settings and ILM Explain but not CAT Shards nor CAT Indices. E.g.

# GET .fleet-actions-results/_ilm/explain
(error above)
# GET *fleet-actions-results*/_ilm/explain
(NULL)
# GET .fleet-actions-results/_settings
(error above)

Due to the nature of how ES diags poll its API, Support does not know they have incomplete/missing data & our diagnostic automations have unexpected behavior.

Steps to Reproduce

1. Create ESS Deployment with Integrations Server, add Fleet-managed Agent
2. Poll API variations above and see results/not.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mosche]

@stefnestor Sorry for the late response on this, we discussed the issue in Core/Infra today. First and foremost, we'd like to better understand what the expectation and intend of this issue is. We're lacking a bit of context, particularly when it comes to the mentioned tooling (Support CURL).

The allowed product origins for fleet system indices are kibana and fleet. In that way, cloud is rightfully rejected. Are you asking to include cloud in the list of allowed products?

The issue you're describing above when attempting to override the product origin header honestly sounds like a bug in the tooling to us. The header is intentionally single valued and - based on our limited understanding - there was objections in the team changing this. But as said initially, we might be missing some context here.

[stefnestor]

Thanks, @mosche ! I appreciate you calling out me potentially not using the right jargon (sorry!) & happy to discuss further offline, but attempting clarification:

Confirming this surfaces when

• Support/user uses Elastic Cloud's ES API Console (aka. Support CURL)
• Support / ECE Admin pulls the Deployment > Operations > ES diag

My expectation is that the ES diag/API works within Cloud the same as on-prem (based on ES permission restrictions not API header restrictions) and that we don't silently drop response data which throws off our diag automations.

I'm not sure on best resolution from Dev's perspective but agree adding cloud to the allowlist sounds like a viable solution to me.

[mosche]

The fleet system indices are owned by the ingest team, changing the allowed product origins would be up to them. I'm honestly not sure about the implications / consequences of doing so with respect to allowing Cloud. @kpollich could your team answer if adding Cloud as allowed product origin is an acceptable path forward here?

Confirming this surfaces when
Support/user uses Elastic Cloud's ES API Console (aka. Support CURL)
Support / ECE Admin pulls the Deployment > Operations > ES diag

Though, as previously hinted, to me this sounds like an issue that is better fixed in above mentioned tools. Rather than just appending another product origin to the already existing Cloud origin, the existing origin header should be replaced by the support tools. If overriding headers is meant to be supported by those, it has to be done the correct way. Unfortunately I don't know who's owning the tools, I'd leave it to you to reach out to the respective team.

[kpollich]

This does seem like an issue that should be fixed in the support tooling rather than in the ES behavior here. I'll also cc @lucabelluccini for context from the support side.

I think there is technically an information leakage concern for action results, as they can contain data about user hosts in the event that a user is executing osquery operations for example that return potentially sensitive data about their environments. This is already somewhat true for other Fleet indicies, though, so I think our existing cloud authentication schemes cover us in terms of policy here.

[lucabelluccini]

Reached out privately, we'll post the outcome here.

[mosche]

The issue with the mutliple values for single-valued header is occurring because somehow there's already X-elastic-product-origin defined and the curl you've attempted is using x-elastic-product-origin: fleet. It seems that somehow that when using X-elastic + x-elastic, the 2 values are merged as an array of values and rejected at validaton. While using X-elastic and X-elastic, the last value overrides the first one.
I tried -H'X-elastic-product-origin:fleet' and it works:

Looks like there is a bug in the way headers are processed. Based on above, this might not always be done in a case INsensitive way.

[mosche]

@lucabelluccini We looked into the the issue you've raised above regarding inconsistent handling of headers if casing differs. Thanks @ldematte for adding various unit tests in the area! We can definitely rule out ES as the source of this bug, it looks like your tooling attempts to deduplicate the product origin header, but doesn't do so in a case INsensitive way. Please pass this issue on to the team maintaining the tools.

From a Core/Infra perspective we currently don't see anything else we can do here. I'm leaving it to you @kpollich, @lucabelluccini and @stefnestor to discuss next steps or to close this.

[stefnestor]

(Followed-up in private thread to share data-sensitive example.)