[Filebeat][Inputs][HTTP JSON] Multi-Step/API Requests

[BenB196]

Describe the enhancement:

I'd like to request the ability to specify running multiple steps (API calls) in a single HTTP JSON input configuration.

Describe a specific use case for the enhancement or feature:

There are some APIs which require you to first authenticate against an auth endpoint, to get an API token, to then use to execute an API call. This requires 2 different URLs/APIs to be used in a single input.

Example Flow

An example of this type of flow.

1. You have a clientId and a secret, and you make a call like:

curl -X POST -u "<clientId>:<secret>" my-api.com/auth

4. This API returns a JSON value like:

{"accessToken": "<some_access_token>"}

5. You then use this token in a second request to auth against the actual API:

curl -X GET -H "Authorization: <accessToken>" my-api.com/realAPI

6. You then get the actual data you wanted.

{"APIData": "<api_data>"}

Reading through the docs, this doesn't seem possible currently.

[cmacknz]

Some support for multi-step or chained httpjson calls was added in #29816. It is tagged for 8.3 but looks like it made the cutoff for the upcoming 8.2 release.

I'm not sure it would support setting the Authorization header as in your example though, it likely needs to be extended further to allow that.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

I have a work-in-progress that would be able to do this in #31233 (though not as part of HTTP JSON). If this goes in, it is likely to be in the new V2 input system.

[andrewkroh]

One use case example is JAMF Protect. https://docs.jamf.com/jamf-protect/documentation/Jamf_Protect_API.html

[ShourieG]

@BenB196 Hey there, are you still looking for this feature to be implemented in httpjson or were you able to solve the issue you were facing ?

[BenB196]

Hi @ShourieG I've unfortunately not have a chance to look into whether #32222 resolved my issue. I'll see if I can test this in the next week or so.

[ShourieG]

@BenB196 any updates on this ?

[BenB196]

Hi @ShourieG I was looking into this, but unless I'm missing something the chain configuration options aren't available via the Fleet Integration UI (I'm on Kibana 8.4.2), so I don't think I can actually test this feature in a custom integration setup currently.

[andrewkroh]

This is likely fully possible with the new CEL input.

[ShourieG]

Closing this issue for the moment, since an alternative solution using CEL is available