[Meta][CrowdStrike] Supporting Event Streaming API

[jamiehynds]

Our existing CrowdStrike integration requires the Falcon SIEM Connector in order to pull detections and audit events from Falcon. This adds complexity and a maintenance overhead for users which could be eliminated if our integration connected directly to Falcon's Event Streaming API to pull the data.

The Event Streaming API works differently to most API's we interact with and our CEL or HTTPJSON inputs cannot currently handle API. In order to get data, you keep one request open indefinitely, while periodically making an independent request to refresh the streaming session to keep it alive.

In order to unblock us from adding support for the Event Streaming API, we need to research if a new input is required or can we bridge the functionality gap via one of our existing inputs.

Tasks

Preview Give feedback

1. [Crowdstrike]Investigate potential support for Event Streaming API in Crowdstrike #10262
   Integration:crowdstrike Team:Security-Service Integrations +2
   
2. Re-work websocket input to support Crowdstrike streaming API format beats#40264
   Team:Security-Service Integrations +1
   
3. Add input/data stream to Crowdstrike integration to make use of streaming input
4. Add input/data stream to Proofpoint integration to make use of streaming input

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[mbudge]

"complexity and a maintenance overhead for users"

We've had it running on a server for 2-3 years with no issues. It handles all the log rotation so it's very easy to maintain.

[jamiehynds]

Thanks for the feedback @mbudge. Based on that, if we supported the Event Streaming API directly would you envisage some value there or likely to remain on SIEM Connector?

[mbudge]

The advantage of using the SIEM connector is you can push customers to use crowdstrike support if there is a problem downloading data.

As this one is a bit complicated with persistent connections I'd prefer to use the SIEM connector.

[NateUT99]

We have used the event stream API with a few other products, and have had good success with it. I would certainly be comfortable moving our production data ingest to this (from SIEM connector) once it becomes available.

[ShourieG]

The last time we had discussions on supporting event stream inputs the following approaches were decided upon :-

• Build a standalone input similar to the websocket input and have support for CEL to process the responses
• Bundle both websocket and event streaming input under a single umbrella sharing common CEL code and have them aliased under say a category like "streaming inputs".

@brijesh-elastic @piyush-elastic, I recall last time we spoke on this, support for CEL was requested because support for response manipulation was required as well as support for URL modification based on cursor values. Could you confirm this and expand on the reason for CEL support ?

cc: @efd6

[piyushw-crest]

@ShourieG - Yes we did discuss on this last time for CrowdStrike only, as we have option to configure event stream API and access it via http endpoint. For more details refer this pdf. Let me know if you need more help on the same.

[ShourieG]

@ShourieG - Yes we did discuss on this last time for CrowdStrike only, as we have option to configure event stream API and access it via http endpoint. For more details refer this pdf. Let me know if you need more help on the same.

@piyush-elastic thanks for the update, I was curious on the CEL support. Do you think it would add value and if what would be the use case here?