proposal: support splunk hec logging plugin

[shuaijinchao]

Issue description

Splunk is a machine data search engine. According to the latest database search engine ranking [1], Splunk has been ranked second, and Splunk can be used to collect, index and retrieve various application data. Splunk, like ElasticSearch, is quasi-real-time and can provide an uninterrupted data stream of search results. I think Apache APISIX can provide a plugin that supports pushing request logs to Splunk, expands the data observability of APISIX, and reduces the cost of Splunk users using APISIX as a gateway.

The following are the design and technical details:

Name

• splunk-hec-logging

Configuration

{
    "endpoint":{
        "uri":"https://hec-splunk.company.com/services/collector",
        "token":"BD274822-96AA-4DA6-90EC-18940FB2414C",
        "channel":"FE0ECFAD-13D5-401B-847D-77833BD77131",
        "ssl_verify":true,
        "timeout": 60
    },
    "inactive_timeout":10,
    "max_retry_count":0,
    "buffer_duration":60,
    "retry_delay":1,
    "batch_max_size":1
}

• endpoint Splunk HTTP Event Collector(HEC) endpoint
• endpoint.uri Splunk HTTP Event Collector(HEC) endpoint request uri
• endpoint.token Splunk HTTP Event Collector(HEC) endpoint request access token [2]
• endpoint.channel Splunk HTTP Event Collector(HEC) endpoint channel identifier (GUID) [3]
• endpoint.ssl_verify Splunk HTTP Event Collector(HEC) endpoint enable SSL verify
• endpoint.timeout Splunk HTTP Event Collector(HEC) endpoint connection timeout (unit: second)
• max_retry_count max number of retries before removing from the processing pipe line
• retry_delay number of seconds the process execution should be delayed if the execution fails
• buffer_duration max age in seconds of the oldest entry in a batch before the batch must be processed
• inactive_timeout max age in seconds when the buffer will be flushed if inactive
• batch_max_size max size of each batch

Details

Configuration process

1. Add and set up the Http Event Controller (HEC) through the Splunk console and get the access token.
2. If HEC enable the indexer to confirm, you must specify a channel and obtain the channel ID.
3. Set the request URI, access token, and channel ID of HEC to the plugin configuration

HTTP Request process

1. Obtain and assemble request information data in the APISIX Log stage, data format refer to [4]
2. Add the assembled request data to the batch queue
3. When the threshold of the batch queue is triggered, the requested data is submitted in batches to Splunk's HEC

Push to HEC data format:

{
    "time":1426279439.123,
    "host":"localhost",
    "source":"apache-apisix-splunk-hec-logging",
    "sourcetype":"_json",
    "event":{
        "request_url":"http:\/\/127.0.0.1:9080\/splunk.do?q=hello&a=world",
        "request_method":"GET",
        "request_headers":{
            "user-agent":"curl\/7.68.0",
            "host":"127.0.0.1:9080",
            "accept":"*\/*"
        },
        "request_query":{
            "a":"world",
            "q":"hello"
        },
        "request_size":100,
        "response_headers":{
            "transfer-encoding":"chunked",
            "connection":"close",
            "content-type":"text\/html; charset=utf-8",
            "server":"APISIX\/2.11.0",
            "date":"Mon, 29 Nov 2021 14:03:26 GMT"
        },
        "response_status":200,
        "response_size":102,
        "latency":0.98,
        "upstream":"127.0.0.1:1980"
    }
}

[1] https://db-engines.com/en/ranking/search+engine
[2] https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Create_an_Event_Collector_token_on_Splunk_Enterprise
[3] https://docs.splunk.com/Documentation/Splunk/latest/Data/AboutHECIDXAck#About_channels_and_sending_data
[4] https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata

Maillist

https://lists.apache.org/thread/1ktwd15mcrh53fdqlgzm3sj3wzhwkyvx