[filebeat] Add CSV decoder to httpjson input

[marc-gr]

What does this PR do?

Adds a text/csv decoder to the httpjson input.

Why is it important?

Some apis will only have the option to return csv content, which we want to convert to several events.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[marc-gr]

/test

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b httpjson-csv-dec upstream/httpjson-csv-dec
git merge upstream/master
git push upstream httpjson-csv-dec

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-10-26T09:02:28.669+0000

• Duration: 93 min 19 sec

• Commit: 8be04d7

Test stats 🧪

Test	Results
Failed	0
Passed	7775
Skipped	1201
Total	8976

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

[mtojek]

Hmm, there are CI errors that look related to this PR. Did it work locally for you?

[leehinman]

Not related to code, but I'm wondering if we should rename the input since we aren't getting json back? It just seems odd to get CSV from httpjson.

Also I'm thinking it might be a good idea to switch to something like the parsers config option that was added to the filestream input.

[marc-gr]

Not related to code, but I'm wondering if we should rename the input since we aren't getting json back? It just seems odd to get CSV from httpjson.

The input will return json, but this adds the ability to convert a CSV response from an API to a list of JSON objects. In any case, I am fine with the eventual renaming (to http maybe?), since this does more and more as time goes by.

Also I'm thinking it might be a good idea to switch to something like the parsers config option that was added to the filestream input.

You mean something that lets us add extra configuration to the parser/decoder? Or directly using parser.Config from libbeat as in filestream?

[leehinman]

Not related to code, but I'm wondering if we should rename the input since we aren't getting json back? It just seems odd to get CSV from httpjson.

The input will return json, but this adds the ability to convert a CSV response from an API to a list of JSON objects. In any case, I am fine with the eventual renaming (to http maybe?), since this does more and more as time goes by.

I like http or http_client, be nice if http_endpoint got renamed to http_server or httpd for consistency.

Also I'm thinking it might be a good idea to switch to something like the parsers config option that was added to the filestream input.

You mean something that lets us add extra configuration to the parser/decoder? Or directly using parser.Config from libbeat as in filestream?

I was thinking it would be nice to to use something like the Reader interface that libbeat uses and to be able to stack them in a particular order like you can with parser config in filestream. So our code just calls Next and the reader knows how to do that. Then we could support things like new line delimited, fixed size, multiline, json, csv, etc..

[leehinman]

We can worry about rename later.