HTTP requests should have an `User-Agent` header

[endersonmaia]

Is your feature request related to a problem? Please describe.

When using fluent-bit's prometheus_scraper input plugin , I found that the service I was trying to scrape requires the HTTP requests to have a User-Agent headed and responds with HTTP 418 if it doesn't.

Looking into fluent-bit HTTP client conde, I found that the User-Agent header is only defined when inside the flb_http_client_proxy_connect() branch.

https://github.com/fluent/fluent-bit/blob/master/src/flb_http_client.c

Describe the solution you'd like

fluent-bit should send the User-Agent HTTP Header for all HTTP requests, or at least provide a global or per plugin configuration where the user can enable/disable the User-Agent Http Header

Additional context

As an evidence of the behavior, you could do this:

Start fluent-bit with:

docker run -ti --rm --net=host fluent/fluent-bit:3.0.6 -i prometheus_scrape -p host=127.0.0.1 -p port=8080  -o stdout

On one terminal, listen at TCP port 8080, and wait for the scrape plugin:

nc -l -p 8080
GET /metrics HTTP/1.1
Host: 127.0.0.1:8080
Content-Length: 0

[endersonmaia]

Ideally, we should have a well formed User-Agent like:

User-Agent: Fluent-Bit/3.0.6 (input: in_prometheus_scrape, Git commit: 9af65e2c36a2c36e75b93cab6a92bf29c307ec9e)

And that would change based on the plugin used to make the requests.