Allow elastic agent in containers to use basic auth to get service token

[michel-laterman]

What does this PR do?

Allow the agent to use basic auth defined by env vars to retrieve a
service token from Elasticsearch and inject it into the config used for
the agent and fleet.

Why is it important?

Allow the agent started in a container to use basic auth to contact Elasticsearch and generate a service_token.
This should fix the tests which rely on docker-compose.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Related issues

• Closes Kibana Fleet setup failed: http POST request to .... fails: Unauthorized #29603

Use cases

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-01-04T18:12:16.783+0000

• Duration: 96 min 32 sec

• Commit: 8ba0d57

Test stats 🧪

Test	Results
Failed	0
Passed	7120
Skipped	16
Total	7136

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[michel-laterman]

Running elastic-package stack up with this agent leads to the fleet-server agent starting successfully. However the elastic-agent (without fleet-server) fails to find it's policy. The logs contain:

Created service_token named: token-1640912065673
Error: unable to find policy named ""

It looks like using a service_token fails to list policies:

elastic-agent|retrieve-service-token-container⚡ ⇒ curl -XPOST localhost:5601/api/fleet/service-tokens -u elastic:changeme -H "kbn-xsrf: value"
{"name":"token-1640912238005","value":"AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NDA5MTIyMzgwMDU6ak53LTZHdkFTZnFrN0dTal9SM3BiUQ"}%
elastic-agent|retrieve-service-token-container⚡ ⇒ curl localhost:5601/api/fleet/agent_policies -H "Authorization: Bearer AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NDA5MTIyMzgwMDU6ak53LTZHdkFTZnFrN0dTal9SM3BiUQ"
{"items":[],"total":0,"page":1,"perPage":20}%
elastic-agent|retrieve-service-token-container⚡ ⇒ curl localhost:5601/api/fleet/agent_policies -u elastic:changeme
{"items":[{"id":"499b5aa7-d214-5b5d-838b-3cd76469844e","namespace":"default","monitoring_enabled":["logs","metrics"],"name":"Default Fleet Server policy","description":"Default Fleet Server agent policy created by Kibana","is_default":false,"is_default_fleet_server":true,"is_preconfigured":true,"status":"active","is_managed":false,"revision":1,"updated_at":"2021-12-31T00:54:04.727Z","updated_by":"system","package_policies":["default-fleet-server-agent-policy"],"agents":1},{"id":"2016d7cc-135e-5583-9758-3ba01f5a06e5","namespace":"default","monitoring_enabled":["logs","metrics"],"name":"Default policy","description":"Default agent policy created by Kibana","is_default":true,"is_preconfigured":true,"status":"active","is_managed":false,"revision":1,"updated_at":"2021-12-31T00:54:02.703Z","updated_by":"system","package_policies":["default-system-policy"],"agents":0}],"total":2,"page":1,"perPage":20}%

@aleksmaus, I noticed you had to adjust the permissions the service token gets in Elasticsearch, do you think we're missing another one?

[simitt]

Related issue: elastic/fleet-server#1048 where priveleges that are defined on the package level are missing.

[aleksmaus]

It looks like using a service_token fails to list policies:

@michel-laterman the fleet service token has full access to the .fleet-policies. In your case you are querying the policies through kibana API, could be a possible defect in Kibana.

[michel-laterman]

I can use basic auth to generate a service_token via Kibana, and use that token to post to /api/fleet/setup and /api/fleet/agents/setup.
However the token will return 0 items with /api/fleet/agent_policies (no failure status or error messages). And I have confirmed that the token returns items when searching the .fleet-policies index in ES.
@joshdover, It looks like this is an issue with the fleet api in Kibana.

[joshdover]

@michel-laterman I think I see the problem and I believe it should be addressed by the work in elastic/kibana#121824. Could you confirm that it works against this branch?

If so, I think we'll probably want a more focused PR to fix this for 8.0, which I can prepare.

[joshdover]

Went ahead and threw up a focused fix, please test against this one: elastic/kibana#122320

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[michel-laterman]

Thanks @joshdover, I've confirmed that a service token can now list policies

[blakerouse]

Nicely done! Looks good.