[System tests] Run system tests for integrations with a non root Elastic Agent

[jlind23]

With the addition of the two features below, integrations will now be able to run on non root Elastic Agent. If not specified otherwise, an integration will be runnable on a non root Agent. In order to ensure that integrations are properly running without problems we need to update Elastic Package system test in order to run a non root Elastic Agent.

Related issues

• Add a "Need to run as root" option in package spec package-spec#537
• [Non-Root] Add support for running without root permissions on Linux elastic-agent#3598

Blocked on #787

cc @nimarezainia @pierrehilbert

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jlind23]

@blakerouse I guess we will need elastic/elastic-agent#3799 to be reverted before being able to run these tests right?

@mrodm @jsoriano I guess we will have to add this flag to the Elastic Agent docker image we use right?

[pierrehilbert]

@jlind23 the flag is only hidden so I think there is no problem to run these tests

[jlind23]

@jsoriano @kpollich After chatting with @blakerouse it seems that we should already be able to do it by changing the user that the docker image relies on.
@jen-huang this is then ready to be picked up whenever someone has time.

[jsoriano]

it seems that we should already be able to do it by changing the user that the docker image relies on.

One challenge we will face is that elastic-package test doesn't manage the Elastic Agent used for tests, it is managed by elastic-package stack and is independent of the package to be tested. So as things are now, we cannot just change the docker user depending on the package.

We have though a number of issues that would be solved if we move the Elastic Agent management from the stack subcommand to the test one. This is specially discussed in #787, that could be a generalization of this issue.

The kubernetes and custom agent service deployers, used by some integrations, circumvent this limitation by managing their own agents, but this exceeds their responsibilities, and cannot be generalized to all integrations.

I think we should take the opportunity to dedicate some time to solve #787. It would help on this issue and other similar ones, and it would allow to pay off some technical debt.

If we cannot dedicate time to this now, we could add a temporary profile setting, that makes elastic-package stack up aware of the root requirement of the package in the current directory. This will likely be enough for CI, as we start one stack per package, but can lead to unexpected situations when working locally on multiple packages with the same stack.

[jlind23]

I'll be more than happy to see @jen-huang prioritising #787 😄 Anyway this is something we should tackle rather soonish to make the unprivileged agents fully testable.

[nimarezainia]

@jsoriano in your opinion what is the level of effort in having #787 resolved ?

[jsoriano]

in your opinion what is the level of effort in having #787 resolved ?

This is probably an L-sized task.

Anyway this is something we should tackle rather soonish to make the unprivileged agents fully testable.

If we are in a hurry, we could go in the way of adding a temporary profile setting by now, at least to have coverage in CI.

[nimarezainia]

@jen-huang @jlind23 should we consider prioritizing #787 as part of the quality work? However it's also tempting to add the temp profile as suggested to move forward with testing for unprivileged agents.

[jlind23]

If we are in a hurry, we could go in the way of adding a temporary profile setting by now, at least to have coverage in CI.

I think we should do it anyway and work on the long term plan at the same time.

[jlind23]

@jen-huang @jlind23 should we consider prioritizing #787 as part of the quality work? However it's also tempting to add the temp profile as suggested to move forward with testing for unprivileged agents.

Sounds good to me.

[cmacknz]

I think there are a few complications to keep in mind if we were to rely completely on the integration system tests to detect if an integration can correctly run with a non-root Elastic Agent:

1. Since the system tests are based on Linux containers, they obviously provide zero test coverage for Windows and integrations that may need to be Admin to function.
2. Our Docker container user doesn't run as root, but is a member of the root group so can read files an agent installed with elastic-agent install --unprivileged cannot. See Remove the Docker elastic-agent user from the root group elastic-agent#4087.
3. The filesystem of the Elastic Agent docker container does not exactly match what would be present in an Ubuntu VM. For example there are not system or auth logs by default, so we can't evaluate if we broke reading from some file sources.

That isn't to say this isn't valuable, it just isn't going to test the worst edge cases.

Technically the tests are already running as a non-root user, and once we fix elastic/elastic-agent#4087 the elastic-agent user in our Docker container will have no root permissions at all.

[jsoriano]

That isn't to say this isn't valuable, it just isn't going to test the worst edge cases.

Another point in favour of #787 🙂 This issue was originally motivated exactly by the lack of testing on different platform configurations.

[jen-huang]

I'm moving this issue to a future sprint and pulling in #787 for the next one.

[kpollich]

This is blocked on #787 in the current sprint, but can be worked on once the custom agent image work is completed.

[mrodm]

With this PR #1815 , those packages that have agent.privileges.root and do not define _dev/deploy/agent (custom agents) will spin up a new Elastic Agent container for the tests.

If these new Elastic Agents (containers) are created (no custom agents), they are going to define user: root in their docker-compose configuration automatically:

services:
  elastic-agent:
    hostname: ${AGENT_HOSTNAME}
    image: ${ELASTIC_AGENT_IMAGE_REF}
    user: root
    ...