[Elastic Agent] Agent Upgrade feature test

[michalpristas]

Scenario should be as follow:

2 versions of agent are available (must differ in last commit)

• first agent is the one which is installed
• second one is made available using some public API or local fileserver (must be available at {uri}:{port}/beats/elastic-agent/{tar_name}

First agent is installed and is ONLINE in fleet.

Second agent artifacts is made available

for local test i'm using fileserver with artifacts at ~/server/beats/elastic-agent/*
then from ~/server i run python3 -m http.server

Update is initiated either using UI (manual tests) or using API (manual/automated tests)

API for initiating upgrade using fileserver serving at localhost

curl -X POST -H 'kbn-xsrf: true'  -u elastic:changeme http://localhost:5601/api/ingest_manager/fleet/agents/{AGENT_ID}/upgrade \
  -H 'Content-Type: application/json' \
 -d '{"source_uri":"http://localhost:8000","version":"8.0.0"}'

If fleet does not send SourceURI it is good to configure agent.download.sourceURI to http://localhost:8000 in elastic-agent.yml

Agent is listed as UPDATING in fleet

• with file server this will be only for a fraction of second or couple of seconds max (update is very fast)

Update action is ACKed in fleet

Agent version is changed and agent is listed as ONLINE in fleet

• state should be running
• all beats has to be started and collecting again based on config assigned

No ERROR should be reported during this procedure.

Prerequisites

elastic/kibana#78810
elastic/beats#21461

[EricDavisX]

UPDATED 10/14, by Eric Davis:

Thanks Michal! Allow me to take a first stab at the feature file, content posted below will be updated as we learn how we can test the feature. Manu and I are happy to review.

The syntax below will be dependent on the stack-branch being new enough that the prior version Agent code is 'upgradeable'. For now we can merge this to 8.0 and 7.11 lines.

The steps below have some steps added back for the filebeat, metricbeat, and endpoint about 'sending new data' to ES. Now is a good time to add those back.

file:
agent_and_endpoint_upgrades.feature

@agent_and_endpoint_upgrades
Feature: Agent with Endpoint and Agent Only Upgrade Tests
Scenarios for Agent to deploy with and without Endpoint and upgrade itself.

@deploy-agent-and-upgrade
Scenario: Deploying an Agent and then upgrading it to the latest
Given an older-than-the-stack upgradeable agent is installed to "centos"
And the agent is listed in Fleet as "online"
When the agent upgrade is initiated to current version
Then agent is listed as UPDATING in fleet
And update action is ACKed in Fleet
And agent version is changed in Fleet
And agent is listed as ONLINE in Fleet
And the "elastic-agent" process is in the "started" state on the host
And the "filebeat" process is in the "started" state on the host
And the "metricbeat" process is in the "started" state on the host
And "filebeat" sends new data to ES
And "metric" sends new data to ES

@deploy-endpoint-with-agent-and-upgrade
Scenario: Deploying Endpoint with Agent and upgrade it
Given an older-than-the-stack upgradeable agent is installed to "centos"
And the agent is listed in Fleet as "online"
And the "Elastic Endpoint Security" integration is "added" in the policy
And the "Elastic Endpoint Security" datasource is shown in the policy as added
And the host name is shown in the Administration view in the Security App as "online"
And Then the policy response will be shown in the Security App
And the agent is listed in Fleet as "online"
When the agent upgrade is initiated to current version
Then agent is listed as UPDATING in fleet
And update action is ACKed in Fleet
And agent version is changed in Fleet
And agent is listed as ONLINE in Fleet
And the "elastic-agent" process is in the "started" state on the host
And the "elastic-endpoint" process is in the "started" state on the host
And "endpoint" sends new data to ES

[ph]

@EricDavisX Just to make sure I understand who will do what on this issue. You wrote the feature file, @michalpristas will write the hooks in the cuccumber framework with the help of @mdelapenya?

cc @KseniaElastic

[EricDavisX]

that's a good plan as I see it.

[EricDavisX]

Note - we should we able to edit the comment above and capture all of the feature file spacing requirements to paste into the real file.

I have been manually testing the feature and we can cite the following tech details to implementing this:
While testing, it may not be possible to test via the UI to validate this.

Using the API, you can initiate the upgrade as follows...

• the upgrade is first supported in 7.10, so it is most easy for us to test with automation using the 8.0 or 7.11 stack and we will plan on only supporting that.

• deploy a 7.10 Agent to 8.0 master (and 7.11) stack. in the future we can find a way to keep this dynamic (current stack version minus one)

• then the upgrade can be initiated via the api as follows:
  /api/fleet/agents/90872928-e61d-43d6-95c8-d5181095588f/upgrade

• the string '90872928-e61d-43d6-95c8-d5181095588f' is the Agent ID, which the framework has examples for how to obtain

• the body for this will be as follows, for 7.11:
  {"source_uri":"https://snapshots.elastic.co/7.11.0-8a4554fc/downloads/","version":"7.11.0-SNAPSHOT"}

• for 8.0, the body for this will be as follows:
  {"source_uri":"https://snapshots.elastic.co/8.0.0-1234567/downloads/","version":"8.0.0-SNAPSHOT"}

the requirement for the API to pass in the source_uri is potentially not needed for all builds, but that is tbd as to when it isn't needed so lets plan to always send it for now. The challenge may be for us to capture the dynamic 'current' Agent build, but I think we have that code or at least can use jk / grep to parse it out from the general snapshot build likes like: https://artifacts-api.elastic.co/v1/search/7.11-SNAPSHOT

• the individual url paths are listed as example:
  url: "https://snapshots.elastic.co/7.11.0-9855d75d/downloads/apm-server/apm-server-7.11.0-SNAPSHOT-i686.rpm"

I think that is all that is new for this feature, if we need to discuss more API usage please don't hesitate to reach out

[EricDavisX]

Further on this, @michalpristas and @mdelapenya and @neptunian ...me and Michal had a quick chat. The above is doable with current architecture, but it doesn't actually cover the code that is executed in the Agent fully. So we're considering how to account further.

I think if we had a -force flag in the API that would remove the Kibana restriction on upgrading version x.y.z to version x.y.z that could suffice. In that case, we could test the current PR agent in the current version of Kibana master could ‘force’ an upgrade to basically the same version, but it will be trying out the code in the Agent PR, that we want. I think that makes sense, thoughts? I don’t think we can test this without more code change to remove the (intentional) restriction.

To re-use the phrasing Michal had started with me earlier, we'll have:

• latest stable GA upgrade to newer snapshot build Agent (nightly e2e test, as cited in comments above)
• current master PR build can upgrade to master SNAPSHOT build (/package call dev testing)
  ... this 2nd test is dependent on removing the restriction as noted. And it will basically run in nightly e2e as well, effectively just as a test of upgrading to itself, which still tests out the code paths even if the versions don’t change. But in this case, we’ll need find a way to set which versions to inspect for. Or maybe we can mark that 2nd test as special and somehow skip it during nightly CI.
  @mdelapenya Do you have ideas on how we could do that or how else we'd choose to handle it?

[neptunian]

@EricDavisX If I understand correctly you need to be able to test to upgrading to the same version in order to test new builds. Does this have to be accessible through the UI (via a button) or would making a call to the API suffice?

[EricDavisX]

@neptunian we could survive with just API usage, as the test here will consume it as such.

[neptunian]

@EricDavisX Could this be something we do for 7.11 or is there a reason we'd need it in 7.10?

[EricDavisX]

The additional API support is preferred for 7.11, not needed or wanted for 7.10. @neptunian we're in good shape for 7.10. :)

[EricDavisX]

we are fully unblocked to implement this now. all feature tests are done and 7.10 is working well.

[EricDavisX]

this is done and can be closed, it was only ever targeted towards 'Agent' upgrade not Fleet, just fyi (hence why I updated the description)