Allow DEB and RPM packages to supply an uninstall token when upgrading with Elastic Defend installed

[cmacknz]

• Relates The elastic-agent upgrade CLI command needs to require an uninstall token when Elastic Defend is installed #6356
• Relates Add integration tests for upgrades that include endpoint security #4720

Describe the enhancement:

For Fleet managed agents that are running Elastic Defend, upgrading the RPM or DEB packages does not send a signed upgrade action to Defend which is what allows Defend to upgrade without believing it is being tampered with. The tamper protection feature of Defend is based on all operations requiring a digital signature signed by a private key in Kibana. There is no way to provide a signed token for DEB or RPM upgrades, so Defend legitimately believes it was being tampered with and orphaned from agent here.

Specifically using the CLI skips this block

elastic-agent/internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go

Lines 58 to 74 in 54932dc

	if h.tamperProtectionFn() {
	// Find inputs that want to receive UPGRADE action
	// Endpoint needs to receive a signed UPGRADE action in order to be able to uncontain itself
	state := h.coord.State()
	ucs := findMatchingUnitsByActionType(state, a.Type())
	if len(ucs) > 0 {
	h.log.Debugf("handlerUpgrade: proxy/dispatch action '%+v'", a)
	err := notifyUnitsOfProxiedAction(ctx, h.log, action, ucs, h.coord.PerformAction)
	h.log.Debugf("handlerUpgrade: after action dispatched '%+v', err: %v", a, err)
	if err != nil {
	return err
	}
	} else {
	// Log and continue
	h.log.Debugf("No components running for %v action type", a.Type())
	}
	}

To make the upgrade command work with tamper protection properly we'd have to allow it to accept an uninstall token to allow endpoint to unprotect itself for a CLI upgrade.

Describe a specific use case for the enhancement or feature:

Upgrading an Elastic Agent installed with either the RPM or DEB packages when Elastic Defend is used. Without this change, the only way to upgrade in this situation is to temporarily disable tamper protection.

What is the definition of done?

• Tests exists proving that the RPM and DEB packages can be upgraded when Elastic Defend is installed without disabling tamper protection.

The two bullet points below are not relevant anymore as the user does not have to provide an uninstall token

• These packages types can ideally indicate to users that they should supply an uninstall token when defend is installed.
• The documentation for the RPM and DEB packages is updated to indicate that an uninstall token is required in this situation.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[intxgo]

@cmacknz what about msi and Tamper Protection?

I forgot, msi accepts INSATLLARGS, example here https://discuss.elastic.co/t/installing-elastic-agent-with-gpo/364659/

Furthermore, the variables table of msi can be edited, so we can embed any argument. @gabriellandau already made a nice PS script https://gist.github.com/gabriellandau/b838192eefd81fc53816f4e973a026d0

[cmacknz]

We don't support upgrading via the msi package today, only installing, so I wouldn't expect to have to support an uninstall token for that purpose there unless that changes.

There is INSTALLARGS for uninstalling while installing.

[intxgo]

writing it here, since somehow it took me a while to realize what's the workaround to upgrade via DEB/RPM

sudo systemctl stop ElasticEndpoint
sudo rm -rf /opt/Elastic/Endpoint/state/vault
# endpoint is unprotected now

Since the tamper protection on Linux is just an uninstall prevention, loosely speaking, A root has to execute the upgrade anyway so I believe the step of removing the vault can be embedded inside DEB/RPM.

This is literally what happens behind the scenes when Endpoint receives UPGRADE action, it just drops it's policy. The installer which is executed next is checking current Endpoint policy for tamper protection, no-policy is a valid state. When newly installed Endpoint starts and connects to Agent it receives the policy and thus re-arm itself if tamper protection is ON in the policy.

[pkoutsovasilis]

@intxgo quite recently although we apply the proposed flow

sudo systemctl stop ElasticEndpoint
sudo rm -rf /opt/Elastic/Endpoint/state/vault
# endpoint is unprotected now

endpoint started return exit code 28 both on systemctl stop and systemctl start command. Do you happen to know if anything changed lately?

We observe these failures on these PRs which essentially bump our CI to the next snapshot DRA, and in the previous ones we don't observe these failures

• main
• 8.19
• 9.1

cc @ebeahan

[intxgo]

Hey @pkoutsovasilis sorry for late reply, I was on a long PTO. Is the issue solved? We've recently changed the verify command behavior to start the service if it's stopped or disabled.