Add integration tests for upgrades that include endpoint security

[cmacknz]

We need to add upgrade integration tests where endpoint security is installed. We need to test both the tamper protected and unprotected cases and the .tar.gz, RPM, and DEB package types.

As part of this addition, the logging around forwarding the upgrade action to endpoint needs to be increased to the info level.

elastic-agent/internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go

Lines 58 to 74 in 49745a7

	if h.tamperProtectionFn() {
	// Find inputs that want to receive UPGRADE action
	// Endpoint needs to receive a signed UPGRADE action in order to be able to uncontain itself
	state := h.coord.State()
	ucs := findMatchingUnitsByActionType(state, a.Type())
	if len(ucs) > 0 {
	h.log.Debugf("handlerUpgrade: proxy/dispatch action '%+v'", a)
	err := notifyUnitsOfProxiedAction(ctx, h.log, action, ucs, h.coord.PerformAction)
	h.log.Debugf("handlerUpgrade: after action dispatched '%+v', err: %v", a, err)
	if err != nil {
	return err
	}
	} else {
	// Log and continue
	h.log.Debugf("No components running for %v action type", a.Type())
	}
	}

There have been several recent cases where tamper protected agent upgrades have failed due to invalid uninstall tokens. The root cause is currently unknown, but missing logging and inadequate automated testing are definitely contributing to these problems.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[intxgo]

It would also help if Agent logged its PID, at least at startup. We can see in the endpoint logs when Agent disconnects and reconnects, and are logging its PID each time it connects, but right now we have to search for matching Agent logs by timestamp only.

[cmacknz]

We should do that already, the log lines contain "process.pid":25920 and look like:

{"log.level":"info","@timestamp":"2024-04-16T09:33:06.250Z","log.origin":{"file.name":"cmd/run.go","file.line":155},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":25920,"agent.version":"8.11.2","ecs.version":"1.6.0"}