Return 503 Service Unavailable when unable to authenticate with Elasticsearch instead of 401

[cmacknz]

• Relates Remove automatic unenrollment after 7 Fleet authentication failures elastic-agent#5428

When we attempt to validate an agent's API key and cannot communicate with ES because Fleet Server's API key/service token is temporarily unavailable we return a 401 to Elastic Agents. This seems correct, but if it happens for 7 check ins in a row will cause the agent to automatically unenroll.

We should evaluate handling "cannot auth with ES" from "explicitly told API key was invalid" separately and returning a 503 when Fleet Server itself cannot auth with ES to avoid unintentional mass unenrollments.

The relevant code block is below:

fleet-server/internal/pkg/apikey/auth.go

Lines 34 to 61 in 6c14449

	// Authenticate will return the SecurityInfo associated with the APIKey (retrieved from Elasticsearch).
	// Note: Prefer the bulk wrapper on this API
	func (k APIKey) Authenticate(ctx context.Context, es *elasticsearch.Client) (*SecurityInfo, error) {
	token := fmt.Sprintf("%s%s", authPrefix, k.Token())
	req := esapi.SecurityAuthenticateRequest{
	Header: map[string][]string{AuthKey: []string{token}},
	}
	res, err := req.Do(ctx, es)
	if err != nil {
	return nil, fmt.Errorf("apikey auth request %s: %w", k.ID, err)
	}
	if res.Body != nil {
	defer res.Body.Close()
	}
	if res.IsError() {
	returnError := ErrUnauthorized
	if res.StatusCode == 429 {
	returnError = ErrElasticsearchAuthLimit
	}
	return nil, fmt.Errorf("%w: %w", returnError, fmt.Errorf("apikey auth response %s: %s", k.ID, res.String()))
	}

[blakerouse]

The actual lines in that block that are wrong are here:

https://github.com/elastic/fleet-server/blob/main/internal/pkg/apikey/auth.go#L54-L60