Bump stack version to 7.15.0

[sebgl]

Using hack/update-stack-version.sh.
upgrade-test-harness and operatorhub related files will be updated separately.

[david-kow]

When java.lang.IllegalStateException: JENKINS-37121: something already locked /var/lib/jenkins/workspace/cloud-on-k8s-pr is fixed for devops-ci, let's make sure we run full pr build for this one.

[sebgl]

run full pr build

[sebgl]

E2E tests fail with #4878

[barkbay]

While it's working as expected with 7.14 Agents are not able to setup when using 7.15 with fleet-custom-logs-integration.yaml :

$ k logs pod/elastic-agent-agent-65rmv -f 
Kibana Fleet setup failed: http POST request to https://kibana-kb-http.default.svc:5601/api/fleet/setup fails: fail to execute the HTTP POST request: Post "https://kibana-kb-http.default.svc:5601/api/fleet/setup": context deadline exceeded (Client.Timeout exceeded while awaiting headers). Response:
Kibana Fleet setup failed: http POST request to https://kibana-kb-http.default.svc:5601/api/fleet/setup fails: Package policy is invalid: inputs.logfile.streams.log.log.vars.paths: Log file path is required: <nil>. Response: {"statusCode":400,"error":"Bad Request","message":"Package policy is invalid: inputs.logfile.streams.log.log.vars.paths: Log file path is required"}
Kibana Fleet setup failed: http POST request to https://kibana-kb-http.default.svc:5601/api/fleet/setup fails: Package policy is invalid: inputs.logfile.streams.log.log.vars.paths: Log file path is required: <nil>. Response: {"statusCode":400,"error":"Bad Request","message":"Package policy is invalid: inputs.logfile.streams.log.log.vars.paths: Log file path is required"}

I tried several ways to specify the paths variable, like the following one, without success:

            vars:
              - name: paths
                type: text
                value:
                  - '/var/log/containers/*${kubernetes.container.id}.log'

Agents are still failing with the same error: Package policy is invalid: inputs.logfile.streams.log.log.vars.paths: Log file path is required

Note that any attempt to access the Fleet UI in Kibana fails with the following error:

[david-kow]

Hey @Zacqary, can you advise on the issue above? It seems the expected config for Custom Logs integration changed between 7.14 and 7.15. I've looked at the integration repo, but I didn't find any changes, so I think it's on Kibana side.

[david-kow]

FYI, I'm still looking into failures on Kibana side.

[kpollich]

Hey folks, I spent some time investigating this in elastic/kibana#113400 and just posted a quick summary in a comment.

We should be able to provide paths as an array value to unblock this, e.g. replace

cloud-on-k8s/config/recipes/elastic-agent/fleet-custom-logs-integration.yaml

Lines 38 to 39 in e25f504

	- name: paths
	value: '/var/log/containers/*${kubernetes.container.id}.log'

with

 - name: paths 
   value: 
     - '/var/log/containers/*${kubernetes.container.id}.log' 

Previously, Fleet would accept this value, but in 7.15 we're applying the same validation client-side and server-side to our package policy creation process. So preconfigured values can no longer skirt validation rules such as "array values must be explicitly provided as an array" like in this case.

The initial Package policy is invalid: inputs.logfile.streams.log.log.vars.paths: Log file path is required was caused by a separate issue that was fixed in elastic/kibana#113204.

Let me know if there are still issues here and I'll happily continue taking a look!

[david-kow]

I tested with Kibana at 7.15.1-056f5838-SNAPSHOT and I can confirm it works. Thanks @kpollich for assistance, I really appreciate it.

@sebgl, I'd propose to stay at 7.14.0 for that manifest (fleet-custom-logs-integration.yaml) only and update everything to 7.15.1 when it's out.

[sebgl]

@david-kow good idea, let's do that! Thanks @david-kow @barkbay @kpollich for the investigation 🙏

[sebgl]

run full pr job

[david-kow]

run full pr build

[david-kow]

Hey @sebgl, I think we are good to merge this :)