Cherry-pick #25423 to 7.13: Do not use unversioned home with watcher binary path

[michalpristas]

Cherry-pick of PR #25423 to 7.13 branch. Original message:

What does this PR do?

The problem is when agent tries to invoke watcher which checks upgraded agent for failures and at the same time unversioned home path is set.

Usually agent took its home path which is {top_path}/data/elastic-agent-{hash}/elastic-agent and invoked watch sub-command on binary at that path.

When unversioned path is set it tries to run watcher from {top_path}/data/elastic-agent and as so it results in binary not found.

Fixes: #25371

Why is it important?

Reduce noise created by failng attempt to run this. This is just a case when running container sub-command which tempers paths.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

cc @michel-laterman

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #25460 opened

• Start Time: 2021-04-30T12:15:50.389+0000

• Duration: 52 min 59 sec

• Commit: 0e74ff0

Test stats 🧪

Test	Results
Failed	1
Passed	6739
Skipped	24
Total	6764

Trends 🧪

Test errors

Expand to view the tests failures

Build&Test / x-pack/elastic-agent-windows-2012-windows-2012-r2 / TestConfigurableFailed – github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/operation

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestConfigurableFailed
2021-04-30 12:37:04.8957917 +0000 GMT m=+9.084002001 write error: failed to rotate backups: failed to rotate backups: rename C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log.1: The process cannot access the file because it is being used by another process.
2021-04-30 12:37:06.2287928 +0000 GMT m=+10.417003001 write error: failed to rotate backups: failed to rotate backups: rename C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log.1: The process cannot access the file because it is being used by another process.
2021-04-30 12:37:07.5774129 +0000 GMT m=+11.765623201 write error: failed to rotate backups: failed to rotate backups: rename C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log.1: The process cannot access the file because it is being used by another process.
2021-04-30 12:37:08.9202131 +0000 GMT m=+13.108423401 write error: failed to rotate backups: failed to rotate backups: rename C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log.1: The process cannot access the file because it is being used by another process.
2021-04-30 12:37:10.2512555 +0000 GMT m=+14.439465701 write error: failed to rotate backups: failed to rotate backups: rename C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log.1: The process cannot access the file because it is being used by another process.
2021-04-30 12:37:11.5764716 +0000 GMT m=+15.764681901 write error: failed to rotate backups: failed to rotate backups: rename C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log.1: The process cannot access the file because it is being used by another process.
2021-04-30 12:37:12.9016495 +0000 GMT m=+17.089859801 write error: failed to rotate backups: failed to rotate backups: rename C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log.1: The process cannot access the file because it is being used by another process.
2021-04-30 12:37:14.2278088 +0000 GMT m=+18.416019001 write error: failed to rotate backups: failed to rotate backups: rename C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log.1: The process cannot access the file because it is being used by another process.
2021-04-30 12:37:15.5549556 +0000 GMT m=+19.743165801 write error: failed to rotate backups: failed to rotate backups: rename C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log.1: The process cannot access the file because it is being used by another process.
2021-04-30 12:37:16.8791864 +0000 GMT m=+21.067396601 write error: failed to rotate backups: failed to rotate backups: rename C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log.1: The process cannot access the file because it is being used by another process.
2021-04-30 12:37:18.2043812 +0000 GMT m=+22.392591401 write error: failed to rotate backups: failed to rotate backups: rename C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log.1: The process cannot access the file because it is being used by another process.
2021-04-30 12:37:19.5305551 +0000 GMT m=+23.718765301 write error: failed to rotate backups: failed to rotate backups: rename C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log C:\Users\jenkins\AppData\Local\Temp\go-build152501978\b768\elastic-agent.log.1: The process cannot access the file because it is being used by another process.
    common_test.go:135: check timed out after 15 second: process never went to back to running
--- FAIL: TestConfigurableFailed (19.01s)
 

Steps errors

Expand to view the steps failures

x-pack/elastic-agent-windows-2012-windows-2012-r2 - mage build unitTest

• Took 1 min 27 sec . View more details on here
• Description: mage build unitTest

gsutil -m -q cp -a public-read test-build-artifacts-x-pack-elastic-agent-windows-2012-windows-2012-r

• Took 0 min 2 sec . View more details on here
• Description: @echo off gsutil -m -q cp -a public-read test-build-artifacts-x-pack-elastic-agent-windows-2012-windows-2012-r2-tgz gs://beats-ci-temp/Beats/beats/PR-25460-1

Error signal

• Took 0 min 0 sec . View more details on here
• Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-04-30T13:05:44.656Z]  OS/Arch:           linux/amd64
[2021-04-30T13:05:44.656Z]  Context:           default
[2021-04-30T13:05:44.656Z]  Experimental:      true
[2021-04-30T13:05:44.656Z] 
[2021-04-30T13:05:44.656Z] Server: Docker Engine - Community
[2021-04-30T13:05:44.656Z]  Engine:
[2021-04-30T13:05:44.656Z]   Version:          20.10.3
[2021-04-30T13:05:44.656Z]   API version:      1.41 (minimum version 1.12)
[2021-04-30T13:05:44.656Z]   Go version:       go1.13.15
[2021-04-30T13:05:44.656Z]   Git commit:       46229ca
[2021-04-30T13:05:44.656Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-30T13:05:44.656Z]   OS/Arch:          linux/amd64
[2021-04-30T13:05:44.656Z]   Experimental:     false
[2021-04-30T13:05:44.656Z]  containerd:
[2021-04-30T13:05:44.656Z]   Version:          1.4.4
[2021-04-30T13:05:44.656Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-30T13:05:44.656Z]  runc:
[2021-04-30T13:05:44.656Z]   Version:          1.0.0-rc93
[2021-04-30T13:05:44.656Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-30T13:05:44.656Z]  docker-init:
[2021-04-30T13:05:44.656Z]   Version:          0.19.0
[2021-04-30T13:05:44.656Z]   GitCommit:        de40ad0
[2021-04-30T13:05:44.656Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-30T13:05:44.656Z] Unable to find image 'alpine:3.4' locally
[2021-04-30T13:05:45.598Z] 3.4: Pulling from library/alpine
[2021-04-30T13:05:45.598Z] c1e54eec4b57: Pulling fs layer
[2021-04-30T13:05:45.858Z] c1e54eec4b57: Verifying Checksum
[2021-04-30T13:05:45.858Z] c1e54eec4b57: Download complete
[2021-04-30T13:05:46.120Z] c1e54eec4b57: Pull complete
[2021-04-30T13:05:46.120Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-04-30T13:05:46.120Z] Status: Downloaded newer image for alpine:3.4
[2021-04-30T13:05:48.029Z] Change permissions with write access of all files inside the specific folder
[2021-04-30T13:05:49.115Z] Running in /var/lib/jenkins/workspace/PR-25460-1-0ca880d8-c1a8-4bdf-a1df-b25800ccbcd7/src/github.com/elastic/beats/build
[2021-04-30T13:05:49.442Z] + rm -rf ve
[2021-04-30T13:05:49.442Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-04-30T13:05:49.835Z] + python .ci/scripts/pre_archive_test.py
[2021-04-30T13:06:04.732Z] Copy ./x-pack/heartbeat/build into build/x-pack/heartbeat/build
[2021-04-30T13:06:04.732Z] Copy ./x-pack/heartbeat/build/golang-crossbuild into build/x-pack/heartbeat/build/golang-crossbuild
[2021-04-30T13:06:04.732Z] Copy ./x-pack/elastic-agent/build into build/x-pack/elastic-agent/build
[2021-04-30T13:06:04.732Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent-ubi8/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent-ubi8/elastic-agent-linux-amd64.docker/docker-build
[2021-04-30T13:06:04.732Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build
[2021-04-30T13:06:04.732Z] Copy ./x-pack/elastic-agent/build/golang-crossbuild into build/x-pack/elastic-agent/build/golang-crossbuild
[2021-04-30T13:06:04.732Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-04-30T13:06:04.732Z] Copy ./x-pack/filebeat/build/package/filebeat-ubi8/filebeat-linux-amd64.docker/docker-build into build/x-pack/filebeat/build/package/filebeat-ubi8/filebeat-linux-amd64.docker/docker-build
[2021-04-30T13:06:04.732Z] Copy ./x-pack/filebeat/build/package/filebeat/filebeat-linux-amd64.docker/docker-build into build/x-pack/filebeat/build/package/filebeat/filebeat-linux-amd64.docker/docker-build
[2021-04-30T13:06:04.732Z] Copy ./x-pack/filebeat/build/golang-crossbuild into build/x-pack/filebeat/build/golang-crossbuild
[2021-04-30T13:06:04.733Z] Copy ./x-pack/metricbeat/build into build/x-pack/metricbeat/build
[2021-04-30T13:06:04.733Z] Copy ./x-pack/metricbeat/build/package/metricbeat-ubi8/metricbeat-linux-amd64.docker/docker-build into build/x-pack/metricbeat/build/package/metricbeat-ubi8/metricbeat-linux-amd64.docker/docker-build
[2021-04-30T13:06:04.733Z] Copy ./x-pack/metricbeat/build/package/metricbeat/metricbeat-linux-amd64.docker/docker-build into build/x-pack/metricbeat/build/package/metricbeat/metricbeat-linux-amd64.docker/docker-build
[2021-04-30T13:06:04.733Z] Copy ./x-pack/metricbeat/build/golang-crossbuild into build/x-pack/metricbeat/build/golang-crossbuild
[2021-04-30T13:06:04.763Z] Running in /var/lib/jenkins/workspace/PR-25460-1-0ca880d8-c1a8-4bdf-a1df-b25800ccbcd7/src/github.com/elastic/beats/build
[2021-04-30T13:06:04.813Z] Recording test results
[2021-04-30T13:06:06.508Z] None of the test reports contained any result
[2021-04-30T13:06:06.517Z] [Checks API] No suitable checks publisher found.
[2021-04-30T13:06:07.017Z] + go clean -modcache
[2021-04-30T13:06:12.807Z] Cleaning up /var/lib/jenkins/workspace/PR-25460-1-0ca880d8-c1a8-4bdf-a1df-b25800ccbcd7
[2021-04-30T13:06:12.807Z] Client: Docker Engine - Community
[2021-04-30T13:06:12.807Z]  Version:           20.10.3
[2021-04-30T13:06:12.807Z]  API version:       1.41
[2021-04-30T13:06:12.807Z]  Go version:        go1.13.15
[2021-04-30T13:06:12.807Z]  Git commit:        48d30b5
[2021-04-30T13:06:12.807Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-04-30T13:06:12.807Z]  OS/Arch:           linux/amd64
[2021-04-30T13:06:12.807Z]  Context:           default
[2021-04-30T13:06:12.807Z]  Experimental:      true
[2021-04-30T13:06:12.807Z] 
[2021-04-30T13:06:12.807Z] Server: Docker Engine - Community
[2021-04-30T13:06:12.807Z]  Engine:
[2021-04-30T13:06:12.807Z]   Version:          20.10.3
[2021-04-30T13:06:12.807Z]   API version:      1.41 (minimum version 1.12)
[2021-04-30T13:06:12.807Z]   Go version:       go1.13.15
[2021-04-30T13:06:12.807Z]   Git commit:       46229ca
[2021-04-30T13:06:12.807Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-30T13:06:12.807Z]   OS/Arch:          linux/amd64
[2021-04-30T13:06:12.807Z]   Experimental:     false
[2021-04-30T13:06:12.807Z]  containerd:
[2021-04-30T13:06:12.807Z]   Version:          1.4.4
[2021-04-30T13:06:12.807Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-30T13:06:12.807Z]  runc:
[2021-04-30T13:06:12.807Z]   Version:          1.0.0-rc93
[2021-04-30T13:06:12.807Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-30T13:06:12.807Z]  docker-init:
[2021-04-30T13:06:12.807Z]   Version:          0.19.0
[2021-04-30T13:06:12.807Z]   GitCommit:        de40ad0
[2021-04-30T13:06:12.807Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-30T13:07:34.274Z] Change permissions with write access of all files inside the specific folder
[2021-04-30T13:07:34.363Z] Running in /var/lib/jenkins/workspace/PR-25460-1-0ca880d8-c1a8-4bdf-a1df-b25800ccbcd7
[2021-04-30T13:07:41.935Z] + gsutil --version
[2021-04-30T13:07:43.417Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-30T13:07:43.925Z] + gcloud auth activate-service-account --key-file ****
[2021-04-30T13:07:44.510Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-04-30T13:07:44.988Z] + gsutil -m -q cp -a public-read eC1wYWNrL2VsYXN0aWMtYWdlbnQtcGFja2FnaW5nLWxpbnV4MGU3NGZmMGUxMTc5ZTliOTIwNDVjYWU5OGU0NzQwNzgxNjcwY2VlZg gs://beats-ci-temp/ci/cache/
[2021-04-30T13:07:47.797Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-30T13:07:48.050Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25460/src/github.com/elastic/beats
[2021-04-30T13:07:48.927Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25460
[2021-04-30T13:07:49.067Z] [INFO] getVaultSecret: Getting secrets
[2021-04-30T13:07:49.144Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-30T13:07:50.164Z] + chmod 755 generate-build-data.sh
[2021-04-30T13:07:50.164Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25460/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25460/runs/1 FAILURE 3119496
[2021-04-30T13:07:50.164Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25460/runs/1/steps/?limit=10000 -o steps-info.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	1
Passed	6739
Skipped	24
Total	6764

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

• Name: Build&Test / x-pack/elastic-agent-windows-2012-windows-2012-r2 / TestConfigurableFailed – github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/operation