Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include date separator in the filename prefix of dateRotator to make sure nothing gets purged accidentally #26176

Merged
merged 3 commits into from
Jun 7, 2021

Conversation

kvch
Copy link
Contributor

@kvch kvch commented Jun 7, 2021

What does this PR do?

This PR changes the log file prefix when using date rotation in Beats. Previously the - was not included, so in the list of rotated files every file that started with the configured file prefix were included.

Why is it important?

If the binary is under the same path as the value configured in logging.files.path and logging.files.name, when the number of rotated log files gets bigger than the one configured in loggin.files.keepfiles, the binary is purged on rotation.

The workaround is to put the log files in a separate folder.

Checklist

  • My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 7, 2021
@kvch kvch force-pushed the fix-agent-do-not-purge-binary branch from 6fa9d32 to 363df47 Compare June 7, 2021 14:40
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 7, 2021

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: Pull request #26176 updated

  • Reason: Aborted from #4

  • Start Time: 2021-06-07T14:50:58.997+0000

  • Duration: 4 min 32 sec

  • Commit: 152e8b0

Trends 🧪

Image of Build Times

Log output

Expand to view the last 100 lines of log output

[2021-06-07T14:53:12.937Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-06-07T14:53:12.945Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-06-07T14:53:12.952Z]  > git rev-parse --verify HEAD # timeout=10
[2021-06-07T14:53:12.956Z] No valid HEAD. Skipping the resetting
[2021-06-07T14:53:12.956Z]  > git clean -fdx # timeout=10
[2021-06-07T14:53:12.967Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-06-07T14:53:12.967Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-07T14:53:12.971Z]  > git fetch --no-tags --progress --prune -- git@github.com:elastic/beats.git +refs/pull/26176/head:refs/remotes/origin/PR-26176 +refs/heads/master:refs/remotes/origin/master # timeout=15
[2021-06-07T14:53:13.909Z]  > git config core.sparsecheckout # timeout=10
[2021-06-07T14:53:13.912Z]  > git checkout -f 152e8b0cc1ac418b192069853ec4330f5f12be26 # timeout=15
[2021-06-07T14:53:15.473Z] Merge succeeded, producing 152e8b0cc1ac418b192069853ec4330f5f12be26
[2021-06-07T14:53:15.474Z] Checking out Revision 152e8b0cc1ac418b192069853ec4330f5f12be26 (PR-26176)
[2021-06-07T14:53:15.443Z]  > git remote # timeout=10
[2021-06-07T14:53:15.447Z]  > git config --get remote.origin.url # timeout=10
[2021-06-07T14:53:15.450Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-07T14:53:15.458Z]  > git merge 8bbb26f437df9fdac6f8504fa03268137e04ba6d # timeout=10
[2021-06-07T14:53:15.468Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-06-07T14:53:15.476Z]  > git config core.sparsecheckout # timeout=10
[2021-06-07T14:53:15.479Z]  > git checkout -f 152e8b0cc1ac418b192069853ec4330f5f12be26 # timeout=15
[2021-06-07T14:53:19.711Z] Commit message: "add changelog entry"
[2021-06-07T14:53:19.713Z]  > git rev-list --no-walk 363df4772c4ad56c9c3a5a435a4a0fb97be1618f # timeout=10
[2021-06-07T14:53:19.736Z] Cleaning workspace
[2021-06-07T14:53:20.640Z] Timeout set to expire in 3 hr 0 min
[2021-06-07T14:53:20.653Z] The timestamps step is unnecessary when timestamps are enabled for all Pipeline builds.
[2021-06-07T14:53:20.874Z] [INFO] Number of builds to be searched 10
[2021-06-07T14:53:20.930Z] [INFO] Let's stop build #2. Aborted from #3
[2021-06-07T14:53:21.646Z] [INFO] 'shallow' is forced to be disabled when running on PullRequests
[2021-06-07T14:53:21.660Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26176@tmp
[2021-06-07T14:53:21.676Z] [INFO] gitCheckout: Checkout SCM PR-26176 with default customisation from the Item.
[2021-06-07T14:53:21.700Z] [INFO] Override default checkout
[2021-06-07T14:53:21.736Z] Sleeping for 10 sec
[2021-06-07T14:53:19.738Z]  > git rev-parse --verify HEAD # timeout=10
[2021-06-07T14:53:19.741Z] Resetting working tree
[2021-06-07T14:53:19.741Z]  > git reset --hard # timeout=10
[2021-06-07T14:53:19.821Z]  > git clean -fdx # timeout=10
[2021-06-07T14:53:31.757Z] The recommended git tool is: git
[2021-06-07T14:53:31.807Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
[2021-06-07T14:53:31.817Z] Wiping out workspace first.
[2021-06-07T14:53:31.827Z] Cloning the remote Git repository
[2021-06-07T14:53:31.827Z] Using shallow clone with depth 10
[2021-06-07T14:53:31.827Z] Avoid fetching tags
[2021-06-07T14:53:31.841Z] Cloning repository git@github.com:elastic/beats.git
[2021-06-07T14:53:31.870Z]  > git init /var/lib/jenkins/workspace/Beats_beats_PR-26176@tmp # timeout=10
[2021-06-07T14:53:31.875Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-06-07T14:53:31.875Z]  > git --version # timeout=10
[2021-06-07T14:53:31.878Z]  > git --version # 'git version 2.17.1'
[2021-06-07T14:53:31.878Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-07T14:53:31.882Z]  > git fetch --no-tags --progress -- git@github.com:elastic/beats.git +refs/heads/*:refs/remotes/origin/* # timeout=15
[2021-06-07T14:53:51.988Z] Cleaning workspace
[2021-06-07T14:53:52.000Z] Using shallow fetch with depth 10
[2021-06-07T14:53:52.001Z] Pruning obsolete local branches
[2021-06-07T14:53:52.916Z] Merging remotes/origin/master commit 8bbb26f437df9fdac6f8504fa03268137e04ba6d into PR head commit 152e8b0cc1ac418b192069853ec4330f5f12be26
[2021-06-07T14:53:51.974Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-06-07T14:53:51.977Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-06-07T14:53:51.984Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-06-07T14:53:51.990Z]  > git rev-parse --verify HEAD # timeout=10
[2021-06-07T14:53:51.993Z] No valid HEAD. Skipping the resetting
[2021-06-07T14:53:51.993Z]  > git clean -fdx # timeout=10
[2021-06-07T14:53:52.003Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-06-07T14:53:52.003Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-07T14:53:52.007Z]  > git fetch --no-tags --progress --prune -- git@github.com:elastic/beats.git +refs/pull/26176/head:refs/remotes/origin/PR-26176 +refs/heads/master:refs/remotes/origin/master # timeout=15
[2021-06-07T14:53:52.918Z]  > git config core.sparsecheckout # timeout=10
[2021-06-07T14:53:52.921Z]  > git checkout -f 152e8b0cc1ac418b192069853ec4330f5f12be26 # timeout=15
[2021-06-07T14:53:54.450Z] Merge succeeded, producing 152e8b0cc1ac418b192069853ec4330f5f12be26
[2021-06-07T14:53:54.451Z] Checking out Revision 152e8b0cc1ac418b192069853ec4330f5f12be26 (PR-26176)
[2021-06-07T14:53:55.158Z] Commit message: "add changelog entry"
[2021-06-07T14:53:55.158Z] Cleaning workspace
[2021-06-07T14:53:54.427Z]  > git remote # timeout=10
[2021-06-07T14:53:54.430Z]  > git config --get remote.origin.url # timeout=10
[2021-06-07T14:53:54.433Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-06-07T14:53:54.436Z]  > git merge 8bbb26f437df9fdac6f8504fa03268137e04ba6d # timeout=10
[2021-06-07T14:53:54.446Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-06-07T14:53:54.453Z]  > git config core.sparsecheckout # timeout=10
[2021-06-07T14:53:54.456Z]  > git checkout -f 152e8b0cc1ac418b192069853ec4330f5f12be26 # timeout=15
[2021-06-07T14:53:55.160Z]  > git rev-parse --verify HEAD # timeout=10
[2021-06-07T14:53:55.163Z] Resetting working tree
[2021-06-07T14:53:55.163Z]  > git reset --hard # timeout=10
[2021-06-07T14:53:55.240Z]  > git clean -fdx # timeout=10
[2021-06-07T14:53:56.432Z] Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[2021-06-07T14:53:57.003Z] + git fetch https://****:****@github.com/elastic/beats.git +refs/pull/*/head:refs/remotes/origin/pr/*
[2021-06-07T14:54:23.624Z] Sending interrupt signal to process
[2021-06-07T14:54:29.164Z] script returned exit code 128
[2021-06-07T14:54:29.181Z] [WARN] gitCmd failed, further details in the archived file '.git/fetch.log'
[2021-06-07T14:54:29.212Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26176@tmp/.git
[2021-06-07T14:54:29.262Z] Archiving artifacts
[2021-06-07T14:54:29.587Z] Stage "Lint" skipped due to earlier failure(s)
[2021-06-07T14:54:29.624Z] Stage "Build&Test" skipped due to earlier failure(s)
[2021-06-07T14:54:29.659Z] Stage "Extended" skipped due to earlier failure(s)
[2021-06-07T14:54:29.693Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-06-07T14:54:29.727Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-06-07T14:54:29.783Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-26176/src/github.com/elastic/beats
[2021-06-07T14:54:30.220Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-26176
[2021-06-07T14:54:30.315Z] [INFO] getVaultSecret: Getting secrets
[2021-06-07T14:54:30.360Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-06-07T14:54:31.123Z] + chmod 755 generate-build-data.sh
[2021-06-07T14:54:31.123Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26176/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26176/runs/3 ABORTED 211864
[2021-06-07T14:54:31.123Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26176/runs/3/steps/?limit=10000 -o steps-info.json
[2021-06-07T14:54:31.123Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-26176/runs/3/tests/?status=FAILED -o tests-errors.json
[2021-06-07T14:54:31.374Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-06-07T14:54:32.285Z] Retry 2/3 exited 22, retrying in 2 seconds...

@kvch kvch marked this pull request as ready for review June 7, 2021 14:47
@kvch kvch changed the title Do not remove binary when log files are purged Include date separator in the filename prefix of dateRotator to make sure nothing gets purged accidentally Jun 7, 2021
Copy link
Contributor

@michalpristas michalpristas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and fixes issue with disappearing binaries

@kvch kvch added backport-v7.13.0 Automated backport with mergify backport-v7.14.0 Automated backport with mergify Team:Elastic-Agent Label for the Agent team labels Jun 7, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 7, 2021
@kvch kvch merged commit 447bac9 into elastic:master Jun 7, 2021
mergify bot pushed a commit that referenced this pull request Jun 7, 2021
…e sure nothing gets purged accidentally (#26176)

## What does this PR do?

This PR changes the log file prefix when using date rotation in Beats. Previously the `-` was not included, so in the list of rotated files every file that started with the configured file prefix were included.

## Why is it important?

If the binary is under the same path as the value configured in `logging.files.path` and `logging.files.name`, when the number of rotated log files gets bigger than the one configured in `loggin.files.keepfiles`, the binary is purged on rotation.

The workaround is to put the log files in a separate folder.

(cherry picked from commit 447bac9)
mergify bot pushed a commit that referenced this pull request Jun 7, 2021
…e sure nothing gets purged accidentally (#26176)

## What does this PR do?

This PR changes the log file prefix when using date rotation in Beats. Previously the `-` was not included, so in the list of rotated files every file that started with the configured file prefix were included.

## Why is it important?

If the binary is under the same path as the value configured in `logging.files.path` and `logging.files.name`, when the number of rotated log files gets bigger than the one configured in `loggin.files.keepfiles`, the binary is purged on rotation.

The workaround is to put the log files in a separate folder.

(cherry picked from commit 447bac9)
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #26176 updated

  • Start Time: 2021-06-07T14:52:21.353+0000

  • Duration: 129 min 33 sec

  • Commit: b2018be

Test stats 🧪

Test Results
Failed 0
Passed 47408
Skipped 5253
Total 52661

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 47408
Skipped 5253
Total 52661

kvch added a commit that referenced this pull request Jun 8, 2021
…e sure nothing gets purged accidentally (#26176)

## What does this PR do?

This PR changes the log file prefix when using date rotation in Beats. Previously the `-` was not included, so in the list of rotated files every file that started with the configured file prefix were included.

## Why is it important?

If the binary is under the same path as the value configured in `logging.files.path` and `logging.files.name`, when the number of rotated log files gets bigger than the one configured in `loggin.files.keepfiles`, the binary is purged on rotation.

The workaround is to put the log files in a separate folder.

(cherry picked from commit 447bac9)
kvch added a commit that referenced this pull request Jun 8, 2021
…e sure nothing gets purged accidentally (#26176)

## What does this PR do?

This PR changes the log file prefix when using date rotation in Beats. Previously the `-` was not included, so in the list of rotated files every file that started with the configured file prefix were included.

## Why is it important?

If the binary is under the same path as the value configured in `logging.files.path` and `logging.files.name`, when the number of rotated log files gets bigger than the one configured in `loggin.files.keepfiles`, the binary is purged on rotation.

The workaround is to put the log files in a separate folder.

(cherry picked from commit 447bac9)
kvch added a commit that referenced this pull request Jun 8, 2021
…e sure nothing gets purged accidentally (#26176) (#26179)

## What does this PR do?

This PR changes the log file prefix when using date rotation in Beats. Previously the `-` was not included, so in the list of rotated files every file that started with the configured file prefix were included.

## Why is it important?

If the binary is under the same path as the value configured in `logging.files.path` and `logging.files.name`, when the number of rotated log files gets bigger than the one configured in `loggin.files.keepfiles`, the binary is purged on rotation.

The workaround is to put the log files in a separate folder.

(cherry picked from commit 447bac9)

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
kvch added a commit that referenced this pull request Jun 8, 2021
…e sure nothing gets purged accidentally (#26176) (#26178)

## What does this PR do?

This PR changes the log file prefix when using date rotation in Beats. Previously the `-` was not included, so in the list of rotated files every file that started with the configured file prefix were included.

## Why is it important?

If the binary is under the same path as the value configured in `logging.files.path` and `logging.files.name`, when the number of rotated log files gets bigger than the one configured in `loggin.files.keepfiles`, the binary is purged on rotation.

The workaround is to put the log files in a separate folder.

(cherry picked from commit 447bac9)

Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v7.13.0 Automated backport with mergify backport-v7.14.0 Automated backport with mergify bug Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants