Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix etcd script to delete other files when there is no space left on device #1617

Merged
merged 1 commit into from
Nov 21, 2022

Conversation

24sama
Copy link
Collaborator

@24sama 24sama commented Nov 21, 2022

What type of PR is this?

/kind bug

What this PR does / why we need it:

Fix etcd script to delete other files when there is no space left on device.
There is a very extreme scenario but also very deadly, this script incorrectly deletes / directory files when the node does not have space to create a directory.

  1. And for the existing cluster installed by kk, suggest manually editing the script:
$ vi /usr/local/bin/kube-scripts/etcd-backup.sh
  1. modify the script like the below:
    1. add set -o xxx
    2. replace the : after the cd command with &&
#!/bin/bash

set -o errexit
set -o nounset
set -o pipefail

ETCDCTL_PATH='/usr/local/bin/etcdctl'
ENDPOINTS='https://192.168.100.3:2379'
ETCD_DATA_DIR="/var/lib/etcd"
BACKUP_DIR="/var/backups/kube_etcd/etcd-$(date +%Y-%m-%d-%H-%M-%S)"
KEEPBACKUPNUMBER='6'
ETCDBACKUPSCIPT='/usr/local/bin/kube-scripts'

ETCDCTL_CERT="/etc/ssl/etcd/ssl/admin-node1.pem"
ETCDCTL_KEY="/etc/ssl/etcd/ssl/admin-node1-key.pem"
ETCDCTL_CA_FILE="/etc/ssl/etcd/ssl/ca.pem"

[ ! -d $BACKUP_DIR ] && mkdir -p $BACKUP_DIR

export ETCDCTL_API=2;$ETCDCTL_PATH backup --data-dir $ETCD_DATA_DIR --backup-dir $BACKUP_DIR

sleep 3

{
export ETCDCTL_API=3;$ETCDCTL_PATH --endpoints="$ENDPOINTS" snapshot save $BACKUP_DIR/snapshot.db \
                                   --cacert="$ETCDCTL_CA_FILE" \
                                   --cert="$ETCDCTL_CERT" \
                                   --key="$ETCDCTL_KEY"
} > /dev/null 

sleep 3

cd $BACKUP_DIR/../ && ls -lt |awk '{if(NR > '$KEEPBACKUPNUMBER'){print "rm -rf "$9}}'|sh
  1. reload the new script:
$ systemctl daemon-reload

Which issue(s) this PR fixes:

Fixes #

Special notes for reviewers:

Does this PR introduced a user-facing change?


Additional documentation, usage docs, etc.:


@ks-ci-bot ks-ci-bot added the kind/bug Categorizes issue or PR as related to a bug. label Nov 21, 2022
@ks-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 24sama

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ks-ci-bot ks-ci-bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 21, 2022
…device

Signed-off-by: 24sama <jacksama@foxmail.com>
@pixiake
Copy link
Collaborator

pixiake commented Nov 21, 2022

/lgtm

@ks-ci-bot ks-ci-bot added the lgtm Indicates that a PR is ready to be merged. label Nov 21, 2022
@ks-ci-bot
Copy link
Collaborator

LGTM label has been added.

Git tree hash: 22864033defb2c9691ca295c8a0173cd0836a05e

@ks-ci-bot ks-ci-bot merged commit 84369a3 into kubesphere:master Nov 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants