Skip to content

mineos‐7

Jump to bottom
Allan Roger Reid edited this page Apr 12, 2024 · 6 revisions

See https://github.com/allanrogerr/public/wiki/kes-walkthru-(bare-metal-minio%E2%80%90kes%E2%80%90vault) for initial setups

After kes-walkthru using KES configuration:

cache:
  expiry:
    any: 600s
    unused: 30s
    offline: 600s

Kill KMS process

pkill vault

==> Vault shutdown triggered
2024-04-10T21:50:52.295Z [INFO]  core: marked as sealed
2024-04-10T21:50:52.295Z [INFO]  core: pre-seal teardown starting
2024-04-10T21:50:52.295Z [INFO]  rollback: stopping rollback manager
2024-04-10T21:50:52.296Z [INFO]  core: pre-seal teardown complete
2024-04-10T21:50:52.296Z [INFO]  core: stopping cluster listeners
2024-04-10T21:50:52.296Z [INFO]  core.cluster-listener: forwarding rpc listeners stopped
2024-04-10T21:50:52.683Z [INFO]  core.cluster-listener: rpc listeners successfully shut down
2024-04-10T21:50:52.684Z [INFO]  core: cluster listeners successfully shut down
2024-04-10T21:50:52.684Z [INFO]  core: vault is sealed

Wait for eviction (after 30 secs)

Attempt to restart minio. Observe failure

ubuntu@kes-minio:~$ mc admin service restart minio
Restart command successfully sent to `minio`. Type Ctrl-C to quit or wait to follow the status of the restart process.
┌────────────────┬────────┐
│ HOST           │ STATUS │
├────────────────┼────────┤
│ 127.0.0.1:9000 │ ✔      │
└────────────────┴────────┘

...!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Fix

Install go

cd ~ && mkdir go && cd go && wget http://go.dev/dl/go1.21.9.linux-amd64.tar.gz && sudo rm -rf /usr/local/bin/go && sudo tar -C /usr/local/bin -xzf go1.21.9.linux-amd64.tar.gz 
/usr/local/bin/go/bin/go version

Compile and install new version of minio

sudo apt-get install git -y 
cd ~ && mkdir github -p && cd github && git clone http://github.com/allanrogerr/minio.git && cd minio && git checkout cache-kms-master-key
CGO_ENABLED=0 /usr/local/bin/go/bin/go build -tags kqueue -trimpath -o minio

Start vault server

sudo setcap cap_ipc_lock=+ep $(readlink -f $(which vault))
vault server -config vault-config.json > out.log &
tail -f out.log

Unseal Vault Server

vault status
vault operator unseal 3JR9AKcJG53S6OGhs6g5g67cs4R2rSbb/oApOobwZEp4
vault operator unseal RlqgGI+mrUNR7883UzdVsyzT4jeLhRajBTXBb64y+MUb
vault operator unseal KfmtXtXagOSCpjDDCAQS+HMBVB/N8idpKeleuUs0YrKd
vault status

Start KES Server

pkill kes
./kes server --config config.yml > out.log &
tail -f out.log

Restart minio

pkill minio 
cd ~
MINIO_KMS_KES_ENDPOINT=https://10.214.226.181:9073 \
MINIO_KMS_KES_KEY_NAME=minio-key \
MINIO_KMS_KES_CAPATH=public.crt \
MINIO_KMS_KES_CERT_FILE=client.crt \
MINIO_KMS_KES_KEY_FILE=client.key \
CI=on \
~/github/minio/minio server /tmp/data --certs-dir ~/.minio/certs --address :9000 --console-address :9090 > out.log &
tail -f out.log

Kill KMS process

pkill vault

==> Vault shutdown triggered
2024-04-10T21:50:52.295Z [INFO]  core: marked as sealed
2024-04-10T21:50:52.295Z [INFO]  core: pre-seal teardown starting
2024-04-10T21:50:52.295Z [INFO]  rollback: stopping rollback manager
2024-04-10T21:50:52.296Z [INFO]  core: pre-seal teardown complete
2024-04-10T21:50:52.296Z [INFO]  core: stopping cluster listeners
2024-04-10T21:50:52.296Z [INFO]  core.cluster-listener: forwarding rpc listeners stopped
2024-04-10T21:50:52.683Z [INFO]  core.cluster-listener: rpc listeners successfully shut down
2024-04-10T21:50:52.684Z [INFO]  core: cluster listeners successfully shut down
2024-04-10T21:50:52.684Z [INFO]  core: vault is sealed

Wait for eviction (after 30 secs)

Attempt to restart minio. Observe success

ubuntu@kes-minio:~$ mc admin service restart minio
Restart command successfully sent to `minio`. Type Ctrl-C to quit or wait to follow the status of the restart process.
┌────────────────┬────────┐
│ HOST           │ STATUS │
├────────────────┼────────┤
│ 127.0.0.1:9000 │ ✔      │
└────────────────┴────────┘

...!
Restarted `minio` successfully in 566 milliseconds

Note

  • MinIO starts and decrypts its IAM data using the key MINIO_KMS_KES_KEY_NAME. Therefore, KES has to fetch the key from the KMS backend (if not cached already).

  • After some time KES evicts the key MINIO_KMS_KES_KEY_NAME from the cache since its not used by MinIO. See usage of .cache.expiry.unused. .cache.expiry.any is out of scope since it always purges all cache if KES verifies the KMS to be online. This is a race condition. .cache.expiry.offline is also out of scope since it always purges all cache if KES verifies the KMS is offline. This is a race condition.

  • The KMS becomes temp. unavailable (network issue, maintenance etc.)

  • MinIO gets restarted. This now fails because KES cannot find the MINIO_KMS_KES_KEY_NAME in its cache and it also cannot fetch it from the KMS backend because the KMS is not available.

  • This fix tests for the ability to encrypt and decrypt using the cached KMS key in KES. It also prevents KES from considering the KMS default key as unused. When both points as in place, and the cached key has not been purged due to a race with .cache.expiry.any or .cache.expiry.offline, then MinIO is OK to startup.

Tests - Modify the KES configuration

Test 0

cache:
  expiry:
    any: 600s
    unused: 30s
    offline: 600s
  • Start vault, verify KES running, start MinIO, create and encrypt bucket with default key, upload data, kill vault, restart minio after 30s
  • Observe minio starts up
image
  • Validate existing minio encrypted data is accessible
image
  • Validate new minio encrypted data is uploadable and accessible
image
  • Validate encryption can be removed and new minio data is uploadable and accessible
image image

Test 1

cache:
  expiry:
    any: 600s
    unused: 20s
    offline: 600s
  • Start vault, verify KES running, start MinIO, create and encrypt bucket, upload data, kill vault, restart minio after 10s
image
  • Observe minio starts up sometimes due to the inherent race condition
image
  • Validate existing minio encrypted data is accessible
image
  • Validate new minio encrypted data is uploadable and accessible
image image
  • Validate encryption can be removed and new minio data is uploadable and accessible
image image

Test 2

cache:
  expiry:
    any: 600s
    unused: 10s
    offline: 600s
  • Start vault, verify KES running, start MinIO, create and encrypt bucket, upload data, kill vault, restart minio after 10s
image
  • Observe minio starts up sometimes due to the inherent race condition
image

Test 3

cache:
  expiry:
    any: 600s
    unused: 5s
    offline: 600s
  • Start vault, verify KES running, start MinIO, create and encrypt bucket, upload data, kill vault, restart minio after 5s
image
  • Observe minio does not start up because KES cache was purged before it could be refreshed by minio
image
Clone this wiki locally