minio: Module broken for existing users due to legacy fs backend no longer being supported

[pinpox]

Describe the bug

The minio module is currently broken and fails to start with this error:

× minio.service - Minio Object Storage
     Loaded: loaded (/etc/systemd/system/minio.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2022-11-02 19:22:56 CET; 15h ago
   Duration: 768ms
    Process: 362940 ExecStart=/nix/store/8jv8bxzfxcs7l22shjmgz4pb3lffc827-minio-2022-10-29T06-21-33Z/bin/minio server --json --address 192.168.7.4:9000 --console-address 192.168.7.4:9001 --config-dir=/mnt/data/minio/config /mnt/data/minio/data (code=exited, status=1/FAILURE)
   Main PID: 362940 (code=exited, status=1/FAILURE)
         IP: 257B in, 196B out
        CPU: 1.013s

Nov 02 19:22:55 birne systemd[1]: Started Minio Object Storage.
Nov 02 19:22:56 birne minio[362940]: {"level":"FATAL","errKind":"","time":"2022-11-02T18:22:56.203200891Z","message":"Invalid arguments specified","error":{"message":"Invalid arguments specified","source":["cmd/prepare-storage.go:290:cmd.waitForFormatErasure()"]}}

The problem seems to be --config-dir which is no longer supported, there might be other changes apart from it, there seem to have been some changes in the CLI upstream.

Steps To Reproduce

Steps to reproduce the behavior:

1. enable minio, e.g.:

    services.minio = {
      enable = true;
      configDir = "/mnt/data/minio/config";
    };

3. rebuild
4. systemctl status minio.service

Expected behavior

Minio is running

Notify maintainers

@bachp @superherointj

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.67, NixOS, 22.11 (Raccoon), 22.11.20221031.d40fea9`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.11.0`
 - channels(pinpox): `"home-manager"`
 - nixpkgs: `/nix/store/c2la594kmk73rs39s8x9k49bmlmxshi5-source`

[flokli]

The bump causing the breakage was reverted in #199279.

[bachp]

I will try to take a look at this on the weekend. I also need to check if the the test under nixos/tests/minio.nix would catch this.

[bachp]

The problem is not --config-dir, it's only deprecated but still there.

It seems there is a different issue, but it only affects users with existing data. That's why the test didn't catch it.

From the documentation

Important

RELEASE.2022-10-29T06-21-33Z fully removes the deprecated Gateway/Filesystem backends. MinIO returns an error if it starts up and detects existing Filesystem backend files.

To migrate from an FS-backend deployment, use mc mirror or mc cp to copy your data over to a new MinIO SNSD deployment. You should also recreate any necessary users, groups, policies, and bucket configurations on the SNSD deployment.

It seems there is no easy way except for the user to manually do the migration.

Any ideas?

[bachp]

It seems the only option is to manually migrate the data from the old instance to the new via mc mirroror mc copy.

One way to tackle this is to keep the current minio version around as minio_legacy_fsor similar. Then add an entry to the breaking change of the nixos with some basic migration instructions which would look something like follow:

1. Set services.minio.package = pkgs.mino_legacy_fs;
2. Do `mc mirror server/ ./local
3. Set services.minio.package = pkgs.mino;
4. Do `mc mirror ./local server/

This way people have the chance to migrate their data.

First I tought we should do this before 22.11 to avoid issues with new installations, but I don't think this is the case as minio seems to create the new backend already with the current version if non exists: https://min.io/docs/minio/linux/operations/install-deploy-manage/deploy-minio-single-node-single-drive.html#pre-existing-data

[pinpox]

The steps listed by @bachp worked for me aswell, but they only copy the data and not the configuration from minio such as users, policies, access keys and buckets. For that there are a few more steps listed in the minio migration guide

This is what I did, leaving it documented here in case anyone comes along and has to do it on nixos:

The following assumes minio is listening on localhost and you have minio-client available (e.g. via nix-shell -p minio-client)

0. Set services.minio.package = pkgs.mino_legacy_fs; so the existing minio starts

1. Create alias. The access key and secrets key are the same as the environment variables MINIO_ROOT_USER and MINIO_ROOT_PASSWORD. I called the instance myminio here.

mc alias set myminio http://127.0.0.1:9000 access-key my-secret-key

2. Export data to local folder

mkdir ./minio-export
mc mirror myminio ./minio-export

3a. Export configuration

mc admin config export myminio > config.txt

3b. Fix configuration. With a text-editor remove all occurances of replication_priority=auto, the value seems to be depreated and the import will fail otherwise.

4. Export bucket metadata

mc admin cluster bucket export myminio

5. Export IAM settings

mc admin cluster iam export myminio

6. Set services.minio.package = pkgs.mino; and start new configuration with a new data-dir and config-dir set

7. Import everything

mc admin config import myminio < config.txt
mc admin cluster bucket import myminio cluster-metadata.zip
mc admin cluster iam import myminio alias-iam-info.zip

7. I had to copy over the buckets data individually.
   Do mc mirror ./minio-export/bucket1 myminio/bucket1 for each bucket.

Hope it helps someone, took me a bit to get it working.

[bachp]

@pinpox Thanks for the detailed guide. Would you mind adding this to the release notes? I think there it might be the most helpful.

[pinpox]

You sure that isn't to specific? This issue is linked in the release notes anyway here

[bachp]

You are right, I think the information in the release notes with a link to the issue and the upstream documentation is enough.