Skip to content

MinIO

Jump to bottom Edit New page
hgy59 edited this page Sep 8, 2024 · 9 revisions

Breaking Changes in MinIO dataformat

MinIO Package Release 2022.10.24-16 is the latest that supports the FS (FileSystem) data mode. Any MinIO version starting at 2022-10-29T06-21-33Z does not support this format anymore.

If your initial minio package installation was with Version 2022.09.07-15 or later, you already have the new format.

To check if you're affected by this change use one of the two methods:

  • check your buckets manually by looking into each folder on your diskstation. If there is exactly a single file per S3 object you need to migrate your data.
  • check the format in the file format.json located in the .minio.sys folder. If the file has an entry "format": "fs" you need to migrate your data.

Otherwise you're already running a single-node single-drive deployment (format is xl or xl-single) and can upgrade to the new MinIO package in future without a migration.

An automated migration is not available.

Existing minio data structures in FS format must be manually migrated as documented in Migrate from Gateway or Filesystem Mode.

For a regular user it is suggested to run an instance of MinIO on a separate system and migrate your data to this new instance. This system could be a desktop, single board computer or a server. If possible you can mount the storage from your diskstation over the network and use it directly during the migration process. After a successful migration replace the data of your current MinIO installation on your diskstation. Please stop MinIO before replacing your data and restart it afterwards.

Standalone Data Migration

You can migrate the data on a single system, alas this is not recommended for huge bucket data, since two minio instances are running on the same system.

The following migration works only with minio package release 2022.10.24-16 and 2022.09.07-15. Those are the only versions released that work with the old FS data format and initialize a new data structure with the SNSD format.
And it contains a minio-client version that works in the guide below.

Prerequisites

  • You have a running minio installation on your diskstation that was originally installed with minio version 2022.05.19-14 or older.

  • Assuming the existing minio share is named "minio_old" and you migrate to a share named "minio_new"

  • Create a new shared folder named "minio_new" in the Control Panel like this: grafik

    and configure Read/Write Access for the system internal user sc-minio grafik

  • you need ssh access to the diskstation

  • This guide works only with MinIO Package Version 2022.10.24-16 and 2022.09.07-15 (we recommend version 2022.10.24-16 that provides additional information in the upgrade wizard)

Step by Step Guide

  1. Update your MinIO installation with package Version 2022.10.24-16

    1. Use the existing minio share (minio_old) in the wizard page "MinIO configuration" and optionally alter the minio root user and password.

    2. On the next page "MinIO - Migration Notes" you will see the text: "This MinIO installation uses the FS data format. ..."

    3. Continue the update and you will get a running minio installation

  2. You can login into minio with http://{diskstation-ip}:9001 with the minio root credentials and navigate to the buckes, the users etc.

  3. SSH into your diskstation and create and run the script minio_new.sh with the following content:

    #!/bin/bash
export MINIO_ROOT_USER="minio root user name"
export MINIO_ROOT_PASSWORD="minio root user password"
PATH=/var/packages/minio/target/bin:${PATH}
HOME=/var/packages/minio/var
/var/packages/minio/bin/minio server --address :9002 --console-address :9003 --anonymous /volume1/minio_new

    run this script in foreground and you will see the output of minio server.
    To continue, you need another shell

  4. You can now login into new minio with http://{diskstation-ip}:9003 with the credentials defined in the script above There are no buckets nor users configured yet.

  5. Configure the minio-client to access minio_old and minio_new

    For all the following minio-client calls you can omit the definition of HOME and use the current users home folder instead.

    If you never run minio client before, run

    HOME=/var/packages/minio/var minio-client alias ls

    This inital call will show the following output:

    minio-client: Configuration written to `/var/packages/minio/var/.minio-client/config.json`. Please update your access credentials.
minio-client: Successfully created `/var/packages/minio/var/.minio-client/share`.
minio-client: Initialized share uploads `/var/packages/minio/var/.minio-client/share/uploads.json` file.
minio-client: Initialized share downloads `/var/packages/minio/var/.minio-client/share/downloads.json` file.
gcs
  URL       : https://storage.googleapis.com
  AccessKey : YOUR-ACCESS-KEY-HERE
  SecretKey : YOUR-SECRET-KEY-HERE
  API       : S3v2
  Path      : dns

local
  URL       : http://localhost:9000
  AccessKey :
  SecretKey :
  API       :
  Path      : auto

play
  URL       : https://play.min.io
  AccessKey : Q3AM3UQ867SPQQA43P2F
  SecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
  API       : S3v4
  Path      : auto

s3
  URL       : https://s3.amazonaws.com
  AccessKey : YOUR-ACCESS-KEY-HERE
  SecretKey : YOUR-SECRET-KEY-HERE
  API       : S3v4
  Path      : dns

    use the command line (use your own credentials, and disable bash history to avoid recorded credentials)

    set +o history
HOME=/var/packages/minio/var minio-client alias set minio_old http://localhost:9000 "root user" "root password"
HOME=/var/packages/minio/var minio-client alias set minio_new http://localhost:9002 "root user" "root password"
set -o history

    You can see the new entries with:
    HOME=/var/packages/minio/var minio-client alias ls

    The following commands should work now

    list the existing buckets in the old minio data:
    HOME=/var/packages/minio/var minio-client ls minio_old

    new minio will show no buckets at this moment:
    HOME=/var/packages/minio/var minio-client ls minio_new

    Validate stati

    HOME=/var/packages/minio/var minio-client ready minio_old
The cluster is ready
HOME=/var/packages/minio/var minio-client ready minio_new
The cluster is ready

  6. You are now ready for the migration

    1. Migrate configurations
      Original Remarks by MinIO (https://min.io/docs/minio/linux/operations/install-deploy-manage/migrate-fs-gateway.html)
      The following Filesystem mode steps presume the existing MinIO Client supports the needed export commands.
      If it does not, recreate users, policies, lifecycle rules, and buckets manually on the new deployment using the new MinIO Client.

      HOME=/var/packages/minio/var minio-client admin config export minio_old > minio_config.txt

HOME=/var/packages/minio/var minio-client admin config import minio_new < minio_config.txt
Setting new key has been successful.
Please restart your server with `mc admin service restart minio_new`.

      Remarks by MinIO:
      If import reports an error for a configuration key, comment it out with # at the beginning of the relevant line and try again. When you are finished migrating the deployment, verify the current syntax for the target MinIO Server version and set any needed keys manually using mc admin config set.

      Restart Minio

      HOME=/var/packages/minio/var minio-client admin service restart minio_new
Restart command successfully sent to `minio_new`. Type Ctrl-C to quit or wait to follow the status of the restart process.
....
Restarted `minio_new` successfully in 2 seconds

    2. Migrate Bucket Metadata

      export metadata:

      HOME=/var/packages/minio/var minio-client admin cluster bucket export minio_old
minio-client: Bucket metadata successfully downloaded as cluster-metadata.zip

      import metadata:

      HOME=/var/packages/minio/var minio-client admin cluster bucket import minio_new cluster-metadata.zip

x/x buckets were imported successfully.

    3. Migrate IAM data

      export:

      HOME=/var/packages/minio/var minio-client admin cluster iam export minio_old
minio-client: IAM info successfully downloaded as minio_old-iam-info.zip

      import:

      HOME=/var/packages/minio/var minio-client admin cluster iam import minio_new minio_old-iam-info.zip
minio-client: IAM info imported to minio_new from minio_old-iam-info.zip

    4. Migrate bucket contents with mirror command

      run the following command for each bucket (the original doc adds the parameter --watch to mirror while watching for changes, but with this parameter, the command never terminates and you need a separate shell for each bucket)

      HOME=/var/packages/minio/var minio-client mirror --preserve  minio_old/bucket-name minio_new/bucket-name

      The mirror command can take a huge amount of time, especially with a lot of small files.

  7. Complete the migration

    Ensure that all buckets are migrated and no client changed any bucket after the mirror commands completed.

    1. stop the minio package (the server running with the old data) in DSM Package Center.

    2. stop the new server:
      stop the running script of step 3 of this guide with Ctrl-C

    3. Modify the settings of the minio package to use the new share with the migrated data

      edit the the file /var/packages/minio/etc/installer-variables

      replace the name of the share "minio_old" with "minio_new" in the lines with SHARE_PATH and SHARE_NAME
      if you use this guide with minio version 2022.05.19-14 you have to adjust the variable WIZARD_DATA_DIRECTORY

    4. Start minio in Package Center.

    5. Check wheter minio and the web console are running and all data is available.

Add a custom footer
Clone this wiki locally