分布式安装部署MinIO

官方文档地址：http://docs.minio.org.cn/docs/master/distributed-minio-quickstart-guide

前提条件：分布式Minio至少需要4个硬盘，使用分布式Minio自动引入了纠删码功能。

注意:

• 分布式Minio里所有的节点需要有同样的access秘钥和secret秘钥，这样这些节点才能建立联接。为了实现这个，你需要在执行minio server命令之前，先将access秘钥和secret秘钥export成环境变量。
• 分布式Minio使用的磁盘里必须是干净的，里面没有数据。
• 下面示例里的IP仅供示例参考，你需要改成你真实用到的IP和文件夹路径。
• 分布式Minio里的节点时间差不能超过3秒，你可以使用NTP 来保证时间一致。
• 在Windows下运行分布式Minio处于实验阶段，请悠着点使用。

这里演示使用一台主机上的4块磁盘，每个磁盘1G

官方步骤：

export MINIO_ACCESS_KEY=<ACCESS_KEY>
export MINIO_SECRET_KEY=<SECRET_KEY>
minio server http://192.168.1.11/export1 http://192.168.1.12/export2 \
               http://192.168.1.13/export3 http://192.168.1.14/export4 \
               http://192.168.1.15/export5 http://192.168.1.16/export6 \
               http://192.168.1.17/export7 http://192.168.1.18/export8

实际操作步骤：

1.在一台主机上创建4个不同的路径模拟4个不同的磁盘来进行

从执行结果来看，这种方式行不通

export MINIO_ACCESS_KEY=admin
export MINIO_SECRET_KEY=12345678

mkdir -p /mnt/{data1,data2,data3,data4}
/usr/local/minio/bin/minio server http://192.168.20.102/mnt/data1 http://192.168.20.102/mnt/data2 http://192.168.20.102/mnt/data3 http://192.168.20.102/mnt/data4 --console-address ":9001"

# 操作完毕后提示信息如下
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
Formatting 1st pool, 1 set(s), 4 drives per set.
WARNING: Host 192.168.20.102 has more than 2 drives of set. A host failure will result in data becoming unavailable.

API: SYSTEM()
Time: 09:24:00 UTC 03/10/2022
Error: Disk `/mnt/data1` is part of root disk, will not be used (*errors.errorString)
       7: cmd/erasure-sets.go:1257:cmd.markRootDisksAsDown()
       6: cmd/format-erasure.go:792:cmd.initFormatErasure()
       5: cmd/prepare-storage.go:222:cmd.connectLoadInitFormats()
       4: cmd/prepare-storage.go:306:cmd.waitForFormatErasure()
       3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
       2: cmd/server-main.go:640:cmd.newObjectLayer()
       1: cmd/server-main.go:491:cmd.serverMain()

API: SYSTEM()
Time: 09:24:00 UTC 03/10/2022
Error: Disk `/mnt/data2` is part of root disk, will not be used (*errors.errorString)
       7: cmd/erasure-sets.go:1257:cmd.markRootDisksAsDown()
       6: cmd/format-erasure.go:792:cmd.initFormatErasure()
       5: cmd/prepare-storage.go:222:cmd.connectLoadInitFormats()
       4: cmd/prepare-storage.go:306:cmd.waitForFormatErasure()
       3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
       2: cmd/server-main.go:640:cmd.newObjectLayer()
       1: cmd/server-main.go:491:cmd.serverMain()

API: SYSTEM()
Time: 09:24:00 UTC 03/10/2022
Error: Disk `/mnt/data3` is part of root disk, will not be used (*errors.errorString)
       7: cmd/erasure-sets.go:1257:cmd.markRootDisksAsDown()
       6: cmd/format-erasure.go:792:cmd.initFormatErasure()
       5: cmd/prepare-storage.go:222:cmd.connectLoadInitFormats()
       4: cmd/prepare-storage.go:306:cmd.waitForFormatErasure()
       3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
       2: cmd/server-main.go:640:cmd.newObjectLayer()
       1: cmd/server-main.go:491:cmd.serverMain()

API: SYSTEM()
Time: 09:24:00 UTC 03/10/2022
Error: Disk `/mnt/data4` is part of root disk, will not be used (*errors.errorString)
       7: cmd/erasure-sets.go:1257:cmd.markRootDisksAsDown()
       6: cmd/format-erasure.go:792:cmd.initFormatErasure()
       5: cmd/prepare-storage.go:222:cmd.connectLoadInitFormats()
       4: cmd/prepare-storage.go:306:cmd.waitForFormatErasure()
       3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
       2: cmd/server-main.go:640:cmd.newObjectLayer()
       1: cmd/server-main.go:491:cmd.serverMain()
Formatting 1st pool, 1 set(s), 4 drives per set.
WARNING: Host 192.168.20.102 has more than 2 drives of set. A host failure will result in data becoming unavailable.

改进步骤：

# 根据上一步的执行提示，更新设置的环境变量名
export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=12345678

# 新加四块磁盘，每个磁盘空间1G，不分区，直接格式化后挂载

# 不重启，直接刷新磁盘数据总线，获取新加的磁盘
for host in $(ls /sys/class/scsi_host) ; do echo "- - -" > /sys/class/scsi_host/$host/scan; done

# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   40G  0 disk
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   39G  0 part
  ├─centos-root 253:0    0 35.1G  0 lvm  /
  └─centos-swap 253:1    0  3.9G  0 lvm  [SWAP]
sdb               8:16   0    1G  0 disk
sdc               8:32   0    1G  0 disk
sdd               8:48   0    1G  0 disk
sde               8:64   0    1G  0 disk 

[root@myrabbit3 ~]# mkfs.ext4 /dev/sdb
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdb is entire device, not just one partition!
无论如何也要继续? (y,n) y
文件系统标签=
OS type: Linux
块大小=4096 (log=2)
分块大小=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
第一个数据块=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Allocating group tables: 完成
正在写入inode表: 完成
Creating journal (8192 blocks): 完成
Writing superblocks and filesystem accounting information: 完成

[root@myrabbit3 ~]# mkfs.ext4 /dev/sdc
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdc is entire device, not just one partition!
无论如何也要继续? (y,n) y
文件系统标签=
OS type: Linux
块大小=4096 (log=2)
分块大小=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
第一个数据块=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Allocating group tables: 完成
正在写入inode表: 完成
Creating journal (8192 blocks): 完成
Writing superblocks and filesystem accounting information: 完成

[root@myrabbit3 ~]# mkfs.ext4 /dev/sdd
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdd is entire device, not just one partition!
无论如何也要继续? (y,n) y
文件系统标签=
OS type: Linux
块大小=4096 (log=2)
分块大小=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
第一个数据块=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Allocating group tables: 完成
正在写入inode表: 完成
Creating journal (8192 blocks): 完成
Writing superblocks and filesystem accounting information: 完成

[root@myrabbit3 ~]# mkfs.ext4 /dev/sde
mke2fs 1.42.9 (28-Dec-2013)
/dev/sde is entire device, not just one partition!
无论如何也要继续? (y,n) y
文件系统标签=
OS type: Linux
块大小=4096 (log=2)
分块大小=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
第一个数据块=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Allocating group tables: 完成
正在写入inode表: 完成
Creating journal (8192 blocks): 完成
Writing superblocks and filesystem accounting information: 完成

[root@myrabbit3 ~]# mount /dev/sdb /mnt/data1/
[root@myrabbit3 ~]# mount /dev/sdc /mnt/data2/
[root@myrabbit3 ~]# mount /dev/sdd /mnt/data3/
[root@myrabbit3 ~]# mount /dev/sde /mnt/data4/

# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   40G  0 disk
├─sda1            8:1    0    1G  0 part /boot
└─sda2            8:2    0   39G  0 part
  ├─centos-root 253:0    0 35.1G  0 lvm  /
  └─centos-swap 253:1    0  3.9G  0 lvm  [SWAP]
sdb               8:16   0    1G  0 disk /mnt/data1
sdc               8:32   0    1G  0 disk /mnt/data2
sdd               8:48   0    1G  0 disk /mnt/data3
sde               8:64   0    1G  0 disk /mnt/data4

# 记得把磁盘挂载信息写道/etc/fstab文件中

# /usr/local/minio/bin/minio server http://192.168.20.102/mnt/data1 http://192.168.20.102/mnt/data2 http://192.168.20.102/mnt/data3 http://192.168.20.102/mnt/data4 --console-address ":9001"
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
Automatically configured API requests per node based on available memory on the system: 55
Status:         4 Online, 0 Offline.
API: http://192.168.20.102:9000  http://127.0.0.1:9000
RootUser: admin
RootPass: 12345678 

Console: http://192.168.20.102:9001 http://127.0.0.1:9001
RootUser: admin
RootPass: 12345678 

Command-line: https://docs.min.io/docs/minio-client-quickstart-guide
   $ mc alias set myminio http://192.168.20.102:9000 admin 12345678

Documentation: https://docs.min.io

扩展现有的分布式集群

例如我们是通过区的方式启动MinIO集群，命令行如下：

export MINIO_ACCESS_KEY=<ACCESS_KEY>
export MINIO_SECRET_KEY=<SECRET_KEY>
minio server http://host{1...32}/export{1...32}

MinIO支持通过命令，指定新的集群来扩展现有集群（纠删码模式），命令行如下：

export MINIO_ACCESS_KEY=<ACCESS_KEY>
export MINIO_SECRET_KEY=<SECRET_KEY>
minio server http://host{1...32}/export{1...32} http://host{33...64}/export{1...32}

新的对象上传请求会自动分配到最少使用的集群上。通过以上扩展策略，您就可以按需扩展您的集群。重新配置后重启集群，会立即在集群中生效，并对现有集群无影响。如上命令中，我们可以把原来的集群看做一个区，新增集群看做另一个区，新对象按每个区域中的可用空间比例放置在区域中。在每个区域内，基于确定性哈希算法确定位置。

说明: 您添加的每个区域必须具有与原始区域相同的磁盘数量（纠删码集）大小，以便维持相同的数据冗余SLA。 例如，第一个区有8个磁盘，您可以将集群扩展为16个、32个或1024个磁盘的区域，您只需确保部署的SLA是原始区域的倍数即可。

多个节点的存储容量和就是分布式Minio的存储容量。