KingbaseES Clusterware 高可用案例之---构建iSCSI共享存储

案例说明：

在KingbaseES Clusterware高可用的架构中，集群节点需要访问共享的存储设备，可以使用FC SAN、iscsi SAN、NAS等存储设备。本案例详细描述了，在Linux系统下构建 iscsi存储服务器及客户端访问的过程，为部署KingbaseES Clusterware做好相应的准备。

如下所示：KingbaseES Clusterware架构

系统架构节点信息：

ISCSI SAN存储架构：

一、iscsi server配置

Tips：

iscsi server提供共享存储，并接受client的访问。

1、系统环境信息

[root@node1 ~]# cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)

2、server端软件包安装及启动服务

#软件包安装
[root@node1 ~]# yum -y install targetd targetcli

#启动server端服务
[root@node1 ~]# systemctl start target

[root@node1 ~]# systemctl enable target
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.

[root@node1 ~]# systemctl status target
● target.service - Restore LIO kernel target configuration
   Loaded: loaded (/usr/lib/systemd/system/target.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2022-11-23 15:46:17 CST; 17s ago
 Main PID: 22094 (code=exited, status=0/SUCCESS)

Nov 23 15:46:15 node1 systemd[1]: Starting Restore LIO kernel target configuration...
Nov 23 15:46:17 node1 target[22094]: No saved config file at /etc/target/saveconfig.json, ok, exiting
Nov 23 15:46:17 node1 systemd[1]: Started Restore LIO kernel target configuration.

3、iscsi server端配置

Tips：

当执行targetcli出现以下错误时，需要安装python-rtslib软件。

安装python-rtslib软件：

[root@node1 ~]# yum install -y python-rtslib

server端配置工具：

[root@node1 ~]# targetcli
Warning: Could not load preferences file /root/.targetcli/prefs.bin.
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> help

AVAILABLE COMMANDS
==================
The following commands are available in the
current path:

  - bookmarks action [bookmark]
  - cd [path]
  - clearconfig [confirm]
  - exit
  - get [group] [parameter...]
  - help [topic]
  - ls [path] [depth]
  - pwd
  - refresh
  - restoreconfig [savefile] [clear_existing] [target] [storage_object]
  - saveconfig [savefile]
  - sessions [action] [sid]
  - set [group] [parameter=value...]
  - status
  - version

4、server端添加共享磁盘

[root@node1 ~]# fdisk -l
 .......
Disk /dev/sdb: 21.6 GB, 21579694080 bytes, 42147840 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

5、server端iscsi配置

1）创建要发布的磁盘（块设备LUN)

[root@node1 ~]# targetcli
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 0]
  o- loopback ......................................................................................................... [Targets: 0]

/> /backstores/block create server0.disk1 /dev/sdb
Created block storage object server0.disk1 using /dev/sdb.
#注：server0.disk1 是给/dev/sdb转化成块的名字。

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
  | | o- server0.disk1 ................................................................. [/dev/sdb (20.1GiB) write-thru deactivated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 0]
  o- loopback ......................................................................................................... [Targets: 0]

2）创建iqn名字即创建iscsi对象

iqn:iSCSI Qualified Name（ISCSI 限定名称）

/> /iscsi create iqn.test.com.bqdn:disk1
WWN not valid as: iqn, naa, eui
# 注意：iqn名称格式

/> /iscsi create iqn.2022-11.com.bqdn:disk1
Created target iqn.2022-11.com.bqdn:disk1.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.

o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
  | | o- server0.disk1 ................................................................. [/dev/sdb (20.1GiB) write-thru deactivated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2022-11.com.bqdn:disk1 ........................................................................................ [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 0]
  |     o- luns .......................................................................................................... [LUNs: 0]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]

3）设置ACL即将ISCSI对象与客户端IP或主机名绑定

/> /iscsi/iqn.2022-11.com.bqdn:disk1/tpg1/acls create iqn.2022-11.com.bdqn:client1
Created Node ACL for iqn.2022-11.com.bdqn:client1

/> ls
o- / ......................................................................................................................... [...]
 o- backstores .............................................................................................................. [...]
 | o- block .................................................................................................. [Storage Objects: 1]
 | | o- server0.disk1 ................................................................. [/dev/sdb (20.1GiB) write-thru deactivated]
 | |   o- alua ................................................................................................... [ALUA Groups: 1]
 | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
 | o- fileio ................................................................................................. [Storage Objects: 0]
 | o- pscsi .................................................................................................. [Storage Objects: 0]
 | o- ramdisk ................................................................................................ [Storage Objects: 0]
 o- iscsi ............................................................................................................ [Targets: 1]
 | o- iqn.2022-11.com.bqdn:disk1 ........................................................................................ [TPGs: 1]
 |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
 |     o- acls .......................................................................................................... [ACLs: 1]
 |     | o- iqn.2022-11.com.bdqn:client1 ......................................................................... [Mapped LUNs: 0]
 |     o- luns .......................................................................................................... [LUNs: 0]
 |     o- portals .................................................................................................... [Portals: 1]
 |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
 o- loopback ......................................................................................................... [Targets: 0]
/>
#注：iqn.2022-11.com.bdqn:client1是 initiator 的名字，需要在客户端中设置的。

4）创建LUN并绑定块

/> /iscsi/iqn.2022-11.com.bqdn:disk1/tpg1/luns create /backstores/block/server0.disk1
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.2022-11.com.bdqn:client1

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
  | | o- server0.disk1 ................................................................... [/dev/sdb (20.1GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2022-11.com.bqdn:disk1 ........................................................................................ [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.2022-11.com.bdqn:client1 ......................................................................... [Mapped LUNs: 1]
  |     |   o- mapped_lun0 ......................................................................... [lun0 block/server0.disk1 (rw)]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0 .............................................................. [block/server0.disk1 (/dev/sdb) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]

二、iscsi client访问iscsi server

1、系统环境信息

[root@node1 ~]# cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)

2、客户端软件包安装

[root@node2 ~]# rpm -qa |grep iscsi
libiscsi-1.9.0-6.el7.x86_64
iscsi-initiator-utils-6.2.0.873-32.el7.x86_64
iscsi-initiator-utils-iscsiuio-6.2.0.873-32.el7.x86_64

3、客户端配置

# 配置iscsi client名称
[root@node2 ~]# cat /etc/iscsi/initiatorname.iscsi
#InitiatorName=iqn.1994-05.com.redhat:e43610d8fc4d
InitiatorName=iqn.2022-11.com.bdqn:client1

4、客户端访问iscsi服务器

1）客户端访问工具iscsiadm

[root@node2 ~]# iscsiadm --help
iscsiadm -m discoverydb [ -hV ] [ -d debug_level ] [-P printlevel] [ -t type -p ip:port -I ifaceN ... [ -Dl ] ] | [ [ -p ip:port -t type] [ -o operation ] [ -n name ] [ -v value ] [ -lD ] ]
iscsiadm -m discovery [ -hV ] [ -d debug_level ] [-P printlevel] [ -t type -p ip:port -I ifaceN ... [ -l ] ] | [ [ -p ip:port ] [ -l | -D ] ]
iscsiadm -m node [ -hV ] [ -d debug_level ] [ -P printlevel ] [ -L all,manual,automatic ] [ -U all,manual,automatic ] [ -S ] [ [ -T targetname -p ip:port -I ifaceN ] [ -l | -u | -R | -s] ] [ [ -o  operation  ] [ -n name ] [ -v value ] ]
iscsiadm -m session [ -hV ] [ -d debug_level ] [ -P  printlevel] [ -r sessionid | sysfsdir [ -R | -u | -s ] [ -o operation ] [ -n name ] [ -v value ] ]
iscsiadm -m iface [ -hV ] [ -d debug_level ] [ -P printlevel ] [ -I ifacename | -H hostno|MAC ] [ [ -o  operation  ] [ -n name ] [ -v value ] ] [ -C ping [ -a ip ] [ -b packetsize ] [ -c count ] [ -i interval ] ]
iscsiadm -m fw [ -d debug_level ] [ -l ]
iscsiadm -m host [ -P printlevel ] [ -H hostno|MAC ] [ [ -C chap [ -x chap_tbl_idx ] ] | [ -C flashnode [ -A portal_type ] [ -x flashnode_idx ] ] | [ -C stats ] ] [ [ -o operation ] [ -n name ] [ -v value ] ]
iscsiadm -k priority

2）连接iscsi server

# discovery iscsi server

[root@node2 ~]# iscsiadm -m discovery -t st -p 192.168.8.200 -l
192.168.8.200:3260,1 iqn.2022-11.com.bqdn:disk1
Logging in to [iface: default, target: iqn.2022-11.com.bqdn:disk1, portal: 192.168.8.200,3260] (multiple)
Login to [iface: default, target: iqn.2022-11.com.bqdn:disk1, portal: 192.168.8.200,3260] successful.

# 查看共享存储
[root@node2 ~]# lsscsi
[0:0:0:0]    disk    ATA      VBOX HARDDISK    1.0   /dev/sda
[2:0:0:0]    cd/dvd  VBOX     CD-ROM           1.0   /dev/sr0
[3:0:0:0]    disk    LIO-ORG  server0.disk1    4.0   /dev/sdb
# 如上所示，已经发现共享的存储/dev/sdb。

3）查看磁盘设备

Tips：

如下所示，可以通过fdisk查看到共享的存储，可以像本地磁盘一样去使用共享存储。

[root@node2 ~]# fdisk -l
.......
Disk /dev/sdb: 21.6 GB, 21579694080 bytes, 42147840 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33550336 bytes

4）启动客户端iscsi服务

[root@node2 ~]# systemctl start iscsi
[root@node2 ~]# systemctl status iscsi
● iscsi.service - Login and scanning of iSCSI devices
   Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2022-11-23 16:17:53 CST; 1s ago
     Docs: man:iscsid(8)
           man:iscsiadm(8)
  Process: 27430 ExecStart=/sbin/iscsiadm -m node --loginall=automatic (code=exited, status=21)
  Process: 27419 ExecStart=/usr/libexec/iscsi-mark-root-nodes (code=exited, status=0/SUCCESS)
 Main PID: 27430 (code=exited, status=21)

Nov 23 16:17:53 node2 systemd[1]: Starting Login and scanning of iSCSI devices...
Nov 23 16:17:53 node2 iscsiadm[27430]: iscsiadm: No records found
Nov 23 16:17:53 node2 systemd[1]: Started Login and scanning of iSCSI devices.

三、iscsi client使用共享存储

1、创建磁盘分区

[root@node2 ~]# fdisk /dev/sdb

2、mount磁盘分区

[root@node2 ~]# mkdir /sharedsk

[root@node2 ~]# mount /dev/sdb1 /sharedsk
[root@node2 ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   39G   23G   16G  61% /
devtmpfs                 906M     0  906M   0% /dev
tmpfs                    921M   84K  920M   1% /dev/shm
........
/dev/sdb1                 21G   33M   21G   1% /sharedsk

#如上所示，在node2节点（iscsi client）已经可以像本地磁盘一样访问共享存储。

四、总结

 通过iscsi存储可以构建高性价比的共享存储设备，在基于共享存储的高可用架构中被广泛使用。