[ Centos 7 iscsi搭建 及 1台客户端同时挂载多台iscsi服务端问题 ]

一、什么是iscsi

iSCSI技术是一种由IBM公司研究开发的，是一个供硬件设备使用的可以在IP协议的上层运行的SCSI指令集，这种指令集合可以实现在IP网络上运行SCSI协议，使其能够在诸如高速千兆以太网上进行路由选择。iSCSI技术是一种新储存技术，该技术是将现有SCSI接口与以太网络(Ethernet)技术结合，使服务器可与使用IP网络的储存装置互相交换资料。

二、需求分析

最近在做虚拟化，需要通过一个主机挂载来自两个iscsi服务器的targets，如图：

三、实现思路
    
    问题：刚开始觉得这个应该问题不大，挂载第一台iscsi服务器的时候没有碰到问题，但是在挂载第二台iscsi时，发现怎样都无法挂载，挂载报错。
    最终找到了正确的解决方法，在centos7/RHEL7上挂载时，两台iscsi服务器端的target命名需要一致，下面进行搭建演示：

系统：centos 7.2
        iscsi客户端：192.168.1.156
        iscsi服务端：192.168.1.157
        iscsi服务端：192.168.1.158

首先要确保3台服务器的selinux和firewalld关闭

[root@iscsi-client ~]# setenforce  ; systemctl stop firewalld ; systemctl disable firewalld

iscsi服务端配置如下：

[root@iscsi-server1 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               :        4K   disk
sda               :       20G   disk
├─sda1            :      500M   part /boot
└─sda2            :     .5G   part
  ├─centos-root :     .5G   lvm  /
  └─centos-swap :        2G   lvm  [SWAP]
sdb               :      10G   disk
sr0              :        4G   rom

使用sdb作为target。为sdb进行分区并使用lvm进行发布出去，使用lvm的好处是后期便于扩展，在rhel7和centos7中，因为采用的是xfs文件系统，
lvm只能扩展，不能缩小，这点要牢记。

[root@iscsi-server1 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x966a38a7.

Command (m for help): n
Partition type:
   p   primary ( primary,  extended,  free)
   e   extended
Select (default p):
Using default response p
Partition number (-, default ):
First sector (-, default ):
Using default value
Last sector, +sectors or +size{K,M,G} (-, default ):
Using default value
Partition  of type Linux and of size  GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@iscsi-server1 ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               :        4K   disk
sda               :       20G   disk
├─sda1            :      500M   part /boot
└─sda2            :     .5G   part
  ├─centos-root :     .5G   lvm  /
  └─centos-swap :        2G   lvm  [SWAP]
sdb               :      10G   disk
└─sdb1            :      10G   part
sr0              :        4G   rom

以上就为sdb做了分区，接下来将sdb1创建为独立的lvm

# 创建卷组
[root@iscsi-server1 ~]# vgcreate vg_iscsi_1_156 /dev/sdb1
  Physical volume "/dev/sdb1" successfully created
  Volume group "vg_iscsi_1_156" successfully created

# 创建逻辑分区

[root@iscsi-server1 ~]# lvcreate -l + -n lv_iscsi_1_156 vg_iscsi_1_156
  Logical volume "lv_iscsi_1_156" created.
[root@iscsi-server1 ~]# lvs
  LV             VG             Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root           centos         -wi-ao---- .47g
  swap           centos         -wi-ao----  .00g
  lv_iscsi_1_156 vg_iscsi_1_156 -wi-a----- .00g 

lvm已经创建完毕，接下来创建target

# 安装target工具和服务
[root@iscsi-server1 ~]# yum install scsi-target-utils -y

# 开始创建target
[root@iscsi-server1 ~]# targetcli
targetcli shell version 2.1.fb41
Copyright - by Datera, Inc and others.
For help on commands, type 'help'.

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: ]
  | o- fileio ................................................................................................. [Storage Objects: ]
  | o- pscsi .................................................................................................. [Storage Objects: ]
  | o- ramdisk ................................................................................................ [Storage Objects: ]
  o- iscsi ............................................................................................................ [Targets: ]
  o- loopback ......................................................................................................... [Targets: ]

创建块设备，这里要注意，因为我们使用的lvm，因此不能将sdb添加到块设备，而是将逻辑卷添加进去

/> backstores/block create node1.disk /dev/sdb1
Cannot configure StorageObject because device /dev/sdb1 is already in use
/> backstores/block create node1.disk /dev/vg_iscsi_1_156/lv_iscsi_1_156
Created block storage object node1.disk using /dev/vg_iscsi_1_156/lv_iscsi_1_156.

在创建wwn的时候，又踩了一个坑。
    wwn命名必须 iqn.2017-01.com.xxx:server   注意这个月份 2017-01 这里必须要01 如果写成1就会报错。

# 错误写法
/> iscsi/ create iqn.-.com.node1:server
WWN not valid as: iqn, naa, eui

# 正确写法
/> iscsi/ create iqn.-.com.node1:server
Created target iqn.-.com.node1:server.
Created TPG .
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port .

# 创建acl，在创建acl这里一定要注意，如果准备在同一客户端挂载两个iscsi这里的acl命名一定要一致
/> iscsi/iqn.-.com.node1:server/tpg1/acls create iqn.-.com.node1:client
Created Node ACL for iqn.-.com.node1:client/> iscsi/iqn.-.com.node1:server/tpg1/luns create /backstores/block/node1.disk
Created LUN .
Created LUN -> mapping in node ACL iqn.-.com.node1:client

# 删除自动生成的监听
/> iscsi/iqn.-.com.node1:server/tpg1/portals/ delete 0.0.0.0
Deleted network portal 0.0.0.0:

# 创建绑定固定ip地址的监听
/> iscsi/iqn.-.com.node1:server/tpg1/portals/ create 192.168.1.157
Using default IP port
Created network portal 192.168.1.157:.
/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: ]
  | | o- node1.disk ............................................ [/dev/vg_iscsi_1_156/lv_iscsi_1_156 (.0GiB) write-thru activated]
  | o- fileio ................................................................................................. [Storage Objects: ]
  | o- pscsi .................................................................................................. [Storage Objects: ]
  | o- ramdisk ................................................................................................ [Storage Objects: ]
  o- iscsi ............................................................................................................ [Targets: ]
  | o- iqn.-.com.node1:server ...................................................................................... [TPGs: ]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: ]
  |     | o- iqn.-.com.node1:client ......................................................................... [Mapped LUNs: ]
  |     |   o- mapped_lun0 ............................................................................ [lun0 block/node1.disk (rw)]
  |     o- luns .......................................................................................................... [LUNs: ]
  |     | o- lun0 .......................................................... [block/node1.disk (/dev/vg_iscsi_1_156/lv_iscsi_1_156)]
  |     o- portals .................................................................................................... [Portals: ]
  |       o- 192.168.1.157: ............................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: ]
/> saveconfig
Last  configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json
/> exit
Global pref auto_save_on_exit=true
Last  configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json

最后一定要执行 saveconfig 再退出

另一台iscsi服务端配置一样，targetcli如下：

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: ]
  | | o- node1:disk ............................................ [/dev/vg_iscsi_1_156/lv_iscsi_1_156 (.0GiB) write-thru activated]
  | o- fileio ................................................................................................. [Storage Objects: ]
  | o- pscsi .................................................................................................. [Storage Objects: ]
  | o- ramdisk ................................................................................................ [Storage Objects: ]
  o- iscsi ............................................................................................................ [Targets: ]
  | o- iqn.-.com.node1:server ...................................................................................... [TPGs: ]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: ]
  |     | o- iqn.-.com.node1:client ......................................................................... [Mapped LUNs: ]
  |     |   o- mapped_lun0 ............................................................................ [lun0 block/node1:disk (rw)]
  |     o- luns .......................................................................................................... [LUNs: ]
  |     | o- lun0 .......................................................... [block/node1:disk (/dev/vg_iscsi_1_156/lv_iscsi_1_156)]
  |     o- portals .................................................................................................... [Portals: ]
  |       o- 192.168.1.158: ............................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: ]

其中acl命名一定要和server1保持一致

这里要注意，target服务建议要配置开机启动。在搭建的时候，有出现过没有开机启动配置丢失的情况。

[root@iscsi-server2 ~]# systemctl start target ; systemctl enable target
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.

iscsi客户端配置：

[root@iscsi-client yum.repos.d]# yum install iscsi* -y

1. 修改/etc/iscsi/initiatorname.iscsi文件

[root@iscsi-client ~]# vim /etc/iscsi/initiatorname.iscsi

# 这里InitiatorName 后面就是iscsi服务端通过targetcli创建的iscsi acl命名，因为要挂载两个iscsi所以名称必须一致
InitiatorName=iqn.-.com.node1:client

2. 启动服务

# 这里客户端依然建议开机启动
[root@iscsi-client ~]# systemctl start iscsid ; systemctl enable iscsid

3. 挂载iscsi

# 挂载192.168.1.，挂载之前必须要发现该iscsi
[root@iscsi-client ~]# iscsiadm -m discovery -t st -p 192.168.1.157
192.168.1.157:, iqn.-.com.node1:server
[root@iscsi-client ~]# iscsiadm -m node -T iqn.-.com.node1:server -p 192.168.1.157 -l
Logging in to [iface: default, target: iqn.-.com.node1:server, portal: 192.168.1.157,] (multiple)
Login to [iface: default, target: iqn.-.com.node1:server, portal: 192.168.1.157,] successful.

# 挂载192.168.1.，挂载之前必须要发现该iscsi
[root@iscsi-client ~]# iscsiadm -m discovery -t st -p 192.168.1.158
192.168.1.158:, iqn.-.com.node1:server
[root@iscsi-client ~]# iscsiadm -m node -T iqn.-.com.node1:server -p 192.168.1.158 -l
Logging in to [iface: default, target: iqn.-.com.node1:server, portal: 192.168.1.158,] (multiple)
Login to [iface: default, target: iqn.-.com.node1:server, portal: 192.168.1.158,] successful.

查看是否生效：

[root@iscsi-client ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               :        4K   disk
sda               :       20G   disk
├─sda1            :      500M   part /boot
└─sda2            :     .5G   part
  ├─centos-root :     .5G   lvm  /
  └─centos-swap :        2G   lvm  [SWAP]
sdb               :      10G   disk
sdc               :      10G   disk
sr0              :        4G   rom  /mnt/iso

sdb 和 sdc都已经生成，说明挂载成功

在配置完成iscsi客户端的时候，建议进行重启服务器进行验证

# 重启成功，查看没问题。
[root@iscsi-client ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               :        4K   disk
sda               :       20G   disk
├─sda1            :      500M   part /boot
└─sda2            :     .5G   part
  ├─centos-root :     .5G   lvm  /
  └─centos-swap :        2G   lvm  [SWAP]
sdb               :      10G   disk
sdc               :      10G   disk
sr0              :        4G   rom 

四、对iscsi进行扩容的操作

对192.168.1.157这台主机添加一块硬盘，并扩容客户端iscsi的容量

192.168.1.157 iscsi服务端操作如下：

# 已经为157添加了一块新的硬盘 sdc，首先将sdc加入到lvm中
[root@iscsi-server1 ~]# lsblk
NAME                              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0                                 :        4K   disk
sda                                 :       20G   disk
├─sda1                              :      500M   part /boot
└─sda2                              :     .5G   part
  ├─centos-root                   :     .5G   lvm  /
  └─centos-swap                   :        2G   lvm  [SWAP]
sdb                                 :      10G   disk
└─sdb1                              :      10G   part
  └─vg_iscsi_1_156-lv_iscsi_1_156 :       10G   lvm
sdc                                 :      10G   disk
sr0                                :        4G   rom 

# 对sdc进行分区
[root@iscsi-server1 ~]# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x68a23069.

Command (m for help): n
Partition type:
   p   primary ( primary,  extended,  free)
   e   extended
Select (default p):
Using default response p
Partition number (-, default ):
First sector (-, default ):
Using default value
Last sector, +sectors or +size{K,M,G} (-, default ):
Using default value
Partition  of type Linux and of size  GiB is set

Command (m for help):
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@iscsi-server1 ~]#
[root@iscsi-server1 ~]# lsblk
NAME                              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0                                 :        4K   disk
sda                                 :       20G   disk
├─sda1                              :      500M   part /boot
└─sda2                              :     .5G   part
  ├─centos-root                   :     .5G   lvm  /
  └─centos-swap                   :        2G   lvm  [SWAP]
sdb                                 :      10G   disk
└─sdb1                              :      10G   part
  └─vg_iscsi_1_156-lv_iscsi_1_156 :       10G   lvm
sdc                                 :      10G   disk
└─sdc1                              :      10G   part
sr0                                :        4G   rom

# 将sdc1添加到之前lvm卷组vg_iscsi_1_156里
[root@iscsi-server1 ~]# vgextend vg_iscsi_1_156 /dev/sdc1
  Physical volume "/dev/sdc1" successfully created
  Volume group "vg_iscsi_1_156" successfully extended

# 扩展逻辑卷
[root@iscsi-server1 ~]# lvresize -l + -n /dev/vg_iscsi_1_156/lv_iscsi_1_156
  Size of logical volume vg_iscsi_1_156/lv_iscsi_1_156 changed from 10.00 GiB ( extents) to 19.99 GiB ( extents).
  Logical volume lv_iscsi_1_156 successfully resized.

# 重读下逻辑卷大小
[root@iscsi-server1 ~]# resize2fs /dev/vg_iscsi_1_156/lv_iscsi_1_156
resize2fs 1.42. (-Dec-)
resize2fs: Device or resource busy while trying to open /dev/vg_iscsi_1_156/lv_iscsi_1_156
Couldn't find valid filesystem superblock.

到此，iscsi服务端扩容操作完毕。

192.168.1.156 iscsi客户端操作如下：

重新加载sdb和sdc容量，可以看到，sdc已经变成了20G

[root@iscsi-client ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               :        4K   disk
sda               :       20G   disk
├─sda1            :      500M   part /boot
└─sda2            :     .5G   part
  ├─centos-root :     .5G   lvm  /
  └─centos-swap :        2G   lvm  [SWAP]
sdb               :      10G   disk
sdc               :      20G   disk
sr0              :        4G   rom

这里验证一个问题，因为挂载了两台iscsi，尝试多次重启，看看sdb和sdc会不会出现互换设备名的情况

经过3次重启的验证，并没有出现过互换设备名的情况，但是在挂载的时候，这里还是建议使用UUID
我在虚拟机上分区后无法查看到UUID，物理设备做iscsi服务端是可以查看的。
[root@db1 ~]# blkid /dev/vda1
/dev/vda1: UUID="e1b26164-3ec5-435a-bf0c-b9f4c0989941" TYPE="xfs"

至此，已经完成了一台服务器上挂载两个iscsi服务端的问题。

五、故障问题

在 192.168.1.156上查看到 sdb 所对应的是server 192.168.1.158

测试1：当sdb分区并挂载至某个目录

当158 down掉，客户端会有怎样的提示，如下验证：

dmesg 报错信息如下：

ls 挂载目录错误如下：

很明显的I/O错误。

解决方案：再次启动server 192.168.1.158, target开机自动启动，客户端需要重新挂载到目录才会恢复正常。

测试2：当sdb分区并挂载到某个目录做为web程序存放点，故障时，看web会返回什么状态码

通过apache测试，当iscsi服务器down掉，web服务器并不会马上故障。大概5分钟后报错

查看错误日志：

所以，当iscsi客户端出现IO错误时，先查看磁盘空间是否不足，再次检查下iscsi服务端网络通信是否正常。