用asmlib方式创建oracle集群ASM磁盘(oracleasm)

创建asm磁盘的几种方式

创建asm方式很多主要有以下几种
1、Faking方式
2、裸设备方式
3、udev方式(它下面有两种方式)
  3.1 uuid方式
  3.2 raw方式（裸设备方式）
4、asmlib方式（oracleasm）

 

系统采样

[root@shdb02 dev]#  lsb_release -a
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:
graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID:    RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 6.4 (Santiago)
Release:    6.4
Codename:    Santiago
[root@shdb02 dev]# uname -rm //查询体系架构
2.6.32-358.el6.x86_64 x86_64
[root@shdb02 dev]# cat /etc/redhat-release   //linux centos系统版本
Red Hat Enterprise Linux Server release 6.4 (Santiago)

[root@shdb02 disks]# su - oracle
  [oracle@shfpdb02 ~]$ echo $ORACLE_HOME
  /u01/app/oracle/product/11.2.0/db_1  //ORACLE版本

 

一、安装asmlib(即使用oracleasm工具)

 在Linux中，加载设备时，因加载顺序问题，共享磁盘在各节点的设备名可能会不一致.
    为解决这个问题，保证各节点的设备名一致，有两种方法:
    <<1>>使用asmlib创建asm磁盘，将会在各分区上做asm磁盘标识，以保证设备名称的唯一性
    首先安装ASM内核软件包
    在oracle官方网站下载asm的软件包安装到各节点主机上.

Oracle ASMLib 2.0

 1、oracleasmlib-2.0.12-1.el6.x86_64.rpm  

  https://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.12-1.el6.x86_64.rpm

 2、oracleasm-support-2.1.8-1.el6.x86_64.rpm

  https://oss.oracle.com/projects/oracleasm-support/dist/files/RPMS/rhel6/amd64/2.1.8/oracleasm-support-2.1.8-1.el6.x86_64.rpm

   https://www.oracle.com/linux/downloads/linux-asmlib-v6-downloads.html

https://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.12-1.el6.x86_64.rpm  (el6代表centos6)

https://www.oracle.com/linux/downloads/linux-asmlib-rhel6-downloads.html

 
安装 Install ASMLIB RPM(这三个包要注意按照次序来)

[root@shdb02 src]# ll
kmod-oracleasm-2.0.6.rh1-2.el6.x86_64.rpm 
oracleasmlib-2.0.4-1.el6.x86_64.rpm 
oracleasm-support-2.1.8-1.el6.x86_64.rpm

[root@shdb02 src]#  cat /etc/issue
Red Hat Enterprise Linux Server release 6.4 (Santiago)
Kernel \\r on an \\m
[root@shdb02 src]#  uname -a
Linux itrus 2.6.32-358.el6.x86_64 #1 SMP Sun Nov 10 22:19:54 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

第一步:
[root@shdb02 src]#  rpm -ivh oracleasm-support-2.1.8-1.el6.x86_64.rpm

warning: oracleasm-support-2.1.8-1.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
  Preparing... ########################################### [100%]
  1:oracleasm-support ########################################### [100%]

第二步：

[root@shdb02 src]# rpm -ivh kmod-oracleasm-2.0.6.rh1-2.el6.x86_64.rpm  //安装时间较长。
warning: kmod-oracleasm-2.0.6.rh1-2.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Preparing... ########################################### [100%]
1:kmod-oracleasm ########################################### [100%]

第三步：
[root@shdb02 src]#  rpm -qa|grep kmod-oracleasm
kmod-oracleasm-2.0.6.rh1-2.el6.x86_64
#[root@shdb02 src]#  rpm -ivh oracleasmlib-2.0.4-1.el6.x86_64.rpm
warning: oracleasmlib-2.0.4-1.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Preparing... ########################################### [100%]
1:oracleasmlib ########################################### [100%]

//检查oracleasm

[root@shdb02 src]# rpm -qa|grep oracleasm
  oracleasmlib-2.0.4-1.el6.x86_64 
  kmod-oracleasm-2.0.6.rh1-2.el6.x86_64
  oracleasm-support-2.1.8-1.el6.x86_64

 二、创建ASM磁盘(启用、配置、创建、删除、查看等操作)

2、在一个节点执行以下命令
# /etc/init.d/oracleasm enable
# /etc/init.d/oracleasm configure
# /etc/init.d/oracleasm createdisk diska /dev/sdb5
# /etc/init.d/oracleasm createdisk diskb /dev/sdb6
# /etc/init.d/oracleasm createdisk diskc /dev/sdb7
将在以下路径/dev/oracleasm/disks/下产生ASM磁盘asm-diskaa,asm-diskab,asm-diskac.  在其它节点执行以下命令，扫描ASM磁盘.
3、具体过程
 [root@shdb02 src]# /etc/init.d/oracleasm -h
  Usage: /etc/init.d/oracleasm
{start|stop|restart|enable|disable|configure|createdisk|deletedisk|querydisk|listdisks|scandisks|status}

[root@shdb02 src]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no

[root@shdb02 src]# oracleasm scandisks
      Reloading disk partitions: done
      Cleaning any stale ASM disks...
      Scanning system for ASM disks...

 [root@shdb02 src]# /etc/init.d/oracleasm configure  //注意 和 oracleasm configure 有区别，前面有/etc/init.d/  否则一个是配置，一个是显示结果
 Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver is
  loaded on boot and what permissions it will have. The current values
  will be shown in brackets ('[]'). Hitting <ENTER> without typing an
  answer will keep that current value. Ctrl-C will abort.

//下面是需要我们配置的,这个过程也比较长

Default user to own the driver interface []: grid
  Default group to own the driver interface []: asmadmin
  Start Oracle ASM library driver on boot (y/n) [n]: y
  Scan for Oracle ASM disks on boot (y/n) [y]: y
  Writing Oracle ASM library driver configuration: done
  Initializing the Oracle ASMLib driver: [ OK ]
  Scanning the system for Oracle ASMLib disks: [ OK ]

//这个过程配合日志看下做了什么

[root@shdb02 dev]# cat /var/log/oracleasm
  Creating /dev/oracleasm mount point: /dev/oracleasm
  Loading module "oracleasm": oracleasm
  Mounting ASMlib driver filesystem: /dev/oracleasm
  Reloading disk partitions: done
  Cleaning any stale ASM disks...
  Scanning system for ASM disks...

[root@shdb02 src]# oracleasm configure
  ORACLEASM_ENABLED=true
  ORACLEASM_UID=oracle
  ORACLEASM_GID=dba
  ORACLEASM_SCANBOOT=true
  ORACLEASM_SCANORDER=""
  ORACLEASM_SCANEXCLUDE=""
  ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"

//如果想重新配置可以使用 oracleasm configure -i来重新配置

[root@shdb02 src]# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks:   //回车
[ OK ]  //这个过程时间比较长耐心等待

//oracleasm 检查状态
[root@shdb02 src]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes

[root@shdb02 src]# /etc/init.d/oracleasm listdisks  //如果配置错误，则会显示不出磁盘，这个要注意。

//注意这里，一旦你安装了oracleasm则会有这个目录出现

[root@shdb02 dev]# ll /dev | grep oracleasm
  drwxr-xr-x 4 root root 0 Mar 15 23:41 oracleasm

oracleasm添加的磁盘通常都会在 /dev/oracleasm/disks

由于系统重新启动后，磁盘的owner会被修改为root，为保证重启后保持磁盘owner不变，可以在/etc/udev/rule下添加一个rules文件
    添加udev的 /etc/udev/rules.d/99-asmdisk-owner.rules文件，添加uuid内容
    当然,asmlib并不只是用于创建asm磁盘，还具有asm存储管理功能，并为应用程序提供接口.

另外会碰到的就是

[root@shdb02 dev]# /etc/init.d/oracleasm createdisk asm-diskjj /dev/sdj  //应该是名称不对
  Marking disk "asm-diskjj" as an ASM disk: [FAILED]  //这种情况

//分区的磁盘才有uuid

[root@shdb02 dev]# scsi_id -g -u -d /dev/sdj  //sdj我已分区,不需要格式化
 369ce37410023334d07dc2c4b00000008

//最后解决。

[root@shdb02 dev]# /usr/sbin/oracleasm querydisk /dev/sdj
  Device "/dev/sdj" is not marked as an ASM disk
 [root@shdb02 dev]# /usr/sbin/oracleasm createdisk diskj /dev/sdj   //换了名称之后
  Device "/dev/sdj" is not a partition
 [root@shdb02 dev]# /usr/sbin/oracleasm createdisk diskj /dev/sdj1
  Writing disk header: done
  Instantiating disk: done

[root@shdb02 dev]# cat /var/log/oracleasm    //查看日志,如果出现问题

//这里开始出现

[root@shdb02 disks]# oracleasm listdisks
 DISKJ

[root@shdb02 disks]# lsblk -f | grep sdj
  sdj
   └─sdj1 oracleasm DISKJ

//另外，lsblk 下出现大量的sdm,sdw等磁盘信息，但是一查确实无用的东西，应该是之前谁弄的错的，应该结合fdisk -l 查看更准确一些。

  当然还有另外一种情况,利用的是uuid方式   

[root@shdb02 disks]# ll /dev/ | grep asm*
  drwxrwx--- 2 root asmadmin 280 Dec 15 11:17 asm
  brw-rw---- 1 grid asmadmin 65, 160 Mar 16 01:35 asm-diskaa
  brw-rw---- 1 grid asmadmin 65, 176 Mar 16 01:35 asm-diskab
  brw-rw---- 1 grid asmadmin 65, 192 Mar 16 01:37 asm-diskac
  brw-rw---- 1 grid asmadmin 65, 208 Mar 16 01:35 asm-diskad
  brw-rw---- 1 grid asmadmin 8, 16 Mar 16 01:37 asm-diskb

 然后需要在每个节点上设置磁盘的访问权限，否则后面配置磁盘时无法发现磁盘.
[root@shdb02 src]#chown grid:dba /dev/asm-diskaa
[root@shdb02 src]#chown grid:dba /dev/asm-diskab
[root@shdb02 src]#chown grid:dba /dev/asm-diskac

//如果改变/dev/oracleasm的情况下，但是这种是可以共存的。

//这种是configure需要配置的时候用户是grid,组是dba,加入到udev,请看下面的99-x-asmdisk.rules文件中的配置

查看系统内可使用的磁盘

[root@shfpdb02 dev]# lsblk -f | grep sa*
sda
├─sda1                         ext4              cf0f3592-a2dc-497c-be07-d4f1e0d42253   /boot
└─sda2                         LVM2_member       nY2JaS-cCZf-qn6E-f4XS-i6Ph-Bf3Y-DxLWF9
  └─vg_bjfpbdb2-lv_swap (dm-1) swap              bcea691d-47c4-49d4-b9bc-eb366bc246f0   [SWAP]
sdb                            oracleasm
sdc                            oracleasm
sdd                            oracleasm
sde                            oracleasm
sdf                            oracleasm
sdg                            oracleasm
sdh                            oracleasm
sdi                            oracleasm
sdj
sdk                            oracleasm
sdl                            oracleasm
sdm                            oracleasm
sdn                            oracleasm
sdo                            oracleasm
sdp                            oracleasm
sdq                            oracleasm
sdr                            oracleasm
sds                            oracleasm
sdt                            oracleasm
sdu                            oracleasm
sdv                            oracleasm
sdw                            oracleasm
sdx                            oracleasm
sdy                            oracleasm
sdz                            oracleasm
sdaa                           oracleasm
sdab                           oracleasm
sdac                           oracleasm
sdad                           oracleasm                

 三、使用udev管理

    将从磁盘获得scsi_id，然后根据scsi_id加载成易读的设备名，保证设备名的唯一性.
    如果使用虚拟机，首先需在虚拟主机的配置文件vmx文件中设置下列参数，以起用
    disk.EnableUUID = "TRUE"
    如果虚拟主机有通过设定更改任何参数，这个参数会自动更改为FALSE，需要重设.
    启动主机后确认udev是否已安装
    # rpm -qa | grep udev
    udev-095-14.27.el5_7.1
    执行以下命令获得scsi id信息
    # /sbin/scsi_id -g -u -s /block/sdb         

    #rhel6使用:# /sbin/scsi_id -g -u -d /dev/sdb
    36000c29fd07c14fdb1ca780130a5ad8e
    配置udev配置文件，添加rules文件(在目录/etc/udev/rules.d/下)并授权.
    # cd /etc/udev/rules.d/
    # vi 99-x-asmdisk.rules
    添加以下参数，注意一个行为一条规则，并且逗号(,)与下一个关键词之间有空格.
    KERNEL=="sd?5", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="36000c29fd07c14fdb1ca780130a5ad8e", NAME="asm-diskaa",
OWNER="grid", GROUP="dba", MODE="0660"
    KERNEL=="sd?6", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="36000c29fd07c14fdb1ca780130a5ad8e", NAME="asm-diskab",
OWNER="oracle", GROUP="dba", MODE="0660"
    KERNEL=="sd?7", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent", RESULT=="36000c29fd07c14fdb1ca780130a5ad8e", NAME="asm-diskac",
OWNER="oracle", GROUP="dba", MODE="0660"
    重新启动udev或重启节点.
    # start_udev
    验证是否配置正常
    # ls -al /dev/asmdisk*
    后续创建磁盘组时可以将搜索字符串ASM_DISKSTRING参数指向‘/dev/asmdisk*'就可以了