Rhel6-pacemaker+drbd配置文档

系统环境: rhel6 x86_64 iptables and selinux disabled

主机: 192.168.122.119 server19.example.com

192.168.122.25 server25.example.com(注:时间需同步)

192.168.122.1 desktop36.example.com

所需的包:drbd-8.4.3.tar.gz

yum仓库配置:

[rhel-source]

name=Red
Hat Enterprise Linux $releasever - $basearch - Source

baseurl=ftp://192.168.122.1/pub/yum

enabled=1

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

[HighAvailability]

name=Instructor
Server Repository

baseurl=ftp://192.168.122.1/pub/yum/HighAvailability

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

enabled=1

[LoadBalancer]

name=Instructor
Server Repository

baseurl=ftp://192.168.122.1/pub/yum/LoadBalancer

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

enabled=1

[ResilientStorage]

name=Instructor
Server Repository

baseurl=ftp://192.168.122.1/pub/yum/ResilientStorage

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

enabled=1

[ScalableFileSystem]

name=Instructor
Server Repository

baseurl=ftp://192.168.122.1/pub/yum/ScalableFileSystem

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

enabled=1

#配置pacemaker

以下步骤在server19和server25上实施：

[root@server19
~]# yum install corosync pacemaker -y

以下步骤在server19或server25上实施：

[root@server19
~]# cd /etc/corosync/

[root@server19
corosync]# corosync-keygen (生成该key需要不断的敲打键盘)

[root@server19
corosync]# cp corosync.conf.example corosync.conf

[root@server19
corosync]# vim corosync.conf

#
Please read the corosync.conf.5 manual page

compatibility:
whitetank

totem
{

version: 2

secauth: off

threads: 0

interface {

ringnumber: 0

bindnetaddr: 192.168.122.0

mcastaddr: 226.94.1.1

mcastport: 5405

ttl: 1

}

}

logging
{

fileline: off

to_stderr: yes

to_logfile: yes

to_syslog: yes

logfile: /var/log/cluster/corosync.log

debug: off

timestamp: on

logger_subsys {

subsys: AMF

debug: off

}

}

amf
{

mode: disabled

}

service
{

ver: 0

name: pacemaker

use_mgmtd: yes

}

[root@server19
corosync]# scp corosync.conf authkey
root@192.168.122.25:/etc/corosync/

以下步骤在server19和server25上实施：

[root@server19
corosync]# /etc/init.d/corosync start

此时查看日志tail
-f /var/log/cluster/corosync.log会有如下错误:

Jul
27 02:31:31 [1461] server19.example.com pengine: notice:
process_pe_message: Configuration ERRORs found during PE processing.
Please run "crm_verify -L" to identify issues.

解决方法如下:

[root@server19
corosync]# crm（注：redhat6.4后crm这个命令没有集成在pacemaker包中,需要另外安装crmsh）

crm(live)#
configure

crm(live)configure#
property stonith-enabled=false

crm(live)configure#
commit

crm(live)configure#
quit

[root@server19
corosync]# crm_verify -L(检测配置是否有错误)

此时执行crm_mon进入监控页面，若两台主机均处于Online状态说明配置成功.

以下配置只需在任意一台机子上实施,所有配置会自动同步到另一台机子上.

#添加虚拟IP

[root@server19 corosync]# crm

crm(live)# configure

crm(live)configure# primitive vip ocf:heartbeat:IPaddr2 params ip=192.168.122.178 cidr_netmask=32 op monitor interval=30s

crm(live)configure# commit

crm(live)configure# quit

#忽略法定人数的检查

[root@server19 corosync]# crm

crm(live)# configure

crm(live)configure# property no-quorum-policy=ignore

crm(live)configure# commit

crm(live)configure# quit

#
添加apache服务

1.以下步骤在server19和server25上实施：

[root@server19
corosync]# vim /etc/httpd/conf/httpd.conf

<Location
/server-status>

SetHandler server-status

Order deny,allow

Deny from all

Allow from 127.0.0.1

</Location>

[root@server19
corosync]# echo `hostname` > /var/www/html/index.html

2.以下步骤在server19或server25上实施：

[root@server19
corosync]# crm

crm(live)#
configure

crm(live)configure#
primitive apache ocf:heartbeat:apache params
configfile=/etc/httpd/conf/httpd.conf op monitor interval=1min

crm(live)configure#
commit

qcrm(live)configure#
quit

此时执行crm_mon有可能会发现vip和apache运行在不同的机子上:

解决方法(将vip和apache绑定):

[root@server19 corosync]# crm

crm(live)# configure

crm(live)configure# colocation apache-with-vip inf: apache vip

crm(live)configure# commit

crm(live)configure# quit

此时访问192.168.122.178可访问到server19上的页面

#配置主备

[root@server19
corosync]# crm

crm(live)#
configure

crm(live)configure#
location master-node apache 10: server19.example.com

crm(live)configure#
commit

crm(live)configure#
quit

#配置fence

以下步骤在desktop36上实施：

[root@desktop36
~]# yum list fence*

[root@desktop36
~]# yum install fence-virtd.x86_64 fence-virtd-libvirt.x86_64
fence-virtd-multicast.x86_64 fence-virt-0.2.3-9.el6.x86_64 -y

[root@desktop36
~]# fence_virtd -c

Module
search path [/usr/lib64/fence-virt]:

Available
backends:

libvirt 0.1

Available
listeners:

multicast 1.1

Listener
modules are responsible for accepting requests

from
fencing clients.

Listener
module [multicast]:

The
multicast listener module is designed for use environments

where
the guests and hosts may communicate over a network using

multicast.

The
multicast address is the address that a client will use to

send
fencing requests to fence_virtd.

Multicast
IP Address [225.0.0.12]:

Using
ipv4 as family.

Multicast
IP Port [1229]:

Setting
a preferred interface causes fence_virtd to listen only

on
that interface. Normally, it listens on the default network

interface.
In environments where the virtual machines are

using
the host machine as a gateway, this *must* be set

(typically
to virbr0).

Set
to 'none' for no interface.

Interface
[none]: virbr0

The
key file is the shared key information which is used to

authenticate
fencing requests. The contents of this file must

be
distributed to each physical host and virtual machine within

a
cluster.

Key
File [/etc/cluster/fence_xvm.key]:

Backend
modules are responsible for routing requests to

the
appropriate hypervisor or management layer.

Backend
module [checkpoint]: libvirt

The
libvirt backend module is designed for single desktops or

servers.
Do not use in environments where virtual machines

may
be migrated between hosts.

Libvirt
URI [qemu:///system]:

Configuration
complete.

===
Begin Configuration ===

backends
{

libvirt
{

uri
= "qemu:///system";

}

}

listeners
{

multicast
{

interface
= "virbr0";

port
= "1229";

family
= "ipv4";

address
= "225.0.0.12";

key_file
= "/etc/cluster/fence_xvm.key";

}

}

fence_virtd
{

module_path
= "/usr/lib64/fence-virt";

backend
= "libvirt";

listener
= "multicast";

}

===
End Configuration ===

Replace
/etc/fence_virt.conf with the above [y/N]? y

注:以上设置除“Interface”处填写虚拟机通信接口和Backend
module填写libvirt外,其他选项均可回车保持默认。

[root@desktop36
~]# mkdir /etc/cluster

[root@desktop36
~]# dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=128 count=1

以下步骤在server19和server25上实施：

[root@server19
corosync]# mkdir /etc/cluster

[root@server19
corosync]# yum install fence-virt-0.2.3-9.el6.x86_64 -y

以下步骤在desktop36上实施：

[root@desktop36
~]# scp /etc/cluster/fence_xvm.key root@192.168.122.119:/etc/cluster/

[root@desktop36
~]# scp /etc/cluster/fence_xvm.key root@192.168.122.25:/etc/cluster/

[root@desktop36
~]# /etc/init.d/fence_virtd start

[root@desktop36
~]# netstat -anuple | grep fence

udp
0 0 0.0.0.0:1229 0.0.0.0:*
0 823705 6320/fence_virtd

端口说明fence_virtd启动成功.

以下步骤在server19或server25上实施：

[root@server19
corosync]# crm

crm(live)#
configure

crm(live)configure#
cib new stonith

crm(stonith)configure#
quit

[root@server19
corosync]# crm

crm(live)#
configure

crm(live)configure#
primitive vmfence stonith:fence_xvm params
pcmk_host_map="server19.example.com:vm1
server25.example.com:vm2" op monitor interval=30s

crm(live)configure#
property stonith-enabled=true

crm(live)configure#
commit

crm(live)configure#
quit

测试：将server19断网或执行echo
c > /proc/sysrq-trigger模拟内核崩溃，看服务是否被接管，并且server19断电重启.

#配置drbd

分别给server19和server25加一块相同大小的虚拟硬盘

以下步骤在server19或server25上实施：

[root@server19 kernel]# yum install kernel-devel make -y

[root@server19 kernel]# tar zxf drbd-8.4.3.tar.gz

[root@server19 kernel]# cd drbd-8.4.3

[root@server19 drbd-8.4.3]# ./configure --enable-spec –with-km

此时会出现如下问题:

(1)configure: error: no acceptable C compiler found in $PATH

(2)configure: error: Cannot build utils without flex, either install flex or pass the --without-utils option.

(3)configure: WARNING: No rpmbuild found, building RPM packages is disabled.

(4)configure: WARNING: Cannot build man pages without xsltproc. You may safely ignore this warning when building from a tarball.

(5)configure: WARNING: Cannot update buildtag without git. You may safely ignore this warning when building from a tarball.

解决方法如下:

(1)[root@server19
drbd-8.4.3]# yum install gcc -y

(2)[root@server19
drbd-8.4.3]# yum install flex -y

(3)[root@server19
drbd-8.4.3]# yum install rpm-build -y

(4)[root@server19
drbd-8.4.3]# yum install libxslt -y

(5)[root@server19
drbd-8.4.3]# yum install git -y

[root@server19
kernel]# mkdir -p ~/rpmbuild/SOURCES

[root@server19
kernel]# cp drbd-8.4.3.tar.gz ~/rpmbuild/SOURCES/

[root@server19
drbd-8.4.3]# rpmbuild -bb drbd.spec

[root@server19
drbd-8.4.3]# rpmbuild -bb drbd-km.spec

[root@server19
drbd-8.4.3]# cd ~/rpmbuild/RPMS/x86_64/

[root@server19
x86_64]# rpm -ivh *

[root@server19
x86_64]# scp ~/rpmbuild/RPMS/x86_64/*
root@192.168.122.25:/root/kernel/

以下步骤在server25上实施：

[root@server25
kernel]# rpm -ivh *

以下步骤在server19和server25上实施：

[root@server19
~]# fdisk -cu /dev/vda

划分分区（一般只划一个分区）,类型为Linux
LVM的

[root@server19
~]# pvcreate /dev/vda1

[root@server19
~]# vgcreate koenvg /dev/vda1

[root@server19
~]# lvcreate -L 1G -n koenlv koenvg

以下步骤在server19或server25上实施：

[root@server19
drbd.d]# cd /etc/drbd.d/

[root@server19
drbd.d]# vim drbd.res

resource
koen {

meta-disk
internal;

device
/dev/drbd1;

syncer
{

verify-alg
sha1;

}

net
{

allow-two-primaries;

}

on
server19.example.com
{

disk

/dev/mapper/koenvg-koenlv;

address
192.168.122.119:7789;

}

on
server25.example.com
{

disk

/dev/mapper/koenvg-koenlv;

address
192.168.122.25:7789;

}

}

[root@server19
drbd.d]# scp drbd.res root@192.168.122.25:/etc/drbd.d/

以下步骤在server19和server25上实施：

[root@server19
drbd.d]# drbdadm create-md koen

[root@server19
drbd.d]# /etc/init.d/drbd start

以下步骤在server19上实施：

[root@server19
drbd.d]# drbdsetup /dev/drbd1 primary –force

(此条命令将server19设置成primary节点，并同步数据)

此时可以执行watch
cat /proc/drbd 查看同步状态，当同步完成后继续往下配置，创建文件系统.

[root@server19
drbd.d]# mkfs.ext4 /dev/drbd1

[root@server19
drbd.d]# mount /dev/drbd1 /var/www/html/

注意:两台主机上的/dev/drbd1
不能同时挂载,只有状态为
primary
时,才能被挂载使
用,而此时另一方的状态为
secondary。

测试：在server19上将/dev/drbd1挂在到/var/www/html/下
,进到/var/www/html/中随意编辑一些文件，然后卸载/dev/drbd1(umount
/var/www/html/)，执行drbdadm
secondary koen和drbdadm
primary
koen将server25设置为主节点，在server25上挂载/dev/drbd1,最后查看/var/www/html/下的内容是否同步,

附:拉伸设备

以下步骤在server19和server25上实施：

[root@server19
~]# lvextend -L +1000M /dev/mapper/koenvg-koenlv

以下步骤在server19或server25上实施：

[root@server19
~]# drbdadm resize koen

以下步骤在primary节点上实施：

[root@server25
~]# mount /dev/drbd1 /var/www/html/

[root@server25
~]# resize2fs /dev/drbd1

#将pacemaker和drbd整合

以下步骤在server19或server25上实施：

[root@server19
~]# crm

crm(live)#
configure

crm(live)configure#
primitive webdata ocf:linbit:drbd params drbd_resource=koen op
monitor interval=60s

crm(live)configure#
ms webdataclone webdata meta master-max=1 master-node-max=1
clone-max=2 clone-node-max=1 notify=true

crm(live)configure#
primitive webfs ocf:heartbeat:Filesystem params device="/dev/drbd1"
directory="/var/www/html" fstype=ext4

crm(live)configure#
group webgroup vip apache webfs

crm(live)configure#
colocation apache-on-webdata inf: webgroup webdataclone:Master

crm(live)configure#
order apache-after-webdata inf: webdataclone:promote webgroup:start

crm(live)configure#
commit

crm(live)configure#
quit

附：使用iscsi存储

以下步骤在desktop36上实施：

[root@desktop36 ~]# yum install scsi-target-utils.x86_64 -y

[root@desktop36 ~]# vim /etc/tgt/targets.conf

<target iqn.2013-07.com.example:server.target1>

backing-store /dev/vg_desktop36/iscsi-test

initiator-address 192.168.122.112

initiator-address 192.168.122.234

</target>

[root@desktop36 ~]# /etc/init.d/tgtd start

以下步骤在server19和server25上实施：

[root@server19
~]# iscsiadm -m discovery -t st -p 192.168.122.1

[root@server19
~]# iscsiadm -m node -l

使用fdisk
-cu对iscsi设备进行分区并且格式化.

注:此操作只需要在一个节点上进行即可,另一个节点会自动同步.

以下步骤在server19或server25上实施：

[root@server19
~]# crm

crm(live)#
configure

crm(live)configure#
primitive iscsi ocf:heartbeat:Filesystem params device=/dev/sda1
directory=/var/www/html fstype=ext4 op monitor
interval=30s

crm(live)configure#
colocation apache-with-iscsi inf: apache iscsi

crm(live)configure#
commit

crm(live)configure#
quit