centos6.5 x86_64安装oracle 11.2.0.3grid



centos6.5 x86_64安装oracle 11.2.0.3grid

1、安装前的准备 工作

1.1、配置node1

1.1.1、配置虚拟机并安装centos

安装node1-----

1、安装node1

1.1、加入30G的硬盘，1G内存，2个处理器，加入两个网卡NAT

1.2、设置主机名同一时候配置网络

node1.localdomain  192.168.0.3/255.255.255.0/192.168.0.1  eth0

node1-priv.localdomain  10.10.10.31/255.255.255.0/10.10.10.1 eth1

1.3、新建一个分区结构

/boot-256m  /-8G  /tmp-2G  swap-3G  /u01-10G 

安装grid的要求：

memory-1.5G  swap-2.7G  tmp-1G

1.4、Minimal安装

安装209个包

1.5、cmd-没有关闭防火墙和selinux

把网卡设置为host only能够ping通

问题：

（1）当中一个网卡没有保存

新增一个网卡host only

# ifconfig -a  ##多出一个HWaddr

# cd /etc/sysconfig/network-scripts/

# cp /etc/sysconfig/network-scripts/ifcfg-eth0 ifcfg-eth1

# ping 10.10.10.31  ##能够ping通

安装node2-----

1、安装node2

1.1、加入30G的硬盘，1G内存，2个处理器，加入两个网卡host only

1.2、设置主机名同一时候配置网络

node2.localdomain  192.168.0.4/255.255.255.0/192.168.0.1  eth0

node2-priv.localdomain  10.10.10.32/255.255.255.0/10.10.10.1 eth1

1.3、新建一个分区结构

/boot-256m  /-8G  /tmp-2G  swap-3G  /u01-10G

1.4、Minimal安装

安装209个包

1.5、cmd

网卡host only：重新启动后，直接ping 192.168.0.4能够ping通

1.1.2、node1-配置yum，并安装X Window和Desktop

1、node1和node2互相ping，都能ping通

[root@node1 ~]# ping 192.168.0.3

[root@node1 ~]# ping 192.168.0.4

[root@node1 ~]# ping 10.10.10.31

[root@node1 ~]# ping 10.10.10.32

[root@node2 ~]# ping 10.10.10.31

[root@node2 ~]# ping 10.10.10.32

[root@node2 ~]# ping 192.168.0.3

[root@node2 ~]# ping 192.168.0.4

2、node1配置yum，并安装X系统和桌面系统

# mount /dev/cdrom /mnt

# cd /etc/yum.repos.d/

# mv CentOS-Base.repo CentOS-Base.repo.bak

# cp CentOS-Base.repo.bak CentOS-Base.repo

# vi CentOS-Base.repo

[base]

name=CentOS-$releasever - Base

#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os

baseurl=file:///mnt/Packages

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

# yum clean all

# rm CentOS-Base.repo

# cp CentOS-Media.repo CentOS-Media.repo.bak

# vi CentOS-Media.repo  ##仅仅需改动baseurl

[c6-media]

name=CentOS-$releasever - Media

baseurl=file:///mnt/

gpgcheck=1

enabled=0

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6

# yum --disablerepo=* --enablerepo=c6-media update

Loaded plugins: fastestmirror

Determining fastest mirrors

c6-media                                                                                                                                        | 4.0 kB     00:00 ...

c6-media/primary_db                                                                                                                             | 4.4 MB     00:00 ...

Setting up Update Process

No Packages marked for Update

# yum --disablerepo=* --enablerepo=c6-media clean all

# yum --disablerepo=* --enablerepo=c6-media makecache

# yum --disablerepo=* --enablerepo=c6-media groupinstall "Desktop"    ##须要安装294个包，禁用全部系统提供的.repo库

# yum --disablerepo=* --enablerepo=c6-media groupinstall "X Window System"

3、安装vnc

# rpm -qa |grep vnc

# yum install tigervnc tigervnc-server

Loaded plugins: fastestmirror, refresh-packagekit

Loading mirror speeds from cached hostfile

Setting up Install Process

No package tigervnc available.

No package tigervnc-server available.

# cd /mnt/Packages/

# rpm -ivh tigervnc*

error: Failed dependencies:

        perl is needed by tigervnc-server-1.1.0-5.el6_4.1.x86_64

        xorg-x11-fonts-misc is needed by tigervnc-server-1.1.0-5.el6_4.1.x86_64

# yum --disablerepo=* --enablerepo=c6-media install vnc

# rpm -qa |grep vnc

tigervnc-1.1.0-5.el6_4.1.x86_64  ##仅仅是安装了tigervnc

# yum --disablerepo=* --enablerepo=c6-media install tigervnc tigervnc-server

Running Transaction

  Installing : 1:perl-Pod-Escapes-1.04-136.el6.x86_64                                                                                                              1/9

  Installing : 4:perl-libs-5.10.1-136.el6.x86_64                                                                                                                   2/9

  Installing : 1:perl-Pod-Simple-3.13-136.el6.x86_64                                                                                                               3/9

  Installing : 3:perl-version-0.77-136.el6.x86_64                                                                                                                  4/9

  Installing : 1:perl-Module-Pluggable-3.90-136.el6.x86_64                                                                                                         5/9

  Installing : 4:perl-5.10.1-136.el6.x86_64                                                                                                                        6/9

  Installing : 1:xorg-x11-font-utils-7.2-11.el6.x86_64                                                                                                             7/9

  Installing : xorg-x11-fonts-misc-7.2-9.1.el6.noarch                                                                                                              8/9

  Installing : tigervnc-server-1.1.0-5.el6_4.1.x86_64  

# vi /etc/sysconfig/vncservers

VNCSERVERS="1:root"

VNCSERVERARGS[2]="-geometry 1024x768 -nolisten tcp"

# vncpasswd

# service vncserver start

Starting VNC server: 1:root xauth:  creating new authority file /root/.Xauthority

xauth: (stdin):1:  bad display name "node1.localdomain:1" in "add" command

New 'node1.localdomain:1 (root)' desktop is node1.localdomain:1

Creating default startup script /root/.vnc/xstartup

Starting applications specified in /root/.vnc/xstartup

Log file is /root/.vnc/node1.localdomain:1.log

[  OK  ]

# cd ~/.vnc/

# vi xstartup

#!/bin/sh

export LANG

#unset SESSION_MANAGER

#exec /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xistartup ] && exec /etc/vnc/xstartup

[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources

xsetroot -solid grey

vncconfig -iconic &

xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &

twm &

gnome-session &

必选项：vnc viewer连接centos服务端时,注意防火墙相应的port是否开放/或者关闭防火墙,否则会报10060(超时错误)，使用以下命令配置规则同意相应porttcp包通过：

# iptables -I INPUT 1 -p tcp --dport 5901 -j ACCEPT

# service iptables save

# service iptables restart

问题

（1）Could not open/read file:///mnt/Packages/repodata/repomd.xml

把baseurl设置为file:///mnt/

1.1.3、node1-改动/etc/hosts文件

1、

1.1、

# vi /etc/hosts

#node1

192.168.0.3     node1.localdomain       node1

192.168.0.5     node1-vip.localdomain   node1-vip

10.10.10.31     node1-priv.localdomain  node1-priv

#node2

192.168.0.4     node2.localdomain       node2

192.168.0.6     node2-vip.localdomain   node2-vip

10.10.10.32     node2-priv.localdomain  node2-privi

#scan-ip

192.168.0.8    scan-cluster.localdomain        scan-cluster

******在node2上做如上同样的操作

1.1.4、node1-配置dns，使scan-ip能够被解析

1、

1.1、安装bind服务

# ping 192.168.0.5  ##ping不通

# ping 192.168.0.15  ##ping不通

[root@node1 ～]# yum --disablerepo=* --enablerepo=c6-media install bind

1.2、配置/var/named/chroot/etc/named.conf 文件

*经过了解dns仅仅需安装bind、bind-chroot、bind-utils

# yum --disablerepo=* --enablerepo=c6-media remove bind*

# yum --disablerepo=* --enablerepo=c6-media install bind bind-chroot bind-utils

1.3、配置/etc/named.conf 文件  不再是/var/named/chroot/etc/named.conf 文件

1.3.1、改动主配置文件

# cp -p /etc/named.conf named.conf.bak

# vi /etc/named.conf

 listen-on port 53 { any; };

 allow-query     { any; };

# cp -p /etc/named.rfc1912.zones named.rfc1912.zones.bak

1.3.2、配置zone文件

# vi /etc/named.rfc1912.zones  ##追加下面信息

zone "localdomain" IN {

        type master;

        file "forward.zone";

        allow-update { none; };

};

zone "0.168.192.in-addr.arpa" IN {

        type master;

        file "reverse.zone";

        allow-update { none; };

};

1.3.3、生成正向解析数据库文件

# cd /var/named/

# cp -p named.localhost forward.zone 

# vi forward.zone  ##正向解析数据库文件末尾增加例如以下

$TTL 1D

@       IN SOA  node1.localdomain. root.localdomain. (

                                        0       ; serial

                                        1D      ; refresh

                                        1H      ; retry

                                        1W      ; expire

                                        3H )    ; minimum

        IN      NS      node1.localdomain.

node1   IN      A       127.0.0.1

scan-cluster.localdomain    IN      A     192.168.0.8

scan-cluster    IN      A       192.168.0.8

~

1.3.4、生成方向解析数据库文件

# cp -p named.loopback reverse.zone

# vi  reverse.zone  ##反向解析数据库文件末尾增加例如以下

$TTL 1D

@       IN SOA  node1.localdomain. root.localdomain. (

                                        0       ; serial

                                        1D      ; refresh

                                        1H      ; retry

                                        1W      ; expire

                                        3H )    ; minimum

        IN      NS      node1.localdomain.

3     IN      PTR     node1.localdomain.

8     IN      PTR     scan-cluster.localdomain.

8     IN      PTR     scan-cluster.

# service named restart

# vi /etc/resolv.conf   ##文件末尾增加例如以下

search localdomain

nameserver 192.168.0.3

1.1.5、检查软件环境需求

1.1、检查操作系统版本号

#  cat /etc/issue

CentOS release 6.5 (Final)

Kernel \r on an \m

# cat /proc/version

Linux version 2.6.32-431.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Fri Nov 22 03:15:09 UTC 2013

1.2、检查内核

# uname -r

2.6.32-431.el6.x86_64

1.3、检查必要的包

binutils-2.20.51.0.2-5.11.el6 (x86_64)

compat-libcap1-1.10-1 (x86_64)

compat-libstdc++-33-3.2.3-69.el6 (x86_64)

compat-libstdc++-33-3.2.3-69.el6.i686

gcc-4.4.4-13.el6 (x86_64)

gcc-c++-4.4.4-13.el6 (x86_64)

glibc-2.12-1.7.el6 (i686)

glibc-2.12-1.7.el6 (x86_64)

glibc-devel-2.12-1.7.el6 (x86_64)

glibc-devel-2.12-1.7.el6.i686

ksh

libgcc-4.4.4-13.el6 (i686)

libgcc-4.4.4-13.el6 (x86_64)

libstdc++-4.4.4-13.el6 (x86_64)

libstdc++-4.4.4-13.el6.i686

libstdc++-devel-4.4.4-13.el6 (x86_64)

libstdc++-devel-4.4.4-13.el6.i686

libaio-0.3.107-10.el6 (x86_64)

libaio-0.3.107-10.el6.i686

libaio-devel-0.3.107-10.el6 (x86_64)

libaio-devel-0.3.107-10.el6.i686

make-3.81-19.el6

sysstat-9.0.4-11.el6 (x86_64)

# rpm -qa |grep binutils

binutils-2.20.51.0.2-5.36.el6.x86_64

# rpm -qa |grep compat-libcap

# yum --disablerepo=* --enablerepo=c6-media install compat-libcap

Loaded plugins: fastestmirror, refresh-packagekit

Loading mirror speeds from cached hostfile

Setting up Install Process

No package compat-libcap available.

Error: Nothing to do

# yum --disablerepo=* --enablerepo=c6-media install compat-libcap1

# rpm -qa |grep compat-libstdc++

# yum --disablerepo=* --enablerepo=c6-media install compat-libstdc++*

# rpm -qa |grep gcc

libgcc-4.4.7-4.el6.x86_64

libgcc-4.4.7-4.el6.i686

# yum --disablerepo=* --enablerepo=c6-media install gcc

# yum --disablerepo=* --enablerepo=c6-media install gcc-c++

# rpm -qa |grep gcc

gcc-4.4.7-4.el6.x86_64

libgcc-4.4.7-4.el6.x86_64

libgcc-4.4.7-4.el6.i686

gcc-c++-4.4.7-4.el6.x86_64

# rpm -qa |grep glibc

glibc-common-2.12-1.132.el6.x86_64

glibc-2.12-1.132.el6.x86_64

glibc-2.12-1.132.el6.i686

glibc-devel-2.12-1.132.el6.x86_64

glibc-headers-2.12-1.132.el6.x86_64

#  yum --disablerepo=* --enablerepo=c6-media install glibc-devel

Loaded plugins: fastestmirror, refresh-packagekit

Loading mirror speeds from cached hostfile

Setting up Install Process

Package glibc-devel-2.12-1.132.el6.x86_64 already installed and latest version

Nothing to do

# yum --disablerepo=* --enablerepo=c6-media install glibc-devel.i686  *****

# rpm -qa |grep libgcc

libgcc-4.4.7-4.el6.x86_64

libgcc-4.4.7-4.el6.i686

# rpm -qa |grep libstdc++

libstdc++-4.4.7-4.el6.x86_64

compat-libstdc++-296-2.96-144.el6.i686

libstdc++-devel-4.4.7-4.el6.x86_64

compat-libstdc++-33-3.2.3-69.el6.x86_64

# yum --disablerepo=* --enablerepo=c6-media install libstdc++.i686

# yum --disablerepo=* --enablerepo=c6-media install libstdc++-devel.i686

# rpm -qa |grep libaio   ##值得注意的安装方法

# yum --disablerepo=* --enablerepo=c6-media install libaio*

# rpm -qa |grep libaio

libaio-devel-0.3.107-10.el6.x86_64

libaio-0.3.107-10.el6.x86_64

# yum --disablerepo=* --enablerepo=c6-media install libaio.i686

# yum --disablerepo=* --enablerepo=c6-media install libaio-devel.i686

# rpm -qa |grep ksh

# yum --disablerepo=* --enablerepo=c6-media install ksh

# rpm -qa |grep make

make-3.81-20.el6.x86_64

# yum --disablerepo=* --enablerepo=c6-media install libXp.i686

# yum --disablerepo=* --enablerepo=c6-media install libXt.i686

1.1.6、配置内核參数

# vi /etc/sysctl.conf  ##原始的shmmax/shmall 比較大

fs.aio-max-nr = 1048576

fs.file-max = 6815744

kernel.shmall = 2097152

kernel.shmmax = 536870912

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default = 262144

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 1048576

# /sbin/sysctl –p

error: "–p" is an unknown key

# sysctl -p

1.1.7、配置会话限制

# vi /etc/pam.d/login  ##末尾加入

session    required     pam_limits.so

1.1.8、软件安装用户的资源限制

# vi /etc/security/limits.conf

oracle              hard    nproc   204800

oracle              soft    nofile  204800

oracle              hard    nofile  204800

oracle              soft    stack   204800

grid                soft    nproc   204800

grid                hard    nproc   204800

grid                soft    nofile  204800

grid                hard    nofile  204800

grid                soft    stack   204800

1.1.9、创建组，创建用户，改动环境变量

# groupadd -g 1000 oinstall

# groupadd -g 1200 asmadmin

# groupadd -g 1201 asmdba

# groupadd -g 1202 asmoper

# useradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper -d /home/grid -s /bin/bash grid

# passwd

# vi /home/grid/.bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

        . ~/.bashrc

fi

# User specific environment and startup programs

export PATH=$PATH:$HOME/bin

export PS1="`/bin/hostname -s`->"

export TMP=/tmp

export TMPDIR=$TMP

export ORACLE_SID=+ASM1

export ORACLE_BASE=/u01/app/grid

export ORACLE_HOME=/u01/app/11.2.0/grid

export TNS_ADMIN=$ORACLE_HOME/network/admin

export PATH=/usr/sbin:$PATH

export PATH=$PATH:$ORACLE_HOME/bin

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

# source /home/grid/.bash_profile

# groupadd -g 1300 dba

# groupadd -g 1301 oper

# useradd -u 1101 -g oinstall -G dba,oper,asmdba -d /home/oracle -s /bin/bash oracle

# passwd

# vi /home/oracle/.bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

        . ~/.bashrc

fi

# User specific environment and startup programs

export PATH=$PATH:$HOME/bin

export PS1="`/bin/hostname -s`->"

export TMP=/tmp

export TMPDIR=$TMP

export ORACLE_HOSTNAME=node1.localdomain

export ORACLE_SID=devdb1

export ORACLE_UNQNAME=devdb

export ORACLE_BASE=/u01/app/oracle

export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1

export TNS_ADMIN=$ORACLE_HOME/network/admin

export PATH=/usr/sbin:$PATH

export PATH=$PATH:$ORACLE_HOME/bin

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

# source /home/oracle/.bash_profile

1.1.10、创建文件夹,改动权限

# mkdir -p /u01/app/grid  ##-p级联

# mkdir -p /u01/app/11.2.0/grid

# chown -R grid:oinstall /u01/app/grid

# chown -R grid:oinstall /u01/app/11.2.0/grid

# mkdir -p /u01/app/oracle

# chown -R oracle:oinstall /u01/app/oracle

最后运行例如以下命令:

# chmod -R 775 /u01

1.1.11、停止ntp服务,11gR2新增的检查项

# service ntpd status

ntpd is stopped

# chkconfig ntpd off

#

# mv /etc/ntp.conf /etc/ntp.conf.bak

1.2、配置node2

在node2上，反复node1的1.2-1.11，同一时候配置环境变量时+ASM2/devdb2

1.3、配置grid/oracle用户ssh对等性

##node1

node1->env |grep ORA

ORACLE_UNQNAME=devdb

ORACLE_SID=devdb1

ORACLE_BASE=/u01/app/oracle

ORACLE_HOSTNAME=node1.localdomain

ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1

##node2

node2->env |grep ORA

ORACLE_UNQNAME=devdb

ORACLE_SID=devdb2

ORACLE_BASE=/u01/app/oracle

ORACLE_HOSTNAME=node1.localdomain

ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1

在集群就绪服务 (CRS) 和 RAC 安装过程中，Oracle Universal Installer (OUI) 必须可以以 oracle 的身份将软件拷贝到全部 RAC 节点

而不提示输入口令

##node1

# su - oracle

node1->mkdir ~/.ssh

node1-> chmod 700 ~/.ssh

node1->ssh-keygen -t rsa  ##一路回车

Generating public/private rsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_rsa.

Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.

The key fingerprint is:

1f:a5:6f:8b:c9:e9:a3:04:4d:99:27:6a:3d:77:4d:1e oracle@node1.localdomain

node1->ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_dsa.

Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.

The key fingerprint is:

fc:83:1c:73:37:52:50:90:50:aa:8b:41:9f:61:b9:6b oracle@node1.localdomain

##node2

# su - oracle

node2->mkdir ~/.ssh

node2->chmod 700 ~/.ssh

node2->ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_rsa.

Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.

The key fingerprint is:

e9:fd:b0:9b:1e:57:56:07:51:9c:2e:99:33:dc:0d:03 oracle@node2.localdomain

node2->ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_dsa.

Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.

The key fingerprint is:

67:f7:9a:09:e1:9e:c2:4f:6b:9a:e3:71:ce:c4:eb:08 oracle@node2.localdomain

##回到node1

node1->cd ~

node1->ll -a

node1->cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

node1->cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

node1->ssh node2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The authenticity of host 'node2 (192.168.0.4)' can't be established.

RSA key fingerprint is 36:5f:59:0d:52:d3:24:68:7d:60:59:da:10:37:b1:47.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'node2,192.168.0.4' (RSA) to the list of known hosts.

oracle@node2's password:

node1->ssh node2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

oracle@node2's password:


*****把公钥文件复制到node2

node1->scp ~/.ssh/authorized_keys node2:~/.ssh/authorized_keys

oracle@node2's password:


authorized_keys                                                                                                                      100% 2040     2.0KB/s   00:00

##验证oracle用户对等性

每一个节点运行例如以下命令，第二次运行时不再须要输入口令

ssh node1 date

ssh node2 date

ssh node1-priv date

ssh node2-priv date

ssh node1.localdomain date

ssh node2.localdomain date

ssh node1-priv.localdomain date

ssh node2-priv.localdomain date

依据以上方法，grid用户配置对等性

1.4、配置共享磁盘

C:\Program Files (x86)\VMware\VMware Workstation>vmware-vdiskmanager.exe -c -s 0.5GB -a lsilogic -t 2 "F:\Virtual Machines\share disk\OCRDISK.vmdk"

C:\Program Files (x86)\VMware\VMware Workstation>vmware-vdiskmanager.exe -c -s 0.5GB -a lsilogic -t 2 "F:\Virtual Machines\share disk\VOTINGDISK.vmdk"

C:\Program Files (x86)\VMware\VMware Workstation>vmware-vdiskmanager.exe -c -s 3GB -a lsilogic -t 2 "F:\Virtual Machines\share disk\ASMDISK1.vmdk"

C:\Program Files (x86)\VMware\VMware Workstation>vmware-vdiskmanager.exe -c -s 3GB -a lsilogic -t 2 "F:\Virtual Machines\share disk\ASMDISK2.vmdk"

C:\Program Files (x86)\VMware\VMware Workstation>vmware-vdiskmanager.exe -c -s 3GB -a lsilogic -t 2 "F:\Virtual Machines\share disk\ASMFLASH.vmdk"

F:\Virtual Machines\node2\node2.vmx，使用ultraedit编辑，保存退出，启动node2，仅仅能启动1块硬盘

disk.locking = "FALSE"

diskLib.dataCacheMaxSize = "0"

scsi1.sharedBus = "virtual"

scsi1:0.present = "TRUE"

scsi1:0.fileName = "F:\Virtual Machines\share disk\OCRDISK.vmdk"

scsi1:0.mode = "independent-persistent"

scsi1:0.deviceType = "disk"

scsi1:1.present = "TRUE"

scsi1:1.fileName = "F:\Virtual Machines\share disk\VOTINGDISK.vmdk"

scsi1:1.mode = "independent-persistent"

scsi1:1.deviceType = "disk"

scsi2:0.present = "TRUE"

scsi2:0.fileName = "F:\Virtual Machines\share disk\ASMDISK1.vmdk"

scsi2:0.mode = "independent-persistent"

scsi2:0.deviceType = "disk"

scsi2:1.present = "TRUE"

scsi2:1.fileName = "F:\Virtual Machines\share disk\ASMDISK2.vmdk"

scsi2:1.mode = "independent-persistent"

scsi2:1.deviceType = "disk"

scsi2:2.present = "TRUE"

scsi2:2.fileName = "F:\Virtual Machines\share disk\ASMFLASH.vmdk"

scsi2:2.mode = "independent-persistent"

scsi2:2.deviceType = "disk"

scsi1.virtualDev = "lsilogic"

改动为：能够启动6个硬盘

disk.locking = "FALSE"

diskLib.dataCacheMaxSize = "0"

scsi1.sharedBus = "virtual"

scsi1.present = "TRUE"

scsi1:0.present = "TRUE"

scsi1:0.fileName = "F:\Virtual Machines\share disk\OCRDISK.vmdk"

scsi1:0.mode = "independent-persistent"

scsi1:0.deviceType = "disk"

scsi1:1.present = "TRUE"

scsi1:1.fileName = "F:\Virtual Machines\share disk\VOTINGDISK.vmdk"

scsi1:1.mode = "independent-persistent"

scsi1:1.deviceType = "disk"

scsi1:2.present = "TRUE"

scsi1:2.fileName = "F:\Virtual Machines\share disk\ASMDISK1.vmdk"

scsi1:2.mode = "independent-persistent"

scsi1:2.deviceType = "disk"

scsi1:3.present = "TRUE"

scsi1:3.fileName = "F:\Virtual Machines\share disk\ASMDISK2.vmdk"

scsi1:3.mode = "independent-persistent"

scsi1:3.deviceType = "disk"

scsi1:4.present = "TRUE"

scsi1:4.fileName = "F:\Virtual Machines\share disk\ASMFLASH.vmdk"

scsi1:4.mode = "independent-persistent"

scsi1:4.deviceType = "disk"

scsi1.virtualDev = "lsilogic"

# fdisk -l  ##6个硬盘

1.5、安装rpm包，配置asm磁盘

1、格式化分区（两个节点任选其一）

2、安装asm包（每一个节点）

# rpm -ivh kmod-oracleasm-2.0.6.rh1-3.el6.x86_64.rpm

# rpm -ivh oracleasm-support-2.1.8-1.el5.x86_64.rpm

# rpm -ivh oracleasmlib-2.0.4-1.el5.x86_64.rpm

# oracleasm configure -i

# oracleasm init

3、清理asm磁盘头

# dd if=/dev/zero of=/dev/sdb1 bs=8192 count=12800

12800+0 records in

12800+0 records out

104857600 bytes (105 MB) copied, 2.18097 s, 48.1 MB/s

# dd if=/dev/zero of=/dev/sdc1 bs=8192 count=12800

12800+0 records in

12800+0 records out

104857600 bytes (105 MB) copied, 1.29817 s, 80.8 MB/s

# dd if=/dev/zero of=/dev/sdd1 bs=8192 count=12800

12800+0 records in

12800+0 records out

104857600 bytes (105 MB) copied, 1.31786 s, 79.6 MB/s

# dd if=/dev/zero of=/dev/sde1 bs=8192 count=12800

12800+0 records in

12800+0 records out

104857600 bytes (105 MB) copied, 1.43146 s, 73.3 MB/s

# dd if=/dev/zero of=/dev/sdf1 bs=8192 count=12800

4、映射分区到asm磁盘（两个节点任选其一）

#  oracleasm scandisks  ##刷新asm磁盘信息，不然不是最新信息

# oracleasm createdisk VOL1 /dev/sdb1

Writing disk header: done

Instantiating disk: done

# oracleasm createdisk VOL2 /dev/sdc1

Writing disk header: done

Instantiating disk: done

# oracleasm createdisk VOL3 /dev/sdd1

Writing disk header: done

# oracleasm createdisk VOL4 /dev/sde1

Writing disk header: done

Instantiating disk: done

# oracleasm createdisk VOL5 /dev/sdf1

5、关闭NetworkManager

# chkconfig --level 123456 NetworkManager off

# service NetworkManager stop

Stopping NetworkManager daemon:                            [  OK  ]

1.6、安装 cvuqdisk包，验证grid环境，并修复failed

1、安装cvuqdisk(每一个节点)

# CVUQDISK_GRP=oinstall; export CVUQDISK_GRP

# rpm -qa |grep cvuqdisk

cvuqdisk-1.0.9-1.x86_64

# ll /home/grid/grid/rpm

total 12

-rwxr-xr-x 1 root root 8551 Sep 22  2011 cvuqdisk-1.0.9-1.rpm

# ls -l /usr/sbin/cvuqdisk

-rwsr-xr-x 1 root oinstall 10928 Sep  4  2011 /usr/sbin/cvuqdisk

2、验证grid环境

node1->find /home/grid/grid/ -name runcluvfy.sh

/home/grid/grid/runcluvfy.sh

Checking ASMLib configuration.

  Node Name                             Status

  ------------------------------------  ------------------------

  node1                                 passed

  node2                                 (failed) ASMLib does not list the disks "[VOL2, VOL1, VOL5, VOL4, VOL3]"

node1->./runcluvfy.sh stage -pre crsinst -fixup -n node1,node2 -verbose >> /home/grid/runcluvfy.check

Check: Package existence for "sysstat"

  Node Name     Available                 Required                  Status   

  ------------  ------------------------  ------------------------  ----------

  node2         missing                   sysstat-5.0.5             failed   

  node1         missing                   sysstat-5.0.5             failed   

Result: Package existence check failed for "sysstat"

Check: Package existence for "pdksh"

  Node Name     Available                 Required                  Status   

  ------------  ------------------------  ------------------------  ----------

  node2         missing                   pdksh-5.2.14              failed   

  node1         missing                   pdksh-5.2.14              failed

Result: Default user file creation mask check passed

Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined

File "/etc/resolv.conf" does not have both domain and search entries defined

Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...

domain entry in file "/etc/resolv.conf" is consistent across nodes

Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...

search entry in file "/etc/resolv.conf" is consistent across nodes

Checking file "/etc/resolv.conf" to make sure that only one search entry is defined

All nodes have one search entry defined in file "/etc/resolv.conf"

Checking all nodes to make sure that search entry is "localdomain" as found on node "node2"

All nodes of the cluster have same value for 'search'

Checking DNS response time for an unreachable node

  Node Name                             Status                 

  ------------------------------------  ------------------------

  node2                                 failed                 

  node1                                 failed                 

PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: node2,node1

（1）其它节点必须运行oracleasm scandisks

# oracleasm scandisks

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

Instantiating disk "VOL1"

Instantiating disk "VOL3"

Instantiating disk "VOL2"

Instantiating disk "VOL4"

Instantiating disk "VOL5"

# oracleasm listdisks

VOL1

VOL2

VOL3

VOL4

VOL5

（2）能够不安装sysstat

# yum --disablerepo=* --enablerepo=c6-media install sysstat -y

（3）卸载ksh安装pdksh

# yum --disablerepo=* --enablerepo=c6-media install pdksh

Loaded plugins: fastestmirror, refresh-packagekit

Loading mirror speeds from cached hostfile

c6-media                                                                                                                                        | 4.0 kB     00:00 ...

Setting up Install Process

No package pdksh available.

Error: Nothing to do

# rpm -e ksh-20120801-10.el6.x86_64

# rpm -ivh pdksh-5.2.14-30.x86_64.rpm

warning: pdksh-5.2.14-30.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 73307de6: NOKEY

Preparing...                ########################################### [100%]

   1:pdksh                  ########################################### [100%]

（3）PRVF-5636和PRVF-5637，rhel6以上版本号的bug引起的，11.2.0.4修复

# sed -i 's/recursion  yes/recursion  no/1' /etc/named.conf  ##解决PRVF-5636

# service named restart

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined

File "/etc/resolv.conf" does not have both domain and search entries defined

Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...

domain entry in file "/etc/resolv.conf" is consistent across nodes

Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...

search entry in file "/etc/resolv.conf" is consistent across nodes

Checking file "/etc/resolv.conf" to make sure that only one search entry is defined

All nodes have one search entry defined in file "/etc/resolv.conf"

Checking all nodes to make sure that search entry is "localdomain" as found on node "node2"

All nodes of the cluster have same value for 'search'

Checking DNS response time for an unreachable node

  Node Name                             Status                 

  ------------------------------------  ------------------------

  node2                                 failed                 

  node1                                 failed                 

PRVF-5637 : DNS response time could not be checked on following nodes: node2,node1

##加入段

# sed -i 's/allow-query     { any; };/&\n\ allow-query-cache\     { any; };/' /etc/named.conf  ##解决PRVF-5637

# vi /etc/named.conf  ##仅改动3行

        allow-query     { any; };

 allow-query-cache     { any; };

        recursion no;

# service named restart

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined

File "/etc/resolv.conf" does not have both domain and search entries defined

Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...

domain entry in file "/etc/resolv.conf" is consistent across nodes

Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...

search entry in file "/etc/resolv.conf" is consistent across nodes

Checking file "/etc/resolv.conf" to make sure that only one search entry is defined

All nodes have one search entry defined in file "/etc/resolv.conf"

Checking all nodes to make sure that search entry is "localdomain" as found on node "node2"

All nodes of the cluster have same value for 'search'

Checking DNS response time for an unreachable node

  Node Name                             Status                 

  ------------------------------------  ------------------------

  node2                                 passed                 

  node1                                 passed                 

The DNS response time for an unreachable node is within acceptable limit on all nodes

File "/etc/resolv.conf" is consistent across nodes

1.7、重新启动虚拟机，建立虚拟机快照

（1）查看resolv.conf

关闭NetworkManager后，不再冲突

（2）查看asm磁盘是否挂载

自己主动

（3）配置named自启动

# chkconfig --level 345 named on

（4）清空/tmp

（4）./runcluvfy.sh stage -pre crsinst -fixup -n node1,node2 -verbose

2、開始安装grid