OCFS2 Fencing
OCFS2 FencingPosted on February 8, 2011 by Abdulhameed Basha
I am very excited to start writing my experience with Oracle products
and solutions. In this blog am going to detail on OCFS2 fencing technology.
We are setting up a new data center for Oracle products,
for which we have got SUN servers X4170,X4270 and X4470,
SUN SAN storage 6180, SAN Switches and Fiber Channel
HBA (Host Bus Adapter). We have configured SAN storage and allocated
LUNs to the servers for Oracle EBS R12.1, OBI and Hyperian Applications.
We created a required partitions and OCFS2 file system on new devices
for our Oracle EBS R12.1 RAC database and shared Applications File system.
OCFS2 is certified for Shared Application file system, check ID .
After configuring our file system and mounted, we started to test
various scenarios to verify the stability of the setup. To our surprise noticed
that server is getting rebooted when we removed all HBAs connected to the
server or powered off SAN switch.
On analysis we found that it is a expected behavior of OCFS2 which is called “Fencing”.
Fencing is the act of forcefully removing a node from a cluster.
A node with OCFS2 file system mounted will fence itself when it
realizes that it does not have quorum in a degraded cluster.
It does this so that other nodes won’t be stuck trying to access its resources.
In earlier versions of OCFS2, User reported that nodes are hanging
during fencing. From version OCFS2 1.2.5, Oracle no longer uses
“panic” state for fencing instead it uses “machine restart”.
Let us now see, how exactly OCFS2 forces kernel to restart on fencing.
* After configuring OCFS2 and started cluster service O2CB, there will be a heartbeat system file in which every node writes its presence every 2 seconds to its block in the file.
* Block offset is equal to its global node number, that is node 0 will write to the first block in the file, node 1 to the second block and so on.
* All nodes will read the heartbeat system file every 2 seconds.
* As long as timestamp is changing that node is consider alive.
* A node self-fences if it fails to update its timestamp for ((O2CB_HEARTBEAT_THRESHOLD – 1) * 2) seconds.
* The [o2hb-xx] kernel thread, after every timestamp write, sets a timer to panic the system after that duration.
* If the next timestamp is written within that duration, as it should, it first cancels old timer before setting up a new one.
* If for some reason the [o2hb-x] kernel thread is unable to update the timestamp for O2CB_HEARTBEAT_THRESHOLD (default=7 or 31) loops and thus be deemed dead by other nodes in the cluster and OCFS2 forces kernel to restart.
* Once a node is deemed dead, the surviving node which manages cluster, lock the dead node’s journal, recovers it by replaying the journal.
From the above steps it is evident that the parameter O2CB_HEARTBEAT_THRESHOLD=(((timeout in secs) / 2) + 1), is very important in defining the time line to restart the server.
The default value is 31, which means if the node does not update the timestamp in the heartbeat system file in 60 sec then that node restarts. This value is quite low for RAC environment.
Assume that we have a 2 controller SAN storage and configured active/passive that is at any particular time only 1 controller path is active other is passive and used for failover. OCFS2 forces the kernel to restart (after the timeout) when a cable to a SAN device is cut even if the SAN configuration was going to perform failover.
To over come this we have to increase the value of O2CB_HEARTBEAT_THRESHOLD. If you want to increase to 120sec then the value should be 61.
Steps to change O2CB_HEARTBEAT_THRESHOLD
=====================================
1. Stop O2CB services in the server
#service O2CB stop
2. Update the O2CB configuration file with required value for O2CB_HEARTBEAT_THRESHOLD
# service o2cb configure
Configuring the O2CB driver.
This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets (‘[]‘). Hitting
without typing an answer will keep that current value. Ctrl-C
will abort.
Load O2CB driver on boot (y/n) [y]:
Cluster stack backing O2CB [o2cb]:
Cluster to start on boot (Enter “none” to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [31]: 61
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
Loading filesystem “configfs”: OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading filesystem “ocfs2_dlmfs”: OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK
#
Alternate way to update the value, is to edit file /etc/sysconfig/o2cb and start O2CB service.
3. Check the status of O2CB
# /etc/init.d/o2cb status
Driver for “configfs”: Loaded
Filesystem “configfs”: Mounted
Driver for “ocfs2_dlmfs”: Loaded
Filesystem “ocfs2_dlmfs”: Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 61
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active
#
References
========
1. OCFS2: A Cluster File System for Linux – User’s Guide for Release 1.4
2. OCFS2 Kernel Panics on SAN Failover [ID ]
3. OCFS2 1.2 – FREQUENTLY ASKED QUESTIONS [ID ]
4. Heartbeat/Voting/Quorum Related Timeout Configuration for Linux, OCFS2, RAC Stack to Avoid Unnecessary Node Fencing, Panic and Reboot [ID ]
In my next blog, I will explain the detail steps for configuring multipath using native device mapper(DM) in Linux.
OCFS2 Fencing的更多相关文章
- ocfs2: 搭建环境
OCFS2是基于共享磁盘的集群文件系统,它在一块共享磁盘上创建OCFS2文件系统,让集群中的其它节点可以对磁盘进行读写操作.OCFS2由两部分内容构成,一部分实现文件系统功能,位于VFS之下和Ext4 ...
- 在Oracle Linux Server release 6.4下配置ocfs2文件系统
① 安装ocfs-tools-1.8 如果是使用RedHat Enterprise Linux 6.4,也可以安装ocfs-tools-1.8的,只是要插入Oracle Linux Server re ...
- USACO Section 5.1 Fencing the Cows(凸包)
裸的凸包..很好写,废话不说,直接贴代码. ----------------------------------------------------------------------------- ...
- Oracle 11g RAC database on ASM, ACFS or OCFS2
I see a lot of questions on shared file systems that can be used when people move from single instan ...
- How To Configure VMware fencing using fence_vmware_soap in RHEL High Availability Add On(RHEL Pacemaker中配置STONITH)
本文主要简单介绍一下如何在RHEL 7 Pacemaker中配置一个fence_vmware_soap类型的STONITH设备(仅供测试学习). STONITH是Shoot-The-Other-Nod ...
- Reboot-less node fencing in Oracle Clusterware 11g Release 2
在进行一次RAC的高可用性测试时,当private网卡的网线被拔掉之后,没有出现传说中的有一个节点被CRS强制重启,取而代之的是node2上面的ASM实例和RDBMS实例被关闭:当网线被重新插上时,n ...
- How To Configure VMware fencing using fence_vmware_soap in RHEL High Availability Add On——RHEL Pacemaker中配置STONITH
本文主要简单介绍一下如何在RHEL 7 Pacemaker中配置一个fence_vmware_soap类型的STONITH设备(仅供测试学习). STONITH是Shoot-The-Other-Nod ...
- USACO 5.1 Fencing the Cows
Fencing the CowsHal Burch Farmer John wishes to build a fence to contain his cows, but he's a bit sh ...
- Red Hat Cluster Suite 组件 fencing FAQ
说明 Red Hat Cluster实现HA的关键组件之一是fencing.没有设置fencing,虽然看上去也能够运行Cluster,但是一旦遇到故障切换就会出现异 常,所以深入理解fencing原 ...
随机推荐
- (转)Android内存泄漏分析及调试
http://blog.csdn.net/gemmem/article/details/13017999 此文承接我的另一篇文章:Android进程的内存管理分析 首先了解一下dalvik的Ga ...
- MEF学习总结(1)---总体架构
用了很久的MEF框架来做依赖注入,最近想把它的原理和机构总结一下,主要包括如下几个方面: 1. 总体架构 2. .Net Composition Primitive 3. Attribute Mode ...
- git 修改提交说明 commit message
修改最近一次的提交说明 1.代码未推送到远程服务器 $ git commit --amend 此指令会打开文本编辑器,第二行就是提交说明,修改完后按 ctrl+x 退出,后面根据提示操作. 2.代码已 ...
- textArea中的maxlength是无效的 解决办法
-------------------------------------------------------------------------------------- <s:texta ...
- linux命令ls -l的total是怎么计算出来的?
自己手写实现ls -l命令纠结了很久的地方,刚才总算搞明白了.直接上代码重要部分 long nTotalBlocks = 0; DIR* dp = opendir(dirPathName); stru ...
- EFCodeFirst使用Nuget更新数据库
在MVC开发中,习惯于使用EF作为数据库操作,相对于传统的Ado.Net的数据库操作方式,EF大大的节省了我们手写SQL语句的时间,即便是传统的使用代码生成的方式.EF操作数据库目前分为两种大的方式. ...
- Linux系统级别能够打开的文件句柄的数file-max命令
简单的说, max-file表示系统级别的能够打开的文件句柄的数量, 而ulimit -n控制进程级别能够打开的文件句柄的数量. man 5 proc, 找到file-max的解释:file-max中 ...
- SharePoint 事件 7363:对象缓存:缓存使用的超级读者帐户没有足够的权限访问SharePoint数据库。
转自MSND:http://technet.microsoft.com/zh-cn/library/ff758656(v=office.14) 对象缓存存储 Microsoft SharePoint ...
- javascript json数据的处理
1.前端页面default.html <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" " ...
- 关于No ManagedConnections available within configured blocking timeout异常的解决
最近由于系统和业务重构需要,需要把线上1亿数据迁移到新库,由于业务变更,新表老表结构有变化,没法直接用dba dump的方式,需要自己写转换程序迁移.今天在调试的时候,碰到一个蛋疼的问题,就是一开始查 ...