Hadoop Kernel tunning

/etc/security/limits.conf

@ochadoop soft nofile 102642
@ochadoop hard nofile 102642
@ochadoop soft nproc unlimited
@ochadoop hard nproc unlimited
@ochadoop soft memlock unlimited
@ochadoop hard memlock unlimited

/etc/sysctl.conf

#尽可能少地让应用把内容交换到硬盘
echo "vm.swappiness = 0" >> /etc/sysctl.conf
#可以让应用在请求内存的时候使用交换
echo 'vm.overcommit_memory = 1' >> /etc/sysctl.conf

Mount options:disabling file access time

The real problem with maintaining access time (or

atime as it’s commonly called) is that every time a file is read, the metadata needs to be

updated. That is, for each read, there’s also a mandatory write. This is relatively expensive at scale and can negatively impact the overall performance of Hadoop

sample:

LABEL=/ / ext3 noatime,nodiratime 1 1
LABEL=/data/1 /data/1 ext3 noatime,nodiratime 1 2
LABEL=/data/2 /data/2 ext3 noatime,nodiratime 1 2
LABEL=/data/3 /data/3 ext3 noatime,nodiratime 1 2
LABEL=/data/4 /data/4 ext3 noatime,nodiratime 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sda2 swap swap defaults 0 0