Greenplum启动失败Error occurred: non-zero rc: 1的修复

某日开发反馈测试环境的集群启动失败

报错内容如下：

[gpadmin@hadoop-test2:/root]
$ gpstart
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-Starting gpstart with args:
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-Gathering information and validating the environment...
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.0.0 build dev'
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-Greenplum Catalog Version: ''
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-Starting Master instance in admin mode
:::: gpstart:hadoop-test2:gpadmin-[CRITICAL]:-Failed to start Master instance in admin mode
:::: gpstart:hadoop-test2:gpadmin-[CRITICAL]:-Error occurred: non-zero rc:
 Command was: 'env GPSESSID=0000000000 GPERA=None $GPHOME/bin/pg_ctl -D /home/gpadmin/gpdata/gpmaster/gpseg-1 -l /home/gpadmin/gpdata/gpmaster/gpseg-1/pg_log/startup.log 
-w -t 600 -o " -p 2346 --gp_dbid=1 --gp_num_contents_in_cluster=0 --silent-mode=true -i -M master --gp_contentid=-1 -x 0 -c gp_role=utility " start'
rc=, stdout='waiting for server to start...................................................................................................................................
...........................................................................................................................................................................
...........................................................................................................................................................................
.................................................................................................................................. stopped waiting
', stderr='could not change directory to "/root"
pg_ctl: could not start server
Examine the log output.

查看启动日志发现：

vim /home/gpadmin/gpdata/gpmaster/gpseg-1/pg_log/startup.log

-- ::24.067241 GMT,,,p5464,th-,,,,,,,seg-,,,,,"WARNING","","""work_mem"": setting is deprecated, and may be removed in a future release.",,,,,,,,"set_config_option","guc.c",,
-- ::24.067612 GMT,,,p5464,th-,,,,,,,seg-,,,,,"WARNING","","""work_mem"": setting is deprecated, and may be removed in a future release.",,,,,,,,"set_config_option","guc.c",,
-- ::24.083813 GMT,,,p5465,th-,,,,,,,seg-,,,,,"LOG","","removing all temporary files",,,,,,,,"RemovePgTempFiles","fd.c",,
-- ::24.098673 GMT,,,p5465,th-,,,,,,,seg-,,,,,"FATAL","XX000","could not create shared memory segment: Invalid argument (pg_shmem.c:183)","Failed system call was shmget(key=2346001, size=177586016, 03600).","This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter.  You can either reduce the request size or reconfigure the kernel with larger SHMMAX.  To reduce the request size (currently 177586016 bytes), reduce PostgreSQL's shared_buffers parameter (currently 4000) and/or its max_connections parameter (currently 253).
If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
The PostgreSQL documentation contains more information about shared memory configuration.",,,,,,"InternalIpcMemoryCreate","pg_shmem.c",183,1

内容大概是说/etc/sysctl.conf设置的内核参数shmmax过小，导致启动失败

查看/etc/sysctl.conf下的配置发现：

kernel.shmmax =
kernel.shmmni =
kernel.shmall =
kernel.sem =
kernel.sysrq =
kernel.core_uses_pid =
kernel.msgmnb =
kernel.msgmax =
kernel.msgmni =
net.ipv4.tcp_syncookies =
net.ipv4.ip_forward =
net.ipv4.conf.default.accept_source_route =
net.ipv4.tcp_tw_recycle =
net.ipv4.tcp_max_syn_backlog =
net.ipv4.conf.all.arp_filter =
net.ipv4.ip_local_port_range =
net.core.netdev_max_backlog =
net.core.rmem_max =
net.core.wmem_max =
vm.overcommit_memory = 

对比官网建议的设置和参数定义以及集群已有的数据量，发现确实过小。于是改成官网建议的设置后启动。

:::: gpstart:hadoop-test2:gpadmin-[INFO]:-----------------------------------------------------
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-   Successful segment starts                                            =
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-   Failed segment starts                                                =
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-   Skipped segment starts (segments are marked down in configuration)   =
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-----------------------------------------------------
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-Successfully started  of  segment instances
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-----------------------------------------------------
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-Starting Master instance hadoop-test2 directory /home/gpadmin/gpdata/gpmaster/gpseg-
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-Command pg_ctl reports Master hadoop-test2 instance active
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-No standby master configured.  skipping...
:::: gpstart:hadoop-test2:gpadmin-[INFO]:-Database successfully started

启动成功。

总结：pg启动相关的内核参数配置与实际情况不匹配时，会导致启动失败。可通过查看日志详细信息查找根源解决问题。

参考文档：

1、官网建议设置 http://gpdb.docs.pivotal.io/4380/prep_os-system-params.html#topic3

2、内核参数含义http://www.oicqzone.com/pc/2012091612901.html