RAC升级后，一个节点无法连接数据库，报ORA-12537: TNS:connection closed

RAC从11.2.0.3升级到11.2.0.4后，一个节点的Public IP、VIP无法连接数据库

SQL> CONN SYS/oracle@192.168.122.101:1521/pplus as sysdba
ERROR:
ORA-12537: TNS:connection closed

查看RAC的监听日志：

# cd $ORACLE_BASE/diag/tnslsnr/racdb2/listener/trace/

# vi listener.log

06-MAY-2019 12:13:52 * (CONNECT_DATA=(SERVICE_NAME=posplus)(CID=(PROGRAM=sqlplus)(HOST=racdb1)(USER=grid))) * (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.122.100)(PORT=40081)) * establish * posplus * 12518
TNS-12518: TNS:listener could not hand off client connection
TNS-12546: TNS:permission denied
TNS-12560: TNS:protocol adapter error
TNS-00516: Permission denied
Linux Error: 13: Permission denied

首先想到GRID和Database的是oracle文件权限的问题，都有s权限，确认没有问题

[grid@racdb1 ~]$ ll $ORACLE_HOME/bin/oracle

-rwsr-s--x 1 grid oinstall 209914479 Mar 25 23:39 /home/grid/app/11.2.0.4/grid/bin/oracle

[oracle@racdb1 ~]$ ll $ORACLE_HOME/bin/oracle

-rwsr-s--x 1 oracle asmadmin 239626641 Mar 26 01:36 /home/oracle/app/oracle/product/11.2.0.4/db_1/bin/oracle

然后再用GRID用户去访问Database的HOME目录

[grid@racdb1 ~]$ cd $ORACLE_HOME/bin
-bash: cd: $ORACLE_HOME/bin: Permission denied

果然没有权限

[grid@racdb2 ~]$ ll /home/
total 8
drwxr-xr-x. 7 root oinstall 4096 Mar 23 16:43 grid
drwx------. 8 oracle oinstall 4096 May 6 11:45 oracle

把/home/oracle的权限修改为

chmod g+rx,o+rx /home/oracle

[grid@racdb1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed May 8 10:41:06 2019

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> CONN SYS/oracle@192.168.122.111:1521/posplus as sysdba
Connected.
SQL> CONN SYS/oracle@192.168.122.101:1521/posplus as sysdba
Connected.
SQL> CONN SYS/oracle@192.168.122.110:1521/posplus as sysdba
Connected.
SQL> CONN SYS/oracle@192.168.122.120:1521/posplus as sysdba

至此问题已解决

-------------------------------------------------------------------------------

参考Oracle官方文档：

ORA-12537 / ORA-12547 or TNS-12518 if Listener (including SCAN Listener) and Database are Owned by Different OS User (文档 ID 1069517.1)

In this Document

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Oracle Database Configuration Assistant - Version 11.1.0.7 and later
Oracle Net Services - Version 11.2.0.2 and later
Information in this document applies to any platform.

SYMPTOMS

In environment where listener home (including SCAN listener which resides in GRID Infrastructure/ASM home) and database home are owned by different OS user, ORA-12537 could happen when connecting through listener, when creating database through DBCA, or when installing database software and creating a database in runInstaller. Job Role Separation is a typical example as SCAN and local grid home listener is owned differently than database.

Error detail

12537, 00000, "TNS:connection closed"
// *Cause: "End of file" condition has been reached; partner has disconnected.
// *Action: None needed; this is an information message.

Before the error is reported, connection could hang for a while, and a core file may also be generated.

Screen output:

SQL> conn system/xxx@OSES
ERROR:
ORA-12537: TNS:connection closed

OR

PRCR-1079 : Failed to start resource ora.db11.db
CRS-5011: Check of resource db11 failed: details at ..
CRS-2674: Start of ora.db11.db on racnode1 failed
ORA-12546: TNS:permission denied

listener_scann.log or listener.log

TNS-12518: TNS:listener could not hand off client connection
TNS-12546: TNS:permission denied
TNS-12560: TNS:protocol adapter error
TNS-00516: Permission denied
Linux Error: 13: Permission denied

OR

TNS-12518: TNS:listener could not hand off client connection
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 32: Broken pipe

runInstaller error if installing

INFO: Starting Output Reader Threads for process /ocw/grid/bin/kfod
INFO: Parsing KFOD-00300: OCI error [-1] [OCI error] [ORA-12547: TNS:lost contact
INFO: Parsing ] [12547]
INFO: Parsing
INFO: The process /ocw/grid/bin/kfod exited with code 1
..
SEVERE: [FATAL] [INS-30502] No ASM disk group found.
CAUSE: There were no disk groups managed by the ASM instance +ASM1.

CAUSE

1. In environments where the listener is not run in the same ORACLE_HOME where the database resides, the listener owner (including SCAN listener) may not be able to access the oracle binary in the database home. This is common in RAC or whenever a GRID_HOME and a database ORACLE_HOME are installed.

As listener owner:

$ ls -l $RDBMS_HOME/bin/oracle
ls: /home/oracle/app/oracle/product/11.2/db/bin/oracle: Permission denied

2. Oracle binary in database home has wrong permission:

ls -l $RDBMS_HOME/bin/oracle
-rwxr-x--x 1 oracle asmadmin 184286251 Aug 9 16:25 /home/oracle/app/oracle/product/11.2/db/bin/oracle

The permission "-rwxr-x--x" is wrong as it's missing suid bit, oracle binary should have permission of 6751:

-rwsr-s--x 1 oracle asmadmin 184286251 Aug 9 16:25 /home/oracle/app/oracle/product/11.2/db/bin/oracle

Note: If Job Role Separation is in place, the group will be <asmadmin>, otherwise it will be <oinstall>

3. File System for database home does not support setuid/suid or has nosetuid/nosuid set:

mount| grep <mount_point_of_ORACLE_HOME>
/home/oracle on /dev/dsk/diskoracle read/write/nosuid..

4. RDBMS_HOME/lib has wrong ownership/permission:

As listener owner:

$ ls -l $RDBMS_HOME/lib
ls: /home/oracle/app/oracle/product/11.2/db/lib: Permission denied

5. Another cause may be permissions on the RDBMS Directory structure or Home directory which needs to be accessed by the CRS user.
Check that the RDBMS $ORACLE_HOME is set to 755.
This can be seen from an OS trace such as strace or truss when using it to trace the CRS user running the "oracle" executable which fails with the "Permission denied" error.

Also:

a) Log in as the "GRID" user on each node, and issue the following (on each directory under the RDBMS Home) :-

[grid@orcl002:+ASM2 ~]$ ls -al /home/oracle/app/oracle
ls: cannot open directory /u01/app/oracle: Permission denied

***NOTE: the Oracle directory has 700 for the permissions, which should be changed to 755:

Current (incorrect) -->

[grid@orcl002:+ASM2 ~]$ ll /home/oracle/app
drwx------. 8 oracle oinstall 4096 Oct 12 08:38 oracle

Should be (correct) -->

[grid@orcl002:+ASM2 ~]$ ll /home/oracle/app
drwxr-xr-x. 8 oracle oinstall 4096 Oct 12 08:38 oracle

b) Likewise the /product directory has 700 perms, so change to 755 -->

[grid@orcl002:+ASM2 ~]$ ls -al /home/oracle/app/oracle/product
ls: cannot open directory /u01/app/oracle/product: Permission denied

Current (incorrect) -->

[grid@orcl002:+ASM2 ~]$ ls -al /home/oracle/app/oracle
drwx------. 3 oracle oinstall 4096 Oct 12 08:58 product

Should be (correct) -->

[grid@orcl002:+ASM2 ~]$ ls -al /u01/app/oracle
drwxr-xr-x. 3 oracle oinstall 4096 Oct 12 08:58 product

c) Now a connection should work:

$ sqlplus system/<pwd>@'(description=(address=(protocol=tcp)(host=orcl002)(port=1521))(connect_data=(server=dedicated)(service_name=ORCL)))'

SQL>

SOLUTION

Solution is to make sure file system for database home has setuid/suid set, database binary($RDBMS_HOME/bin/oracle) has correct ownership and permission, and listener owner is able to access database oracle binary (as listener owner, "ls -l $RDBMS_HOME/bin/oracle" will tell)

If its Job Role Separation environment, please refer to the following document:

Oracle® Grid Infrastructure
Installation Guide
11g Release 2 (11.2)

Oracle ASM Job Role Separation Option with SYSASM