ORA-16401: archivelog rejected by RFS

ORA-16401: archivelog rejected by RFS

无线出单系统邮件告警10.111.20.1

1.     报错

SYS > ! oerr ora 16041

16041, 00000, "Remote File Server fatal error"

// *Cause:  The Remote File Server (RFS) process at a redo transport

//          destination encountered a fatal error and was no longer

//          available. Further redo transmission to this destination may not be

//          possible.

// *Action: Refer to the appropriate RFS trace file located at the redo

//          transport destination for details regarding the error encountered

//          and take corrective action.

远程传输错误，RFS服务器，与参数fal_server,fal_client参数有关

2.     搜索百度

通常此问题是由于主库的fal_client参数与备库的log_archive_dest_n中的service参数不匹配造成的。

3.     查询参数

Primary

--查询DG成员

SQL> show parameter config

log_archive_config          rac,dgwless,wxcdrac

--减少自己本身，可能存在两个DG成员

--查询主库db_unique_name

SQL> show parameter db_unique_name

db_unique_name                wxcdrac

--查询主库归档参数

SQL> show parameter archive

NAME                 VALUE

---------------------------------------------------

log_archive_dest_2   service=dgwless  LGWR ASYNC va

lid_for=(online_logfiles,prima

ry_role) db_unique_name=dgwless

log_archive_dest_3   SERVICE=RAC LGWR ASYNC VALID_F

OR=(ONLINE_LOGFILES,PRIMARY_RO

LE) db_unique_name=rac

--查询归档线程参数状态，以上归档线程3对应的参数可以过滤，因为此DG可能已被废弃

--除非是人为或者误操作置为defer，否则基本判断问题，排除此DG

log_archive_dest_state_2             ENABLE

log_archive_dest_state_3             DEFER

--查询主库fal参数

SQL> show parameter fal

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

fal_client                           string      wxcdrac2

fal_server                           string      dgwless

--通过主库参数，查询备库IP地址

SQL> host tnsping dgwless

Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.135.101.23)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dgwless)))

OK (90 msec)

--连接备库后，查询验证主备关系

SQL> select dbid,name, database_role,open_mode,FLASHBACK_ON  from v$database;

DBID NAME            DATABASE_ROL OPEN_MODE  FLASHBACK_ON

----------- --------------- ------------ ---------- ------------------

2283108100 RAC             PRIMARY      READ WRITE NO

SQL> select dbid,name, database_role,open_mode,FLASHBACK_ON  from v$database;

DBID NAME            DATABASE_ROL OPEN_MODE  FLASHBACK_ON

----------- --------------- ------------ ---------- ------------------

2283108100 RAC             PHYSICAL STANDBY   MOUNTED    NO

--standby

SQL> show parameter fal

NAME          VALUE

------------- ------------------------------

fal_client    dgwless

fal_server    wxcdrac  --发现与主库fal_client参数不同

--查询备库归档参数

log_archive_dest_1   location=/picclife/app/oracle/archdg/

log_archive_dest_2   SERVICE=WXCDRAC LGWR ASYNC VAL  --发现tns也与主库fal_client不同

ID_FOR=(ONLINE_LOGFILES,PRIMAR

Y_ROLE) db_unique_name=wxcdrac

--主库fal_client 在11g已被弃用，按照道理将主库fal_client与备库log_archvie_service不同，无异常，查询版本，

当前10G

Enterprise Edition Release 10.2.0.3.0

--查询主库集群状态

wxcdrac2:/home/oracle$ export ORACLE_SID=+ASM1

wxcdrac2:/home/oracle$ cat .profile

export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1

export ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crs_1

wxcdrac2:/home/oracle$ cd $ORA_CRS_HOME/bin

wxcdrac2:/picclife/app/oracle/product/10.2.0/crs_1/bin$

wxcdrac2:/picclife/app/oracle/product/10.2.0/crs_1/bin$ crs_stat -t

Name           Type           Target    State     Host

------------------------------------------------------------

ora.wxcdrac.db application    ONLINE    ONLINE    wxcdrac1

ora....c1.inst application    ONLINE    ONLINE    wxcdrac1

ora....c2.inst application    ONLINE    ONLINE    wxcdrac2

ora....SM1.asm application    ONLINE    ONLINE    wxcdrac1

ora....C1.lsnr application    ONLINE    ONLINE    wxcdrac1

ora....ac1.gsd application    ONLINE    ONLINE    wxcdrac1

ora....ac1.ons application    ONLINE    ONLINE    wxcdrac1

ora....ac1.vip application    ONLINE    ONLINE    wxcdrac1

ora....SM2.asm application    ONLINE    ONLINE    wxcdrac2

ora....C2.lsnr application    ONLINE    ONLINE    wxcdrac2

ora....ac2.gsd application    ONLINE    ONLINE    wxcdrac2

ora....ac2.ons application    ONLINE    ONLINE    wxcdrac2

ora....ac2.vip application    ONLINE    ONLINE    wxcdrac2

4.     查询日志

--备库日志

Wed Aug  8 01:59:19 2018

Redo Shipping Client Connected as PUBLIC

-- Connected User is Valid

RFS[2586]: Assigned to RFS process 29818962

RFS[2586]: Identified database type as 'physical standby'

Primary database is in MAXIMUM PERFORMANCE mode

Primary database is in MAXIMUM PERFORMANCE mode

RFS[2586]: Successfully opened standby log 15: '/picclife/app/oracle/oradata/dgwless/std02_1.log'

Wed Aug  8 02:04:42 2018

RFS[2579]: Archivelog thread 2 sequence 3661 cannot be reused

Wed Aug  8 02:04:42 2018

Errors in file /picclife/app/oracle/admin/dgwless/udump/dgwless_rfs_13238468.trc:

ORA-16401: archivelog rejected by RFS

Redo Shipping Client Connected as PUBLIC

-- Connected User is Valid

--查询trace文件

KCRRSNPS

Identified standby redo log 12 for implicit mid-log reconnect

*** 2018-08-08 02:04:42.324

*** 2018-08-08 02:04:42.324 61287 kcrr.c

RFS[2579]: Archivelog thread 2 sequence 3661 cannot be reused

This problem can occur as the result of multiple primary and/or

standby database(s) trying to archive to this standby database.

This is not allowed, since successfully completed archivelogs

would be overwritten.

ORA-16401: archivelog rejected by RFS

*** 2018-08-08 02:13:43.562

Dispatching RFS shutdown notification

以上备库查询得到的信息是，归档接受出现网络问题，但不明确

查询主库Alert日志

--查询邮件告警所在的数据库Alert日志

SQL> show parameter background_dump_dest

NAME                  VALUE

--------------------- ------------------------------

background_dump_dest  /picclife/app/oracle/admin/wxcdrac/bdump

Wed Aug  8 01:02:57 2018

LNS: Standby redo logfile selected for thread 1 sequence 3993 for destination LOG_ARCHIVE_DEST_2

Wed Aug  8 01:03:32 2018

Starting control autobackup

Control autobackup written to SBT_TAPE device

comment 'API Version 2.0,MMS Version 10.0.0.116',

media 'V_89405_737884'

handle 'c-2283108100-20180808-01'

Wed Aug  8 01:08:19 2018

Errors in file /picclife/app/oracle/admin/wxcdrac/bdump/wxcdrac1_arc2_10682458.trc:

ORA-16401: archivelog rejected by RFS

Wed Aug  8 01:11:58 2018

ARC0: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (3135)

ARC0: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned

Wed Aug  8 01:11:58 2018

Errors in file /picclife/app/oracle/admin/wxcdrac/bdump/wxcdrac1_arc0_11337924.trc:

ORA-03135: connection lost contact

FAL[server, ARC0]: Error 3135 creating remote archivelog file 'dgwless'

FAL[server, ARC0]: FAL archive failed, see trace file.

Wed Aug  8 01:11:58 2018

Errors in file /picclife/app/oracle/admin/wxcdrac/bdump/wxcdrac1_arc0_11337924.trc:

ORA-16055: FAL request rejected

ARCH: FAL archive failed. Archiver continuing

Wed Aug  8 01:11:58 2018

ORACLE Instance wxcdrac1 - Archival Error. Archiver continuing.

*** 2018-08-08 01:08:19.693

Error 16401 creating standby archive log file at host 'dgwless'

ORA-16401: archivelog rejected by RFS

--从以上的信息还是未明确发现由于参数问题导致的报错

5.     搜索MOS

1183143.1

ORA-16401: ARCHIVELOG REJECTED BY RFS error occurs when incorrect naming convention for fal_client parameter is set.

Version 10.2.0.4 to 11.1.0.7 [Release 10.2 to 11.1]

配置主备不一致的fal_client时会发生此错误

***** Primary *****

log_archive_config DG_CONFIG=(p1ncm1,p1ncm2)

log_archive_dest_1 location=/u99/ORACLE/p1ncm1/archive valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=p1ncm1

log_archive_dest_2 service=p1ncm2_dgfal.csm.fub.com optional reopen=60 valid_for=(ONLINE_LOGFILES,PRIMARY_ROLE) db_unique_name=p1ncm2

fal_client p1ncm1_fc.csm.fub.com

fal_server p1ncm2_fs.csm.fub.com

********* Standby ********

log_archive_config DG_CONFIG=(p1ncm1,p1ncm2)

log_archive_dest_1 location=/u99/ORACLE/p1ncm2/archive valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=p1ncm2

log_archive_dest_2 service=p1ncm1_dgfal.csm.fub.com optional reopen=60 valid_for=(ONLINE_LOGFILES,PRIMARY_ROLE) db_unique_name=p1ncm1

fal_client p1ncm2_fc.csm.fub.com

fal_server p1ncm1_fs.csm.fub.com

SOLUTION

standby

alter system set fal_client=p1ncm2_dgfal.csm.fub.com scope=both sid='*';

6.     处理

-修改主库的fal_client参数，与备库保持一致

--修改前需要进行验证

SQL> host tnsping wxcdrac2

SQL> host tnsping wxcdrac

SQL> alter system set fal_client=wxcdrac;

--第二次，在其它客户遇到同样报错，问题是tnsping 主库fal_client不通，端口错误。

7.     补充说明

MOS ID 288074.1

Versions 11.2, 12.1  对于11g12c而言，很可能此消息是一个提示作用

Error:  ORA-16401 archive log rejected by Remote File Server (RFS)

---------------------------------------------------------------------------

Cause:  An attempt was made to re-archive an existing archive log.

Action: See alert log and trace file for more details. No action is necessary.

This is an informational statement provided to record the event for

diagnostic purposes.

Versions 9.2, 10.1, 10.2, 11.1

Error:  ORA-16401 archivelog rejected by RFS  对于10g而言，此报错，很可能为fal_client参数配置错误

---------------------------------------------------------------------------

Cause:  An attempt was made to re-archive an existing archivelog. This usually

happens because either a multiple primary database or standby

database(s) or both are trying to archive to this standby database.

Action: See alert log and trace file for more details. No action is necessary;

this is an informational statement provided to record the event for

diagnostic purposes.