IIS不定期Crash和Oracle“未处理的内部错误(-2)”的问题分析

问题描述：系统不定期报出Oracle“未处理的内部错误(-2)”，严重时IIS会Crash

典型异常日志如下：

Exception type:   System.AccessViolationException

Message:          尝试读取或写入受保护的内存。这通常指示其他内存已损坏。

InnerException:   <none>

StackTrace (generated):

SP               IP               Function

000000000C388FE0 0000000000000001 System_Data_OracleClient_ni!System.Data.Common.UnsafeNativeMethods.OCILobCreateTemporary(System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle, UInt16, CHARSETFORM, LOB_TYPE, Int32, DURATION)+0x2

000000000C3890F0 000007FEF4627DEF System_Data_OracleClient_ni!System.Data.OracleClient.TracedNativeMethods.OCILobCreateTemporary(System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle, UInt16, CHARSETFORM, LOB_TYPE, Int32, DURATION)+0xef

000000000C3891A0 000007FEF464FC82 System_Data_OracleClient_ni!System.Data.OracleClient.OracleLob..ctor(System.Data.OracleClient.OracleConnection, System.Data.OracleClient.OracleType)+0xc2

000000000C389210 000007FEF463ED33 System_Data_OracleClient_ni!System.Data.OracleClient.OracleParameterBinding.CreateTemporaryLobForValue(System.Data.OracleClient.OracleConnection, System.Data.OracleClient.OracleType, System.Object)+0x63

000000000C389260 000007FEF463FE68 System_Data_OracleClient_ni!System.Data.OracleClient.OracleParameterBinding.PrepareForBind(System.Data.OracleClient.OracleConnection, Int32 ByRef)+0x178

000000000C3892D0 000007FEF46322A1 System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.Execute(System.Data.OracleClient.OciStatementHandle, System.Data.CommandBehavior, Boolean, System.Data.OracleClient.OciRowidDescriptor ByRef, System.Collections.ArrayList ByRef)+0x381

000000000C3893C0 000007FEF46328C6 System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.ExecuteNonQueryInternal(Boolean, System.Data.OracleClient.OciRowidDescriptor ByRef)+0xa6

000000000C389450 000007FEF46327DE System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.ExecuteNonQuery()+0x10e

。。。。。。。。。。。

。。。。。。。。。。。

000000000C38E590 000007FEF8FCF8A6 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x286

000000000C38E6F0 000007FEF8FCF60A mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0xa

000000000C38E720 000007FEF8F7B220 mscorlib_ni!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()+0x70

000000000C38E770 000007FEF8F7B82B mscorlib_ni!System.Threading.ThreadPoolWorkQueue.Dispatch()+0x1eb

Nested exception -------------------------------------------------------------

Exception object: 0000000182836198

Exception type:   System.InvalidOperationException

Message:          未处理的内部错误(-2)。

InnerException:   <none>

StackTrace (generated):

SP               IP               Function

000000000C38DDD0 000007FEF469DE51 System_Data_OracleClient_ni!System.Data.OracleClient.OracleConnection.CheckError(System.Data.OracleClient.OciErrorHandle, Int32)+0x71951

000000000C38DE10 000007FEF463244B System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.Execute(System.Data.OracleClient.OciStatementHandle, System.Data.CommandBehavior, Boolean, System.Data.OracleClient.OciRowidDescriptor ByRef, System.Collections.ArrayList ByRef)+0x52b

000000000C38DF00 000007FEF4631F14 System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.Execute(System.Data.OracleClient.OciStatementHandle, System.Data.CommandBehavior, System.Collections.ArrayList ByRef)+0x24

000000000C38DF50 000007FEF4633E00 System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.ExecuteReader(System.Data.CommandBehavior)+0x130

000000000C38E010 000007FEF46326B9 System_Data_OracleClient_ni!System.Data.OracleClient.OracleCommand.ExecuteDbDataReader(System.Data.CommandBehavior)+0x9

000000000C38E040 000007FEF60C67A5 System_Data_ni!System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(System.Data.CommandBehavior)+0x15

000000000C38E070 000007FEF60AA555 System_Data_ni!System.Data.Common.DbDataAdapter.FillInternal(System.Data.DataSet, System.Data.DataTable[], Int32, Int32, System.String, System.Data.IDbCommand, System.Data.CommandBehavior)+0x145

000000000C38E120 000007FEF60AA394 System_Data_ni!System.Data.Common.DbDataAdapter.Fill(System.Data.DataSet, Int32, Int32, System.String, System.Data.IDbCommand, System.Data.CommandBehavior)+0x1a4

000000000C38E1D0 000007FEF60AA196 System_Data_ni!System.Data.Common.DbDataAdapter.Fill(System.Data.DataSet)+0x116

原因分析：

查阅资料发现使用DBLink查询时，oracle会开启隐式分布式事务，进而会引发ORA-24784和内部错误(-2)的异常。

与业务组同事交流，项目上确实使用了DBLink；

分析服务器的事件查看器和业务系统日志，也发现了ORA-24784的Exception，如下：

{"code":24784,"ClassName":"System.Data.OracleClient.OracleException","Message":"ORA-24784: 事务处理已存在\u0000","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"

System.Data.OracleClient.OciEnlistContext.Join(OracleInternalConnection internalConnection, Transaction indigoTransaction)\r\n

System.Data.OracleClient.OracleInternalConnection.Enlist(String userName, String password, String serverName, Transaction transaction, Boolean manualEnlistment)\r\n

System.Data.OracleClient.OracleInternalConnection.Activate(Transaction transaction)\r\n

System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)\r\n

System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)\r\n

System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)\r\n

System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)\r\n

System.Data.OracleClient.OracleConnection.Open()\r\n

Demo验证：

       经过分析测试验证，使用DBLink的查询确实会开启一个隐式的事务，该数据库连接被重用后如果再显示的开启一个分布式事务时，系统会产生ORA-24784: 事务处理已存在的错误。

问题结论

在并发的混合场景下，通过抓取first chance Exception dump，发现ORA-24784与(-2)、IIS Crash相伴而生，具有很强的关联性。

代码排查所有DBLink的使用，DBLink查询后增加显式的commit调用，以提交因oracle DBLink产生的隐式事务。

相关资料：

oracle官方文档：http://files.cnblogs.com/files/zhaoguan_wang/ORA-24784_Oracle_XA%E6%96%87%E6%A1%A3_ID_1271726.1.pdf