Troubleshooting 'library cache: mutex X' Waits. (Doc ID 1357946.1)

In this Document

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.1.0.6 and later
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Information in this document applies to any platform.

PURPOSE

The purpose of the article is to help troubleshoot contention for the wait event 'library cache: mutex X'.

TROUBLESHOOTING STEPS

What is a 'library cache: mutex X' wait?

The mutex feature is a mechanism to control access to in memory structures. It is used in a number of areas including the library cache.

The library cache is a memory area that holds parsed cursor structures needed to execute SQL.

Waits for 'library cache: mutex X' are similar to a library cache waits in earlier versions. 'library cache: mutex X' may be caused by many issues (including application issues, lack of sharing resulting in high version counts etc.) but essentially something is holding the mutex for "too long" such that other session have to wait for the resource. If there is contention on the latches/mutexes that protect the library cache structures this means that there is stress on the parsing system. Parsing of SQL takes longer because it cannot get the resources they need. This delays other operations and generally slows the system.

Because of the varied causes, it is important to find the correct cause; so that the right solution can be implemented.

What causes 'library cache: mutex X' wait?

Frequent Hard Parses - If the frequency of Hard Parsing is extremely high, then contention can occur on this pin.
High Version Counts - When Version counts become excessive, a long chain of versions needs to be examined and this can lead to contention on this event
Invalidations - An invalidation is a measure of the number of times a cached cursor is deleted from the cache because it is no longer valid. A cursor is invalidated because something has changed such that the copy of the cursor in memory is not valid any more. For example, regathering the statistics on an object or modifying a table definition is enough to invalidate a cursor for a query that is based on that object. When a cursor is invalidated, any sessions wanting to use the cursor need to wait for a valid version to be loaded. If there is excessive or unnecessary invalidation then significant waits for 'library cache: mutex X' can be seen.
Reloads - Reload is a count of the number of times a cursor that previously existed in the cache, was searched for, found to not be there (because it had aged out etc) and then had to be re-compiled and re-loaded in to the library cache. High reloads are a bad thing because they indicate that you are doing work that you would not have had to do if your cache was setup appropriately so as not to remove the cursor in the first place. If a cursor is being reloaded then it cannot be grabbed for work by a session and this can lead to waits for 'library cache: mutex X'.
Known Bugs

Name of events in 12c and higher

In 12, the events have been split further into three separate events:

* library cache: mutex X -- for handle objects

* library cache: bucket mutex X -- for library cache hash buckets

* library cache: dependency mutex X -- for dependencies

How to diagnose the cause.

1. Check to see if anything has changed:
     a. increased load?
     b. any change in the application, os, or middle tier?
     c. any os changes?

2. Is there a trend to the waits for 'library cache: mutex X':
a. is there a certain time of the day when this wait is seen?
b. does something trigger this wait?

3. During the time of the issue, run AWR and ADDM. Also obtain the baseline to compare the load, parameter changes, and any other differences.
To gather this it is suggested to run AWR and ADDM for half an hour to an hour interval as follows:

SQL> @$ORACLE_HOME/rdbms/admin/awrrpt.sql
SQL> @$ORACLE_HOME/rdbms/admin/addmrpt.sql

See:

Document 1680075.1 How to Generate and Check an ADDM report
Document 1903158.1 How to Collect Standard Diagnostic Information Using AWR Reports for Performance Issues

4. Sometimes system state dump is necessary to match known issues. For example, if there is no obvious candidate SQL in AWR, capturing holder or waiter processes in systemstate allows you to focus in on potential problems. Run system state when processes appear hung on 'library cache: mutex X':

(a) Non-Rac

sqlplus "/ as sysdba"

oradebug setmypid
oradebug unlimit
oradebug dump systemstate 266
wait 90 seconds
oradebug dump systemstate 266
wait 90 seconds
oradebug dump systemstate 266
quit

(b) RAC

$ sqlplus '/ as sysdba'
oradebug setmypid
oradebug unlimit
oradebug setinst all
oradebug -g all hanganalyze 4
oradebug -g all dump systemstate 266
quit

5. Errorstacks: Another way to obtain process information is with errorstack. Assuming you can identify a blocker, taking errorstacks will provide much the same information as systemstates but with a much reduced disk footprint for trace. Once the ospid of the blocker has been found, an errorstack can be generated:

$ sqlplus
SQL> oradebug setospid <p.spid from above>
oradebug dump errorstack 3
<< wait 1min>>
oradebug dump errorstack 3
<< wait 1min>>
oradebug dump errorstack 3
exit

In particular, the stack from the resultant trace can be used to match known issues.
The system state and errorstacks are not easily readable; so a Service Request may need to be opened to read the files.

6. Sometimes it is not feasible to run system state dump, as it may be resource intensive. So the following sql can also be ran in interval:

select s.sid, t.sql_text
from v$session s, v$sql t
where s.event like '%mutex%'
and t.sql_id = s.sql_id

Check to see what sessions are waiting on.

7. In 11g RAC, there is another less resource intensive tool that can be used when compared with taking system state dumps:

Document 459694.1 Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes

How to Examine the Diagnostics.

1. Normally, the top wait event will be the library cache: mutex X in the problematic AWR:

2. First look for high parsing and high version counts from AWR.
Click on *SQL Statistics under Main Report of AWR:

Then, under SQL Statistics click on 'SQL ordered by Parse Calls' and 'SQL ordered by Version Count' to view that information:

Check for high parse calls.

Check to see if there is high parse calls to execute. Ideally, there should be less parse to executions. Notice there is as many parses as executes, indicating cursors are not used well in the application. Once the cursor is opened and parsed, it should be kept open. Check with the application developer on how to keep the cursor opened to re execute the sqls.

Next, check the version count of the sql:

From this list, investigate the SQLs with the high version count. What are the reasons that these statements are not shared? Can this be addressed?
Check V$SQL_SHARED_CURSOR to see the potential reason for the high version count using:

Document 438755.1 Formated V$SQL_SHARED_CURSOR Report by SQLID or Hash Value
Document 296377.1 Troubleshooting: High Version Count Issues

Potential Solutions

1. Check for high hard parsing, as this can cause can reloads in the sql area. Check the hard parse under the load profile:

This load shows 26.3 hard parses per second, indicating high hard parsing. Check to see if the application is sharing the sql. If application is mostly using literals, see if the sqls can be shared by using bind variables. Furthermore, review the 'Over Parsing' section of the following note:

Document 33089.1 TROUBLESHOOTING: Possible Causes of Poor SQL Performance

Also check for high reloads in the sql area:

If there is a high number of reloads, then look to see if cursors are being shared efficiently (remember reloads counts cursors that were once cached but are no longer there). If they are, then check to see if the shared pool or sga_target is large enough; the cursors may be being aged out because there is insufficient space for them. Remember that inefficient sharing means that the library cache will fill up with non-reuseable cursors which may then cause reuseable ones to be flushed out. These will cause reloads when they are re-executed.
If sharing is efficient and the shared pool is too small, then shareable SQL statements will age out and hard parses will be higher. In most cases however this is not the case and the problem is inefficient sharing. The following note is helpful in tuning the shared pool:

Document 62143.1 Understanding and Tuning the Shared Pool

2. Check for invalidations under Library Cache Activity. If the invalidation has high number, then check ddl's performed during the time such as truncate, drop, grants, dbms_stats, etc.

3. Check the following note for relevant bugs under 'Known Bugs' and relevant version:

Document 727400.1 WAITEVENT: "library cache: mutex X"

4. For 11g, make sure cursor_sharing is not similar, as it has been deprecated. This may also cause mutex waits:

Document 1169017.1 ANNOUNCEMENT: Deprecating the cursor_sharing = ‘SIMILAR’ setting

5. If the database has been migrated from 10g to 11g and mutex performance issue surfaces, please consider the 11.2.0.2.2 psu + fix for Unpublished Bug 12431716. Many mutex fixes are already included in this patch:

Document 1291879.1 Oracle Database Patch Set Update 11.2.0.2.2 Known Issues

Troubleshooting heavy contention for 'library cache: mutex X' waits for 11g and later

For known issues and troubleshooting 'library cache: mutex X' wait for 11g and later, review following note:

Document 2051456.1 Troubleshooting Databases Hang Due to Heavy Contention for 'library cache: mutex X' Waits (Oracle 11.2 and Later)

REFERENCES

NOTE:459694.1 - Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes

NOTE:62143.1 - Troubleshooting: Understanding and Tuning the Shared Pool
NOTE:727400.1 - WAITEVENT: "library cache: mutex X"
NOTE:1291879.1 - Oracle Database Patch Set Update 11.2.0.2.2 Known Issues
NOTE:1169017.1 - ANNOUNCEMENT: Deprecating the Cursor_Sharing = 'SIMILAR' Setting
NOTE:296377.1 - Troubleshooting: High Version Count Issues
NOTE:33089.1 - * TROUBLESHOOTING: Possible Causes of Poor SQL Performance
NOTE:438755.1 - High SQL Version Counts - Script to determine reason(s)
NOTE:2051456.1 - Troubleshooting Databases Hang Due to Heavy Contention for 'library cache: mutex X' Waits (Oracle 11.2 and Later)