Luca Canali on 26 May 2016

Topic: In this post you will find a short discussion and pointers to the code of a few sample scripts that I have written using Linux BPF/bcc and uprobes for Oracle tracing.

Previous work and motivations

Tools for dynamic tracing are very useful for troubleshooting and internals investigations of Oracle workloads. Dynamic tracing probes on the OS/kernel, can be used to measure the details for I/O latency for example. Moreover probes on the Oracle userspace functions can be be used to complement Oracle instrumentation and probe deeper into the internals of the Oracle engine, when needed. For example in a work done in collaboration with Frits Hoogland(link is external) we have investigated how to link Oracle wait event interface with tools able to probe the userspace, such as gdb and dynamic tracing tools as DTrace, SystemTap and Perf. More details and examples on this topic can be found in previous work: Modern Linux Tools for Oracle Troubleshooting (presentation at UKOUG Tech14)Life of an Oracle I/O: Tracing Logical and Physical I/O with SystemTap(link is external)Linux Perf Probes for Oracle Tracing(link is external)Flame Graphs for Oracle(link is external).

What's new with BPF/bcc

BPF together with his frontend bcc(link is external) are new and exciting technologies in the Linux dynamic tracing ecosystem. If you are not familiar with them, you can get up to speed using the excellent material in Brendan Gregg's blog(link is external). While the tools are still in development as I write this, a large amount of important features are already available starting from kernel 4.5, notably the support for uprobes that I have used in the scripts discussed later in this post.

I have started experimenting with porting a few probes for the Oracle userspace that I had written using SystemTap and discussed in this blog. While the language/syntax is completely different between SystemTap and BPF/bcc, porting the probes has turned out to be relatively straightforward. The work has been made substantially easier by the availability of a large selection of well-written and documented scripts in the tools directory of bcc(link is external). At present writing BPF/bcc scripts is a mixture of coding in Python and C, which feels to me both familiar and at the same time strangely low-level compared to the more mature environments for example of DTrace and SystemTap.

The example scripts that I have written using BPF/bcc are on Github(link is external). Here is a list with comments:

Script name

Short description

ora_sqlparse_trace.py(link is external)

Tracing of Oracle SQL parsing. This script traces SQL hard parsing on Oracle binaries hooking on the Oracle function "opiprs" and reads from function arguments (CPU registers) and from process memory.

ora_wait_trace.py(link is external)

Tracing of Oracle wait events. This script traces Oracle sessions by hooking on the functions "kskthewt" and "kews_update_wait_time" and reads from function arguments (CPU registers).

ora_logicalIO_histogram.py(link is external)

Logical IO latency histograms. This script measures the latency between call and return time for the Oracle function "kcbgtcr", which is an important part of the logical IO processing for consistent reads.

ora_wait_histogram.py(link is external)

Wait event latency histograms. This script traces Oracle sessions by hooking on the functions "kskthewt" and "kews_update_wait_time" and reads from function arguments (CPU registers). BPF computes the latency histogram for the wait events and the script prints the values on stdout.

An example of the usage of ora_wait_histogram.py to measure and display wait event latency:

# stdbuf -oL ./ora_wait_histogram.py 10 10|sed -e 's/event# = /event#=/g' -f eventsname.sed

Start tracing oracle wait events... Hit Ctrl-C to end.

event=db file sequential read

wait time, microsec : count     distribution

0 -> 1          : 0        |                                        |

2 -> 3          : 0        |                                        |

4 -> 7          : 0        |                                        |

8 -> 15         : 0        |                                        |

16 -> 31         : 0        |                                        |

32 -> 63         : 0        |                                        |

64 -> 127        : 25       |                                        |

128 -> 255        : 24521    |********************                    |

256 -> 511        : 46788    |****************************************|

512 -> 1023       : 12169    |**********                              |

1024 -> 2047       : 1132     |                                        |

2048 -> 4095       : 660      |                                        |

4096 -> 8191       : 248      |                                        |

8192 -> 16383      : 29       |                                        |

Latency heat maps to display histograms collected with BPF/bcc

PyLatencyMap (link is external)is a command-line tool for visualizing latency histograms using heat maps on terminal graphics, using ANSI escape codes. PyLatencyMap can be used to investigate I/O performance for random I/O, especially suited for the cases of multiple modes of response time from the storage (SSD cache, HDD, latency outliers). The original idea for PyLatencyMap comes from the work of Brendan Gregg(link is external) on latency heat maps. I have added to the PyLatencyMap repository(link is external) a modified version of the biolatency.py(link is external) script to measure I/O latency histograms: this is the link to the script pylatencymap_biolatency.py(link is external) and an example of heat maps generated with PyLatencyMap with the script Example11_BPF-bcc_blockIO_latency.sh(link is external):

The test workload has been generated using Kevin Closson's SLOB(link is external). Additional references with a more detailed discussion of the topic of testing Oracle I/O with SLOB and measuring latency heat maps are:

Heat Map Visualization of I/O Latency with SystemTap and PyLatencyMap(link is external) and OraLatencyMap v1.1 and Testing I/O with SLOB 2(link is external).

Notes on the test environment

The scripts discussed in this post have been developed on Fedora 24 (alpha) running Linux kernel version 4.6 (using the rawhide kernel) and have been tested on workloads generated using Oracle version 11.2.0.4. This is not an Oracle-supported configuration and  the scripts are intended mainly as a demonstration of the technology and for learning purposes.

Here are some pointers on the steps I used to setup a lab environment for testing:

Note in particular the step for configuring the rawhide kernel, probably a good choice when testing BPF, as new features are being added on a regular basis as I write this:
# sudo dnf update
 
Additional pointers and recipes on how to install Oracle 11.2 on Fedora can be found on Tim Hall(link is external)'s website at: https://oracle-base.com/articles/11g/articles-11g(link is external)
 
 

Conclusions

BPF with its bcc frontend are new and powerful tools for dynamic tracing for Linux. A discussion of the merits and shortfalls of BPF/bcc vs other existing solutions in the Linux dynamic tracing ecosystem is beyond the scope of this post. What you can find in this post are a few example scripts that I have written for tracing Oracle using BPF/bcc anduprobes and an additional script for integrating BPF/bcc with PyLatencyMap, which provides visualization as heat maps of the histograms generated using BPF/bcc.

Although the BPF/bcc environment is currently under evolution, it appears already a very useful addition to the toolbox for troubleshooting and performance investigations of Linux workloads. A set of example scripts/tools that come with the bcc repository are of great help for getting started both with using BPF/bcc and with writing new scripts. BPF/bcc can only run on relatively new kernels (as I write this, I have tested the scripts discussed in this post on kernel version 4.6) and this is an obstacle for its adoption in many environments, at least in the short term.

Credits and acknowledgements

Many of the original ideas and tools discussed here are inspired or directly derived from the awesome published work of Brendan Gregg(link is external).

Many thanks also to the development teams of BPF and bcc(link is external) for providing and supporting this new powerful tools. In particular thanks to Brenden Blanco(link is external) for his work on uprobes for bcc and for his support on issue #478(link is external).

The work of investigating Oracle userspace with dynamic tracing tools has been done in collaboration with Frits Hoogland(link is external).

Linux BPF/bcc for Oracle Tracing的更多相关文章

  1. Linux Perf Probes for Oracle Tracing

    Luca Canali on 21 Jan 2016 Topic: this post is about Linux perf and uprobes for tracing and profilin ...

  2. 在Ubuntu和Linux Mint上安装Oracle JDK

    在Ubuntu和Linux Mint上安装Oracle JDK 使用下面的命令安装,只需一些时间,它就会下载许多的文件,所及你要确保你的网络环境良好: sudo add-apt-repository ...

  3. Linux 6.5(oracle 11.2.0.4)单实例ASM安装

    Linux 6.5(oracle 11.2.0.4) 1.解析主机.配置网络等 /etc/hosts /etc/sysconfig/network /etc/init.d/NetworkManager ...

  4. Linux/Unix shell 监控Oracle监听器(monitor listener)

    使用shell脚本实现对Oracle数据库的监控与管理将大大简化DBA的工作负担,如常见的对实例的监控,监听的监控,告警日志的监控,以及数据库的备份,AWR report的自动邮件等.本文给出Linu ...

  5. Linux/Unix shell 监控Oracle告警日志(monitor alter log file)

    使用shell脚本实现对Oracle数据库的监控与管理将大大简化DBA的工作负担,如常见的对实例的监控,监听的监控,告警日志的监控,以及数据库的备份,AWR report的自动邮件等.本文给出Linu ...

  6. Linux服务器下对Oracle作Rman备份

    由于工作需要,最近要对几台Linux系统下的Oracle数据库进行Rman备份,就在操作的同时,整理了一下,方便今后作为资料进行查阅. ------------------------Linux服务器 ...

  7. 转://Linux大内存页Oracle数据库优化

    PC Server发展到今天,在性能方面有着长足的进步.64位的CPU在数年前都已经进入到寻常的家用PC之中,更别说是更高端的PC Server:在Intel和AMD两大处理器巨头的努力下,x86 C ...

  8. Linux下的启动oracle的EM的命令

    Linux下的启动oracle的EM的命令 1.启动数据库 su - oracle $sqlplus / as sysdba sql>startup 2.启动监听 $lsnrctl LSNRCT ...

  9. Linux服务器中创建Oracle数据库实例

    紧接上篇,在Linux服务器已经完成对Oracle数据库软件的安装后,接下来要创建Oracle实例,看图说话: [su – oracle echo $DISPLAY export DISPLAY=10 ...

随机推荐

  1. POJ:2342-Anniversary party(树形dp入门题目)

    传送门:http://poj.org/problem?id=2342 Anniversary party Time Limit: 1000MS Memory Limit: 65536K Descrip ...

  2. CF 219 D:Choosing Capital for Treeland(树形dp)

    D. Choosing Capital for Treeland 链接:http://codeforces.com/problemset/problem/219/D   The country Tre ...

  3. Python之code对象与pyc文件(二)

    上一节:Python之code对象与pyc文件(一) 创建pyc文件的具体过程 前面我们提到,Python在通过import或from xxx import xxx时会对module进行动态加载,如果 ...

  4. Python3下基于bs4和sqlalchemy的爬虫实现

    本文来自网易云社区 作者:王贝 小学生现在都在学python了,作为专业程序员当然不能落下了,所以,快马加鞭,周六周末在家学起了python3,python3的基本语法比较简单,相比于Java开发更加 ...

  5. Wordpress入门笔记

    简单介绍一下wordpress个人操作,建议安装中文版. 登入后台管理者页面, 浏览器地址栏输入           (线上) http://XXXX.com/wp-login.php (本地) ht ...

  6. 【Luogu】P2596书架(Splay)

    题目链接 通过这题我加深了对Splay的理解,原来Splay的子树也是可以接来接去接到别的点上的,而不是只能旋转qwq 具体接的办法就是swap大法. 对于Top操作我们把当前节点Splay到根,然后 ...

  7. 移动端可拖动导航菜单小demo

    <!DOCTYPE html> <html lang="en"> <head> <title>移动端滑动导航菜单</title ...

  8. 《常见问题集》Maven

    1.Maven Eclipse插件要不要安装? [解决方法] 打开你的Eclipse,如果已经有Maven了就不用装插件了. 方法一:没有的话或者下载最新的Eclipse(maven插件,eclips ...

  9. 多态Poly中的向上/下转型 Upcast/Downcast

    [参考资料]马克-to-win java中多态polymorphism,向上转型和动态方法调度有什么用? java中什么叫多态,动态方法调度(dynamic method dispatch)? jav ...

  10. dubbo面试题之dubbo-client 和dubbo-server同时配置超时时间,是以哪个为准;

    根据官网表示: 以 timeout 为例,下图显示了配置的查找顺序,其它 retries, loadbalance, actives 等类似: 方法级优先,接口级次之,全局配置再次之. 如果级别一样, ...