DTrace patch for Python 2.7.x and 3.x

DTrace patch for Python 2.7.x and 3.x

Última Actualización: 21 de septiembre de 2015

https://www.jcea.es/artic/python_dtrace.htm

You can follow this work in the Python bugtracker on issue 13405 (original work on issue 4111).

How to get the patch

You can clone my mercurial repository. Interesting branches are dtrace-issue13405_2.7, dtrace-issue13405_3.4, dtrace-issue13405_3.5 and dtrace-issue13405 (future 3.6). This configuration is recommended if you plan to contribute.

If you just want to compile your Python interpreter with dtrace patch applied, you only need to apply the matching patch over a pristine copy of the canonical Python source tree:

• Python 3.5.1: 209b219a8b5b
  • Python 3.5.0: a7da156226da
  • Python 3.5.0b2: 20ac4207134f
  • Python 3.5.0a1: 6a85b8f9fe18
• Python 3.4.4: c450f9cba38a
  • Python 3.4.3: d406e844623e
  • Python 3.4.2: 55be19e94859
  • Python 3.4.1: a1b9322b6d8c
  • Python 3.4.0: 0de6441eedb7
  • Python 3.4.0rc1: 1c6ba78665ce
  • Python 3.4.0b1: 654264da62f2
• Python 2.7.11: 8c5948409bbe (fully compatible with computed gotos)
  • Python 2.7.11: 733833f7789d (You must configure with --without-computed-gotos)
  • Python 2.7.10: b5158e9b7e17
  • Python 2.7.9: 05d8fd4c57a1
  • Python 2.7.8: 0b37c3647b82
  • Python 2.7.7: af8ecf2352e1
  • Python 2.7.6: 0b375fd8628e
  • Python 2.7.5: f96ea83cd766
  • Python 2.7.3: 1a89922027e1
• Python 3.3.5: b8da4a55237d

  • Python 3.3.4: 1c7158230a62
  • Python 3.3.3: 93eb37b54d6f
  • Python 3.3.2: 23dafaf73d29
  • Python 3.3.0: 3f5aab499e15

How to compile and test

Do "autoconf".

Add "--with-dtrace" to your "configure" command line and compile as usual. If you have previously compiled Python in the same directory, remember to do "make distclean" first.

When done, you should run the complete testsuite. If you just want to run dtrace tests, do

# LD_LIBRARY_PATH=`pwd` ./python Lib/test/regrtest.py -v test_dtrace.py

You should get something like

[1/1] test_dtrace
test_function_entry_return (test.test_dtrace.DTraceTestsNormal) ... ok
test_garbage_collection (test.test_dtrace.DTraceTestsNormal) ... ok
test_instance_creation_destruction (test.test_dtrace.DTraceTestsNormal) ... ok
test_line (test.test_dtrace.DTraceTestsNormal) ... ok
test_stack (test.test_dtrace.DTraceTestsNormal) ... ok
test_unicode_function_entry_return (test.test_dtrace.DTraceTestsNormal) ... ok
test_unicode_stack (test.test_dtrace.DTraceTestsNormal) ... ok
test_verify_opcodes (test.test_dtrace.DTraceTestsNormal) ... ok

----------------------------------------------------------------------
Ran 8 tests in 5.335s

OK
test_function_entry_return (test.test_dtrace.DTraceTestsOptimize) ... ok
test_garbage_collection (test.test_dtrace.DTraceTestsOptimize) ... ok
test_instance_creation_destruction (test.test_dtrace.DTraceTestsOptimize) ... ok
test_line (test.test_dtrace.DTraceTestsOptimize) ... ok
test_stack (test.test_dtrace.DTraceTestsOptimize) ... ok
test_unicode_function_entry_return (test.test_dtrace.DTraceTestsOptimize) ... ok
test_unicode_stack (test.test_dtrace.DTraceTestsOptimize) ... ok
test_verify_opcodes (test.test_dtrace.DTraceTestsOptimize) ... ok

----------------------------------------------------------------------
Ran 8 tests in 5.455s

OK
1 test OK.

The first set of tests run under non-optimizing Python, while the second set run under optimizing Python (-O).

Documentation

:mod:`dtrace` --- DTrace probes for Python
===============================================

.. module:: dtrace
   :synopsis: DTrace probes for Python.

**Source code:** :source:`Lib/dtrace.py`

--------------

The :mod:`dtrace` module indicates if the CPython executable currently
running has been compiled with DTrace probes support.

.. impl-detail::

   DTrace probes are implementation details of the CPython interpreter!
   No garantees are made about probe compatibility between versions of
   CPython. DTrace scripts can stop working or work incorrectly without
   warning when changing CPython versions.

The :mod:`dtrace` module defines the following variable:

.. data:: available

   The variable will be ``True`` if the current CPython interpreter was
   compiled with DTrace probe support. ``False`` if not.

DTrace probes
-------------

DTrace scripts are run externally to CPython. DTrace probes export
selected events inside CPython interpreter in order to make them
accessible to external scripts.

The probes are exported through the "python" provider. The available
probes are defined in the file :file:`Include/pydtrace.d`.

To learn how to use DTrace, read `DTrace User Guide
`_.

.. opcode:: function-entry (arg0, arg1, arg2)

   Fires when python code enters a new function. *arg0* is sourcecode
   file path, *arg1* is the name of the funcion called, and *arg2* is
   line number.

   The probe is not fired if Python code calls C functions.

.. opcode:: function-return (arg0, arg1, arg2)

   Fires when Python code finishes execution of a function. Parameters
   are the same as in ``function-entry``.

   The probe is not fired if the finishing function is written in C.

.. opcode:: line (arg0, arg1, arg2)

   Fires when Python code changes the execution line. Parameters are the
   same as in ``function-entry``.

   The probe is not fired in C functions.

.. opcode:: gc-start (arg0)

   Fires when the Python interpreter starts a garbage collection cycle.
   *arg0* is the generation to scan, like :func:`gc.collect()`.

.. opcode:: gc-done (arg0)

   Fires when the Python interpreter finishes a garbage collection
   cycle. *arg0* is the number of collected objects.

.. opcode:: instance-new-start (arg0, arg1)

   Fires when an object instanciation starts. *arg0* is the class name,
   *arg1* is the filename where the class is defined.

   The probe is not fired for most C code object creations.

.. opcode:: instance-new-done (arg0, arg1)

   Fires when an object instanciation finishes. Parameters are the same
   as in ``instance-new-done``.

   The probe is not fired for most C code object creations.

.. opcode:: instance-delete-start (arg0, arg1)

   Fires when an object instance is going to be destroyed. Parameters
   are the same as in ``instance-new-done``.

   The probe is not fired for most C code object destructions.

.. opcode:: instance-delete-done (arg0, arg1)

   Fires when an object instance has been destroyed. parameters are the
   same as in ``instance-new-done``.

   Between an ``instance-delete-start`` and corresponding
   ``instance-delete-done`` others probes can fire if, for instance,
   deletion of an instance creates a deletion cascade.

   The probe is not fired for most C code object destructions.

Python stack
------------

When a DTrace probe is fired, the DTrace script can examine the stack.
Since CPython is a Python interpreter coded in C, the stack will show C
functions, with no direct relation to the Python code currently being
executed.

Using the special "jstack()" DTrace function, the user will be given
hints about the python program stack, if possible. In particular, the
augmented stack will show python function calls, filename, name
of the function or method, and the line number.

DTrace scripts examples
-----------------------

DTrace python provider is suffixed by the pid of the process to monitor.
In the examples, the pid will be 9876.

Show the time spent doing garbage collection (in nanoseconds)::

  python9876:::gc-start
  {
      self->t = timestamp;
  }

  python9876:::gc-done
  /self->t/
  {
      printf("%d", timestamp-self->t);
      self->t = 0;
  }

Count how many instances are created of each class::

  python9876:::instance-new-start
  {
      @v[copyinstr(arg1), copyinstr(arg0)] = count();
  }

Observe time spent in object destruction, useful if datastructures are
complicated and deletion of an object can create a cascade effect::

  python9876:::instance-delete-start
  /self->t==0/
  {
      self->t = timestamp;
      self->level = 0;
  }

  python9876:::instance-delete-start
  /self->t/
  {
      self->level += 1;
  }

  python9876:::instance-delete-done
  /(self->level) && (self->t)/
  {
      self->level -= 1;
  }

  python9876:::instance-delete-done
  /(self->level==0) && (self->t)/
  {
      @time = quantize(timestamp-self->t);
      self->t = 0;
  }

To know which python source code lines create new TCP/IP connections::

  pid9876::sock_connect:entry
  {
      @conn[jstack()] = count();
  }