Exploring Python Code Objects

Exploring Python Code Objects

https://late.am/post/2012/03/26/exploring-python-code-objects.html

Inspired by David Beazley's Keynote at PyCon, I've been digging around in code objects in Python lately. I don't have a particular axe to grind, nor some particular task to solve (yet?), so consider this post just some notes and ramblings that might be of interest (and my apologies if not).

Disclaimer: This post is about CPython version 2.7, though much of it is also likely true for other CPython versions (including 3.x). I make no claims to its accuracy or applicability to PyPy, Jython, IronPython, etc.

Step 0: What?

So first of all, what is a code object? Many people (particularly Python haters) claim that Python is an interpreted language, but all your Python code is actually compiled before it is ever executed. This goes even for code you write interactively in the Python shell. CPython implements a virtual machine that executes a stack-based bytecode. At runtime, executable things (functions, methods, modules, class bodies, lambdas, statements, expressions, etc) are all executed as bytecode by the Python virtual machine.

Code objects, then, are Python objects which represent some piece of bytecode, along with all that it needs to execute: a declaration of the expected argument types and counts, a list (not dictionary! more about which later) of locals, information about the source code from which the bytecode was generated (for debugging and printing stack traces), etc -- oh, and also (perhaps obviously), the bytecode itself, as a str (or, in Python3, bytes).

Though code objects represent some piece of executable code, they are not, by themselves, directly callable. To execute a code object, you must use the exec keyword or eval() function.

Step 1: Make some Code

Most of the time, you won't encounter code objects in ordinary Python programming. When you do, there's a very good chance that they were created and are managed for you by Python, without any special attention. In some cases, you might want to create code objects yourself, like in this post where we'll be experimenting with them:

>>> code_str = """
... print "Hello, world"
... """
>>> code_obj = compile(code_str, '<string>', 'exec')
>>> code_obj
<code object <module> at 0x1054c74b0, file "<string>", line 2>

Woohoo, your first code object!

The first argument to compile() is the string of Python code to be compiled, which should be obvious. The second defines the "filename" of the piece of code (here, as is conventional, we use <string> to indicate code attained from the interactive shell). The third is the type of compilation, which most often will be exec as you see here. The other choices for mode are eval, which is used for strings containing only a single expression, orsingle, in which the generated code object is expected to contain a single statement, whose return value is printed if it is not None (like in the interactive shell).

When using eval mode, if the code contains statements (as our example above does, it contains a printstatement), compilation will fail with a syntax error:

>>> code_str = """
... print "Hello, world"
... """
>>> code_obj = compile(code_str, '<string>', 'eval')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 2
    print "Hello, world"
        ^
SyntaxError: invalid syntax

When using single, only a single statement is processed; multiple statements (or non-statements) will be ignored:

>>> code_str = """
... print "Hello, world"
... print "Goodbye, world"
... """
>>> code_obj = compile(code_str, '<string>', 'single')
>>> exec code_obj
Hello, world

What happened to my "goodbye"?

For the rest of the post, we'll stick with exec, which is the type of compilation Python does for you when importing modules.

Step 2: Open 'er Up

Let's go back to our first example, and have a look inside the code object to see what we have:

>>> code_str = """
... print "Hello, world"
... """
>>> code_obj = compile(code_str, '<string>', 'exec')
>>> dir(code_obj)
# dunder attributes excluded for readability
['co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename',
 'co_firstlineno', 'co_flags', 'co_freevars', 'co_lnotab', 'co_name',
 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames']

These attributes are documented in the inspect module, but I'll highlight a few cool ones here:

First, we can see where our second argument to compile() ended up:

>>> code_obj.co_filename
'<string>'

And, perhaps surprisingly, our code represents an anonymous module (code compiled with exec mode is always treated as module-level code, though, of course, it can contain function or class definitions, or any other valid Python):

>>> code_obj.co_name
'<module>'

And, as we expect, a code object representing a Python module (that's effectively what our code string was -- a series of statements at the top-most level, that is, not indented at all) takes no arguments:

>>> code_obj.co_argcount
0
>>> code_obj.co_varnames
()

If we were to take a piece of code from a function which does have arguments, we'd see them here:

>>> def foo(x, y):
...     print x, y
...
>>> foo.func_code
<code object foo at 0x1054b9830, file "<stdin>", line 1>
>>> foo.func_code.co_varnames
('x', 'y')
>>> foo.func_code.co_argcount
2

If you're curious, you can also see the raw bytecode that will be processed by the Python virtual machine:

>>> code_obj.co_code
'd\x00\x00GHd\x01\x00S'

I don't recommend trying to learn to read that directly, there's an easier way (hint: see the next section).

Finally, we have one constant object within scope, the string "Hello, world", which is printed by our code:

>>> code_obj.co_consts
('Hello, world', None)

Wait. Where's that None coming from?

A Detour into Code Disassembly

We can see exactly what's going on in our code object with the dis module, which disassembles code objects into a human readable series of bytecode instructions:

>>> dis.dis(code_obj)
  2           0 LOAD_CONST           0 ('Hello, world')
              3 PRINT_ITEM
              4 PRINT_NEWLINE
              5 LOAD_CONST           1 (None)
              8 RETURN_VALUE

Reading disassembled Python code requires a bit of experience, so let me walk you through it. The LOAD_CONSTinstruction reads a value from the co_consts tuple, and pushes it onto the top of the stack. The PRINT_ITEMinstruction pops the top of the stack, and prints the string representation. PRINT_NEWLINE should be pretty self-explanatory.

Next we see the mysterious None. It turns out this is a bit of a quirk of the implementation details of the CPython virtual machine. Since function calls in Python (including "hidden" function calls, like those behind animport statement) are implemented with function calls in C in the Python virtual machine, modules actually have a return value -- this indicates to the Python virtual machine that execution of the module has completed, and control can be returned to the calling scope (i.e. the module in which the import statement appeared). I won't embarrass myself by trying to explain this further -- if you are interested, see Larry Hastingss PyCon presentation Stepping through CPython around 44:22 -- that video covers Python 3.x, but Python 2.7 does the same thing. If you're interested in this sort of implementation detail, then you should definitely watch the entirety of this video, and David Beazley's keynote as well.

Step 3: Interesting Internals

Many of the features we've looked at are clearly useful for a running Python virtual machine, but what about the human side of the story? What if we want to interactively debug code (using pdb or a similar tool), or get helpful, readable tracebacks from exceptions?

It turns out, code objects support this as well. As we've already seen, code objects indicate from which file they were generated, and this will obviously help in looking up source code; they also indicate the line number on which the source code for this code object begins:

>>> code_obj.co_firstlineno
2

And the mysterious co_lnotab attribute. To illustrate its purpose, we'll need a larger code snippet:

>>> code_str = """
... x = 1
... y = 2
... print x + y
... """
>>> code_obj = compile(code_str, '<string>', 'exec')
>>> code_obj.co_lnotab
'\x06\x01\x06\x01'

Hm, so what are we to make of this? Perhaps the dis module can help here again:

>>> dis.dis(code_obj)
  2           0 LOAD_CONST           0 (1)
              3 STORE_NAME           0 (x)

  3           6 LOAD_CONST           1 (2)
              9 STORE_NAME           1 (y)

  4          12 LOAD_NAME            0 (x)
             15 LOAD_NAME            1 (y)
             18 BINARY_ADD
             19 PRINT_ITEM
             20 PRINT_NEWLINE
             21 LOAD_CONST           2 (None)
             24 RETURN_VALUE

At the far left of (some) lines, is the line number of the Python source from which this code object was created (notice that 2 here corresponds to the value of code_obj.co_firstlineno). The next column is the offset into the code of the bytecode instruction, 0 bytes for the first instruction, 3 bytes for the second, and so on. The third column is the instruction name itself, and the fourth is the argument to the instruction, if any, along with the value of the argument in parentheses.

Now we can put this together with the co_lnotab (which stands for "line number table", by the way) to see how Python makes sense of the code objects' relation to their original source code:

>>> code_obj.co_lnotab
'\x06\x01\x06\x01'

After a little tinkering and trial and error, I realized that this is a series of pairs of bytes: the first is a length offset into the bytecode (6 bytes, which advances us to the second LOAD_CONST as seen in our disassembly), followed by a number of source lines of code that the skipped instructions appeared on.

We can confirm this theory by slightly modifying our source code, recompiling, and examining the co_lnotabattribute of the resulting code object:

>>> code_str2 = """
... x = 1
...
... y = 2
... print x + y
... """
>>> code_obj2 = compile(code_str2, '<string>', 'exec')
>>> code_obj2.co_lnotab
'\x06\x02\x06\x01'

We've moved the second assignment down one line, so we see that in the second byte of co_lnotab, we are incrementing the "current line number" by two instead of by one.

We can also verify that the bytecode resulting from these two (slightly) different source codes is identical:

>>> code_obj2.co_code == code_obj.co_code
True

Since both the bytecode offset and line number offset at single (unsigned) bytes, one might wonder what happens if you have, say, 257 (or more) blank lines between statements in a Python source file? Let's see:

>>> thousand_blanks = '\n' * 1000
>>> code_str = """
... x = 1
... """ + thousand_blanks + """
... y = 2
... print x + y
... """
>>> code_obj = compile(code_str, '<string>', 'exec')
>>> code_obj.co_lnotab
'\x06\xff\x00\xff\x00\xff\x00\xec\x06\x01'

Since both the bytecode offsets and line offsets are, well, offsets, having large empty spaces just means that some of the interleaved offsets are 0-length offsets. Here we have a 6-byte offset into bytecode, followed by a 255-line offset into the source code, then a 0-byte offset into the bytecode, another 255 lines of source, another 0 bytes into the bytecode, yet another 255 lines of source, one more 0-byte offset into bytecode, and a final 236 lines of offset into the source code (then the usual, expected 6 bytes of bytecode and 1 line of source code for the final print statement). Neat!