hybrid programming based on python and C/C++

Python/C API Reference Manual¶　　

https://docs.python.org/3/c-api/index.html

Extending and Embedding the Python Interpreter¶

https://docs.python.org/3/extending/

Abstract:

Reference counting:
 1) Stealing a reference means that when you pass a reference to a function, that function assumes that it now owns that reference,you are not responsible for it any longer;
 2) When no ownership is transferred, the caller is said to borrow the reference. Nothing needs to be done for a borrowed reference.
 3) in many cases, the returned object is created on the fly, and the reference you get is the only reference to the object. 
 4) A borrowed reference can be changed into an owned reference by calling Py_INCREF() (create a new owned one)

Details:
Most functions that return a reference to an object pass on ownership with the reference. In particular, all functions whose function it is to create a new object,
such as PyLong_FromLong() and Py_BuildValue(), pass ownership to the receiver. Even if the object is not actually new, you still receive ownership of a new reference to that object.

Many functions that extract objects from other objects also Transfer ownership with the reference, e.g. PyObject_GetAttrString(), which means the extracted obj's ownship transfered
but therr are Eexceptions: PyTuple_GetItem(), PyList_GetItem(), PyDict_GetItem(), and PyDict_GetItemString() all return references that you borrow from the tuple, list or dictionary.

When you pass an object reference into another function, in general, the function borrows the reference from you — if it needs to store it,
it will use Py_INCREF() to become a independent owner. There are exactly two important exceptions to this rule: PyTuple_SetItem() and PyList_SetItem().
These functions take over ownership of the item passed to them — even if they fail!

Extending and Embedding the Python Interpreter
 Extend Python Interpreter with c++ : C dynamic lib called in Python script
 Embedding Python interpreter in C : invoking Python script in C execuatable
 
 1) an C extension for python is not compatible among different implementations of Python, such as CPython ..
 2) you must include Python.h before any standard headers are included.
 3) In order to avoid name clashes with other extension modules, all symbols should be static;
 4) and symbole which should be accessible from other extension modules must be exported in a different way, that is Capsules machinism

Exceptions:
 Exceptions are stored in a static global variable inside the interpreter
 An important convention is : when a function fails, it should set an exception condition and return an error value
 Set an exception including : exception, associated value, traceback
 How to set in extension ? use PyErr_SetXXX or create an error object and then add it to the module manually

Others:
 1) Different Python versions contain different lib packages(or syntax), six AND python-future are two packages to provide compatibility layer
 2) pip introduce the idea of Requirements files, a list of pip install arguments
 3) To support cyclic garbage collection, flag in the type object should be enabled
 4) use "pythonX.Y-config --cflags" will give you the recommended flags when compiling/linking

Types:
 1) Python runtime sees all Python objects as variables of type PyObject* (base type, only contains the refcount and a pointer to the object’s “type obobject")
 2) Tyep object of an Object determine the Object's action(which function get called)

Notes: the follow code was from https://docs.python.org/3/extending/embedding.html

1 embed python sript in C language:

#!/usr/bin/python3
# sample2.pydef func2():
    print ("hello world")

def func3(s):
    print ("hello ", s)

class student:
    def SetName(self, name):
        self._name = name;
    def PrintName(self):
        print(self._name)

# should not run func when this was called by C excutable
# func(

#include <Python.h>
#include <iostream>
#include<string>
using namespace std;

void funct()
{
    // before call python, initialize with following func first
    Py_Initialize();
    PyRun_SimpleString("import sys");
    PyRun_SimpleString("sys.path.append('./')");

    PyObject * pModule = NULL;
    PyObject * pFunc = NULL;
    PyObject * pClass = NULL;
    PyObject * pInstance = NULL;

    pModule = PyImport_ImportModule("sample2");
    if ( pModule == NULL)
        printf("dddddddddddddddddddd");
    pFunc = PyObject_GetAttrString(pModule, "func2");
    PyEval_CallObject(pFunc, NULL);
    Py_DECREF(pFunc);

    pFunc = PyObject_GetAttrString(pModule, "func3");
    PyObject_CallFunction(pFunc, "s", "zhengji");
    Py_DECREF(pFunc);

    pClass = PyObject_GetAttrString(pModule, "student");
    if (!pClass)
    {
        cout << "can't find student instance\n";
        return ;
    }

    pInstance = PyObject_CallObject(pClass, NULL);
    //PyClass
    if (!pInstance)
    {
        cout << "can't create student instance\n";
        return;
    }

    PyObject_CallMethod(pInstance, "SetName", "s", "my family");
    PyObject_CallMethod(pInstance, "PrintName", NULL, NULL);

    Py_Finalize();
}

int main()
{
    funct();
    return 0;
}

build it : g++ embedone.cpp -o embed -I/usr/include/python3.4m -lpython3.4m

2 build c/c++ extension for Python:

// file spammodule1.cpp
#include <Python.h>
/*    target
    >>> import spam
    >>> status = spam.system("ls -l")
 */

#ifdef __cplusplus
extern "C" {
#endif

    // all variable and function should be static
    static PyObject *SpamError;

    // C function to be called
    int function(const char * command)
    {
        return system(command);
    }

    // wrapper for C function
    static PyObject *spam_system(PyObject *self, PyObject *args)
    {
        const char *command;
        int sts;

        if (!PyArg_ParseTuple(args, "s", &command))
            return NULL;
        sts = system(command);
        if (sts < )
        {
            PyErr_SetString(SpamError, "System command failed");
            return NULL;
        }
        return PyLong_FromLong(sts);
        // if return void , use:
        //        Py_INCREF(Py_None);
        //        return Py_None;
    }

    // list its name and address in a method table
    static PyMethodDef SpamMethods[] = {
        { "system", spam_system, METH_VARARGS,
            "Execute a shell command." },
        { NULL, NULL, , NULL }        /* Sentinel */
    };

    // put method talbe in module definition structure
    static struct PyModuleDef spammodule = {
        PyModuleDef_HEAD_INIT,
        "spam",                            /* name of module */
        "spam_doc, which intoduce...",    /* module documentation, may be NULL */
        -,                                /* size of per-interpreter state of the module, or -1 if the module keeps state in global variables. */
        SpamMethods
    };

    // The initialization function must be named in the format PyInit_name()
    // The init function must return the module object to its caller, so that it then gets inserted into sys.modules.
    PyMODINIT_FUNC    PyInit_spam(void)
    {
        PyObject *m;

        m = PyModule_Create(&spammodule);
        if (m == NULL)
        {
            SpamError = PyErr_NewException("spam.error", NULL, NULL);
            Py_INCREF(SpamError);
            PyModule_AddObject(m, "error", SpamError);
            return NULL;
        }
        return m;
    }

#ifdef __cplusplus
}
#endif

/*

// if this module is called in another C code (enbedding Python), use PyImport_AppendInittab() before it's imported
int function(int argc, char *argv[])
{
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL)
    {
        fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
        exit(1);
    }
    // Add a built-in module, before Py_Initialize
    PyImport_AppendInittab("spam", PyInit_spam);
    // Pass argv[0] to the Python interpreter ,inform the interpreter about paths to Python run-time libraries
    Py_SetProgramName(program);
    // Initialize the Python interpreter.  Required.
    Py_Initialize();
    // Optionally import the module; alternatively, import can be deferred until the embedded script imports it.
    PyImport_ImportModule("spam");
    // program was mallocated by Python, so free it with Py_free
    PyMem_RawFree(program);
    return 0;
}

*/
//    http://www.swig.org/papers/PyTutorial98/PyTutorial98.pdf

build it : g++ spammodule1.cpp -o spam.so -shared -fpic -I/usr/include/python3.4m

or create file setup.py and run : python3 setup.py build

#!/usr/bin/python3

from distutils.core import setup, Extension

# Extention is a collection of attributes/every thing that needed to build a module
# init(self, name,sources, include_dirs=NONE, libraries, arguments, macros...)
module1 = Extension('spam',sources=['spammodule1.cpp'])

# https://docs.python.org/3.4/distutils/apiref.html?highlight=setup#distutils.core.setup
# name        : package name
# packages    : A list of Python packages that distutils will manipulate, contain .py files
# py_modules  : A list of Python modules that distutils will manipulate, usually .py files
# ext_modules : a list of instances of distutils.core.Extension to be built (module)
setup (name = 'spam1',version = '1.0', description = 'This is a demo package',ext_modules = [module1])

#!/usr/bin/python3
# test.py
import spam
spam.system("ls -a")

3 extended embedding python

// file : pythonembedding.cpp
#include <Python.h>
#include <iostream>
using namespace std;

/*
    code are copied from https://docs.python.org/3/extending/embedding.html
    extend Embedded Python, which means
    python extension: accept python objects, funcs are called in python script
    embeded python  : call python function with c objects
*/
static int numargs = ;

/* Return the number of arguments of the application command line */
static PyObject* emb_numargs(PyObject *self, PyObject *args)
{
    if (!PyArg_ParseTuple(args, ":numargs"))
        return NULL;
    return PyLong_FromLong(numargs);
}

static PyMethodDef EmbMethods[] = {
    { "numargs", emb_numargs, METH_VARARGS,
    "Return the number of arguments received by the process." },
    { NULL, NULL, , NULL }
};

static PyModuleDef EmbModule = {
    PyModuleDef_HEAD_INIT, "emb", NULL, -, EmbMethods,
    NULL, NULL, NULL, NULL
};

static PyObject* PyInit_emb(void)
{
    return PyModule_Create(&EmbModule);
}

int  main(int argc, char *argv[])
{
    PyObject *pName, *pModule, *pDict, *pFunc;
    PyObject *pArgs, *pValue;
    int i;

    if (argc < )
    {
        fprintf(stderr, "Usage: call pythonfile funcname [args]\n");
        fprintf(stderr, "examp: ./xx sample getname dd\n");
        return ;
    }

    numargs = argc;
    PyImport_AppendInittab("emb", &PyInit_emb);

    Py_Initialize();

    // add current path so it can import module in current directory
    PyRun_SimpleString("import sys");
    PyRun_SimpleString("sys.path.append('./')");

    pName = PyUnicode_DecodeFSDefault(argv[]);
    /* Error checking of pName left out */

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);

    if (pModule != NULL)
    {
        pFunc = PyObject_GetAttrString(pModule, argv[]);
        /* pFunc is a new reference */

        if (pFunc && PyCallable_Check(pFunc))
        {
            pArgs = PyTuple_New(argc - );
            for (i = ; i < argc - ; ++i)
            {
                pValue = PyLong_FromLong(atoi(argv[i + ]));
                if (!pValue)
                {
                    Py_DECREF(pArgs);
                    Py_DECREF(pModule);
                    fprintf(stderr, "Cannot convert argument\n");
                    return ;
                }
                /* pValue reference stolen here: */
                PyTuple_SetItem(pArgs, i, pValue);
            }
            pValue = PyObject_CallObject(pFunc, pArgs);
            Py_DECREF(pArgs);
            if (pValue != NULL)
            {
                printf("Result of call: %ld\n", PyLong_AsLong(pValue));
                Py_DECREF(pValue);
            }
            else
            {
                Py_DECREF(pFunc);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr, "Call failed\n");
                return ;
            }
        }
        else
        {
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function \"%s\"\n", argv[]);
        }
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    }
    else
    {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", argv[]);
        return ;
    }
    // included in version 3.6
    // if (Py_FinalizeEx() < 0) { return 120; }
    Py_Finalize();
    return ;
}

#!/usr/bin/python3
# sample.py
import emb
print("number of arguments", emb.numargs())
def func(x):
    print(" the imput arguments is ", x )

sample.py invokes module "emb" defined in pythonembedding.cpp, meanwhile this python file (module) will be invoked in pythonembedding

build it : g++ pythonembedding.cpp  -o pythonembedding -I/usr/include/python3.4m/ -lpython3.4m 

run : ./pythonembedding