path: root/sphinx/using.rst

.. _ref-using:

Using SIP
=========

Bindings are generated by the SIP code generator from a number of specification
files, typically with a ``.sip`` extension.  Specification files look very
similar to C and C++ header files, but often with additional information (in
the form of a *directive* or an *annotation*) and code so that the bindings
generated can be finely tuned.


.. _ref-simple-c++-example:

A Simple C++ Example
--------------------

We start with a simple example.  Let's say you have a (fictional) C++ library
that implements a single class called ``Word``.  The class has one constructor
that takes a ``\0`` terminated character string as its single argument.  The
class has one method called ``reverse()`` which takes no arguments and returns
a ``\0`` terminated character string.  The interface to the class is defined in
a header file called ``word.h`` which might look something like this::

    // Define the interface to the word library.

    class Word {
        const char *the_word;

    public:
        Word(const char *w);

        char *reverse() const;
    };

The corresponding SIP specification file would then look something like this::

    // Define the SIP wrapper to the word library.

    %Module word 0

    class Word {

    %TypeHeaderCode
    #include <word.h>
    %End

    public:
        Word(const char *w);

        char *reverse() const;
    };

Obviously a SIP specification file looks very much like a C++ (or C) header
file, but SIP does not include a full C++ parser.  Let's look at the
differences between the two files.

    - The :directive:`%Module` directive has been added [#]_.  This is used to
      name the Python module that is being created and to give it a
      *generation* number.  In this example these are ``word`` and ``0``
      respectively.  The generation number is effectively the version number of
      the module.

    - The :directive:`%TypeHeaderCode` directive has been added.  The text
      between this and the following :directive:`%End` directive is included
      literally in the code that SIP generates.  Normally it is used, as in
      this case, to ``#include`` the corresponding C++ (or C) header file [#]_.

    - The declaration of the private variable ``this_word`` has been removed.
      SIP does not support access to either private or protected instance
      variables.

If we want to we can now generate the C++ code in the current directory by
running the following command::

    sip -c . word.sip

However, that still leaves us with the task of compiling the generated code and
linking it against all the necessary libraries.  It's much easier to use the
:ref:`SIP build system <ref-build-system>` to do the whole thing.

Using the SIP build system is simply a matter of writing a small Python script.
In this simple example we will assume that the ``word`` library we are wrapping
and it's header file are installed in standard system locations and will be
found by the compiler and linker without having to specify any additional
flags.  In a more realistic example your Python script may take command line
options, or search a set of directories to deal with different configurations
and installations.

This is the simplest script (conventionally called ``configure.py``)::

    import os
    import sipconfig

    # The name of the SIP build file generated by SIP and used by the build
    # system.
    build_file = "word.sbf"

    # Get the SIP configuration information.
    config = sipconfig.Configuration()

    # Run SIP to generate the code.
    os.system(" ".join([config.sip_bin, "-c", ".", "-b", build_file, "word.sip"]))
    
    # Create the Makefile.
    makefile = sipconfig.SIPModuleMakefile(config, build_file)

    # Add the library we are wrapping.  The name doesn't include any platform
    # specific prefixes or extensions (e.g. the "lib" prefix on UNIX, or the
    # ".dll" extension on Windows).
    makefile.extra_libs = ["word"]

    # Generate the Makefile itself.
    makefile.generate()

Hopefully this script is self-documenting.  The key parts are the
``Configuration`` and ``SIPModuleMakefile`` classes.  The build system contains
other Makefile classes, for example to build programs or to call other
Makefiles in sub-directories.

After running the script (using the Python interpreter the extension module is
being created for) the generated C++ code and ``Makefile`` will be in the
current directory.

To compile and install the extension module, just run the following
commands [#]_::

    make
    make install

That's all there is to it.

See :ref:`ref-distutils` for an example of how to build this example using
distutils.

.. [#] All SIP directives start with a ``%`` as the first non-whitespace
       character of a line.
.. [#] SIP includes many code directives like this.  They differ in where the
       supplied code is placed by SIP in the generated code.
.. [#] On Windows you might run ``nmake`` or ``mingw32-make`` instead.


A Simple C Example
------------------

Let's now look at a very similar example of wrapping a fictional C library::

    /* Define the interface to the word library. */

    struct Word {
        const char *the_word;
    };

    struct Word *create_word(const char *w);
    char *reverse(struct Word *word);

The corresponding SIP specification file would then look something like this::

    /* Define the SIP wrapper to the word library. */

    %CModule word 0

    struct Word {

    %TypeHeaderCode
    #include <word.h>
    %End

        const char *the_word;
    };

    struct Word *create_word(const char *w) /Factory/;
    char *reverse(struct Word *word);

Again, let's look at the differences between the two files.

    - The :directive:`%CModule` directive has been added.  This has the same
      syntax as the :directive:`%Module` directive used in the previous example
      but tells SIP that the library being wrapped is implemented in C rather
      than C++.

    - The :directive:`%TypeHeaderCode` directive has been added.

    - The :fanno:`Factory` annotation has been added to the ``create_word()``
      function.  This tells SIP that a newly created structure is being
      returned and it is owned by Python.

The ``configure.py`` build system script described in the previous example can
be used for this example without change.


A More Complex C++ Example
--------------------------

In this last example we will wrap a fictional C++ library that contains a class
that is derived from a Qt class.  This will demonstrate how SIP allows a class
hierarchy to be split across multiple Python extension modules, and will
introduce SIP's versioning system.

The library contains a single C++ class called ``Hello`` which is derived from
Qt's ``QLabel`` class.  It behaves just like ``QLabel`` except that the text
in the label is hard coded to be ``Hello World``.  To make the example more
interesting we'll also say that the library only supports Qt v4.2 and later,
and also includes a function called ``setDefault()`` that is not implemented
in the Windows version of the library.

The ``hello.h`` header file looks something like this::

    // Define the interface to the hello library.

    #include <qlabel.h>
    #include <qwidget.h>
    #include <qstring.h>

    class Hello : public QLabel {
        // This is needed by the Qt Meta-Object Compiler.
        Q_OBJECT

    public:
        Hello(QWidget *parent = 0);

    private:
        // Prevent instances from being copied.
        Hello(const Hello &);
        Hello &operator=(const Hello &);
    };

    #if !defined(Q_OS_WIN)
    void setDefault(const QString &def);
    #endif

The corresponding SIP specification file would then look something like this::

    // Define the SIP wrapper to the hello library.

    %Module hello 0

    %Import QtGui/QtGuimod.sip

    %If (Qt_4_2_0 -)

    class Hello : QLabel {

    %TypeHeaderCode
    #include <hello.h>
    %End

    public:
        Hello(QWidget *parent /TransferThis/ = 0);

    private:
        Hello(const Hello &);
    };

    %If (!WS_WIN)
    void setDefault(const QString &def);
    %End

    %End

Again we look at the differences, but we'll skip those that we've looked at in
previous examples.

    - The :directive:`%Import` directive has been added to specify that we are
      extending the class hierarchy defined in the file ``QtGui/QtGuimod.sip``.
      This file is part of PyQt.  The build system will take care of finding
      the file's exact location.

    - The :directive:`%If` directive has been added to specify that everything
      [#]_ up to the matching :directive:`%End` directive only applies to Qt
      v4.2 and later.  ``Qt_4_2_0`` is a *tag* defined in ``QtCoremod.sip``
      [#]_ using the :directive:`%Timeline` directive.  :directive:`%Timeline`
      is used to define a tag for each version of a library's API you are
      wrapping allowing you to maintain all the different versions in a single
      SIP specification.  The build system provides support to ``configure.py``
      scripts for working out the correct tags to use according to which
      version of the library is actually installed.

    - The ``public`` keyword used in defining the super-classes has been
      removed.  This is not supported by SIP.

    - The :aanno:`TransferThis` annotation has been added to the constructor's
      argument.  It specifies that if the argument is not 0 (i.e. the ``Hello``
      instance being constructed has a parent) then ownership of the instance
      is transferred from Python to C++.  It is needed because Qt maintains
      objects (i.e. instances derived from the ``QObject`` class) in a
      hierachy.  When an object is destroyed all of its children are also
      automatically destroyed.  It is important, therefore, that the Python
      garbage collector doesn't also try and destroy them.  This is covered in
      more detail in :ref:`ref-object-ownership`.  SIP provides many other
      annotations that can be applied to arguments, functions and classes.
      Multiple annotations are separated by commas.  Annotations may have
      values.

    - The ``=`` operator has been removed.  This operator is not supported by
      SIP.

    - The :directive:`%If` directive has been added to specify that everything
      up to the matching :directive:`%End` directive does not apply to Windows.
      ``WS_WIN`` is another tag defined by PyQt, this time using the
      :directive:`%Platforms` directive.  Tags defined by the
      :directive:`%Platforms` directive are mutually exclusive, i.e. only one
      may be valid at a time [#]_.

One question you might have at this point is why bother to define the private
copy constructor when it can never be called from Python?  The answer is to
prevent the automatic generation of a public copy constructor.

We now look at the ``configure.py`` script.  This is a little different to the
script in the previous examples for two related reasons.

Firstly, PyQt includes a pure Python module called ``pyqtconfig`` that extends
the SIP build system for modules, like our example, that build on top of PyQt.
It deals with the details of which version of Qt is being used (i.e. it
determines what the correct tags are) and where it is installed.  This is
called a module's configuration module.

Secondly, we generate a configuration module (called ``helloconfig``) for our
own ``hello`` module.  There is no need to do this, but if there is a chance
that somebody else might want to extend your C++ library then it would make
life easier for them.

Now we have two scripts.  First the ``configure.py`` script::

    import os
    import sipconfig
    from PyQt4 import pyqtconfig

    # The name of the SIP build file generated by SIP and used by the build
    # system.
    build_file = "hello.sbf"

    # Get the PyQt configuration information.
    config = pyqtconfig.Configuration()

    # Get the extra SIP flags needed by the imported PyQt modules.  Note that
    # this normally only includes those flags (-x and -t) that relate to SIP's
    # versioning system.
    pyqt_sip_flags = config.pyqt_sip_flags

    # Run SIP to generate the code.  Note that we tell SIP where to find the qt
    # module's specification files using the -I flag.
    os.system(" ".join([config.sip_bin, "-c", ".", "-b", build_file, "-I", config.pyqt_sip_dir, pyqt_sip_flags, "hello.sip"]))

    # We are going to install the SIP specification file for this module and
    # its configuration module.
    installs = []

    installs.append(["hello.sip", os.path.join(config.default_sip_dir, "hello")])

    installs.append(["helloconfig.py", config.default_mod_dir])

    # Create the Makefile.  The QtGuiModuleMakefile class provided by the
    # pyqtconfig module takes care of all the extra preprocessor, compiler and
    # linker flags needed by the Qt library.
    makefile = pyqtconfig.QtGuiModuleMakefile(
        configuration=config,
        build_file=build_file,
        installs=installs
    )

    # Add the library we are wrapping.  The name doesn't include any platform
    # specific prefixes or extensions (e.g. the "lib" prefix on UNIX, or the
    # ".dll" extension on Windows).
    makefile.extra_libs = ["hello"]

    # Generate the Makefile itself.
    makefile.generate()

    # Now we create the configuration module.  This is done by merging a Python
    # dictionary (whose values are normally determined dynamically) with a
    # (static) template.
    content = {
        # Publish where the SIP specifications for this module will be
        # installed.
        "hello_sip_dir":    config.default_sip_dir,

        # Publish the set of SIP flags needed by this module.  As these are the
        # same flags needed by the qt module we could leave it out, but this
        # allows us to change the flags at a later date without breaking
        # scripts that import the configuration module.
        "hello_sip_flags":  pyqt_sip_flags
    }

    # This creates the helloconfig.py module from the helloconfig.py.in
    # template and the dictionary.
    sipconfig.create_config_module("helloconfig.py", "helloconfig.py.in", content)

Next we have the ``helloconfig.py.in`` template script::

    from PyQt4 import pyqtconfig

    # These are installation specific values created when Hello was configured.
    # The following line will be replaced when this template is used to create
    # the final configuration module.
    # @SIP_CONFIGURATION@

    class Configuration(pyqtconfig.Configuration):
        """The class that represents Hello configuration values.
        """
        def __init__(self, sub_cfg=None):
            """Initialise an instance of the class.

            sub_cfg is the list of sub-class configurations.  It should be None
            when called normally.
            """
            # This is all standard code to be copied verbatim except for the
            # name of the module containing the super-class.
            if sub_cfg:
                cfg = sub_cfg
            else:
                cfg = []

            cfg.append(_pkg_config)

            pyqtconfig.Configuration.__init__(self, cfg)

    class HelloModuleMakefile(pyqtconfig.QtGuiModuleMakefile):
        """The Makefile class for modules that %Import hello.
        """
        def finalise(self):
            """Finalise the macros.
            """
            # Make sure our C++ library is linked.
            self.extra_libs.append("hello")

            # Let the super-class do what it needs to.
            pyqtconfig.QtGuiModuleMakefile.finalise(self)

Again, we hope that the scripts are self documenting.

.. [#] Some parts of a SIP specification aren't subject to version control.
.. [#] Actually in ``versions.sip``.  PyQt uses the :directive:`%Include`
       directive to split the SIP specification for Qt across a large number of
       separate ``.sip`` files.
.. [#] Tags can also be defined by the :directive:`%Feature` directive.  These
       tags are not mutually exclusive, i.e. any number may be valid at a time.


.. _ref-object-ownership:

Ownership of Objects
--------------------

When a C++ instance is wrapped a corresponding Python object is created.  The
Python object behaves as you would expect in regard to garbage collection - it
is garbage collected when its reference count reaches zero.  What then happens
to the corresponding C++ instance?  The obvious answer might be that the
instance's destructor is called.  However the library API may say that when the
instance is passed to a particular function, the library takes ownership of the
instance, i.e. responsibility for calling the instance's destructor is
transferred from the SIP generated module to the library.

Ownership of an instance may also be associated with another instance.  The
implication being that the owned instance will automatically be destroyed if
the owning instance is destroyed.  SIP keeps track of these relationships to
ensure that Python's cyclic garbage collector can detect and break any
reference cycles between the owning and owned instances.  The association is
implemented as the owning instance taking a reference to the owned instance.

The TransferThis, Transfer and TransferBack annotations are used to specify
where, and it what direction, transfers of ownership happen.  It is very
important that these are specified correctly to avoid crashes (where both
Python and C++ call the destructor) and memory leaks (where neither Python and
C++ call the destructor).

This applies equally to C structures where the structure is returned to the
heap using the ``free()`` function.

See also :cfunc:`sipTransferTo()`, :cfunc:`sipTransferBack()` and
:cfunc:`sipTransferBreak()`.


.. _ref-types-metatypes:

Types and Meta-types
--------------------

Every Python object (with the exception of the :class:`object` object itself)
has a meta-type and at least one super-type.  By default an object's meta-type
is the meta-type of its first super-type.

SIP implements two super-types, :class:`sip.simplewrapper` and
:class:`sip.wrapper`, and a meta-type, :class:`sip.wrappertype`.

:class:`sip.simplewrapper` is the super-type of :class:`sip.wrapper`.  The
super-type of :class:`sip.simplewrapper` is :class:`object`.

:class:`sip.wrappertype` is the meta-type of both :class:`sip.simplewrapper`
and :class:`sip.wrapper`.  The super-type of :class:`sip.wrappertype` is
:class:`type`.

:class:`sip.wrapper` supports the concept of object ownership described in
:ref:`ref-object-ownership` and, by default, is the super-type of all the types
that SIP generates.

:class:`sip.simplewrapper` does not support the concept of object ownership but
SIP generated types that are sub-classed from it have Python objects that take
less memory.

SIP allows a class's meta-type and super-type to be explicitly specified using
the :canno:`Metatype` and :canno:`Supertype` class annotations.

SIP also allows the default meta-type and super-type to be changed for a module
using the :directive:`%DefaultMetatype` and :directive:`%DefaultSupertype`
directives.  Unlike the default super-type, the default meta-type is inherited
by importing modules.

If you want to use your own meta-type or super-type then they must be
sub-classed from one of the SIP provided types.  Your types must be registered
using :cfunc:`sipRegisterPyType()`.  This is normally done in code specified
using the :directive:`%InitialisationCode` directive.

As an example, PyQt4 uses :directive:`%DefaultMetatype` to specify a new
meta-type that handles the interaction with Qt's own meta-type system.  It also
uses :directive:`%DefaultSupertype` to specify that the smaller
:class:`sip.simplewrapper` super-type is normally used.  Finally it uses
:canno:`Supertype` as an annotation of the ``QObject`` class to override the
default and use :class:`sip.wrapper` as the super-type so that the parent/child
relationships of ``QObject`` instances are properly maintained.


.. _ref-lazy-type-attributes:

Lazy Type Attributes
--------------------

Instead of populating a wrapped type's dictionary with its attributes (or
descriptors for those attributes) SIP only creates objects for those attributes
when they are actually needed.  This is done to reduce the memory footprint and
start up time when used to wrap large libraries with hundreds of classes and
tens of thousands of attributes.

SIP allows you to extend the handling of lazy attributes to your own attribute
types by allowing you to register an attribute getter handler (using
:cfunc:`sipRegisterAttributeGetter()`).  This will be called just before a
type's dictionary is accessed for the first time.


Support for Python's Buffer Interface
-------------------------------------

SIP supports Python's buffer interface in that whenever C/C++ requires a
``char`` or ``char *`` type then any Python type that supports the buffer
interface (including ordinary Python strings) can be used.

If a buffer is made up of a number of segments then all but the first will be
ignored.


Support for Wide Characters
---------------------------

SIP v4.6 introduced support for wide characters (i.e. the ``wchar_t`` type).
Python's C API includes support for converting between unicode objects and wide
character strings and arrays.  When converting from a unicode object to wide
characters SIP creates the string or array on the heap (using memory allocated
using :cfunc:`sipMalloc()`).  This then raises the problem of how this memory
is subsequently freed.

The following describes how SIP handles this memory in the different situations
where this is an issue.

    - When a wide string or array is passed to a function or method then the
      memory is freed (using :cfunc:`sipFree()`) after than function or method
      returns.

    - When a wide string or array is returned from a virtual method then SIP
      does not free the memory until the next time the method is called.

    - When an assignment is made to a wide string or array instance variable
      then SIP does not first free the instance's current string or array.


.. _ref-gil:

The Python Global Interpreter Lock
----------------------------------

Python's Global Interpretor Lock (GIL) must be acquired before calls can be
made to the Python API.  It should also be released when a potentially
blocking call to C/C++ library is made in order to allow other Python threads
to be executed.  In addition, some C/C++ libraries may implement their own
locking strategies that conflict with the GIL causing application deadlocks.
SIP provides ways of specifying when the GIL is released and acquired to
ensure that locking problems can be avoided.

SIP always ensures that the GIL is acquired before making calls to the Python
API.  By default SIP does not release the GIL when making calls to the C/C++
library being wrapped.  The :fanno:`ReleaseGIL` annotation can be used to
override this behaviour when required.

If SIP is given the :option:`-g <sip -g>` command line option then the default
behaviour is changed and SIP releases the GIL every time is makes calls to the
C/C++ library being wrapped.  The :fanno:`HoldGIL` annotation can be used to
override this behaviour when required.


.. _ref-incompat-apis:

Managing Incompatible APIs
--------------------------

.. versionadded:: 4.9

Sometimes it is necessary to change the way something is wrapped in a way that
introduces an incompatibility.  For example a new feature of Python may
suggest that something may be wrapped in a different way to exploit that
feature.

SIP's :directive:`%Feature` directive could be used to provide two different
implementations.  However this would mean that the choice between the two
implementations would have to be made when building the generated module
potentially causing all sorts of deployment problems.  It may also require
applications to work out which implementation was available and to change
their behaviour accordingly.

Instead SIP provides limited support for providing multiple implementations
(of classes, mapped types and functions) that can be selected by an
application at run-time.  It is then up to the application developer how they
want to manage the migration from the old API to the new, incompatible API.

This support is implemented in three parts.

Firstly the :directive:`%API` directive is used to define the name of an API
and its default version number.  The default version number is the one used if
an application doesn't explicitly set the version number to use.

Secondly the :canno:`API class <API>`, :manno:`mapped type <API>` or
:fanno:`function <API>` annotation is applied accordingly to specify the API
and range of version numbers that a particular class, mapped type or function
implementation should be enabled for.

Finally the application calls :func:`sip.setapi` to specify the version number
of the API that should be enabled.  This call must be made before any module
that has multiple implementations is imported for the first time.

Note this mechanism is not intended as a way or providing equally valid
alternative APIs.  For example::

    %API MyAPI 1

    class Foo
    {
    public:
        void bar();
    };

    class Baz : Foo
    {
    public:
        void bar() /API=MyAPI:2-/;
    };

If the following Python code is executed then an exception will be raised::

    b = Baz()
    b.bar()

This is because when version 1 of the *MyAPI* API (the default) is enabled
there is no *Baz.bar()* implementation and *Foo.bar()* will not be called
instead as might be expected.