Lifetime of temporary objects in SWIG's Python wrappers (?)

Question

Edited 12 Feb

I've just recently come up with an odd crash using some SWIG-generated Python wrappers for some C++ classes. It seems that the combination of SWIG and Python together are somewhat eager to clean up temporary values. So eager, in fact, that they're cleaned up while they're still being used. A significantly condensed version looks like this:

/* Example.hpp */
struct Foo {
    int value;
    ~Foo();
};

struct Bar {
    Foo theFoo;
    Bar();
};

/* Example.cpp */
#include "Example.hpp"
Bar::Bar()  {theFoo.value=1;}
Foo::~Foo() {value=0;}

/* Example.i */
%module Example
%{
#include "Example.hpp"
%}
%include "Example.hpp"

I run SWIG (1.3.37) on the .i file, and then in Python, have:

Python 2.4.3 (#1, Sept 17 2008, 16:07:08)
[GCC 4.1.2 20071124 (Red Hat 4.1.2-41)] on linux2
Type "help", "copyright", "credits", or "license" for more information.
>>> from Example import Bar
>>> b=Bar()
>>> print b.theFoo.value      # expect '1', since Bar's constructor sets this
1
>>> print Bar().theFoo.value  # expect '1', since we're still using the Foo object
26403424

It seems that in the second instance, the temporary Bar object is destroyed before we ever get to read theFoo's value field. Chasing things around in gdb, this is clearly what's happening. And so by the time we read .value from Bar().theFoo, C++ has already destroyed (and overwritten with some other heap allocation) .theFoo. In my actual situation, this is causing a segfault.

Is there any SWIG directive or trick that I can add to my Example.i file to make Bar().theFoo.value return 1 here?

Answer 1

Second Update:

Well we know that the basic problem is that python destroys Bar immediately. When Bar is implemented in python, python's gc knows that there's still a reference to theFoo, and so does not destroy it. But when Bar is implemented in c++, python calls the c++ destructor, which automatically destroys theFoo along with Bar.

So the obvious solution is to prevent python from destroying Bar prematurely. Here's a slightly hackish solution involving subclassing Bar:

class PersistentBar(swigexample.Bar):
    lastpbar = None
    def __init__(self):
        super(PersistentBar, self).__init__()
        PersistentBar.lastpbar = self

This saves a reference to the last Bar created so that it isn't destroyed right away. When a new Bar is created, the old one is deleted. (My old version was silly; no need to override __del__ for this.) Here's the output (with cout << "deleting Foo " in Foo's destructor):

>>> from test import persistentBar
>>> persistentBar().theFoo.value
1
>>> persistentBar().theFoo.value
deleting Foo 1
>>> persistentBar().theFoo.value
deleting Foo 1

I still don't love this. It might be better to sequester the "persistent" behavior in a decorator; I tried that too and it worked (if you want to see the code let me know). It would definitely be better to somehow tell python to handle destroying theFoo itself, but I can't figure out how to do that.

First Update:

The wrap code told me nothing, so I looked in swigexample.py. That also yielded nothing. Things became clearer when I tried duplicating Bar in pure python:

# pyfoobar.py
class Foo(object):
    def __init__(self):
        self.value = -1

class Bar(object):
    def __init__(self):
        self.theFoo = Foo()
        self.theFoo.value = 1
    def __del__(self):
        self.theFoo.value = 0

Now we import Bar from pyfoobar:

>>> from pyfoobar import Bar
>>> b = Bar()
>>> b.theFoo.value
1
>>> Bar().theFoo.value
0

This behavior is coming from Python!

Original Answer:

It seems like there's definitely some garbage collection combat at work here... Here's some related information on SWIG Memory Management. Based on this, it looks like the %newobject directive might be what you're looking for; but I tried several variations and couldn't get it to give python control over theFoo:

>>> from swigexample import Bar
>>> b = Bar()
>>> b.theFoo.value
1
>>> b.theFoo.thisown
False
>>> Bar().theFoo.value
0
>>> Bar().theFoo.thisown
False

I'm beginning to suspect that this is intentional; seems like this line from the above link is relevant here:

C is now holding a reference to the object---you probably don't want Python to destroy it.

But I'm not certain. I'm going to look at the swigexample_wrap code to see if I can figure out when ~Bar is being called.

Answer 2

The solution is to add %naturalvar to your .i file like this:

%naturalvar Bar::theFoo;
%include "Example.hpp"

This causes SWIG to return a copy of Foo instead of a reference to it, which solves the problem of aggressive temporary object cleanup that Python does.