Where should I put Py_INCREF and Py_DECREF on this block in Python C Extension?

Question

Whenever I called my function, memory usage is increased around +10M per call, so I think there is some memory leak here.

....
PyObject *pair = PyTuple_New(2), *item = PyList_New(0);

PyTuple_SetItem(pair, 0, PyInt_FromLong(v[j]));

if(v[j] != DISTANCE_MAX && (p[j] || d[0][j])){
  jp=j;
  while(jp!=istart) {
    PyList_Append(item, PyInt_FromLong(jp));
    jp=p[jp];
  }

  PyList_Append(item, PyInt_FromLong(jp));

  PyList_Reverse(item);
}

PyTuple_SetItem(pair, 1, item);

return pair;
....

When I read document, some calls like

void
bug(PyObject *list)
{
    PyObject *item = PyList_GetItem(list, 0);

    PyList_SetItem(list, 1, PyInt_FromLong(0L));
    PyObject_Print(item, stdout, 0); /* BUG! */
}

need to put reference counts like this

void
no_bug(PyObject *list)
{
    PyObject *item = PyList_GetItem(list, 0);

    Py_INCREF(item);
    PyList_SetItem(list, 1, PyInt_FromLong(0L));
    PyObject_Print(item, stdout, 0);
    Py_DECREF(item);
}

So, Where should I put Py_INCREF and Py_DECREF on my function?

Answer 1

The objects you create with PyInt_FromLong() and you add to the list should be kept in a local variable.

The reason are the ownership rules: PyInt_FromLong() generates a reference that you own. In the call to PyTuple_SetItem(), you lose this ownership again, because PyTuple_SetItem() "steals" it from you, so you don't have to care about. But PyList_Append() doesn't do so, it increases the refcount. In order to have the object GC'ed correctly, you have to release your ownership by DECREF'ing.

So, instead of PyList_Append(item, PyInt_FromLong(jp)), you do the following:

PyObject * jpo = PyInt_FromLong(jp);
// do some error checking here
PyList_Append(item, jpo);
Py_DECREF(jpo);

This will make the program do the right thing.

Answer 2

When an object is created, its refcount will be 1, so after this:

my_item  = PyInt_FromLong(jp)

the object at my_item will have a refcount of 1.

When you store an item into a container, the item's reference count is incremented so that the item will be retained, so after this:

PyList_Append(my_list, my_item);

the object at my_item will have a refcount of 2.

Therefore, this line:

PyList_Append(item, PyInt_FromLong(jp));

will create an object with refcount 1 and store it in the list, incrementing the object's refcount to 2.