Recently I completed a project where I used PyObject_CallFunctionObjArgs extensively with the NLTK library from a program written in NASM, with no problems.  Now I am on a new project where I call the Python random library.  I use the same setup as
before, but I am getting a segfault with random.seed. 

At the start of the NASM program I call a C API program that gets PyObject pointers to “seed” and “randrange” in the same way as I did before:

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

if (pMod_random == 0x0){
PyErr_Print();
return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

Later in the same program I call a C API program to call random.seed:

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_1);

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

return return_val;
}

The first program correctly imports “random” and gets pointers to “seed” and “randrange.”  I verified that the same pointer is correctly passed into C_API_2, and the seed value (1234) is passed as  Py_ssize_t value_1.  But I get this
segfault:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at ../Include/object.h:459 459     ../Include/object.h: No such file or directory.

So I tried Py_INCREF in the first program: 

Py_INCREF(pMod_random);
Py_INCREF(pAttr_seed);

Then I moved Py_INCREF(pAttr_seed) to the second program.  Same segfault.

Finally, I initialized “random” and “seed” in the second program, where they are used.  Same segfault. 

The segfault refers to Py_INCREF, so this seems to do with reference counting, but Py_INCREF didn’t solve it.   

I’m using Python 3.8 on Ubuntu. 

Thanks for any ideas on how to solve this. 

Jen

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2022-09-29 16:54, Jen Kris via Python-list wrote:

Recently I completed a project where I used PyObject_CallFunctionObjArgs extensively with the NLTK library from a program written in NASM, with no problems.  Now I am on a new project where I call the Python random library.  I use the same setup as

before, but I am getting a segfault with random.seed.

At the start of the NASM program I call a C API program that gets PyObject pointers to “seed” and “randrange” in the same way as I did before:

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Both PyUnicode_FromString and PyImport_Import return new references or
null pointers.

if (pMod_random == 0x0){
PyErr_Print();

You're leaking a reference here (pName_random).

return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

You're leaking 2 references here (pName_random and pMod_random).

return 0;
}

Later in the same program I call a C API program to call random.seed:

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_1);

It's expecting all of the arguments to be PyObject*, but value_1 is
Py_ssize_t instead of PyObject* (a pointer to a _Python_ int).

The argument list must end with a null pointer.

It returns a new reference or a null pointer.

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

You're leaking a reference here (p_seed_calc).

return return_val;
}

The first program correctly imports “random” and gets pointers to “seed” and “randrange.”  I verified that the same pointer is correctly passed into C_API_2, and the seed value (1234) is passed as  Py_ssize_t value_1.  But I get this

segfault:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at ../Include/object.h:459 459     ../Include/object.h: No such file or directory.

So I tried Py_INCREF in the first program:

Py_INCREF(pMod_random);
Py_INCREF(pAttr_seed);

Then I moved Py_INCREF(pAttr_seed) to the second program.  Same segfault.

Finally, I initialized “random” and “seed” in the second program, where they are used.  Same segfault.

The segfault refers to Py_INCREF, so this seems to do with reference counting, but Py_INCREF didn’t solve it.

I’m using Python 3.8 on Ubuntu.

Thanks for any ideas on how to solve this.

Jen

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Thanks very much to @MRAB for taking time to answer.  I changed my code to conform to your answer (as best I understand your comments on references), but I still get the same error.  My comments continue below the new code immediately below. 

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Py_INCREF(pName_random);
Py_INCREF(pMod_random);

if (pMod_random == 0x0){
PyErr_Print();
return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");

Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = (PyObject * )value_1;
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr, NULL);

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

return return_val;
}

So I incremented the reference to all objects in Get_LibModules, but I still get the same segfault at PyObject_CallFunctionObjArgs.  Unfortunately, reference counting is not well documented so I’m not clear what’s wrong. 




Sep 29, 2022, 10:06 by python@mrabarnett.plus.com:

On 2022-09-29 16:54, Jen Kris via Python-list wrote:

Recently I completed a project where I used PyObject_CallFunctionObjArgs extensively with the NLTK library from a program written in NASM, with no problems.  Now I am on a new project where I call the Python random library.  I use the same setup as

before, but I am getting a segfault with random.seed.

At the start of the NASM program I call a C API program that gets PyObject pointers to “seed” and “randrange” in the same way as I did before:

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Both PyUnicode_FromString and PyImport_Import return new references or null pointers.

if (pMod_random == 0x0){
PyErr_Print();

You're leaking a reference here (pName_random).

return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

You're leaking 2 references here (pName_random and pMod_random).

return 0;
}

Later in the same program I call a C API program to call random.seed:

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_1); >>

It's expecting all of the arguments to be PyObject*, but value_1 is Py_ssize_t instead of PyObject* (a pointer to a _Python_ int).

The argument list must end with a null pointer.

It returns a new reference or a null pointer.

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

You're leaking a reference here (p_seed_calc).

return return_val;
}

The first program correctly imports “random” and gets pointers to “seed” and “randrange.”  I verified that the same pointer is correctly passed into C_API_2, and the seed value (1234) is passed as  Py_ssize_t value_1.  But I get this

segfault:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at ../Include/object.h:459
459     ../Include/object.h: No such file or directory.

So I tried Py_INCREF in the first program:

Py_INCREF(pMod_random);
Py_INCREF(pAttr_seed);

Then I moved Py_INCREF(pAttr_seed) to the second program.  Same segfault. >>
Finally, I initialized “random” and “seed” in the second program, where they are used.  Same segfault.

The segfault refers to Py_INCREF, so this seems to do with reference counting, but Py_INCREF didn’t solve it.

I’m using Python 3.8 on Ubuntu.

Thanks for any ideas on how to solve this.

Jen

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

To update my previous email, I found the problem, but I have a new problem. 

Previously I cast PyObject * value_ptr = (PyObject * )value_1 but that's not correct.  Instead I used PyObject * value_ptr = PyLong_FromLong(value_1) and that works.  HOWEVER, while PyObject_CallFunctionObjArgs does work now, it returns -1, which is
not the right answer for random.seed.  I use "long return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long. 

So my question is why do I get -1 as return value?  When I query p_seed calc : get:

(gdb) p p_seed_calc
$2 = (PyObject *) 0x7ffff69be120 <_Py_NoneStruct>

Thanks again.

Jen




Sep 29, 2022, 13:02 by python-list@python.org:

Thanks very much to @MRAB for taking time to answer.  I changed my code to conform to your answer (as best I understand your comments on references), but I still get the same error.  My comments continue below the new code immediately below. 

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Py_INCREF(pName_random);
Py_INCREF(pMod_random);

if (pMod_random == 0x0){
PyErr_Print();
return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");

Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = (PyObject * )value_1;
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr, NULL);

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

return return_val;
}

So I incremented the reference to all objects in Get_LibModules, but I still get the same segfault at PyObject_CallFunctionObjArgs.  Unfortunately, reference counting is not well documented so I’m not clear what’s wrong. 

Sep 29, 2022, 10:06 by python@mrabarnett.plus.com:

On 2022-09-29 16:54, Jen Kris via Python-list wrote:

Recently I completed a project where I used PyObject_CallFunctionObjArgs extensively with the NLTK library from a program written in NASM, with no problems.  Now I am on a new project where I call the Python random library.  I use the same setup as

before, but I am getting a segfault with random.seed.

At the start of the NASM program I call a C API program that gets PyObject pointers to “seed” and “randrange” in the same way as I did before:

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Both PyUnicode_FromString and PyImport_Import return new references or null pointers.

if (pMod_random == 0x0){
PyErr_Print();

You're leaking a reference here (pName_random).

return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

You're leaking 2 references here (pName_random and pMod_random).

return 0;
}

Later in the same program I call a C API program to call random.seed:

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_1); >>>

It's expecting all of the arguments to be PyObject*, but value_1 is Py_ssize_t instead of PyObject* (a pointer to a _Python_ int).

The argument list must end with a null pointer.

It returns a new reference or a null pointer.

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

You're leaking a reference here (p_seed_calc).

return return_val;
}

The first program correctly imports “random” and gets pointers to “seed” and “randrange.”  I verified that the same pointer is correctly passed into C_API_2, and the seed value (1234) is passed as  Py_ssize_t value_1.  But I get this

segfault:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at ../Include/object.h:459
459     ../Include/object.h: No such file or directory.

So I tried Py_INCREF in the first program:

Py_INCREF(pMod_random);
Py_INCREF(pAttr_seed);

Then I moved Py_INCREF(pAttr_seed) to the second program.  Same segfault. >>>
Finally, I initialized “random” and “seed” in the second program, where they are used.  Same segfault.

The segfault refers to Py_INCREF, so this seems to do with reference counting, but Py_INCREF didn’t solve it.

I’m using Python 3.8 on Ubuntu.

Thanks for any ideas on how to solve this.

Jen

--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2022-09-29 21:47, Jen Kris wrote:

To update my previous email, I found the problem, but I have a new
problem.

Previously I cast PyObject * value_ptr = (PyObject * )value_1 but
that's not correct.  Instead I used PyObject * value_ptr = PyLong_FromLong(value_1) and that works.  HOWEVER, while PyObject_CallFunctionObjArgs does work now, it returns -1, which is
not the right answer for random.seed.  I use "long return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long.

random.seed returns None, so when you call PyObject_CallFunctionObjArgs
it returns a new reference to Py_None.

If you then pass to PyLong_AsLong a reference to something that's not a
PyLong, it'll set an error and return -1.

So my question is why do I get -1 as return value?  When I query
p_seed calc : get:

(gdb) p p_seed_calc
$2 = (PyObject *) 0x7ffff69be120 <_Py_NoneStruct>

Exactly. It's Py_None, not a PyLong.

Thanks again.

Jen

Sep 29, 2022, 13:02 by python-list@python.org:

Thanks very much to @MRAB for taking time to answer.  I changed my
code to conform to your answer (as best I understand your comments
on references), but I still get the same error.  My comments
continue below the new code immediately below.

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Py_INCREF(pName_random);
Py_INCREF(pMod_random);

if (pMod_random == 0x0){
PyErr_Print();
return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
"randrange");

Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = (PyObject * )value_1;
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
value_ptr, NULL);

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

return return_val;
}

So I incremented the reference to all objects in Get_LibModules,
but I still get the same segfault at
PyObject_CallFunctionObjArgs.  Unfortunately, reference counting
is not well documented so I’m not clear what’s wrong.

Sep 29, 2022, 10:06 by python@mrabarnett.plus.com:

On 2022-09-29 16:54, Jen Kris via Python-list wrote:

Recently I completed a project where I used
PyObject_CallFunctionObjArgs extensively with the NLTK
library from a program written in NASM, with no problems.
Now I am on a new project where I call the Python random
library.  I use the same setup as before, but I am getting
a segfault with random.seed.

At the start of the NASM program I call a C API program
that gets PyObject pointers to “seed” and “randrange” in
the same way as I did before:

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Both PyUnicode_FromString and PyImport_Import return new
references or null pointers.

if (pMod_random == 0x0){
PyErr_Print();

You're leaking a reference here (pName_random).

return 1;}

PyObject * pAttr_seed =
PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange =
PyObject_GetAttrString(pMod_random, "randrange");

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

You're leaking 2 references here (pName_random and pMod_random).

return 0;
}

Later in the same program I call a C API program to call
random.seed:

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * p_seed_calc =
PyObject_CallFunctionObjArgs(pAttr_seed, value_1);

It's expecting all of the arguments to be PyObject*, but
value_1 is Py_ssize_t instead of PyObject* (a pointer to a
_Python_ int).

The argument list must end with a null pointer.

It returns a new reference or a null pointer.

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

You're leaking a reference here (p_seed_calc).

return return_val;
}

The first program correctly imports “random” and gets
pointers to “seed” and “randrange.”  I verified that the
same pointer is correctly passed into C_API_2, and the
seed value (1234) is passed as  Py_ssize_t value_1.  But I
get this segfault:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at
../Include/object.h:459
459     ../Include/object.h: No such file or directory.

So I tried Py_INCREF in the first program:

Py_INCREF(pMod_random);
Py_INCREF(pAttr_seed);

Then I moved Py_INCREF(pAttr_seed) to the second program. 
Same segfault.

Finally, I initialized “random” and “seed” in the second
program, where they are used.  Same segfault.

The segfault refers to Py_INCREF, so this seems to do with
reference counting, but Py_INCREF didn’t solve it.

I’m using Python 3.8 on Ubuntu.

Thanks for any ideas on how to solve this.

Jen

--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I just solved this C API problem, and I’m posting the answer to help anyone else who might need it. 

The errors were:

(1) we must call Py_INCREF on each object when it’s created.

(2) in C_API_2 (see below) we don’t cast value_1 as I did before with PyObject * value_ptr = (PyObject * )value_1.  Instead we use PyObject * value_ptr = PyLong_FromLong(value_1);

(3) The command string to PyObject_CallFunctionObjArgs must be null terminated.

Here’s the revised code:

First we load the modules, and increment the reference to each object: 

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Py_INCREF(pName_random);
Py_INCREF(pMod_random);

if (pMod_random == 0x0){
PyErr_Print();
return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");

Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

Next we call a program to initialize the random number generator with random.seed(), and increment the reference to its return value p_seed_calc:

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = PyLong_FromLong(value_1);
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr, NULL);

// _________

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

Py_INCREF(p_seed_calc);

return 0;
}

Now we call another program to get a random number:

int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1)
{
PyObject * value_ptr = PyLong_FromLong(value_1);
PyObject * p_randrange_calc = PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);

if (p_randrange_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_randrange_calc);

return return_val;
}

That returns 28, which is what I get from the Python command line. 

Thanks again to MRAB for helpful comments. 

Jen


Sep 29, 2022, 15:31 by python@mrabarnett.plus.com:

On 2022-09-29 21:47, Jen Kris wrote:

To update my previous email, I found the problem, but I have a new problem. >>
Previously I cast PyObject * value_ptr = (PyObject * )value_1 but that's not correct.  Instead I used PyObject * value_ptr = PyLong_FromLong(value_1) and that works.  HOWEVER, while PyObject_CallFunctionObjArgs does work now, it returns -1, which is

not the right answer for random.seed.  I use "long return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long.

random.seed returns None, so when you call PyObject_CallFunctionObjArgs it returns a new reference to Py_None.

If you then pass to PyLong_AsLong a reference to something that's not a PyLong, it'll set an error and return -1.

So my question is why do I get -1 as return value?  When I query p_seed calc : get:

(gdb) p p_seed_calc
$2 = (PyObject *) 0x7ffff69be120 <_Py_NoneStruct>

Exactly. It's Py_None, not a PyLong.

Thanks again.

Jen




Sep 29, 2022, 13:02 by python-list@python.org:

Thanks very much to @MRAB for taking time to answer.  I changed my
code to conform to your answer (as best I understand your comments
on references), but I still get the same error.  My comments
continue below the new code immediately below.

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Py_INCREF(pName_random);
Py_INCREF(pMod_random);

if (pMod_random == 0x0){
PyErr_Print();
return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
"randrange");

Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = (PyObject * )value_1;
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
value_ptr, NULL);

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

return return_val;
}

So I incremented the reference to all objects in Get_LibModules,
but I still get the same segfault at
PyObject_CallFunctionObjArgs.  Unfortunately, reference counting
is not well documented so I’m not clear what’s wrong.




Sep 29, 2022, 10:06 by python@mrabarnett.plus.com:

On 2022-09-29 16:54, Jen Kris via Python-list wrote:

Recently I completed a project where I used
PyObject_CallFunctionObjArgs extensively with the NLTK
library from a program written in NASM, with no problems.
Now I am on a new project where I call the Python random
library.  I use the same setup as before, but I am getting
a segfault with random.seed.

At the start of the NASM program I call a C API program
that gets PyObject pointers to “seed” and “randrange” in
the same way as I did before:

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Both PyUnicode_FromString and PyImport_Import return new
references or null pointers.

if (pMod_random == 0x0){
PyErr_Print();


You're leaking a reference here (pName_random).

return 1;}

PyObject * pAttr_seed =
PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange =
PyObject_GetAttrString(pMod_random, "randrange");

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;


You're leaking 2 references here (pName_random and pMod_random).

return 0;
}

Later in the same program I call a C API program to call
random.seed:

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * p_seed_calc =
PyObject_CallFunctionObjArgs(pAttr_seed, value_1);


It's expecting all of the arguments to be PyObject*, but
value_1 is Py_ssize_t instead of PyObject* (a pointer to a
_Python_ int).

The argument list must end with a null pointer.

It returns a new reference or a null pointer.


if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

You're leaking a reference here (p_seed_calc).

return return_val;
}

The first program correctly imports “random” and gets
pointers to “seed” and “randrange.”  I verified that the
same pointer is correctly passed into C_API_2, and the
seed value (1234) is passed as  Py_ssize_t value_1.  But I
get this segfault:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at
../Include/object.h:459
459     ../Include/object.h: No such file or directory.

So I tried Py_INCREF in the first program:

Py_INCREF(pMod_random);
Py_INCREF(pAttr_seed);

Then I moved Py_INCREF(pAttr_seed) to the second program. 
Same segfault.

Finally, I initialized “random” and “seed” in the second
program, where they are used.  Same segfault.

The segfault refers to Py_INCREF, so this seems to do with
reference counting, but Py_INCREF didn’t solve it.

I’m using Python 3.8 on Ubuntu.

Thanks for any ideas on how to solve this.

Jen


-- https://mail.python.org/mailman/listinfo/python-list


-- https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2022-09-29 23:41, Jen Kris wrote:

I just solved this C API problem, and I’m posting the answer to help
anyone else who might need it.

The errors were:

(1) we must call Py_INCREF on each object when it’s created.

Some functions return an object that has already been incref'ed ("new reference"). This occurs when it has either created a new object (the
refcount will be 1) or has returned a pointer to an existing object (the refcount will be > 1 because it has been incref'ed).

Other functions return an object that hasn't been incref'ed. This occurs
when you're looking up something, for example, looking at a member of a
list or the value of an attribute.

(2) in C_API_2 (see below) we don’t cast value_1 as I did before with PyObject * value_ptr = (PyObject * )value_1.  Instead we use PyObject
* value_ptr = PyLong_FromLong(value_1);

(3) The command string to PyObject_CallFunctionObjArgs must be null terminated.

Always read the docs carefully!

Here’s the revised code:

First we load the modules, and increment the reference to each object:

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Possible null pointers here.

Py_INCREF(pName_random);
Py_INCREF(pMod_random);

Py_INCREF will fail on a null pointer, which you haven't check for yet.

If they aren't null, then they're new references (incref'ed) that you've incref'ed.

if (pMod_random == 0x0){
PyErr_Print();

Leaks here because of the refcount.

return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");

Possible null pointers here too.

Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);

Same issue as above.

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

Next we call a program to initialize the random number generator with random.seed(), and increment the reference to its return value
p_seed_calc:

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = PyLong_FromLong(value_1);

New reference.

PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
value_ptr, NULL);

// _________

if (p_seed_calc == 0x0){
    PyErr_Print();

Leak.

    return 1;}

Py_INCREF(p_seed_calc);

p_seed_calc will be a new reference to Py_None, incref'ed again, so leak.

return 0;
}

Now we call another program to get a random number:

int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1)
{
PyObject * value_ptr = PyLong_FromLong(value_1);
PyObject * p_randrange_calc =
PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);

if (p_randrange_calc == 0x0){

Leak.

    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_randrange_calc);

Leak.

return return_val;
}

That returns 28, which is what I get from the Python command line.

Thanks again to MRAB for helpful comments.

Jen

Getting refcounting right can be difficult!

Sep 29, 2022, 15:31 by python@mrabarnett.plus.com:

On 2022-09-29 21:47, Jen Kris wrote:

To update my previous email, I found the problem, but I have a
new problem.

Previously I cast PyObject * value_ptr = (PyObject * )value_1
but that's not correct.  Instead I used PyObject * value_ptr =
PyLong_FromLong(value_1) and that works. HOWEVER, while
PyObject_CallFunctionObjArgs does work now, it returns -1,
which is not the right answer for random.seed.  I use "long
return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long.

random.seed returns None, so when you call
PyObject_CallFunctionObjArgs it returns a new reference to Py_None.

If you then pass to PyLong_AsLong a reference to something that's
not a PyLong, it'll set an error and return -1.

So my question is why do I get -1 as return value?  When I
query p_seed calc : get:

(gdb) p p_seed_calc
$2 = (PyObject *) 0x7ffff69be120 <_Py_NoneStruct>

Exactly. It's Py_None, not a PyLong.

Thanks again.

Jen

Sep 29, 2022, 13:02 by python-list@python.org:

Thanks very much to @MRAB for taking time to answer.  I changed my
code to conform to your answer (as best I understand your comments
on references), but I still get the same error.  My comments
continue below the new code immediately below.

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Py_INCREF(pName_random);
Py_INCREF(pMod_random);

if (pMod_random == 0x0){
PyErr_Print();
return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random,
"seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
"randrange");

Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = (PyObject * )value_1;
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
value_ptr, NULL);

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

return return_val;
}

So I incremented the reference to all objects in Get_LibModules,
but I still get the same segfault at
PyObject_CallFunctionObjArgs.  Unfortunately, reference counting
is not well documented so I’m not clear what’s wrong.

Sep 29, 2022, 10:06 by python@mrabarnett.plus.com:

On 2022-09-29 16:54, Jen Kris via Python-list wrote:

Recently I completed a project where I used
PyObject_CallFunctionObjArgs extensively with the NLTK
library from a program written in NASM, with no problems.
Now I am on a new project where I call the Python random
library.  I use the same setup as before, but I am getting
a segfault with random.seed.

At the start of the NASM program I call a C API program
that gets PyObject pointers to “seed” and “randrange” in
the same way as I did before:

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Both PyUnicode_FromString and PyImport_Import return new
references or null pointers.

if (pMod_random == 0x0){
PyErr_Print();

You're leaking a reference here (pName_random).

return 1;}

PyObject * pAttr_seed =
PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange =
PyObject_GetAttrString(pMod_random, "randrange");

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

You're leaking 2 references here (pName_random and pMod_random).

return 0;
}

Later in the same program I call a C API program to call
random.seed:

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * p_seed_calc =
PyObject_CallFunctionObjArgs(pAttr_seed, value_1);

It's expecting all of the arguments to be PyObject*, but
value_1 is Py_ssize_t instead of PyObject* (a pointer to a
_Python_ int).

The argument list must end with a null pointer.

It returns a new reference or a null pointer.

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

You're leaking a reference here (p_seed_calc).

return return_val;
}

The first program correctly imports “random” and gets
pointers to “seed” and “randrange.”  I verified that the
same pointer is correctly passed into C_API_2, and the
seed value (1234) is passed as  Py_ssize_t value_1.  But I
get this segfault:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at
../Include/object.h:459
459     ../Include/object.h: No such file or directory.

So I tried Py_INCREF in the first program:

Py_INCREF(pMod_random);
Py_INCREF(pAttr_seed);

Then I moved Py_INCREF(pAttr_seed) to the second program.
Same segfault.

Finally, I initialized “random” and “seed” in the second
program, where they are used.  Same segfault.

The segfault refers to Py_INCREF, so this seems to do with
reference counting, but Py_INCREF didn’t solve it.

I’m using Python 3.8 on Ubuntu.

Thanks for any ideas on how to solve this.

Jen

-- https://mail.python.org/mailman/listinfo/python-list

-- https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2022-09-30 01:02, MRAB wrote:

On 2022-09-29 23:41, Jen Kris wrote:

I just solved this C API problem, and I’m posting the answer to help
anyone else who might need it.

[snip]

What I like to do is write comments that state which variables hold a reference, followed by '+' if it's a new reference (incref'ed) and '?'
if it could be null. '+?' means that it's probably a new reference but
could be null. Once I know that it's not null, I can remove the '?', and
once I've decref'ed it (if required) and no longer need it, I remobe it
from the comment.

Clearing up references, as soon as they're not needed, helps to keep the
number of current references more manageable.


int64_t Get_LibModules(int64_t * return_array) {
PyObject * pName_random = PyUnicode_FromString("random");
//> pName_random+?
if (!pName_random) {
PyErr_Print();
return 1;
}

//> pName_random+
PyObject * pMod_random = PyImport_Import(pName_random);
//> pName_random+ pMod_random+?
Py_DECREF(pName_random);
//> pMod_random+?
if (!pMod_random) {
PyErr_Print();
return 1;
}

//> pMod_random+
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
//> pMod_random+ pAttr_seed?
if (!pAttr_seed) {
Py_DECREF(pMod_random);
PyErr_Print();
return 1;
}

//> pMod_random+ pAttr_seed
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");
//> pMod_random+ pAttr_seed pAttr_randrange?
Py_DECREF(pMod_random);
//> pAttr_seed pAttr_randrange?
if (!pAttr_randrange) {
PyErr_Print();
return 1;
}

//> pAttr_seed pAttr_randrange
return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
PyObject * value_ptr = PyLong_FromLong(value_1);
//> value_ptr+?
if (!!value_ptr) {
PyErr_Print();
return 1;
}

//> value_ptr+
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr, NULL);
//> value_ptr+ p_seed_calc+?
Py_DECREF(value_ptr);
//> p_seed_calc+?
if (!p_seed_calc) {
PyErr_Print();
return 1;
}

//> p_seed_calc+
Py_DECREF(p_seed_calc);
return 0;
}

int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
PyObject * value_ptr = PyLong_FromLong(value_1);
//> value_ptr+?
if (!value_ptr) {
PyErr_Print();
return 1;
}

//> value_ptr+
PyObject * p_randrange_calc = PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
//> value_ptr+ p_randrange_calc+?
Py_DECREF(value_ptr);
//> p_randrange_calc+?
if (!p_randrange_calc) {
PyErr_Print();
return 1;
}

//Prepare return values
//> p_randrange_calc+
return_val = PyLong_AsLong(p_randrange_calc);
Py_DECREF(p_randrange_calc);

return return_val;
}

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Thanks very much for your detailed reply.  I have a few followup questions. 

You said, “Some functions return an object that has already been incref'ed ("new reference"). This occurs when it has either created a new object (the refcount will be 1) or has returned a pointer to an existing object (the refcount will be > 1 because
it has been incref'ed).  Other functions return an object that hasn't been incref'ed. This occurs when you're looking up something, for example, looking at a member of a list or the value of an attribute.” 

In the official docs some functions show “Return value: New reference” and others do not.  Is there any reason why I should not just INCREF on every new object, regardless of whether it’s a new reference or not, and DECREF when I am finished with
it?  The answer at https://stackoverflow.com/questions/59870703/python-c-extension-need-to-py-incref-a-borrowed-reference-if-not-returning-it-to says “With out-of-order execution, the INCREF/DECREF are basically free operations, so performance is no
reason to leave them out.”  Doing so means I don’t have to check each object to see if it needs to be INCREF’d or not, and that is a big help. 

Also: 

What is a borrowed reference, and how does it effect reference counting?  According to https://jayrambhia.com/blog/pythonc-api-reference-counting, “Use Py_INCREF on a borrowed PyObject pointer you already have. This increments the reference count on
the object, and obligates you to dispose of it properly.”  So I guess it’s yes, but I’m confused by “pointer you already have.” 

What does it mean to steal a reference?  If a function steals a reference does it have to decref it without incref (because it’s stolen)?

Finally, you said:

if (pMod_random == 0x0){
    PyErr_Print();
Leaks here because of the refcount

Assuming pMod_random is not null, why would this leak? 

Thanks again for your input on this question. 

Jen



Sep 29, 2022, 17:33 by python@mrabarnett.plus.com:

On 2022-09-30 01:02, MRAB wrote:

On 2022-09-29 23:41, Jen Kris wrote:

I just solved this C API problem, and I’m posting the answer to help anyone else who might need it.

[snip]

What I like to do is write comments that state which variables hold a reference, followed by '+' if it's a new reference (incref'ed) and '?' if it could be null. '+?' means that it's probably a new reference but could be null. Once I know that it's not

null, I can remove the '?', and once I've decref'ed it (if required) and no longer need it, I remobe it from the comment.

Clearing up references, as soon as they're not needed, helps to keep the number of current references more manageable.

int64_t Get_LibModules(int64_t * return_array) {
PyObject * pName_random = PyUnicode_FromString("random");

pName_random+?

if (!pName_random) {
PyErr_Print();
return 1;
}

pName_random+

PyObject * pMod_random = PyImport_Import(pName_random);

pName_random+ pMod_random+?

Py_DECREF(pName_random);

pMod_random+?

if (!pMod_random) {
PyErr_Print();
return 1;
}

pMod_random+

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");

pMod_random+ pAttr_seed?

if (!pAttr_seed) {
Py_DECREF(pMod_random);
PyErr_Print();
return 1;
}

pMod_random+ pAttr_seed

PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");

pMod_random+ pAttr_seed pAttr_randrange?

Py_DECREF(pMod_random);

pAttr_seed pAttr_randrange?

if (!pAttr_randrange) {
PyErr_Print();
return 1;
}

pAttr_seed pAttr_randrange

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
PyObject * value_ptr = PyLong_FromLong(value_1);

value_ptr+?

if (!!value_ptr) {
PyErr_Print();
return 1;
}

value_ptr+

PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr, NULL);

value_ptr+ p_seed_calc+?

Py_DECREF(value_ptr);

p_seed_calc+?

if (!p_seed_calc) {
PyErr_Print();
return 1;
}

p_seed_calc+

Py_DECREF(p_seed_calc);
return 0;
}

int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
PyObject * value_ptr = PyLong_FromLong(value_1);

value_ptr+?

if (!value_ptr) {
PyErr_Print();
return 1;
}

value_ptr+

PyObject * p_randrange_calc = PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);

value_ptr+ p_randrange_calc+?

Py_DECREF(value_ptr);

p_randrange_calc+?

if (!p_randrange_calc) {
PyErr_Print();
return 1;
}

//Prepare return values

p_randrange_calc+

return_val = PyLong_AsLong(p_randrange_calc);
Py_DECREF(p_randrange_calc);

return return_val;
}

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2022-09-30 17:02, Jen Kris wrote:

Thanks very much for your detailed reply.  I have a few followup
questions.

You said, “Some functions return an object that has already been
incref'ed ("new reference"). This occurs when it has either created a
new object (the refcount will be 1) or has returned a pointer to an
existing object (the refcount will be > 1 because it has been
incref'ed).  Other functions return an object that hasn't been
incref'ed. This occurs when you're looking up something, for example,
looking at a member of a list or the value of an attribute.”

In the official docs some functions show “Return value: New reference” and others do not.  Is there any reason why I should not just INCREF
on every new object, regardless of whether it’s a new reference or
not, and DECREF when I am finished with it?  The answer at https://stackoverflow.com/questions/59870703/python-c-extension-need-to-py-incref-a-borrowed-reference-if-not-returning-it-to
says “With out-of-order execution, the INCREF/DECREF are basically
free operations, so performance is no reason to leave them out.” 
Doing so means I don’t have to check each object to see if it needs to
be INCREF’d or not, and that is a big help.

It's OK to INCREF them, provided that you DECREF them when you no longer
need them, and remember that if it's a "new reference" you'd need to
DECREF it twice.

Also:

What is a borrowed reference, and how does it effect reference
counting?  According to https://jayrambhia.com/blog/pythonc-api-reference-counting, “Use
Py_INCREF on a borrowed PyObject pointer you already have. This
increments the reference count on the object, and obligates you to
dispose of it properly.”  So I guess it’s yes, but I’m confused by “pointer you already have.”

A borrowed reference is when it hasn't been INCREFed.

You can think of INCREFing as a way of indicating ownership, which is
often shared ownership (refcount > 1). When you're borrowing a
reference, you're using it temporarily, but not claiming ownership. When
the last owner releases its ownership (DECREF reduces the refcount to
0), the object can be garbage collected.

When, say, you lookup an attribute, or get an object from a list with PyList_GetItem, it won't have been INCREFed. You're using it
temporarily, just borrowing a reference.

What does it mean to steal a reference?  If a function steals a
reference does it have to decref it without incref (because it’s stolen)?

When function steals a reference, it's claiming ownership but not
INCREFing it.

Finally, you said:

if (pMod_random == 0x0){
    PyErr_Print();
Leaks here because of the refcount

Assuming pMod_random is not null, why would this leak?

It's pName_random that's the leak.

PyUnicode_FromString("random") will either create and return a new
object for the string "random" (refcount == 1) or return a reference to
an existing object (refcount > 1). You need to DECREF it before
returning from the function.

Suppose it created a new object. You call the function, it creates an
object, you use it, then return from the function. The object still
exists, but there's no reference to it. Now call the function again. It
creates another object, you use it, then return from the function. You
now have 2 objects with no reference to them.

Thanks again for your input on this question.

Jen

Sep 29, 2022, 17:33 by python@mrabarnett.plus.com:

On 2022-09-30 01:02, MRAB wrote:

On 2022-09-29 23:41, Jen Kris wrote:

I just solved this C API problem, and I’m posting the
answer to help anyone else who might need it.

[snip]

What I like to do is write comments that state which variables
hold a reference, followed by '+' if it's a new reference
(incref'ed) and '?' if it could be null. '+?' means that it's
probably a new reference but could be null. Once I know that it's
not null, I can remove the '?', and once I've decref'ed it (if
required) and no longer need it, I remobe it from the comment.

Clearing up references, as soon as they're not needed, helps to
keep the number of current references more manageable.

int64_t Get_LibModules(int64_t * return_array) {
PyObject * pName_random = PyUnicode_FromString("random");
//> pName_random+?
if (!pName_random) {
PyErr_Print();
return 1;
}

//> pName_random+
PyObject * pMod_random = PyImport_Import(pName_random);
//> pName_random+ pMod_random+?
Py_DECREF(pName_random);
//> pMod_random+?
if (!pMod_random) {
PyErr_Print();
return 1;
}

//> pMod_random+
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
//> pMod_random+ pAttr_seed?
if (!pAttr_seed) {
Py_DECREF(pMod_random);
PyErr_Print();
return 1;
}

//> pMod_random+ pAttr_seed
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
"randrange");
//> pMod_random+ pAttr_seed pAttr_randrange?
Py_DECREF(pMod_random);
//> pAttr_seed pAttr_randrange?
if (!pAttr_randrange) {
PyErr_Print();
return 1;
}

//> pAttr_seed pAttr_randrange
return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
PyObject * value_ptr = PyLong_FromLong(value_1);
//> value_ptr+?
if (!!value_ptr) {
PyErr_Print();
return 1;
}

//> value_ptr+
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
value_ptr, NULL);
//> value_ptr+ p_seed_calc+?
Py_DECREF(value_ptr);
//> p_seed_calc+?
if (!p_seed_calc) {
PyErr_Print();
return 1;
}

//> p_seed_calc+
Py_DECREF(p_seed_calc);
return 0;
}

int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
PyObject * value_ptr = PyLong_FromLong(value_1);
//> value_ptr+?
if (!value_ptr) {
PyErr_Print();
return 1;
}

//> value_ptr+
PyObject * p_randrange_calc =
PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
//> value_ptr+ p_randrange_calc+?
Py_DECREF(value_ptr);
//> p_randrange_calc+?
if (!p_randrange_calc) {
PyErr_Print();
return 1;
}

//Prepare return values
//> p_randrange_calc+
return_val = PyLong_AsLong(p_randrange_calc);
Py_DECREF(p_randrange_calc);

return return_val;
}

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 1/10/22 8:18 am, MRAB wrote:

It's OK to INCREF them, provided that you DECREF them when you no longer
need them, and remember that if it's a "new reference" you'd need to
DECREF it twice.

Which means there would usually be no point in doing the extra
INCREF/DECREF. You still need to know whether it's a new reference
or not and treat it accordingly.

--
Greg

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

That's great.  It clarifies things a lot for me, particularly re ref count for new references.  I would have had trouble if I didn't decref it twice. 

Thanks very much once again. 


Sep 30, 2022, 12:18 by python@mrabarnett.plus.com:

On 2022-09-30 17:02, Jen Kris wrote:

Thanks very much for your detailed reply.  I have a few followup questions. >>
You said, “Some functions return an object that has already been incref'ed ("new reference"). This occurs when it has either created a new object (the refcount will be 1) or has returned a pointer to an existing object (the refcount will be > 1

because it has been incref'ed).  Other functions return an object that hasn't been incref'ed. This occurs when you're looking up something, for example, looking at a member of a list or the value of an attribute.”

In the official docs some functions show “Return value: New reference” and others do not.  Is there any reason why I should not just INCREF on every new object, regardless of whether it’s a new reference or not, and DECREF when I am finished

with it?  The answer at https://stackoverflow.com/questions/59870703/python-c-extension-need-to-py-incref-a-borrowed-reference-if-not-returning-it-to says “With out-of-order execution, the INCREF/DECREF are basically free operations, so performance is
no reason to leave them out.”  Doing so means I don’t have to check each object to see if it needs to be INCREF’d or not, and that is a big help.

It's OK to INCREF them, provided that you DECREF them when you no longer need them, and remember that if it's a "new reference" you'd need to DECREF it twice.

Also:

What is a borrowed reference, and how does it effect reference counting?  According to https://jayrambhia.com/blog/pythonc-api-reference-counting, “Use Py_INCREF on a borrowed PyObject pointer you already have. This increments the reference count

on the object, and obligates you to dispose of it properly.”  So I guess it’s yes, but I’m confused by “pointer you already have.”

A borrowed reference is when it hasn't been INCREFed.

You can think of INCREFing as a way of indicating ownership, which is often shared ownership (refcount > 1). When you're borrowing a reference, you're using it temporarily, but not claiming ownership. When the last owner releases its ownership (DECREF

reduces the refcount to 0), the object can be garbage collected.

When, say, you lookup an attribute, or get an object from a list with PyList_GetItem, it won't have been INCREFed. You're using it temporarily, just borrowing a reference.

What does it mean to steal a reference?  If a function steals a reference does it have to decref it without incref (because it’s stolen)?

When function steals a reference, it's claiming ownership but not INCREFing it.

Finally, you said:

if (pMod_random == 0x0){
    PyErr_Print();
Leaks here because of the refcount

Assuming pMod_random is not null, why would this leak?

It's pName_random that's the leak.

PyUnicode_FromString("random") will either create and return a new object for the string "random" (refcount == 1) or return a reference to an existing object (refcount > 1). You need to DECREF it before returning from the function.

Suppose it created a new object. You call the function, it creates an object, you use it, then return from the function. The object still exists, but there's no reference to it. Now call the function again. It creates another object, you use it, then

return from the function. You now have 2 objects with no reference to them.

Thanks again for your input on this question.

Jen



Sep 29, 2022, 17:33 by python@mrabarnett.plus.com:

On 2022-09-30 01:02, MRAB wrote:

On 2022-09-29 23:41, Jen Kris wrote:


I just solved this C API problem, and I’m posting the
answer to help anyone else who might need it.

[snip]

What I like to do is write comments that state which variables
hold a reference, followed by '+' if it's a new reference
(incref'ed) and '?' if it could be null. '+?' means that it's
probably a new reference but could be null. Once I know that it's
not null, I can remove the '?', and once I've decref'ed it (if
required) and no longer need it, I remobe it from the comment.

Clearing up references, as soon as they're not needed, helps to
keep the number of current references more manageable.


int64_t Get_LibModules(int64_t * return_array) {
PyObject * pName_random = PyUnicode_FromString("random");

pName_random+?

if (!pName_random) {
PyErr_Print();
return 1;
}

pName_random+

PyObject * pMod_random = PyImport_Import(pName_random);

pName_random+ pMod_random+?

Py_DECREF(pName_random);

pMod_random+?

if (!pMod_random) {
PyErr_Print();
return 1;
}

pMod_random+

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");

pMod_random+ pAttr_seed?

if (!pAttr_seed) {
Py_DECREF(pMod_random);
PyErr_Print();
return 1;
}

pMod_random+ pAttr_seed

PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
"randrange");

pMod_random+ pAttr_seed pAttr_randrange?

Py_DECREF(pMod_random);

pAttr_seed pAttr_randrange?

if (!pAttr_randrange) {
PyErr_Print();
return 1;
}

pAttr_seed pAttr_randrange

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
PyObject * value_ptr = PyLong_FromLong(value_1);

value_ptr+?

if (!!value_ptr) {
PyErr_Print();
return 1;
}

value_ptr+

PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
value_ptr, NULL);

value_ptr+ p_seed_calc+?

Py_DECREF(value_ptr);

p_seed_calc+?

if (!p_seed_calc) {
PyErr_Print();
return 1;
}

p_seed_calc+

Py_DECREF(p_seed_calc);
return 0;
}

int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
PyObject * value_ptr = PyLong_FromLong(value_1);

value_ptr+?

if (!value_ptr) {
PyErr_Print();
return 1;
}

value_ptr+

PyObject * p_randrange_calc =
PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);

value_ptr+ p_randrange_calc+?

Py_DECREF(value_ptr);

p_randrange_calc+?

if (!p_randrange_calc) {
PyErr_Print();
return 1;
}

//Prepare return values

p_randrange_calc+

return_val = PyLong_AsLong(p_randrange_calc);
Py_DECREF(p_randrange_calc);

return return_val;
}

-- https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)