I am writing a spot speedup in assembly language for a short but computation-intensive Python loop, and I discovered something about Python array handling that I would like to clarify. 

For a simplified example, I created a matrix mx1 and assigned the array arr1 to the third row of the matrix:

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

The pointers to these are now the same:

ida = id(mx1[2]) - 140260325306880
idb = id(arr1) - 140260325306880

That’s great because when I encounter this in assembly or C, I can just borrow the pointer to row 3 for the array arr1, on the assumption that they will continue to point to the same object.  Then when I do any math operations in arr1 it will be
reflected in both arrays because they are now pointing to the same array:

arr1[0] += 2
print(mx1[2]) - [9, 8, 9]
print(arr1) - [9, 8, 9]

Now mx1 looks like this:

[ 1, 2, 3 ]
[ 4, 5, 6 ]
[ 9, 8, 9 ]

and it stays that way for remaining iterations.  

But on the next iteration we assign arr1 to something else:

arr1 = [ 10, 11, 12 ]
idc = id(arr1) – 140260325308160
idd = id(mx1[2]) – 140260325306880

Now arr1 is no longer equal to mx1[2], and any subsequent operations in arr1 will not affect mx1.  So where I’m rewriting some Python code in a low level language, I can’t assume that the two objects are equal because that equality will not remain
if either is reassigned.  So if I do some operation on one array I have to conform the two arrays for as long as they remain equal, I can’t just do it in one operation because I can’t rely on the objects remaining equal. 

Is my understanding of this correct?  Is there anything I’m missing? 

Thanks very much. 

Jen

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Jen Kris <jenkris@tutanota.com> writes:

So where I’m rewriting some Python code in a low level
language, I can’t assume that the two objects are equal
because that equality will not remain if either is reassigned.

I have not carefully read all of your message, so I might
miss something! But from the sentence quoted above, I wonder
whether you might mix names and objects.

When you are talking about objects, there is no assignment.
One can /not/ assign or reassign an object. One can only
assign /names/.

Also, "two objects are equal" is ambiguous. Better use
one of these two expression for these two different situations:

A Two different objects have the same value.
B Two names (or other expression) evaluate to the same object.

A

x =( 1, 2 )
y =( 1, 2 )
x is y

|False

x == y

|True

x and y are two different objects. They have the same value.

B

x =( 1, 2 )
y = x
x is y

|True

x == y

|True

"x" and "y" are two different names for the same object.
There is only ever /one/ object here. There are not "two
different objects which are the same", because ...

|Roughly speaking, to say of two things that they are
|identical is nonsense, and to say of one thing that it is
|identical with itself is to say nothing at all.

Tractatus 5.5303 - Ludwig Wittgenstein (1889/1951).

When two objects are equal in sense A, reassigning names
does not change this relationship between the objects. For
example,

x =( 1, 2 )
y =( 1, 2 )
x == y

|True

a = x
b = y
del x
del y

The interpreter was started, and then two objects were
created. These two objects are equal. There are only these
two objects, so when I say "these two objects", it's clear
which two objects I'm referring to. Then, I mix up the names
a bit. This does not affect these two objects. Theses two
objects are still equal. These two objects will be equal
until the end of their existence no matter what they are
called.

|You may call me Terry, you may call me Timmy
|You may call me Bobby, you may call me Zimmy
Bob Dylan (1941/)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, 11 Jan 2023 at 07:41, Jen Kris <jenkris@tutanota.com> wrote:

Thanks for your comments. I'd like to make one small point. You say:

"Assignment in Python is a matter of object references. It's not
"conform them as long as they remain equal". You'll have to think in
terms of object references the entire way."

But where they have been set to the same object, an operation on one will affect the other as long as they are equal (in Python). So I will have to conform them in those cases because Python will reflect any math operation in both the array and the

matrix.


It's not that "an operation on one will affect the other" - it's that,
no matter how you refer to that object, you're affecting *that one
single object*. It's like when you're washing a window; the inside and
outside of the window are the exact same window, so regardless of
where you're looking at it from, it's the same single window and
changes affect it equally.

So you shouldn't have to replicate any changes. What should be
happening is that you find the right object to mutate, and mutate
that. For example:

stuff = [[1, 2, 3], [4, 5, 6]]
stuff.append(stuff[0])
print(stuff)

You now have two references to the same list, inside another list. Any
change to stuff[0] is a change to stuff[2], because they're the exact
same list. When you append "a reference to this list over here" (which
you found by asking for stuff[0]) to the outer list, you get that
list.

That's Python's object model, and trying to cheat it by copying
changes is just going to cause untold nightmares of desynchronization.

ChrisA

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, 11 Jan 2023 at 07:14, Jen Kris via Python-list
<python-list@python.org> wrote:

I am writing a spot speedup in assembly language for a short but computation-intensive Python loop, and I discovered something about Python array handling that I would like to clarify.

For a simplified example, I created a matrix mx1 and assigned the array arr1 to the third row of the matrix:

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

The pointers to these are now the same:

ida = id(mx1[2]) - 140260325306880
idb = id(arr1) - 140260325306880

That’s great because when I encounter this in assembly or C, I can just borrow the pointer to row 3 for the array arr1, on the assumption that they will continue to point to the same object. Then when I do any math operations in arr1 it will be

reflected in both arrays because they are now pointing to the same array:


That's not an optimization; what you've done is set arr1 to be a
reference to that object.

But on the next iteration we assign arr1 to something else:

arr1 = [ 10, 11, 12 ]
idc = id(arr1) – 140260325308160
idd = id(mx1[2]) – 140260325306880

Now arr1 is no longer equal to mx1[2], and any subsequent operations in arr1 will not affect mx1.

Yep, you have just set arr1 to be a completely different object.

So where I’m rewriting some Python code in a low level language, I can’t assume that the two objects are equal because that equality will not remain if either is reassigned. So if I do some operation on one array I have to conform the two arrays

for as long as they remain equal, I can’t just do it in one operation because I can’t rely on the objects remaining equal.

Is my understanding of this correct? Is there anything I’m missing?

Assignment in Python is a matter of object references. It's not
"conform them as long as they remain equal". You'll have to think in
terms of object references the entire way.

ChrisA

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Thanks for your comments.  I'd like to make one small point.  You say:

"Assignment in Python is a matter of object references. It's not
"conform them as long as they remain equal". You'll have to think in
terms of object references the entire way."

But where they have been set to the same object, an operation on one will affect the other as long as they are equal (in Python).  So I will have to conform them in those cases because Python will reflect any math operation in both the array and the
matrix. 



Jan 10, 2023, 12:28 by rosuav@gmail.com:

On Wed, 11 Jan 2023 at 07:14, Jen Kris via Python-list <python-list@python.org> wrote:

I am writing a spot speedup in assembly language for a short but computation-intensive Python loop, and I discovered something about Python array handling that I would like to clarify.

For a simplified example, I created a matrix mx1 and assigned the array arr1 to the third row of the matrix:

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

The pointers to these are now the same:

ida = id(mx1[2]) - 140260325306880
idb = id(arr1) - 140260325306880

That’s great because when I encounter this in assembly or C, I can just borrow the pointer to row 3 for the array arr1, on the assumption that they will continue to point to the same object. Then when I do any math operations in arr1 it will be

reflected in both arrays because they are now pointing to the same array:

That's not an optimization; what you've done is set arr1 to be a
reference to that object.

But on the next iteration we assign arr1 to something else:

arr1 = [ 10, 11, 12 ]
idc = id(arr1) – 140260325308160
idd = id(mx1[2]) – 140260325306880

Now arr1 is no longer equal to mx1[2], and any subsequent operations in arr1 will not affect mx1.

Yep, you have just set arr1 to be a completely different object.

So where I’m rewriting some Python code in a low level language, I can’t assume that the two objects are equal because that equality will not remain if either is reassigned. So if I do some operation on one array I have to conform the two arrays

for as long as they remain equal, I can’t just do it in one operation because I can’t rely on the objects remaining equal.

Is my understanding of this correct? Is there anything I’m missing?

Assignment in Python is a matter of object references. It's not
"conform them as long as they remain equal". You'll have to think in
terms of object references the entire way.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Jen Kris via Python-list schreef op 10/01/2023 om 21:41:

But where they have been set to the same object, an operation on one will affect the other as long as they are equal (in Python).

As long as they are *identical*, not equal. Identical as in having the
same identity as Python defines it.
I advise you to read Ned Batchelder's explanation about names and values
in Python, or watch his presentation, to get a good understanding. See https://nedbatchelder.com/text/names1.html

--
"Don't Panic."
-- Douglas Adams, The Hitchhiker's Guide to the Galaxy

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2023-01-10 20:41, Jen Kris via Python-list wrote:

Thanks for your comments.  I'd like to make one small point.  You say:

"Assignment in Python is a matter of object references. It's not
"conform them as long as they remain equal". You'll have to think in
terms of object references the entire way."

But where they have been set to the same object, an operation on one will affect the other as long as they are equal (in Python).  So I will have to conform them in those cases because Python will reflect any math operation in both the array and the

matrix.

It's not a 2D matrix, it's a 1D list containing references to 1D lists,
each of which contains references to Python ints.

In CPython, references happen to be pointers, but that's just an
implementation detail.

Jan 10, 2023, 12:28 by rosuav@gmail.com:

On Wed, 11 Jan 2023 at 07:14, Jen Kris via Python-list
<python-list@python.org> wrote:

I am writing a spot speedup in assembly language for a short but computation-intensive Python loop, and I discovered something about Python array handling that I would like to clarify.

For a simplified example, I created a matrix mx1 and assigned the array arr1 to the third row of the matrix:

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

The pointers to these are now the same:

ida = id(mx1[2]) - 140260325306880
idb = id(arr1) - 140260325306880

That’s great because when I encounter this in assembly or C, I can just borrow the pointer to row 3 for the array arr1, on the assumption that they will continue to point to the same object. Then when I do any math operations in arr1 it will be

reflected in both arrays because they are now pointing to the same array:

That's not an optimization; what you've done is set arr1 to be a
reference to that object.

But on the next iteration we assign arr1 to something else:

arr1 = [ 10, 11, 12 ]
idc = id(arr1) – 140260325308160
idd = id(mx1[2]) – 140260325306880

Now arr1 is no longer equal to mx1[2], and any subsequent operations in arr1 will not affect mx1.

Yep, you have just set arr1 to be a completely different object.

So where I’m rewriting some Python code in a low level language, I can’t assume that the two objects are equal because that equality will not remain if either is reassigned. So if I do some operation on one array I have to conform the two arrays

for as long as they remain equal, I can’t just do it in one operation because I can’t rely on the objects remaining equal.

Is my understanding of this correct? Is there anything I’m missing?

Assignment in Python is a matter of object references. It's not
"conform them as long as they remain equal". You'll have to think in
terms of object references the entire way.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

For clarification, equality is not identity in Python. e.g.

x = 7
y = 7.0
print(x == y)
print(x is y)

Will return
True
False

Full explanation at https://docs.python.org/3/reference/expressions.html#comparisons


From: Python-list <python-list-bounces+gweatherby=uchc.edu@python.org> on behalf of Chris Angelico <rosuav@gmail.com>
Date: Tuesday, January 10, 2023 at 3:47 PM
To: Python List <python-list@python.org>
Subject: Re: To clarify how Python handles two equal objects
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***

On Wed, 11 Jan 2023 at 07:41, Jen Kris <jenkris@tutanota.com> wrote:

Thanks for your comments. I'd like to make one small point. You say:

"Assignment in Python is a matter of object references. It's not
"conform them as long as they remain equal". You'll have to think in
terms of object references the entire way."

But where they have been set to the same object, an operation on one will affect the other as long as they are equal (in Python). So I will have to conform them in those cases because Python will reflect any math operation in both the array and the

matrix.


It's not that "an operation on one will affect the other" - it's that,
no matter how you refer to that object, you're affecting *that one
single object*. It's like when you're washing a window; the inside and
outside of the window are the exact same window, so regardless of
where you're looking at it from, it's the same single window and
changes affect it equally.

So you shouldn't have to replicate any changes. What should be
happening is that you find the right object to mutate, and mutate
that. For example:

stuff = [[1, 2, 3], [4, 5, 6]]
stuff.append(stuff[0])
print(stuff)

You now have two references to the same list, inside another list. Any
change to stuff[0] is a change to stuff[2], because they're the exact
same list. When you append "a reference to this list over here" (which
you found by asking for stuff[0]) to the outer list, you get that
list.

That's Python's object model, and trying to cheat it by copying
changes is just going to cause untold nightmares of desynchronization.

ChrisA
-- https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!iORRBJFui8Kn7WzY0ZRPfSdGKcmnDV81UffITsv7ExEAbBXEtv86qC3BOvGaDXCAY708Q4QbDXh0_wo7$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-
list__;!!Cn_UX_p3!iORRBJFui8Kn7WzY0ZRPfSdGKcmnDV81UffITsv7ExEAbBXEtv86qC3BOvGaDXCAY708Q4QbDXh0_wo7$>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Just to add a possibly picky detail to what others have said, Python
does not have an "array" type. It has a "list" type, as well as some
other, not necessarily mutable, sequence types.

If you want to speed up list and matrix operations, you might use NumPy.
Its arrays and matrices are heavily optimized for fast processing and
provide many useful operations on them. No use calling out to C code
yourself when NumPy has been refining that for many years.

On 1/10/2023 4:10 PM, MRAB wrote:

On 2023-01-10 20:41, Jen Kris via Python-list wrote:

Thanks for your comments.  I'd like to make one small point.  You say:

"Assignment in Python is a matter of object references. It's not
"conform them as long as they remain equal". You'll have to think in
terms of object references the entire way."

But where they have been set to the same object, an operation on one
will affect the other as long as they are equal (in Python).  So I
will have to conform them in those cases because Python will reflect
any math operation in both the array and the matrix.

It's not a 2D matrix, it's a 1D list containing references to 1D lists,
each of which contains references to Python ints.

In CPython, references happen to be pointers, but that's just an implementation detail.

Jan 10, 2023, 12:28 by rosuav@gmail.com:

On Wed, 11 Jan 2023 at 07:14, Jen Kris via Python-list
<python-list@python.org> wrote:

I am writing a spot speedup in assembly language for a short but
computation-intensive Python loop, and I discovered something about
Python array handling that I would like to clarify.

For a simplified example, I created a matrix mx1 and assigned the
array arr1 to the third row of the matrix:

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

The pointers to these are now the same:

ida = id(mx1[2]) - 140260325306880
idb = id(arr1) - 140260325306880

That’s great because when I encounter this in assembly or C, I can
just borrow the pointer to row 3 for the array arr1, on the
assumption that they will continue to point to the same object.
Then when I do any math operations in arr1 it will be reflected in
both arrays because they are now pointing to the same array:

That's not an optimization; what you've done is set arr1 to be a
reference to that object.

But on the next iteration we assign arr1 to something else:

arr1 = [ 10, 11, 12 ]
idc = id(arr1) – 140260325308160
idd = id(mx1[2]) – 140260325306880

Now arr1 is no longer equal to mx1[2], and any subsequent operations
in arr1 will not affect mx1.

Yep, you have just set arr1 to be a completely different object.

So where I’m rewriting some Python code in a low level language, I
can’t assume that the two objects are equal because that equality
will not remain if either is reassigned.  So if I do some operation
on one array I have to conform the two arrays for as long as they
remain equal, I can’t just do it in one operation because I can’t
rely on the objects remaining equal.

Is my understanding of this correct?  Is there anything I’m missing? >>>>

Assignment in Python is a matter of object references. It's not
"conform them as long as they remain equal". You'll have to think in
terms of object references the entire way.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, 11 Jan 2023 at 09:08, Thomas Passin <list1@tompassin.net> wrote:

Just to add a possibly picky detail to what others have said, Python
does not have an "array" type. It has a "list" type, as well as some
other, not necessarily mutable, sequence types.

Just to be even pickier, Python DOES have an array type, but it's not
the one the OP was using :)

https://docs.python.org/3/library/array.html

ChrisA

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

There are cases where NumPy would be the best choice, but that wasn’t the case here with what the loop was doing. 

To sum up what I learned from this post, where one object derives from another object (a = b[0], for example), any operation that would alter one will alter the other.  When either is assigned to something else, then they no longer point to the same
memory location and they’re once again independent.   I hope the word "derives" sidesteps the semantic issue of whether they are "equal."   

Thanks to all who replied to this post. 

Jen


Jan 10, 2023, 13:59 by list1@tompassin.net:

Just to add a possibly picky detail to what others have said, Python does not have an "array" type. It has a "list" type, as well as some other, not necessarily mutable, sequence types.

If you want to speed up list and matrix operations, you might use NumPy. Its arrays and matrices are heavily optimized for fast processing and provide many useful operations on them. No use calling out to C code yourself when NumPy has been refining

that for many years.

On 1/10/2023 4:10 PM, MRAB wrote:

On 2023-01-10 20:41, Jen Kris via Python-list wrote:

Thanks for your comments.  I'd like to make one small point.  You say: >>>
"Assignment in Python is a matter of object references. It's not
"conform them as long as they remain equal". You'll have to think in
terms of object references the entire way."

But where they have been set to the same object, an operation on one will affect the other as long as they are equal (in Python).  So I will have to conform them in those cases because Python will reflect any math operation in both the array and the

matrix.

It's not a 2D matrix, it's a 1D list containing references to 1D lists, each of which contains references to Python ints.

In CPython, references happen to be pointers, but that's just an implementation detail.

Jan 10, 2023, 12:28 by rosuav@gmail.com:

On Wed, 11 Jan 2023 at 07:14, Jen Kris via Python-list
<python-list@python.org> wrote:

I am writing a spot speedup in assembly language for a short but computation-intensive Python loop, and I discovered something about Python array handling that I would like to clarify.

For a simplified example, I created a matrix mx1 and assigned the array arr1 to the third row of the matrix:

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

The pointers to these are now the same:

ida = id(mx1[2]) - 140260325306880
idb = id(arr1) - 140260325306880

That’s great because when I encounter this in assembly or C, I can just borrow the pointer to row 3 for the array arr1, on the assumption that they will continue to point to the same object. Then when I do any math operations in arr1 it will be

reflected in both arrays because they are now pointing to the same array:

That's not an optimization; what you've done is set arr1 to be a
reference to that object.

But on the next iteration we assign arr1 to something else:

arr1 = [ 10, 11, 12 ]
idc = id(arr1) – 140260325308160
idd = id(mx1[2]) – 140260325306880

Now arr1 is no longer equal to mx1[2], and any subsequent operations in arr1 will not affect mx1.

Yep, you have just set arr1 to be a completely different object.

So where I’m rewriting some Python code in a low level language, I can’t assume that the two objects are equal because that equality will not remain if either is reassigned.  So if I do some operation on one array I have to conform the two

arrays for as long as they remain equal, I can’t just do it in one operation because I can’t rely on the objects remaining equal.

Is my understanding of this correct?  Is there anything I’m missing? >>>>>

Assignment in Python is a matter of object references. It's not
"conform them as long as they remain equal". You'll have to think in
terms of object references the entire way.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 1/10/2023 5:21 PM, Jen Kris wrote:

There are cases where NumPy would be the best choice, but that wasn’t
the case here with what the loop was doing.

To sum up what I learned from this post, where one object derives from another object (a = b[0], for example), any operation that would alter
one will alter the other.

Let's make sure we're clear here. The way you were doing it, it *looks
like* "one alters the other". But in reality, both are the same thing,
and so when that thing gets altered in some way, both variables will
show the change because they are in fact references to the same object.

As an analogy, if you dye your hair purple and look in a mirror, you
will see your image with the new purple hair. If you look in a
different mirror, you will also see your image with the new purple hair.
They are both reflections of the same object, namely you with your new
purple hair.

The point about the identity of objects contained within other objects
is echoed by the copy and deepcopy operations. copy() copies the
references, deepcopy() makes new objects that are equal to the original
ones. After making a deep copy of a list and assigning it to new
variable, changes in one will no longer show up in the other because the elements are are no longer the same elements.

When either is assigned to something else,
then they no longer point to the same memory location and they’re once again independent.

This is right except that in Python, it's better not to think about
their memory locations, because that would basically be an
implementation detail (well, except that if you are going to access them
with C you will possibly need actual locations). Their logical identity
would be better to think about.

I hope the word "derives" sidesteps the semantic
issue of whether they are "equal."

Thanks to all who replied to this post.

Jen

Jan 10, 2023, 13:59 by list1@tompassin.net:

Just to add a possibly picky detail to what others have said, Python
does not have an "array" type. It has a "list" type, as well as some
other, not necessarily mutable, sequence types.

If you want to speed up list and matrix operations, you might use
NumPy. Its arrays and matrices are heavily optimized for fast
processing and provide many useful operations on them. No use
calling out to C code yourself when NumPy has been refining that for
many years.

On 1/10/2023 4:10 PM, MRAB wrote:

On 2023-01-10 20:41, Jen Kris via Python-list wrote:

Thanks for your comments.  I'd like to make one small
point.  You say:

"Assignment in Python is a matter of object references. It's not
"conform them as long as they remain equal". You'll have to
think in
terms of object references the entire way."

But where they have been set to the same object, an
operation on one will affect the other as long as they are
equal (in Python).  So I will have to conform them in those
cases because Python will reflect any math operation in both
the array and the matrix.

It's not a 2D matrix, it's a 1D list containing references to 1D
lists, each of which contains references to Python ints.

In CPython, references happen to be pointers, but that's just an
implementation detail.

Jan 10, 2023, 12:28 by rosuav@gmail.com:

On Wed, 11 Jan 2023 at 07:14, Jen Kris via Python-list
<python-list@python.org> wrote:

I am writing a spot speedup in assembly language for
a short but computation-intensive Python loop, and I
discovered something about Python array handling
that I would like to clarify.

For a simplified example, I created a matrix mx1 and
assigned the array arr1 to the third row of the matrix:

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

The pointers to these are now the same:

ida = id(mx1[2]) - 140260325306880
idb = id(arr1) - 140260325306880

That’s great because when I encounter this in
assembly or C, I can just borrow the pointer to row
3 for the array arr1, on the assumption that they
will continue to point to the same object. Then when
I do any math operations in arr1 it will be
reflected in both arrays because they are now
pointing to the same array:

That's not an optimization; what you've done is set arr1
to be a
reference to that object.

But on the next iteration we assign arr1 to
something else:

arr1 = [ 10, 11, 12 ]
idc = id(arr1) – 140260325308160
idd = id(mx1[2]) – 140260325306880

Now arr1 is no longer equal to mx1[2], and any
subsequent operations in arr1 will not affect mx1.

Yep, you have just set arr1 to be a completely different
object.

So where I’m rewriting some Python code in a low
level language, I can’t assume that the two objects
are equal because that equality will not remain if
either is reassigned.  So if I do some operation on
one array I have to conform the two arrays for as
long as they remain equal, I can’t just do it in one
operation because I can’t rely on the objects
remaining equal.

Is my understanding of this correct?  Is there
anything I’m missing?

Assignment in Python is a matter of object references.
It's not
"conform them as long as they remain equal". You'll have
to think in
terms of object references the entire way.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 1/10/23 12:03, Jen Kris via Python-list wrote:

I am writing a spot speedup in assembly language for a short but computation-intensive Python
loop, and I discovered something about Python array handling that I would like to clarify.

But on the next iteration we assign arr1 to something else:

arr1 = [ 10, 11, 12 ]
idc = id(arr1) – 140260325308160
idd = id(mx1[2]) – 140260325306880

Now arr1 is no longer equal to mx1[2]...

If you want to have `arr1` to still be `mx1[2]` (and consequently for `mx1[2]` to now be `[10, 11, 12]` you need to
mutate `arr1` instead of reassigning it:

arr1[:] = [10, 11, 12]

--
~Ethan~

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2023-01-10 22:21, Jen Kris via Python-list wrote:

There are cases where NumPy would be the best choice, but that wasn’t the case here with what the loop was doing.

To sum up what I learned from this post, where one object derives from another object (a = b[0], for example), any operation that would alter one will alter the other.  When either is assigned to something else, then they no longer point to the same

memory location and they’re once again independent.   I hope the word "derives" sidesteps the semantic issue of whether they are "equal."

[snip]
In C terms (and in CPython), a 'list' is a resizable array of pointers
to objects, so after "a=b[0]", the name "a" will point to the same
object that b[0] points to. That object might or might not be mutable.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 1/10/2023 5:11 PM, Chris Angelico wrote:

On Wed, 11 Jan 2023 at 09:08, Thomas Passin <list1@tompassin.net> wrote:

Just to add a possibly picky detail to what others have said, Python
does not have an "array" type. It has a "list" type, as well as some
other, not necessarily mutable, sequence types.

Just to be even pickier, Python DOES have an array type, but it's not
the one the OP was using :)

https://docs.python.org/3/library/array.html

Ha! And here all this time I thought you got them from an add-on package.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 11/01/23 11:21 am, Jen Kris wrote:

where one object derives from another object (a = b[0], for example), any operation that would alter one will alter the other.

I think you're still confused. In C terms, after a = b[0], a and b[0]
are pointers to the same block of memory. If you change that block of
memory, then of course you will see the change through either pointer.

Here's a rough C translation of some of your Python code:

/* mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] */
int **mx1 = (int **)malloc(3 * sizeof(int *));
mx1[0] = (int *)malloc(3 * sizeof(int));
mx1[0][0] = 1;
mx1[0][1] = 2;
mx1[0][2] = 3;
mx1[1] = (int *)malloc(3 * sizeof(int));
mx1[1][0] = 4;
mx1[1][1] = 5;
mx1[1][2] = 6;
mx1[2] = (int *)malloc(3 * sizeof(int));
mx1[2][0] = 7;
mx1[2][1] = 8;
mx1[2][2] = 9;

/* arr1 = mx1[2] */
int *arr1 = mx[2];

/* arr1 = [ 10, 11, 12 ] */
arr1 = (int *)malloc(3 * sizeof(int));
arr1[0] = 10;
arr1[1] = 11;
arr1[2] = 12;

Does that help your understanding?

--
Greg

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Yes, I did understand that.  In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they’re the same memory block).  My issue in Python came up because Python can dynamically change one or the
other to a different object (memory block) so I have to be aware of that when handing this kind of situation. 


Jan 10, 2023, 17:31 by greg.ewing@canterbury.ac.nz:

On 11/01/23 11:21 am, Jen Kris wrote:

where one object derives from another object (a = b[0], for example), any operation that would alter one will alter the other.

I think you're still confused. In C terms, after a = b[0], a and b[0]
are pointers to the same block of memory. If you change that block of
memory, then of course you will see the change through either pointer.

Here's a rough C translation of some of your Python code:

/* mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ] */
int **mx1 = (int **)malloc(3 * sizeof(int *));
mx1[0] = (int *)malloc(3 * sizeof(int));
mx1[0][0] = 1;
mx1[0][1] = 2;
mx1[0][2] = 3;
mx1[1] = (int *)malloc(3 * sizeof(int));
mx1[1][0] = 4;
mx1[1][1] = 5;
mx1[1][2] = 6;
mx1[2] = (int *)malloc(3 * sizeof(int));
mx1[2][0] = 7;
mx1[2][1] = 8;
mx1[2][2] = 9;

/* arr1 = mx1[2] */
int *arr1 = mx[2];

/* arr1 = [ 10, 11, 12 ] */
arr1 = (int *)malloc(3 * sizeof(int));
arr1[0] = 10;
arr1[1] = 11;
arr1[2] = 12;

Does that help your understanding?

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list:

Yes, I did understand that.  In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they’re the same memory block).

Sorry if you feel I'm being overly pedantic, but your explanation "an
operation on one is an operation on the other (because they’re the same memory block)" still feels a bit misguided. "One" and "other" still make
it sound like there are two objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations.
Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity
or convenience, sometimes we really need to be precise. I think this is
a case where we need to be precise.

So, to be precise: there is only one object, with possible multiple
names to it. We can change the object, using one of the names. That is
one and only one operation on one and only one object. Since the
different names refer to the same object, that change will of course be
visible through all of them.
Note that 'name' in that sentence doesn't just refer to variables (mx1,
arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...),
loop variables, function arguments.

The correct mental model is important here, and I do think you're on
track or very close to it, but the way you phrase things does give me
that nagging feeling that you still might be just a bit off.

--
"Peace cannot be kept by force. It can only be achieved through understanding."
-- Albert Einstein

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Jen Kris <jenkris@tutanota.com> writes:

Each name has the same pointer

... from the C programmer's point of view.

From the Python programmer's point of view, there are no "pointers".

Instead, names or other targets are /assigned/ or /bound/ to objects.

This is the language of the venerable "Python Language Reference":

|... the name is bound to the object ...
|
|... the sequence is asked to assign the assigned object to
| its item with that index ...
|
section "7.2 Assignment statements"
of "The Python Language Reference".

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Thanks for your comments.  After all, I asked for clarity so it’s not pedantic to be precise, and you’re helping to clarify. 

Going back to my original post,

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer).  Similarly, if I write "mx1[2][1] += 5" then again both names will be updated. 

That’s what I meant by "an operation on one is an operation on the other."  To be more precise, an operation on one name will be reflected in the other name.  The difference is in the names,  not the pointers.  Each name has the same pointer in my
example, but operations can be done in Python using either name. 




Jan 11, 2023, 09:13 by roel@roelschroeven.net:

Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list:

Yes, I did understand that.  In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they’re the same memory block).

Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they’re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects,

and "an operation on one" and "an operation on the other" make it sound like there are two operations.

Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise.

So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that

change will of course be visible through all of them.

Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments.

The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off.

--
"Peace cannot be kept by force. It can only be achieved through understanding."
-- Albert Einstein

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, 10 Jan 2023 16:59:59 -0500, Thomas Passin <list1@tompassin.net> declaimed the following:

Just to add a possibly picky detail to what others have said, Python
does not have an "array" type. It has a "list" type, as well as some
other, not necessarily mutable, sequence types.

However, it has long had https://docs.python.org/3/library/array.html



--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I think the key point is that "the operation" doesn't act on "names" but
on "objects" (which are different sort of things), and thus there isn't
an "the other" when talking about the object being operated on.

Thinking of an operation being on a "name" is the mental model error.
The only operations that operate on "names" are assignment operations.

Augmented assignment operations are more complicatd, as they can either
work on the object the name points on, if that object is mutable, or
rebinds the name to a new object if it isn't.

Thus a += b is NOT neccessarilily the same as a = a + b, as a += b might
just mutate the object that a is bound to or might rebind in the manner
of a = a + b;

Thus:

a = [ 1, 2, 3]
b = a
a += [4]

will change the single list that a and b are bound to into [1, 2, 3, 4],
while

a = "foo"
b = a
a += "bar"

will change a to be bound to the string object "foobar" but not b, since
the string object "foo" wasn't mutable.

Brings up the point, that you need to be careful with augmented
assignment operators.

On 1/11/23 1:28 PM, Jen Kris via Python-list wrote:

Thanks for your comments.  After all, I asked for clarity so it’s not pedantic to be precise, and you’re helping to clarify.

Going back to my original post,

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer).  Similarly, if I write "mx1[2][1] += 5" then again both names will be updated.

That’s what I meant by "an operation on one is an operation on the other."  To be more precise, an operation on one name will be reflected in the other name.  The difference is in the names,  not the pointers.  Each name has the same pointer in

my example, but operations can be done in Python using either name.

Jan 11, 2023, 09:13 by roel@roelschroeven.net:

Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list:

Yes, I did understand that.  In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they’re the same memory block).

Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they’re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two

objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations.

Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise.

So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that

change will of course be visible through all of them.

Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments.

The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off.

--
"Peace cannot be kept by force. It can only be achieved through understanding."
-- Albert Einstein

--
https://mail.python.org/mailman/listinfo/python-list

--
Richard Damon

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2023-01-11 18:49:14 +0000, Stefan Ram wrote:

Jen Kris <jenkris@tutanota.com> writes:

Each name has the same pointer

... from the C programmer's point of view.

From the Python programmer's point of view, there are no "pointers".

That's just window dressing. Pointers are evil, so we can't have
pointers. So we'll just call them by a different name.

Instead, names or other targets are /assigned/ or /bound/ to objects.

This is the wrong way around. The name isn't assigned to the object. The
object is assigned to the name. As you quoted:

|... the sequence is asked to assign the assigned object to
| its item with that index ...

Also, while the word "bind" is used in this manner in the official docs

|... the name is bound to the object ...

and some members of this list, I think it is really misleading. It
sounds like the name is now an attribute of the object. But it isn't.
There is no way to the name (or the names) from the object. But there is
a way to get from the name the object. I can't think of a snappy,
unambiguous single verb to describe what's happening here, but do we
need one? We already have words like "assign", "refer", "point" with
fairly standardized meaning in IT. We can write that "a reference to an
object is assigned to a variable", and that after the assignment "the
variable refers to the object". Or we can even use the P-word. And since
we know that variables in Python can only contain references and never
values, we can abbreviate "a reference to an object" as "an object" in
most contexts.

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAmO/G2kACgkQ8g5IURL+ KF0TNRAAj1h6YUqVWOI28saVkuymZXDLV93/lVz8wNQkwY0yAFxCOcvlTFl9dkC+ n0pf93F64ra4zHpKAELn74hIuEKAPaKjwiX8QF4klljD/VAXqZVcxC0IsbixFJV5 v+FGtvort5g4cSIgrF6X4fRh94aJCoXiE+R5k0iGzb4qN81k98lkhtzuhSPiSJNk v1M4g2GuOFZApDaqVnEzq6MuUgo2ApvObeTOqCHei//WOf0ntxLOaewjRgh9jwMT MmWFErJ5NjVL2rYLGXTbQ+VqwgsSOIHg40NM8XV8wTxHNOKCMHximSiE/A0O4WPB fTSaiW5Ie1pcGPlQhtlRc9tqeQHxmwygH6U6D7Vx81a3R6KfV0bN7oq76+Xxn+FN GnNu4RUnBdq6wKFz0Chwqw/U6t8SEGP64e8sBVvPJvYe5PpD4Fx1Uwipc/niow1K Vv+t/CAUN0n58Wl3Cvq/vrKjrG8KxZrYhdzXJX46YO70a8cOnui2oS/pQ4TGMl+L 0fnvehhNqT9V81xsbjdjKqYwnNbfJ/F3f1HOV0RGKqWrTF9xJx1AdHoAR9ffeKYI 3oWJ9OX/BMHG5bQS3ZwZhW1Gq30xn5OYxhHKtVdMrX3MA98gvO+tHDkIEt3lRNy+ xE0YELqXkw/bBS4Z1t73TbqQ6KCbYYYT5NVbfxP

Jen,

It is dangerous territory you are treading as there are times all or parts of objects are copied, or changed in place or the method you use to make a view is not doing quite what you want.

As an example, you can create a named slice such as:

middle_by_two = slice(5, 10, 2)

The above is not in any sense pointing at anything yet. But given a long enough list or other such objects, it will take items (starting at index 0) starting with item that are at indices 5 then 7 then 9 as in this:

nums = [n for n in range(12)]
nums[middle_by_two]

[5, 7, 9]

The same slice will work on anything else:

list('abcdefghijklmnopqrstuvwxyz')[middle_by_two]
['f', 'h', 'j']

So although you may think the slice is bound to something, it is not. It is an object that only later is briefly connected to whatever you want to apply it to.

If I later change nums, above, like this:

nums = [-3, -2, -1] + nums
nums
[-3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
nums[middle_by_two]
[2, 4, 6]

In the example, you can forget about whether we are talking about pointers directly or indirectly or variable names and so on. Your "view" remains valid ONLY as long as you do not change either the slice or the underlying object you are applying to -- at
least not the items you want to extract.

Since my example inserted three new items at the start using negative numbers for illustration, you would need to adjust the slice by making a new slice designed to fit your new data. The example below created an adjusted slice that adds 3 to the start
and stop settings of the previous slice while copying the step value and then it works on the elongated object:

middle_by_two_adj = slice(middle_by_two.start + 3, middle_by_two.stop + 3, middle_by_two.step)
nums[middle_by_two_adj]
[5, 7, 9]

A suggestion is that whenever you are not absolutely sure that the contents of some data structure might change without your participation, then don't depend on various kinds of aliases to keep the contents synchronized. Make a copy, perhaps a deep
copy and make sure the only thing ever changing it is your code and later, if needed, copy the result back to any other data structure. Of course, if anything else is accessing the result in the original in between, it won't work.

Just FYI, a similar analysis applies to uses of the numpy and pandas and other modules if you get some kind of object holding indices to a series such as integers or Booleans and then later try using it after the number of items or rows or columns have
changed. Your indices no longer match.

Avi

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On Behalf Of Jen Kris via Python-list
Sent: Wednesday, January 11, 2023 1:29 PM
To: Roel Schroeven <roel@roelschroeven.net>
Cc: python-list@python.org
Subject: Re: To clarify how Python handles two equal objects

Thanks for your comments. After all, I asked for clarity so it’s not pedantic to be precise, and you’re helping to clarify.

Going back to my original post,

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then again both names will be updated.

That’s what I meant by "an operation on one is an operation on the other." To be more precise, an operation on one name will be reflected in the other name. The difference is in the names, not the pointers. Each name has the same pointer in my
example, but operations can be done in Python using either name.




Jan 11, 2023, 09:13 by roel@roelschroeven.net:

Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list:

Yes, I did understand that. In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they’re the same memory block).

Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they’re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects,

and "an operation on one" and "an operation on the other" make it sound like there are two operations.

Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise.

So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that

change will of course be visible through all of them.

Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments.

The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off.

--
"Peace cannot be kept by force. It can only be achieved through understanding."
-- Albert Einstein

--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

<avi.e.gross@gmail.com> writes:

As an example, you can create a named slice such as:

middle_by_two = slice(5, 10, 2)

The above is not in any sense pointing at anything yet.

From a functional programming point of view this just looks like a
partially applied function, and with this in mind the behaviour to me
seems to be completely as expected. No surprises here, or do I miss
something?

Best regards

Axel

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Axel Reichert <mail@axel-reichert.de> writes:

<avi.e.gross@gmail.com> writes:

As an example, you can create a named slice such as:
middle_by_two = slice(5, 10, 2)
The above is not in any sense pointing at anything yet.

From a functional programming point of view this just looks like a
partially applied function,

A partially applied function is a partially applied function
from any point of view.

But "slice" has only three parameters, and in "slice( 5, 10, 2 )",
there are three arguments. So, one might as well say that
this is a /totally/ applied function since for each parameter
an argument value is given.

But what "partial" also does is to delay the evaluation of
the application object until it is called, even if immediate
evaluation would be possible. So one might call it a
"delayed application" or "separated evaluation".

main.py

import functools
import math

call = functools.partial( math.cos, 0 )
print( call )
product = call()
print( product )

output

functools.partial(<built-in function cos>, 0)
1.0

One also could write a class for this.

main.py

import math

class cos:
def __init__( self, argumentvalue ):
self.argumentvalue = argumentvalue
def value( self ):
if 'val' not in dir( self ):
self.val = math.cos( self.argumentvalue )
return self.val
def __str__( self ):
return f'application of cos to {self.argumentvalue}'

call = cos( 0 )
print( call )
product = call.value()
print( product )

output

application of cos to 0
1.0

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Avi,

Thanks for your comments.  You make a good point. 

Going back to my original question, and using your slice() example: 

middle_by_two = slice(5, 10, 2)
nums = [n for n in range(12)]
q = nums[middle_by_two]
x = id(q)
b = q
y = id(b)

If I assign "b" to "q", then x and y match – they point to the same memory until "b" OR "q" are  reassigned to something else.  If "q" changes during the lifetime of "b" then it’s not safe to use the pointer to "q" for "b", as in:

nums = [n for n in range(2, 14)]
q = nums[middle_by_two]
x = id(q)
y = id(b)

Now "x" and "y" are different, as we would expect.  So when writing a spot speed up in a compiled language, you can see in the Python source if either is reassigned, so you’ll know how to handle it.  The motivation behind my question was that in a
compiled extension it’s faster to borrow a pointer than to move an entire array if it’s possible, but special care must be taken. 

Jen



Jan 12, 2023, 20:51 by avi.e.gross@gmail.com:

Jen,

It is dangerous territory you are treading as there are times all or parts of objects are copied, or changed in place or the method you use to make a view is not doing quite what you want.

As an example, you can create a named slice such as:

middle_by_two = slice(5, 10, 2)

The above is not in any sense pointing at anything yet. But given a long enough list or other such objects, it will take items (starting at index 0) starting with item that are at indices 5 then 7 then 9 as in this:

nums = [n for n in range(12)]
nums[middle_by_two]

[5, 7, 9]

The same slice will work on anything else:

list('abcdefghijklmnopqrstuvwxyz')[middle_by_two]
['f', 'h', 'j']

So although you may think the slice is bound to something, it is not. It is an object that only later is briefly connected to whatever you want to apply it to.

If I later change nums, above, like this:

nums = [-3, -2, -1] + nums
nums
[-3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
nums[middle_by_two]
[2, 4, 6]

In the example, you can forget about whether we are talking about pointers directly or indirectly or variable names and so on. Your "view" remains valid ONLY as long as you do not change either the slice or the underlying object you are applying to --

at least not the items you want to extract.

Since my example inserted three new items at the start using negative numbers for illustration, you would need to adjust the slice by making a new slice designed to fit your new data. The example below created an adjusted slice that adds 3 to the start

and stop settings of the previous slice while copying the step value and then it works on the elongated object:

middle_by_two_adj = slice(middle_by_two.start + 3, middle_by_two.stop + 3, middle_by_two.step)
nums[middle_by_two_adj]
[5, 7, 9]

A suggestion is that whenever you are not absolutely sure that the contents of some data structure might change without your participation, then don't depend on various kinds of aliases to keep the contents synchronized. Make a copy, perhaps a deep

copy and make sure the only thing ever changing it is your code and later, if needed, copy the result back to any other data structure. Of course, if anything else is accessing the result in the original in between, it won't work.

Just FYI, a similar analysis applies to uses of the numpy and pandas and other modules if you get some kind of object holding indices to a series such as integers or Booleans and then later try using it after the number of items or rows or columns have

changed. Your indices no longer match.

Avi

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On Behalf Of Jen Kris via Python-list
Sent: Wednesday, January 11, 2023 1:29 PM
To: Roel Schroeven <roel@roelschroeven.net>
Cc: python-list@python.org
Subject: Re: To clarify how Python handles two equal objects

Thanks for your comments. After all, I asked for clarity so it’s not pedantic to be precise, and you’re helping to clarify.

Going back to my original post,

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then again both names will be updated.

That’s what I meant by "an operation on one is an operation on the other." To be more precise, an operation on one name will be reflected in the other name. The difference is in the names, not the pointers. Each name has the same pointer in my

example, but operations can be done in Python using either name.

Jan 11, 2023, 09:13 by roel@roelschroeven.net:

Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list:

Yes, I did understand that. In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they’re the same memory block).

Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they’re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two

objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations.

Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise.

So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that

change will of course be visible through all of them.

Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments.

The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off.

--
"Peace cannot be kept by force. It can only be achieved through understanding."
-- Albert Einstein

--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

It seems to me that the the entire concept of relying on python's idea of
where an object is stored is just plain dangerous. A most simple example
might be:
>>> a=1
>>> b=1
>>> a is b
True
>>> a=1234
>>> b=1234
>>> a is b
False

Not sure what happens if you manipulate the data referenced by 'b' in the
first example thinking you are changing something referred to by 'a' ...
but you might be smart to NOT think that you know.



On Fri, Jan 13, 2023 at 9:00 AM Jen Kris via Python-list < python-list@python.org> wrote:

Avi,

Thanks for your comments. You make a good point.

Going back to my original question, and using your slice() example:

middle_by_two = slice(5, 10, 2)
nums = [n for n in range(12)]
q = nums[middle_by_two]
x = id(q)
b = q
y = id(b)

If I assign "b" to "q", then x and y match – they point to the same memory until "b" OR "q" are reassigned to something else. If "q" changes during the lifetime of "b" then it’s not safe to use the pointer to "q" for "b", as in:

nums = [n for n in range(2, 14)]
q = nums[middle_by_two]
x = id(q)
y = id(b)

Now "x" and "y" are different, as we would expect. So when writing a spot speed up in a compiled language, you can see in the Python source if either is reassigned, so you’ll know how to handle it. The motivation behind my question was that in a compiled extension it’s faster to borrow a pointer than to move an entire array if it’s possible, but special care must be taken.

Jen

Jan 12, 2023, 20:51 by avi.e.gross@gmail.com:

Jen,

It is dangerous territory you are treading as there are times all or

parts of objects are copied, or changed in place or the method you use to make a view is not doing quite what you want.

As an example, you can create a named slice such as:

middle_by_two = slice(5, 10, 2)

The above is not in any sense pointing at anything yet. But given a long

enough list or other such objects, it will take items (starting at index 0) starting with item that are at indices 5 then 7 then 9 as in this:

nums = [n for n in range(12)]
nums[middle_by_two]

[5, 7, 9]

The same slice will work on anything else:

list('abcdefghijklmnopqrstuvwxyz')[middle_by_two]
['f', 'h', 'j']

So although you may think the slice is bound to something, it is not. It

is an object that only later is briefly connected to whatever you want to apply it to.

If I later change nums, above, like this:

nums = [-3, -2, -1] + nums
nums
[-3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
nums[middle_by_two]
[2, 4, 6]

In the example, you can forget about whether we are talking about

pointers directly or indirectly or variable names and so on. Your "view" remains valid ONLY as long as you do not change either the slice or the underlying object you are applying to -- at least not the items you want to extract.

Since my example inserted three new items at the start using negative

numbers for illustration, you would need to adjust the slice by making a
new slice designed to fit your new data. The example below created an adjusted slice that adds 3 to the start and stop settings of the previous slice while copying the step value and then it works on the elongated
object:

middle_by_two_adj = slice(middle_by_two.start + 3, middle_by_two.stop +

3, middle_by_two.step)

nums[middle_by_two_adj]
[5, 7, 9]

A suggestion is that whenever you are not absolutely sure that the

contents of some data structure might change without your participation,
then don't depend on various kinds of aliases to keep the contents synchronized. Make a copy, perhaps a deep copy and make sure the only
thing ever changing it is your code and later, if needed, copy the result back to any other data structure. Of course, if anything else is accessing the result in the original in between, it won't work.

Just FYI, a similar analysis applies to uses of the numpy and pandas and

other modules if you get some kind of object holding indices to a series
such as integers or Booleans and then later try using it after the number
of items or rows or columns have changed. Your indices no longer match.

Avi

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org>

On Behalf Of Jen Kris via Python-list

Sent: Wednesday, January 11, 2023 1:29 PM
To: Roel Schroeven <roel@roelschroeven.net>
Cc: python-list@python.org
Subject: Re: To clarify how Python handles two equal objects

Thanks for your comments. After all, I asked for clarity so it’s not

pedantic to be precise, and you’re helping to clarify.

Going back to my original post,

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be

changed because while they are different names, they are the assigned same memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then
again both names will be updated.

That’s what I meant by "an operation on one is an operation on the

other." To be more precise, an operation on one name will be reflected in the other name. The difference is in the names, not the pointers. Each name has the same pointer in my example, but operations can be done in
Python using either name.

Jan 11, 2023, 09:13 by roel@roelschroeven.net:

Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list:

Yes, I did understand that. In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they’re the same memory block).

Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they’re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects, and "an operation on one" and "an

operation on the other" make it sound like there are two operations.

Sometimes it doesn't matter if we're a bit sloppy for sake of

simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise.

So, to be precise: there is only one object, with possible multiple

names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different
names refer to the same object, that change will of course be visible
through all of them.

Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...),

loop variables, function arguments.

The correct mental model is important here, and I do think you're on

track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off.

--
"Peace cannot be kept by force. It can only be achieved through understanding."
-- Albert Einstein

--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

--

**** Listen to my FREE CD at http://www.mellowood.ca/music/cedars ****
Bob van der Poel ** Wynndel, British Columbia, CANADA **
EMAIL: bob@mellowood.ca
WWW: http://www.mellowood.ca

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Bob van der Poel <bob@mellowood.ca> writes:

a=1234
b=1234
a is b

False
Not sure what happens if you manipulate the data referenced by 'b'

One example executed with one version of CPython:

import ctypes

j = 65

def deref( addr, typ ):
return ctypes.cast( addr, ctypes.POINTER( typ ))

deref( id( 2 ), ctypes.c_int )[ 6 ] = j

print( 2 ) # prints "65"

.

... and now, there's the word "POINTER"!

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Bob,

Your examples show a and b separately defined.  My example is where the definition is a=1; b = a.  But I'm only interested in arrays.  I would not rely on this for integers, and there's not likely to be any real cost savings there.  


Jan 13, 2023, 08:45 by bob@mellowood.ca:

It seems to me that the the entire concept of relying on python's idea of where an object is stored is just plain dangerous. A most simple example might be:
   >>> a=1
   >>> b=1
   >>> a is b
  True
  >>> a=1234
  >>> b=1234
  >>> a is b
  False

Not sure what happens if you manipulate the data referenced by 'b' in the first example thinking you are changing something referred to by 'a' ... but you might be smart to NOT think that you know.

On Fri, Jan 13, 2023 at 9:00 AM Jen Kris via Python-list <> python-list@python.org> > wrote:

Avi,

Thanks for your comments.  You make a good point. 

Going back to my original question, and using your slice() example: 

middle_by_two = slice(5, 10, 2)
nums = [n for n in range(12)]
q = nums[middle_by_two]
x = id(q)
b = q
y = id(b)

If I assign "b" to "q", then x and y match – they point to the same memory until "b" OR "q" are  reassigned to something else.  If "q" changes during the lifetime of "b" then it’s not safe to use the pointer to "q" for "b", as in:

nums = [n for n in range(2, 14)]
q = nums[middle_by_two]
x = id(q)
y = id(b)

Now "x" and "y" are different, as we would expect.  So when writing a spot speed up in a compiled language, you can see in the Python source if either is reassigned, so you’ll know how to handle it.  The motivation behind my question was that in

a compiled extension it’s faster to borrow a pointer than to move an entire array if it’s possible, but special care must be taken. 

Jen



Jan 12, 2023, 20:51 by >> avi.e.gross@gmail.com>> :

Jen,

It is dangerous territory you are treading as there are times all or parts of objects are copied, or changed in place or the method you use to make a view is not doing quite what you want.

As an example, you can create a named slice such as:

  middle_by_two = slice(5, 10, 2)

The above is not in any sense pointing at anything yet. But given a long enough list or other such objects, it will take items (starting at index 0) starting with item that are at indices 5 then 7 then 9  as in this:

  nums = [n for n in range(12)]
  nums[middle_by_two]

[5, 7, 9]

The same slice will work on anything else:

  list('abcdefghijklmnopqrstuvwxyz')[middle_by_two]
['f', 'h', 'j']

So although you may think the slice is bound to something, it is not. It is an object that only later is briefly connected to whatever you want to apply it to.

If I later change nums, above, like this:

  nums = [-3, -2, -1] + nums
  nums
[-3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
  nums[middle_by_two]
[2, 4, 6]

In the example, you can forget about whether we are talking about pointers directly or indirectly or variable names and so on. Your "view" remains valid ONLY as long as you do not change either the slice or the underlying object you are applying to

-- at least not the items you want to extract.

Since my example inserted three new items at the start using negative numbers for illustration, you would need to adjust the slice by making a new slice designed to fit your new data. The example below created an adjusted slice that adds 3 to the

start and stop settings of the previous slice while copying the step value and then it works on the elongated object:

  middle_by_two_adj = slice(middle_by_two.start + 3, middle_by_two.stop + 3, middle_by_two.step)
  nums[middle_by_two_adj]
[5, 7, 9]

A suggestion is  that whenever you are not absolutely sure that the contents of some data structure might change without your participation, then don't depend on various kinds of aliases to keep the contents synchronized. Make a copy, perhaps  a

deep copy and make sure the only thing ever changing it is your code and later, if needed, copy the result back to any other data structure. Of course, if anything else is accessing the result in the original in between, it won't work.

Just FYI, a similar analysis applies to uses of the numpy and pandas and other modules if you get some kind of object holding indices to a series such as integers or Booleans and then later try using it after the number of items or rows or columns

have changed. Your indices no longer match.

Avi

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=>> gmail.com@python.org>> > On Behalf Of Jen Kris via Python-list
Sent: Wednesday, January 11, 2023 1:29 PM
To: Roel Schroeven <>> roel@roelschroeven.net>> >
Cc: >> python-list@python.org
Subject: Re: To clarify how Python handles two equal objects

Thanks for your comments.  After all, I asked for clarity so it’s not pedantic to be precise, and you’re helping to clarify.

Going back to my original post,

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer).  Similarly, if I write "mx1[2][1] += 5" then again both names will be updated.

That’s what I meant by "an operation on one is an operation on the other."  To be more precise, an operation on one name will be reflected in the other name.  The difference is in the names,  not the pointers.  Each name has the same pointer

in my example, but operations can be done in Python using either name.

Jan 11, 2023, 09:13 by >> roel@roelschroeven.net>> :

Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list:

Yes, I did understand that.  In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they’re the same memory block).

Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they’re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two

objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations.

Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise.

So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object,

that change will of course be visible through all of them.

Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments.

The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off.

--
"Peace cannot be kept by force. It can only be achieved through understanding."
  -- Albert Einstein

--

https://mail.python.org/mailman/listinfo/python-list

--

https://mail.python.org/mailman/listinfo/python-list

--

https://mail.python.org/mailman/listinfo/python-list

--

**** Listen to my FREE CD at > http://www.mellowood.ca/music/cedars> **** Bob van der Poel ** Wynndel, British Columbia, CANADA **
EMAIL: > bob@mellowood.ca
WWW:   > http://www.mellowood.ca

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2023-01-13 16:57:45 +0100, Jen Kris via Python-list wrote:

Thanks for your comments.  You make a good point. 

Going back to my original question, and using your slice() example: 

middle_by_two = slice(5, 10, 2)
nums = [n for n in range(12)]
q = nums[middle_by_two]
x = id(q)
b = q
y = id(b)

If I assign "b" to "q",

You don't asssign b to q. You assign q to b. Assignment is not
commutative, the direction matters.

then x and y match – they point to the same memory until "b" OR "q"
are  reassigned to something else.

Correct.

  If "q" changes during the lifetime of "b" then it’s not safe to use
the pointer to "q" for "b", as in:

There is no pointer to q[1]. q is a pointer to something (an object of type list with 3 elements).

b is a pointer to the same object at this point. Of course, if you
assign a different pointer to q, then q will point to the new object
from that point on while b will continue to point to the original object
(until it is also re-assigned).

nums = [n for n in range(2, 14)]
q = nums[middle_by_two]
x = id(q)
y = id(b)

Now "x" and "y" are different, as we would expect.
So when writing a spot speed up in a compiled language,

If you are writing a Python compiler you certainly will have to adjust
for the fact that the same variable may hold pointers to very different
objects over its lifetime. But as far as I understand it, you aren't
trying to write a Python compiler, just an extension, so that shouldn't
concern you.

The motivation behind my question was that in a compiled extension
it’s faster to borrow a pointer than to move an entire array if it’s possible, but special care must be taken. 

I still don't understand what you mean by that.

hp

[1] Not quite true. The run-time system must keep track of the
variables, so there likely is a pointer to q somewhere. But it's (IMHO) irrelevant to this discussion, unless the extension is trying to look up variables by name or something like that.

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAmPBs0MACgkQ8g5IURL+ KF0gPQ/+KUNaKGqxKRU+vINP7Mn07Qe+7YxjKDAljheEuPgP/ys0FW9d1UNyr8mn y8/bejUHhfxFwtM0WKsh47x9g2mjntRZBFfTexkhNiwdlrA7AJubfIfDkIlAFvGm YVk8ImyfxIh/9hIrBu/toVXO8GW52UgGufloPhExLjPkuZIvVxFMvLxMl6ttMqfV TtAOmfZrx9KGq+3qohL/sv/9tDo/z/s66jmu+N2wegTNHGTUdLhAJbE2wOGhdZwC x1sOji5SdVXPiGdNsfjxQ9yYtzZAzgKyENkG1aQwtPSyHtOQqiSMyG8KLKVUKwwy PNiJ50ociGVdRyt7B32zWVntVc7eWD025ImkmamFUmSQf5ZOPThsBvV6U9xX9/jd rY/Xcc0WUXvdl/NPOig4MbJP3d4pefC02lQtyVnFUr5NGuiCHU1Ce6FAXLRv/X10 +KZBE3q27UNCepxntMqetjhPJa2bLlFXJb/iQyz1WEndsYnk84pxoZZCnvgDlRkK z1jJ0f/knNUp426owIz9KMPYZadTU58/adN+YsGDWNmir22lQvAEpgpBv+ptdIi9 LtDb1iVzBCIoQnSzJZ88ruCGls6kUwhOnCE3LbnlZeVGNbStth0wSLwQobZRBqlE 9vLdjtNNi2oTjsQ0OI9qCPR2fiv1c84zzTtbMbv

Axel and others,

I can appreciate the comparison to a partially applied function but not in
this case. Not that it matters, but this example is more like creating an object in something like machine learning and initializing parameters
without adding data. Only when you ad data and call upon some transforms and
so on, does it do something.

This case is even more general. You create an object that does NOTHING. It simply holds a start/end/step set of up to three values. Lots of other functions will take this object as an argument. It can be used and reused
any number of times. Strictly speaking, code like name[5:10:1] just creates
a transient slice object and then uses that to get the answer. It is not delayed or partial as much as making one does nothing.

Stefan mentioned functools.partial and that does create a bit of a curried function that wraps the data and holds on to it so invoking it sort of wakes the function up, with some or all data already accessible. A slice does not
do that and needs some other functionality to use IT alongside whatever
object you want to see a slice of.

No special behavior was intended by me. I was illustrating how some methods
of providing a selected view of an object are equally sensitive to the underlying data changing. A partially applied function that still takes an argument later, would have a similar problem if underlying data outside the what is stored within the function, changed, or if the saved was a reference
to something that changed.

But this is really far from unique. In the example given of creating a
partial call, what if you made a second copy to that call then the first variable to the partial function was re-defined.

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On Behalf Of Axel Reichert
Sent: Friday, January 13, 2023 3:22 AM
To: python-list@python.org
Subject: Re: To clarify how Python handles two equal objects

<avi.e.gross@gmail.com> writes:

As an example, you can create a named slice such as:

middle_by_two = slice(5, 10, 2)

The above is not in any sense pointing at anything yet.

From a functional programming point of view this just looks like a partially applied function, and with this in mind the behaviour to me seems to be completely as expected. No surprises here, or do I miss something?

Best regards

Axel
--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Jen,



Can a compiler, or spot compiler, always know if something that was a reference is changed?



Obvious examples may be if a change happens in non-deterministic ways such as in a fork chosen at random or from user input but also sometimes levels of indirection such as deleting an object that internally contains a reference the other, perhaps even
more indirectly.



I know programmers often have their code overhauled and their assumptions go away such as someone deciding to create a variable name in an inner scope and thus hiding the same variable they were using in an outer scope. So even if your code is currently
valid, after it changes, a later compiler if it detected some change might not want to do your speedup.



I think the subject line of the message we keep exchanging is now a bit misleading. It is not about two objects nor really about how python handles them. There seem to be one object and possibly multiple views of it and you may not want to pass the
entire object around or manipulate it a certain way. I am not so certain your methods necessarily speed things up as certain views simply do calculations on the many places they need to read or change to supply what you want.



From: Jen Kris <jenkris@tutanota.com>
Sent: Friday, January 13, 2023 10:58 AM
To: avi.e.gross@gmail.com
Cc: python-list@python.org
Subject: RE: To clarify how Python handles two equal objects





Avi,



Thanks for your comments. You make a good point.



Going back to my original question, and using your slice() example:



middle_by_two = slice(5, 10, 2)

nums = [n for n in range(12)]

q = nums[middle_by_two]

x = id(q)

b = q

y = id(b)



If I assign "b" to "q", then x and y match – they point to the same memory until "b" OR "q" are reassigned to something else. If "q" changes during the lifetime of "b" then it’s not safe to use the pointer to "q" for "b", as in:



nums = [n for n in range(2, 14)]

q = nums[middle_by_two]

x = id(q)

y = id(b)



Now "x" and "y" are different, as we would expect. So when writing a spot speed up in a compiled language, you can see in the Python source if either is reassigned, so you’ll know how to handle it. The motivation behind my question was that in a
compiled extension it’s faster to borrow a pointer than to move an entire array if it’s possible, but special care must be taken.



Jen







Jan 12, 2023, 20:51 by avi.e.gross@gmail.com <mailto:avi.e.gross@gmail.com> :

Jen,



It is dangerous territory you are treading as there are times all or parts of objects are copied, or changed in place or the method you use to make a view is not doing quite what you want.



As an example, you can create a named slice such as:



middle_by_two = slice(5, 10, 2)



The above is not in any sense pointing at anything yet. But given a long enough list or other such objects, it will take items (starting at index 0) starting with item that are at indices 5 then 7 then 9 as in this:



nums = [n for n in range(12)]

nums[middle_by_two]



[5, 7, 9]



The same slice will work on anything else:



list('abcdefghijklmnopqrstuvwxyz')[middle_by_two]

['f', 'h', 'j']



So although you may think the slice is bound to something, it is not. It is an object that only later is briefly connected to whatever you want to apply it to.



If I later change nums, above, like this:



nums = [-3, -2, -1] + nums

nums

[-3, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

nums[middle_by_two]

[2, 4, 6]



In the example, you can forget about whether we are talking about pointers directly or indirectly or variable names and so on. Your "view" remains valid ONLY as long as you do not change either the slice or the underlying object you are applying to -- at
least not the items you want to extract.



Since my example inserted three new items at the start using negative numbers for illustration, you would need to adjust the slice by making a new slice designed to fit your new data. The example below created an adjusted slice that adds 3 to the start
and stop settings of the previous slice while copying the step value and then it works on the elongated object:



middle_by_two_adj = slice(middle_by_two.start + 3, middle_by_two.stop + 3, middle_by_two.step)

nums[middle_by_two_adj]

[5, 7, 9]



A suggestion is that whenever you are not absolutely sure that the contents of some data structure might change without your participation, then don't depend on various kinds of aliases to keep the contents synchronized. Make a copy, perhaps a deep copy
and make sure the only thing ever changing it is your code and later, if needed, copy the result back to any other data structure. Of course, if anything else is accessing the result in the original in between, it won't work.



Just FYI, a similar analysis applies to uses of the numpy and pandas and other modules if you get some kind of object holding indices to a series such as integers or Booleans and then later try using it after the number of items or rows or columns have
changed. Your indices no longer match.



Avi



-----Original Message-----

From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org <mailto:python-list-bounces+avi.e.gross=gmail.com@python.org> > On Behalf Of Jen Kris via Python-list

Sent: Wednesday, January 11, 2023 1:29 PM

To: Roel Schroeven <roel@roelschroeven.net <mailto:roel@roelschroeven.net> >

Cc: python-list@python.org <mailto:python-list@python.org>

Subject: Re: To clarify how Python handles two equal objects



Thanks for your comments. After all, I asked for clarity so it’s not pedantic to be precise, and you’re helping to clarify.



Going back to my original post,



mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]

arr1 = mx1[2]



Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then again both names will be updated.



That’s what I meant by "an operation on one is an operation on the other." To be more precise, an operation on one name will be reflected in the other name. The difference is in the names, not the pointers. Each name has the same pointer in my example,
but operations can be done in Python using either name.









Jan 11, 2023, 09:13 by roel@roelschroeven.net <mailto:roel@roelschroeven.net> :

Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list:

Yes, I did understand that. In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they’re the same memory block).



Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they’re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects,
and "an operation on one" and "an operation on the other" make it sound like there are two operations.

Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise.



So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that
change will of course be visible through all of them.

Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments.



The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off.



--

"Peace cannot be kept by force. It can only be achieved through understanding."

-- Albert Einstein



--

https://mail.python.org/mailman/listinfo/python-list



--

https://mail.python.org/mailman/listinfo/python-list



--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Jen,

This may not be on target but I was wondering about your needs in this category. Are all your data in a form where all in a cluster are the same object type, such as floating point?

Python has features designed to allow you to get multiple views on such objects such as memoryview that can be used to say see an array as a matrix of n rows by m columns, or m x n, or any other combo. And of course the fuller numpy package has quite a
few features.

However, as you note, there is no guarantee that any reference to the data may not shift away from it unless you build fairly convoluted logic or data structures such as having an object that arranges to do something when you try to remove it, such as
tinkering with the __del__ method as well as whatever method is used to try to set it to a new value. I guess that might make sense for something like asynchronous programming including when setting locks so multiple things cannot overlap when being done.

Anyway, some of the packages like numpy are optimized in many ways but if you want to pass a subset of sorts to make processing faster, I suspect you could do things like pass a memoryview but it might not be faster than what you build albeit probably
more reliable and portable.

I note another odd idea that others may have mentioned, with caution.

If you load the sys module, you can CAREFULLY use code like this.

a="Something Unique"
sys.getrefcount(a)
2

Note if a==1 you will get some huge number of references and this is meaningless. The 2 above is because asking about how many references also references it.

So save what ever number you have and see what happens when you make a second reference or a third, and what happens if you delete or alter a reference:

a="Something Unique"
sys.getrefcount(a)
2
b = a
sys.getrefcount(a)
3
sys.getrefcount(b)
3
c = b
d = a
sys.getrefcount(a)
5
sys.getrefcount(d)
5
del(a)
sys.getrefcount(d)
4
b = "something else"
sys.getrefcount(d)
3

So, in theory, you could carefully write your code to CHECK the reference count had not changed but there remain edge cases where a removed reference is replaced by yet another new reference and you would have no idea.

Avi


-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On Behalf Of Jen Kris via Python-list
Sent: Wednesday, January 11, 2023 1:29 PM
To: Roel Schroeven <roel@roelschroeven.net>
Cc: python-list@python.org
Subject: Re: To clarify how Python handles two equal objects

Thanks for your comments. After all, I asked for clarity so it’s not pedantic to be precise, and you’re helping to clarify.

Going back to my original post,

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then again both names will be updated.

That’s what I meant by "an operation on one is an operation on the other." To be more precise, an operation on one name will be reflected in the other name. The difference is in the names, not the pointers. Each name has the same pointer in my
example, but operations can be done in Python using either name.




Jan 11, 2023, 09:13 by roel@roelschroeven.net:

Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list:

Yes, I did understand that. In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they’re the same memory block).

Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they’re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two objects,

and "an operation on one" and "an operation on the other" make it sound like there are two operations.

Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise.

So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that

change will of course be visible through all of them.

Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments.

The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off.

--
"Peace cannot be kept by force. It can only be achieved through understanding."
-- Albert Einstein

--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun, 15 Jan 2023 at 10:32, Jen Kris via Python-list
<python-list@python.org> wrote:

The situation I described in my original post is limited to a case such as x = y ... the assignment can be done simply by "x" taking the pointer to "y" rather than moving all the data from "y" into the memory buffer for "x"

It's not simply whether it *can* be done. It, in fact, *MUST* be done
that way. The ONLY meaning of "x = y" is that you now have a name "x"
which refers to whatever object is currently found under the name "y".
This is not an optimization, it is a fundamental of Python's object
model. This is true regardless of what kind of object this is; every
object must behave this way.

ChrisA

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Avi,

Your comments go farther afield than my original question, but you made some interesting additional points.  For example, I sometimes work with the C API and sys.getrefcount may be helpful in deciding when to INCREF and DECREF.  But that’s another
issue. 

The situation I described in my original post is limited to a case such as x = y where both "x" and "y" are arrays – whether they are lists in Python, or from the array module – and the question in a compiled C extension is whether the assignment can
be done simply by "x" taking the pointer to "y" rather than moving all the data from "y" into the memory buffer for "x" which, for a wide array, would be much more time consuming than just moving a pointer.  The other advantage to doing it that way is
if, as in my case, we perform a math operation on any element in "x" then Python expects that the same change to be reflected in "y."  If I don’t use the same pointers then I would have to perform that operation twice – once for "x" and once  for "
y" – in addition to the expense of moving all the data. 

The answers I got from this post confirmed that it I can use the pointer if "y" is not re-defined to something else during the lifespan of "x."  If it is then "x" has to be restored to its original pointer.  I did it that way, and helpfully the
compiler did not overrule me. 


Jan 13, 2023, 18:41 by avi.e.gross@gmail.com:

Jen,

This may not be on target but I was wondering about your needs in this category. Are all your data in a form where all in a cluster are the same object type, such as floating point?

Python has features designed to allow you to get multiple views on such objects such as memoryview that can be used to say see an array as a matrix of n rows by m columns, or m x n, or any other combo. And of course the fuller numpy package has quite a

few features.

However, as you note, there is no guarantee that any reference to the data may not shift away from it unless you build fairly convoluted logic or data structures such as having an object that arranges to do something when you try to remove it, such as

tinkering with the __del__ method as well as whatever method is used to try to set it to a new value. I guess that might make sense for something like asynchronous programming including when setting locks so multiple things cannot overlap when being done.

Anyway, some of the packages like numpy are optimized in many ways but if you want to pass a subset of sorts to make processing faster, I suspect you could do things like pass a memoryview but it might not be faster than what you build albeit probably

more reliable and portable.

I note another odd idea that others may have mentioned, with caution.

If you load the sys module, you can CAREFULLY use code like this.

a="Something Unique"
sys.getrefcount(a)
2

Note if a==1 you will get some huge number of references and this is meaningless. The 2 above is because asking about how many references also references it.

So save what ever number you have and see what happens when you make a second reference or a third, and what happens if you delete or alter a reference:

a="Something Unique"
sys.getrefcount(a)
2
b = a
sys.getrefcount(a)
3
sys.getrefcount(b)
3
c = b
d = a
sys.getrefcount(a)
5
sys.getrefcount(d)
5
del(a)
sys.getrefcount(d)
4
b = "something else"
sys.getrefcount(d)
3

So, in theory, you could carefully write your code to CHECK the reference count had not changed but there remain edge cases where a removed reference is replaced by yet another new reference and you would have no idea.

Avi

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On Behalf Of Jen Kris via Python-list
Sent: Wednesday, January 11, 2023 1:29 PM
To: Roel Schroeven <roel@roelschroeven.net>
Cc: python-list@python.org
Subject: Re: To clarify how Python handles two equal objects

Thanks for your comments. After all, I asked for clarity so it’s not pedantic to be precise, and you’re helping to clarify.

Going back to my original post,

mx1 = [ [ 1, 2, 3 ], [ 4, 5, 6 ], [ 7, 8, 9 ] ]
arr1 = mx1[2]

Now if I write "arr1[1] += 5" then both arr1 and mx1[2][1] will be changed because while they are different names, they are the assigned same memory location (pointer). Similarly, if I write "mx1[2][1] += 5" then again both names will be updated.

That’s what I meant by "an operation on one is an operation on the other." To be more precise, an operation on one name will be reflected in the other name. The difference is in the names, not the pointers. Each name has the same pointer in my

example, but operations can be done in Python using either name.

Jan 11, 2023, 09:13 by roel@roelschroeven.net:

Op 11/01/2023 om 16:33 schreef Jen Kris via Python-list:

Yes, I did understand that. In your example, "a" and "b" are the same pointer, so an operation on one is an operation on the other (because they’re the same memory block).

Sorry if you feel I'm being overly pedantic, but your explanation "an operation on one is an operation on the other (because they’re the same memory block)" still feels a bit misguided. "One" and "other" still make it sound like there are two

objects, and "an operation on one" and "an operation on the other" make it sound like there are two operations.

Sometimes it doesn't matter if we're a bit sloppy for sake of simplicity or convenience, sometimes we really need to be precise. I think this is a case where we need to be precise.

So, to be precise: there is only one object, with possible multiple names to it. We can change the object, using one of the names. That is one and only one operation on one and only one object. Since the different names refer to the same object, that

change will of course be visible through all of them.

Note that 'name' in that sentence doesn't just refer to variables (mx1, arr1, ...) but also things like indexed lists (mx1[0], mx1[[0][0], ...), loop variables, function arguments.

The correct mental model is important here, and I do think you're on track or very close to it, but the way you phrase things does give me that nagging feeling that you still might be just a bit off.

--
"Peace cannot be kept by force. It can only be achieved through understanding."
-- Albert Einstein

--
https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun, 15 Jan 2023 at 11:38, Jen Kris <jenkris@tutanota.com> wrote:

Yes, in fact I asked my original question – "I discovered something about Python array handling that I would like to clarify" -- because I saw that Python did it that way.

Yep. This is not specific to arrays; it is true of all Python objects.
Also, I suspect you're still thinking about things backwards, and am
trying to lead you to a completely different way of thinking that
actually does align with Python's object model.

ChrisA

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Yes, in fact I asked my original question – "I discovered something about Python array handling that I would like to clarify" -- because I saw that Python did it that way. 



Jan 14, 2023, 15:51 by rosuav@gmail.com:

On Sun, 15 Jan 2023 at 10:32, Jen Kris via Python-list <python-list@python.org> wrote:

The situation I described in my original post is limited to a case such as x = y ... the assignment can be done simply by "x" taking the pointer to "y" rather than moving all the data from "y" into the memory buffer for "x"

It's not simply whether it *can* be done. It, in fact, *MUST* be done
that way. The ONLY meaning of "x = y" is that you now have a name "x"
which refers to whatever object is currently found under the name "y".
This is not an optimization, it is a fundamental of Python's object
model. This is true regardless of what kind of object this is; every
object must behave this way.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Chris Angelico schreef op 15/01/2023 om 1:41:

On Sun, 15 Jan 2023 at 11:38, Jen Kris <jenkris@tutanota.com> wrote:

Yes, in fact I asked my original question – "I discovered something about Python array handling that I would like to clarify" -- because I saw that Python did it that way.

Yep. This is not specific to arrays; it is true of all Python objects.
Also, I suspect you're still thinking about things backwards, and am
trying to lead you to a completely different way of thinking that
actually does align with Python's object model.

Indeen, I also still have the impression that Jen is thinking in terms
of variables that are possible aliased such as you can have in a
language like C, instead of objects with one or more names like we have
in Python. Jens, in the Python model you really have to think of the
objects largely independently of the names that are or are not
referencing the objects.

--
"Ever since I learned about confirmation bias, I've been seeing
it everywhere."
-- Jon Ronson

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2023-01-15 4:36 AM, Roel Schroeven wrote:

Chris Angelico schreef op 15/01/2023 om 1:41:

On Sun, 15 Jan 2023 at 11:38, Jen Kris <jenkris@tutanota.com> wrote:

Yes, in fact I asked my original question – "I discovered something

about Python array handling that I would like to clarify" -- because I
saw that Python did it that way.

Yep. This is not specific to arrays; it is true of all Python objects.
Also, I suspect you're still thinking about things backwards, and am
trying to lead you to a completely different way of thinking that
actually does align with Python's object model.

Indeen, I also still have the impression that Jen is thinking in terms
of variables that are possible aliased such as you can have in a
language like C, instead of objects with one or more names like we have
in Python. Jens, in the Python model you really have to think of the
objects largely independently of the names that are or are not
referencing the objects.

My 'aha' moment came when I understood that a python object has only
three properties - a type, an id, and a value. It does *not* have a name.

Frank Millman

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Jen Kris wrote:

Avi,

Your comments go farther afield than my original question, but you made some interesting additional points.  For example, I sometimes work with the C API and sys.getrefcount may be helpful in deciding when to INCREF and DECREF.  But that’s another

issue.

The situation I described in my original post is limited to a case such as x = y where both "x" and "y" are arrays – whether they are lists in Python, or from the array module – and the question in a compiled C extension is whether the assignment

can be done simply by "x" taking the pointer to "y" rather than moving all the data from "y" into the memory buffer for "x" which, for a wide array, would be much more time consuming than just moving a pointer.  The other advantage to doing it that way
is if, as in my case, we perform a math operation on any element in "x" then Python expects that the same change to be reflected in "y."  If I don’t use the same pointers then I would have to perform that operation twice – once for "x" and once 
for "y" – in addition to the expense of moving all the data.

The answers I got from this post confirmed that it I can use the pointer if "y" is not re-defined to something else during the lifespan of "x."  If it is then "x" has to be restored to its original pointer.  I did it that way, and helpfully the

compiler did not overrule me.

I haven't done much with C extensions, but I don't think you'd need to
do anything with "x" in that case. If something else is assigned to
"y", "x" would still be a reference to the original object - why would
it need to be "restored" to anything? Unless I've misunderstood what's
going on here...

--
Mark.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

<<< Frank Millman>>> My 'aha' moment came when I understood that a python object has only three properties - a type, an id, and a value. It does *not* have a name.

Yes, Frank, it is a bit like how some people need to wrap their minds around a concept like an anonymous function. It has no name and for many purposes needs no name, but at the same time can be bound to one or more names if needed.

Some people will encounter a situation where you need to pass a function to be called such as a key function for determining the sort order and they will create a new function first the usual way with a name and then pass it along by name. That works
fine but the function then persists while not being used again and even choosing a name can have consequences if the same name is already in use and so on.

Experienced programmers might use some kind of lambda expression in-line as the function as an object is then created, and passed along as a reference to be used and probably discarded once nothing refers to it.

Similar things can happen if an object is created as part of a larger complex such as a list or deque of them. Each item can be referenced unambiguously without having a formal name. You can pop off the next one an use it or ask to take the fifth. It may
acquire and lose names as the program runs but sometimes can have many names or none.

Getting people to see that an object exists whether it has no name and to speak about them with that understanding can be a problem. We as humans seem to think we are our names. But I know my names have changed over the years partially because I moved
between countries and I have several additional names used just for special purposes, and yet for some purposes I am simply a number. If I raise my class in a lecture hall where my name is not known, they may point at me or ask the guy in the blue short
to talk. At the Department of Motor Vehicles, my name, like everyone else, is NEXT!

The real point is what Python does, not what other situations require. Two objects are equal does not always mean what you think or want it to mean. What we are discussing here is two objects loosely of type "name" that are being compared not for whether
THEY are themselves equal as in the same symbols composing them and perhaps living in the same namespace. The question was about two objects that contained references to another object or even to different parts of another object. As it happens, that can
be a deeper question in which some parts are not a required aspect of the language and may be part of one of many possible implementations. And, as noted, the names can be detached from that reference too so a link is not permanent.

Have we beaten this one to death yet?


-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On Behalf Of Frank Millman
Sent: Sunday, January 15, 2023 12:47 AM
To: python-list@python.org
Subject: Re: To clarify how Python handles two equal objects





--
https://mail.python.org/mailman/listinfo/python-list

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)