Simulating Pointers in Python

Question

I'm trying to cross compile an in house language(ihl) to Python.

One of the ihl features is pointers and references that behave like you would expect from C or C++.

For instance you can do this:

a = [1,2];  // a has an array 
b = &a;     // b points to a
*b = 2;     // derefernce b to store 2 in a
print(a);   // outputs 2
print(*b);   // outputs 2

Is there a way to duplicate this functionality in Python.

I should point out that I think I've confused a few people. I don't want pointers in Python. I just wanted to get a sense from the Python experts out there, what Python I should generate to simulate the case I've shown above

My Python isn't the greatest but so far my exploration hasn't yielded anything promising:(

I should point out that we are looking to move from our ihl to a more common language so we aren't really tied to Python if someone can suggest another language that may be more suitable.

Answer 1

This can be done explicitly.

class ref:
    def __init__(self, obj): self.obj = obj
    def get(self):    return self.obj
    def set(self, obj):      self.obj = obj

a = ref([1, 2])
b = a
print(a.get())  # => [1, 2]
print(b.get())  # => [1, 2]

b.set(2)
print(a.get())  # => 2
print(b.get())  # => 2

Answer 2

You may want to read Semantics of Python variable names from a C++ perspective. The bottom line: All variables are references.

More to the point, don't think in terms of variables, but in terms of objects which can be named.

Answer 3

If you're compiling a C-like language, say:

func()
{
    var a = 1;
    var *b = &a;
    *b = 2;
    assert(a == 2);
}

into Python, then all of the "everything in Python is a reference" stuff is a misnomer.

It's true that everything in Python is a reference, but the fact that many core types (ints, strings) are immutable effectively undoes this for many cases. There's no direct way to implement the above in Python.

Now, you can do it indirectly: for any immutable type, wrap it in a mutable type. Ephemient's solution works, but I often just do this:

a = [1]
b = a
b[0] = 2
assert a[0] == 2

(I've done this to work around Python's lack of "nonlocal" in 2.x a few times.)

This implies a lot more overhead: every immutable type (or every type, if you don't try to distinguish) suddenly creates a list (or another container object), so you're increasing the overhead for variables significantly. Individually, it's not a lot, but it'll add up when applied to a whole codebase.

You could reduce this by only wrapping immutable types, but then you'll need to keep track of which variables in the output are wrapped and which aren't, so you can access the value with "a" or "a[0]" appropriately. It'll probably get hairy.

As to whether this is a good idea or not--that depends on why you're doing it. If you just want something to run a VM, I'd tend to say no. If you want to be able to call to your existing language from Python, I'd suggest taking your existing VM and creating Python bindings for it, so you can access and call into it from Python.

Answer 4

Almost exactly like ephemient answer, which I voted up, you could use Python's builtin property function. It will do something nearly similar to the ref class in ephemient's answer, except now, instead of being forced to use get and set methods to access a ref instance, you just call the attributes of your instance which you've assigned as properties in the class definition. From Python docs (except I changed C to ptr):

class ptr(object):
    def __init__(self):
        self._x = None
    def getx(self):
        return self._x
    def setx(self, value):
        self._x = value
    def delx(self):
        del self._x
    x = property(getx, setx, delx, "I'm the 'x' property.")

Both methods work like a C pointer, without resorting to global. For example if you have a function that takes a pointer:

def do_stuff_with_pointer(pointer, property, value):
    setattr(pointer, property, value)

For example

a_ref = ptr()      # make pointer
a_ref.x = [1, 2]   # a_ref pointer has an array [1, 2]
b_ref = a_ref      # b_ref points to a_ref
# pass ``ptr`` instance to function that changes its content
do_stuff_with_pointer(b_ref, 'x', 3)
print a_ref.x      # outputs 3
print b_ref.x      # outputs 3

Another, and totally crazy option would be to use Python's ctypes. Try this:

from ctypes import *
a = py_object([1,2]) # a has an array 
b = a                # b points to a
b.value = 2          # derefernce b to store 2 in a
print a.value        # outputs 2
print b.value        # outputs 2

or if you want to get really fancy

from ctypes import *
a = py_object([1,2])   # a has an array 
b = pointer(a)         # b points to a
b.contents.value = 2   # derefernce b to store 2 in a
print a.value          # outputs 2
print b.contents.value # outputs 2

which is more like OP's original request. crazy!

Answer 5

Everything in Python is pointers already, but it's called "references" in Python. This is the translation of your code to Python:

a = [1,2]  // a has an array 
b = a     // b points to a
a = 2      // store 2 in a.
print(a)   // outputs 2
print(b)  // outputs [1,2]

"Dereferencing" makes no sense, as it's all references. There isn't anything else, so nothing to dereference to.

Answer 6

As others here have said, all Python variables are essentially pointers.

The key to understanding this from a C perspective is to use the unknown by many id() function. It tells you what address the variable points to.

>>> a = [1,2]
>>> id(a)
28354600

>>> b = a
>>> id(a)
28354600

>>> id(b)
28354600

Answer 7

This is goofy, but a thought...

# Change operations like:
b = &a

# To:
b = "a"

# And change operations like:
*b = 2

# To:
locals()[b] = 2


>>> a = [1,2]
>>> b = "a"
>>> locals()[b] = 2
>>> print(a)
2
>>> print(locals()[b])
2

But there would be no pointer arithmetic or such, and no telling what other problems you might run into...

Answer 8

class Pointer(object):
    def __init__(self, target=None):
        self.target = target

    _noarg = object()

    def __call__(self, target=_noarg):
        if target is not self._noarg:
            self.target = target
        return self.target

a = Pointer([1, 2])
b = a

print a() # => [1, 2]
print b() # => [1, 2]

b(2)
print a()  # => 2
print b()  # => 2

Answer 9

I think that this example is short and clear.

Here we have class with implicit list:

class A: 
   foo = []
a, b = A(), A()
a.foo.append(5)
b.foo
ans: [5]

Looking at this memory profile (using: from memory_profiler import profile), my intuition tells me that this may somehow simulate pointers like in C:

Filename: F:/MegaSync/Desktop/python_simulate_pointer_with_class.py

Line #    Mem usage    Increment   Line Contents
================================================
     7     31.2 MiB      0.0 MiB   @profile
     8                             def f():
     9     31.2 MiB      0.0 MiB       a, b = A(), A()
    10                                 #here memoery increase and is coupled
    11     50.3 MiB     19.1 MiB       a.foo.append(np.arange(5000000))
    12     73.2 MiB     22.9 MiB       b.foo.append(np.arange(6000000))
    13     73.2 MiB      0.0 MiB       return a,b


[array([      0,       1,       2, ..., 4999997, 4999998, 4999999]), array([      0,       1,       2, ..., 5999997, 5999998, 5999999])] [array([      0,       1,       2, ..., 4999997, 4999998, 4999999]), array([      0,       1,       2, ..., 5999997, 5999998, 5999999])]
Filename: F:/MegaSync/Desktop/python_simulate_pointer_with_class.py

Line #    Mem usage    Increment   Line Contents
================================================
    14     73.4 MiB      0.0 MiB   @profile
    15                             def g():
    16                                 #clearing b.foo list clears a.foo
    17     31.5 MiB    -42.0 MiB       b.foo.clear()
    18     31.5 MiB      0.0 MiB       return a,b


[] []
Filename: F:/MegaSync/Desktop/python_simulate_pointer_with_class.py

Line #    Mem usage    Increment   Line Contents
================================================
    19     31.5 MiB      0.0 MiB   @profile
    20                             def h():
    21                                 #and here mem. coupling is lost ;/
    22     69.6 MiB     38.1 MiB       b.foo=np.arange(10000000)
    23                                 #memory inc. when b.foo is replaced
    24    107.8 MiB     38.1 MiB       a.foo.append(np.arange(10000000))
    25                                 #so its seams that modyfing items of
    26                                 #existing object of variable a.foo,
    27                                 #changes automaticcly items of b.foo
    28                                 #and vice versa,but changing object
    29                                 #a.foo itself splits with b.foo
    30    107.8 MiB      0.0 MiB       return b,a


[array([      0,       1,       2, ..., 9999997, 9999998, 9999999])] [      0       1       2 ..., 9999997 9999998 9999999]

And here we have explicit self in class:

class A: 
    def __init__(self): 
        self.foo = []
a, b = A(), A()
a.foo.append(5)
b.foo
ans: []

Filename: F:/MegaSync/Desktop/python_simulate_pointer_with_class.py

Line #    Mem usage    Increment   Line Contents
================================================
    44    107.8 MiB      0.0 MiB   @profile
    45                             def f():
    46    107.8 MiB      0.0 MiB       a, b = B(), B()
    47                                 #here some memory increase
    48                                 #and this mem. is not coupled
    49    126.8 MiB     19.1 MiB       a.foo.append(np.arange(5000000))
    50    149.7 MiB     22.9 MiB       b.foo.append(np.arange(6000000))
    51    149.7 MiB      0.0 MiB       return a,b


[array([      0,       1,       2, ..., 5999997, 5999998, 5999999])] [array([      0,       1,       2, ..., 4999997, 4999998, 4999999])]
Filename: F:/MegaSync/Desktop/python_simulate_pointer_with_class.py

Line #    Mem usage    Increment   Line Contents
================================================
    52    111.6 MiB      0.0 MiB   @profile
    53                             def g():
    54                                 #clearing b.foo list
    55                                 #do not clear a.foo
    56     92.5 MiB    -19.1 MiB       b.foo.clear()
    57     92.5 MiB      0.0 MiB       return a,b


[] [array([      0,       1,       2, ..., 5999997, 5999998, 5999999])]
Filename: F:/MegaSync/Desktop/python_simulate_pointer_with_class.py

Line #    Mem usage    Increment   Line Contents
================================================
    58     92.5 MiB      0.0 MiB   @profile
    59                             def h():
    60                                 #and here memory increse again ;/
    61    107.8 MiB     15.3 MiB       b.foo=np.arange(10000000)
    62                                 #memory inc. when b.foo is replaced
    63    145.9 MiB     38.1 MiB       a.foo.append(np.arange(10000000))
    64    145.9 MiB      0.0 MiB       return b,a


[array([      0,       1,       2, ..., 9999997, 9999998, 9999999])] [      0       1       2 ..., 9999997 9999998 9999999]

ps: I'm self learning programming (started with Python) so please do not hate me if I'm wrong. Its just mine intuition, that let me think that way, so do not hate me!

Answer 10

Negative, no pointers. You should not need them with the way the language is designed. However, I heard a nasty rumor that you could use the: ctypes module to use them. I haven't used it, but it smells messy to me.