ctypes - Simulating integer overflow in Python

Question

Python 2 has two integer datatypes int and long, and automatically converts between them as necessary, especially in order to avoid integer overflow.

I am simulating a C function in Python and am wondering if there are standard ways to re-enable integer overflow. For the nonce, I've used

overflow_point = maxint + 1
if value > overflow_point:
    value -= 2 * overflow_point

Is there a more standard way to do the same thing?

Answer 1

I think the basic idea is sound, but needs some tweaks:

1. your function doesn't overflow on sys.maxint+1, but it should;
2. sys.maxint can be exceeded several times over as a result of a single operation;
3. negative values below -sys.maxint-1 also need to be considered.

With this in mind, I came up with the following:

import sys

def int_overflow(val):
  if not -sys.maxint-1 <= val <= sys.maxint:
    val = (val + (sys.maxint + 1)) % (2 * (sys.maxint + 1)) - sys.maxint - 1
  return val

Answer 2

This function should convert your numbers to look like hardware integers. Depending on your application, you might need to apply this function between each stage of your operations.

def correct(value, bits, signed):
    base = 1 << bits
    value %= base
    return value - base if signed and value.bit_length() == bits else value

The following shortcut functions may come in handy for "casting" values to their appropriate range:

byte, sbyte, word, sword, dword, sdword, qword, sqword = (
    lambda v: correct(v, 8, False), lambda v: correct(v, 8, True),
    lambda v: correct(v, 16, False), lambda v: correct(v, 16, True),
    lambda v: correct(v, 32, False), lambda v: correct(v, 32, True),
    lambda v: correct(v, 64, False), lambda v: correct(v, 64, True)
)

As an example of how you might use them, a bug can reproduced that one might see in C. If one were to write a for loop using a byte to print out 0 - 255, the loop might never end. The following program demonstrates this problem:

#! /usr/bin/env python3
def main():
    counter = 0
    while counter < 256:
        print(counter)
        counter = byte(counter + 1)


def correct(value, bits, signed):
    base = 1 << bits
    value %= base
    return value - base if signed and value.bit_length() == bits else value


byte, sbyte, word, sword, dword, sdword, qword, sqword = (
    lambda v: correct(v, 8, False), lambda v: correct(v, 8, True),
    lambda v: correct(v, 16, False), lambda v: correct(v, 16, True),
    lambda v: correct(v, 32, False), lambda v: correct(v, 32, True),
    lambda v: correct(v, 64, False), lambda v: correct(v, 64, True)
)


if __name__ == '__main__':
    main()

Answer 3

Does your function use division or right bit-shifting? If not then you don't need to worry about overflows at each stage of the calculation because you will always get the "correct" answer modulo 2^32 or 2^64. Before returning the result (or before doing division or right bit-shifting) you can normalize back to the standard integer range using something like

import sys

HALF_N = sys.maxint + 1
N = HALF_N * 2

def normalize(value):
    return (value + HALF_N) % N - HALF_N

Answer 4

I don't know that there's a convenient way to do this natively because it's not normally considered a problem, so it's not something the python devs would want to build in. I think the way you're doing it is fine. You could even subclass the int built-in type and override the __add__(), __sub__(), etc operator methods to include your functionality, but that might be overkill.

Answer 5

Use NumPy (which is written in C and exposes native machine-level integers). NumPy has these integer types:

Signed integers:

• np.int8
• np.int16
• np.int32

Unsigned integers:

• np.uint8
• np.uint16
• np.uint32

For example (notice it overflows after integer value 127):

import numpy as np

counter = np.int8(0)
one = np.int8(1)

for i in range(130):
    print(type(counter), counter)
    counter = counter + one

At any point you can convert back to a Python integer with just int(counter), for example.