Global dict, not all threads adding keys to it?

Question

I am confused about global variables in Python. Sometimes a global variable is shared among all instances of my program and sometimes an instance will create its own version of the global variable.

What I need is to have one handler that handles items put into a global dictionary. There is only one function that adds items to the global dictionary, but this function is run multiple times concurrently.

In views:

global_dict = {}

def handler():
    global global_dict 

    print "Starting handler"
    while True:
        local_dict = dict(global_dict) 
        for key, v in local_dict.iteritems():
            handle_the_item(v)
            del global_dict[key]
            print "Handled: ", key
        time.sleep(0.05)

def some_function(function_number)
    global global_dict

    print "Starting function", function_number
    for x in y:
        key = random.randint(0, 5000000)
        print function_number, "giving to handler:", key
        global_dict[key] = some_item

I start:

(I use Django so each function is started by calling a url with some parameter)

handler()
some_function(1)
some_function(2)
some_function(3)

It prints:

Starting handler
Starting function 1
Starting function 2
Starting function 3

1 giving to handler 111111
2 giving to handler 222222
3 giving to handler 333333
1 giving to handler 444444
2 giving to handler 555555
3 giving to handler 666666

Handled: 111111
Handled: 222222
Handled: 444444
Handled: 555555

The handler never handles the items added by function 3. As far as I can tell, this is because the handler and function 3 each have their own instances of the (supposedly) global variable global_dict. I also verified this by printing out the length of global_dict in some_function(). global_dict in function 3 keeps growing as more items are added.

Note that it is random which functions share an instance of global_dict and which have their own. If I stop everything and run it again, it may be that all instances share global_dict. Or none. Or 3 and 1 along with the handler.

I noticed that you don't check for key in global_dict. It is possible that Python's random number generator is the cause of your problems and that you are overwriting keys already in global_dict. — James Brewer
Had the same thought and checked that. I print out the key number that I am giving to the handler and they are different (and more random than my easier-to-read example numbers here). — user984003
Your some_function definition is missing a semicolon. Is this your actual code? — Kevin
You have so many race conditions I don't even know where to start. — Wessie
I just wrote a long post explaining exactly why this happens. You may want to check it out here — jeffknupp

Duncan Duncan · Accepted Answer · 2013-02-22T20:49:32

When you run an application such as Django behind a web server the web server will usually run multiple instances of your application as separate long processes. If several requests come in together they will be distributed across the application processes.

This means that you cannot depend on global variables being shared across requests. You might be lucky and the requests come in to the same process, or you might be unlucky and different requests go to different instances of the application.

If you need to share state between web requests then you should use some other mechanism such as a database or memcached.

Global dict, not all threads adding keys to it?

1 Answers