How to pass/exchange data from asynchronous methods/threads?

Question

Could someone explain the concept behind how to pass data between asynchronous calls in python?

I have this scenario:
I have a main procedure (mainthread) and then I run another async call that adds two numbers() and sleeps for some time. The expectation is to have main thread wait until calc and sleeping are done. In terms of pseudo code this might look like this:

def example():
    def my_calc(x,y):
        z = x+ y
        time.sleep(2)
        return z  #this should get pushed to queue, but for simplicity we use 'return z'

    z = asyncSomething.callInThread(my_calc, 2, 20)  #assume we get z from queue
    return z+10

def runner():  #our main function
    print 'start'
    z = example()
    print 'result is {0}'.format(z)

How do I made the last print wait for z? I tried to use threading.Event() and played with set, wait, and clear, but they block everything.

I do not think my case is unique and there has to be a neat systematic way of doing it. I looked into twisted reactor but it was partially successful. My actual code involved GUI that has to wait for results from async processes and then self- and auto- update...but I think the example depicts the bulk of the problem

I think I am missing the actual concept of how to work asynchronously in python.

Are you using Python 2? Looks like Python 3's async and await keywords can't help here ._. — GeeTransit
@Geetransit, it is 2.7. I used 'async' figuratively not literally — JavaFan
@opa, ecosystem I work with is still using 2.7 in 2019; hence my question — JavaFan

manveti manveti · Accepted Answer · 2019-01-09T01:03:21

If your main thread has progressed as far as it meaningfully can without the result from the worker thread and you want to block, you can either use a threading.Event to wait for a certain point in the worker thread's execution or thread.join (where thread is a threading.Thread instance) to wait for the worker thread to complete:

class Example(object):
    def my_calc(self, x, y):
        time.sleep(2)
        self._z = x + y # or push the result to a queue

    def example(self):
        thread = threading.Thread(target=self.my_calc, args=(2, 20))
        thread.start()
        # any other useful work could go here while the worker runs
        thread.join()
        return self._z + 10 # or grab the result from a queue

def runner():
    print "start"
    z = Example().example()
    print "result is {0}".format(z)

However, in the context of a GUI, it is unlikely that your main thread has progressed as far as it can -- it is probably constantly busy keeping the GUI responsive. In that case, as was mentioned in the comments on the question, it's better to just pack anything you want done after the worker's calculation is finished into a callback for the worker thread to call:

def my_calc(x, y):
    time.sleep(2)
    return x + y

def my_calc_thread(x, y, callback):
    z = my_calc(x, y)
    # depending on your GUI framework, you may need to do something
    # like call_in_main_thread(callback, z) if callback touches GUI
    # elements
    callback(z)

def example():
    def finish(z):
        print "result is {0}".format(z)
    t = threading.Thread(target=my_calc_thread, args=(2, 20, finish))
    t.start()

def runner():
    print "start"
    example()

How to pass/exchange data from asynchronous methods/threads?

2 Answers