python: how-to do a periodic non-blocking lookup

Question

Could you please advise on how to periodically execute a task that it takes more time to execute than the periodic interval ?

For example:

def lookup():
    # do some lookups, retrieve info, let's assume it takes 60sec to complete
    msg = {'abc':123}
    time.sleep(60)
    return msg

class Publisher(object):
    def __init__(self):
        self._TIMEIT = 0
        self._INTERVAL = 5
        self._counter = 0

    def xxx():
        t_start = time.time()
        msg = lookup()
        # do something with the value returned
        save_msg_to_db(msg)
        self._counter += 1
        t_end = time.time()
        self._TIMEIT = int(math.ceil(t_end - t_start))

    def run():
        while True:
            # let's do the lookup every 5sec, but remember that lookup takes 60sec to complete
            time.sleep(self._INTERVAL)
            # the call to xxx() should be non-blocking
            xxx()

But the run method is responsible for scheduling the periodic task, and as it iterates it should not block when calling the function xxx.

I am thinking to create an event loop on every call to xxx function, as described in A Bad Coroutine Example but how to do the call to xxx non-blocking ?

PS. I am using Python3.4 new to asyncio (was using gevent in the past), not sure if I am asking sth stupid here.

So lookup will create a async loop that will take let's say 60sec to complete. But, in the run method there is a endless loop running that i would like it to do the lookup every 5sec, in other words I would like to (1) how often i call the lookup function, independent of (2) how long it takes lookup to complete

Is your lookup() function CPU-intensive or it just takes long time to complete due to I/O operations? — zwer
: "it just takes long time to complete due to I/O operations" — nskalis
And what happens while the lookup() is running? If you want to run it in the background (non-blocking) and it takes 60 seconds to complete and you call it every 5 seconds pretty soon you will have thousands of them running. So something has got to give - either your checking interval (extending it) or the calls to the lookup() function (don't make another call if the previous didn't complete). — zwer
@zwer well i am counting on the fact that there will 60/5=12 tasks more or less all the time and i am perfectly fine with it — nskalis

zwer zwer · Accepted Answer · 2017-06-27T08:26:00

Since your lookup() is primarily I/O intensive, you can run your xxx() method as a thread and be perfectly fine (code shortened for brevity):

import threading
import time

class Publisher(object):

    def __init__(self):
        self._INTERVAL = 5
        self._counter = 0
        self._mutex = threading.Lock()

    def xxx(self):
        msg = lookup()
        save_msg_to_db(msg)
        with self._mutex:  # make sure only one thread is modifying counter at a given time
            self._counter += 1

    def run(self):
        while True:
            time.sleep(self._INTERVAL)
            t = threading.Thread(target=self.xxx)
            t.setDaemon(True)  # so we don't need to track/join threads
            t.start()  # start the thread, this is non-blocking

python: how-to do a periodic non-blocking lookup

1 Answers