Rate Limit API Calls to Shopify API with Django on Google App Engine

Question

I am trying to use the Shopify API from my Django app hosted on Google App Engine.

For my local single threaded scripts I am using a modified version of this to make sure that I don't go over Shopify's rate limit:

# Setup local bucket to limit API Calls
bucket = TokenBucket(40, .5)

api_call_success = False

while not api_call_success:
    if bucket.tokens < 1:
        sleep(.5)
    else:
        [... do an API call ...]
        bucket.consume(1)
        api_call_success = True

This works for my local scripts, but it won't work for my Google App Engine hosted application where there may be multiple tenants, and multiple sessions occurring at once.

I have been trying to research the best way to handle this rate limiting, and presently was going to try to constantly write each users/stores request response header to memcache so that I could always check the 'x-shopify-shop-api-call-limit' to see what the previous call limit (and time of the call) was. So I tried something like this:

    fill_rate = .5
    capacity = 40   

    # get memcache key info
    last_call_time = memcache.get(memKey+"_last_call_time")
    last_call_value = memcache.get(memKey+"_last_call_value")

    # Calculate how many tokens should be available
    now = datetime.datetime.utcnow()
    delta = fill_rate * ((now - last_call_time).seconds)
    tokensAvailable = min(capacity, delta + last_call_value)

    # Check if we can perform operation
    if tokensAvailble > 1:
        [... Some work involving Shopify API call ...]
        # Do some work and then update memcache
        memcache.set_multi( {"_last_call_time": datetime.datetime.strptime(resp_dict['date'], '%a, %d %b %Y %H:%M:%S %Z'), "_last_call_value": resp_dict['x-shopify-shop-api-call-limit'].split('/',1)[0]}, key_prefix=memKey, time=120)
    else:
        [... wait ...]

Can anyone recommend a better way to manage this rate limiting?

Julien Julien · Accepted Answer · 2016-10-26T20:54:51

I essentially have the same logic as you (using Redis), however instead of doing this inline everywhere, I've monkey patched shopify.base.ShopifyConnection like so:

from time import sleep
from django.conf import settings
from pyactiveresource.activeresource import formats
from pyactiveresource.connection     import (
    Connection,
    ConnectionError,
    ServerError,
)
import shopify


class ShopifyConnection(Connection, object):
    response = None

    def __init__(self, site, user=None, password=None, timeout=None,
                 format=formats.JSONFormat):
        super(ShopifyConnection, self).__init__(site, user, password, timeout, format)

    def consume_token(uid, capacity, rate, min_interval=0):
        # Your rate limiting logic here

    def _open(self, *args, **kwargs):
        uid = self.site.split("https://")[-1].split(".myshopify.com")[0]
        self.response = None
        retries = 0
        while True:
            try:
                self.consume_token(uid, 40, 1.95, 0.05)
                self.response = super(ShopifyConnection, self)._open(*args, **kwargs)
                return self.response
            except (ConnectionError, ServerError) as err:
                retries += 1
                if retries > settings.SHOPIFY_MAX_RETRIES:
                    self.response = err.response
                    raise
                sleep(settings.SHOPIFY_RETRY_WAIT)


shopify.base.ShopifyConnection = ShopifyConnection

You'll want this code to live in a file within your app's directory, and then import the file into your app's __init__.py. This way, I can write the rest of my code without worrying about any rate limiting logic. The only thing you'll have to take care of is to check if the ShopifyConnection class changes when you're updating your shopify module, and update the monkey patch accordingly. It's not a big issue, as the module doesn't update very often.

(As you can see, I've taken this monkey patch as an opportunity to insert retry logic, as about 1/1000 requests fail for no reason. I'm defining SHOPIFY_MAX_RETRIES and SHOPIFY_RETRY_WAIT in settings.py and pull in the values here.)

Rate Limit API Calls to Shopify API with Django on Google App Engine

3 Answers