Fetch all the products (more than 50,000) using the Shopify API

Question

I am trying to fetch all the products of a Shopify store in my app using the Shopify API. I have the following code to fetch all the products page-wise. This script is breaking at some point in time, but I am not sure where. How can I make a thorough script which will fetch all the products in one go without breaking.

Here is what I have thus far:

$ch = curl_init();
for ($i=1; $i<=1000; $i++)
{
    curl_setopt($ch, CURLOPT_URL, "https://API:PASS@store/products.json?page=$i");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
    curl_setopt($ch, CURLOPT_TIMEOUT, 20);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 20);
    $res = curl_exec($ch);
    $res=json_decode($res, true);
    foreach($res as $products) { // A different example
        foreach($products as $product) {
            $title=$product['title'];
        }
     }
     sleep(1); 
}
curl_close($ch);

Shopify have option to increase and improve API call. This option is available for Shopify Plus account. You contact with shopify account manager to improve API call. — kaushal sharma

alexandresaiz alexandresaiz · Accepted Answer · 2015-03-11T09:50:06

You're probably reaching the Shopify API limit call. So you'd probably need to work with queues. Here is how the Shopify API limit calculations work:

Shopify uses a leaky bucket algorithm with a bucket size of 40 calls, and a leak rate of 2 per second.

How does that work:

Each call you make will increase the level of a virtual bucket level by one
Every half a second the bucket leaks, decreasing the current level by one
Your calls are processed immediately as long as the bucket has room, otherwise you get a 429 error
The X-Shopify-Shop-Api-Call-Limit header will be indicating the current level and the bucket size as 'x/40'

What does that mean for your app:

You can do bursts of up to 40 calls in a second. They will all be processed immediately
Once you hit the limit it will take 500 ms before another call will be allowed through
As long as you stay under an average of 2 calls per second you won't hit the limit
This actually maxes out to around 600 calls per store per 5 minutes

Fetch all the products (more than 50,000) using the Shopify API

1 Answers