Fast way to import 24000 single variation products to Shopify

Question

I want to import ~24000 (single variation) products to Shopify every night at 04:00.

The problem with the default Admin > Products > Import > CSV method is that it takes an unacceptably long time to finish. So I started working with the Shopify API using their Ruby gem.

The Shopify API has an API call limit of 2 calls per second, with room for a burst of 40 calls at once. You can read more about the API call limit here. I don't understand the "room for a burst of 40 calls at once" part.

A Shopify Employee told me the following about importing products via the API:

If you were to make Product creation API requests that includes the variant information in them (assuming 3 variants per product) and using a default API limit (refresh of 2 requests per second, or 2 concurrent requests) you could probably finish creating all those products in about 67 minutes.

I wrote the following Ruby script to import all products to Shopify:

require 'shopify_api'
require 'open-uri'
require 'json'

begin_time = Time.now

shop_url  = "https://<API_KEY>:<PASSWORD>@<SHOPNAME>.myshopify.com/admin"

ShopifyAPI::Base.site = shop_url

raw_product_data = JSON.parse(open('<LINK>') {|f| f.read }.force_encoding('UTF-8'))
raw_product_data_size = raw_product_data.size

puts '========================================================================='
puts "#{raw_product_data_size} products loaded. Starting import now..."
puts '-------------------------------------------------------------------------'

raw_product_data.each_with_index do |item, index|
  single_product_begin_time = Time.now

  # Store item data in variables
  <DATA>

  # Create new product
  new_product = ShopifyAPI::Product.new
  new_product.title = title
  new_product.body_html = body
  new_product.product_type = type
  new_product.vendor = vendor
  new_product.tags = tags

  new_product.variants = [
    {
      "option_1" => "First",
      "price" => variant_price,
      "sku" => variant_sku
    }
  ]

  new_product.images = [
    {
      src: image_src
    }
  ]

  new_product.save

  creation_time = Time.now - single_product_begin_time

  puts "#{((index / raw_product_data_size.to_f) * 100).round(2)}% - #{index} - #{variant_sku} added in #{creation_time.round(2)} seconds"
end

puts '-------------------------------------------------------------------------'
puts "Done. It took #{begin_time - Time.now} minutes."
puts '========================================================================='

But I'm not getting anywhere near the 67 minutes.

It currently takes around 3 seconds to add a single product. At this rate it would take 20 hours to add 24000 products. That's unacceptable. Everything under 2 hours is acceptable.

What could I try to speed things up?

Its probably because they host the API on the same servers as the regular web app, which is dog slow and probably oversold * 10. — mxmissile

narzero narzero · Accepted Answer · 2014-10-20T19:20:42

Short and sweet answer, use multithreading.

Since a single product took ~3 seconds to import, my solution is having 5 workers (threads) running the product import and checking for 429 errors. If they hit a 429, they pause for 10 seconds and then retry. It currently takes ~3,6 hours to import ~26000 product.

Fast way to import 24000 single variation products to Shopify

2 Answers