Ruby Twitter Gem, rate limit exceeded

Question

I am trying to use the ruby gem Twitter (https://github.com/sferik/twitter) to fetch the followers of a user from twitter api.

As per the documentation (https://dev.twitter.com/docs/api/1/get/followers/ids), twitter returns 5000 users in one request. As per the rate limit settings twitter allows me to make 15 calls per 15 minutes (https://dev.twitter.com/docs/rate-limiting/1.1/limits)

Problem:

When I retrieve followers of a user with more than 75000 followers (5000*15) I get a 'Rate limit exceeded' error.

When I user the gem method Twitter.followers_id(user), I get all the followers automagically and not in buckets of 5000 with cursor. I guess the gem takes care of this internally, and hence I am unable to control or delay these requests.

In the gem documentation there is an example of rate limiting (https://github.com/sferik/twitter#rate-limiting), but I dont know if that takes care of the already retrieved users or again starts from fresh.

My question is How and when do I apply a throttling mechanism to this to get all the followers?

Regards

Answer 1

The workaround for this problem is explained very well here.

MAX_ATTEMPTS = 3
num_attempts = 0
begin
  num_attempts += 1
  retweets = Twitter.retweeted_by_user("sferik")
rescue Twitter::Error::TooManyRequests => error
  if num_attempts <= MAX_ATTEMPTS
    # NOTE: Your process could go to sleep for up to 15 minutes but if you
    # retry any sooner, it will almost certainly fail with the same exception.
        sleep error.rate_limit.reset_in
    retry
  else
    raise
  end
end

Answer 2

Add a sleep command to the cursor_from_response_with_user method in the Twitter gem which is located in lib/twitter/api/utils.rb

https://github.com/sferik/twitter/blob/master/lib/twitter/api/utils.rb

def cursor_from_response_with_user(collection_name, klass, request_method, path, args, method_name)
  puts "sleeping 60"
  sleep 60
  arguments = Twitter::API::Arguments.new(args)
  merge_user!(arguments.options, arguments.pop || screen_name) unless arguments.options[:user_id] || arguments.options[:screen_name]
  cursor_from_response(collection_name, klass, request_method, path, arguments.options, method_name)
end

Each cursor response will take a minimum of 60 seconds so you won't go over the 15 requests per 15 minutes. It's a bit of a hack but it will work until this issue is sorted for 75K+ follower_ids.