How to determine how many more requests my app can make before it hits the Twitter API's rate limit?

Question

My application habitually makes a request to the Twitter API for a user's timeline (the user's tweets).

As an aside, I'm using the twitter gem and configured a Twitter client in order to enable my app to make the request:

client = Twitter::REST::Client.new do |config|
  config.consumer_key        = ENV["TWITTER_CONSUMER_KEY"]
  config.consumer_secret     = ENV["TWITTER_CONSUMER_SECRET"]
  config.access_token        = ENV["TWITTER_ACCESS_TOKEN"]
  config.access_token_secret = ENV["TWITTER_ACCESS_TOKEN_SECRET"]
end

The endpoint that I'm making a request to -- https://api.twitter.com/1.1/statuses/user_timeline.json -- has a rate limit of 900 requests every 15 minutes.

If I understand it correctly, there are 2 common ways to determine if my app has hit its rate limit -- but neither is what I need:

First Approach:

This is a function I wrote that attempts to make the request for a user's tweets, and if the app has it the rate limit, it'll raise an exception.

def get_tweets_from_TwitterAPI(twitter_handle)
  tweets = []
  begin
    tweets = CLIENT.user_timeline(twitter_handle)
  rescue Twitter::Error::TooManyRequests => error
    raise error
  end
  return tweets
end

The problem with that approach is I'd like to find out how many more requests my app can safely make before I even make the request to the Twitter API. I fear that this approach would have my app pinging the Twitter API long after it hit the rate limit, and that would open my app up to punitive actions by Twitter (like my app being blacklisted for instance.)

Second Approach The second approach is to make a request to this Twitter endpoint --https://api.twitter.com/1.1/application/rate_limit_status.json -- that sends data back about where the application's status for a given rate limit.

But again, this endpoint also has its own rate limit (180 requests every 15 minutes) -- which isn't very high. My app would blow past that limit. In an ideal world, I would like to determine my app's current rate-limit status before I make a request to the API at all. And so:

def start_tweets_fetch
  number_of_requests_in_last_15 minutes = WHERE DO I GET THIS NUMBER FROM??
  if number_of_requests_in_last_15 minutes <= 900
    get_tweets_from_TwitterAPI(twitter_handle)
  end
end

I'm imagining I would have to increment some number that I've persisted to my database to keep track of requests to the API. Or is there an easier way?

Answer 1

I can't speak for the gem you are using, but a way to track your request limits without having to additionally call the rate_limit_status endpoint is to examine the X-Rate-Limit-Remaining headers on each API call. I don't know whether that data is available on the Ruby gem you're using, though.

Answer 2

Edit

This is in response to Andy Piper's answer which I think is the simplest way to keep track of the remaining calls.

Assuming you're using this Twitter gem, it looks like each response from the gem will populate a Twitter::RateLimit object with the information from the rate limiting headers like Andy has suggested.

You should be able to access that information like this:

tweets = CLIENT.user_timeline(twitter_handle)

remaining_calls = tweets.rate_limit.remaining

From there you can save that value to check it the next time you want to make a request. How you save it and check is up to you but the rest of my answer may still be useful for that.

---

Note: I haven't tried this method before but it's one of the first things I would try in your situation if I didn't to permanently store request logs.

One way might to be to use Rails' built in Cache API. This will allow you to store any value you wish in a cache store which should be faster and lighter than a database.

number_of_requests_in_last_15 = Rails.cache.fetch("twitter_requests", expires_in: 15.minutes) { 0 }
if number_of_requests_in_last_15 minutes <= 900
  get_tweets_from_TwitterAPI(twitter_handle)
  Rails.cache.increment("twitter_requests")
end

Let's break this down

Rails.cache.fetch("twitter_requests", expires_in: 15.minutes) { 0 }:

• The fetch method on Rails.cache will attempt to pull the value for the key twitter_requests.
• If the key doesn't exist, it will evaluate the block and set the return value as the key's new value and return that. In this case, if the key twitter_requests doesn't exist, the new key value will be 0.
• The expires_in: 15.minutes option passed to the fetch method says to automatically clear this key (twitter_requests) every 15 minutes.

Rails.cache.increment("twitter_requests"):

• Increments the value in the twitter_requests key by 1.

Notes

• By default, Rails will use an in memory datastore. This should work without issue but any values stored in the cache will be reset every time you restart the rails server.
• The backend of the cache is configurable and can be changed to other popular systems (i.e. memcache, redis) but those will also need to be running and accessible by Rails.
• You may want to increment the cache before calling the API to reduce the chance of the cache expiring between when you checked it and when you increment it. Incrementing a key that doesn't exist will return nil.