How to send Base64 image to Google Cloud Vision API label detection in Ruby?

3
votes

Hi I'm building a program in Ruby to generate alt attributes for images on a webpage. I'm scraping the page for the images then sending their src, in other words a URL, to google-cloud-vision for label detection and other Cloud Vision methods. It takes about 2-6 seconds per image. I'm wondering if there's any way to reduce response time. I first used TinyPNG to compress the images. Cloud Vision was a tad faster but the time it took to compress more than outweighed the improvement. How can I improve response time? I'll list some ideas.

1) Since we're sending a URL to Google Cloud, it takes time for Google Cloud to receive a response, that is from the img_src, before it can even analyze the image. Is it faster to send a base64 encoded image? What's the fastest form in which to send (or really, for Google to receive) an image?

cloud_vision = Google::Cloud::Vision.new project: PROJECT_ID
@vision = cloud_vision.image(@file_name)
@vision.labels #or @vision.web, etc.

2) My current code for label detection. First question: is it faster to send a JSON request rather than call Ruby (label or web) methods on a Google Cloud Project? If so, should I limit responses? Labels with less than a 0.6 confidence score don't seem of much help. Would that speed up image rec/processing time?

Open to any suggestions on how to speed up response time from Cloud Vision.

2
rubygoogle-app-engineweb-scrapinggoogle-cloud-vision
Please leave feedback on the answer provided. Did it work for you or not? If not, why? – karnesJ.R
You should better save image to file and then send it -- ruby Cloud Vision library has method for that. If you send the image as a String object, you consume the RAM for it and I won't be surprised if Net::HTTP library anyway converts the payload to tempfile anyway if it's enough large. – Nakilon

2 Answers

2
votes

TL;DR - You can take advantage of the batching supporting in the annotation API for Cloud Vision.

Longer version

Google Cloud Vision API supports batching multiple requests in a single call to the images:annotate API. There are also these limits which are enforced for Cloud Vision:

  • Maximum of 16 images per request
  • Maximum 4 MB per image
  • Maximum of 8 MB total request size.

You could reduce the number of requests by batching 16 at a time (assuming you do not exceed any of the image size restrictions within the request):

#!/usr/bin/env ruby

require "google/cloud/vision"

image_paths = [
  ...
  "./wakeupcat.jpg",
  "./cat_meme_1.jpg",
  "./cat_meme_2.jpg",
  ...
]

vision = Google::Cloud::Vision.new
length = image_paths.length

start = 0
request_count = 0
while start < length do
  last = [start + 15, length - 1].min
  current_image_paths = image_paths[start..last]
  printf "Sending %d images in the request. start: %d last: %d\n", current_image_paths.length, start, last
  result = vision.annotate *current_image_paths, labels: 1
  printf "Result: %s\n", result
  start += 16
  request_count += 1
end

printf "Made %d requests\n", request_count
1
votes

So you're using Ruby to scrape some images off a page and then send the image to Google, yeah?

Why you might not want to base64 encode the image:

  • Headless scraping becomes more network intensive. You have to download the image to then process it.
  • Now you also have to worry about adding in the base64 encode process
  • Potential storage concerns if you aren't just holding the image in memory (and if you do this, debugging becomes somewhat more challenging

Why you might want to base64 encode the image:

  • The image is not publicly accessible
  • You have to store the image anyway

Once you have weighed the choices, if you still want to get the image into base64 here is how you do it:

require 'base64'
Base64.encode(image_binary)

It really is that easy.

But how do I get that image in binary?

require 'curb'
# This line is an example and is not intended to be valid
img_binary = Curl::Easy.perform("http://www.imgur.com/sample_image.png").body_str

How do I send that to Google?

Google has a pretty solid write-up of this process here: Make a Vision API Request in JSON

If you can't click it (or are too lazy to) I have provided a zero-context copy-and-paste of what a request body should look like to their API here:

request_body_json = {
  "requests":[
    {
      "image":{
        "content":"/9j/7QBEUGhvdG9...image contents...eYxxxzj/Coa6Bax//Z"
      },
      "features":[
        {
          "type":"LABEL_DETECTION",
          "maxResults":1
        }
      ]
    }
  ]
}

So now we know what a request should look like in the body. If you're already sending the img_src in a POST request, then it's as easy as this:

require 'base64'
require 'curb'
requests = []
for image in array_of_image_urls
  img_binary = Curl::Easy.perform(image).body_str
  image_in_base64 = Base64.encode(image_binary)
  requests << { "image" => { "content" : image_in_base64 }, "imageContext" => "<OPTIONAL: SEE REFERENCE LINK>", "features" => [ {"type" => "LABEL_DETECTION", "maxResults" => 1 }]}
end

# Now just POST requests.to_json with your Authorization and such (You did read the reference right?)

Play around with the hash formatting and values as required. This is the general idea which is the best I can give you when your question is SUPER vague.