Ruby Google Cloud PubSub async pull returns GRPC::DeadlineExceeded error

Question

Hi I have a subscription set up in Google PubSub and I am trying to pull messages asynchronously using the "official" google-cloud-ruby library. Here is my code which will be executed from a rake task which passes in subscription_name:

def pull!
  creds = Google::Cloud::PubSub::Credentials.new(
    GCP_CREDENTIALS_KEYFILE_PATH,
    scope: "https://www.googleapis.com/auth/pubsub"
  )
  messages = []
  pubsub = Google::Cloud::PubSub.new(
    project_id: GOOGLE_PROJECT_ID,
    credentials: creds
  )

  subscription = pubsub.subscription(subscription_name)

  subscription.pull(immediate: true).each do |received_message|
    puts "Received message: #{received_message.data}"
    received_message.acknowledge!
    messages.push(received_message)
  end

  # Return the collected messages
  messages
  
rescue => error
  Rails.logger error

  messages.presence
end

The Google::Cloud::PubSub::Credentials part references a working keyfile. I know the JSON keyfile is good since I can use it to generate a working Bearer token using oauth2l and pull from the PubSub using cURL, postman, Net::HTTP, etc. Using the same JSON credentials for a separate Google::Cloud::Storage service and that works fine also.

But for some reason using Google::Cloud::PubSub it just hangs and won't respond. After about 60 seconds I get the following error:

GRPC::DeadlineExceeded: 4:Deadline Exceeded. debug_error_string:{"created":"@1602610740.445195000","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":69,"grpc_status":4}
/Users/bbulpet/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grpc-1.32.0-universal-darwin/src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status'
/Users/bbulpet/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grpc-1.32.0-universal-darwin/src/ruby/lib/grpc/generic/active_call.rb:180:in `attach_status_results_and_complete_call'
/Users/bbulpet/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grpc-1.32.0-universal-darwin/src/ruby/lib/grpc/generic/active_call.rb:376:in `request_response'
/Users/bbulpet/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grpc-1.32.0-universal-darwin/src/ruby/lib/grpc/generic/client_stub.rb:172:in `block (2 levels) in request_response'
/Users/bbulpet/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grpc-1.32.0-universal-darwin/src/ruby/lib/grpc/generic/interceptors.rb:170:in `intercept!'
/Users/bbulpet/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/grpc-1.32.0-universal-darwin/src/ruby/lib/grpc/generic/client_stub.rb:171:in `block in request_response'
/Users/bbulpet/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/gapic-common-0.3.4/lib/gapic/grpc/service_stub/rpc_call.rb:121:in `call'
/Users/bbulpet/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/gapic-common-0.3.4/lib/gapic/grpc/service_stub.rb:156:in `call_rpc'
/Users/bbulpet/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/google-cloud-pubsub-v1-0.1.2/lib/google/cloud/pubsub/v1/subscriber/client.rb:503:in `get_subscription'
/Users/bbulpet/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/google-cloud-pubsub-2.1.0/lib/google/cloud/pubsub/service.rb:154:in `get_subscription'
/Users/bbulpet/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/google-cloud-pubsub-2.1.0/lib/google/cloud/pubsub/project.rb:286:in `subscription'

Adding a debugger shows that the following line is what hangs and results in an error:

subscription = pubsub.subscription(subscription_name)

I've tried everything I can think of based on the documentation. Updated all related gems and even tried using deprecated syntax out of desparation. If anyone has at least an idea of where to start that would be most appreciated, thanks!

UPDATE

So it turns out that for some reason running this locally was not able to connect, but when shipped to a deployed environment the above code connected flawlessly to pubsub and was able to pull and ack messages. Further, upon making the initial connection in the deployed environment, I am now able to connect locally as well using the same credentials.

Link to Github issue conversation for context around the troubleshooting process and quartzmo suggestion to try deploying to another environment.

Answer 1

Are you still having this issue? I just tried to reproduce it using Ruby 2.6.5p114, google-cloud-pubsub 2.1.0 and grpc 1.32.0 (same versions as you), but I cannot reproduce it. Here is my code (slightly modified to run in a Minitest spec context) for comparison:

  GCP_CREDENTIALS_KEYFILE_PATH = "/Users/quartzmo/my-project.json"
  GOOGLE_PROJECT_ID = "my-project-id"

  def pull! topic_name, subscription_name
    creds = Google::Cloud::PubSub::Credentials.new(
      GCP_CREDENTIALS_KEYFILE_PATH,
      scope: "https://www.googleapis.com/auth/pubsub"
    )
    messages = []
    pubsub = Google::Cloud::PubSub.new(
      project_id: GOOGLE_PROJECT_ID,
      credentials: creds
    )
    topic = pubsub.create_topic topic_name
    topic.subscribe subscription_name

    topic.publish "A test message from #{topic_name} to #{subscription_name}"
  
    subscription = pubsub.subscription(subscription_name)
  
    subscription.pull(immediate: true).each do |received_message|
      puts "Received message: #{received_message.data}"
      received_message.acknowledge!
      messages.push(received_message)
    end
  
    # Return the collected messages
    messages
  end

  focus
  it "pull!" do
    topic_name = random_topic_name
    subscription_name = random_subscription_name
    messages = pull! topic_name, subscription_name
    assert_equal 1, messages.count
    assert_equal "A test message from #{topic_name} to #{subscription_name}", messages[0].data
  end

And this is the output:

% bundle exec rake test
Run options: --junit --junit-filename=sponge_log.xml --seed 30984

# Running:

Received message: A test message from ruby-pubsub-samples-test-topic-7cb10bde to ruby-pubsub-samples-test-subscription-f47f2eaa
.

Finished in 6.529219s, 0.1532 runs/s, 0.3063 assertions/s.

1 runs, 2 assertions, 0 failures, 0 errors, 0 skips

Update (2020-10-20): This issue was resolved when executing the code in a different environment, although the reason is unknown. See comment on GitHub issue.