Run Kafka Consumers on Google Cloud Run

3
votes

I have a large application with lots of microservices that communicate through Kafka. Right now it's working on GKE.

We are moving Kafka to confluent.io and we were planning to move some of the microservices to Google Cloud Run (fully managed).

BUT,... it looks like Google Cloud Run (fully managed) does not support listening to kafka events, right? Are there any plans to support it? Is there a workaround?

EDIT:

This post shared by andres-s, shows that you can implement your own cloud run and have it connected to confluent kafka, in Anthos.

It would be great to have this option in the fully-managed Google Cloud Run service.

But in the meantime, the question would be: is it possible to implement it in a regular GKE cluster (not Anthos)?

2
apache-kafkagoogle-kubernetes-enginegoogle-cloud-run
AFAIK, Cloud Run fully managed is compliant with Knative serving API, not with eventing. And so, you can't link both world without a wrapper/translator - guillaume blaquiere
Sorry, my bad,... you're right, it's built using Knative serving API. I was confused, I thought it was using Knative eventing API. I'm going to update the question. - csalazar
Anthos is marketing in your case!! Anthos means "manage several clusters (GKE, EKS, AKS, on prem,...) in a single place". If you use simply GKE, the sticker Anthos is just marketing! - guillaume blaquiere

2 Answers

2
votes

Google Cloud has a fully managed Kafka solution through SaaS partner Confluent, which uses Cloud Run for Anthos (with GKE)

Google Pub/Sub is the GCP alternative for Kafka, but through Confluent you can use kafka on GCP

2
votes

Cloud Run is just Knative SERVING. It is stateless and spins up when it receives events. Due to this, it can't really subscribe to a topic and pull an event.

Knative Eventing is more stateful in nature and handles the pulls and subsequently triggers the pods running Knative Serving. They, ideally, are used together to give you the full serverless experience.

The good news is, there is a "hack". You can do Kafka to PubSub then PubSub to Cloud Run. If you are adventurous and don't mind OSS software, there are a number of Knative Eventing tutorials at serverlesseventing.com.