Does Google Cloud Pub/Sub service support “data residency”?

Question

My application needs to make sure that the data contained in the messages sent through Pub/Sub does not leave the boundaries of the European Union. This is in connection with the so-called "data residency" requirements.

Pub/Sub is described as a global service[1] where the clients of the service can publish/subscribe messages without knowing the specific location where the messages are stored and processed.

Can I use Pub/Sub for my application?

[1] https://cloud.google.com/pubsub/architecture

Answer 1

The version of the Google Cloud Pub/Sub service available today does not provide any "data residency" guarantees. However, it is possible to engineer a solution with Pub/Sub that minimizes the risks of violating "data residency" objectives such that the possibility of Pub/Sub message data getting stored outside of the EU multi-region becomes highly unlikely. Specifically:

1. When a publisher (i.e. Pub/Sub client code) residing in a EU region (e.g. eu-west1) publishes a message to a Pub/Sub topic, the message will be stored in the same region, unless a) Pub/Sub in the region suffers an outage; b) network connectivity between the VM and Pub/Sub experiences a latency spike; or c) the subscriber to the message’s topic connects from a different region.

2. It is unlikely that a message published by a Compute Engine VM residing in a EU region (e.g. eu-west1) will be copied outside of the EU because Pub/Sub load balancer uses a measure of network latency (ping) when deciding whether to store a message to a different region. Network latency from a EU region to a different EU region is likely to be lower than from a EU region to a non-EU region. So even in the events of outages (i.e. 1a above) and intra-regional network latency spikes (1b above), the message is likely to stay in the EU.

3. In the unlikely event that a message is copied outside of a EU region, the message will be stored in the non-EU region for a maximum of 7 days after which is it deleted. The message retention time can be reduced to fewer than 7 days.

Answer 2

Google Cloud Pub/Sub does not support data residency. The recommended solution is to ensure you don't send any customer data using Pub/Sub. Instead, you can send pointers to the actual data stored in Datastore, GCS, etc. Those services do respect data residency and can be configured to ensure data does not leave the EU.

Answer 3

Correction: Turns out we can't actually set the data residency anywhere. The MessageStoragePolicy type is only used on the read

Answer from later 2019: Now available using UI, CLI and API

Answer from early 2019: Golang API has a MessageStoragePolicy type https://godoc.org/cloud.google.com/go/pubsub#MessageStoragePolicy

type MessageStoragePolicy struct {
// The list of GCP regions where messages that are published to the topic may
// be persisted in storage. Messages published by publishers running in
// non-allowed GCP regions (or running outside of GCP altogether) will be
// routed for storage in one of the allowed regions. An empty list indicates a
// misconfiguration at the project or organization level, which will result in
// all Publish operations failing.
AllowedPersistenceRegions []string }