Pubsub emulator with apache beam python sdk

2
votes

Is it possible to use topic created in pub sub emulator as data source with direct runner in apache beam python?

I've set the environment variables(within the script as well):

set PUBSUB_EMULATOR_HOST=localhost:8085
set PUBSUB_PROJECT_ID=<proj_id>

It doesn't seem to be connecting to it because even if I set the emulator host to anything else it gives the same error and if not set, it looks for credentials.

Tried reading messages using topic and subscription id(pull).

Using input topic :

File "lib\site-packages\apache_beam\runners\direct\transform_evaluator.py", line 646, in get_subscription topic_name = sub_client.topic_path(project, short_topic_name) AttributeError: 'SubscriberClient' object has no attribute 'topic_path'

Using subscription id(pull) :

lib\site-packages\apache_beam\runners\direct\transform_evaluator.py", line 694, in _read_from_pubsub sub_client.api.transport.channel.close() AttributeError: 'SubscriberClient' object has no attribute 'transport'

For java, use of emulator has been mentioned in the docs(setPubsubRootUrl(String)) but couldn't find anything for python.

1
pythonapache-beamgoogle-cloud-pubsub
Did you figure out what the root cause was? – Hartimer
This seems related issues.apache.org/jira/browse/BEAM-11318 – Hartimer

1 Answers

0
votes

You have to set the environment variables on the command:

dev_appserver.py app.yaml --env_var PUBSUB_EMULATOR_HOST=${PUBSUB_EMULATOR_HOST}