how to get bigquery sql result in Spark dataframe?

0
votes

I am querying bigquery table in my dataproc code,

fetch_history_query = 'SELECT * FROM {} WHERE ({}) = ("{}")'.format(bq_table, Columns, filer_value)
print(fetch_history_query)
client = bigquery.Client()
result_history = client.query(fetch_history_query).result().to_dataframe()
print(len(result_history))

Above code create result_history as pandas.core.frame.DataFrame. How Can I get result in pyspark.sql.dataframe.DataFrame ?

1
pythonpandaspysparkapache-spark-sqlgoogle-bigquery

1 Answers

0
votes

You can use big query connector with spark.

Refer to below documentations for examples and more details:

https://cloud.google.com/dataproc/docs/tutorials/bigquery-connector-spark-example

https://github.com/GoogleCloudDataproc/spark-bigquery-connector