Create a Spark Data Frame with only one column having number from 1 to 1 million

1
votes

How can I create a Spark Data Frame with only one column having numbers from 1 to 10 million?

I have tried using Python to create a list of numbers ranging from 1 to 10 million but my solution is failing. Please advise, thank you in advance.

I am using below query:

df_base1 = sqlc.range(0, 1000000)

But it's failing:

An error was encountered: Invalid status code '500' from http://:8999/sessions/401/statements with error payload: "java.lang.IllegalStateException: RPC channel is closed."

1
pythonpython-3.xapache-sparkpysparkspark-dataframe
Please provide a code example of your current solution buddy. – Rann Lifshitz
There is nothing wrong with your code. It is the right way to do it. The exception is probably a result of something else. Could you post full traceback, please? – Alper t. Turker

1 Answers

0
votes

df_base1 = sqlc.range(0, 1000000)

Works fine. There was some environment issue.