Error in pyspark word count: ValueError: empty separator

0
votes

Here is the s84.txt which is on hdfs and I want to do wordcount on it:

[paslechoix@gw03 ~]$ hdfs dfs -cat s84.txt Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s,

when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting,

remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus

PageMaker including versions of Lorem Ipsum. 

s84RDD = sc.textFile("p84.txt") 
nonempty_lines = s84RDD.filter(lambda x: len(x) > 0)
words = nonempty_lines.flatMap(lambda x: x.split("")) 
wc = words.map(lambda x: (x,1)).reduceByKey(lambda x, y: x+y).map(lambda x: (x[1], x[0])).sortByKey(False)

Error:

ValueError: empty separator

What am I missing here? Thank you very much.

1
pythonsplitpyspark
What are you trying to split() on? An empty string is not a valid argument, no argument defaults to whitespace, any explicit argument is used for the split. Note: your test for an empty line may not be correct. BTW you show s84.txt but are loading p84.txt. – AChampion
Thank you AChampion for noticing the missing space and the confusing file name – Choix

1 Answers

4
votes

From the official docs:

Return a list of the words in the string, using sep as the delimiter string.

split() needs a separator. You are using split("") with argument as "" which is an empty string.

>>> '1,2,3'.split(',')
['1', '2', '3']

>>> '1 2 3'.split()
['1', '2', '3']

>>> "Lorem ipsum sit dolor amet".split("")
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    x.split("")
ValueError: empty separator

>>> "Lorem ipsum sit dolor amet".split(" ")
['Lorem', 'ipsum', 'sit', 'dolor', 'amet']