How to get the size of an RDD in Pyspark?

Question

I am relatively new to Apache Spark and Python and was wondering how to get the size of a RDD. I have a RDD that looks like this:

[[‘ID: 6993.1066',
  'Time: 15:53:43',
  'Lab: West',
  'Lab-Tech: Nancy McNabb, ',
  '\tBob Jones, Harry Lim, ',
  '\tSue Smith, Will Smith, ',
  '\tTerry Smith, Nandini Chandra, ',
  ]]

Is there a method or function in pyspark that can give the size how many tuples in a RDD? The one above has 7.

Scala has something like: myRDD.length.

Bala Bala · Accepted Answer · 2018-02-21T07:29:29

For RDD individual element's size, this appears to be the way

>>> rdd = sc.parallelize([(1,2,'the'),(5,2,5),(1,1,'apple')])
>>> rdd.map(lambda x: len(x)).collect()
[3, 3, 3]

For overall element count within RDD

>>> rdd.count()
3

How to get the size of an RDD in Pyspark?

1 Answers