Spark - there is no type "Dataframe" in Java

0
votes

I am going through this and this . I clearly see the type"Dataframe" is defined for scala and not for Java. Does that mean type "Dataframe" is not technically supported but rather use Dataset[T] insetad ?

And also, from this , please share inputs about what does it mean - Does that signify the same fact Java doesnt have the construct for dataframe ?

Scala Dataset[T] & DataFrame (alias for Dataset[Row])

Java Dataset[T]

1
scalaapache-sparkapache-spark-sql
All Scala classes are visible to Java and vice versa. Have you added the spark and scala libraries to your Maven/Gradle/SBT? E.g. for Maven to your pom.xml file. - Danny Varod
I did , i was referring to the javadoc and scaladoc and also the other question wrt DataFrame (alias for Dataset[Row]) - it is not mentioned in Java - Nag
@DannyVarod DataFrame is a type alias, not a class. - Alexey Romanov
@AlexeyRomanov I know, for Dataset[Row], forgot that java can't see these though. - Danny Varod
@Nag why not use a scala class in your code? It will still compile into your jar, however, you will be able to use all the scala "goodies" such as lambdas, implicits, typetags, spark-encoders etc. - Danny Varod

1 Answers

1
votes

Java has no concept of a type alias, so in Java, you would need to use

Dataset<org.apache.spark.sql.Row>

to refer to a Dataframe.