Spark ML (Dataframe) and dimension reduction

Question

There does not appear to be classes available to perform dimension reduction if using the newer Dataframe APIs (i.e., not RDD). Nor is there any documentation (https://spark.apache.org/docs/latest/ml-guide.html).

Can someone tell me the approach for dimension reduction using classes from spark.ml (Dataframe) packages rather than spark.mllib (RDD)...? Is it possible?

Answer 1

PCA is available for new ML API. You'll find details in:

• ML guide
• API docs (Scala, Python)