I just got introduced to Spark SQL higher order functions transform(), filter() etc. I searched online, but couldn't find much advanced use-cases leveraging these functions.
Can anyone please explain transform() with a couple of advanced real-life use-case using sql query. Does it always need to work on nested complex types (arrays, struct etc) ? Or can it be used to process simple data-type records as well ?
Any help is appreciated.
Thanks
