Can we process multiple rows using U-SQL User defined operators?

Question

I was trying to develop U-SQL User defined operators using this link. It looks like we can read one row, process it and write it as a single row using UDO.

In my scenario I have to read multiple consecutive rows and write multiple consecutive rows and that seems not possible using the help provided in blog.

In another scenario, I have to process single row and break into multiple and then write to output.

I am wondering if it is possible to process multiple rows using U-SQL UDO or if there is any other way to do it in U-SQL?

Answer 1

You can write a custom applier to take a single row and return several rows. You invoke it with CROSS APPLY.

You can write a custom reducer (or a user-defined aggregator) to take several rows (cells) and return a single row (cell).

What do you want to do by reading several rows, see them all and then return several rows? Would that be similar to a self-join (you could use a combiner)?