Data Flow Partition by Column Value Not Writing Unique Column Values to Each Folder

1
votes

I am reading an SQL DB as source and it outputs the following table.

enter image description here

My intention is to use data flow to save each unique type into a data lake folder partition probably named as specific type.

I somehow manage to create individual folders but my data flow saves the entire table with all types into each of the folders.

my data flow

enter image description here

Source

enter image description here

Window

enter image description here

enter image description here

Sink enter image description here

enter image description here

Any ideas?

2
azure-data-factoryazure-data-lake-gen2azure-data-flowexpressionbuilder
What's the output you expect? How did you set the windows sort/ windows columns? Did you check the sink output data preview? – Leon Yue
I think you need a Aggregate active get you want. – Leon Yue
@Leon Yue Invited you on chat because of the sensitivity of info – wwnde
Hi @wwnde, I'm in the chatting room – Leon Yue
chat.stackoverflow.com/rooms/230297/room-for-wwnde-and-leon-yue – wwnde

2 Answers

1
votes

I create a same csv source and it works well, please ref my example.

Windows settings: enter image description here enter image description here

Sink settings: choose the file name option like this enter image description here

Note, please don't set optmize again in sink side.

The output folder schema we can get:

enter image description here

Just for now, Data Factory Data Flow doesn't support custom the output file name.

HTH.

-1
votes

You can also try "Name folder as column data" using the OpType column instead of using partitioning. This is a property in the Sink settings.

enter image description here