I have a function that returns very high volume of data. I am loading the output of the function into a target table. So this is how it goes:-
.append async MyTable with(tags=...) <| MyFunction(..arguments..)
Because of the very high volume , I would like to fire multiple instance of the query (function basically) and load into the same target table -- I know some column which is a good candidate for uniform distribution of data. Lets call it dept no , which I can pass to the function or filter on top of the function. So this is something I am trying to do :-
.append async MyTable with(tags=...) <| MyFunction(..arguments..) | where deptno == 11
.append async MyTable with(tags=...) <| MyFunction(..arguments..) | where deptno == 12
.
.
and so on
So lets say if I have 10 of these dept numbers, I will want to fire up 10 different ingestions from 10 different queries.
My question is , is there a way to automatically do it in ADX instead of firing 10 different control commands manually?
I read about partition operator which seems to be doing something logically similar but I am not sure if it runs entire thing as part of a single query or actually creates multiple queries with their own OperationId -- because if it's a single query , if some branch fails , entire operation will fail and nothing will get loaded to the target table , wasting even those query branches which have succeeded.
