U-SQL Custom reducer could not find file

Question

I'm writing a custom reducer in U-SQL that requires a lookup file as an argument into the constructor of the reducer. The problem is when I do the local run, it can not locate the lookup file that's included in the project under /data/ directory. Here is the code snippet:

DECLARE @EventType string = @"/data/EventType.csv"; 
...
@red =
REDUCE @filtered
ON id
PRODUCE 
...
USING new USQLApplication1.MyReducer(@EventType);

And here is the error message I'm getting. Whatever the solution, I'd like it to work not just in the local run, but also in the azure cloud. Thanks.

Running step 1 of 1

Unhandled Exception: System.Exception: {"diagnosticCode":195887132,"severity":"E rror","component":"RUNTIME","source":"User","errorId":"E_RUNTIME_USER_UNHANDLED_ EXCEPTION_FROM_USER_CODE","message":"An unhandled exception from user code has b een reported when invoking the method 'Constructor' on the user type 'USQLApplic ation1.MyReducer'","description":"Unhandled exception from user code: \"Could not find file 'D:\Users\jbfp\Documents\Visual Studio 2015\Projects\USQL Application1\USQLApplication1\bin\Debug\ECB1C361139EE2D8\Script2_2E96CB20AE 368E7B\EventType.csv'.\"

Answer 1

I can't comment so I am going to post my question this way.

Are you using DEPLOY RESOURCE to deploy your lookup file to your working directory? The file will need to be deployed in each of the vertices that your operation is taking place. Hence an explicit deployment is necessary. There may be some quirks with the local deploy but let's clarify that first.

Answer 2

To extend on chi's answer.

U-SQL operates on two store "layers" (for lack of a better term):

1. The globally persisted store layer. Mainly Azure Data Lake Storage or Windows Azure Blob Store.
2. Each node that runs a user-defined function or operator also provides access to the node's local file system's working directory.

Custom code cannot access the globally persisted store layer directly with file IOs. Instead U-SQL provides the so called UDO model that will provide parts of the file to each node through a well-defined interface.

So how can you access files and resources stored in the data lake inside your user code if you want to read the file content directly? You need to copy (deploy) the file into every node's local working directory. This is done by either using the ADDITIONAL_FILE option on CREATE ASSEMBLY, or by using the DEPLOY RESOURCE statement that chi mentioned.

Now having said that, local execution mode seems to have a bug in this area that we have fixed but not yet released (it will probably be released by the end of the month). If you need the fix earlier, please let us know and we will see if we can deploy a hotfix.