Azure Batch Pool Start up task to download resource file from Blob FileShare

Question

I'm new to Azure, and I'm trying to build a simple Azure batch case here.

I'm stacking at the Batch Pool Start up task...

I created a batch account and a storage account in East US, and then I created a general -- Fileshare in the storage account with a container. I manually updated a file called Test.txt.

What I want to do is to ask the Batch Pool to download this file at the startup task.

So Code goes:

string storageConnectionString = String.Format("DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}", StorageAccountName, StorageAccountKey);

// Retrieve the storage account
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);
CloudFileClient fileClient = storageAccount.CreateCloudFileClient();
SharedAccessFilePolicy fileShareConstraint = new SharedAccessFilePolicy
{
    SharedAccessExpiryTime = DateTime.UtcNow.AddHours(48),
    Permissions = SharedAccessFilePermissions.Read,
};
var fileShare = fileClient.GetShareReference(inputContainerName);
var rootDir = fileShare.GetRootDirectoryReference();
var testFile = rootDir.GetFileReference("test.txt");
var sasUrl = fileShare.GetSharedAccessSignature(fileShareConstraint);
var fileUrl = string.Format("{0}{1}", testFile.StorageUri.PrimaryUri, sasUrl);

var list = new List<ResourceFile>();
var testResourceFile = new ResourceFile(fileUrl, "test.txt");

list.Add(testResourceFile );
await CreatePoolAsync(batchClient, PoolId, list);

Then the CreatePoolAsync method:

private static async Task CreatePoolAsync(BatchClient batchClient, string poolId, IList<ResourceFile> resourceFiles)
{

    Console.WriteLine("Creating pool [{0}]...", poolId);

    // Create the unbound pool. Until we call CloudPool.Commit() or CommitAsync(), no pool is actually created in the
    // Batch service. This CloudPool instance is therefore considered "unbound," and we can modify its properties.

    //if(await batchClient.PoolOperations.GetPoolAsync(poolId) == null)
    //{

    CloudPool pool = batchClient.PoolOperations.CreatePool(
        poolId: poolId,
        targetDedicated: 1,                                                         // 3 compute nodes
        virtualMachineSize: "small",                                                // single-core, 1.75 GB memory, 225 GB disk
        cloudServiceConfiguration: new CloudServiceConfiguration(osFamily: "4"));   // Windows Server 2012 R2
    pool.MaxTasksPerComputeNode = 2;

    // Create and assign the StartTask that will be executed when compute nodes join the pool.
    // In this case, we copy the StartTask's resource files (that will be automatically downloaded
    // to the node by the StartTask) into the shared directory that all tasks will have access to.
    pool.StartTask = new StartTask
    {
        // Specify a command line for the StartTask that copies the task application files to the
        // node's shared directory. Every compute node in a Batch pool is configured with a number
        // of pre-defined environment variables that can be referenced by commands or applications
        // run by tasks.

        // Since a successful execution of robocopy can return a non-zero exit code (e.g. 1 when one or
        // more files were successfully copied) we need to manually exit with a 0 for Batch to recognize
        // StartTask execution success.

        CommandLine = "cmd /c (robocopy %AZ_BATCH_TASK_WORKING_DIR% %AZ_BATCH_NODE_SHARED_DIR%) ^& IF %ERRORLEVEL% LEQ 1 exit 0",
        //CommandLine = "cmd /c net use E: \\krisblob2.file.core.windows.net\\krisfilecontiner1 /u:krisblob2 aqTFKyPqcpeI3BrEnlx8RTBAmDaN5FK+mxpBtdgn3v6IT+IbPgDhVU4ojRA1wAmMpYPEHQ9Gzh/A1mAHtxNs+A==",
        //CommandLine = $@"cmd /c net use Z: \\{StorageAccountName}.file.core.windows.net\krisfilecontiner1 /u:{StorageAccountName} {StorageAccountKey}",
        //CommandLine = "cmd /c %AZ_BATCH_TASK_WORKING_DIR%\\WinPcap_4_1_3.exe /passive",
        ResourceFiles = resourceFiles,
        WaitForSuccess = true
    };
}     

The inputcontainer was the name I gave to the container in the file share.

When I ran the code, the startup task always failed with error:

BlobDownloadMiscError Message  

Miscellaneous error encountered while downloading one of the specified Azure Blob(s) Details   The value for one of the HTTP headers is not in the correct format. RequestId:944807de-001a-00bb-73ae-4ac746000000 Time:2016-11-30T02:04:59.8679984Z

Could anyone help me to solve this problem?

Thanks!

Answer 1

Azure Batch resource files can only be sourced from Azure Blob Storage and not Azure File Storage.

You will need to net use mount the share and copy the files manually or move your files to Azure Blob storage instead.