Azure Batch mass untar on attached storage

Question

I need to untar hundreds of tar files of a size between 30 and 140GB as soon as they become available. I've created a pool with (currently) 5 "standard a1" single core Linux nodes, " microsoft-azure-batch centos-container 7-4 (latest)"

The batch has an associated storage account. The storage contains the tar files which I wish to untar into that same store. There is also a job called, "untarjob"

Since the batch allows for an association to a store I made an assumption that would be naturally available to each compute node. However, the assumption now is perhaps that isn't true? As such I'm trying to mount the store and run the unix tar command to untar as shown below.

This isn't working:

 az batch task create --task-id t80000 --job-id untarjob --command- 
 line "mount -t cifs //dummy.file.core.windows.net/s01 /tarfiles -o  
 vers=3.0,username=dummy,password=Pv4<snipp>N8Imu34reseEEe==,dir_mode=0777, 
 file_mode=0777,sec=ntlmssp ; tar -xvf /tarfiles/80000.tar"

Could you tell me how I could solve this please?

Many thanks

Kevin

fpark fpark · Accepted Answer · 2018-06-06T16:03:10

First, you probably shouldn't use a container-specific VM image if your tasks don't need containers and need to run directly on the host.

For the issue, wrap your command in a shell invocation:

/bin/sh -c 'sudo mkdir /tarfiles; sudo mount ...; tar ...'

Note the single quotes since you're using double quotes for the --command-line arg. Also, if your command needs to run privileged entirely, consider using user identities to avoid having to use sudo.

Azure Batch mass untar on attached storage

1 Answers