How do I Unzip and Execute a Batch Service job as part of Azure Data Factory

1
votes

Azure Data Factory can execute custom activities as Batch Service Jobs. These jobs can run from an .exe (and associated dependencies) in a storage account which are copied across prior to execution.

There is a limitation on the files in the storage account that can be used:

Total size of resourceFiles cannot be more than 32768 characters

The solution appears to be to zip the files in the storage account and unzip as part of the command. This post suggests running the Batch Service Command in Azure Data Factory as:

Unzip.exe [myZipFilename] && MyExeName.exe [cmdLineArgs]

Running this locally on a Windows 10 machine works fine. Setting this as the Command parameter on the batch service custom activity (using a Cloud Services Windows Server 2019 OS Image App Pool) results in:

caution: filename not matched: &&

It feels like something basic that I'm missing but I've tried various permutations and cannot get it to work.

4
azurecmdunzipazure-data-factoryazure-batch
Instead of zipping and unzipping files, you may also want to consider creating the resource files with the storageContainerUrl property, rather than specifying each individuality. Assuming some of the files are in the same container, this should drastically reduce the number of characters in your request. – brklein
@brklein I'm not sure I understand what you mean? In the Data Factory Custom activity I connect to the batch service and app pool then in the settings set a 'Resource linked service' to the storage account and container with a folder path for it to copy the application from. This is as granular as it appears to be? whatever it is doing behind the scenes when the activity runs to retrieve those files and put them on the batch node is essentially a black box to me. – Cargowire
That is my bad. I didn't realize you were using ADF which only exposes a subset of Batch functionality. – brklein

4 Answers

3
votes

Sharing few possibilities which could be happenening, I agree that something small is missing, if you want to share the exact commandline you are trying then feel free to share.

Few ideas which could be causing this behavioour:

  • Please try your command-line under quotes from batch point of view like: cmd /c "Unzip.exe [myZipFilename] && MyExeName.exe [cmdLineArgs]"

  • Make sure your file exist i.e. one possibility is : that in the end the command is trying to run an empty string with && i.e. cmd /c "unzip.exe "empty" && ...

Hope one of the above 2 fixes, or feel free to add more detail et. al.

2
votes

Without full knowledge of the context in which ADF runs Custom Activity Commands on a Windows Batch Service Node I changed my setup to avoid expecting Unzip.exe to exist (which it appears not to when running under cmd /c "Unzip.exe" rather than with just Unzip.exe as the command).

Now my storage account contents backing the custom activity has:

  • executable.zip (my .NET Core Console application published for windows with all dependencies)
  • unzip.exe (taken from Git Bash on my local machine)
    • including the msys-2.0.dll and msys-bz2-1.dll dependencies

The command in ADF is then:

cmd /c "Unzip.exe executable-with-deps.zip && executable.exe"
0
votes

I had the same problem and solved it with a short powershell script, that extracts my zip and start my Program. The generic script look like (called unzipandrun.ps1):

[CmdletBinding()]
param (    
    [Parameter(Mandatory = $true)][String]$zipFile,    
    [Parameter(Mandatory = $true)][String]$executable,    
    [Parameter(Mandatory = $true)][String]$prm
)
#######
# Unzip
#######
Add-Type -AssemblyName System.IO.Compression.FileSystem
function Unzip
{
    param([string]$zipfile, [string]$outpath)
    [System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $outpath)
}
"Start Unzip $zipFile" | Write-Host
Unzip $zipFile $pwd
###########
# Start exe
###########
"Start $executable $prm" | Write-Host
$cmd = "& $executable $prm"
Invoke-Expression $cmd

So if I want to start myapp.exe with parameter /t 123 on azure batch I would start:

PowerShell -NoProfile -ExecutionPolicy Bypass -Command "unzipandrun.ps1 '/t 123'"
0
votes

Another option, which doesn't require copying additional files along with zip file:

cmd /c "tar -xf MyExe.zip & MyExe.exe"

tar -xf MyExe.zip & MyExe.exe gives an error, but cmd /c "tar -xf MyExe.zip & MyExe.exe" works fine on windows 10