Run a python script on Azure Batch

Question

Iam trying to execute a python script on azure batch which is a linux dsvm so that the script can install python packages and then execute the python script.

Below is the code i used:

try:
   from pip import main as pipmain
except ImportError:
   from pip._internal import main as pipmain

try:
    import pandas as pd
except:

   pipmain(['install', 'pandas',"])

import pandas

When i run the python script on azure Batch command line , the pool task errors out at the last statement(import pandas) eventhough i can see in the stdout log file that the pandas, numpy etc packages are installed.

It seems that the packages are installed at some other location and while trying to import it is trying to import from some other location. It gives the error ImportError: No module named pandas in the stderr.txt file on the azure batch pool tasks.

The reason why iam trying to install python packages and importing it the same script is because the azure batch command line doesnt allow me to write 2 commands , something like

pip install pandas
python test.py

where it first install the packages and then invokes script where it just does the import of pandas library.

I have also used the command in the pip install pandas and pip install --install-option="--prefix=$AZ_BATCH_TASK_WORKING_DIR" pandas at the start task of the batch pool. AZ_BATCH_TASK_WORKING_DIR as per my understanding is working directory to which the task and script has access when the task batch runs

Is there a way i run the python script successfully on Azure Batch. At the momemt iam only running one command: import pandas

With respect to "The reason why iam trying to install python packages and importing it the same script is because the azure batch command line doesnt allow me to write 2 commands", you should be able to specify multiple commands by either embedding them in a batch/shell script or using standard && operators. Also make sure you prepend your tasks with '/bin/bash -c' or 'cmd /c' to open up a command shell. — brklein
Thanks brklein '/bin/bash command worked. Is there a way to install pip on the azure batch. I had installed it by using ssh into the nodes of the pool and then ran the pip command to install the libraries in the azure batch command line. — Matt_85
You can add 'curl bootstrap.pypa.io/get-pip.py -o get-pip.py && python get-pip.py' — brklein
Hi brklein, i tried /bin/bash -c "curl bootstrap.pypa.io/get-pip.py -o get-pip.py && python get-pip.py && pip install pandas". But the pip doesnt get installed. — Matt_85

fpark fpark · Accepted Answer · 2019-04-18T17:34:45

You need to provide a inline shell script to run your multiple commands and take advantage of shell expansion. Please see this doc. You'll want to run your two commands like:

/bin/bash -c "pip install pandas && python test.py"

Additionally, tasks are run under context-specific directories (i.e., a start task runs in the start task directory versus a normal task will run in a different directory, although $AZ_BATCH_TASK_WORKING_DIR is named the same) and user identities can also modify the user context for which a task is run.

Run a python script on Azure Batch

2 Answers