Environment Variables when python script run by cron

28
votes

I have been looking at other stack overflow questions but couldn't get any to work. I have a python script which uses environment variables. This script works exactly as planned when run directly however, I would like to run it as a cron job every minute for the time being.

Currently in my cron.d directory I have a file called scrapers containing:

* * * * * root /usr/bin/python3.5 /code/scraper.py

This runs the python script but the script fails as in the script I use two environment variables.

I read I should add SHELL=/bin/bash to the cron file so I did but this didn't help.

SHELL=/bin/bash
* * * * * root /usr/bin/python3.5 /code/scraper.py

Then I read

In the crontab, before you command, add . $HOME/.profile. 

SHELL=/bin/bash
* * * * * . $HOME/.profile; root /usr/bin/python3.5 /code/scraper.py

but this caused the cron to stop running altogether. What is the best way of 'sending' the env variables to the cron?

5
pythoncronenvironment-variables
try adding source ~/.bashrc && command or the file where your env variables are declared and see if it works. - franklinsijo
@franklinsijo Sorry, I'm not quite sure what I should be doing. The env variables are declared in docker-compose file when setting up the containers. - user6723321
and those variables are the ones to be used by scraper.py? - franklinsijo
@franklinsijo yes. - user6723321
AFAIK, env variables has to be set in a file (profile files) to be available even after the session is closed. Or you can use export command to set the variable for that session. - franklinsijo

5 Answers

10
votes

Instead of executing the whole ~/.profile what I'd do is move the variables that must be shared between your cron jobs and the account that has the profile, then I'd source these both in ~/.profile and in the cron job.

The last attempt you show in the question is not properly formatted. The user id should be coming right after the scheduling information, but you've added the sourcing of the profile before the user id, which surely cannot work.

Here's an example setup that I've tested here:

*/1 * * * * someuser . /tmp/t10/setenv && /usr/bin/python /tmp/t10/test.py

I've set it to execute every minute for testing purposes. Replace someuser with something that makes sense. The /tmp/t10/setenv script I used had this:

export FOO=foovalue
export BAR=barvalue

The /tmp/t10/test.py file had this:

import os

print os.environ["FOO"], os.environ["BAR"]

My cron emails me the output of the scripts it runs. I got an email with this output:

foovalue barvalue
4
votes

You can set the env variable inline:

* * * * * root ENV_VAR=VALUE /usr/bin/python3.5 /code/scraper.py

Another way is use honcho that you can pass a file with env variables.

honcho -e /path/to/.env run /code/scraper.py

2
votes

You can specify your two environment variables by this:

* * * * * root env A=1 B=2 /usr/bin/python3.5 /code/scraper.py

env is a system program that runs a specified program with additional variables:

$ env A=1 B=2 /bin/sh -c 'echo $A$B'  # or just 'sh': would search in $PATH
12
1
votes

You can add it to the top of your crontab and keep it out of version control. Let's say the environment variable causing you difficulty is export DJANGO_SECRET_KEY="FOOBAR_1241243124312341234":

crontab

DJANGO_SECRET_KEY="FOOBAR_1241243124312341234"

SCRIPT_NAME = my_cool_script
20 21 * * 1-5 bash ~/git_repo/cronjobs/$SCRIPT_NAME.sh 2&>1 | tee ~/git_repo/cronjobs/logs/$SCRIPT_NAME.log

my_cool_script.sh

#!/usr/bin/env bash
~/anaconda3/envs/django/bin/python ~/git_repo/django_project/manage.py run_command

This has worked well for me when the environment variables in question need to be kept secret and the loading of existing .bashrc does not play nice for whatever reason.

0
votes

This is one of the approach I like, write a script to set environment and execute the script with its parameters as its parameters

set_env_to_process.sh

#!/usr/bin/env bash
echo "TEST_VAR before export is: <$TEST_VAR>"

export TEST_VAR=/opt/loca/netcdf
echo "TEST_VAR after export is: <$TEST_VAR>"
export PATH=$PATH:/usr/bin/python3.5
export PYTHTONPATH=$PYTHONPATH:/my/installed/pythonpath

# execute command and its parameters as input for this script
if [ $# -eq 0 ]; then
    echo "No command to execute"
else
    echo "Execute commands with its parameters: $@"
    eval $@
fi

usage

/usr/bin/python3.5 /code/scraper.pyare taken as input for set_env_to_process.sh set_env_to_process.sh set the correct env for script to run

It could be used as command line, cron, sudo, ssh to setup env

 * * * * * root set_env_to_process.sh /usr/bin/python3.5 /code/scraper.py