Urllib2 runs fine if i run the program independently but throws error when i add it to a cronjob

1
votes
url = "www.someurl.com"

request = urllib2.Request(url,header={"User-agent" : "Mozilla/5.0"})

contentString = urllib2.url(request).read()

contentFile = StringIO.StringIO(contentString)

for i in range(0,2):
    html = contentFile.readline()

print html

The above code runs fine from commandline but if i add it to a cron job it throws the following error:

  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.6/urllib2.py", line 1186, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib64/python2.6/urllib2.py", line 1161, in do_open
    raise URLError(err)
urllib2.URLError:

I did look at some tips on the other forums and tried it but it has been of no use.

Any help will be much appreciated.

1
pythonurllib2python-2.6
I would check the following: 1. Make sure that the same python version and python path is used in both cases. Insert a print sys.version; print sys.path at the top of the program and compare 2. Compare environment variables (print os.environ). Check if a HTTP_PROXY environment variable is present. - codeape
The version is the same and yes HTTP_PROXY is present. is that the problem? - Nischal Hp
I bet it is a problem with your proxy settings. URLError: <urlopen error [Errno 3] name does not exist> - Kiwisauce
But the thing that is confusing is on the same machine how does the python script run from commandline? Shouldn the proxy cause a problem even then? or are the Os settings used only after i create a cron job? - Nischal Hp
Double check that the HTTP_PROXY environment variable is seen by the cronjob task. Do a print os.environ from the program and make sure that output is redirected so that you can inspect the value. - codeape

1 Answers

1
votes

The environment variables that were used by crontab and from the command line were different.

I fixed this by adding */15 * * * * . $HOME/.profile; /path/to/command.

This made the crontab to pick up enivronment variables that were specified for the system.