shell - Why do people write #!/usr/bin/env python on the first line of a Python script?

Question

It seems to me like the files run the same without that line.

Answer 1

If you have several versions of Python installed, /usr/bin/env will ensure the interpreter used is the first one on your environment's $PATH. The alternative would be to hardcode something like #!/usr/bin/python; that's ok, but less flexible.

In Unix, an executable file that's meant to be interpreted can indicate what interpreter to use by having a #! at the start of the first line, followed by the interpreter (and any flags it may need).

If you're talking about other platforms, of course, this rule does not apply (but that "shebang line" does no harm, and will help if you ever copy that script to a platform with a Unix base, such as Linux, Mac, etc).

Answer 2

That is called the shebang line. As the Wikipedia entry explains:

In computing, a shebang (also called a hashbang, hashpling, pound bang, or crunchbang) refers to the characters "#!" when they are the first two characters in an interpreter directive as the first line of a text file. In a Unix-like operating system, the program loader takes the presence of these two characters as an indication that the file is a script, and tries to execute that script using the interpreter specified by the rest of the first line in the file.

See also the Unix FAQ entry.

Even on Windows, where the shebang line does not determine the interpreter to be run, you can pass options to the interpreter by specifying them on the shebang line. I find it useful to keep a generic shebang line in one-off scripts (such as the ones I write when answering questions on SO), so I can quickly test them on both Windows and ArchLinux.

The env utility allows you to invoke a command on the path:

The first remaining argument specifies the program name to invoke; it is searched for according to the PATH environment variable. Any remaining arguments are passed as arguments to that program.

Answer 3

Expanding a bit on the other answers, here's a little example of how your command line scripts can get into trouble by incautious use of /usr/bin/env shebang lines:

$ /usr/local/bin/python -V
Python 2.6.4
$ /usr/bin/python -V
Python 2.5.1
$ cat my_script.py 
#!/usr/bin/env python
import json
print "hello, json"
$ PATH=/usr/local/bin:/usr/bin
$ ./my_script.py 
hello, json
$ PATH=/usr/bin:/usr/local/bin
$ ./my_script.py 
Traceback (most recent call last):
  File "./my_script.py", line 2, in <module>
    import json
ImportError: No module named json

The json module doesn't exist in Python 2.5.

One way to guard against that kind of problem is to use the versioned python command names that are typically installed with most Pythons:

$ cat my_script.py 
#!/usr/bin/env python2.6
import json
print "hello, json"

If you just need to distinguish between Python 2.x and Python 3.x, recent releases of Python 3 also provide a python3 name:

$ cat my_script.py 
#!/usr/bin/env python3
import json
print("hello, json")

Answer 4

In order to run the python script, we need to tell the shell three things:

1. That the file is a script
2. Which interpreter we want to execute the script
3. The path of said interpreter

The shebang #! accomplishes (1.). The shebang begins with a # because the # character is a comment marker in many scripting languages. The contents of the shebang line are therefore automatically ignored by the interpreter.

The env command accomplishes (2.) and (3.). To quote "grawity,"

A common use of the env command is to launch interpreters, by making use of the fact that env will search $PATH for the command it is told to launch. Since the shebang line requires an absolute path to be specified, and since the location of various interpreters (perl, bash, python) may vary a lot, it is common to use:

#!/usr/bin/env perl  instead of trying to guess whether it is /bin/perl, /usr/bin/perl, /usr/local/bin/perl, /usr/local/pkg/perl, /fileserver/usr/bin/perl, or /home/MrDaniel/usr/bin/perl on the user's system...

On the other hand, env is almost always in /usr/bin/env. (Except in cases when it isn't; some systems might use /bin/env, but that's a fairly rare occassion and only happens on non-Linux systems.)

Answer 5

The exec system call of the Linux kernel understands shebangs (#!) natively

When you do on bash:

./something

on Linux, this calls the exec system call with the path ./something.

This line of the kernel gets called on the file passed to exec: https://github.com/torvalds/linux/blob/v4.8/fs/binfmt_script.c#L25

if ((bprm->buf[0] != '#') || (bprm->buf[1] != '!'))

It reads the very first bytes of the file, and compares them to #!.

If the comparison is true, then the rest of the line is parsed by the Linux kernel, which makes another exec call with:

• executable: /usr/bin/env
• first argument: python
• second argument: script path

therefore equivalent to:

/usr/bin/env python /path/to/script.py

env is an executable that searches PATH to e.g. find /usr/bin/python, and then finally calls:

/usr/bin/python /path/to/script.py

The Python interpreter does see the #! line in the file, but # is the comment character in Python, so that line just gets ignored as a regular comment.

And yes, you can make an infinite loop with:

printf '#!/a\n' | sudo tee /a
sudo chmod +x /a
/a

Bash recognizes the error:

-bash: /a: /a: bad interpreter: Too many levels of symbolic links

#! just happens to be human readable, but that is not required.

If the file started with different bytes, then the exec system call would use a different handler. The other most important built-in handler is for ELF executable files: https://github.com/torvalds/linux/blob/v4.8/fs/binfmt_elf.c#L1305 which checks for bytes 7f 45 4c 46 (which also happens to be human readable for .ELF). Let's confirm that by reading the 4 first bytes of /bin/ls, which is an ELF executable:

head -c 4 "$(which ls)" | hd 

output:

00000000  7f 45 4c 46                                       |.ELF|
00000004                                                                 

So when the kernel sees those bytes, it takes the ELF file, puts it into memory correctly, and starts a new process with it. See also: How does kernel get an executable binary file running under linux?

Finally, you can add your own shebang handlers with the binfmt_misc mechanism. For example, you can add a custom handler for .jar files. This mechanism even supports handlers by file extension. Another application is to transparently run executables of a different architecture with QEMU.

I don't think POSIX specifies shebangs however: https://unix.stackexchange.com/a/346214/32558 , although it does mention in on rationale sections, and in the form "if executable scripts are supported by the system something may happen". macOS and FreeBSD also seem to implement it however.

PATH search motivation

Likely, one big motivation for the existence of shebangs is the fact that in Linux, we often want to run commands from PATH just as:

basename-of-command

instead of:

/full/path/to/basename-of-command

But then, without the shebang mechanism, how would Linux know how to launch each type of file?

Hardcoding the extension in commands:

 basename-of-command.py

or implementing PATH search on every interpreter:

python basename-of-command

would be a possibility, but this has the major problem that everything breaks if we ever decide to refactor the command into another language.

Shebangs solve this problem beautifully.

Major use case of env: pyenv and other version managers

One major use case of why you should use #!/usr/bin/env python instead of just /usr/bin/python is that of version managers with pyenv.

pyenv allows you to easily install multiple python versions on a single machine, to be able to better reproduce other projects without virtualization.

Then, it manages the "current" python version by setting its order in the PATH: e.g. as shown at apt-get install for different python versions a pyenv managed python could be located at:

/home/ciro/.pyenv/shims/python

so nowhere close to /usr/bin/python, which some systems might deal with via update-alternatives symlinks.

Answer 6

Perhaps your question is in this sense:

If you want to use: $python myscript.py

You don't need that line at all. The system will call python and then python interpreter will run your script.

But if you intend to use: $./myscript.py

Calling it directly like a normal program or bash script, you need write that line to specify to the system which program use to run it, (and also make it executable with chmod 755)

Answer 7

Technically, in Python, this is just a comment line.

This line is only used if you run the py script from the shell (from the command line). This is know as the "Shebang!", and it is used in various situations, not just with Python scripts.

Here, it instructs the shell to start a specific version of Python (to take care of the rest of the file.

Answer 8

The main reason to do this is to make the script portable across operating system environments.

For example under mingw, python scripts use :

#!/c/python3k/python 

and under GNU/Linux distribution it is either:

#!/usr/local/bin/python 

or

#!/usr/bin/python

and under the best commercial Unix sw/hw system of all (OS/X), it is:

#!/Applications/MacPython 2.5/python

or on FreeBSD:

#!/usr/local/bin/python

However all these differences can make the script portable across all by using:

#!/usr/bin/env python

Answer 9

It probably makes sense to emphasize one thing that the most have missed, which may prevent immediate understanding. When you type python in terminal you don't normally provide a full path. Instead, the executable is up looked in PATH environment variable. In turn, when you want to execute a Python program directly, /path/to/app.py, one must tell the shell what interpreter to use (via the hashbang, what the other contributors are explaining above).

Hashbang expects full path to an interpreter. Thus to run your Python program directly you have to provide full path to Python binary which varies significantly, especially considering a use of virtualenv. To address portability the trick with /usr/bin/env is used. The latter is originally intended to alter environment in-place and run a command in it. When no alteration is provided it runs the command in current environment, which effectively results in the same PATH lookup which does the trick.

Source from unix stackexchange

Answer 10

This is a shell convention that tells the shell which program can execute the script.

#!/usr/bin/env python

resolves to a path to the Python binary.

Answer 11

It's recommended way, proposed in documentation:

2.2.2. Executable Python Scripts

On BSD’ish Unix systems, Python scripts can be made directly executable, like shell scripts, by putting the line

#! /usr/bin/env python3.2

from http://docs.python.org/py3k/tutorial/interpreter.html#executable-python-scripts

Answer 12

You can try this issue using virtualenv

Here is test.py

#! /usr/bin/env python
import sys
print(sys.version)

Create virtual environments

virtualenv test2.6 -p /usr/bin/python2.6
virtualenv test2.7 -p /usr/bin/python2.7

activate each environment then check the differences

echo $PATH
./test.py

Answer 13

It just specifies what interpreter you want to use. To understand this, create a file through terminal by doing touch test.py, then type into that file the following:

#!/usr/bin/env python3
print "test"

and do chmod +x test.py to make your script executable. After this when you do ./test.py you should get an error saying:

  File "./test.py", line 2
    print "test"
               ^
SyntaxError: Missing parentheses in call to 'print'

because python3 doesn't supprt the print operator.

Now go ahead and change the first line of your code to:

#!/usr/bin/env python2

and it'll work, printing test to stdout, because python2 supports the print operator. So, now you've learned how to switch between script interpreters.

Answer 14

It seems to me like the files run the same without that line.

If so, then perhaps you're running the Python program on Windows? Windows doesn't use that line—instead, it uses the file-name extension to run the program associated with the file extension.

However in 2011, a "Python launcher" was developed which (to some degree) mimics this Linux behaviour for Windows. This is limited just to choosing which Python interpreter is run — e.g. to select between Python 2 and Python 3 on a system where both are installed. The launcher is optionally installed as py.exe by Python installation, and can be associated with .py files so that the launcher will check that line and in turn launch the specified Python interpreter version.

Answer 15

This is meant as more of historical information than a "real" answer.

Remember that back in the day you had LOTS of unix like operating systems whose designers all had their own notion of where to put stuff, and sometimes didn't include Python, Perl, Bash, or lots of other GNU/Open Source stuff at all.

This was even true of different Linux distributions. On Linux--pre-FHS[1]-you might have python in /usr/bin/ or /usr/local/bin/. Or it might not have been installed, so you built your own and put it in ~/bin

Solaris was the worst I ever worked on, partially as the transition from Berkeley Unix to System V. You could wind up with stuff in /usr/, /usr/local/, /usr/ucb, /opt/ etc. This could make for some really long paths. I have memories of the stuff from Sunfreeware.com installing each package in it's own directory, but I can't recall if it symlinked the binaries into /usr/bin or not.

Oh, and sometimes /usr/bin was on an NFS server[2].

So the env utility was developed to work around this.

Then you could write #!/bin/env interpreter and as long as the path was proper things had a reasonable chance of running. Of course, reasonable meant (for Python and Perl) that you had also set the appropriate environmental variables. For bash/ksh/zsh it just worked.

This was important because people were passing around shell scripts (like perl and python) and if you'd hard coded /usr/bin/python on your Red Hat Linux workstation it was going to break bad on a SGI...well, no, I think IRIX put python in the right spot. But on a Sparc station it might not run at all.

I miss my sparc station. But not a lot. Ok, now you've got me trolling around on E-Bay. Bastages.

[1] File-system Hierarchy Standard. https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

[2] Yes, and sometimes people still do stuff like that. And no, I did not wear either a turnip OR an onion on my belt.

Answer 16

If you're running your script in a virtual environment, say venv, then executing which python while working on venv will display the path to the Python interpreter:

~/Envs/venv/bin/python

Note that the name of the virtual environment is embedded in the path to the Python interpreter. Therefore, hardcoding this path in your script will cause two problems:

• If you upload the script to a repository, you're forcing other users to have the same virtual environment name. This is if they identify the problem first.
• You won't be able to run the script across multiple virtual environments even if you had all required packages in other virtual environments.

Therefore, to add to Jonathan's answer, the ideal shebang is #!/usr/bin/env python, not just for portability across OSes but for portability across virtual environments as well!

Answer 17

The line #!/bin/bash/python3 or #!/bin/bash/python specifies which python compiler to use. You might have multiple python versions installed. For example,
a.py :

#!/bin/bash/python3
print("Hello World")

is a python3 script, and
b.py :

#!/bin/bash/python
print "Hello World"

is a python 2.x script
In order to run this file ./a.py or ./b.py is used, you need to give the files execution privileges before hand, otherwise executing will lead to Permission denied error.
For giving execution permission,

chmod +x a.py

Answer 18

Considering the portability issues between python2 and python3, you should always specify either version unless your program is compatible with both.

Some distributions are shipping python symlinked to python3 for a while now - do not rely on python being python2.

This is emphasized by PEP 394:

In order to tolerate differences across platforms, all new code that needs to invoke the Python interpreter should not specify python, but rather should specify either python2 or python3 (or the more specific python2.x and python3.x versions; see the Migration Notes). This distinction should be made in shebangs, when invoking from a shell script, when invoking via the system() call, or when invoking in any other context.

Answer 19

It tells the interpreter which version of python to run the program with when you have multiple versions of python.

Answer 20

It allows you to select the executable that you wish to use; which is very handy if perhaps you have multiple python installs, and different modules in each and wish to choose. e.g.

#!/bin/sh
#
# Choose the python we need. Explanation:
# a) '''\' translates to \ in shell, and starts a python multi-line string
# b) "" strings are treated as string concat by python, shell ignores them
# c) "true" command ignores its arguments
# c) exit before the ending ''' so the shell reads no further
# d) reset set docstrings to ignore the multiline comment code
#
"true" '''\'
PREFERRED_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python
ALTERNATIVE_PYTHON=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
FALLBACK_PYTHON=python3

if [ -x $PREFERRED_PYTHON ]; then
    echo Using preferred python $ALTERNATIVE_PYTHON
    exec $PREFERRED_PYTHON "$0" "$@"
elif [ -x $ALTERNATIVE_PYTHON ]; then
    echo Using alternative python $ALTERNATIVE_PYTHON
    exec $ALTERNATIVE_PYTHON "$0" "$@"
else
    echo Using fallback python $FALLBACK_PYTHON
    exec python3 "$0" "$@"
fi
exit 127
'''

__doc__ = """What this file does"""
print(__doc__)
import platform
print(platform.python_version())

Answer 21

this tells the script where is python directory !

#! /usr/bin/env python