Sometimes I download the python source code from github
and don't know how to install all the dependencies. If there is no requirements.txt
file I have to create it by hands.
The question is:
Given the python source code directory is it possible to create requirements.txt
automatically from the import section?
14 Answers
You can use the following code to generate a requirements.txt file:
pip install pipreqs
pipreqs /path/to/project
more info related to pipreqs can be found here.
Sometimes you come across pip freeze
, but this saves all packages in the environment including those that you don't use in your current project.
In my case, I use Anaconda, so running the following command from conda terminal inside my environment solved it, and created this requirements.txt
file for me automatically:
conda list -e > requirements.txt
This was taken from this Github link pratos/condaenv.txt
If an error been seen, and you are using anaconda, try to use the .yml
option:
conda env export > <environment-name>.yml
For other person to use the environment or if you are creating a new enviroment on another machine:
conda env create -f <environment-name>.yml
Firstly, your project file must be a py file which is direct python file. If your file is in ipynb format, you can convert it to py type by using the line of code below:
jupyter nbconvert --to=python
Then, you need to install pipreqs library from cmd (terminal for mac).
pip install pipreqs
Now we can create txt file by using the code below. If you are in the same path with your file, you can just write ./ . Otherwise you need to give path of your file.
pipreqs ./
or
pipreqs /home/project/location
That will create a requirements.txt file for your project.
Make sure to run pip3 for python3.7.
pip3 freeze >> yourfile.txt
Before executing the above command make sure you have created a virtual environment.
python3:
pip3 install virtualenv
python3 -m venv <myenvname>
python2:
pip install virtualenv
virtualenv <myenvname>
After that put your source code in the directory. If you run the python file now, probably it won't launch if you are using non-native modules. You can install those modules by running pip3 install <module>
or pip install <module>
.
This will not affect you entire module list except the environment you are in.
Now you can execute the command at the top and now you have a requirements file which contains only the modules you installed in the virtual environment. Now you can run the command at the top.
I advise everyone to use environments as it makes things easier when it comes to stuff like this.
If Facing the same issue as mine i.e. not on the virtual environment and wants requirements.txt for a specific project or from the selected folder(includes children) and pipreqs is not supporting.
You can use :
import os
import sys
from fuzzywuzzy import fuzz
import subprocess
path = "C:/Users/Username/Desktop/DjangoProjects/restAPItest"
files = os.listdir(path)
pyfiles = []
for root, dirs, files in os.walk(path):
for file in files:
if file.endswith('.py'):
pyfiles.append(os.path.join(root, file))
stopWords = ['from', 'import',',','.']
importables = []
for file in pyfiles:
with open(file) as f:
content = f.readlines()
for line in content:
if "import" in line:
for sw in stopWords:
line = ' '.join(line.split(sw))
importables.append(line.strip().split(' ')[0])
importables = set(importables)
subprocess.call(f"pip freeze > {path}/requirements.txt", shell=True)
with open(path+'/requirements.txt') as req:
modules = req.readlines()
modules = {m.split('=')[0].lower() : m for m in modules}
notList = [''.join(i.split('_')) for i in sys.builtin_module_names]+['os']
new_requirements = []
for req_module in importables:
try :
new_requirements.append(modules[req_module])
except KeyError:
for k,v in modules.items():
if len(req_module)>1 and req_module not in notList:
if fuzz.partial_ratio(req_module,k) > 90:
new_requirements.append(modules[k])
new_requirements = [i for i in set(new_requirements)]
new_requirements
with open(path+'/requirements.txt','w') as req:
req.write(''.join(new_requirements))
P.S: It may have a few additional libraries as it checks on fuzzylogic.
I blindly followed the accepted answer of using pip3 freeze > requirements.txt
It generated a huge file that listed all the dependencies of the entire solution, which is not what I wanted.
So you need to figure out what sort of requirements.txt you are trying to generate.
If you need a requirements.txt file that has ALL the dependencies, then use the pip3
pip3 freeze > requirements.txt
However, if you want to generate a minimal requirements.txt that only lists the dependencies you need, then use the pipreqs package. Especially helpful if you have numerous requirements.txt files in per component level in the project and not a single file on the solution wide level.
pip install pipreqs
pipreqs [path to folder]
e.g. pipreqs .
pipreqs . --force --ignore=tests (Overwrites exisiting requirements.txt, ignores the tests directory)
If you have installed many dependencies in your system and you need requirements.txt for a specific project, you can install first pipreqs:
$ pip install pipreqs
and execute the below command under the project folder.
$ pipreqs
This command will generate requirements.txt file for the particular project.
Simple Pythonic Way
To get a list of all the REQUIREMENTS in a standard requirements.txt
file, you can use the following command.
pip freeze > requirements.txt
Now, this should automatically create a standard requirements file with all of the packages installed alongside their corresponding versions.
Pretty Print on Terminal
If you just want to get a pretty print on the terminal you can use the following approach.
pip list
This lists all of the installed packages, in a pretty print format.
Custom Dependency
If you have a project folder like say, a Github Repo, and you want to get a custom requirements.txt for project You can use the following Package.
https://pypi.org/project/pipreqs/ pipreqs
Usage
$ pipreqs /home/project/location
Successfully saved requirements file in /home/project/location/requirements.txt
Contents of requirements.txt
wheel==0.23.0
Yarg==0.1.9
docopt==0.6.2
pip freeze > requirements.txt
will save all your python libraries with current version intorequirements.txt
file – Shaikhulpip freeze
will dump current versions ofall
the installed modules on that system irrespective of there usage in the current project. b) pip will only list modules that have been installed via pip – akskap