1
votes

I have this git repo structure:

.gitignore
JSONs/subdirA/some.json
JSONs/subdirB/other.json
MyPackage/__init__.py
MyPackage/myModule.py

How do I properly pack the JSONs folder into MyPackage/JSONs, without moving it there permanently (mostly because customers use this git repo directly for non-python usage, and the folder at the top of the repo is easy/intuitive... But now I also want to release this same dir into my PyPi package)?

I've tried adding it to the MANIFEST.in and then playing with data_files in setup.py as well as package_data... But to no avail. Maybe some .pyc or cached build files got the best of me... But I haven't figured it out from all the other (not quite) duplicate questions as they don't specifically call out their directory structure and desired final location.

I've tried resorting to os.walk and shutil.copy before the call to setup and then deleting that directory after setup... While it seems to work locally, when pushing to our local devpi package server, something goes wrong. Is my goal totally off the radar for setuptools/pip ideals??? Or am I just missing some key understanding? Please enlighten me!

1
Why not simply using symlinks to make it look like this JSONs directory is inside MyPackage? Then the rest is pretty standard (here for example). Do not use data_files, use MANIFEST.in and package_data. - sinoroc
can symlinks be committed to GIT though? In a cross-platform way (The python package works on Windows primarily, but the C code that uses the JSONs works on linux) - nmz787
I guess it does. But otherwise you can use setuptools's package_dir to rewrite the path to the JSONs directory. Something like package_dir = {'MyPackage.JSONs': 'JSONs'}, assuming your setup.py is in the same directory as .gitignore (the root directory). -- docs.python.org/3/distutils/… - sinoroc
Thanks, I'll give package_dir a try now. The confusing thing about the page you linked earlier is, the data files are inside the package already, while in my case they're not. So my earlier attempts were leaving the JSONs folder in either sys.prefix or site-packages depending on what I was trying. - nmz787
Yeah, that's the whole point of package_dir, it will transform ./JSONs into MyPackage/JSONs, when it does the build, i.e. in the wheel it has the new path, but in the sdist it still has the old structure. - sinoroc

1 Answers

4
votes

Something like the following could help:

First we need to make sure that the json files are added to the source distribution.

MANIFEST.in:

recursive-include JSONs *.json

Then in the actual setup script, the list of packages has to be modified on the fly to take into account the target package structure.

setup.py:

#!/usr/bin/env python3

import setuptools

PACKAGES = (
    setuptools.find_packages(exclude=['JSONs*'])
    +
    [
        f'MyPackage.{package}'
        for package
        in setuptools.find_namespace_packages(include=['JSONs*'])
    ]
)

setuptools.setup(
    packages=PACKAGES,
    package_dir={
        'MyPackage.JSONs': 'JSONs',
    },
    include_package_data=True,
    #
    name='Something',
    version='1.2.3',
)

JSONs/subdirA/some.json :

{"Marco": "Polo"}

Such package data can be read like so:

MyPackage/myModule.py:

import pkgutil

print(pkgutil.get_data('MyPackage', 'JSONs/subdirA/some.json').decode())

And use it like in the following:

$ python -m pip install .
$ # Move to another directory to prevent that the current working directory
$ # ... overshadows the installed project
$ cd ..
$ python -m MyPackage.myModule
{"Marco": "Polo"}