How do I pip install Linux packages on a Windows machine (for AWS Lambda)?

Question

I'm trying to write a Python AWS Lambda script. The Python code works locally in Windows, but it's using the Windows packages installed through pip. When uploading to AWS Lambda, I need to include the Linux packages.

For example, when I run pip install pandas I get:

Downloading pandas-1.0.1-cp37-cp37m-win_amd64.whl

But what I need (for uploading to AWS Lambda) is:

pandas-1.0.1-cp37-cp37m-manylinux1_x86_64.whl

My Attempt

I have tried to use Docker to simulate a Linux environment in Windows. My idea is to pip install the Linux packages in Docker and then copy them to my local machine. I think this can be done through a Docker Volume. I have tried to do this using the Dockerfile:

FROM python:3.7-slim-buster

WORKDIR /usr/src/app

# Download python packages to /usr/src/app/lib
RUN mkdir -p /usr/src/app/lib
RUN pip3 install pandas -t /usr/src/app/lib

# Copy the python pacakges to local machine
VOLUME host:/myvol
RUN mkdir /myvol
COPY /usr/src/app/lib /myvol

But when I run docker build I get the error:

COPY failed: stat /var/lib/docker/tmp/docker-builder233015161/usr/src/app/lib: no such file or directory

I don't think COPY will run with full paths (assuming you're running docker build . -t some_image), since it will just look in the current build context. So even if /usr/src/app/lib did exist in your windows environment, it won't find it — C.Nivs
/usr/src/app/lib is a path inside the Docker container. The pip3 install command works fine. I'm trying attempting to copy out the folder inside the Docker container (/usr/src/app/lib) to the Windows folder ('host'). The 'host' directory does exist. — maurera
@maurera no, COPY <host_path> <container_path> is how the command works. You are looking for /usr/src/app/lib on your host machine the way it's currently written — C.Nivs
You are probably looking for RUN mv /usr/src/app/lib /host/. I'll check to see if that will actually work, because I'm doubtful tbh — C.Nivs

maurera maurera · Accepted Answer · 2020-03-09T17:39:21

Thanks to explanation from @C.Nivs, this can be done using Docker interactively:

First run docker interactively with docker run -it python:3.6-slim bash. Then create a folder 'target' and pip install into it (this installs the linux packages). Make note of the container id (my command line shows root@31d9be68deec:/#. The container id is 31d9be68deec)

mkdir /target 
pip install pandas -t /target

Then open a new command prompt and use docker cp to copy files from container to local. The following copies from the target folder in the container to the local folder libs.

docker cp <container_id>:/target libs

That's it. The python packages are now available in the local folder libs.

Note from @C.Nivs: "COPY doesn't do what it's name might suggest. COPY is a build step that copies files from the build context (where the image is being built) to the image itself. It doesn't allow things to go the other way".

How do I pip install Linux packages on a Windows machine (for AWS Lambda)?

1 Answers