How to include files outside of Docker's build context?

Question

How can I include files from outside of Docker's build context using the "ADD" command in the Docker file?

From the Docker documentation:

The path must be inside the context of the build; you cannot ADD ../something/something, because the first step of a docker build is to send the context directory (and subdirectories) to the docker daemon.

I do not want to restructure my whole project just to accommodate Docker in this matter. I want to keep all my Docker files in the same sub-directory.

Also, it appears Docker does not yet (and may not ever) support symlinks: Dockerfile ADD command does not follow symlinks on host #1676.

The only other thing I can think of is to include a pre-build step to copy the files into the Docker build context (and configure my version control to ignore those files). Is there a better workaround for than that?

Answer 1

The best way to work around this is to specify the Dockerfile independently of the build context, using -f.

For instance, this command will give the ADD command access to anything in your current directory.

docker build -f docker-files/Dockerfile .

Update: Docker now allows having the Dockerfile outside the build context (fixed in 18.03.0-ce). So you can also do something like

docker build -f ../Dockerfile .

Answer 2

I often find myself utilizing the --build-arg option for this purpose. For example after putting the following in the Dockerfile:

ARG SSH_KEY
RUN echo "$SSH_KEY" > /root/.ssh/id_rsa

You can just do:

docker build -t some-app --build-arg SSH_KEY="$(cat ~/file/outside/build/context/id_rsa)" .

But note the following warning from the Docker documentation:

Warning: It is not recommended to use build-time variables for passing secrets like github keys, user credentials etc. Build-time variable values are visible to any user of the image with the docker history command.

Answer 3

I spent a good time trying to figure out a good pattern and how to better explain what's going on with this feature support. I realized that the best way to explain it was as follows...

• Dockerfile: Will only see files under its own relative path
• Context: a place in "space" where the files you want to share and your Dockerfile will be copied to

So, with that said, here's an example of the Dockerfile that needs to reuse a file called start.sh

Dockerfile

It will always load from its relative path, having the current directory of itself as the local reference to the paths you specify.

COPY start.sh /runtime/start.sh

Files

Considering this idea, we can think of having multiple copies for the Dockerfiles building specific things, but they all need access to the start.sh.

./all-services/
   /start.sh
   /service-X/Dockerfile
   /service-Y/Dockerfile
   /service-Z/Dockerfile
./docker-compose.yaml

Considering this structure and the files above, here's a docker-compose.yml

docker-compose.yaml

• In this example, your shared context directory is the runtime directory.
  • Same mental model here, think that all the files under this directory are moved over to the so-called context.
  • Similarly, just specify the Dockerfile that you want to copy to that same directory. You can specify that using dockerfile.
• The directory where your main content is located is the actual context to be set.

The docker-compose.yml is as follows

version: "3.3"
services:
  
  service-A
    build:
      context: ./all-service
      dockerfile: ./service-A/Dockerfile

  service-B
    build:
      context: ./all-service
      dockerfile: ./service-B/Dockerfile

  service-C
    build:
      context: ./all-service
      dockerfile: ./service-C/Dockerfile

• all-service is set as the context, the shared file start.sh is copied there as well the Dockerfile specified by each dockerfile.
• Each gets to be built their own way, sharing the start file!

Answer 4

On Linux you can mount other directories instead of symlinking them

mount --bind olddir newdir

See https://superuser.com/questions/842642 for more details.

I don't know if something similar is available for other OSes. I also tried using Samba to share a folder and remount it into the Docker context which worked as well.

Answer 5

If you read the discussion in the issue 2745 not only docker may never support symlinks they may never support adding files outside your context. Seems to be a design philosophy that files that go into docker build should explicitly be part of its context or be from a URL where it is presumably deployed too with a fixed version so that the build is repeatable with well known URLs or files shipped with the docker container.

I prefer to build from a version controlled source - ie docker build -t stuff http://my.git.org/repo - otherwise I'm building from some random place with random files.

fundamentally, no.... -- SvenDowideit, Docker Inc

Just my opinion but I think you should restructure to separate out the code and docker repositories. That way the containers can be generic and pull in any version of the code at run time rather than build time.

Alternatively, use docker as your fundamental code deployment artifact and then you put the dockerfile in the root of the code repository. if you go this route probably makes sense to have a parent docker container for more general system level details and a child container for setup specific to your code.

Answer 6

I believe the simpler workaround would be to change the 'context' itself.

So, for example, instead of giving:

docker build -t hello-demo-app .

which sets the current directory as the context, let's say you wanted the parent directory as the context, just use:

docker build -t hello-demo-app ..

Answer 7

You can also create a tarball of what the image needs first and use that as your context.

https://docs.docker.com/engine/reference/commandline/build/#/tarball-contexts

Answer 8

As is described in this GitHub issue the build actually happens in /tmp/docker-12345, so a relative path like ../relative-add/some-file is relative to /tmp/docker-12345. It would thus search for /tmp/relative-add/some-file, which is also shown in the error message.*

It is not allowed to include files from outside the build directory, so this results in the "Forbidden path" message."

Answer 9

Using docker-compose, I accomplished this by creating a service that mounts the volumes that I need and committing the image of the container. Then, in the subsequent service, I rely on the previously committed image, which has all of the data stored at mounted locations. You will then have have to copy these files to their ultimate destination, as host mounted directories do not get committed when running a docker commit command

You don't have to use docker-compose to accomplish this, but it makes life a bit easier

# docker-compose.yml

version: '3'
  services:
    stage:
      image: alpine
      volumes:
        - /host/machine/path:/tmp/container/path
      command: bash -c "cp -r /tmp/container/path /final/container/path"
    setup:
      image: stage

# setup.sh

# Start "stage" service
docker-compose up stage

# Commit changes to an image named "stage"
docker commit $(docker-compose ps -q stage) stage

# Start setup service off of stage image
docker-compose up setup

Answer 10

An easy workaround might be to simply mount the volume (using the -v or --mount flag) to the container when you run it and access the files that way.

example:

docker run -v /path/to/file/on/host:/desired/path/to/file/in/container/ image_name

for more see: https://docs.docker.com/storage/volumes/

Answer 11

Workaround with links:

ln path/to/file/outside/context/file_to_copy ./file_to_copy

On Dockerfile, simply:

COPY file_to_copy /path/to/file

Answer 12

I had this same issue with a project and some data files that I wasn't able to move inside the repo context for HIPAA reasons. I ended up using 2 Dockerfiles. One builds the main application without the stuff I needed outside the container and publishes that to internal repo. Then a second dockerfile pulls that image and adds the data and creates a new image which is then deployed and never stored anywhere. Not ideal, but it worked for my purposes of keeping sensitive information out of the repo.

Answer 13

One quick and dirty way is to set the build context up as many levels as you need - but this can have consequences. If you're working in a microservices architecture that looks like this:

./Code/Repo1
./Code/Repo2
...

You can set the build context to the parent Code directory and then access everything, but it turns out that with a large number of repositories, this can result in the build taking a long time.

An example situation could be that another team maintains a database schema in Repo1 and your team's code in Repo2 depends on this. You want to dockerise this dependency with some of your own seed data without worrying about schema changes or polluting the other team's repository (depending on what the changes are you may still have to change your seed data scripts of course) The second approach is hacky but gets around the issue of long builds:

Create a sh (or ps1) script in ./Code/Repo2 to copy the files you need and invoke the docker commands you want, for example:

#!/bin/bash
rm -r ./db/schema
mkdir ./db/schema

cp  -r ../Repo1/db/schema ./db/schema

docker-compose -f docker-compose.yml down
docker container prune -f
docker-compose -f docker-compose.yml up --build

In the docker-compose file, simply set the context as Repo2 root and use the content of the ./db/schema directory in your dockerfile without worrying about the path. Bear in mind that you will run the risk of accidentally committing this directory to source control, but scripting cleanup actions should be easy enough.

Answer 14

In my case, my Dockerfile is written like a template containing placeholders which I'm replacing with real value using my configuration file.

So I couldn't specify this file directly but pipe it into the docker build like this:

sed "s/%email_address%/$EMAIL_ADDRESS/;" ./Dockerfile | docker build -t katzda/bookings:latest . -f -;

But because of the pipe, the COPY command didn't work. But the above way solves it by -f - (explicitly saying file not provided). Doing only - without the -f flag, the context AND the Dockerfile are not provided which is a caveat.

Answer 15

Create a wrapper docker build shell script that grabs the file then calls docker build then removes the file.

a simple solution not mentioned anywhere here from my quick skim:

• have a wrapper script called docker_build.sh
• have it create tarballs, copy large files to the current working directory
• call docker build
• clean up the tarballs, large files, etc

this solution is good because (1.) it doesn't have the security hole from copying in your SSH private key (2.) another solution uses sudo bind so that has another security hole there because it requires root permission to do bind.

Answer 16

How to share typescript code between two Dockerfiles

I had this same problem, but for sharing files between two typescript projects. Some of the other answers didn't work for me because I needed to preserve the relative import paths between the shared code. I solved it by organizing my code like this:

api/
  Dockerfile
  src/
    models/
      index.ts

frontend/
  Dockerfile
  src/
    models/
      index.ts

shared/
  model1.ts
  model2.ts
  index.ts

.dockerignore

Note: After extracting the shared code into that top folder, I avoided needing to update the import paths because I updated api/models/index.ts and frontend/models/index.ts to export from shared: (eg export * from '../../../shared)

Since the build context is now one directory higher, I had to make a few additional changes:

1. Update the build command to use the new context:

   docker build -f Dockerfile .. (two dots instead of one)

2. Use a single .dockerignore at the top level to exclude all node_modules. (eg **/node_modules/**)

3. Prefix the Dockerfile COPY commands with api/ or frontend/

4. Copy shared (in addition to api/src or frontend/src)

   WORKDIR /usr/src/app
   
   COPY api/package*.json ./     <---- Prefix with api/
   RUN npm ci
   
   COPY api/src api/ts*.json ./  <---- Prefix with api/
   COPY shared usr/src/shared    <---- ADDED
   RUN npm run build
   

This was the easiest way I could send everything to docker, while preserving the relative import paths in both projects. The tricky (annoying) part was all the changes/consequences caused by the build context being up one directory.