Docker Build Secrets

Sometimes, one needs to pass sensitive data to a docker build.

Most likely, it would be a secret token to download a library from a private repository (company internal python registry for example)

In this case, a build-arg is actually a bad security practice, because if used in an env var it will most likely be persisted in the final image or in a docker build cache layer.

To remediate this, one can use docker build secrets

In AIchor, one can configure build time environment variables in the Environment Variable tab and Build variables menu:

For this example, we set the following value to the environment variable PIP_INSTALL_ARG :

_--index-url https://__token__:<token>@gitlab.example.com/api/v4/groups/<group_id>/-/packages/pypi/simple_

Once this is done, you can add something similar to the following to your Dockerfile:

RUN --mount=type=secret,id=_env,dst=/etc/secrets/.env . /etc/secrets/.env \
        && pip install $PIP_INSTALL_ARG ridl==x.x.x

Let’s unpack the above a bit.

• --mount=type=secret,id=_env,dst=/etc/secrets/.env is docker build secret boilerplate;

The id of the build secret passed by AIchor as explained above will always be _env.

The dst must be a valid file path, and can be changed as you see fit.

Docker build secrets do not allow to set environment variables natively, so AIchor generates a .env type file with this content:

PIP_INSTALL_ARG="--index-url https://__token__:<token>@gitlab.example.com/api/v4/groups/<group_id>/-/packages/pypi/simple"
OTHER_ENV_VAR="Something else"

The file will be available to the current RUN instruction only. It will not be saved in the layer or in the build cache

• . /etc/secrets/.env this is the POSIX way to populate environment variables from a .env file. From this point the environment variables are available;

• pip install $PIP_INSTALL_ARG private-lib==0.0.1 .

This pip install will use the custom made —-index-url defined in the PIP_INSTALL_ARG environment variable to install the private-lib

💡 Caveat: Since the build secret is not included in the cache, changes to the build environment variables will be ignored in your following builds. If you add a new build environment variable for a second token for another library, please make sure to update the `pip install` command to invalidate the cache. You most likely will need to add the second library to the pip install copmmand or requirement.txt file anyway. This is also why pinning the version of libraries and updating it at times of changes is also a good practice.

Note: In a CI, this allows for strategies of one time tokens without changing the docker cache when the library version is unchanged.

Note: If you separately need to build your dockerfile

When building a dockerfile containing a RUN --mount=type=secret,id=_env instruction, please make sure to do something similar to this:

echo "PIP_INSTALL_ARG=\"$PIP_INSTALL_ARG\"" > /tmp/.env
DOCKER_BUILDKIT=1 docker build --secret id=_env,src=/tmp/.env .

You need to previously set your environment variable PIP_INSTALL_ARG. It will be used to generate a local .env file

DOCKER_BUILDKIT=1 activates Buildkit, the most up to date docker build system, which is required for this docker secrets feature.

Tips: Using secret file

If you need the secrets to be in a file you can encode it in base64

cat my_secret_file.json | base64 

Then in your Dockerfile use something like this:

RUN --mount=type=secret,id=_env,dst=/etc/secrets/.env . /etc/secrets/.env && echo "$GCP_SA_BASE64" | base64 -d > /gcp_sa.json && pip install pcb-envpool --extra-index-url https://europe-west4-python.pkg.dev/instadeep/pcb-envpool/simple/ ; ret=$?; rm /gcp_sa.json ; exit $ret