Production Docker images built small scanned and actually rebuilt

The Dockerfile that works in development and the Dockerfile that belongs in production are different documents, and the difference is not style. The development image is built from the base the laptop had handy, runs as root because the permissions just worked, contains the compiler toolchain the application never needed at runtime, and ships with the tag latest because versions felt like ceremony. That image in production is a larger attack surface than the application it wraps, a slower cold start than the application deserves, and a supply chain nobody can audit. This article builds the production image properly, the multi stage build that separates the build from the runtime, the base image choice with its tradeoffs, the security configuration that runs as nobody, and the supply chain practices, scanning, digest pinning and the rebuild discipline, that keep the image honest a year after it shipped.

Multi stage builds, the compiler never rides to production

The single stage Dockerfile carries its build tools into runtime because the layers have nowhere else to go, and the multi stage build is the mechanism that finally gave them somewhere:

FROM golang:1.24 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /out/app ./cmd/server

FROM gcr.io/distroless/static-debian12
COPY --from=build /out/app /app
USER nonroot:nonroot
ENTRYPOINT ["/app"]

The first stage carries the toolchain, the source, the build cache, and only the artifact crosses the boundary through the copy from the build stage. The runtime image contains the binary and the base's minimal runtime, megabytes where the single stage carried gigabytes, and the size is not vanity, smaller images pull faster on scale up, fit the node's cache better, and present less surface to scan and to exploit. The interpreted languages get the same shape, the Python or Node image with the build stage compiling the native dependencies and the runtime stage copying the installed tree:

FROM node:22-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:22-alpine
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER node
CMD ["node", "dist/server.js"]

The layer ordering inside each stage is the caching discipline, the dependency manifests copy and install before the source copies, so the source change does not invalidate the dependency layer, and the dockerignore file completes the caching hygiene by keeping the local build output, the git directory and the environment files out of the build context entirely.

Base images, the deliberate minimum

The base image choice is a real tradeoff with three named positions. The distroless family carries the runtime and nothing else, no shell, no package manager, the smallest surface and the smallest attack story, at the cost of debugging inside the container being effectively impossible, which is the correct price for most production services. The alpine family carries a small musl based userspace with a shell and a package manager, convenient for the images that need an occasional tool, with the musl compatibility caveats on native extensions that the Python and Ruby ecosystems documented extensively. The slim variants of the standard distributions carry the familiar glibc userspace trimmed of tooling, the compatibility choice for the applications with native dependencies that assume the standard library, and the honest practice is the one the team can defend, the smallest base the application actually runs on, chosen once and recorded.

FROM python:3.12-slim AS build

The version pinning continues through the whole file, the base by tag or digest, the dependencies by lockfile, and the digest pinning is the strictest form, the immutable identifier that makes the build reproducible even when the tag is quietly re-pointed upstream:

FROM python:3.12-slim@sha256:9c4b5c...

Running as nobody, the non root default

The root default is the base image inheritance that production images refuse. The application does not need root, and the container's root is the real root of the host's user namespace in the standard configurations, so the compromised root container is a host compromise's first step. The production image creates and switches to an unprivileged user, the distroless base's nonroot or the explicit creation:

RUN groupadd -r app && useradd -r -g app app
USER app

The supporting hardening rides the compose or the pod spec, the read only root filesystem with the tmpfs for the write paths, the dropped capabilities, the seccomp profile the runtime applies by default, and the image's job is to make the hardening possible, the application that writes nowhere but its tmpfs and needs no capability is the application that runs under the strictest policy without an incident.

The supply chain, scanning pinning and the rebuild calendar

The image that ships is the image that must be auditable and the practices are three. The scan runs in the pipeline on every build, Trivy or Grype against the image's layers, the critical findings gate the merge, and the scan's output includes the fix available, which turns most findings into a base image bump rather than an investigation. The digest pinning from the previous section covers the base and the dependencies, the lockfile in the repository is the dependency digest, and the build's reproducibility is the audit's foundation, the image that can be rebuilt byte identical is the image whose contents are known.

The rebuild calendar is the practice that separates the shipped image from the archived one, because the image that does not rebuild decays. The base's vulnerabilities accumulate, the runtime's patches ship, and the monthly rebuild with the bumped bases is the maintenance that keeps the scan output short, the digest bump in the repository is the change the pipeline verifies and the registry receives. The SBOM completes the audit story, the software bill of materials generated at build and stored with the image, the answer to the question that arrives with every new CVE, which of our images contain the affected package, answerable by query rather than by rebuild-and-check.

One service's images before and after the discipline

The before and after makes the discipline concrete. The single stage Python image, FROM python:3.12 with the apt install of the build toolchain and the pip install in one layer, ships at 1.4 gigabytes, pulls on a cold node for forty seconds, and its scan output runs to three hundred findings, mostly in the toolchain the runtime never calls. The multi stage rebuild on the slim base, the build stage with the compilers, the runtime stage with the installed packages copied across, ships at 180 megabytes, a 87 percent reduction, pulls in under five seconds, and the scan output drops to the dozen findings that actually live in the runtime tree, all with fixes available in the bumped base.

The scale up moment is where the numbers pay, the autoscaler's cold start is the image pull plus the container start, and the fleet that scales by twenty nodes on a Monday morning waits seven hundred cold node seconds at 1.4 gigabytes or eighty at 180 megabytes, the same event on the same cluster with different user experiences. The memory story repeats the disk story in slower motion, the larger image's layers occupy the node's cache, the eviction cycles evict the application's neighbors, and the small image is the polite tenant whose cache footprint lets the fleet stay warm.

Build cache optimization, the pipeline's speed budget

The build's speed is the discipline's adoption cost, and the modern build tooling pays it. The cache mounts mount the package manager's cache directory across builds, the pip cache or the npm cache that turns the dependency layer install from a download into a copy:

RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

The cache mounts keep the heavy dependency resolution out of the layer stack entirely, the layer holds the result and the cache holds the downloads, and the build that ran twelve minutes on the cold runner runs ninety seconds on the warm one. The remote cache backend completes the story across runners, the build cache exported to the registry or the dedicated cache service, so the cache warmth follows the pipeline rather than the runner, and the pull request's build starts where the main branch's build ended.

Debugging the distroless image from outside the shell

The no shell base's debugging story is the objection every team raises, and the answer is that the debugging belongs outside the image. The application's logs go to stdout by the container contract, structured, the observability stack collects them, and the first debugging tool is the log stream, not the shell session. The second is the ephemeral debug container, the operator attaches a toolbox image to the pod's namespaces, shares its filesystem view and its network, and inspects the distroless container from a sibling that has the tools:

kubectl debug -it target-pod --image=busybox --target=app

The debug container pattern keeps the production image pure and the debugging capability intact, the shell exists when the incident needs one, in a container that leaves when the session ends, and the image that never had the shell is the image that never had the shell's vulnerabilities. The teams that adopt the pattern report the same transition, the first month misses the exec session, and the second month has the log query and the debug container wired into the runbook, and the shell in the production image turns out to be the comfort food of container debugging rather than its requirement.

The registry and the runtime image contract

The registry practices complete the production discipline, the immutable tags that the orchestration layer relies on, the tag per commit or per build, never latest, because latest in a manifest is a race between the deploy and the push. The retention policy keeps the registry affordable, the last N builds per branch plus the production tags forever, and the garbage collection runs on the registry's schedule rather than waiting for the invoice. The promotion story writes itself once the tags are immutable, the staging cluster runs the digest, the production cluster runs the same digest, and the promotion is a manifest update in the deployment repository, the verified artifact moving forward without a rebuild anywhere in the pipeline. the tag per commit or per build, never latest, because latest in a manifest is a race between the deploy and the push. The runtime contract between the image and its orchestrator is worth writing down, the image declares its ports, its healthcheck path and its user, the orchestrator declares the resources and the policy, and the contract keeps the image portable, the same artifact through staging and production, verified once, run everywhere, which is the whole promise the container made in the first place.

The working rules compress the whole discipline, and the list is short:

1. Build in stages, the toolchain never rides to runtime, and order the layers for the cache;
2. Pin by digest or lockfile, and treat latest as the tag that means unspecified;
3. Run as the unprivileged user, with the read only filesystem and the dropped capabilities;
4. Scan in the pipeline, gate on critical, and bump the base monthly rather than investigating quarterly;
5. Generate the SBOM at build, and store it where the CVE question can query it;
6. Rebuild on a calendar, the image that has not been rebuilt in ninety days is the debt the scan will name.

The production image is an artifact with a supply chain, an audit trail and a maintenance calendar, and the Dockerfile that treats it as one, small, pinned, unprivileged and rebuilt, is the difference between the container as the packaging convenience and the container as the platform it promised to be.