Part 8: Docker and docker compose: ship the app, run the whole stack
A 23 MB non-root image, a one-command environment with Postgres, Prometheus, Grafana, Jaeger and Loki, and the reasons behind each line.
Series: Build a production-grade REST API in Go · Part 8 of 17 Where we are: the observable API from Part 7. This post adds no Go code: it packages the app and runs the tools that display what Part 7 emits.
"It works on my machine" is the oldest deployment bug. A container image packages the program and everything it needs so it runs identically on your laptop, in CI and in production. docker compose describes a set of containers (our app, the database, the monitoring tools) that start together.
By the end of this post, one command starts everything:
make up
Part A: the image
Create Dockerfile
# syntax=docker/dockerfile:1
# ---- Stage 1: build ---------------------------------------------------------
FROM golang:1.27-alpine AS build
WORKDIR /src
# Copy ONLY the dependency manifests first and download. Docker caches each
# layer; as long as go.mod/go.sum are unchanged this slow step is skipped, even
# when source code changes on every build.
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
# CGO_ENABLED=0 -> a fully static binary (pgx is pure Go), so it runs on a
# minimal image with no libc.
# -trimpath -> strip local file paths from the binary (reproducible, no leaks)
# -s -w -> drop symbol/debug tables (~30% smaller)
RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/server ./cmd/rest
# ---- Stage 2: runtime -------------------------------------------------------
# distroless/static: no shell, no package manager, no curl: almost nothing for
# an attacker to use. ":nonroot" runs as uid 65532 instead of root.
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/server /server
EXPOSE 7800 9090
# distroless has no curl, so the binary probes itself (see `healthcheck` mode).
HEALTHCHECK --interval=10s --timeout=4s --start-period=10s --retries=3 \
CMD ["/server", "healthcheck"]
ENTRYPOINT ["/server"]
This is a multi-stage build. Two FROM lines mean two stages:
buildusesgolang:alpine, an image with the full Go toolchain (hundreds of MB), to compile the program.- The final stage starts from
distroless/staticand copies only the compiled binary from the first stage. The toolchain is thrown away.
The result is a 23 MB image instead of ~800 MB. Smaller is not just about disk: it's about attack surface. Distroless has no shell, no package manager, no curl, almost nothing an intruder could use if they got in. And it runs as nonroot (uid 65532), so a compromise doesn't own the container.
Line-by-line reasons:
- Copy
go.mod/go.sumfirst, thengo mod download, then copy the source. Docker caches each instruction as a layer and reuses a layer if its inputs are unchanged. Dependencies change rarely; source changes constantly. By splitting them, a code change only re-runs the (fast) compile step, not the slow download. --mount=type=cachekeeps Go's module and build caches between builds.CGO_ENABLED=0produces a fully static binary with no C library dependency. This is why we chose pure-Go drivers (pgx) all along: a static binary runs on a minimal image with no libc.-trimpath -ldflags="-s -w"removes local file paths and debug tables: smaller, and doesn't leak your directory layout.- The Go version must match.
FROM golang:1.27-alpinehas to be at least the version in yourgo.mod(go mod initwrites the version of the Go on your machine). If the build fails withgo.mod requires go >= 1.27.3 (running go 1.27.2), your cached image is stale: rundocker pull golang:1.27-alpine(ordocker compose build --pull) and try again. HEALTHCHECK: the image has nocurl, so the binary checks itself. Ourmainhas ahealthchecksubcommand that calls its own/readyz. One binary, three modes:serve,migrate,healthcheck.
And the file that keeps junk out of the build context:
Create .dockerignore
.git
bin/
*.db*
.env
docs/
deploy/
docker-compose.yml
(Excluding .git, local databases and .env also matters for security: you never want secrets baked into an image layer.)
Part B: docker compose, service by service
Create docker-compose.yml
# Full local stack. `make up` -> API :7800, Grafana :3000, Prometheus :9091, Jaeger :16686
name: rest-go
x-logging: &logging # cap container log files so disks don't fill up
driver: json-file
options: { max-size: "10m", max-file: "3" }
services:
# ---------------------------------------------------------------- database
postgres:
image: postgres:17-alpine
environment:
POSTGRES_USER: ${POSTGRES_USER:-app}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-change-me}
POSTGRES_DB: ${POSTGRES_DB:-app}
volumes:
- pgdata:/var/lib/postgresql/data # named volume: data survives `down`
ports:
- "127.0.0.1:5432:5432" # localhost only, never 0.0.0.0
healthcheck: # lets other services wait until it is READY
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 5s
timeout: 3s
retries: 10
logging: *logging
# One-shot job: apply migrations, then exit 0. The API starts only after it
# succeeded, so the schema is always ready, and migrations run once (not once
# per API replica).
migrate:
build: .
command: ["migrate"]
environment: &dburl
DATABASE_URL: postgres://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-change-me}@postgres:5432/${POSTGRES_DB:-app}?sslmode=disable
depends_on:
postgres: { condition: service_healthy }
restart: "no"
logging: *logging
# --------------------------------------------------------------------- api
api:
build: .
environment:
<<: *dburl
ADDR: ":7800"
ADMIN_ADDR: ":9090"
OTLP_ENDPOINT: jaeger:4317
LOG_LEVEL: info
ports:
- "7800:7800" # public API only; admin port 9090 stays internal
depends_on:
migrate: { condition: service_completed_successfully }
jaeger: { condition: service_started }
restart: unless-stopped
read_only: true # container filesystem is immutable
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]
stop_grace_period: 30s # > SHUTDOWN_TIMEOUT so Docker doesn't SIGKILL mid-drain
logging: *logging
# ----------------------------------------------------------- observability
prometheus:
image: prom/prometheus:latest
volumes:
- ./deploy/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./deploy/prometheus/alerts.yml:/etc/prometheus/alerts.yml:ro
- promdata:/prometheus
ports: ["127.0.0.1:9091:9090"]
depends_on: [api]
logging: *logging
postgres-exporter: # turns Postgres internals into metrics
image: prometheuscommunity/postgres-exporter:latest
environment:
DATA_SOURCE_NAME: postgres://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-change-me}@postgres:5432/${POSTGRES_DB:-app}?sslmode=disable
depends_on:
postgres: { condition: service_healthy }
logging: *logging
jaeger: # trace storage + UI; receives OTLP on :4317
image: jaegertracing/all-in-one:latest
ports: ["127.0.0.1:16686:16686"]
logging: *logging
loki: # log storage (indexes labels only, so it is cheap)
image: grafana/loki:latest
volumes: [lokidata:/loki]
logging: *logging
alloy: # log shipper: tails every container's stdout -> Loki
image: grafana/alloy:latest
command: ["run", "--server.http.listen-addr=0.0.0.0:12345", "--storage.path=/var/lib/alloy/data", "/etc/alloy/config.alloy"]
volumes:
- ./deploy/alloy/config.alloy:/etc/alloy/config.alloy:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
depends_on: [loki]
logging: *logging
grafana:
image: grafana/grafana:latest
environment:
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-change-me}
GF_USERS_ALLOW_SIGN_UP: "false"
volumes:
- ./deploy/grafana/provisioning:/etc/grafana/provisioning:ro
- ./deploy/grafana/dashboards:/var/lib/grafana/dashboards:ro
- grafanadata:/var/lib/grafana
ports: ["127.0.0.1:3000:3000"]
depends_on: [prometheus, loki, jaeger]
logging: *logging
volumes:
pgdata:
promdata:
lokidata:
grafanadata:
The database
postgres:
image: postgres:17-alpine
volumes: [pgdata:/var/lib/postgresql/data]
ports: ["127.0.0.1:5432:5432"]
healthcheck: ...
- Named volume
pgdata: data outlives the container.docker compose downkeeps it;down -v(ourmake reset) deletes it. 127.0.0.1:5432:5432: publish the port only on localhost. Writing5432:5432would expose your database to every machine that can reach your laptop. Default to the narrowest binding.healthcheckrunspg_isready. This is the key to ordering: "container started" is not "database ready to accept connections".
The migration job
migrate:
command: ["migrate"]
depends_on: { postgres: { condition: service_healthy } }
restart: "no"
A one-shot job: it applies migrations and exits. Why not migrate inside the API at startup, as in Part 6? Because when you run three API replicas, all three would race to migrate. A single job per deploy runs once, and the API waits for it:
api:
depends_on:
migrate: { condition: service_completed_successfully }
The schema is therefore guaranteed to exist before any API instance takes traffic.
The API
Notice the hardening block:
read_only: true # container filesystem is immutable
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL] # drop every Linux capability
stop_grace_period: 30s
A read-only filesystem means even a compromised process cannot write a backdoor to disk. Dropping all capabilities removes privileges the app never needed. stop_grace_period must exceed our SHUTDOWN_TIMEOUT (15 s): when Docker stops a container it sends SIGTERM, waits, then SIGKILL. If the wait is shorter than our graceful shutdown, in-flight requests get killed mid-drain. (Remember Part 3? This is the other half of that contract.)
The admin port 9090 is not published. Only Prometheus, inside the compose network, can reach it. From the host, curl localhost:9090/metrics fails. That's the point.
Configuration comes from the environment
environment:
DATABASE_URL: postgres://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-change-me}@postgres:5432/...
${VAR:-default} reads from your shell or a .env file (git-ignored, with .env.example committed as a template). Notice the hostname postgres: inside a compose network, containers reach each other by service name.
Part C: the observability stack
Everything in Part 7 now has somewhere to go.
Prometheus: metrics
Create deploy/prometheus/prometheus.yml
global:
scrape_interval: 15s # how often to pull /metrics
evaluation_interval: 15s # how often to evaluate alert rules
rule_files:
- /etc/prometheus/alerts.yml
scrape_configs:
- job_name: api
static_configs:
- targets: ["api:9090"] # the ADMIN port, reachable only inside the compose network
- job_name: postgres
static_configs:
- targets: ["postgres-exporter:9187"]
- job_name: prometheus
static_configs:
- targets: ["localhost:9090"]
Prometheus pulls: every 15 seconds it scrapes /metrics from each target. (Pull rather than push means a dead target is itself a signal: up == 0.) The api target is api:9090, the admin port, reachable only on the internal network.
Alerts
Create deploy/prometheus/alerts.yml
# Alert rules are evaluated by Prometheus. Delivering them (email/Slack/pager)
# needs Alertmanager, which is intentionally not part of this learning stack;
# firing alerts are still visible at http://localhost:9091/alerts.
groups:
- name: api
rules:
- alert: ApiDown
expr: up{job="api"} == 0
for: 1m
labels: { severity: page }
annotations: { summary: "API target is down" }
- alert: HighErrorRate # symptom-based: users see 5xx
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ clamp_min(sum(rate(http_requests_total[5m])), 1) > 0.05
for: 5m
labels: { severity: page }
annotations: { summary: "More than 5% of requests fail with 5xx" }
- alert: HighLatencyP95
expr: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) > 1
for: 10m
labels: { severity: ticket }
annotations: { summary: "p95 latency above 1s" }
- alert: DbPoolExhausted # requests are queueing for a connection
expr: rate(db_pool_empty_acquire_total[5m]) > 0
for: 5m
labels: { severity: ticket }
annotations: { summary: "DB pool is exhausted; consider raising DB_MAX_CONNS or fixing slow queries" }
Good alerts are symptom-based: they describe what users experience (error ratio, latency, the service being down), not internal causes (CPU 80%). Each rule has a for: duration so a ten-second blip doesn't wake anyone. The last alert is a nice example of connecting two parts of the series: db_pool_empty_acquire_total is the pool metric from Part 7, and a sustained rate means requests are queueing for database connections. (Prometheus evaluates these rules and shows them at /alerts; delivering them to Slack or a pager needs Alertmanager, which we leave out.)
Loki and Alloy: logs
loki: # stores logs, indexing only labels (cheap)
alloy: # agent: tails every container's stdout and ships it to Loki
Create deploy/alloy/config.alloy
// Alloy = Grafana's telemetry agent. Pipeline: discover containers -> label -> tail logs -> push to Loki.
discovery.docker "containers" {
host = "unix:///var/run/docker.sock"
}
discovery.relabel "containers" {
targets = discovery.docker.containers.targets
// Turn Docker metadata into Loki labels (keep labels FEW and LOW-cardinality;
// never use request_id or trace_id as a label).
rule {
source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
target_label = "service"
}
}
loki.source.docker "containers" {
host = "unix:///var/run/docker.sock"
targets = discovery.relabel.containers.output
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint {
url = "http://loki:3100/loki/api/v1/push"
}
}
Our app writes JSON logs to stdout. That's the 12-factor approach: the app doesn't manage log files; the platform collects the stream. Alloy discovers containers through the Docker socket, labels each stream with its compose service, and pushes to Loki. Keep Loki labels few and low-cardinality (the same rule as Prometheus!). Search inside lines with LogQL instead: {service="api"} | json | status >= 500.
Jaeger: traces
One container, receiving spans on port 4317 (the OTLP_ENDPOINT: jaeger:4317 we set on the API) and serving a UI on 16686.
Grafana: one window for all three
Create deploy/grafana/provisioning/datasources/datasources.yml
apiVersion: 1
datasources:
- name: Prometheus
uid: prometheus
type: prometheus
url: http://prometheus:9090
isDefault: true
- name: Jaeger
uid: jaeger
type: jaeger
url: http://jaeger:16686
- name: Loki
uid: loki
type: loki
url: http://loki:3100
jsonData:
# Makes trace_id in a log line a clickable link that opens the trace in
# Jaeger: the log -> trace jump.
derivedFields:
- name: trace_id
matcherRegex: '"trace_id":"(\w+)"'
url: '$${__value.raw}'
datasourceUid: jaeger
Provisioning means Grafana is configured from files at startup: no clicking, and it works identically after make reset. The derivedFields block is the best trick in the series: it teaches Loki that "trace_id":"abc123" in a log line is a link into Jaeger. You can read a log line, click the trace id, and see the full request tree. That is the connected observability we promised in Part 7.
Grafana also needs to be told where to find dashboards:
Create deploy/grafana/provisioning/dashboards/dashboards.yml
apiVersion: 1
providers:
- name: default
type: file
options:
path: /var/lib/grafana/dashboards
and here is a compact first dashboard. Each panel is just a PromQL query from Part 7 (request rate, error ratio, p50/p95/p99 latency, requests in flight, DB pool usage and waits) plus a live log panel from Loki:
Create deploy/grafana/dashboards/api-overview.json
{
"uid": "api-overview",
"title": "API overview",
"schemaVersion": 39,
"refresh": "10s",
"time": { "from": "now-30m", "to": "now" },
"panels": [
{ "id": 1, "type": "timeseries", "title": "Request rate by route (req/s)",
"gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
"targets": [ { "refId": "A", "legendFormat": "{{route}}", "expr": "sum by (route) (rate(http_requests_total[1m]))" } ] },
{ "id": 2, "type": "timeseries", "title": "Error ratio (share of 5xx)",
"gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
"targets": [ { "refId": "A", "legendFormat": "5xx", "expr": "(sum(rate(http_requests_total{status=~\"5..\"}[5m])) or vector(0)) / clamp_min(sum(rate(http_requests_total[5m])), 1)" } ] },
{ "id": 3, "type": "timeseries", "title": "Latency p50 / p95 / p99",
"gridPos": { "x": 0, "y": 8, "w": 12, "h": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
"targets": [
{ "refId": "A", "legendFormat": "p50", "expr": "histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))" },
{ "refId": "B", "legendFormat": "p95", "expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))" },
{ "refId": "C", "legendFormat": "p99", "expr": "histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))" } ] },
{ "id": 4, "type": "timeseries", "title": "Requests in flight",
"gridPos": { "x": 12, "y": 8, "w": 12, "h": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [ { "refId": "A", "legendFormat": "in flight", "expr": "http_requests_in_flight" } ] },
{ "id": 5, "type": "timeseries", "title": "DB pool connections",
"gridPos": { "x": 0, "y": 16, "w": 12, "h": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [
{ "refId": "A", "legendFormat": "in use", "expr": "db_pool_acquired_conns" },
{ "refId": "B", "legendFormat": "idle", "expr": "db_pool_idle_conns" },
{ "refId": "C", "legendFormat": "max", "expr": "db_pool_max_conns" } ] },
{ "id": 6, "type": "timeseries", "title": "DB pool waits per second (exhaustion signal)",
"gridPos": { "x": 12, "y": 16, "w": 12, "h": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"targets": [ { "refId": "A", "legendFormat": "waits/s", "expr": "rate(db_pool_empty_acquire_total[1m])" } ] },
{ "id": 7, "type": "logs", "title": "API logs (click a trace_id to open the trace)",
"gridPos": { "x": 0, "y": 24, "w": 24, "h": 10 },
"datasource": { "type": "loki", "uid": "loki" },
"options": { "enableLogDetails": true, "wrapLogMessage": true, "sortOrder": "Descending" },
"targets": [ { "refId": "A", "expr": "{service=\"api\"}" } ] }
]
}
Complete code for this part
Besides the files above, four small things change. Paste them, then you are ready to run the stack.
Already given earlier in this post: .dockerignore, Dockerfile, deploy/alloy/config.alloy, deploy/grafana/provisioning/dashboards/dashboards.yml, deploy/grafana/provisioning/datasources/datasources.yml, deploy/prometheus/alerts.yml, deploy/prometheus/prometheus.yml, docker-compose.yml.
Create .env.example
# Copy to .env (git-ignored) and change values. docker compose reads it automatically.
POSTGRES_USER=app
POSTGRES_PASSWORD=change-me
POSTGRES_DB=app
GRAFANA_ADMIN_PASSWORD=change-me
Replace the contents of .gitignore with
users.db
users.db-shm
users.db-wal
bin/
.env
Replace the contents of makefile with
.PHONY: db run build test test-integration up down logs reset migrate psql load clean
BINARY := bin/server
# Local (non-Docker) default; matches the compose Postgres published on localhost.
export DATABASE_URL ?= postgres://app:change-me@localhost:5432/app?sslmode=disable
TEST_DATABASE_URL ?= $(DATABASE_URL)
# ---- Go ----------------------------------------------------------------------
run: ## run the API locally (needs: make db)
go run ./cmd/rest
build:
mkdir -p bin
CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o $(BINARY) ./cmd/rest
migrate: ## apply migrations to $$DATABASE_URL
go run ./cmd/rest migrate
test: ## unit tests (database tests are skipped)
go vet ./...
go test -race ./...
test-integration: ## all tests incl. database tests, against compose Postgres
docker compose up -d --wait postgres
TEST_DATABASE_URL="$(TEST_DATABASE_URL)" go test -race -count=1 ./...
# ---- Docker ------------------------------------------------------------------
db: ## only Postgres (for running the API with `make run`)
docker compose up -d --wait postgres
up: ## whole stack: API + Postgres + Prometheus + Grafana + Jaeger + Loki
docker compose up -d --build --wait
down: ## stop, KEEP data
docker compose down
reset: ## stop and DELETE all data volumes
docker compose down -v
logs:
docker compose logs -f api
psql:
docker compose exec postgres sh -c 'psql -U $$POSTGRES_USER -d $$POSTGRES_DB'
load: ## generate some traffic so dashboards have data
./scripts/load.sh
clean:
rm -rf bin
Create scripts/load.sh
#!/usr/bin/env bash
# Generates mixed traffic (including errors) so the dashboards have something to show.
set -euo pipefail
API=${API:-http://localhost:7800}
for i in $(seq 1 ${N:-200}); do
curl -s -o /dev/null -XPOST "$API/users" -d "{\"name\":\"user$i\"}"
curl -s -o /dev/null "$API/users"
curl -s -o /dev/null "$API/users/$((RANDOM % 50 + 1))" # many 404s
curl -s -o /dev/null -XPOST "$API/orders" -d "{\"user_id\":$((RANDOM % 20 + 1)),\"item\":\"book\",\"quantity\":2}"
curl -s -o /dev/null -XPOST "$API/users" -d '{}' # 422s
done
echo "done"
Try it
$ cp .env.example .env
$ make up # builds the image, starts 9 containers, waits until they are healthy
Container rest-go-postgres-1 Healthy
Container rest-go-migrate-1 Exited ← one-shot job finished
Container rest-go-api-1 Healthy
Container rest-go-prometheus-1 Healthy
Container rest-go-grafana-1 Healthy
...
$ chmod +x scripts/load.sh # once: a file you created by hand is not executable yet
$ make load # generates mixed traffic, including errors
Then take the tour:
- Grafana → http://localhost:3000 (admin / the password in
.env) → API overview. - Explore → Loki:
{service="api"} | json | status=422. Clicktrace_idon a line. - Jaeger opens that trace: an HTTP span with
pool.acquireandINSERTchildren. - Prometheus →
/targets(allUP) and/alerts. - Break it on purpose:
docker compose pause postgres. Watch/readyzreturn 503, requests fail after the 10 s timeout, and the error ratio climb.docker compose unpause postgresand watch recovery. Nothing teaches monitoring like watching it notice an outage you caused.
Verify graceful shutdown: docker compose stop api && docker compose logs api shows shutdown signal received then server stopped cleanly, and docker inspect reports exit code 0.
Recap
- Multi-stage builds: build with the toolchain, ship only the binary; distroless + non-root.
- Order dependent services with healthchecks and
depends_on: condition. - Run migrations as a one-shot job, not inside every replica.
- Publish only what must be public; bind tools to
127.0.0.1. - Harden containers: read-only filesystem, no capabilities, adequate stop grace period.
- Provision monitoring as code so it's reproducible.
Exercises
- Run
docker image lsand compare your image to one builtFROM golang:alpineas the final stage. Thendocker run --rm -it --entrypoint sh <image>: why does it fail on the distroless one? - Add an alert for "p99 latency above 2 s for 10 minutes" and trigger it by adding
pg_sleep(3)to a query. - Change
MaxConnsto 1 in the compose environment, runmake loadfrom two terminals, and find the effect on the DB pool waits panel and in a trace'spool.acquirespan.
Comments
Join the discussionNo comments yet. Be the first to comment.
Build a production-grade REST API in Go
A hands-on series: start with under 200 lines of in-memory CRUD and end with Postgres, JWT auth, rate limiting, TLS and a full observability stack.