# Part 8: Docker and docker compose: ship the app, run the whole stack

*A 23 MB non-root image, a one-command environment with Postgres, Prometheus, Grafana, Jaeger and Loki, and the reasons behind each line.*

---

**Series:** Build a production-grade REST API in Go · **Part 8 of 17** **Where we are:** the observable API from Part 7. This post adds no Go code: it packages the app and runs the tools that display what Part 7 emits.

"It works on my machine" is the oldest deployment bug. A **container image** packages the program and everything it needs so it runs identically on your laptop, in CI and in production. **docker compose** describes a set of containers (our app, the database, the monitoring tools) that start together.

By the end of this post, one command starts everything:

```bash
make up
```

## Part A: the image

**Create** `Dockerfile`

```dockerfile
# syntax=docker/dockerfile:1

# ---- Stage 1: build ---------------------------------------------------------
FROM golang:1.27-alpine AS build
WORKDIR /src

# Copy ONLY the dependency manifests first and download. Docker caches each
# layer; as long as go.mod/go.sum are unchanged this slow step is skipped, even
# when source code changes on every build.
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download

COPY . .
# CGO_ENABLED=0 -> a fully static binary (pgx is pure Go), so it runs on a
#                  minimal image with no libc.
# -trimpath    -> strip local file paths from the binary (reproducible, no leaks)
# -s -w        -> drop symbol/debug tables (~30% smaller)
RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/server ./cmd/rest

# ---- Stage 2: runtime -------------------------------------------------------
# distroless/static: no shell, no package manager, no curl: almost nothing for
# an attacker to use. ":nonroot" runs as uid 65532 instead of root.
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/server /server
EXPOSE 7800 9090

# distroless has no curl, so the binary probes itself (see `healthcheck` mode).
HEALTHCHECK --interval=10s --timeout=4s --start-period=10s --retries=3 \
    CMD ["/server", "healthcheck"]

ENTRYPOINT ["/server"]
```

This is a **multi-stage build**. Two `FROM` lines mean two stages:

1. `build` uses `golang:alpine`, an image with the full Go toolchain (hundreds of MB), to compile the program.
2. **The final stage** starts from `distroless/static` and copies *only the compiled binary* from the first stage. The toolchain is thrown away.

The result is a **23 MB** image instead of ~800 MB. Smaller is not just about disk: it's about *attack surface*. Distroless has no shell, no package manager, no `curl`, almost nothing an intruder could use if they got in. And it runs as `nonroot` (uid 65532), so a compromise doesn't own the container.

Line-by-line reasons:

- **Copy** `go.mod`**/**`go.sum` **first, then** `go mod download`**, then copy the source.** Docker caches each instruction as a layer and reuses a layer if its inputs are unchanged. Dependencies change rarely; source changes constantly. By splitting them, a code change only re-runs the (fast) compile step, not the slow download.
- `--mount=type=cache` keeps Go's module and build caches between builds.
- `CGO_ENABLED=0` produces a fully static binary with no C library dependency. This is why we chose pure-Go drivers (pgx) all along: a static binary runs on a minimal image with no libc.
- `-trimpath -ldflags="-s -w"` removes local file paths and debug tables: smaller, and doesn't leak your directory layout.
- **The Go version must match.** `FROM golang:1.27-alpine` has to be at least the version in your `go.mod` (`go mod init` writes the version of the Go on your machine). If the build fails with `go.mod requires go >= 1.27.3 (running go 1.27.2)`, your cached image is stale: run `docker pull golang:1.27-alpine` (or `docker compose build --pull`) and try again.
- `HEALTHCHECK`: the image has no `curl`, so *the binary checks itself*. Our `main` has a `healthcheck` subcommand that calls its own `/readyz`. One binary, three modes: `serve`, `migrate`, `healthcheck`.

And the file that keeps junk out of the build context:

**Create** `.dockerignore`

```plaintext
.git
bin/
*.db*
.env
docs/
deploy/
docker-compose.yml
```

(Excluding `.git`, local databases and `.env` also matters for security: you never want secrets baked into an image layer.)

## Part B: docker compose, service by service

**Create** `docker-compose.yml`

```yaml
# Full local stack. `make up` -> API :7800, Grafana :3000, Prometheus :9091, Jaeger :16686
name: rest-go

x-logging: &logging   # cap container log files so disks don't fill up
  driver: json-file
  options: { max-size: "10m", max-file: "3" }

services:
  # ---------------------------------------------------------------- database
  postgres:
    image: postgres:17-alpine
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-app}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-change-me}
      POSTGRES_DB: ${POSTGRES_DB:-app}
    volumes:
      - pgdata:/var/lib/postgresql/data      # named volume: data survives `down`
    ports:
      - "127.0.0.1:5432:5432"                # localhost only, never 0.0.0.0
    healthcheck:                             # lets other services wait until it is READY
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 3s
      retries: 10
    logging: *logging

  # One-shot job: apply migrations, then exit 0. The API starts only after it
  # succeeded, so the schema is always ready, and migrations run once (not once
  # per API replica).
  migrate:
    build: .
    command: ["migrate"]
    environment: &dburl
      DATABASE_URL: postgres://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-change-me}@postgres:5432/${POSTGRES_DB:-app}?sslmode=disable
    depends_on:
      postgres: { condition: service_healthy }
    restart: "no"
    logging: *logging

  # --------------------------------------------------------------------- api
  api:
    build: .
    environment:
      <<: *dburl
      ADDR: ":7800"
      ADMIN_ADDR: ":9090"
      OTLP_ENDPOINT: jaeger:4317
      LOG_LEVEL: info
    ports:
      - "7800:7800"                          # public API only; admin port 9090 stays internal
    depends_on:
      migrate: { condition: service_completed_successfully }
      jaeger: { condition: service_started }
    restart: unless-stopped
    read_only: true                          # container filesystem is immutable
    security_opt: ["no-new-privileges:true"]
    cap_drop: [ALL]
    stop_grace_period: 30s                   # > SHUTDOWN_TIMEOUT so Docker doesn't SIGKILL mid-drain
    logging: *logging

  # ----------------------------------------------------------- observability
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./deploy/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./deploy/prometheus/alerts.yml:/etc/prometheus/alerts.yml:ro
      - promdata:/prometheus
    ports: ["127.0.0.1:9091:9090"]
    depends_on: [api]
    logging: *logging

  postgres-exporter:                         # turns Postgres internals into metrics
    image: prometheuscommunity/postgres-exporter:latest
    environment:
      DATA_SOURCE_NAME: postgres://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-change-me}@postgres:5432/${POSTGRES_DB:-app}?sslmode=disable
    depends_on:
      postgres: { condition: service_healthy }
    logging: *logging

  jaeger:                                    # trace storage + UI; receives OTLP on :4317
    image: jaegertracing/all-in-one:latest
    ports: ["127.0.0.1:16686:16686"]
    logging: *logging

  loki:                                      # log storage (indexes labels only, so it is cheap)
    image: grafana/loki:latest
    volumes: [lokidata:/loki]
    logging: *logging

  alloy:                                     # log shipper: tails every container's stdout -> Loki
    image: grafana/alloy:latest
    command: ["run", "--server.http.listen-addr=0.0.0.0:12345", "--storage.path=/var/lib/alloy/data", "/etc/alloy/config.alloy"]
    volumes:
      - ./deploy/alloy/config.alloy:/etc/alloy/config.alloy:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on: [loki]
    logging: *logging

  grafana:
    image: grafana/grafana:latest
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-change-me}
      GF_USERS_ALLOW_SIGN_UP: "false"
    volumes:
      - ./deploy/grafana/provisioning:/etc/grafana/provisioning:ro
      - ./deploy/grafana/dashboards:/var/lib/grafana/dashboards:ro
      - grafanadata:/var/lib/grafana
    ports: ["127.0.0.1:3000:3000"]
    depends_on: [prometheus, loki, jaeger]
    logging: *logging

volumes:
  pgdata:
  promdata:
  lokidata:
  grafanadata:
```

### The database

```yaml
postgres:
  image: postgres:17-alpine
  volumes: [pgdata:/var/lib/postgresql/data]
  ports: ["127.0.0.1:5432:5432"]
  healthcheck: ...
```

- **Named volume** `pgdata`: data outlives the container. `docker compose down` keeps it; `down -v` (our `make reset`) deletes it.
- `127.0.0.1:5432:5432`: publish the port *only on localhost*. Writing `5432:5432` would expose your database to every machine that can reach your laptop. Default to the narrowest binding.
- `healthcheck` runs `pg_isready`. This is the key to ordering: "container started" is not "database ready to accept connections".

### The migration job

```yaml
migrate:
  command: ["migrate"]
  depends_on: { postgres: { condition: service_healthy } }
  restart: "no"
```

A **one-shot job**: it applies migrations and exits. Why not migrate inside the API at startup, as in Part 6? Because when you run three API replicas, all three would race to migrate. A single job per deploy runs once, and the API waits for it:

```yaml
api:
  depends_on:
    migrate: { condition: service_completed_successfully }
```

The schema is therefore guaranteed to exist before any API instance takes traffic.

### The API

Notice the hardening block:

```yaml
read_only: true                       # container filesystem is immutable
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]                       # drop every Linux capability
stop_grace_period: 30s
```

A read-only filesystem means even a compromised process cannot write a backdoor to disk. Dropping all capabilities removes privileges the app never needed. `stop_grace_period` must exceed our `SHUTDOWN_TIMEOUT` (15 s): when Docker stops a container it sends `SIGTERM`, waits, then `SIGKILL`. If the wait is shorter than our graceful shutdown, in-flight requests get killed mid-drain. (Remember Part 3? This is the other half of that contract.)

The admin port `9090` is **not published**. Only Prometheus, inside the compose network, can reach it. From the host, `curl localhost:9090/metrics` fails. That's the point.

### Configuration comes from the environment

```yaml
environment:
  DATABASE_URL: postgres://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-change-me}@postgres:5432/...
```

`${VAR:-default}` reads from your shell or a `.env` file (git-ignored, with `.env.example` committed as a template). Notice the hostname `postgres`: inside a compose network, containers reach each other **by service name**.

## Part C: the observability stack

Everything in Part 7 now has somewhere to go.

### Prometheus: metrics

**Create** `deploy/prometheus/prometheus.yml`

```yaml
global:
  scrape_interval: 15s       # how often to pull /metrics
  evaluation_interval: 15s   # how often to evaluate alert rules

rule_files:
  - /etc/prometheus/alerts.yml

scrape_configs:
  - job_name: api
    static_configs:
      - targets: ["api:9090"]       # the ADMIN port, reachable only inside the compose network
  - job_name: postgres
    static_configs:
      - targets: ["postgres-exporter:9187"]
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
```

Prometheus **pulls**: every 15 seconds it scrapes `/metrics` from each target. (Pull rather than push means a dead target is itself a signal: `up == 0`.) The `api` target is `api:9090`, the admin port, reachable only on the internal network.

### Alerts

**Create** `deploy/prometheus/alerts.yml`

```yaml
# Alert rules are evaluated by Prometheus. Delivering them (email/Slack/pager)
# needs Alertmanager, which is intentionally not part of this learning stack;
# firing alerts are still visible at http://localhost:9091/alerts.
groups:
  - name: api
    rules:
      - alert: ApiDown
        expr: up{job="api"} == 0
        for: 1m
        labels: { severity: page }
        annotations: { summary: "API target is down" }

      - alert: HighErrorRate           # symptom-based: users see 5xx
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / clamp_min(sum(rate(http_requests_total[5m])), 1) > 0.05
        for: 5m
        labels: { severity: page }
        annotations: { summary: "More than 5% of requests fail with 5xx" }

      - alert: HighLatencyP95
        expr: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) > 1
        for: 10m
        labels: { severity: ticket }
        annotations: { summary: "p95 latency above 1s" }

      - alert: DbPoolExhausted         # requests are queueing for a connection
        expr: rate(db_pool_empty_acquire_total[5m]) > 0
        for: 5m
        labels: { severity: ticket }
        annotations: { summary: "DB pool is exhausted; consider raising DB_MAX_CONNS or fixing slow queries" }
```

Good alerts are **symptom-based**: they describe what *users* experience (error ratio, latency, the service being down), not internal causes (CPU 80%). Each rule has a `for:` duration so a ten-second blip doesn't wake anyone. The last alert is a nice example of connecting two parts of the series: `db_pool_empty_acquire_total` is the pool metric from Part 7, and a sustained rate means requests are queueing for database connections. (Prometheus *evaluates* these rules and shows them at `/alerts`; delivering them to Slack or a pager needs Alertmanager, which we leave out.)

### Loki and Alloy: logs

```yaml
loki:   # stores logs, indexing only labels (cheap)
alloy:  # agent: tails every container's stdout and ships it to Loki
```

**Create** `deploy/alloy/config.alloy`

```hcl
// Alloy = Grafana's telemetry agent. Pipeline: discover containers -> label -> tail logs -> push to Loki.
discovery.docker "containers" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "containers" {
  targets = discovery.docker.containers.targets

  // Turn Docker metadata into Loki labels (keep labels FEW and LOW-cardinality;
  // never use request_id or trace_id as a label).
  rule {
    source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
    target_label  = "service"
  }
}

loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.containers.output
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}
```

Our app writes JSON logs to stdout. That's the 12-factor approach: *the app doesn't manage log files; the platform collects the stream.* Alloy discovers containers through the Docker socket, labels each stream with its compose `service`, and pushes to Loki. Keep Loki labels **few and low-cardinality** (the same rule as Prometheus!). Search *inside* lines with LogQL instead: `{service="api"} | json | status >= 500`.

### Jaeger: traces

One container, receiving spans on port 4317 (the `OTLP_ENDPOINT: jaeger:4317` we set on the API) and serving a UI on 16686.

### Grafana: one window for all three

**Create** `deploy/grafana/provisioning/datasources/datasources.yml`

```yaml
apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

  - name: Jaeger
    uid: jaeger
    type: jaeger
    url: http://jaeger:16686

  - name: Loki
    uid: loki
    type: loki
    url: http://loki:3100
    jsonData:
      # Makes trace_id in a log line a clickable link that opens the trace in
      # Jaeger: the log -> trace jump.
      derivedFields:
        - name: trace_id
          matcherRegex: '"trace_id":"(\w+)"'
          url: '$${__value.raw}'
          datasourceUid: jaeger
```

**Provisioning** means Grafana is configured from files at startup: no clicking, and it works identically after `make reset`. The `derivedFields` block is the best trick in the series: it teaches Loki that `"trace_id":"abc123"` in a log line is a link into Jaeger. You can read a log line, click the trace id, and see the full request tree. That is the *connected* observability we promised in Part 7.

Grafana also needs to be told *where to find dashboards*:

**Create** `deploy/grafana/provisioning/dashboards/dashboards.yml`

```yaml
apiVersion: 1
providers:
  - name: default
    type: file
    options:
      path: /var/lib/grafana/dashboards
```

and here is a compact first dashboard. Each panel is just a PromQL query from Part 7 (request rate, error ratio, p50/p95/p99 latency, requests in flight, DB pool usage and waits) plus a live log panel from Loki:

**Create** `deploy/grafana/dashboards/api-overview.json`

```json
{
  "uid": "api-overview",
  "title": "API overview",
  "schemaVersion": 39,
  "refresh": "10s",
  "time": { "from": "now-30m", "to": "now" },
  "panels": [
    { "id": 1, "type": "timeseries", "title": "Request rate by route (req/s)",
      "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
      "targets": [ { "refId": "A", "legendFormat": "{{route}}", "expr": "sum by (route) (rate(http_requests_total[1m]))" } ] },
    { "id": 2, "type": "timeseries", "title": "Error ratio (share of 5xx)",
      "gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
      "targets": [ { "refId": "A", "legendFormat": "5xx", "expr": "(sum(rate(http_requests_total{status=~\"5..\"}[5m])) or vector(0)) / clamp_min(sum(rate(http_requests_total[5m])), 1)" } ] },
    { "id": 3, "type": "timeseries", "title": "Latency p50 / p95 / p99",
      "gridPos": { "x": 0, "y": 8, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
      "targets": [
        { "refId": "A", "legendFormat": "p50", "expr": "histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))" },
        { "refId": "B", "legendFormat": "p95", "expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))" },
        { "refId": "C", "legendFormat": "p99", "expr": "histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))" } ] },
    { "id": 4, "type": "timeseries", "title": "Requests in flight",
      "gridPos": { "x": 12, "y": 8, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "targets": [ { "refId": "A", "legendFormat": "in flight", "expr": "http_requests_in_flight" } ] },
    { "id": 5, "type": "timeseries", "title": "DB pool connections",
      "gridPos": { "x": 0, "y": 16, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "targets": [
        { "refId": "A", "legendFormat": "in use", "expr": "db_pool_acquired_conns" },
        { "refId": "B", "legendFormat": "idle", "expr": "db_pool_idle_conns" },
        { "refId": "C", "legendFormat": "max", "expr": "db_pool_max_conns" } ] },
    { "id": 6, "type": "timeseries", "title": "DB pool waits per second (exhaustion signal)",
      "gridPos": { "x": 12, "y": 16, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "targets": [ { "refId": "A", "legendFormat": "waits/s", "expr": "rate(db_pool_empty_acquire_total[1m])" } ] },
    { "id": 7, "type": "logs", "title": "API logs (click a trace_id to open the trace)",
      "gridPos": { "x": 0, "y": 24, "w": 24, "h": 10 },
      "datasource": { "type": "loki", "uid": "loki" },
      "options": { "enableLogDetails": true, "wrapLogMessage": true, "sortOrder": "Descending" },
      "targets": [ { "refId": "A", "expr": "{service=\"api\"}" } ] }
  ]
}
```

## Complete code for this part

Besides the files above, four small things change. Paste them, then you are ready to run the stack.

*Already given earlier in this post:* `.dockerignore`*,* `Dockerfile`*,* `deploy/alloy/config.alloy`*,* `deploy/grafana/provisioning/dashboards/dashboards.yml`*,* `deploy/grafana/provisioning/datasources/datasources.yml`*,* `deploy/prometheus/alerts.yml`*,* `deploy/prometheus/prometheus.yml`*,* `docker-compose.yml`*.*

**Create** `.env.example`

```plaintext
# Copy to .env (git-ignored) and change values. docker compose reads it automatically.
POSTGRES_USER=app
POSTGRES_PASSWORD=change-me
POSTGRES_DB=app
GRAFANA_ADMIN_PASSWORD=change-me
```

**Replace the contents of** `.gitignore` **with**

```plaintext
users.db
users.db-shm
users.db-wal
bin/
.env
```

**Replace the contents of** `makefile` **with**

```makefile
.PHONY: db run build test test-integration up down logs reset migrate psql load clean

BINARY := bin/server
# Local (non-Docker) default; matches the compose Postgres published on localhost.
export DATABASE_URL ?= postgres://app:change-me@localhost:5432/app?sslmode=disable
TEST_DATABASE_URL ?= $(DATABASE_URL)

# ---- Go ----------------------------------------------------------------------
run:                      ## run the API locally (needs: make db)
	go run ./cmd/rest

build:
	mkdir -p bin
	CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o $(BINARY) ./cmd/rest

migrate:                  ## apply migrations to $$DATABASE_URL
	go run ./cmd/rest migrate

test:                     ## unit tests (database tests are skipped)
	go vet ./...
	go test -race ./...

test-integration:         ## all tests incl. database tests, against compose Postgres
	docker compose up -d --wait postgres
	TEST_DATABASE_URL="$(TEST_DATABASE_URL)" go test -race -count=1 ./...

# ---- Docker ------------------------------------------------------------------
db:                       ## only Postgres (for running the API with `make run`)
	docker compose up -d --wait postgres

up:                       ## whole stack: API + Postgres + Prometheus + Grafana + Jaeger + Loki
	docker compose up -d --build --wait

down:                     ## stop, KEEP data
	docker compose down

reset:                    ## stop and DELETE all data volumes
	docker compose down -v

logs:
	docker compose logs -f api

psql:
	docker compose exec postgres sh -c 'psql -U $$POSTGRES_USER -d $$POSTGRES_DB'

load:                     ## generate some traffic so dashboards have data
	./scripts/load.sh

clean:
	rm -rf bin
```

**Create** `scripts/load.sh`

```bash
#!/usr/bin/env bash
# Generates mixed traffic (including errors) so the dashboards have something to show.
set -euo pipefail
API=${API:-http://localhost:7800}
for i in $(seq 1 ${N:-200}); do
  curl -s -o /dev/null -XPOST "$API/users" -d "{\"name\":\"user$i\"}"
  curl -s -o /dev/null "$API/users"
  curl -s -o /dev/null "$API/users/$((RANDOM % 50 + 1))"          # many 404s
  curl -s -o /dev/null -XPOST "$API/orders" -d "{\"user_id\":$((RANDOM % 20 + 1)),\"item\":\"book\",\"quantity\":2}"
  curl -s -o /dev/null -XPOST "$API/users" -d '{}'                 # 422s
done
echo "done"
```

## Try it

```bash
$ cp .env.example .env
$ make up          # builds the image, starts 9 containers, waits until they are healthy
 Container rest-go-postgres-1   Healthy
 Container rest-go-migrate-1    Exited       ← one-shot job finished
 Container rest-go-api-1        Healthy
 Container rest-go-prometheus-1 Healthy
 Container rest-go-grafana-1    Healthy
 ...

$ chmod +x scripts/load.sh   # once: a file you created by hand is not executable yet
$ make load        # generates mixed traffic, including errors
```

Then take the tour:

1. **Grafana** → <http://localhost:3000> (admin / the password in `.env`) → *API overview*.
2. **Explore → Loki**: `{service="api"} | json | status=422`. Click `trace_id` on a line.
3. **Jaeger** opens that trace: an HTTP span with `pool.acquire` and `INSERT` children.
4. **Prometheus** → `/targets` (all `UP`) and `/alerts`.
5. **Break it on purpose**: `docker compose pause postgres`. Watch `/readyz` return 503, requests fail after the 10 s timeout, and the error ratio climb. `docker compose unpause postgres` and watch recovery. Nothing teaches monitoring like watching it notice an outage you caused.

Verify graceful shutdown: `docker compose stop api && docker compose logs api` shows `shutdown signal received` then `server stopped cleanly`, and `docker inspect` reports exit code 0.

## Recap

- Multi-stage builds: build with the toolchain, ship only the binary; distroless + non-root.
- Order dependent services with **healthchecks** and `depends_on: condition`.
- Run migrations as a one-shot job, not inside every replica.
- Publish only what must be public; bind tools to `127.0.0.1`.
- Harden containers: read-only filesystem, no capabilities, adequate stop grace period.
- Provision monitoring as code so it's reproducible.

## Exercises

1. Run `docker image ls` and compare your image to one built `FROM golang:alpine` as the final stage. Then `docker run --rm -it --entrypoint sh <image>`: why does it fail on the distroless one?
2. Add an alert for "p99 latency above 2 s for 10 minutes" and trigger it by adding `pg_sleep(3)` to a query.
3. Change `MaxConns` to 1 in the compose environment, run `make load` from two terminals, and find the effect on the *DB pool waits* panel and in a trace's `pool.acquire` span.