Skip to main content

Part 8: Docker and docker compose: ship the app, run the whole stack

Updated 17 min readMarkdown
Part of seriesBuild a production-grade REST API in Go

A 23 MB non-root image, a one-command environment with Postgres, Prometheus, Grafana, Jaeger and Loki, and the reasons behind each line.


Series: Build a production-grade REST API in Go · Part 8 of 17 Where we are: the observable API from Part 7. This post adds no Go code: it packages the app and runs the tools that display what Part 7 emits.

"It works on my machine" is the oldest deployment bug. A container image packages the program and everything it needs so it runs identically on your laptop, in CI and in production. docker compose describes a set of containers (our app, the database, the monitoring tools) that start together.

By the end of this post, one command starts everything:

make up

Part A: the image

Create Dockerfile

# syntax=docker/dockerfile:1

# ---- Stage 1: build ---------------------------------------------------------
FROM golang:1.27-alpine AS build
WORKDIR /src

# Copy ONLY the dependency manifests first and download. Docker caches each
# layer; as long as go.mod/go.sum are unchanged this slow step is skipped, even
# when source code changes on every build.
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download

COPY . .
# CGO_ENABLED=0 -> a fully static binary (pgx is pure Go), so it runs on a
#                  minimal image with no libc.
# -trimpath    -> strip local file paths from the binary (reproducible, no leaks)
# -s -w        -> drop symbol/debug tables (~30% smaller)
RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/server ./cmd/rest

# ---- Stage 2: runtime -------------------------------------------------------
# distroless/static: no shell, no package manager, no curl: almost nothing for
# an attacker to use. ":nonroot" runs as uid 65532 instead of root.
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/server /server
EXPOSE 7800 9090

# distroless has no curl, so the binary probes itself (see `healthcheck` mode).
HEALTHCHECK --interval=10s --timeout=4s --start-period=10s --retries=3 \
    CMD ["/server", "healthcheck"]

ENTRYPOINT ["/server"]

This is a multi-stage build. Two FROM lines mean two stages:

  1. build uses golang:alpine, an image with the full Go toolchain (hundreds of MB), to compile the program.
  2. The final stage starts from distroless/static and copies only the compiled binary from the first stage. The toolchain is thrown away.

The result is a 23 MB image instead of ~800 MB. Smaller is not just about disk: it's about attack surface. Distroless has no shell, no package manager, no curl, almost nothing an intruder could use if they got in. And it runs as nonroot (uid 65532), so a compromise doesn't own the container.

Line-by-line reasons:

  • Copy go.mod/go.sum first, then go mod download, then copy the source. Docker caches each instruction as a layer and reuses a layer if its inputs are unchanged. Dependencies change rarely; source changes constantly. By splitting them, a code change only re-runs the (fast) compile step, not the slow download.
  • --mount=type=cache keeps Go's module and build caches between builds.
  • CGO_ENABLED=0 produces a fully static binary with no C library dependency. This is why we chose pure-Go drivers (pgx) all along: a static binary runs on a minimal image with no libc.
  • -trimpath -ldflags="-s -w" removes local file paths and debug tables: smaller, and doesn't leak your directory layout.
  • The Go version must match. FROM golang:1.27-alpine has to be at least the version in your go.mod (go mod init writes the version of the Go on your machine). If the build fails with go.mod requires go >= 1.27.3 (running go 1.27.2), your cached image is stale: run docker pull golang:1.27-alpine (or docker compose build --pull) and try again.
  • HEALTHCHECK: the image has no curl, so the binary checks itself. Our main has a healthcheck subcommand that calls its own /readyz. One binary, three modes: serve, migrate, healthcheck.

And the file that keeps junk out of the build context:

Create .dockerignore

.git
bin/
*.db*
.env
docs/
deploy/
docker-compose.yml

(Excluding .git, local databases and .env also matters for security: you never want secrets baked into an image layer.)

Part B: docker compose, service by service

Create docker-compose.yml

# Full local stack. `make up` -> API :7800, Grafana :3000, Prometheus :9091, Jaeger :16686
name: rest-go

x-logging: &logging   # cap container log files so disks don't fill up
  driver: json-file
  options: { max-size: "10m", max-file: "3" }

services:
  # ---------------------------------------------------------------- database
  postgres:
    image: postgres:17-alpine
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-app}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-change-me}
      POSTGRES_DB: ${POSTGRES_DB:-app}
    volumes:
      - pgdata:/var/lib/postgresql/data      # named volume: data survives `down`
    ports:
      - "127.0.0.1:5432:5432"                # localhost only, never 0.0.0.0
    healthcheck:                             # lets other services wait until it is READY
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 5s
      timeout: 3s
      retries: 10
    logging: *logging

  # One-shot job: apply migrations, then exit 0. The API starts only after it
  # succeeded, so the schema is always ready, and migrations run once (not once
  # per API replica).
  migrate:
    build: .
    command: ["migrate"]
    environment: &dburl
      DATABASE_URL: postgres://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-change-me}@postgres:5432/${POSTGRES_DB:-app}?sslmode=disable
    depends_on:
      postgres: { condition: service_healthy }
    restart: "no"
    logging: *logging

  # --------------------------------------------------------------------- api
  api:
    build: .
    environment:
      <<: *dburl
      ADDR: ":7800"
      ADMIN_ADDR: ":9090"
      OTLP_ENDPOINT: jaeger:4317
      LOG_LEVEL: info
    ports:
      - "7800:7800"                          # public API only; admin port 9090 stays internal
    depends_on:
      migrate: { condition: service_completed_successfully }
      jaeger: { condition: service_started }
    restart: unless-stopped
    read_only: true                          # container filesystem is immutable
    security_opt: ["no-new-privileges:true"]
    cap_drop: [ALL]
    stop_grace_period: 30s                   # > SHUTDOWN_TIMEOUT so Docker doesn't SIGKILL mid-drain
    logging: *logging

  # ----------------------------------------------------------- observability
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./deploy/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./deploy/prometheus/alerts.yml:/etc/prometheus/alerts.yml:ro
      - promdata:/prometheus
    ports: ["127.0.0.1:9091:9090"]
    depends_on: [api]
    logging: *logging

  postgres-exporter:                         # turns Postgres internals into metrics
    image: prometheuscommunity/postgres-exporter:latest
    environment:
      DATA_SOURCE_NAME: postgres://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-change-me}@postgres:5432/${POSTGRES_DB:-app}?sslmode=disable
    depends_on:
      postgres: { condition: service_healthy }
    logging: *logging

  jaeger:                                    # trace storage + UI; receives OTLP on :4317
    image: jaegertracing/all-in-one:latest
    ports: ["127.0.0.1:16686:16686"]
    logging: *logging

  loki:                                      # log storage (indexes labels only, so it is cheap)
    image: grafana/loki:latest
    volumes: [lokidata:/loki]
    logging: *logging

  alloy:                                     # log shipper: tails every container's stdout -> Loki
    image: grafana/alloy:latest
    command: ["run", "--server.http.listen-addr=0.0.0.0:12345", "--storage.path=/var/lib/alloy/data", "/etc/alloy/config.alloy"]
    volumes:
      - ./deploy/alloy/config.alloy:/etc/alloy/config.alloy:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on: [loki]
    logging: *logging

  grafana:
    image: grafana/grafana:latest
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-change-me}
      GF_USERS_ALLOW_SIGN_UP: "false"
    volumes:
      - ./deploy/grafana/provisioning:/etc/grafana/provisioning:ro
      - ./deploy/grafana/dashboards:/var/lib/grafana/dashboards:ro
      - grafanadata:/var/lib/grafana
    ports: ["127.0.0.1:3000:3000"]
    depends_on: [prometheus, loki, jaeger]
    logging: *logging

volumes:
  pgdata:
  promdata:
  lokidata:
  grafanadata:

The database

postgres:
  image: postgres:17-alpine
  volumes: [pgdata:/var/lib/postgresql/data]
  ports: ["127.0.0.1:5432:5432"]
  healthcheck: ...
  • Named volume pgdata: data outlives the container. docker compose down keeps it; down -v (our make reset) deletes it.
  • 127.0.0.1:5432:5432: publish the port only on localhost. Writing 5432:5432 would expose your database to every machine that can reach your laptop. Default to the narrowest binding.
  • healthcheck runs pg_isready. This is the key to ordering: "container started" is not "database ready to accept connections".

The migration job

migrate:
  command: ["migrate"]
  depends_on: { postgres: { condition: service_healthy } }
  restart: "no"

A one-shot job: it applies migrations and exits. Why not migrate inside the API at startup, as in Part 6? Because when you run three API replicas, all three would race to migrate. A single job per deploy runs once, and the API waits for it:

api:
  depends_on:
    migrate: { condition: service_completed_successfully }

The schema is therefore guaranteed to exist before any API instance takes traffic.

The API

Notice the hardening block:

read_only: true                       # container filesystem is immutable
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]                       # drop every Linux capability
stop_grace_period: 30s

A read-only filesystem means even a compromised process cannot write a backdoor to disk. Dropping all capabilities removes privileges the app never needed. stop_grace_period must exceed our SHUTDOWN_TIMEOUT (15 s): when Docker stops a container it sends SIGTERM, waits, then SIGKILL. If the wait is shorter than our graceful shutdown, in-flight requests get killed mid-drain. (Remember Part 3? This is the other half of that contract.)

The admin port 9090 is not published. Only Prometheus, inside the compose network, can reach it. From the host, curl localhost:9090/metrics fails. That's the point.

Configuration comes from the environment

environment:
  DATABASE_URL: postgres://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-change-me}@postgres:5432/...

${VAR:-default} reads from your shell or a .env file (git-ignored, with .env.example committed as a template). Notice the hostname postgres: inside a compose network, containers reach each other by service name.

Part C: the observability stack

Everything in Part 7 now has somewhere to go.

Prometheus: metrics

Create deploy/prometheus/prometheus.yml

global:
  scrape_interval: 15s       # how often to pull /metrics
  evaluation_interval: 15s   # how often to evaluate alert rules

rule_files:
  - /etc/prometheus/alerts.yml

scrape_configs:
  - job_name: api
    static_configs:
      - targets: ["api:9090"]       # the ADMIN port, reachable only inside the compose network
  - job_name: postgres
    static_configs:
      - targets: ["postgres-exporter:9187"]
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

Prometheus pulls: every 15 seconds it scrapes /metrics from each target. (Pull rather than push means a dead target is itself a signal: up == 0.) The api target is api:9090, the admin port, reachable only on the internal network.

Alerts

Create deploy/prometheus/alerts.yml

# Alert rules are evaluated by Prometheus. Delivering them (email/Slack/pager)
# needs Alertmanager, which is intentionally not part of this learning stack;
# firing alerts are still visible at http://localhost:9091/alerts.
groups:
  - name: api
    rules:
      - alert: ApiDown
        expr: up{job="api"} == 0
        for: 1m
        labels: { severity: page }
        annotations: { summary: "API target is down" }

      - alert: HighErrorRate           # symptom-based: users see 5xx
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / clamp_min(sum(rate(http_requests_total[5m])), 1) > 0.05
        for: 5m
        labels: { severity: page }
        annotations: { summary: "More than 5% of requests fail with 5xx" }

      - alert: HighLatencyP95
        expr: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) > 1
        for: 10m
        labels: { severity: ticket }
        annotations: { summary: "p95 latency above 1s" }

      - alert: DbPoolExhausted         # requests are queueing for a connection
        expr: rate(db_pool_empty_acquire_total[5m]) > 0
        for: 5m
        labels: { severity: ticket }
        annotations: { summary: "DB pool is exhausted; consider raising DB_MAX_CONNS or fixing slow queries" }

Good alerts are symptom-based: they describe what users experience (error ratio, latency, the service being down), not internal causes (CPU 80%). Each rule has a for: duration so a ten-second blip doesn't wake anyone. The last alert is a nice example of connecting two parts of the series: db_pool_empty_acquire_total is the pool metric from Part 7, and a sustained rate means requests are queueing for database connections. (Prometheus evaluates these rules and shows them at /alerts; delivering them to Slack or a pager needs Alertmanager, which we leave out.)

Loki and Alloy: logs

loki:   # stores logs, indexing only labels (cheap)
alloy:  # agent: tails every container's stdout and ships it to Loki

Create deploy/alloy/config.alloy

// Alloy = Grafana's telemetry agent. Pipeline: discover containers -> label -> tail logs -> push to Loki.
discovery.docker "containers" {
  host = "unix:///var/run/docker.sock"
}

discovery.relabel "containers" {
  targets = discovery.docker.containers.targets

  // Turn Docker metadata into Loki labels (keep labels FEW and LOW-cardinality;
  // never use request_id or trace_id as a label).
  rule {
    source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
    target_label  = "service"
  }
}

loki.source.docker "containers" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.containers.output
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Our app writes JSON logs to stdout. That's the 12-factor approach: the app doesn't manage log files; the platform collects the stream. Alloy discovers containers through the Docker socket, labels each stream with its compose service, and pushes to Loki. Keep Loki labels few and low-cardinality (the same rule as Prometheus!). Search inside lines with LogQL instead: {service="api"} | json | status >= 500.

Jaeger: traces

One container, receiving spans on port 4317 (the OTLP_ENDPOINT: jaeger:4317 we set on the API) and serving a UI on 16686.

Grafana: one window for all three

Create deploy/grafana/provisioning/datasources/datasources.yml

apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true

  - name: Jaeger
    uid: jaeger
    type: jaeger
    url: http://jaeger:16686

  - name: Loki
    uid: loki
    type: loki
    url: http://loki:3100
    jsonData:
      # Makes trace_id in a log line a clickable link that opens the trace in
      # Jaeger: the log -> trace jump.
      derivedFields:
        - name: trace_id
          matcherRegex: '"trace_id":"(\w+)"'
          url: '$${__value.raw}'
          datasourceUid: jaeger

Provisioning means Grafana is configured from files at startup: no clicking, and it works identically after make reset. The derivedFields block is the best trick in the series: it teaches Loki that "trace_id":"abc123" in a log line is a link into Jaeger. You can read a log line, click the trace id, and see the full request tree. That is the connected observability we promised in Part 7.

Grafana also needs to be told where to find dashboards:

Create deploy/grafana/provisioning/dashboards/dashboards.yml

apiVersion: 1
providers:
  - name: default
    type: file
    options:
      path: /var/lib/grafana/dashboards

and here is a compact first dashboard. Each panel is just a PromQL query from Part 7 (request rate, error ratio, p50/p95/p99 latency, requests in flight, DB pool usage and waits) plus a live log panel from Loki:

Create deploy/grafana/dashboards/api-overview.json

{
  "uid": "api-overview",
  "title": "API overview",
  "schemaVersion": 39,
  "refresh": "10s",
  "time": { "from": "now-30m", "to": "now" },
  "panels": [
    { "id": 1, "type": "timeseries", "title": "Request rate by route (req/s)",
      "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
      "targets": [ { "refId": "A", "legendFormat": "{{route}}", "expr": "sum by (route) (rate(http_requests_total[1m]))" } ] },
    { "id": 2, "type": "timeseries", "title": "Error ratio (share of 5xx)",
      "gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
      "targets": [ { "refId": "A", "legendFormat": "5xx", "expr": "(sum(rate(http_requests_total{status=~\"5..\"}[5m])) or vector(0)) / clamp_min(sum(rate(http_requests_total[5m])), 1)" } ] },
    { "id": 3, "type": "timeseries", "title": "Latency p50 / p95 / p99",
      "gridPos": { "x": 0, "y": 8, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
      "targets": [
        { "refId": "A", "legendFormat": "p50", "expr": "histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))" },
        { "refId": "B", "legendFormat": "p95", "expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))" },
        { "refId": "C", "legendFormat": "p99", "expr": "histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))" } ] },
    { "id": 4, "type": "timeseries", "title": "Requests in flight",
      "gridPos": { "x": 12, "y": 8, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "targets": [ { "refId": "A", "legendFormat": "in flight", "expr": "http_requests_in_flight" } ] },
    { "id": 5, "type": "timeseries", "title": "DB pool connections",
      "gridPos": { "x": 0, "y": 16, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "targets": [
        { "refId": "A", "legendFormat": "in use", "expr": "db_pool_acquired_conns" },
        { "refId": "B", "legendFormat": "idle", "expr": "db_pool_idle_conns" },
        { "refId": "C", "legendFormat": "max", "expr": "db_pool_max_conns" } ] },
    { "id": 6, "type": "timeseries", "title": "DB pool waits per second (exhaustion signal)",
      "gridPos": { "x": 12, "y": 16, "w": 12, "h": 8 },
      "datasource": { "type": "prometheus", "uid": "prometheus" },
      "targets": [ { "refId": "A", "legendFormat": "waits/s", "expr": "rate(db_pool_empty_acquire_total[1m])" } ] },
    { "id": 7, "type": "logs", "title": "API logs (click a trace_id to open the trace)",
      "gridPos": { "x": 0, "y": 24, "w": 24, "h": 10 },
      "datasource": { "type": "loki", "uid": "loki" },
      "options": { "enableLogDetails": true, "wrapLogMessage": true, "sortOrder": "Descending" },
      "targets": [ { "refId": "A", "expr": "{service=\"api\"}" } ] }
  ]
}

Complete code for this part

Besides the files above, four small things change. Paste them, then you are ready to run the stack.

Already given earlier in this post: .dockerignore, Dockerfile, deploy/alloy/config.alloy, deploy/grafana/provisioning/dashboards/dashboards.yml, deploy/grafana/provisioning/datasources/datasources.yml, deploy/prometheus/alerts.yml, deploy/prometheus/prometheus.yml, docker-compose.yml.

Create .env.example

# Copy to .env (git-ignored) and change values. docker compose reads it automatically.
POSTGRES_USER=app
POSTGRES_PASSWORD=change-me
POSTGRES_DB=app
GRAFANA_ADMIN_PASSWORD=change-me

Replace the contents of .gitignore with

users.db
users.db-shm
users.db-wal
bin/
.env

Replace the contents of makefile with

.PHONY: db run build test test-integration up down logs reset migrate psql load clean

BINARY := bin/server
# Local (non-Docker) default; matches the compose Postgres published on localhost.
export DATABASE_URL ?= postgres://app:change-me@localhost:5432/app?sslmode=disable
TEST_DATABASE_URL ?= $(DATABASE_URL)

# ---- Go ----------------------------------------------------------------------
run:                      ## run the API locally (needs: make db)
	go run ./cmd/rest

build:
	mkdir -p bin
	CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o $(BINARY) ./cmd/rest

migrate:                  ## apply migrations to $$DATABASE_URL
	go run ./cmd/rest migrate

test:                     ## unit tests (database tests are skipped)
	go vet ./...
	go test -race ./...

test-integration:         ## all tests incl. database tests, against compose Postgres
	docker compose up -d --wait postgres
	TEST_DATABASE_URL="$(TEST_DATABASE_URL)" go test -race -count=1 ./...

# ---- Docker ------------------------------------------------------------------
db:                       ## only Postgres (for running the API with `make run`)
	docker compose up -d --wait postgres

up:                       ## whole stack: API + Postgres + Prometheus + Grafana + Jaeger + Loki
	docker compose up -d --build --wait

down:                     ## stop, KEEP data
	docker compose down

reset:                    ## stop and DELETE all data volumes
	docker compose down -v

logs:
	docker compose logs -f api

psql:
	docker compose exec postgres sh -c 'psql -U $$POSTGRES_USER -d $$POSTGRES_DB'

load:                     ## generate some traffic so dashboards have data
	./scripts/load.sh

clean:
	rm -rf bin

Create scripts/load.sh

#!/usr/bin/env bash
# Generates mixed traffic (including errors) so the dashboards have something to show.
set -euo pipefail
API=${API:-http://localhost:7800}
for i in $(seq 1 ${N:-200}); do
  curl -s -o /dev/null -XPOST "$API/users" -d "{\"name\":\"user$i\"}"
  curl -s -o /dev/null "$API/users"
  curl -s -o /dev/null "$API/users/$((RANDOM % 50 + 1))"          # many 404s
  curl -s -o /dev/null -XPOST "$API/orders" -d "{\"user_id\":$((RANDOM % 20 + 1)),\"item\":\"book\",\"quantity\":2}"
  curl -s -o /dev/null -XPOST "$API/users" -d '{}'                 # 422s
done
echo "done"

Try it

$ cp .env.example .env
$ make up          # builds the image, starts 9 containers, waits until they are healthy
 Container rest-go-postgres-1   Healthy
 Container rest-go-migrate-1    Exited       ← one-shot job finished
 Container rest-go-api-1        Healthy
 Container rest-go-prometheus-1 Healthy
 Container rest-go-grafana-1    Healthy
 ...

$ chmod +x scripts/load.sh   # once: a file you created by hand is not executable yet
$ make load        # generates mixed traffic, including errors

Then take the tour:

  1. Grafana → http://localhost:3000 (admin / the password in .env) → API overview.
  2. Explore → Loki: {service="api"} | json | status=422. Click trace_id on a line.
  3. Jaeger opens that trace: an HTTP span with pool.acquire and INSERT children.
  4. Prometheus → /targets (all UP) and /alerts.
  5. Break it on purpose: docker compose pause postgres. Watch /readyz return 503, requests fail after the 10 s timeout, and the error ratio climb. docker compose unpause postgres and watch recovery. Nothing teaches monitoring like watching it notice an outage you caused.

Verify graceful shutdown: docker compose stop api && docker compose logs api shows shutdown signal received then server stopped cleanly, and docker inspect reports exit code 0.

Recap

  • Multi-stage builds: build with the toolchain, ship only the binary; distroless + non-root.
  • Order dependent services with healthchecks and depends_on: condition.
  • Run migrations as a one-shot job, not inside every replica.
  • Publish only what must be public; bind tools to 127.0.0.1.
  • Harden containers: read-only filesystem, no capabilities, adequate stop grace period.
  • Provision monitoring as code so it's reproducible.

Exercises

  1. Run docker image ls and compare your image to one built FROM golang:alpine as the final stage. Then docker run --rm -it --entrypoint sh <image>: why does it fail on the distroless one?
  2. Add an alert for "p99 latency above 2 s for 10 minutes" and trigger it by adding pg_sleep(3) to a query.
  3. Change MaxConns to 1 in the compose environment, run make load from two terminals, and find the effect on the DB pool waits panel and in a trace's pool.acquire span.

No comments yet. Be the first to comment.

More from this blog