# Part 15: Idempotency keys: making retries safe *TLS stops anyone from reading the traffic. It can't stop your own client from sending the same order twice. How to make* `POST` *safe to retry, and why the key must be claimed inside the same database transaction as the write.* --- **Series:** Build a production-grade REST API in Go · **Part 15 of 17** **Where we are:** the TLS-protected API from Part 14. Here is a bug that no amount of validation, authentication or encryption prevents: 1. A customer taps **Buy**. The app sends `POST /orders`. 2. The server creates the order and sends `201 Created`. 3. The phone enters a tunnel. The response never arrives. 4. The app times out. What should it do? If it **gives up**, the customer may never get the order they paid for. If it **retries**, they may get two. The client cannot tell which, because *the network cannot tell it*: "the request was lost" and "the response was lost" look identical from the outside. Mobile clients, load balancers, API gateways and HTTP libraries all retry automatically, so this is not a rare edge case. It is Tuesday. The fix is to make the operation **idempotent**: performing it N times has the same effect as performing it once. - `GET`, `PUT` and `DELETE` are idempotent by definition (deleting something twice leaves it deleted). - `POST` **is not.** Two identical POSTs create two things. An `Idempotency-Key` header turns a POST into a safe-to-retry operation. This is how payment APIs have worked for years; the IETF has a draft standard for it. ## The contract ```plaintext POST /orders Idempotency-Key: 7c9e6679-7425-40de-944b-e07fc1f90ae7 ← the client generates a unique value (a UUID) {"item":"book","quantity":2} ``` | Situation | Response | | --- | --- | | first request with this key | `201`, order created | | same key, same request (a retry) | `201` with the **original** order and `Idempotent-Replayed: true`. Nothing new is created | | same key, **different** request | `422`: a key identifies one request; reusing it is a client bug | | two identical requests at the same instant | one creates the order; the other waits and gets the same result | | invalid key (empty, over 255 chars, spaces, non-ASCII) | `400` | | the first attempt **failed** (5xx, validation error, deleted account) | nothing is stored; retrying with the same key just tries again | | no header at all | works exactly as before | The last row is a deliberate choice: the key is optional here. Payment-style APIs often make it mandatory; that is a one-line change in the handler. ## The naive design, and its hole The obvious implementation is middleware: > If I've seen this key, return the saved response. Otherwise run the handler and save its response. It looks right and has three problems. 1. **A crash window.** "Run the handler" and "save the response" are two separate steps. If the process dies in between, the order exists but nothing records the key. The client's retry runs the write **again**. You've built a duplicate-creating bug *inside* your duplicate-prevention code. 2. **Concurrent duplicates.** A double-click sends two identical requests at once. Both check, both find nothing, both run. To prevent this you need an "in progress" marker, which brings its own problems: what if the process that set it dies? 3. **Stale locks.** Now you need lock timeouts and cleanup, and the timeout becomes a tuning knob that can be wrong in both directions. ## The better design: claim the key inside the business transaction The insight: you already have a tool that makes several writes succeed or fail together, a **database transaction**. If the *key* and the *order* are written in the same transaction, "the key exists" and "the order exists" become one atomic fact. ![](https://cdn.hashnode.com/uploads/covers/6398c90a452f4e3f4cb65bac/c8fdb240-fefd-4fd0-965e-1809b7e781fe.png align="center") This quietly solves all three problems: - **Crash safety.** A crash anywhere before `COMMIT` rolls back *both* the order and the key. The retry starts clean. - **Concurrency, for free.** PostgreSQL guarantees that a second `INSERT ... ON CONFLICT` on a key whose first insert is still uncommitted **waits** for that transaction. If the first commits, the second sees the conflict and replays the stored result. If the first rolls back, the second wins the claim. We wrote no locking code. - **No stale locks.** There is no separate "in progress" state to abandon. An uncommitted transaction vanishes when its connection dies. ### Step 1: the table **Create** `internal/storage/migrations/00005_idempotency_keys.sql` ```sql -- +goose Up -- One row per (user, Idempotency-Key). The row is inserted in the SAME -- transaction that performs the business write, so "the key exists" and "the -- write happened" are one atomic fact: either both are committed or neither is. CREATE TABLE idempotency_keys ( user_id BIGINT NOT NULL REFERENCES users (id) ON DELETE CASCADE, key TEXT NOT NULL, -- SHA-256 of the request's meaning. Reusing a key with a DIFFERENT request is a client bug we must reject. fingerprint BYTEA NOT NULL, -- The stored response, replayed verbatim on retries. response_status INTEGER, response_body BYTEA, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), PRIMARY KEY (user_id, key) ); -- The purge job deletes by age; without this it would scan the whole table. CREATE INDEX idx_idempotency_keys_created_at ON idempotency_keys (created_at); -- +goose Down DROP TABLE idempotency_keys; ``` - The **primary key is** `(user_id, key)`: keys are scoped to the user. Two users can choose the same key string without colliding, and nobody can probe whether *another* user's key exists. (Remember IDOR from Part 12? This is the same principle applied to a new table.) - `fingerprint` and the stored `response_*` columns support replay and misuse detection. - The index on `created_at` makes the periodic cleanup cheap. ### Step 2: the transactional helper `internal/storage/idempotency.go (excerpt)` ```go // ErrIdempotencyKeyReuse: the same Idempotency-Key was sent with a different // request. That is a client bug (or an attack), never a retry, so we refuse // instead of guessing which request the client meant. var ErrIdempotencyKeyReuse = errors.New("idempotency key reused with a different request") // IdempotentResponse is the stored outcome of an operation. type IdempotentResponse struct { Status int Body []byte Replayed bool // true when this came from an earlier request with the same key } // RunIdempotent executes fn at most once per (userID, key), and returns the // original result to every later caller. // // THE DESIGN: claim the key first, inside the same database transaction as the // business write. // // 1. INSERT the key row. ON CONFLICT DO NOTHING tells us whether we won. // 2. Won: run fn (which writes using the SAME tx), store its response on the // key row, COMMIT. The business write and the key become visible together. // 3. Lost: someone already used this key. Compare fingerprints and replay. // // Why this is better than a middleware that records responses separately: // - Crash safety. If the process dies after the business write but before the // response is recorded, a separate-record design leaves a window where a // retry would run the write AGAIN. Here the tx rolls back as a unit. // - Concurrency for free. Two simultaneous requests with one key: the second // INSERT blocks on the first's uncommitted row (a Postgres guarantee). When // the first commits, the second sees a conflict and replays; if the first // rolls back, the second wins the claim. No "in progress" state, no lock // timeouts, no stale locks to clean up. // // Errors from fn roll everything back, including the key, so failed attempts // are naturally retryable and never replayed. func RunIdempotent( ctx context.Context, pool *pgxpool.Pool, userID int, key string, fingerprint []byte, ttl time.Duration, fn func(tx pgx.Tx) (status int, body []byte, err error), ) (IdempotentResponse, error) { tx, err := pool.Begin(ctx) if err != nil { return IdempotentResponse{}, fmt.Errorf("begin: %w", err) } defer tx.Rollback(ctx) // no-op after Commit // An expired key counts as unused. (The purge job removes old rows in bulk; // this makes expiry exact for the one key we are looking at.) if _, err := tx.Exec(ctx, "DELETE FROM idempotency_keys WHERE user_id = $1 AND key = $2 AND created_at < now() - make_interval(secs => $3)", userID, key, ttl.Seconds()); err != nil { return IdempotentResponse{}, fmt.Errorf("expire key: %w", err) } tag, err := tx.Exec(ctx, "INSERT INTO idempotency_keys (user_id, key, fingerprint) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING", userID, key, fingerprint) if isForeignKeyViolation(err) { // The key row references users(id): a deleted user cannot own a key. Same // meaning as the business insert failing for a missing user. return IdempotentResponse{}, ErrUserNotFound } if err != nil { return IdempotentResponse{}, fmt.Errorf("claim key: %w", err) } if tag.RowsAffected() == 0 { // lost the claim: replay var ( stored []byte status *int body []byte ) err := tx.QueryRow(ctx, "SELECT fingerprint, response_status, response_body FROM idempotency_keys WHERE user_id = $1 AND key = $2", userID, key).Scan(&stored, &status, &body) if err != nil { return IdempotentResponse{}, fmt.Errorf("load key: %w", err) } if !bytes.Equal(stored, fingerprint) { return IdempotentResponse{}, ErrIdempotencyKeyReuse } if status == nil { // cannot happen: rows are committed together with their response return IdempotentResponse{}, errors.New("idempotency key has no stored response") } return IdempotentResponse{Status: *status, Body: body, Replayed: true}, nil } status, body, err := fn(tx) if err != nil { return IdempotentResponse{}, err // deferred Rollback drops the key too } if _, err := tx.Exec(ctx, "UPDATE idempotency_keys SET response_status = $3, response_body = $4 WHERE user_id = $1 AND key = $2", userID, key, status, body); err != nil { return IdempotentResponse{}, fmt.Errorf("store response: %w", err) } if err := tx.Commit(ctx); err != nil { return IdempotentResponse{}, fmt.Errorf("commit: %w", err) } return IdempotentResponse{Status: status, Body: body}, nil } ``` Read it as the diagram above, made concrete. Points worth noticing: - `defer tx.Rollback(ctx)` after `Begin`. After a successful `Commit` it's a no-op; on any early return it undoes everything, including the claimed key. This is why a failed attempt consumes nothing. - `fn` **receives the** `tx`**.** The business write must use *this* transaction, not the pool, or the atomicity is an illusion. - The helper is **generic**: it returns a status and body for any operation. Protecting another endpoint is a matter of passing a different closure. - Expiry is exact: an expired key for *this* user and key string is deleted before claiming, so a stale row never causes a wrong replay. ### Step 3: sharing the insert between "plain" and "in a transaction" `OrderStore.Create` used to talk to the pool directly. We need the same SQL to run on a transaction. A two-method interface that both a pool and a transaction satisfy is all it takes: `internal/storage/orders.go (excerpt)` ```go // queryRower is satisfied by both *pgxpool.Pool and pgx.Tx, so the same SQL can // run on its own or inside a larger transaction. type queryRower interface { QueryRow(ctx context.Context, sql string, args ...any) pgx.Row } func insertOrder(ctx context.Context, q queryRower, userID int, item string, quantity int) (*Order, error) { o, err := scanOrder(q.QueryRow(ctx, ` INSERT INTO orders (user_id, item, quantity) SELECT $1::bigint, $2::text, $3::int WHERE EXISTS (SELECT 1 FROM users WHERE id = $1) RETURNING `+orderCols, userID, item, quantity)) if errors.Is(err, pgx.ErrNoRows) { return nil, ErrUserNotFound } if err != nil { return nil, fmt.Errorf("insert order: %w", err) } return o, nil } // CreateIdempotent is Create made safe to retry. The same (user, key) with the // same fingerprint creates the order once and returns that same order to every // caller; replayed reports whether this call was such a retry. func (s *OrderStore) CreateIdempotent(ctx context.Context, userID int, key string, fingerprint []byte, item string, quantity int) (o *Order, replayed bool, err error) { res, err := RunIdempotent(ctx, s.pool, userID, key, fingerprint, s.idempotencyTTL, func(tx pgx.Tx) (int, []byte, error) { created, err := insertOrder(ctx, tx, userID, item, quantity) // same tx as the key if err != nil { return 0, nil, err } body, err := json.Marshal(created) return statusCreated, body, err }) if err != nil { return nil, false, err } var out Order if err := json.Unmarshal(res.Body, &out); err != nil { return nil, false, fmt.Errorf("decode stored response: %w", err) } return &out, res.Replayed, nil } ``` (`pgxpool.Pool` and `pgx.Tx` both have `QueryRow`, so both satisfy `queryRower` without either knowing about it. Go interfaces at their most useful.) A layering detail: the storage layer must not import `net/http`, so the stored status is the plain constant `201` with a comment rather than `http.StatusCreated`. ## Step 4: the HTTP side Validate the key, compute a fingerprint, call the store, and map the outcomes: `internal/handlers/orders.go (excerpt)` ```go func (h *OrderHandler) Create(w http.ResponseWriter, r *http.Request) { var in orderInput if !h.decodeJSON(w, r, &in) { return } in.Item = strings.TrimSpace(in.Item) switch { case in.Item == "" || utf8.RuneCountInString(in.Item) > h.limits.MaxNameLen: h.writeError(w, r, http.StatusUnprocessableEntity, fmt.Sprintf("item is required and at most %d characters", h.limits.MaxNameLen)) return case in.Quantity < 1 || in.Quantity > maxQuantity: h.writeError(w, r, http.StatusUnprocessableEntity, fmt.Sprintf("quantity must be between 1 and %d", maxQuantity)) return } p, _ := auth.FromContext(r.Context()) var ( o *storage.Order replayed bool err error ) if key := r.Header.Get("Idempotency-Key"); key != "" { if !validIdempotencyKey(key) { h.writeError(w, r, http.StatusBadRequest, "Idempotency-Key must be 1-255 visible ASCII characters (a UUID is ideal)") return } o, replayed, err = h.store.CreateIdempotent(r.Context(), p.UserID, key, fingerprintOrder(in), in.Item, in.Quantity) } else { o, err = h.store.Create(r.Context(), p.UserID, in.Item, in.Quantity) } switch { case errors.Is(err, storage.ErrIdempotencyKeyReuse): // 422 per the IETF Idempotency-Key draft: the key is fine, the request // that came with it is not the one it was first used for. h.writeError(w, r, http.StatusUnprocessableEntity, "Idempotency-Key was already used with a different request; send a new key for a new request") return case errors.Is(err, storage.ErrUserNotFound): // The token is valid but its user was deleted since: treat as unauthenticated. h.writeError(w, r, http.StatusUnauthorized, "account no longer exists") return case err != nil: h.internalError(w, r, "create order", err) return } if replayed { w.Header().Set("Idempotent-Replayed", "true") // tells the client "this is the original result" h.log.InfoContext(r.Context(), "idempotent replay", "order_id", o.ID) if h.onReplay != nil { h.onReplay() } } h.writeJSON(w, r, http.StatusCreated, o) } // validIdempotencyKey: bounded length and visible ASCII only. The key is stored // and may be logged, so we do not accept control characters or unbounded text. func validIdempotencyKey(k string) bool { if len(k) == 0 || len(k) > 255 { return false } for i := 0; i < len(k); i++ { if k[i] < 0x21 || k[i] > 0x7e { return false } } return true } // fingerprintOrder identifies the MEANING of a request. It hashes the validated, // normalised fields, not the raw bytes, so harmless differences (whitespace, key // order, " book " vs "book") still count as the same request. The owner is not // part of it: keys are already scoped per user in the database. func fingerprintOrder(in orderInput) []byte { b, _ := json.Marshal(struct { Op string `json:"op"` Item string `json:"item"` Quantity int `json:"quantity"` }{"POST /orders", in.Item, in.Quantity}) sum := sha256.Sum256(b) return sum[:] } ``` **Validating the key.** It is stored, may be logged, and is attacker-controlled, so it is bounded (1-255) and restricted to visible ASCII. No control characters, no unbounded text. **The fingerprint is the *meaning* of the request.** We hash the validated, normalised fields (`op`, `item`, `quantity`), not the raw body bytes. So: ```plaintext {"item":"book","quantity":2} { "quantity": 2, "item": " book " } ``` are the *same* request (same key → replay), while changing the quantity under the same key is a different request (→ `422`). Hashing raw bytes would make harmless re-serialization (a different JSON library on the retry path, reordered fields) look like misuse. The user is deliberately *not* part of the hash, because the table's primary key already scopes by user. **Why** `422` **for key reuse?** The IETF draft uses it: the key is syntactically fine and the server understood the request; the problem is that this request is not the one the key was created for. **Failed attempts are not remembered.** If the first try fails (database error, deleted account) the transaction rolls back and no key is stored, so the client's retry with the same key simply attempts the operation again. Only *successful* creations are replayed. That is both safe (a failure didn't commit anything) and what clients expect. ## Step 5: expiry, cleanup and metrics Keys only need to outlive the longest realistic client retry window. They expire after `IDEMPOTENCY_TTL` (default 24 h): `internal/storage/idempotency.go (excerpt)` ```go // PurgeIdempotencyKeys deletes keys older than ttl and returns how many. Run it // periodically; keys only need to outlive the longest realistic client retry window. func PurgeIdempotencyKeys(ctx context.Context, pool *pgxpool.Pool, ttl time.Duration) (int64, error) { tag, err := pool.Exec(ctx, "DELETE FROM idempotency_keys WHERE created_at < now() - make_interval(secs => $1)", ttl.Seconds()) if err != nil { return 0, fmt.Errorf("purge idempotency keys: %w", err) } return tag.RowsAffected(), nil } ``` A goroutine in `main` runs it every 10 minutes. We also count replays (`idempotent_replays_total`): a steady trickle is healthy (clients do retry), but a spike points at an upstream network problem or a client retry bug. ## Tests: prove the guarantees Features like this are only worth having if the guarantees are *tested*, especially the ugly ones. **Exactly once under concurrency.** Twelve goroutines are released simultaneously with one key: `internal/storage/idempotency_test.go (excerpt)` ```go // The core promise: N simultaneous requests with one key create exactly one order. func TestIdempotentCreateIsExactlyOnceUnderConcurrency(t *testing.T) { ctx := context.Background() pool := testdb.New(t) u := mkUser(t, storage.NewUserStore(pool), "ann") orders := storage.NewOrderStore(pool) const n = 12 var ( wg sync.WaitGroup mu sync.Mutex ids = map[int]int{} originals int errs []error ) start := make(chan struct{}) for i := 0; i < n; i++ { wg.Add(1) go func() { defer wg.Done() <-start // release everyone at once o, replayed, err := orders.CreateIdempotent(ctx, u.ID, "race-key", fp1, "book", 1) mu.Lock() defer mu.Unlock() if err != nil { errs = append(errs, err) return } ids[o.ID]++ if !replayed { originals++ } }() } close(start) wg.Wait() if len(errs) > 0 { t.Fatalf("errors: %v", errs) } if len(ids) != 1 || originals != 1 { t.Fatalf("want 1 distinct order and 1 original, got ids=%v originals=%d", ids, originals) } if got := countRows(t, ctx, pool, "orders"); got != 1 { t.Fatalf("%d orders in the database, want 1", got) } } ``` The assertion is the whole point: one distinct order id, one non-replayed response, one row in the database. **Atomicity.** Simulate a crash *after* the business write but *before* the response is recorded, the exact window that breaks the naive design: `internal/storage/idempotency_test.go (excerpt)` ```go // The business write and the key commit together or not at all. Simulate a crash // AFTER the write but BEFORE the response is recorded (the window that makes // separate-record designs create duplicates). func TestBusinessWriteAndKeyAreAtomic(t *testing.T) { ctx := context.Background() pool := testdb.New(t) u := mkUser(t, storage.NewUserStore(pool), "ann") boom := errors.New("crash after the write") _, err := storage.RunIdempotent(ctx, pool, u.ID, "key-1", fp1, time.Hour, func(tx pgx.Tx) (int, []byte, error) { if _, err := tx.Exec(ctx, "INSERT INTO orders (user_id, item, quantity) VALUES ($1, 'x', 1)", u.ID); err != nil { return 0, nil, err } return 0, nil, boom }) if !errors.Is(err, boom) { t.Fatalf("got %v", err) } if o, k := countRows(t, ctx, pool, "orders"), countRows(t, ctx, pool, "idempotency_keys"); o != 0 || k != 0 { t.Fatalf("partial state survived: %d orders, %d keys", o, k) } // ...and the retry then succeeds normally. res, err := storage.RunIdempotent(ctx, pool, u.ID, "key-1", fp1, time.Hour, func(tx pgx.Tx) (int, []byte, error) { return 201, []byte(`{}`), nil }) if err != nil || res.Replayed { t.Fatalf("retry after crash: %+v %v", res, err) } } ``` Neither the order nor the key survives, and the retry then succeeds normally. The HTTP layer has its own end-to-end versions: retry with different whitespace gets the same body; reuse with another payload gets `422` and creates nothing; two users with the same key don't interact; eight concurrent identical requests produce one order. We ran these three times in a row under the race detector to make sure none are flaky. ### A bug the tests caught The first run of the "failed attempts are retryable" test failed with a raw database error. We claim the key *before* inserting the order, and the key row has a foreign key to `users`. If the user was deleted after their token was issued, the **claim** fails the foreign key before the order insert gets a chance to produce its friendly `ErrUserNotFound` (which the handler turns into a `401`). The result would have been a `500` for a case that already had a clean answer. The fix maps PostgreSQL error `23503` (foreign-key violation) on the claim to the same `ErrUserNotFound`: `internal/storage/users.go (excerpt)` ```go // isForeignKeyViolation reports Postgres error 23503. func isForeignKeyViolation(err error) bool { var pg *pgconn.PgError return errors.As(err, &pg) && pg.Code == "23503" } ``` A second, quieter lesson repeated from Part 10: the new config default (`IDEMPOTENCY_TTL`) initially didn't take effect because of a mistyped edit, and only a test that asserts the default caught it. **Every config field deserves a test that reads its default back.** ## Complete code for this part Everything to type, file by file. The new migration and `idempotency.go` hold the idea; the rest is wiring. Then `go build ./...` (no new dependencies). *Already given earlier in this post:* `internal/storage/migrations/00005_idempotency_keys.sql`*.* **Change** `internal/config/config.go` ```diff --- a/internal/config/config.go +++ b/internal/config/config.go @@ -36,6 +36,10 @@ type Config struct { // WriteTimeout so we can still send a reply. RequestTimeout time.Duration + // How long an Idempotency-Key is remembered. Must outlive the longest + // realistic client retry window (retries with backoff, queued mobile requests). + IdempotencyTTL time.Duration + MaxBodyBytes int64 MaxNameLen int DefaultPageSize int @@ -95,6 +99,7 @@ func Load() (Config, error) { IdleTimeout: 60 * time.Second, ShutdownTimeout: 15 * time.Second, RequestTimeout: 10 * time.Second, + IdempotencyTTL: 24 * time.Hour, MaxBodyBytes: 1 << 20, MaxNameLen: 100, DefaultPageSize: 20, @@ -140,6 +145,7 @@ func Load() (Config, error) { {"REFRESH_TOKEN_TTL", &c.RefreshTokenTTL}, {"LOGIN_FAILURE_REFILL", &c.LoginFailureRefill}, {"HSTS_MAX_AGE", &c.HSTSMaxAge}, + {"IDEMPOTENCY_TTL", &c.IdempotencyTTL}, } for _, d := range durations { if v := os.Getenv(d.key); v != "" { @@ -227,6 +233,8 @@ func Load() (Config, error) { return c, fmt.Errorf("TLS_MIN_VERSION must be 1.2 or 1.3") case c.HSTSMaxAge < 0: return c, fmt.Errorf("HSTS_MAX_AGE must be >= 0") + case c.IdempotencyTTL <= 0: + return c, fmt.Errorf("IDEMPOTENCY_TTL must be positive") case c.BcryptCost < 4 || c.BcryptCost > 16: return c, fmt.Errorf("BCRYPT_COST must be within [4,16]") case c.AccessTokenTTL <= 0 || c.RefreshTokenTTL <= c.AccessTokenTTL: ``` **Change** `internal/handlers/orders.go` ```diff --- a/internal/handlers/orders.go +++ b/internal/handlers/orders.go @@ -2,6 +2,8 @@ package handlers import ( "context" + "crypto/sha256" + "encoding/json" "errors" "fmt" "log/slog" @@ -20,12 +22,20 @@ type OrderStore interface { List(ctx context.Context, userID, afterID, limit int) ([]*storage.Order, bool, error) // userID 0 = all Get(ctx context.Context, id, ownerID int) (*storage.Order, error) // ownerID 0 = any Create(ctx context.Context, userID int, item string, quantity int) (*storage.Order, error) + CreateIdempotent(ctx context.Context, userID int, key string, fingerprint []byte, item string, quantity int) (*storage.Order, bool, error) Delete(ctx context.Context, id, ownerID int) error } type OrderHandler struct { base - store OrderStore + store OrderStore + onReplay func() // optional: called when a request is answered from a stored result +} + +// WithReplayHook registers a callback for idempotent replays (used for a metric). +func (h *OrderHandler) WithReplayHook(f func()) *OrderHandler { + h.onReplay = f + return h } func NewOrderHandler(store OrderStore, log *slog.Logger, limits Limits) *OrderHandler { @@ -108,19 +118,76 @@ func (h *OrderHandler) Create(w http.ResponseWriter, r *http.Request) { } p, _ := auth.FromContext(r.Context()) - o, err := h.store.Create(r.Context(), p.UserID, in.Item, in.Quantity) - if errors.Is(err, storage.ErrUserNotFound) { + + var ( + o *storage.Order + replayed bool + err error + ) + if key := r.Header.Get("Idempotency-Key"); key != "" { + if !validIdempotencyKey(key) { + h.writeError(w, r, http.StatusBadRequest, + "Idempotency-Key must be 1-255 visible ASCII characters (a UUID is ideal)") + return + } + o, replayed, err = h.store.CreateIdempotent(r.Context(), p.UserID, key, fingerprintOrder(in), in.Item, in.Quantity) + } else { + o, err = h.store.Create(r.Context(), p.UserID, in.Item, in.Quantity) + } + + switch { + case errors.Is(err, storage.ErrIdempotencyKeyReuse): + // 422 per the IETF Idempotency-Key draft: the key is fine, the request + // that came with it is not the one it was first used for. + h.writeError(w, r, http.StatusUnprocessableEntity, + "Idempotency-Key was already used with a different request; send a new key for a new request") + return + case errors.Is(err, storage.ErrUserNotFound): // The token is valid but its user was deleted since: treat as unauthenticated. h.writeError(w, r, http.StatusUnauthorized, "account no longer exists") return - } - if err != nil { + case err != nil: h.internalError(w, r, "create order", err) return } + if replayed { + w.Header().Set("Idempotent-Replayed", "true") // tells the client "this is the original result" + h.log.InfoContext(r.Context(), "idempotent replay", "order_id", o.ID) + if h.onReplay != nil { + h.onReplay() + } + } h.writeJSON(w, r, http.StatusCreated, o) } +// validIdempotencyKey: bounded length and visible ASCII only. The key is stored +// and may be logged, so we do not accept control characters or unbounded text. +func validIdempotencyKey(k string) bool { + if len(k) == 0 || len(k) > 255 { + return false + } + for i := 0; i < len(k); i++ { + if k[i] < 0x21 || k[i] > 0x7e { + return false + } + } + return true +} + +// fingerprintOrder identifies the MEANING of a request. It hashes the validated, +// normalised fields, not the raw bytes, so harmless differences (whitespace, key +// order, " book " vs "book") still count as the same request. The owner is not +// part of it: keys are already scoped per user in the database. +func fingerprintOrder(in orderInput) []byte { + b, _ := json.Marshal(struct { + Op string `json:"op"` + Item string `json:"item"` + Quantity int `json:"quantity"` + }{"POST /orders", in.Item, in.Quantity}) + sum := sha256.Sum256(b) + return sum[:] +} + func (h *OrderHandler) Delete(w http.ResponseWriter, r *http.Request) { id, ok := h.parseID(w, r) if !ok { ``` **Change** `internal/observability/metrics.go` ```diff --- a/internal/observability/metrics.go +++ b/internal/observability/metrics.go @@ -20,6 +20,7 @@ type Metrics struct { duration *prometheus.HistogramVec inFlight prometheus.Gauge rateLimited *prometheus.CounterVec + replays prometheus.Counter } func NewMetrics(pool *pgxpool.Pool) *Metrics { @@ -45,8 +46,12 @@ func NewMetrics(pool *pgxpool.Pool) *Metrics { Help: "Requests rejected with 429, by limiter name.", }, []string{"limiter"}), } + m.replays = prometheus.NewCounter(prometheus.CounterOpts{ + Name: "idempotent_replays_total", + Help: "Requests answered from a stored result because their Idempotency-Key was seen before.", + }) m.Registry.MustRegister( - m.requests, m.duration, m.inFlight, m.rateLimited, + m.requests, m.duration, m.inFlight, m.rateLimited, m.replays, collectors.NewGoCollector(), // goroutines, GC, heap collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}), // fds, cpu, rss ) @@ -56,6 +61,11 @@ func NewMetrics(pool *pgxpool.Pool) *Metrics { return m } +// IdempotentReplay is the callback that counts one replayed request. A steady +// rate is normal (clients retry); a sudden spike usually means a network problem +// upstream or a client bug. +func (m *Metrics) IdempotentReplay() { m.replays.Inc() } + // RateLimited returns a callback that counts a rejection for the named limiter. // The limiter name is a small fixed set ("ip", "user", ...): safe as a label. func (m *Metrics) RateLimited(limiter string) func() { ``` **Create** `internal/storage/idempotency.go` ```go package storage import ( "bytes" "context" "errors" "fmt" "time" "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgxpool" ) // ErrIdempotencyKeyReuse: the same Idempotency-Key was sent with a different // request. That is a client bug (or an attack), never a retry, so we refuse // instead of guessing which request the client meant. var ErrIdempotencyKeyReuse = errors.New("idempotency key reused with a different request") // IdempotentResponse is the stored outcome of an operation. type IdempotentResponse struct { Status int Body []byte Replayed bool // true when this came from an earlier request with the same key } // RunIdempotent executes fn at most once per (userID, key), and returns the // original result to every later caller. // // THE DESIGN: claim the key first, inside the same database transaction as the // business write. // // 1. INSERT the key row. ON CONFLICT DO NOTHING tells us whether we won. // 2. Won: run fn (which writes using the SAME tx), store its response on the // key row, COMMIT. The business write and the key become visible together. // 3. Lost: someone already used this key. Compare fingerprints and replay. // // Why this is better than a middleware that records responses separately: // - Crash safety. If the process dies after the business write but before the // response is recorded, a separate-record design leaves a window where a // retry would run the write AGAIN. Here the tx rolls back as a unit. // - Concurrency for free. Two simultaneous requests with one key: the second // INSERT blocks on the first's uncommitted row (a Postgres guarantee). When // the first commits, the second sees a conflict and replays; if the first // rolls back, the second wins the claim. No "in progress" state, no lock // timeouts, no stale locks to clean up. // // Errors from fn roll everything back, including the key, so failed attempts // are naturally retryable and never replayed. func RunIdempotent( ctx context.Context, pool *pgxpool.Pool, userID int, key string, fingerprint []byte, ttl time.Duration, fn func(tx pgx.Tx) (status int, body []byte, err error), ) (IdempotentResponse, error) { tx, err := pool.Begin(ctx) if err != nil { return IdempotentResponse{}, fmt.Errorf("begin: %w", err) } defer tx.Rollback(ctx) // no-op after Commit // An expired key counts as unused. (The purge job removes old rows in bulk; // this makes expiry exact for the one key we are looking at.) if _, err := tx.Exec(ctx, "DELETE FROM idempotency_keys WHERE user_id = $1 AND key = $2 AND created_at < now() - make_interval(secs => $3)", userID, key, ttl.Seconds()); err != nil { return IdempotentResponse{}, fmt.Errorf("expire key: %w", err) } tag, err := tx.Exec(ctx, "INSERT INTO idempotency_keys (user_id, key, fingerprint) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING", userID, key, fingerprint) if isForeignKeyViolation(err) { // The key row references users(id): a deleted user cannot own a key. Same // meaning as the business insert failing for a missing user. return IdempotentResponse{}, ErrUserNotFound } if err != nil { return IdempotentResponse{}, fmt.Errorf("claim key: %w", err) } if tag.RowsAffected() == 0 { // lost the claim: replay var ( stored []byte status *int body []byte ) err := tx.QueryRow(ctx, "SELECT fingerprint, response_status, response_body FROM idempotency_keys WHERE user_id = $1 AND key = $2", userID, key).Scan(&stored, &status, &body) if err != nil { return IdempotentResponse{}, fmt.Errorf("load key: %w", err) } if !bytes.Equal(stored, fingerprint) { return IdempotentResponse{}, ErrIdempotencyKeyReuse } if status == nil { // cannot happen: rows are committed together with their response return IdempotentResponse{}, errors.New("idempotency key has no stored response") } return IdempotentResponse{Status: *status, Body: body, Replayed: true}, nil } status, body, err := fn(tx) if err != nil { return IdempotentResponse{}, err // deferred Rollback drops the key too } if _, err := tx.Exec(ctx, "UPDATE idempotency_keys SET response_status = $3, response_body = $4 WHERE user_id = $1 AND key = $2", userID, key, status, body); err != nil { return IdempotentResponse{}, fmt.Errorf("store response: %w", err) } if err := tx.Commit(ctx); err != nil { return IdempotentResponse{}, fmt.Errorf("commit: %w", err) } return IdempotentResponse{Status: status, Body: body}, nil } // PurgeIdempotencyKeys deletes keys older than ttl and returns how many. Run it // periodically; keys only need to outlive the longest realistic client retry window. func PurgeIdempotencyKeys(ctx context.Context, pool *pgxpool.Pool, ttl time.Duration) (int64, error) { tag, err := pool.Exec(ctx, "DELETE FROM idempotency_keys WHERE created_at < now() - make_interval(secs => $1)", ttl.Seconds()) if err != nil { return 0, fmt.Errorf("purge idempotency keys: %w", err) } return tag.RowsAffected(), nil } ``` **Change** `internal/storage/orders.go` ```diff --- a/internal/storage/orders.go +++ b/internal/storage/orders.go @@ -2,6 +2,7 @@ package storage import ( "context" + "encoding/json" "errors" "fmt" "time" @@ -23,11 +24,18 @@ type Order struct { } type OrderStore struct { - pool *pgxpool.Pool + pool *pgxpool.Pool + idempotencyTTL time.Duration } func NewOrderStore(pool *pgxpool.Pool) *OrderStore { - return &OrderStore{pool: pool} + return &OrderStore{pool: pool, idempotencyTTL: 24 * time.Hour} +} + +// WithIdempotencyTTL sets how long an Idempotency-Key is remembered. +func (s *OrderStore) WithIdempotencyTTL(d time.Duration) *OrderStore { + s.idempotencyTTL = d + return s } const orderCols = "id, user_id, item, quantity, created_at" @@ -99,12 +107,26 @@ func (s *OrderStore) Get(ctx context.Context, id, ownerID int) (*Order, error) { return o, nil } +// statusCreated is stored with the response; the storage layer deliberately +// does not import net/http, so the number is spelled out. +const statusCreated = 201 + +// queryRower is satisfied by both *pgxpool.Pool and pgx.Tx, so the same SQL can +// run on its own or inside a larger transaction. +type queryRower interface { + QueryRow(ctx context.Context, sql string, args ...any) pgx.Row +} + // Create inserts only if the user exists, in ONE atomic statement. A separate // "SELECT user, then INSERT" would race with a concurrent user delete. Zero // rows inserted surfaces as pgx.ErrNoRows from RETURNING. The FOREIGN KEY // remains as a backstop. func (s *OrderStore) Create(ctx context.Context, userID int, item string, quantity int) (*Order, error) { - o, err := scanOrder(s.pool.QueryRow(ctx, ` + return insertOrder(ctx, s.pool, userID, item, quantity) +} + +func insertOrder(ctx context.Context, q queryRower, userID int, item string, quantity int) (*Order, error) { + o, err := scanOrder(q.QueryRow(ctx, ` INSERT INTO orders (user_id, item, quantity) SELECT $1::bigint, $2::text, $3::int WHERE EXISTS (SELECT 1 FROM users WHERE id = $1) @@ -118,6 +140,30 @@ func (s *OrderStore) Create(ctx context.Context, userID int, item string, quanti return o, nil } +// CreateIdempotent is Create made safe to retry. The same (user, key) with the +// same fingerprint creates the order once and returns that same order to every +// caller; replayed reports whether this call was such a retry. +func (s *OrderStore) CreateIdempotent(ctx context.Context, userID int, key string, fingerprint []byte, + item string, quantity int) (o *Order, replayed bool, err error) { + res, err := RunIdempotent(ctx, s.pool, userID, key, fingerprint, s.idempotencyTTL, + func(tx pgx.Tx) (int, []byte, error) { + created, err := insertOrder(ctx, tx, userID, item, quantity) // same tx as the key + if err != nil { + return 0, nil, err + } + body, err := json.Marshal(created) + return statusCreated, body, err + }) + if err != nil { + return nil, false, err + } + var out Order + if err := json.Unmarshal(res.Body, &out); err != nil { + return nil, false, fmt.Errorf("decode stored response: %w", err) + } + return &out, res.Replayed, nil +} + // Delete removes an order; ownerID > 0 restricts it to that user's orders (see Get). func (s *OrderStore) Delete(ctx context.Context, id, ownerID int) error { tag, err := s.pool.Exec(ctx, ``` **Change** `internal/storage/users.go` ```diff --- a/internal/storage/users.go +++ b/internal/storage/users.go @@ -57,6 +57,12 @@ func isUniqueViolation(err error) bool { return errors.As(err, &pg) && pg.Code == "23505" } +// isForeignKeyViolation reports Postgres error 23503. +func isForeignKeyViolation(err error) bool { + var pg *pgconn.PgError + return errors.As(err, &pg) && pg.Code == "23503" +} + // Every method takes a context and passes it to the pool. When the client // disconnects or the request deadline passes, the in-flight query is // cancelled server-side and the connection returns to the pool. ``` **Change** `cmd/rest/main.go` ```diff --- a/cmd/rest/main.go +++ b/cmd/rest/main.go @@ -255,11 +255,30 @@ func serve() error { authHandler.WithLoginGuard(failures, metrics.RateLimited("login_account")) } + // Housekeeping: expired idempotency keys are deleted in the background. + go func() { + t := time.NewTicker(10 * time.Minute) + defer t.Stop() + for { + select { + case <-t.C: + if n, err := storage.PurgeIdempotencyKeys(ctx, pool, cfg.IdempotencyTTL); err != nil { + log.Error("purge idempotency keys", "err", err) + } else if n > 0 { + log.Info("purged idempotency keys", "count", n) + } + case <-ctx.Done(): + return + } + } + }() + mux := http.NewServeMux() routes.SetupRoutes(mux, routes.Deps{ - Auth: authHandler, - Users: handlers.NewUserHandler(userStore, log, limits), - Orders: handlers.NewOrderHandler(storage.NewOrderStore(pool), log, limits), + Auth: authHandler, + Users: handlers.NewUserHandler(userStore, log, limits), + Orders: handlers.NewOrderHandler(storage.NewOrderStore(pool).WithIdempotencyTTL(cfg.IdempotencyTTL), log, limits). + WithReplayHook(metrics.IdempotentReplay), Authenticate: middleware.Authenticate(tm, log), UserLimit: userLimit, AuthLimit: authLimit, ``` ## Try it ```bash $ T=... # an access token from /auth/login $ curl --cacert certs/ca.pem -i -XPOST https://localhost:7800/orders \ -H "Authorization: Bearer $T" -H 'Idempotency-Key: 7c9e6679-7425-40de-944b-e07fc1f90ae7' \ -d '{"item":"book","quantity":2}' HTTP/2 201 {"id":4,"user_id":3,"item":"book","quantity":2,"created_at":"2026-10-18T08:07:19.633586Z"} ``` The retry, same body: ```plaintext HTTP/2 201 idempotent-replayed: true {"id":4,"user_id":3,"item":"book","quantity":2,"created_at":"2026-10-18T08:07:19.633586Z"} ``` Same id, same `created_at`: the original order, not a new one. Now misuse: ```plaintext # same key, different request HTTP/2 422 {"error":"Idempotency-Key was already used with a different request; send a new key for a new request"} # malformed key HTTP/2 400 {"error":"Idempotency-Key must be 1-255 visible ASCII characters (a UUID is ideal)"} ``` And the double-click: ten simultaneous requests with one fresh key. ```bash $ for i in $(seq 10); do curl ... -H 'Idempotency-Key: double-click-1' -d '{"item":"mug","quantity":1}' & done 201 201 201 201 201 201 201 201 201 201 $ curl ... /orders # how many mugs? 2 orders: [(4, 'book'), (5, 'mug')] ← one mug, not ten ``` Ten `201`s, one order. Prometheus agrees: `idempotent_replays_total = 10` (one sequential retry plus nine concurrent duplicates). ## What a client should do An idempotency key is a *contract with the client*; the server cannot enforce good behavior. The rules to document: 1. Generate a **new random key per logical operation** (a UUID), for example when the user taps "Buy", and keep it for every retry of *that* operation. 2. Retry on network errors, timeouts, `5xx` and `429` (honoring `Retry-After`) with the **same key and the same body**. 3. Never reuse a key for a different operation. The `422` is there to tell you that you did. 4. Do not derive the key from the payload alone: two genuinely separate identical purchases are two operations. ## Limits and pitfalls - **Side effects outside the database** (charging a card, sending an email, calling a third party) are *not* covered by a database transaction. Either pass the same idempotency key through to the downstream provider (Stripe and others accept one), or use the **transactional outbox** pattern: write "send this email" as a row in the same transaction and let a worker perform it exactly once. - **Replays return the original creation response**, even if the order has since been changed or deleted. That is deliberate and matches most APIs. - **Only this endpoint is protected.** `/auth/register` and others are not; `RunIdempotent` is generic so extending it is cheap. - **Waiting duplicates hold a connection.** A concurrent duplicate waits for the first transaction; that wait is bounded by the request timeout and `statement_timeout`, but under extreme duplicate floods you'd rather reject quickly (`409`) after a short `lock_timeout`. - **Storage growth**: one row per key for 24 h. Size the TTL deliberately. ## Recap - Networks can't tell a lost request from a lost response, so clients must be able to retry `POST` safely. - An idempotency key plus a stored response makes retries return the original result. - **Claim the key in the same transaction as the write.** It gives crash-safety, concurrency control and no stale locks without writing locking code. - Scope keys per user; fingerprint the *meaning* of the request; store only successes. - Test the ugly guarantees: concurrency and the crash window. ## Exercises 1. Protect `POST /auth/register` with the same helper. What is the right key scope there, given that the caller isn't authenticated yet? 2. Make the key **mandatory** behind an env flag (`IDEMPOTENCY_REQUIRED`). What status code should a missing key return? (The IETF draft has an opinion.) 3. Add a `lock_timeout` to the claim and return `409 Conflict` with `Retry-After` when a concurrent duplicate would wait longer than 2 seconds. 4. Break it on purpose: comment out the `ON CONFLICT DO NOTHING` and observe which tests fail first.