Part 10: Rate limiting, and the spoofing trap that breaks most implementations
One client in a tight loop can starve everyone else. A token-bucket limiter protects fairness, the database, and your bill, if you identify clients correctly.
Series: Build a production-grade REST API in Go · Part 10 of 17 Where we are: the paginated API from Part 9.
Pagination protected us from one request being too big. Rate limiting protects us from too many requests: a buggy client stuck in a retry loop, a scraper, a script kiddie. Without limits, a single abusive client can use up all database connections and CPU, and every legitimate user suffers.
Rate limiting answers: how many requests may one client make per unit of time? Two parts make that hard:
- the algorithm (how to count), and
- the identity (who is "one client"?).
The second part is where almost every first implementation is broken.
The algorithm: token bucket
Picture a bucket that holds up to burst tokens and is refilled at rate tokens per second. Each request takes one token. No token? Reject (429 Too Many Requests) and tell the client when to retry.
- Burst lets a client do a quick flurry (a page load firing ten API calls).
- Rate caps the sustained speed.
Other algorithms exist: fixed window ("100 per minute") is simple but allows 2x bursts at window boundaries; sliding window is smoother but costs more memory; leaky bucket smooths output into a steady stream. Token bucket is the common default because it's cheap and bursts are usually desirable. We use golang.org/x/time/rate, a well-tested token bucket, and build the per-client bookkeeping around it.
go get golang.org/x/time
Step 1: a limiter per client
internal/ratelimit/ratelimit.go (excerpt)
type entry struct {
lim *rate.Limiter
lastSeen time.Time
}
type Limiter struct {
rate rate.Limit
burst int
ttl time.Duration // forget keys idle for longer than this
max int // hard cap on tracked keys (memory-exhaustion guard)
now func() time.Time
mu sync.Mutex
entries map[string]*entry
}
// New creates a limiter allowing rps sustained requests/second per key with the
// given burst. Keys idle for ttl are evicted; at most maxKeys are tracked.
func New(rps float64, burst int, ttl time.Duration, maxKeys int) *Limiter {
return &Limiter{
rate: rate.Limit(rps), burst: burst, ttl: ttl, max: maxKeys,
now: time.Now, entries: make(map[string]*entry),
}
}
// Allow spends a token for key. If none is available it returns false and how
// long the caller should wait before retrying.
func (l *Limiter) Allow(key string) (ok bool, retryAfter time.Duration) {
now := l.now()
l.mu.Lock()
e, found := l.entries[key]
if !found {
if len(l.entries) >= l.max {
l.cleanupLocked(now)
}
if len(l.entries) >= l.max {
// Still full of ACTIVE keys (e.g. an attack with many source
// addresses). Failing closed would lock out legitimate users and turn
// the limiter into the outage; fail open and rely on other layers.
l.mu.Unlock()
return true, 0
}
e = &entry{lim: rate.NewLimiter(l.rate, l.burst)}
l.entries[key] = e
}
e.lastSeen = now
l.mu.Unlock() // rate.Limiter is itself goroutine-safe; don't hold our lock while using it
// Reserve + Cancel (instead of Allow) lets us learn the wait time without
// consuming a token when the request is rejected.
r := e.lim.ReserveN(now, 1)
if !r.OK() {
return false, time.Second
}
if d := r.DelayFrom(now); d > 0 {
r.CancelAt(now)
return false, d
}
return true, 0
}
Read Allow carefully:
- One bucket per key (the key will be the client's IP for now). A mutex-guarded map holds them.
Reserve+Cancelinstead of the simplerAllow: we want the client to learn how long to wait (Retry-After), and we must not use up a token for a request we're rejecting. Reserve, look at the delay, and cancel if it's non-zero.- Memory is bounded. An attacker who sends requests from a million different addresses would otherwise create a million map entries. Two defenses: a janitor evicts keys idle for
ttl(Run), and a hard capmax. - Fail open at the cap. If the map is full of active keys, we allow the request instead of rejecting. That sounds backwards, so think it through: failing closed would lock out legitimate users during exactly the kind of attack that fills the table, turning our protection into the outage. Volumetric attacks need defenses elsewhere (a CDN or load balancer).
internal/ratelimit/ratelimit.go (excerpt)
// Run evicts idle keys until done is closed. Without it the map would grow
// forever: one entry per distinct client ever seen.
func (l *Limiter) Run(done <-chan struct{}) {
t := time.NewTicker(l.ttl / 2)
defer t.Stop()
for {
select {
case <-t.C:
l.Cleanup()
case <-done:
return
}
}
}
// RetryAfterSeconds rounds a wait UP to whole seconds for the Retry-After header
// (it only carries integers; rounding down would invite an immediate retry).
func RetryAfterSeconds(d time.Duration) int {
return max(1, int(math.Ceil(d.Seconds())))
}
(Retry-After carries whole seconds; we round up, since rounding down invites an immediate retry that will fail again.)
Step 2: who is a "client"? The X-Forwarded-For trap
The obvious key is the connection's IP address (r.RemoteAddr). But in production your server sits behind a load balancer or reverse proxy, so RemoteAddr is always the proxy's address. Everyone would share a single bucket!
Proxies solve this by adding the original client address to an X-Forwarded-For header. So we should read that header... and now we've walked into the trap:
X-Forwarded-Foris just an HTTP header. Any client can send it. If we trust it blindly, an attacker sendsX-Forwarded-For: 1.2.3.4, then1.2.3.5, then1.2.3.6... and gets a fresh bucket every single request. The rate limit is bypassed completely.
The correct rule has two parts:
- Only believe
X-Forwarded-Forwhen the direct peer is a proxy you control (TRUSTED_PROXIES, a list of CIDR ranges). - Even then, take the right-most address that is not itself a trusted proxy: that's the one our own infrastructure appended. Anything to its left was supplied by the client and can be forged.
internal/ratelimit/ratelimit.go (excerpt)
// ClientIP returns the caller's address for use as a rate-limit key.
//
// SECURITY: X-Forwarded-For is just a header; any client can send
// "X-Forwarded-For: 1.2.3.4" and, if we believed it, pick a fresh identity per
// request and bypass the limit entirely. We therefore honour it ONLY when the
// direct peer is one of our trusted proxies (load balancer), and then take the
// right-most address that is not itself a trusted proxy: the one our own
// infrastructure appended, which the client cannot forge.
//
// IPv6: a single user commonly controls a whole /64, so we key on the /64 prefix.
// Otherwise one client could mint billions of "different" addresses.
func ClientIP(r *http.Request, trusted []netip.Prefix) string {
peer := parseAddr(r.RemoteAddr)
ip := peer
if isTrusted(peer, trusted) {
parts := strings.Split(strings.Join(r.Header.Values("X-Forwarded-For"), ","), ",")
for i := len(parts) - 1; i >= 0; i-- {
a, err := netip.ParseAddr(strings.TrimSpace(parts[i]))
if err != nil {
break // garbage: fall back to the peer
}
ip = a
if !isTrusted(a, trusted) {
break
}
}
}
if !ip.IsValid() {
return r.RemoteAddr
}
if ip.Is4In6() {
ip = ip.Unmap()
}
if ip.Is6() {
p, _ := ip.Prefix(64)
return p.Addr().String() + "/64"
}
return ip.String()
}
func isTrusted(a netip.Addr, trusted []netip.Prefix) bool {
for _, p := range trusted {
if p.Contains(a) {
return true
}
}
return false
}
Two more details in there:
- IPv6: a single home subscriber typically controls an entire
/64(2^64 addresses). If we keyed on the full address, one user could mint billions of "different" clients. So IPv6 clients are keyed by their /64 prefix. Is4In6()/Unmap()normalizes IPv4 addresses that arrive in IPv6 notation (::ffff:1.2.3.4), so one client doesn't get two buckets.
The unit tests are a table of exactly these scenarios:
internal/ratelimit/ratelimit_test.go (excerpt)
func TestClientIP(t *testing.T) {
trusted, _ := ParseCIDRs("10.0.0.0/8")
tests := []struct {
name, remote, xff, want string
}{
{"direct client", "203.0.113.5:4000", "", "203.0.113.5"},
{"untrusted peer cannot spoof XFF", "203.0.113.5:4000", "1.2.3.4", "203.0.113.5"},
{"trusted proxy: use forwarded client", "10.0.0.2:80", "198.51.100.7", "198.51.100.7"},
{"client-supplied prefix is ignored (rightmost untrusted wins)", "10.0.0.2:80", "1.1.1.1, 198.51.100.7", "198.51.100.7"},
{"chain of trusted proxies", "10.0.0.2:80", "198.51.100.7, 10.0.0.9", "198.51.100.7"},
{"garbage header falls back to peer", "10.0.0.2:80", "not-an-ip", "10.0.0.2"},
{"ipv6 keyed by /64", "[2001:db8:1:2:aaaa:bbbb:cccc:dddd]:80", "", "2001:db8:1:2::/64"},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
r := httptest.NewRequest("GET", "/", nil)
r.RemoteAddr = tc.remote
if tc.xff != "" {
r.Header.Set("X-Forwarded-For", tc.xff)
}
if got := ClientIP(r, trusted); got != tc.want {
t.Fatalf("got %q want %q", got, tc.want)
}
})
}
}
Misconfiguration cuts both ways: if you forget TRUSTED_PROXIES behind a load balancer, all users share one bucket and get throttled together; if you trust too broadly, spoofing works. Keep it exact.
Step 3: the middleware
We keep the limiter generic: the middleware takes a key function, so the same code will later limit by user id or by account (Part 13).
internal/middleware/middleware.go (excerpt)
// RateLimitOptions configures RateLimit. The key function is what makes the
// middleware reusable: "per IP" today, "per authenticated user" later, "per
// account being logged into" for brute-force protection: same machinery,
// different key.
type RateLimitOptions struct {
Limiter *ratelimit.Limiter
// Key returns the bucket for this request. Return "" to skip limiting.
Key func(r *http.Request) string
// OnReject is called for every rejected request (used to count them).
OnReject func(r *http.Request)
}
// RateLimit answers 429 Too Many Requests with a Retry-After header when the
// caller's bucket is empty, before any expensive work happens.
func RateLimit(o RateLimitOptions) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
key := o.Key(r)
if key == "" {
next.ServeHTTP(w, r)
return
}
ok, wait := o.Limiter.Allow(key)
if !ok {
if o.OnReject != nil {
o.OnReject(r)
}
w.Header().Set("Retry-After", strconv.Itoa(ratelimit.RetryAfterSeconds(wait)))
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusTooManyRequests)
_, _ = w.Write([]byte(`{"error":"rate limit exceeded"}` + "\n"))
return
}
next.ServeHTTP(w, r)
})
}
}
The response is the standard one: status 429, a Retry-After header, and our usual JSON error body, consistent with all other errors.
Step 4: wire it in, with metrics
cmd/rest/main.go (lines 149-163)
// Rate limiting: one bucket per client IP, applied to every API request.
var limiters []func(http.Handler) http.Handler
if cfg.RateLimitRPS > 0 {
trusted, err := ratelimit.ParseCIDRs(cfg.TrustedProxies)
if err != nil {
return fmt.Errorf("TRUSTED_PROXIES: %w", err)
}
ipLimiter := ratelimit.New(cfg.RateLimitRPS, cfg.RateLimitBurst, 10*time.Minute, 100_000)
go ipLimiter.Run(ctx.Done()) // evicts idle clients until shutdown
limiters = append(limiters, middleware.RateLimit(middleware.RateLimitOptions{
Limiter: ipLimiter,
Key: func(r *http.Request) string { return ratelimit.ClientIP(r, trusted) },
OnReject: func(*http.Request) { metrics.RateLimited("ip")() },
}))
}
The limiter sits innermost in the middleware chain, so rejected requests still appear in access logs, traces and the http_requests_total{status="429"} metric (invisible rejections are a debugging nightmare), but never reach a handler or the database. We also count rejections:
internal/observability/metrics.go (excerpt)
// RateLimited returns a callback that counts a rejection for the named limiter.
// The limiter name is a small fixed set ("ip", "user", ...): safe as a label.
func (m *Metrics) RateLimited(limiter string) func() {
c := m.rateLimited.WithLabelValues(limiter)
return c.Inc
}
and the config gained RATE_LIMIT_RPS / RATE_LIMIT_BURST (0 disables it). An alert fires if limiting is sustained, because that means either abuse or a limit that's too low for real traffic.
A configuration lesson worth stealing. This feature has several inputs (
RATE_LIMIT_RPS,RATE_LIMIT_BURST,TRUSTED_PROXIES), and a config field can be broken in two silent ways: the variable is never read into the struct, or the struct has no default. Either one leaves the limiter quietly switched off (or ignoring your proxy list) while everything still starts and passes its tests, and docker compose can hide it by setting the variables explicitly. The fix is cheap: a test that loads the config with no environment and asserts every default.c, _ := Load() if c.RateLimitRPS != 20 || c.RateLimitBurst != 40 { t.Fatalf("defaults not applied: %+v", c) }
Complete code for this part
Everything to type, file by file: the new ratelimit package in full, and small changes elsewhere. Then go mod tidy && go build ./....
Change internal/config/config.go
--- a/internal/config/config.go
+++ b/internal/config/config.go
@@ -41,6 +41,11 @@ type Config struct {
DefaultPageSize int
MaxPageSize int
+ // Rate limiting (per client IP). RateLimitRPS == 0 disables it.
+ RateLimitRPS float64
+ RateLimitBurst int
+ TrustedProxies string // CIDRs whose X-Forwarded-For we believe, e.g. "10.0.0.0/8"
+
ServiceName string
LogLevel slog.Level
OTLPEndpoint string // host:port of an OTLP/gRPC collector; empty disables tracing
@@ -70,6 +75,9 @@ func Load() (Config, error) {
ServiceName: env("SERVICE_NAME", "rest-go"),
OTLPEndpoint: os.Getenv("OTLP_ENDPOINT"),
TraceSampleRatio: 1.0,
+ RateLimitRPS: 20,
+ RateLimitBurst: 40,
+ TrustedProxies: os.Getenv("TRUSTED_PROXIES"),
}
var err error
@@ -127,6 +135,16 @@ func Load() (Config, error) {
}
}
}
+ if v := os.Getenv("RATE_LIMIT_RPS"); v != "" {
+ if c.RateLimitRPS, err = strconv.ParseFloat(v, 64); err != nil {
+ return c, fmt.Errorf("RATE_LIMIT_RPS: %w", err)
+ }
+ }
+ if v := os.Getenv("RATE_LIMIT_BURST"); v != "" {
+ if c.RateLimitBurst, err = strconv.Atoi(v); err != nil {
+ return c, fmt.Errorf("RATE_LIMIT_BURST: %w", err)
+ }
+ }
if v := os.Getenv("TRACE_SAMPLE_RATIO"); v != "" {
if c.TraceSampleRatio, err = strconv.ParseFloat(v, 64); err != nil {
return c, fmt.Errorf("TRACE_SAMPLE_RATIO: %w", err)
@@ -148,6 +166,8 @@ func Load() (Config, error) {
return c, fmt.Errorf("REQUEST_TIMEOUT (%s) must be below WRITE_TIMEOUT (%s)", c.RequestTimeout, c.WriteTimeout)
case c.DefaultPageSize < 1 || c.MaxPageSize < c.DefaultPageSize:
return c, fmt.Errorf("need 1 <= DEFAULT_PAGE_SIZE <= MAX_PAGE_SIZE")
+ case c.RateLimitRPS < 0 || (c.RateLimitRPS > 0 && c.RateLimitBurst < 1):
+ return c, fmt.Errorf("RATE_LIMIT_RPS must be >= 0 and RATE_LIMIT_BURST >= 1 when enabled")
case c.DBMaxConns < 1 || c.DBMinConns < 0 || c.DBMinConns > c.DBMaxConns:
return c, fmt.Errorf("need 0 <= DB_MIN_CONNS <= DB_MAX_CONNS and DB_MAX_CONNS >= 1")
case c.TraceSampleRatio < 0 || c.TraceSampleRatio > 1:
Change internal/middleware/middleware.go
--- a/internal/middleware/middleware.go
+++ b/internal/middleware/middleware.go
@@ -9,6 +9,7 @@ import (
"encoding/hex"
"log/slog"
"net/http"
+ "strconv"
"time"
"github.com/klauspost/compress/gzhttp"
@@ -16,6 +17,7 @@ import (
"go.opentelemetry.io/otel/trace"
"go-rest/internal/observability"
+ "go-rest/internal/ratelimit"
)
// RequestID gives every request a correlation id: it reuses the caller's
@@ -127,6 +129,44 @@ func Recover(log *slog.Logger) func(http.Handler) http.Handler {
}
}
+// RateLimitOptions configures RateLimit. The key function is what makes the
+// middleware reusable: "per IP" today, "per authenticated user" later, "per
+// account being logged into" for brute-force protection: same machinery,
+// different key.
+type RateLimitOptions struct {
+ Limiter *ratelimit.Limiter
+ // Key returns the bucket for this request. Return "" to skip limiting.
+ Key func(r *http.Request) string
+ // OnReject is called for every rejected request (used to count them).
+ OnReject func(r *http.Request)
+}
+
+// RateLimit answers 429 Too Many Requests with a Retry-After header when the
+// caller's bucket is empty, before any expensive work happens.
+func RateLimit(o RateLimitOptions) func(http.Handler) http.Handler {
+ return func(next http.Handler) http.Handler {
+ return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ key := o.Key(r)
+ if key == "" {
+ next.ServeHTTP(w, r)
+ return
+ }
+ ok, wait := o.Limiter.Allow(key)
+ if !ok {
+ if o.OnReject != nil {
+ o.OnReject(r)
+ }
+ w.Header().Set("Retry-After", strconv.Itoa(ratelimit.RetryAfterSeconds(wait)))
+ w.Header().Set("Content-Type", "application/json")
+ w.WriteHeader(http.StatusTooManyRequests)
+ _, _ = w.Write([]byte(`{"error":"rate limit exceeded"}` + "\n"))
+ return
+ }
+ next.ServeHTTP(w, r)
+ })
+ }
+}
+
// Chain applies middlewares so the first listed is the outermost.
func Chain(h http.Handler, mws ...func(http.Handler) http.Handler) http.Handler {
for i := len(mws) - 1; i >= 0; i-- {
Change internal/observability/metrics.go
--- a/internal/observability/metrics.go
+++ b/internal/observability/metrics.go
@@ -16,9 +16,10 @@ import (
type Metrics struct {
Registry *prometheus.Registry
- requests *prometheus.CounterVec
- duration *prometheus.HistogramVec
- inFlight prometheus.Gauge
+ requests *prometheus.CounterVec
+ duration *prometheus.HistogramVec
+ inFlight prometheus.Gauge
+ rateLimited *prometheus.CounterVec
}
func NewMetrics(pool *pgxpool.Pool) *Metrics {
@@ -39,9 +40,13 @@ func NewMetrics(pool *pgxpool.Pool) *Metrics {
Name: "http_requests_in_flight",
Help: "Requests currently being served.",
}),
+ rateLimited: prometheus.NewCounterVec(prometheus.CounterOpts{
+ Name: "rate_limited_requests_total",
+ Help: "Requests rejected with 429, by limiter name.",
+ }, []string{"limiter"}),
}
m.Registry.MustRegister(
- m.requests, m.duration, m.inFlight,
+ m.requests, m.duration, m.inFlight, m.rateLimited,
collectors.NewGoCollector(), // goroutines, GC, heap
collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}), // fds, cpu, rss
)
@@ -51,6 +56,13 @@ func NewMetrics(pool *pgxpool.Pool) *Metrics {
return m
}
+// RateLimited returns a callback that counts a rejection for the named limiter.
+// The limiter name is a small fixed set ("ip", "user", ...): safe as a label.
+func (m *Metrics) RateLimited(limiter string) func() {
+ c := m.rateLimited.WithLabelValues(limiter)
+ return c.Inc
+}
+
// Handler serves the registry in Prometheus text format.
func (m *Metrics) Handler() http.Handler {
return promhttp.HandlerFor(m.Registry, promhttp.HandlerOpts{})
Create internal/ratelimit/ratelimit.go
// Package ratelimit implements an in-memory, per-key token-bucket limiter.
//
// Token bucket: each key owns a bucket holding up to `burst` tokens that refills
// at `rate` tokens/second. A request spends one token; no token means "wait".
// That allows short bursts (a page load firing 10 calls) while capping the
// sustained rate. It is the algorithm behind most API rate limits.
package ratelimit
import (
"math"
"net"
"net/http"
"net/netip"
"strings"
"sync"
"time"
"golang.org/x/time/rate"
)
type entry struct {
lim *rate.Limiter
lastSeen time.Time
}
type Limiter struct {
rate rate.Limit
burst int
ttl time.Duration // forget keys idle for longer than this
max int // hard cap on tracked keys (memory-exhaustion guard)
now func() time.Time
mu sync.Mutex
entries map[string]*entry
}
// New creates a limiter allowing rps sustained requests/second per key with the
// given burst. Keys idle for ttl are evicted; at most maxKeys are tracked.
func New(rps float64, burst int, ttl time.Duration, maxKeys int) *Limiter {
return &Limiter{
rate: rate.Limit(rps), burst: burst, ttl: ttl, max: maxKeys,
now: time.Now, entries: make(map[string]*entry),
}
}
// Limit and Burst expose the configuration (for headers and docs).
func (l *Limiter) Limit() float64 { return float64(l.rate) }
func (l *Limiter) Burst() int { return l.burst }
// Allow spends a token for key. If none is available it returns false and how
// long the caller should wait before retrying.
func (l *Limiter) Allow(key string) (ok bool, retryAfter time.Duration) {
now := l.now()
l.mu.Lock()
e, found := l.entries[key]
if !found {
if len(l.entries) >= l.max {
l.cleanupLocked(now)
}
if len(l.entries) >= l.max {
// Still full of ACTIVE keys (e.g. an attack with many source
// addresses). Failing closed would lock out legitimate users and turn
// the limiter into the outage; fail open and rely on other layers.
l.mu.Unlock()
return true, 0
}
e = &entry{lim: rate.NewLimiter(l.rate, l.burst)}
l.entries[key] = e
}
e.lastSeen = now
l.mu.Unlock() // rate.Limiter is itself goroutine-safe; don't hold our lock while using it
// Reserve + Cancel (instead of Allow) lets us learn the wait time without
// consuming a token when the request is rejected.
r := e.lim.ReserveN(now, 1)
if !r.OK() {
return false, time.Second
}
if d := r.DelayFrom(now); d > 0 {
r.CancelAt(now)
return false, d
}
return true, 0
}
// Cleanup evicts idle keys. Run it periodically (see Run).
func (l *Limiter) Cleanup() {
l.mu.Lock()
l.cleanupLocked(l.now())
l.mu.Unlock()
}
func (l *Limiter) cleanupLocked(now time.Time) {
for k, e := range l.entries {
if now.Sub(e.lastSeen) > l.ttl {
delete(l.entries, k)
}
}
}
// Run evicts idle keys until done is closed. Without it the map would grow
// forever: one entry per distinct client ever seen.
func (l *Limiter) Run(done <-chan struct{}) {
t := time.NewTicker(l.ttl / 2)
defer t.Stop()
for {
select {
case <-t.C:
l.Cleanup()
case <-done:
return
}
}
}
// Len reports tracked keys (for tests and metrics).
func (l *Limiter) Len() int {
l.mu.Lock()
defer l.mu.Unlock()
return len(l.entries)
}
// RetryAfterSeconds rounds a wait UP to whole seconds for the Retry-After header
// (it only carries integers; rounding down would invite an immediate retry).
func RetryAfterSeconds(d time.Duration) int {
return max(1, int(math.Ceil(d.Seconds())))
}
// ---- client identification ------------------------------------------------
// ClientIP returns the caller's address for use as a rate-limit key.
//
// SECURITY: X-Forwarded-For is just a header; any client can send
// "X-Forwarded-For: 1.2.3.4" and, if we believed it, pick a fresh identity per
// request and bypass the limit entirely. We therefore honour it ONLY when the
// direct peer is one of our trusted proxies (load balancer), and then take the
// right-most address that is not itself a trusted proxy: the one our own
// infrastructure appended, which the client cannot forge.
//
// IPv6: a single user commonly controls a whole /64, so we key on the /64 prefix.
// Otherwise one client could mint billions of "different" addresses.
func ClientIP(r *http.Request, trusted []netip.Prefix) string {
peer := parseAddr(r.RemoteAddr)
ip := peer
if isTrusted(peer, trusted) {
parts := strings.Split(strings.Join(r.Header.Values("X-Forwarded-For"), ","), ",")
for i := len(parts) - 1; i >= 0; i-- {
a, err := netip.ParseAddr(strings.TrimSpace(parts[i]))
if err != nil {
break // garbage: fall back to the peer
}
ip = a
if !isTrusted(a, trusted) {
break
}
}
}
if !ip.IsValid() {
return r.RemoteAddr
}
if ip.Is4In6() {
ip = ip.Unmap()
}
if ip.Is6() {
p, _ := ip.Prefix(64)
return p.Addr().String() + "/64"
}
return ip.String()
}
func parseAddr(hostport string) netip.Addr {
host, _, err := net.SplitHostPort(hostport)
if err != nil {
host = hostport
}
a, _ := netip.ParseAddr(host)
return a
}
func isTrusted(a netip.Addr, trusted []netip.Prefix) bool {
for _, p := range trusted {
if p.Contains(a) {
return true
}
}
return false
}
// ParseCIDRs parses a comma-separated list like "10.0.0.0/8,172.16.0.0/12".
func ParseCIDRs(s string) ([]netip.Prefix, error) {
var out []netip.Prefix
for _, part := range strings.Split(s, ",") {
part = strings.TrimSpace(part)
if part == "" {
continue
}
p, err := netip.ParsePrefix(part)
if err != nil {
return nil, err
}
out = append(out, p)
}
return out, nil
}
Change cmd/rest/main.go
--- a/cmd/rest/main.go
+++ b/cmd/rest/main.go
@@ -18,6 +18,7 @@ import (
"go-rest/internal/handlers"
"go-rest/internal/middleware"
"go-rest/internal/observability"
+ "go-rest/internal/ratelimit"
"go-rest/internal/routes"
"go-rest/internal/storage"
)
@@ -145,17 +146,35 @@ func serve() error {
mux := http.NewServeMux()
routes.SetupRoutes(mux, users, orders)
+ // Rate limiting: one bucket per client IP, applied to every API request.
+ var limiters []func(http.Handler) http.Handler
+ if cfg.RateLimitRPS > 0 {
+ trusted, err := ratelimit.ParseCIDRs(cfg.TrustedProxies)
+ if err != nil {
+ return fmt.Errorf("TRUSTED_PROXIES: %w", err)
+ }
+ ipLimiter := ratelimit.New(cfg.RateLimitRPS, cfg.RateLimitBurst, 10*time.Minute, 100_000)
+ go ipLimiter.Run(ctx.Done()) // evicts idle clients until shutdown
+ limiters = append(limiters, middleware.RateLimit(middleware.RateLimitOptions{
+ Limiter: ipLimiter,
+ Key: func(r *http.Request) string { return ratelimit.ClientIP(r, trusted) },
+ OnReject: func(*http.Request) { metrics.RateLimited("ip")() },
+ }))
+ }
+
// First listed = outermost. ORDER MATTERS:
// 1. otelhttp, RequestID, Timeout all COPY the request (r.WithContext), so
// they go first.
// 2. Logging and Metrics read r.Pattern after the mux has routed. The mux
// sets it on the exact request object it receives, so these must pass
// the request through unchanged: they come after every copier.
- // 3. Recover is innermost so a panic becomes a 500 that Logging/Metrics
+ // 3. Recover sits inside Logging/Metrics so a panic becomes a 500 that they
// still record.
+ // 4. The rate limiter is innermost: its 429 replies are logged and counted
+ // like any response, yet rejected requests never reach the mux.
api := &http.Server{
Addr: cfg.Addr,
- Handler: middleware.Chain(mux,
+ Handler: middleware.Chain(mux, append([]func(http.Handler) http.Handler{
func(next http.Handler) http.Handler { return otelhttp.NewHandler(next, "http.server") },
middleware.RequestID,
middleware.Gzip,
@@ -165,7 +184,7 @@ func serve() error {
middleware.Logging(log),
metrics.Middleware,
middleware.Recover(log),
- ),
+ }, limiters...)...),
// The zero-value http.Server has NO timeouts, so slow or malicious
// clients (e.g. Slowloris) can hold connections open forever.
ReadHeaderTimeout: cfg.ReadHeaderTimeout,
Replace the contents of deploy/prometheus/alerts.yml with
# Alert rules are evaluated by Prometheus. Delivering them (email/Slack/pager)
# needs Alertmanager, which is intentionally not part of this learning stack;
# firing alerts are still visible at http://localhost:9091/alerts.
groups:
- name: api
rules:
- alert: ApiDown
expr: up{job="api"} == 0
for: 1m
labels: { severity: page }
annotations: { summary: "API target is down" }
- alert: HighErrorRate # symptom-based: users see 5xx
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ clamp_min(sum(rate(http_requests_total[5m])), 1) > 0.05
for: 5m
labels: { severity: page }
annotations: { summary: "More than 5% of requests fail with 5xx" }
- alert: HighLatencyP95
expr: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) > 1
for: 10m
labels: { severity: ticket }
annotations: { summary: "p95 latency above 1s" }
- alert: DbPoolExhausted # requests are queueing for a connection
expr: rate(db_pool_empty_acquire_total[5m]) > 0
for: 5m
labels: { severity: ticket }
annotations: { summary: "DB pool is exhausted; consider raising DB_MAX_CONNS or fixing slow queries" }
- alert: SustainedRateLimiting # a client (or attack) is hitting the limit continuously
expr: sum(rate(rate_limited_requests_total[5m])) > 1
for: 10m
labels: { severity: ticket }
annotations: { summary: "Requests are being rate limited continuously; check for abuse or a too-low limit" }
Change docker-compose.yml
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -46,6 +46,10 @@ services:
ADMIN_ADDR: ":9090"
OTLP_ENDPOINT: jaeger:4317
LOG_LEVEL: info
+ # Per-IP limit. Generous locally so `make load` isn't throttled; lower it to see 429s.
+ RATE_LIMIT_RPS: ${RATE_LIMIT_RPS:-200}
+ RATE_LIMIT_BURST: ${RATE_LIMIT_BURST:-400}
+ # TRUSTED_PROXIES: 10.0.0.0/8 # set when behind a load balancer, or every client shares its IP
ports:
- "7800:7800" # public API only; admin port 9090 stays internal
depends_on:
Try it
Run with a tiny limit so it's easy to trigger (2 requests/second sustained, bursts of 5):
RATE_LIMIT_RPS=2 RATE_LIMIT_BURST=5 go run ./cmd/rest
$ for i in $(seq 1 12); do curl -s -o /dev/null -w "%{http_code} " localhost:7800/users; done
200 200 200 200 200 429 429 429 429 429 429 429
$ curl -si localhost:7800/users
HTTP/1.1 429 Too Many Requests
Retry-After: 1
{"error":"rate limit exceeded"}
Five requests used the burst, then the bucket was empty. Wait two seconds and a couple of requests pass again as tokens refill.
Now the spoofing attempt. We did not configure TRUSTED_PROXIES, so the header must be ignored:
$ for i in 1 2 3; do curl -s -o /dev/null -w "%{http_code} " -H "X-Forwarded-For: 9.9.9.$i" localhost:7800/users; done
429 429 429
Still limited: the fake addresses bought nothing. And in Prometheus: rate_limited_requests_total{limiter="ip"} counts the rejections.
Known limits of this design (be honest about them)
- State is in memory, per instance. With three replicas, a client can make up to 3x the limit. Fine for protecting each instance; for an exact global limit you need a shared store (Redis) or to limit at the gateway.
- IP is a weak identity. Many people share an office or mobile-carrier address and would throttle each other; an attacker can have many addresses. IP limiting is the outer safety net. In Parts 11-13 we add per-user and per-account limits once we know who is calling.
- Pick limits from data. Start generous, watch
rate_limited_requests_total, tighten gradually.
Recap
- Token bucket: cheap, allows bursts, caps sustained rate.
- Bound the limiter's memory; fail open at the cap rather than lock everyone out.
- Never trust
X-Forwarded-Forfrom untrusted peers; use the right-most untrusted entry; key IPv6 by /64. - Reject with
429+Retry-After; keep rejections visible in logs and metrics. - Make the limiter generic: the key function is the extension point.
Exercises
- Write a test for the middleware using a key function that reads a header, so you can simulate two clients.
- Add
RateLimit-LimitandRateLimit-Remainingresponse headers (look up the IETF draft). What's awkward about "remaining" withReserve? - Behind a real proxy you'd set
TRUSTED_PROXIES. Start twonginxcontainers in a chain and verify the right-most-untrusted rule.
Comments
Join the discussionNo comments yet. Be the first to comment.
Build a production-grade REST API in Go
A hands-on series: start with under 200 lines of in-memory CRUD and end with Postgres, JWT auth, rate limiting, TLS and a full observability stack.