Resiliency Patterns

Networks fail. Dependencies get slow. Downstream services fall over right when you need them. Resiliency patterns are how a service degrades gracefully instead of cascading into an outage. None of them are exotic — they're a small set of techniques you compose, and Go's context makes most of them natural.

We'll build up: timeouts, retries with backoff and jitter, circuit breakers, rate limiting, and bulkheads — and the order they should be layered.

Start with timeouts — the foundation

The first resiliency bug in almost every system is a missing timeout. A call with no deadline can hang forever, and one hung call ties up a goroutine, a connection, and eventually your whole capacity. Every outbound call needs a deadline.

Use a per-call context (post 5), not a global client timeout, so each operation gets a bound appropriate to it:

func (c *Client) GetUser(ctx context.Context, id string) (*User, error) {
    ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
    defer cancel()

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.url(id), nil)
    if err != nil {
        return nil, err
    }
    resp, err := c.http.Do(req) // aborts when the deadline fires
    // ...
}

A subtle but vital point: timeouts must propagate. If your caller already has 1 second left on its context, don't override it with a fresh 5-second timeout — derive from the incoming context so the budget shrinks as it flows down the call tree. context.WithTimeout(ctx, ...) takes the minimum of the two deadlines, which is exactly what you want.

Retries — but carefully

Transient failures (a dropped connection, a brief blip) often succeed on a second try. But naïve retries are dangerous: they amplify load exactly when a dependency is already struggling, and retrying non-idempotent operations can double-charge a customer. The rules:

1. Only retry idempotent operations (or ones made idempotent with a key).
2. Only retry transient errors — timeouts, 503s, connection resets. Never retry a 400 or a validation error; it'll fail every time.
3. Back off exponentially, and add jitter so clients don't retry in lockstep and create a thundering herd.
4. Cap the attempts and respect the context deadline.

func retry(ctx context.Context, attempts int, base time.Duration, op func() error) error {
    var err error
    for i := 0; i < attempts; i++ {
        if err = op(); err == nil {
            return nil
        }
        if !isRetryable(err) {
            return err // permanent failure — don't waste attempts
        }
        // Exponential backoff with full jitter.
        backoff := base * (1 << i)               // base, 2×, 4×, 8×...
        sleep := time.Duration(rand.Int63n(int64(backoff))) // jitter in [0, backoff)

        select {
        case <-time.After(sleep):
        case <-ctx.Done():
            return ctx.Err() // give up if the overall deadline passed
        }
    }
    return fmt.Errorf("after %d attempts: %w", attempts, err)
}

"Full jitter" — sleeping a random duration in [0, backoff) — is the variant that best spreads out retries. Without jitter, every client that failed at the same instant retries at the same instant, and you get synchronized spikes that keep knocking the dependency over.

Mark which errors are worth retrying with a small predicate (post 3's errors.As/Is is the tool):

func isRetryable(err error) bool {
    if errors.Is(err, context.DeadlineExceeded) {
        return true
    }
    var nerr net.Error
    if errors.As(err, &nerr) && nerr.Timeout() {
        return true
    }
    return false // default: don't retry unknown errors
}

Circuit breakers — stop hammering a dead dependency

Retries handle blips. When a dependency is genuinely down, retrying just piles on. A circuit breaker detects sustained failure and "trips" — failing fast without even attempting the call — then periodically tests whether the dependency has recovered. It has three states:

• Closed — normal operation; calls pass through, failures are counted.
• Open — the breaker tripped; calls fail immediately (fast), sparing both you and the struggling dependency. After a cooldown, move to half-open.
• Half-open — let a limited number of trial calls through. Success closes the breaker; failure re-opens it.

A sketch of the core logic (use a battle-tested library in production — there are good ones — rather than shipping your own):

func (cb *Breaker) Call(op func() error) error {
    if !cb.allowRequest() { // open and still cooling down?
        return ErrCircuitOpen
    }
    err := op()
    cb.record(err) // updates failure count / state transitions
    return err
}

The win is twofold: callers fail fast (no waiting on doomed timeouts), and the ailing dependency gets breathing room to recover instead of being pinned under retry load. Pair it with a fallback — serve stale cache, a default, or a degraded response — so an open circuit degrades gracefully rather than erroring outright.

Rate limiting — protect yourself and others

Rate limiting caps how fast operations happen — to respect a downstream's limits, to protect your own service from overload, or to enforce fairness. Go's standard golang.org/x/time/rate implements a token-bucket limiter:

import "golang.org/x/time/rate"

// 100 events/sec, with bursts of up to 20.
limiter := rate.NewLimiter(100, 20)

func (s *Service) handle(ctx context.Context, req Request) error {
    if err := limiter.Wait(ctx); err != nil {
        return err // context cancelled while waiting for a token
    }
    return s.process(req)
}

Wait blocks until a token is available (or the context is cancelled); Allow() is the non-blocking variant for "reject immediately if over limit." Token buckets allow short bursts while bounding the sustained rate — usually what you want for both client-side throttling and server-side protection.

Bulkheads — isolate failures

Named after a ship's watertight compartments: partition resources so a failure in one area can't sink the whole vessel. Concretely, give each dependency its own bounded pool of concurrency so a slow one can't consume every goroutine and starve the others. A semaphore is the simplest bulkhead:

// Each downstream gets its own concurrency budget.
var paymentSem = semaphore.NewWeighted(10)
var searchSem  = semaphore.NewWeighted(50)

func callPayment(ctx context.Context) error {
    if err := paymentSem.Acquire(ctx, 1); err != nil {
        return err
    }
    defer paymentSem.Release(1)
    return payment.Do(ctx)
}

If payment goes slow and saturates its 10 slots, callers to payment queue or fail — but search still has its own 50 slots and keeps serving. The failure is contained to one compartment.

Layer them in the right order

These patterns compose, and the nesting order matters. A useful default, from outermost to innermost:

   rate limiter        ← shed load before doing any work
     └─ bulkhead       ← bound concurrency per dependency
         └─ circuit breaker  ← fail fast if the dependency is down
             └─ retry        ← handle transient blips
                 └─ timeout  ← bound every individual attempt
                     └─ the actual call

The reasoning: reject excess load early (rate limit), isolate what you do admit (bulkhead), skip calls to known-dead dependencies (breaker), retry the transient failures that remain, and bound each attempt with a timeout. Putting retry inside the breaker means a tripped circuit stops the retries too — exactly what you want.

The takeaway