Introduction
### The Retry Mirage Ever think slapping retries onto your async payment gateway will fix everything? It’s like using a sledgehammer to crack a walnut—sure, you’ll get results, but hello, collateral damage. Payment timeouts, network blips, downstream hiccups: retries handle the failure bit, not the side effects. Without coordination, your client may retry a charge and accidentally bill the same user twice. ### Why Mutexes Matter Distributed systems are messy. Naïve retries risk duplicate processing, race conditions, and that all-too-real nightmare: a request flood. Mutexes (locks) and idempotency keys act as the velvet rope that lets only one request pass through the critical “debit this account” section—no VIP double-charges, no messy rollbacks. ## Key Insights ### Retry vs. Idempotency – Retries recover from transient errors, not state consistency.
- Idempotency guarantees repeated calls leave the system unchanged.
- You need both: retries to recover, mutexes to coordinate. ### The Mutex Club Pattern This umbrella term covers distributed mutexes (e.g., Hazelcast’s FencedLock), leader-election locks, or protocol-layer idempotency keys (Stripe, PayPal). They ensure at-most-once execution—even if clients hammer your API after a crash or timeout. ## Real-World Impact ### Metrics That Matter One fintech team combined exponential backoff retries, circuit breakers, and distributed locks—timeouts dropped 60%, circuit-breaker trips fell 90%, and thundering-herd retries turned into polite, staggered requests. ### Hazelcast’s FencedLock Using Raft-based fencing, FencedLock assigns sequential tokens to lock holders. A retry after a crash carries an outdated token and is rejected, guaranteeing “exactly-once” delivery without complex dedupe logic. ## Conclusion Retries alone are the hammer; mutexes are the safe. Are your payment flows wielding both? Or is your API about to double-send the bill? — References: https://hazelcast.com/blog/testing-the-cp-subsystem-with-jepsen/, https://leapcell.io/blog/seven-retry-patterns
