Webhook Delivery Platform
Durable, multi-tenant event delivery
HLD · Distributed systems
A webhook platform built around an uncomfortable reality: a receiver can accept an event just before the sender crashes. Designed for durable admission, bounded retries, and recoverable delivery.
Design decisions
- Commits events and delivery records atomically in PostgreSQL, eliminating a database-to-queue dual-write gap.
- Coordinates workers with row locks, tenant concurrency limits, expiring leases, and tokens that reject stale completions.
- Signs payloads with HMAC, supports overlapping secret rotation, and validates destination IPs with a total request deadline.
- Preserves attempt history through exponential-backoff retries, failure exhaustion, and idempotent manual replay.
Tradeoffs
At-least-once attempts, with explicit receiver deduplication. No promise of exactly-once remote side effects or event ordering.
Local verification
29 tests · signed delivery verified: 503 → 503 → 204
Read the verification notes ↗