Make webhook consumers safe to retry

A webhook sender cannot know whether your service processed an event if the connection drops before it receives the response. Many providers retry. Some events arrive late or out of order. A reliable consumer assumes duplicates are normal and makes its side effects safe.

Verify before trusting the payload

Validate the provider signature against the raw request body using the documented algorithm and a constant-time comparison. Check the timestamp window if the provider supports it. Keep signing secrets in a secret manager and support deliberate key rotation. A valid signature authenticates the sender; it does not make every field safe to use in SQL, a file path, or a command.

Persist before acknowledging

For short synchronous work, verify the event, record its unique provider ID, and perform a bounded transaction before returning success. For longer work, persist the verified event into a queue or inbox table and return success only once it is durably accepted. If the process crashes after acknowledgment but before persistence, the event may be lost.

Make side effects idempotent

Use a unique constraint on the provider's event ID or a carefully designed idempotency key. Let the database arbitrate concurrent duplicate deliveries. A handler can then return success for an already-processed event without charging twice, sending two emails, or creating duplicate records.

INSERT INTO received_events (provider, event_id, received_at)
VALUES (?, ?, CURRENT_TIMESTAMP)
ON DUPLICATE KEY UPDATE event_id = event_id;

Handle ordering and recovery

Do not assume creation time is delivery order. Where state transitions matter, use provider sequence numbers or fetch the canonical object after receiving an event. Track attempts, last error, and a terminal/dead-letter state. Retry transient failures with backoff; surface permanent failures for review.

• Return promptly; do not run unbounded work in the request handler.
• Keep event payload retention proportional to debugging and privacy needs.
• Test duplicate, concurrent, delayed, malformed, and replayed events.

The exact mechanism depends on the provider and database, but the invariant is stable: if the sender retries, your system should converge on one intended business effect.