Arihant Global Ventures Inc. Machine-readable Markdown for this page: send Accept text/markdown, or open /guides/webhook-reliability-patterns/index.md. Site overview for language models: /llms.txt. Full guide text: /llms-full.txt.

Canadian enterprise & public-sector advisoryPrincipal-led deliveryToronto, ON, Canada

Arihant Global Ventures Inc.
Discuss an Assessment
← All guides

Guide · October 11, 2026 · 7 min read

Webhook Reliability Patterns: Ingestion That Survives the Real World

How to build webhook ingestion that survives retries, duplicates, outages, and bad payloads: signature verification, fast-ack processing, idempotency, raw payload storage, and poll fallbacks.

Enterprise ArchitectureIntegrationReliabilityAPIs

Webhooks are the easiest integration pattern to build badly. A provider fires an HTTP POST at your endpoint, you process it inline, you return 200 — and it works right up until it doesn’t: a retry storm after your database stalls, a duplicate delivery that double-credits a customer, a payload format that changed silently on a Tuesday. The difference between an ingestion pipeline that runs quietly for months and one that pages you at 2 AM is five patterns. None of them are clever. All of them are load-bearing.

1. Verify the Signature Before You Do Anything Else

Every webhook endpoint should start the same way: prove the payload came from the provider before parsing it, logging it, or touching the database. Payment providers sign their webhooks — typically HMAC-SHA256 over the raw body with a shared secret — and skipping verification because “the URL is unguessable” is how you end up processing forged events.

Three rules:

  • Verify against the raw body bytes, not the parsed JSON. Parsing and re-serializing can change whitespace or key order, which breaks the HMAC. Read the raw body, compute the signature, compare with a timing-safe comparison, and only then parse.
  • Store one secret per connection, not one per provider. When multiple merchants each connect their own provider account, each connection has its own signing secret. The verification step looks up the secret by the connection identifier in the webhook URL or headers — which means the URL must carry enough information to identify the connection before verification, without leaking anything sensitive.
  • Fail closed. If the signature is missing or invalid, return an error status and log the attempt. Do not process the payload “just in case.” A provider misconfiguration is a support ticket; a forged event that credits points is an incident.

2. Fast-Ack, Then Process Asynchronously

The single most important architectural decision: never do real work inside the webhook request. The endpoint’s job is to accept the payload durably and return 200 as fast as possible. Processing happens asynchronously.

Why this matters in practice: providers retry aggressively on timeouts. If your handler takes ten seconds because it is calling a reward engine, writing to three tables, and sending an email, the provider’s 5-second timeout fires, it retries, and now you have two copies of the same event being processed concurrently. A slow downstream dependency can turn one delayed batch into thousands of duplicate deliveries.

The pattern:

  1. Webhook arrives. Verify signature.
  2. Store the raw payload in a durable queue or table, with the provider name, event type, and arrival timestamp. Return 200.
  3. A background worker picks up queued events and runs the real pipeline: parse, normalize, check idempotency, apply business logic.

The queue can be a database table, a message broker, or a managed queue — the mechanism matters less than the guarantee: the HTTP response is decoupled from the processing. A slow database or a bug in your business logic must never cause the provider to retry, because retries multiply every other problem.

3. Idempotency Keys With Database Unique Constraints

Even with fast-ack, duplicates happen. Providers retry on network blips. Your own poll fallback (see pattern 5) can deliver an event the webhook already processed. Two workers can pick up the same queued event. Idempotency is not optional; it is the core correctness guarantee of the pipeline.

The pattern that works: derive a deterministic idempotency key from the event itself — typically {provider}:{merchantId}:{providerTransactionId} — and enforce it with a database unique constraint, not application logic.

CREATE UNIQUE INDEX ON idempotency_keys (provider, merchant_id, provider_txn_id);

The insert-or-skip happens before any business logic runs. If the key already exists, the event is a duplicate: acknowledge it and move on. Application-level “check then insert” has a race window; the unique constraint does not. This is one of those cases where the database is the right place for the guarantee.

Two details that matter:

  • Give keys a TTL. Thirty days is a reasonable default for payment events. Old keys can be purged; the dedup window only needs to cover the provider’s retry horizon plus your poll overlap.
  • Make the key tenant-scoped. In a multi-tenant system, the same provider transaction ID can legitimately appear under different merchants. The unique constraint must include the tenant or merchant identifier, or you will silently drop real events.

4. Store Raw Payloads for Audit and Replay

Before normalization, before business logic, store the original payload exactly as received. This table is append-only and never updated. It is the cheapest insurance in the entire pipeline.

Raw payload storage pays for itself three ways:

  • Debugging. When a merchant reports a missing transaction, the first question is “did the provider send it, and what did it contain?” Without the raw payload, you are reconstructing history from logs. With it, you query one table.
  • Replay. When you fix a bug in the normalization or reward logic, you can reprocess historical events through the corrected pipeline. Without raw payloads, the data is gone — the provider will not resend six weeks of webhooks because you asked nicely.
  • Disputes and audit. Payment-adjacent systems get questioned. “Show me exactly what the provider sent” is an answer you want to have, byte for byte.

Store the provider name, event type, connection identifier, arrival timestamp, and the raw body. Index by connection and time range — the queries you will run are “show me everything from this merchant in this window.”

5. Poll Fallback With Per-Connection Watermarks

Webhooks are the primary path, but they are not reliable enough to be the only path. Providers have outages. Your endpoint has deploys. A firewall rule changes and silently drops traffic for a day. Every webhook integration needs a poll fallback: a scheduled job that asks the provider’s API “what happened since I last checked?”

The watermark pattern:

  • Each provider connection stores a watermark — the timestamp (or cursor) of the most recent event successfully ingested.
  • The poll job runs on a schedule, fetches events newer than the watermark, processes them through the same pipeline (including the idempotency check — this is why pattern 3 exists), and advances the watermark.
  • Prefer the provider’s creation timestamp over arrival time for the watermark, and add an overlap window (two hours is a reasonable default) so late-arriving events are not missed.

The poll path and the webhook path must converge on the same processing pipeline. Two code paths that do the same thing will diverge; one pipeline with two entry points will not. The idempotency key is what makes the overlap safe: events delivered by both paths are processed exactly once.

One more consideration: poll on a per-connection basis with per-connection failure isolation. If one merchant’s token expires, that connection’s poll job should fail loudly and visibly without blocking every other merchant’s ingestion. Token refresh failures are the most common poll outage — build the refresh daemon and the alerting for it from day one.

Putting It Together

The five patterns form a single pipeline: verify the signature, accept fast and queue, dedupe on a deterministic key, keep the raw evidence, and backstop everything with polling. Each pattern covers the failure mode the others cannot: verification stops forgery, fast-ack stops retry storms, idempotency stops double-processing, raw storage stops data loss, polling stops silent gaps.

The cost is real — a queue, a raw events table, an idempotency table, a poll scheduler — but it is a fraction of the cost of the alternative: a loyalty or payments platform that double-credits customers, loses transactions silently, or cannot explain what happened when a merchant asks. Build the pipeline once, and it runs quietly. Skip it, and every provider integration becomes a pager.

For the token lifecycle that keeps poll-based ingestion running — encryption at rest, refresh daemons, expiry alerting — see our POS integration playbook. If you are scoping an integration project and want these patterns designed into the architecture from the start, review our engagement models or contact us.

Start with a conversation.

A free 30-minute introduction to discuss your AI governance challenge, constraints, and whether we can help. No project brief required.

Contact us for current availability. Scope, timing, and fees are agreed before paid work begins.