Shopify Webhook Patterns: Reliable Event-Driven Architecture
Webhooks are the nervous system of any subscription commerce stack. When a customer updates their payment method in ReCharge (or Skio, which now shares an owner with ReCharge), when Shopify processes a recurring order, when a subscription is cancelled at midnight on a Sunday — your backend needs to know about it instantly and handle it correctly. Get this wrong and you end up with phantom subscriptions, missed fulfilments, and customer data that silently drifts out of sync.
Having built and maintained webhook handlers for subscription backends processing tens of thousands of events daily, I can tell you the difference between a fragile integration and a reliable one comes down to a handful of patterns. This article covers the architectural decisions that matter most.
typical Shopify webhook delivery rate under normal conditions
of deliveries are duplicates in production — rising to 30%+ during platform incidents
response window before Shopify and ReCharge mark a delivery as failed
The Webhook Pipeline
A reliable webhook architecture separates reception from processing. The HTTP handler acknowledges the event immediately, a message queue buffers it, and an asynchronous worker handles the business logic with its own retry mechanism. Here is the full request chain.
Shopify / ReCharge
Event source
Your Server
Verify + ACK
Queue
Messenger / SQS
Worker
Business logic
Database
MariaDB / Doctrine
The full webhook pipeline: events are verified and acknowledged immediately, then processed asynchronously via a message queue.
Start With HMAC Verification — No Exceptions
Every webhook endpoint you expose is a public URL. Without verification, anyone who discovers it can send fabricated payloads — creating fake orders, triggering bogus cancellations, or corrupting your customer records. Shopify signs every webhook with an HMAC-SHA256 hash using your app’s shared secret. ReCharge does the same with its own client secret.
The verification process is straightforward: read the raw request body (before any JSON parsing), compute the HMAC using your secret key, then compare it against the signature header using a timing-safe comparison function. In PHP, the hash_equals function handles this correctly. In Symfony, you would typically implement this as an event listener or middleware that runs before the controller, rejecting any request where the signature does not match.
HMAC verification in a Symfony event listener
public function onKernelRequest(RequestEvent $event): void
{
$request = $event->getRequest();
$rawBody = $request->getContent();
$hmacHeader = $request->headers->get('X-Shopify-Hmac-Sha256', '');
$computed = base64_encode(
hash_hmac('sha256', $rawBody, $this->shopifySecret, true)
);
if (!hash_equals($computed, $hmacHeader)) {
$event->setResponse(new Response('', 401));
}
}A common mistake is reading the request body after Symfony has already parsed it into a parameter bag. You need the raw bytes exactly as they arrived. Use the Request object’s getContent method before anything else touches the payload.
One critical detail: always return a 200 or 401 response promptly. If your verification takes too long because you are doing database lookups first, the webhook provider may time out and schedule a retry, leading to duplicate processing.
Idempotency: The Single Most Important Pattern
Webhooks will be delivered more than once. Shopify explicitly retries failed deliveries up to 19 times over 48 hours. ReCharge retries for up to 5 days. Network blips, load balancer timeouts, and deployment windows all cause retries even when your handler actually succeeded. Your handlers must produce the same result whether they process an event once or five times.
duplicate delivery rate
In production systems I have monitored, between 2% and 8% of webhook deliveries are duplicates. During Shopify platform incidents, that figure can spike to 30% or higher for a brief period. Without idempotency, every duplicate creates a risk of double-processing — duplicate charges, duplicate fulfilments, or corrupted state.
The simplest idempotency strategy uses a unique event identifier. Shopify includes an X-Shopify-Webhook-Id header with every delivery. Store this identifier in a database table when you begin processing. Before doing any work, check whether that identifier already exists. If it does, return 200 immediately and skip all processing.
For ReCharge webhooks, which do not always include a unique delivery identifier, you need to construct your own idempotency key from the payload. Combine the webhook topic, the resource ID, and the updated_at timestamp. This gives you a composite key that is unique per state change rather than per delivery.
In Doctrine ORM, I use a dedicated WebhookEvent entity with a unique constraint on the idempotency key column. Attempting to persist a duplicate triggers a unique constraint violation, which the handler catches and treats as a no-op. This is more reliable than a check-then-insert pattern because it eliminates the race condition window between the SELECT and INSERT.
Respond Fast, Process Later
Both Shopify and ReCharge expect a 2xx response within 5 seconds. If your handler needs to call external APIs, run complex business logic, or update multiple database tables, you will routinely exceed that window. The solution is to separate acknowledgement from processing.
The pattern is simple: validate the HMAC, store the raw payload in a queue table with a “pending” status, and return 200. A separate worker process picks up pending events and handles the actual business logic. In Symfony, the Messenger component is ideal for this. Dispatch a message containing the webhook payload and topic, then let a worker consume it asynchronously.
This decoupling has a secondary benefit: if your processing logic throws an exception, the webhook delivery is not marked as failed. You avoid triggering the provider’s retry mechanism (which compounds the problem) and instead rely on your own retry logic with proper backoff.
Exponential backoff retry schedule
Each retry waits progressively longer. After all retries are exhausted, the event is moved to a dead letter queue for manual inspection.
Event Ordering Is Not Guaranteed
A customer might update their address and then cancel their subscription in quick succession. Your backend might receive the cancellation webhook before the address update. If your cancellation handler deletes the subscription record, the subsequent address update handler will fail with a “not found” error or, worse, recreate a record that should not exist.
There are two approaches to handling out-of-order delivery. The first is timestamp-based conflict resolution: include an updated_at field on your local records and only apply changes from webhooks with a newer timestamp. If a webhook arrives with an older timestamp than your current record, discard it.
The second approach, which I prefer for critical subscription state, is to treat webhooks as notifications rather than data carriers. When you receive a subscription/updated webhook, do not blindly apply the payload data. Instead, use the webhook as a trigger to fetch the current state directly from the ReCharge or Shopify API. This guarantees you always have the latest data regardless of delivery order.
Dead Letter Queues and Failure Monitoring
When a webhook handler fails after your own retry attempts are exhausted, the event needs to go somewhere visible. A dead letter queue (DLQ) is a holding area for events that could not be processed. In practice, this is often a database table with the original payload, the error message, the number of attempts, and the timestamp of the last failure.
Symfony Messenger has built-in support for failure transports, which function as a DLQ. Configure a failure transport backed by Doctrine, and any message that exceeds your retry limit is automatically moved there. You can then build an admin interface to inspect, replay, or discard failed events.
Monitoring is equally important. Set up alerts for DLQ growth rate, not just size. A steady trickle of failures might be acceptable, but a sudden spike usually indicates a systemic issue — a changed API response format, an expired credential, or a database connection problem. I pipe these alerts through Sentry with custom fingerprinting so that distinct error types are grouped separately.
The Reliability Checklist
HMAC verification
Validate every payload against the provider signature before any processing. Use timing-safe comparison.
Idempotency keys
Store a unique event identifier per delivery. Use database unique constraints to prevent race conditions.
Queue-first processing
Acknowledge immediately, dispatch to a message queue, and process asynchronously via a worker.
Retry with backoff
Exponential backoff on your own retry logic. Never rely solely on the provider’s retry schedule.
Dead letter queue
Events that exhaust retries are moved to a DLQ for inspection, replay, or manual resolution.
Nightly reconciliation
Poll the source API daily to catch missed events, deployment gaps, and silent webhook failures.
Webhooks vs Polling for Subscription State
Webhooks provide near-real-time updates but are inherently unreliable delivery mechanisms. Polling is slow but comprehensive. The best subscription backends use both: webhooks for immediate responsiveness and periodic polling as a reconciliation layer.
I typically run a nightly reconciliation job that fetches all subscriptions modified in the last 24 hours from the ReCharge API and compares them against local records. Any discrepancies are logged and corrected. This catches the edge cases that webhooks miss: events during deployment windows, payloads that were valid but contained unexpected data shapes, and the rare Shopify platform glitch where webhooks simply are not sent.
The polling job also serves as a health check on your webhook pipeline. If the reconciliation consistently finds zero discrepancies, your webhooks are reliable. If it starts finding dozens, something is broken upstream.
Common Pitfalls With ReCharge Webhooks
ReCharge has some specific behaviours that catch developers off guard. First, their webhook payloads differ between API versions. If you register webhooks on the 2021-11 API but your handler expects the 2021-01 payload format, fields will be missing or renamed. Always ensure your webhook registration and handler agree on the API version.
Second, ReCharge sends a charge/created webhook when a recurring charge is generated, but this does not mean the charge has been processed. You need to listen for charge/paid (or charge/failed) to know the actual payment outcome. Building fulfilment logic on charge/created is a recipe for shipping orders that were never paid for.
Third, subscription/updated fires for every field change, including internal metadata updates that ReCharge makes on its own. Filter on the specific fields your business logic cares about. Otherwise, you will process hundreds of no-op updates per day that consume resources and clutter your logs.
Finally, test your handlers against actual ReCharge webhook payloads, not fabricated ones. The shape of the data in ReCharge’s documentation does not always match what arrives in production, particularly for edge cases like subscriptions with custom properties or bundled items.
Building It Right From Day One
If I were starting a subscription backend today, I would implement these patterns in order: HMAC verification first (non-negotiable security), then the acknowledge-and-queue pattern (prevents timeout failures), then idempotency (prevents duplicate processing), and finally the reconciliation layer (catches everything else).
Each pattern builds on the previous one. Together they form a webhook pipeline that handles the messy reality of distributed systems: messages arrive late, arrive twice, arrive out of order, or do not arrive at all. Your backend needs to be resilient to all four scenarios.
The investment pays for itself quickly. A reliable webhook pipeline means fewer support tickets about missing orders, fewer manual corrections in the database, and fewer 3am alerts about state inconsistencies. It is the kind of infrastructure work that is invisible when done well — and catastrophically visible when done poorly.