What happens when webhooks drop or reorder during a host cutover? A one-to-two minute gap can trigger duplicate charges, missed orders, or broken automations for downstream integrators. Engineers juggling cost, uptime, and ordering guarantees need a migration that avoids lost or reordered events and that minimizes disruption.
Migrating webhooks and third-party integrations during host moves: plan and execute a staged migration that keeps existing webhook delivery while switching to the new host. Dual-deliver to old and new endpoints. Implement idempotent handlers and deterministic dedupe. Replay historical events via ETL scripts. Use health checks and SLIs for cutover gating. Prepare rollback toggles and communication templates for third-party integrators.
Summary of the process
Start a staged migration that keeps the old endpoint active while the new host receives the same events. Then validate, promote, and deprecate the old endpoint after monitoring clears. Run an ordered ETL replay for historical events and close the window.
- Inventory sources, auth, and delivery history.
- Stand up new endpoints and enable dual-delivery fan-out.
- Implement idempotent handlers and deterministic dedupe.
- Run smoke, integration, and E2E tests tied to SLIs.
- Promote via automated gates and backfill with ETL replay.
- Deprecate the old endpoint after the replay and monitoring window.
Take one small pause and verify your checklist.
Step 1: inventory and prepare
List every webhook subscription, auth method, and rate limit before any changes. This inventory takes between 30 and 120 minutes depending on the number of providers. The deliverables are an export of endpoints, auth types, and allowed IP ranges.
List subscriptions and auth
Query provider APIs to list webhook endpoints and signing methods. For Stripe, GitHub, and Twilio use their REST endpoints to export configs. Example: run a curl command to fetch endpoints from Stripe using a secret key.
Bash
curl -s -u 'sk_test_xxx:' /
https://api.stripe.com/v1/webhook_endpoints | jq '.' > stripe_webhooks.json
Export delivery history
Export delivery logs for the last 30 to 90 days including event_id and timestamp. Preserve provider delivery IDs and retry metadata in the export. Stripe maintains webhook events and delivery attempts that help reconstruct ordering and failure reasons (Stripe Webhook docs).
Export event_id, timestamp, original_signature, and delivery_attempt in the ETL export. Keep payloads encrypted at rest during transfer for compliance.
Step 2: dual-delivery and idempotency
Enable fan-out so each incoming webhook posts to both old and new endpoints at the same time. Dual-delivery avoids lost events during DNS or routing changes. Implement idempotency and deterministic dedupe before enabling fan-out.
Fan-out proxy and examples
Use a small proxy to forward incoming requests concurrently to two backends. The proxy must only acknowledge receipt to the provider after confirming durable enqueue or successful acknowledgement from the target backends. If durable enqueue cannot be confirmed for both targets, return a non-2xx to trigger provider retry; return 202 Accepted only when the proxy guarantees reliable persistence and later redelivery.
This preserves the provider's retry semantics and downstream delivery guarantees. A proxy removes dependence on provider retries and centralizes signature checks.
Python
import requests
def fan_out(req):
- body = req.get_data() headers = {'X-Forwarded-For': req.remote_addr} urls = ['https://old.example.com/webhook','https:/new.example.com/webhook'] Replace with behavior that checks responses and persists before ack: for each target, attempt enqueue to a local durable queue or POST with retry/backoff and verify a 2xx enqueue response
- if both enqueues succeed, return 200 to the provider
- if one fails transiently, retry with exponential backoff and alert on repeated failures
- if persistent failure occurs, return a 5xx so the provider retries and escalate
In short, the example should show confirmation of durable persistence or coordinated retries rather than an unconditional immediate OK.
Deduplication design
Design idempotency keys from provider event_id plus consumer id and webhook version. Store keys in Redis with a TTL longer than the replay window. Use a consistent first-write policy for ordering-sensitive consumers.
Sql
-- PostgreSQL upsert pattern
INSERT INTO processed_events (id, event_id, consumer, processed_at)
VALUES ($1,$2,NOW())
ON CONFLICT (event_id, consumer) DO NOTHING;
The most common error at this point is enabling fan-out without idempotency. That error creates duplicates and breaks downstream state machines.
Small pause to review your idempotency key logic.
Step 3: ETL replay and backfill
Extract raw delivery logs while preserving event IDs, timestamps, and provider metadata. Transform logs into newline-delimited JSON and include replay_metadata with original timestamps. Load by replaying into the fan-out proxy or directly to the consumer with dedupe headers.
Fetch provider event history via API and save in compressed NDJSON. Keep the original event_id and timestamp fields unchanged. Use field mapping rules to avoid changing semantics during transform.
Bash
cat stripe_events.json | jq -c '.data[] | {event_id:.id, ts:.created, payload:.data}' > events.ndjson
gzip events.ndjson
Replay with ordering preserved
Replay in strict timestamp or sequence order for systems that require ordering. Use batching sized by consumer rate limits. Implement exponential backoff on 429 responses and respect rate limits during replay.
Bash
cat events.ndjson | while read line; do
event_id=$(echo "$line" | jq -r '.event_id')
curl -s -X POST -H "X-Original-Event:$event_id" --data "$line" https://new.example.com/replay
sleep 0.01
done
What most guides omit is preserving event IDs and ordering metadata during replay. Naive replay causes duplicates and breaks workflows.
1. Inventory
→
2. Fan-out
→
3. Test & Monitor
→
4. Promote & Replay
Take a short break and confirm replay order.
Define automated gates that require SLIs to pass before changing DNS or removing the old host. Gates should check delivery success rate, latency, duplicate rate, and ordering violations. The promotion window should be at least 30 to 60 minutes of stable metrics.
Test matrix and smoke tests
Run unit tests for handlers, integration tests for signature verification, and E2E tests for end-to-end delivery. A smoke test must send synthetic events and assert single application of side effects. Plan duration: unit tests 5-20 minutes. Integration and E2E tests take 20-90 minutes depending on systems.
Automate promotion via a CI/CD job that runs smoke tests and evaluates SLIs. Use a time-windowed evaluation such as 15 minutes of aggregated metrics. If any gate fails, the pipeline blocks promotion and triggers rollback playbooks.
The recommendation is simple: require automated SLI gates before any endpoint swap, even for low-volume services. This works if the team enforces idempotent handlers and monitors duplicate rates. If SLI gates fail repeatedly, stop and run replay diagnostics before promoting.
A runnable test checklist reduces ambiguity during gating. Example checklist entries include concrete assertions and commands. Each item shows the command, expected result, and the threshold to pass.
Pause to confirm the checklist commands run cleanly.
Step 5: cutover, rollback, and cleanup
Promote after gates pass and shift traffic gradually when possible. Keep the old endpoint live for the replay window and signal deprecation in the response header. Remove the old endpoint only after the replay and monitoring window closes.
Gradual traffic shift and deprecation
If the load balancer supports weight-based routing, shift traffic in steps: 5%, 25%, 75%, 100%. Each step should run for the SLI window duration. After 100% and stable metrics, mark the old endpoint deprecated but not deleted.
Rollback strategy and toggles
Keep API versioning or a feature flag to revert instantly to the old endpoint. Re-enable dual-delivery to restore safety and run replay for missed events. If rollback is permanent, rotate tokens and secret material for the new host.
Cleanup and final replay
Run a final ETL replay for any remaining events and track dedupe hits. Archive logs for 90 days as a compliance safety net. Document the event counts and SLI history for the postmortem.
In-flight webhook handling and endpoint versioning must be explicit for true zero-downtime migration. Before shifting traffic, enable connection draining on old hosts and configure the fan-out proxy to durably enqueue incoming requests. Only acknowledge the provider once the event is durably persisted.
During cutover, perform weight-based routing: route 95% to old and 5% to new, then increment weights after each SLI window. For in-flight requests, set a drain period equal to the maximum processing timeout plus a safety margin. Use versioned endpoint names like /v1/webhook and /v2/webhook so rollback is a routing change.
Maintain a rollback playbook with commands to re-enable dual-delivery, switch weights back, and trigger an immediate replay. These measures preserve delivery ordering and prevent dropped or partially-processed in-flight webhooks.
Short pause to ensure runbooks and playbooks are available.
Scripts and templates
Provide runnable snippets for fan-out, ETL export, dedupe, Terraform, and CI. Copy and paste these into your pipeline and replace secrets from your secret store. Each snippet is ready to adapt and run in staging.
Fan-out proxy minimal node example
- js // fanout.js minimal example const http = require('http')
- const fetch = require('node-fetch')
- http.createServer(async (req,res)=>{ let body=''
- req.on('data',d=>body+=d)
- req.on('end', async ()=>{ const urls=['https://old.example.com/webhook','https:/new.example.com/webhook']
- await Promise.all(urls.map(u=>fetch(u,{method:'POST',body})).catch(()=>null))
- res.writeHead(200)
- res.end('OK')
- })
- }).listen(8080)
ETL replay pattern
python
import requests, json
with open('events.ndjson') as f:
for line in f:
obj=json.loads(line)
headers={'X-Original-Event': obj['event_id']}
requests.post('https://new.example.com/replay', json=obj['payload'], headers=headers)
Idempotency redis lua example
lua
-- KEYS[1]=idempotency_key, ARGV[1]=ttl_seconds
if redis.call('SETNX', KEYS[1], 1) == 1 then
redis.call('EXPIRE', KEYS[1], tonumber(ARGV[1]))
return 1
end
return 0
hcl
resource "aws_route53_record" "webhook_alias" {
zone_id = var.zone_id
name = "webhooks.example.com"
type = "CNAME"
ttl = 300
records = [aws_lb.webhook.dns_name]
}
GitHub actions snippet for gating
yaml
name: webhook-cutover
on: workflow_dispatch
jobs:
gate:
runs-on: ubuntu-latest
steps:
- name: Run smoke tests
run: ./scripts/run_smoke_tests.sh
- name: Check SLIs
run: ./scripts/check_slis.sh
Pre-cutover, cutover-day, and post-cutover communications should be explicit, timestamped, and copy-ready. For example: Subject: "Scheduled webhook cutover for webhooks.example.com: [DATE/TIME UTC]". Body: "We will perform a zero-downtime migration at [UTC window]. During the window we will dual-deliver events to both your current endpoint and a new endpoint: https://old.integration.com/webhook and https://new.integration.com/webhook.
No action is required to continue receiving events, but please verify that requests signed with your existing HMAC key are accepted and that your handler is idempotent. If you maintain IP allowlists, please add these CIDR ranges: 203.0.113.0/24 and 198.51.100.0/24 for the migration window. After the migration completes, we will send a verification request and a deprecation notice at +48h.
If you prefer a maintenance window instead of dual-delivery, reply with your available slots. Include the contact for on-call support: ops@example.com (pager)." This template gives integrators concrete endpoints, IPs, verification steps, and an explicit rollback contact so downstream teams can prepare and validate.
Short pause to verify contacts and IP lists.
Errors that ruin the result
Switching endpoint URLs without fan-out is the fastest way to lose events. That mistake leads to permanent data loss if providers stop retries. Always enable fan-out or a reverse proxy before DNS changes.
Common pitfalls
Relying only on provider retries without idempotency creates duplicate side effects. Not preserving event IDs during replay causes state machines to misbehave. Forgetting to update IP allowlists breaks deliveries for locked integrators.
Recovery tactics
If duplicates occurred, rely on idempotent handlers and process logs to reconcile state. If ordering broke, replay events with sequence metadata and apply compensating actions. If deliveries stop, re-enable dual-delivery and open a replay window.
Pause to document recovery steps and owners.
When not to apply this method
If the team uses a managed integration product that does not allow replay or dual-targeting, prefer a provider-level migration plan instead. If third parties refuse pre-approved IP ranges or mutual TLS, use a scheduled maintenance window and post-migration replay instead.
If a quick runbook review would reduce risk, run the included smoke tests and replay on your staging environment before production. If a single CTA is helpful to finalize this plan, paste your cutover window and a list of top five integrators into the kickoff checklist above and run the smoke tests in a staging environment.
Frequently asked questions
How to validate webhooks after moving hosts?
Run automated smoke tests that send synthetic events and assert single processing. Then run integration tests that validate HMAC signatures and payload schemas. Compare delivery rates, retry counts, and dedupe hits between old and new endpoints over a 30 to 60 minute window.
How to replay historical events safely?
Export provider logs preserving event_id and timestamps. Replay in original order with dedupe headers and exponential backoff. For ordering-sensitive systems, replay into a queue that enforces sequence before processing.
How long should the replay window be?
A typical replay window runs 24 to 72 hours depending on business needs. Low-risk systems can use 24 hours. Financial workflows often need 72 hours. Keep logs and replay idempotency keys for the full window.
How to preserve API keys and signing secrets?
Rotate secrets only after both endpoints validate the previous secret. Share new signing certificates to integrators ahead of time. Use vaults for secret injection and audit all secret rotations during the cutover.
Block promotion if success rate drops below 99.9 percent, p95 latency exceeds 500ms, duplicate rate exceeds 0.1 percent, or ordering violations appear. Automate gating and require manual review for repeated failures.
How much does reconfiguring webhooks usually cost?
Costs vary widely: simple DNS changes cost near zero, while enterprise migrations with VPNs and support windows cost thousands of dollars. Expect specialist time of 8 to 40 hours for moderate migrations, depending on complexity and the number of integrators.
Pause to gather cost approvals and budget owners.
Closing resources and references
The practices described follow provider guidance and security standards, including NIST SP 800-53 for handling logs and changes (NIST SP 800-53). Stripe webhook recommendations for replay and signing are documented by Stripe (2023). The SLI targets above are operational guidance used in current SRE runbooks and validated in live migrations.
If webhooks are low-volume and downtime is acceptable, or if the provider forbids dual-delivery, this staged fan-out and replay plan is not applicable. Use scheduled maintenance and a replay-only approach instead when appropriate.
Will dual-delivery create duplicate side effects?
Only if handlers are not idempotent or dedupe is missing. Design idempotency keys using provider event_id and consumer id to avoid duplicates. Track dedupe hits and keep duplicate rate under 0.1 percent as an SLI.
