The payment webhook failure I had to inject on purpose

Most webhook testing stops at the happy path plus the signature check. That covers the cases where the provider misbehaves. It does not cover the case where your own storage misbehaves halfway through a write. The order update lands, the event bookkeeping does not, and the system now disagrees with itself about whether a customer paid.

I built a small local lab for payment webhook failures to answer one question: when the worst failure happens between two writes, does anything catch it? The failure I care about most is one I inject on purpose. This post walks through the injection, the containment, and the recovery path for when the automatic attempts run out.

The lab in one paragraph

It is Python standard library only. A WSGI endpoint at POST /webhooks/payment accepts a signed JSON event, verifies an HMAC-SHA256 signature over the exact raw bytes, validates the schema, then applies the event to a SQLite order store. No live provider, no credentials, a synthetic secret called sandbox-secret, and fixtures signed by a shell script. The whole failure switch is one attribute:

service.failure = "partial_db_failure"

Set it, send the fixture, and the storage layer fails in a very specific spot: after the first write instead of before both.

What partial failure actually means here

The lab runs the event handler against two tables in one transaction. Order status flips from pending to paid in the first write. The event ledger insert, the record that says event X was processed at time T, comes second. The failure switch kills the connection between the two. On restart the order says paid, the ledger says nothing, and the idempotency check that would normally block a duplicate replay has no row to check against.

This is the state providers warn about when they say webhooks are at-least-once. The retry will come. The question is whether your system can absorb it without double-applying the payment or silently dropping it.

The recovery path, step by step

First, the replay arrives and the signature verifies. The handler looks up the event id in the ledger and finds nothing. A naive implementation treats missing as unprocessed and re-applies the payment mutation. The lab treats missing-but-paid as suspicious: it checks whether the order was already flipped by an event with this id, using a deterministic correlation key stored on the order row itself, not in the ledger.

If the correlation key matches, the handler writes the ledger row with the original timestamp from the event payload, marks it reconciled, and does not touch the order again. If it does not match, the event goes to a review queue with the full payload attached. No automatic mutation happens on ambiguity.

The review queue is boring on purpose: a table, a status, an authorized replay endpoint that requires a human decision. When I replay from the queue, the lab applies the mutation with the same idempotency guarantees as the first attempt. The queue drains through a decision, not a timer.

What the injection proved

Three findings came out of running this failure on purpose.

One: the signature check and the schema check both pass on the replay. Verification of authenticity and validation of shape have nothing to say about whether storage is consistent. The dangerous case is invisible to both.

Two: the correlation key saved the day, and it only existed because I put it there. The default version of the lab, without the key, double-applied the payment on replay and the tests stayed green, because nothing asserted that a paid order stays paid after an unknown-ledger replay. The bug was in the assertion list before it was in the code.

Three: bounded retries on the first delivery attempt are necessary and insufficient. They handle the provider timing out before your response. They do nothing for the case where your process dies mid-transaction. That case needs the reconciliation path, not more retries.

The checklist I took from this

Before you trust a webhook handler in production, check for these:

• The event ledger write and the business mutation share one transaction, or there is an explicit reconciliation path between them
• A deterministic correlation key exists outside the ledger, so a replay can be matched even when the ledger row is missing
• Missing-ledger plus already-paid is a distinct branch with its own handling, not an exception that falls through to re-apply
• Ambiguous events go to a review queue with the full payload, and replay from the queue is authorized and audited
• A test exists that injects the failure between the two writes and asserts the order is not double-applied after recovery

The part that generalize

The pattern here is bigger than payments. Any pipeline with two writes where the second depends on the first has this exposure: order systems, ledger entries, audit logs, session stores, notification records. The failure is rare, the tests are green, and the blast radius is money or trust. Injecting the failure on purpose in a lab is how you find out which side of that line your system is on before a real outage answers for you.

The lab code is small enough to read in one sitting and it runs anywhere Python runs. I would rather keep it that way than turn it into a framework. If you want the same coverage for your own webhook flow, the checklist above is the shortest path: the code is the easy part, the assertion list is where the bugs hide.

If this was useful, my scanner and other MCP security tooling are linked on my profile. I write one post like this most weeks, usually about the failure modes that only show up when something breaks between two writes.

Originally published on DEV. Discussion happens there.