The Valley of Webhooks: Why Rebuilding the Same System Three Times Made Me Question the Primitive
After building the same webhook integration system at three different companies, the author exposes the hidden costs of relying on webhooks for data replication. From signature verification and dedup tables to bootstrap importers and reconciliation crons, each layer compensates for webhooks' inherent flaws: no ordering, no completeness, no bootstrap, and no verifiability. The article argues that webhooks are a local optimum, and proposes flipping the arrow: instead of providers pushing notifications, consumers should pull from an ordered, cursor-addressed change log, making the entire stack simpler and more reliable.
It’s a jigsaw puzzle where the manufacturer had the original picture, cut it up, mailed me the pieces one at a time, lost a few in the post, mailed some twice, and printed nothing on the box.
- toomim
This is a nice writeup of the problems in using Webhooks for State Synchronization. I also noticed that the proposed solution is a pseudo IETF-style draft protocol called SCROLL... that happens to be remarkably similar to an actual IETF draft I am bringing to IETF 127 this November called "Braid-HTTP Subscriptions."
Both drafts request a subscription with a GET plus a header:
Scroll Request:
GET /scroll/feed/customers
Prefer: stream
Braid Request:
GET /customers
Subscribe:
In both systems, the GET leaves its response open to stream events. SCROLL responds with application/x-ndjson. Braid subscriptions are a 209 Multiresponse, with content-type application/http-history. This lets them support more than just JSON. You can send updates to the state of CSV, or PNGs, XML, HTML, plain text, or any media type.
The author noted that it's hard to get adoption. Well, the reason that Webhooks are so common is that they are bog-standard HTTP. For this to get adopted, we need to put it into bog-standard HTTP. So we need to go to the IETF, and and extend HTTP in a general way to support state synchronization. It should just work for any existing HTTP media type (not just JSON), and any resource/URL (not just special /scroll/* URLs), and any way of marking timestamps (not just the ordered strings proposed in SCROLL).
Then we can bake this stuff into HTTP, and thus into all our bog-standard libraries, utilities, and code, and you won't have to r […]
- tlonny
I much prefer cursor paginated API requests vs. webhooks. The obvious downside being that in order to not get 429'd you need a respectable poll frequency - meaning you lose reactivity to new events.
Thus I think webhooks still have a place - but as a simple "poke" that can be sent to the client to tell them something has changed - supplementing a default low frequency polling interval.
This gives us the best of both worlds:
1. No need to bother de-duping/retrying pokes - if you miss a webhook you will shortly recover anyway when you next poll.
2. No need for any local-specific tunnelling/tooling - the local app will work just fine with the default poll interval.
3. No need to keep a connection live for each client.
4. All the good stuff OP mentioned in his blog post.
- alt227
I had the exact same thing with the Quickbooks api recently. You cannot trust the responses or webhooks at all.
On create a user or invoice for example sometimes it will return an error, yet it actually created the entity. This means you have to check manually after creating everything to know if its created properly.
Then you have the issue that sometimes quickbooks takes a while to update, and locks the company file while it does some background magic. This means you cannot immediately do the existence check, and also sometimes the check errors or times out which essentially means you need to keep checking forever until you can properly reconcile your db against theirs. But with hundreds/thousands of transactions per minute this state is never reached. You perpetually live in a state of trying to catch up but never managing it.
When I brought it up with Quickbooks dev support their response was literally "Its your job to make sure things are created properly in our system".
How did we get to this place where we started putting up with systems that cannot ever be trusted?
- bytesandbots
With the proposed solution every consumer will have a persistent connection to the server irrespective of the frequency of events. This setup seems inefficient unless you have a very high volume of events coming in. Many CDN networks have a limit on how long a connection you can open. And data providers will not prefer serving persistent requests.
Problems listed are signatures, dedup, buffering, bootstrap, cron. Everything other than signatures and bootstrap, can be solved by having a counter in every webhook payload. It will increment each time. When you receive a webhook and the counter does not match, the consumer can fetch the missing data from the events API.
I agree with the author that providers simply saying "at least once delivery" is insufficient. they should have solutions that does not require an architecture diagram.
Bootstrap is better served with a bulk events API so you don't make one call per request. It can have an after/cursor pagination. Solutions that work for our internal Kafka might not be suited to work across services, over the internet.