Design a web hook service

Posted on | In | Comments:

Description

Compared to traditional API services which is pull based, a push based webhook service stands out in many ways:

  • Performance: In real time scenarios (for example, you want to receive real time update of information, metrics or alerts), polling(in an interval) is not efficient.
    • Polling too often may waste resources, polling too late may not process the data in a timely manner.
  • User experience: Instead of working around the API to pull data, it’s much easier to just handle the data pushed to you based on defined schema.
    Only process when data is available.

Many product provide web hooks to strengthen their features like SendGrid webhook, Sparkpost web hook.

Design

To have a web hook service working, 3 problems need to be engineered.

Setup subscription

Allow clients to subscribe to the web hook, an authentication method preferred. Use OAuth 2 or built your own

Event Categorization

Make it capable for clients to subscribe to different kinds of event categories.

At the minimum, clients need to provide the following informations to subscribe to the web hook:

  • callback_url: String, For where the web hook send events to
  • event categories: List, For what events web hook should send, eg: unsubscribe events.
  • requester: String, For security check.
  • status: String, Status of the subscription, subscribe, suspend, unsubscribe

Sending events

Once a subscription is validated, you should send events as soon as them arrive.

There are different triggering mechanisms for a web hook:

  • realtime: Always on sending action.
  • scheduled: Useful if only hourly/daily/monthly data required

Event schema

For events sending, we also needs to design a universal event schema.

Events Delivery Guarantee

Client needs to send a response to acknowledge they received the events.

If web hook failed to send events or client replied back with a non success response, we should resent events.

Back pressure in place in case client’s response rate is lag behind the web hook publish rate

Manage Web hook status

Since web hook is push based, you should always keep your clients up-to-date the current health status of the web hook.

Implementation

This implementation is based on the requirements of serving push campaign metrics, which will show up in push management dashboard

TODO

References

https://mandrill.zendesk.com/hc/en-us/articles/205583257-How-to-Authenticate-Webhook-Requests