99.99%

Your Guide To Webhooks

Introduction

When developers create apps that involve sending email, one of the most important decisions they have to make is how to track each message and respond to issues to maintain their sender reputation and deliverability rates. Webhooks are an extremely flexible way for developers to monitor the health of their email messaging in real time, analyze deliverability data and program apps to handle unsubscribes, spam reports and bounces instantly. This white paper will introduce webhooks, show what they can do for email senders and provide detailed instructions for deploying them.

What Are Webhooks And Why Use Them?

WHAT ARE WEBHOOKS AND WHY USE THEM?
Webhooks are user-defined HTTP callbacks typically triggered by an event, such as a successful email delivery or bounce notification. When the event occurs, the source site makes an HTTP POST request to a URI the developer has configured to receive the webhook. Users can configure them to cause events on one site to invoke behavior on another. The best use for email webhooks is to get information about how effectively messages are delivered to recipients, and once they receive it, how favorably they respond via opens and clicks.

Webhooks are useful for many reasons. First, they’re far more flexible than the typical ESP dashboard. Users aren’t limited to the reports and analyses offered by their ESP. Instead, developers receive data directly and can report it, analyze it, respond to it – whatever they want. Second, webhooks are more efficient than API calls. Think of the difference between a phone call and a text. When a user sends a call to an API, they’re essentially asking their ESP for data. But a webhook is more like a text message that the system sends automatically as soon as a triggering event occurs. Since webhooks are based on HTTP POST, they are easy to use. What’s more, hook scripts can be written in just about any scripting language developers prefer, including curl, Ruby, Python, PHP, Java, C# and Go. Once the data has been captured, it can be stored in a database and used to gauge the effectiveness of email campaigns or augment recipient profiles.

Let’s start by exploring the POST request, which can be encoded as application/x-www-form-urlencoded for most messages, and multipart/form-data if there’s an attachment included. The POST request method is designed to request that a web server accept the data enclosed in the request message’s body for storage.

Here’s an example of an HTTP POST made by Mailgun to a URI at Runescope. Note that the Content Type header is set as “application/x-www-form-urlencoded”

ACCEPT: */*
ACCEPT-ENCODING: GZIP
CONNECTION: CLOSE
CONTENT-LENGTH: 1325
CONTENT-TYPE: APPLICATION/X-WWW-FORM-URLENCODED
HOST:
USER-AGENT: MAILGUN/TREQ-0.2.

The body of the message contains parameters stored as key-value pairs. (We’ll go into greater detail about the data that is posted in the section below.) Since the data is encoded, it will simply appear as gibberish. Here’s what the decoded body might look like:

DOMAIN: BEATNIKZ.NET
EVENT: DELIVERED
MESSAGE-HEADERS: [["RECEIVED", "BY LUNA.MAILGUN.NET WITH HTTP; WED, 07 JAN 2015 00:44:03 +0000"], ["MIME-VERSION", "1.0"], ["CONTENT-TYPE", ["TEXT/PLAIN", {"CHARSET": "ASCII"}]], ["SUBJECT", "HELLO"], ["FROM", "TAG TEST <NOLAN@YBEATNIKZ.NET>"], ["TO", "MGBOX01@GMAIL.COM"], ["X-MAILGUN-TAG", "SEPTEMBER NEWSLETTER"], ["X-MAILGUN-TAG", "NEWSLETTERS"], ["MESSAGE-ID", "<20150107004403.125880.28353@BEATNIKZ.NET>"], ["X-MAILGUN-SID", "WYI3NGU3NYISICJTZ2JVEDAXQGDTYWLSLMNVBSISICI0MGRKIL0="], ["DATE", "WED, 07 JAN 2015 00:44:11 +0000"], ["SENDER", "NOLAN=YBEATNIKZ.NET@BEATNIKZ.NET"], ["CONTENT-TRANSFER-ENCODING", ["7BIT", {}]]]
MESSAGE-ID: <20150107004403.125880.28353@BEATNIKZ.NET>
RECIPIENT: MGBOX01@GMAIL.COM
SIGNATURE: EB9FE5C673522299A2259052E56487E54F4D2486A0F1582E91D2C17114A6398
TIMESTAMP: 1420591452
TOKEN: E40542A95B5A6989B5E226CC0E9BB2F120478AF8EE594F884C
X-MAILGUN-SID: WYI3NGU3NYISICJTZ2JVEDAXQGDTYWLSLMNVBSISICI0MGRKIL0=
X-MAILGUN-TAG: SEPTEMBER NEWSLETTER
X-MAILGUN-TAG: NEWSLETTERS

Events And Parameters

As explained above, webhooks are triggered by specific events. In the email realm, these events include opens, clicks, unsubscribe requests and other events resulting from attempted or successful email delivery. Here’s a complete breakdown:

Open: This event occurs every time a recipient opens a message. Open tracking is enabled by using the O:TRACKING or O:TRACKING-OPENS parameters when sending a message.
Click: This event tracks every time a recipient clicks on links in an email message. Enable click tracking by using the O:TRACKING or O:TRACKING-CLICKS parameters when sending a message. As with opens, the appropriate CNAME records must be included in the user’s DNS.
Unsubscribe: This event occurs when a recipient clicks on the “unsubscribe” link in a message.
Spam Complaint: Not every ISP supports Feedback Loop (“FBL”) notifications for spam complaints, but developers should make sure that they get data from all of the ones that do.
Bounce: An email message is said to “bounce” if it is rejected by the recipient SMTP server. These are often classified as hard or soft bounces as follows:
Hard bounces (permanent failure): Recipient is not found, and the recipient email server specifies the recipient does not exist. The app should stop attempting delivery to invalid recipients after one hard bounce.
Soft bounces (temporary failure): Email is not delivered because of a temporary issue, such as a full inbox. Apps can programmatically respond to soft bounces by reattempting a set number of times before removing the recipient address from the list.
Failure: Failures consist of both hard bounces and soft bounces. Depending on its capabilities, an ESP may notify users through a webhook when a message is dropped (i.e., stop retries) for any of several reasons.
Delivery: A successful delivery occurs when the recipient email server responds that it has accepted the message.
Depending on the event, webhooks can deliver a variety of parameters to help identify and describe the message in question. This data can then be parsed via scripts for analysis. Common parameters include:

Event
Recipient
Sending Domain
Message Headers
Recipient Identifying Details, such as country, region, city, device, email client and OS
Depending on the ESP, other parameters may include custom variables, tags and campaign names and IDs, among others. In addition, certain events offer more detail, such as a URL clicked; a reason for/description of a negative event or special codes providing specific event details.

Securing Webhooks

A receiving URI must be public, so webhooks should be secured with a signature, time stamp and token to create a hash map using an API key to verify that the data is coming from the developer’s ESP. Users should program their application to check that hash map and compare it to that of the ESP, and then allow the post to be made only if it matches.

To verify the webhook is originating from their ESP, users should concatenate time stamp and token values, encode the resulting string with the HMAC algorithm (using the ESP’s supplied API Key as a key and SHA256 digest mode), and compare the resulting hexdigest to the signature. Optionally, users can cache the token value locally and not honor any subsequent request with the same token. This will prevent replay attacks.

Another level of security would be to programmatically check the timestamp to verify that the POST attempt has been made within a certain timeframe.

Below is a Python code sample used to verify the signature.

IMPORT HASHLIB, HMAC
DEF VERIFY(API_KEY, TOKEN, TIMESTAMP, SIGNATURE):
RETURN SIGNATURE == HMAC.NEW(
KEY=API_KEY,
MSG='{}{}'.FORMAT(TIMESTAMP, TOKEN),
DIGESTMOD=HASHLIB.SHA256).HEXDIGEST()

Steps To Using Webhooks

Now that we’ve described the uses for webhooks, let’s detail how to use them. After that, we’ll walk through a couple of useful examples.

1. Choose the desired data. The first decision developers must make as they plan out their email tracking and response is exactly what data they’ll be looking for. For example, if one is only interested in knowing when sent emails are bouncing, the user URI could perform a script on incoming POSTs to capture and save the email address in a local database. The same script could be augmented to capture recipient name, subject or any other parameter provided by the webhook.

2. Attach data to messages. When sending an email, some ESPs permit users to attach data to their messages by passing custom data to the API or SMTP endpoints. The data will be represented as a header within the email, and is typically formatted in JSON. This custom data would then be included in any webhook events related to the email containing it. Several such headers may be included and their values will be combined.

Example:

X-MAILGUN-VARIABLES: {"FIRST_NAME": "JOHN", "LAST_NAME": "SMITH"}
X-MAILGUN-VARIABLES: {"MY_MESSAGE_ID": 123}

To add this header to a message:

USING API: PASS THE FOLLOWING PARAMETER, "V:MY-CUSTOM-DATA" => "{"MY_MESSAGE_ID": 123}".
USING SMTP: ADD THE FOLLOWING HEADER TO THE EMAIL, "X-MAILGUN-VARIABLES: {"MY_MESSAGE_ID": 123}".

Sometimes it’s helpful to categorize outgoing email traffic based on some criteria, perhaps separate sign-up emails from password recovery emails or from user comments. The ESP may permit tagging each outgoing message with a custom value. The user can then access deliverability statistics aggregated by these tags. To attach a tag to a message, supply one or more O:TAG to it.

Tagging Code Sample:

CURL -S --USER 'API:YOUR_API_KEY' \

HTTPS://API.MAILGUN.NET/V3/YOUR_DOMAIN_NAME/MESSAGES \
-F FROM='SENDER BOB <SBOB@YOUR_DOMAIN_NAME>' \
-F TO='ALICE@EXAMPLE.COM' \
-F SUBJECT='HELLO' \
-F TEXT='TESTING SOME MAILGUN AWESOMNESS!' \
-F O:TAG='SEPTEMBER NEWSLETTER' \
-F O:TAG='NEWSLETTERS'

3. Set up the URI. In order to receive the data from a webhook, users must give their ESP a URI to deliver requests to. This means that they also need to set up the URI in their app, so it is accessible from the public web (hence the need for security). The ESP’s webhooks will then POST data to the URI as application/x-www-form-urlencoded or multipart/form-data.

4. Create scripts to capture data. The final step is to add scripts to the URI that capture the data provided by the webhooks, and process it in any way the developer sees fit. See the use cases below for specific examples of these scripts.

Use Case: Tracking Email Bounces

A use case for an application might be to capture an attachment and store the file locally. For example, the following code uses a combination of a microframework for Python, Flask and Requests HTTP library. Here’s a quick Flask App to capture a file from a Bounce webhook, keep the native filename and store it locally on your webserver.

FROM FLASK IMPORT FLASK
FROM FLASK IMPORT REQUEST
FROM WERKZEUG IMPORT SECURE_FILENAME
APP = FLASK(__NAME__)

@APP.ROUTE('/WEBHOOK', METHODS=['GET', 'POST'])
DEF TRACKING():
#CHECKS IF THE REQUEST IS A POST
IF REQUEST.METHOD == 'POST':
F = REQUEST.FILES['ATTACHMENT-1']
# OBTAINS THE FILESTORAGE INSTANCE FROM REQUEST
FILENAME = SECURE_FILENAME(F.FILENAME)
F.SAVE('/HOME/DIRECTORY/WEBHOOK/'+ FILENAME)
PRINT FILENAME
RETURN "OK"
IF __NAME__ == '__MAIN__':
APP.RUN(HOST='0.0.0.0', PORT=100, DEBUG=TRUE)

To see it in action, run your application and paste the URL (For example: http://yourdomainhere.com:100/webhook) into the Bounce webhook and click to “Test Webhook.” You’ll get a file named “message.mime” with …

RECEIVED: BY LUNA.MAILGUN.NET WITH SMTP MGRT 8734663311733; FRI, 03 MAY 2013 18:26:27 +0000
CONTENT-TYPE: MULTIPART/ALTERNATIVE; BOUNDARY="EB663D73AE0A4D6C9153CC0AEC8B7520"
MIME-VERSION: 1.0
SUBJECT: TEST BOUNCES WEBHOOK
FROM: BOB <BOB@AWESOME.MAILGUN.ORG>
TO: ALICE <ALICE@EXAMPLE.COM>
MESSAGE-ID: <20130503182626.18666.16540@BEATNIKZ.MAILGUN.ORG>
LIST-UNSUBSCRIBE: <MAILTO:U+NA6TMY3EGE4TGNLDMYYTQOJQMFSDEMBYME3TMY3CHA4WCNDBGAYDQYRGOI6WSZDPOVRHI5DINFZW63TFMV4GS43UOMSTIMDHNVQWS3BOMNXW2JTUHUSTEQJGMQ6TM@BEATNIKZ.MAILGUN.ORG>
X-MAILGUN-SID: WYIWNZI5MCISICJHBGLJZUBLEGFTCGXLLMNVBSISICI2IL0=
X-MAILGUN-VARIABLES: {"MY_VAR_1": "MAILGUN VARIABLE #1", "MY-VAR-2": "AWESOME"}
DATE: FRI, 03 MAY 2013 18:26:27 +0000
SENDER: BOB@AWESOME.MAILGUN.ORG
--EB663D73AE0A4D6C9153CC0AEC8B7520
MIME-VERSION: 1.0
CONTENT-TYPE: TEXT/PLAIN; CHARSET="ASCII"
CONTENT-TRANSFER-ENCODING: 7BIT

HI ALICE, DO YOU EXIST ON THIS DOMAIN?
--EB663D73AE0A4D6C9153CC0AEC8B7520
MIME-VERSION: 1.0
CONTENT-TYPE: TEXT/PLAIN; CHARSET="ASCII"
CONTENT-TRANSFER-ENCODING: 7BIT

HI ALICE, DO YOU EXIST ON THIS DOMAIN?

--EB663D73AE0A4D6C9153CC0AEC8B7520--

Conclusion

While there are several methods for accessing the data generated by email delivery, including ESP dashboards and API calls, the most flexible and efficient by far is webhooks. Rather than pulling data from their ESP, developers can receive continuous push data related to email in real time.

For more detailed information on how to use webhooks with Mailgun, check out our documentation on webhooks.

Integrate email into your app
in minutes with our API

Sign up free

Want better deliverability?
Learn more about our managed email service.

Share
Tweet
Share
Share
Tweet
Share