Back to main menu

Product

Mailgun authentication service: Post mortem July 2018

A review of the July 2018 Authentication Service downtime. Read more...

PUBLISHED ON

PUBLISHED ON

This was originally posted on July 18, 2018.

The stuff of urban legends? An uncanny coincidence? Perhaps. What we do know is that this past Friday the 13th was not a great day for us.

Some of our customers were impacted by downtime, and we took immediate action to determine the root cause. We would like to be transparent and take a moment to share the details of our findings:

This is what happened

As a part of ongoing work by our engineering teams, several of our internal and external services were updated to delegate authentication to a centralized authentication service. One of those updated services was deployed at just after 10:00 UTC.

At 11:00 UTC on Friday, July 13, Mailgun engineering began receiving alerts of problems with several services. Our initial investigation suggested that the problem was related to this software change released earlier in the day, and we initiated immediate efforts to roll back that release.

Continued investigation revealed that, despite the roll back, our authentication services were still not responding in a timely manner. Authentication (and related) services were restarted, and systems began to resume normal operations. By 12:44 UTC, all services were fully functional again.

Why did this happen? What did you do about it?

Before this release, we had deployed an unrelated set of changes to the authentication service. This introduced additional latency to the authentication flow and reduced the rate at which requests could be serviced. Combined with the additional load generated by our updated services, the queue of authentication requests grew faster than they could be serviced. Additionally, failed requests were being retried, which further compounded the load problem.

We worked to reduce the impact and took several immediate measures to restore services by:

  • reducing authentication load by reverting the most recently updated service

  • removing the circular dependency to reduce latency

  • restarting authentication services to clear request backlog

Lessons learned

Mailgun engineering has performed a comprehensive root cause analysis of this incident, and we have identified several actions we’ll be taking to reduce the likelihood of future incidents.

In addition to code and configuration changes made to remove unnecessary response latency, we are also in the process of formalizing SLOs. This will help increase our visibility into service latency and introduce more comprehensive data collection, monitoring, and alerting to aid in SLO enforcement.

We are also developing tooling to identify potential problem areas earlier in the development and release cycle in order to keep incidents like this from impacting our customers.

We really appreciate the understanding from our customers while we worked to resolve the issue quickly. We’d be happy to answer any questions or address concerns for impacted accounts – just open a support ticket, and our team will get back to you.

Related readings

The golden age of scammers: AI-powered phishing

Long live the prince of Nigeria, he had a good run. Gone is the age where scammers wield the same mediocre power as a snake oil salesman, reliant on their own persuasion and...

Read more

An expanded Mailgun product suite to transform email deliverability

Today marks a special day for Sinch Mailgun. For over a decade, our focus has been to provide the best email experience for businesses all around the world. Now, we take...

Read more

What are SYN flood attacks and how can you defend against them?

“We’re under attack!” It’s a line that could very well be taken directly from Star Wars or The Matrix, but it’s also a cyber security reality. These attacks are not only sneaky but can be...

Read more

Popular posts

Email inbox.

Build Laravel 10 email authentication with Mailgun and Digital Ocean

When it was first released, Laravel version 5.7 added a new capability to verify user’s emails. If you’ve ever run php artisan make:auth within a Laravel app you’ll know the...

Read more

Mailgun statistics.

Sending email using the Mailgun PHP API

It’s been a while since the Mailgun PHP SDK came around, and we’ve seen lots of changes: new functionalities, new integrations built on top, new API endpoints…yet the core of PHP...

Read more

Statistics on deliverability.

Here’s everything you need to know about DNS blocklists

The word “blocklist” can almost seem like something out of a movie – a little dramatic, silly, and a little unreal. Unfortunately, in the real world, blocklists are definitely something you...

Read more

See what you can accomplish with the world's best email delivery platform. It's easy to get started.Let's get sending
CTA icon