Back to main menu


Mailgun’s active defense against Log4j

Mailgun's response to the vulnerability, dubbed Log4Shell which left assets vulnerable to a (ridiculously) simple exploit that led to Remote Code Execution (RCE).



On December 10, 2021 the world was rocked by a huge vulnerability that left millions of IT and security professionals scrambling. The vulnerability, dubbed Log4Shell, left assets vulnerable to a (ridiculously) simple exploit that led to Remote Code Execution (RCE). This blog post isn’t a technical article about the vulnerability itself– there are many great resources for that, just a Google search away. Instead, we want to share how Mailgun treated the vulnerability as an immediate incident and pivot into an active defense model within hours of recognizing our own exposure... 


As soon as we recognized the impact of this vulnerability, we mobilized our in-house incident response (IR) team. We generally handle new vulnerabilities through our standard vulnerability management process, which would have seen a high level of Common Vulnerabilities and Exposures (CVEs) patched within seven days or critical vulnerabilities patched within 48 hours. However, this was no “normal” vulnerability! 

Within two hours of notification, our IR team had reviewed our asset inventory and software bill of material. We found that no public assets for Mailgun were affected by this vulnerability. However, there were multiple internal assets that did have some aspects of the impacted software present. Some of these assets were protected due to the use of Java Security Manager. 

Another often missed component of such a critical vulnerability response is to understand your third-party exposure. During our initial identification phase, our compliance team kicked off a vendor review. Contacting all of your vendors is not a simple process, but is critical to understanding whether some of your third parties need to have limitations imposed against their access to data. 

After this initial review, the incident response team moved into a remediation and active defense phase. 


Again, within hours of understanding our exposure, the engineering teams at Mailgun had rolled out the first patches for this vulnerability. Due to the critical nature of some of these assets, we had to work through a standard patching process, hitting our development and staging environments first. However, it is worth noting how critical those environments are. Many vulnerability management programs miss these “non-critical” environments in their efforts to achieve an agile stance. Attackers don’t care– they will still compromise those environments and abuse them to hurt you! 

Unfortunately, over the course of the next several days, more CVEs and exploits were let loose into the world, and we had to continually re-evaluate our assets and deploy new patches or mitigations. Our engineering teams were up to the challenge, though, and continued patching and reviewing reports throughout the following days. 

After the patching effort had begun and our engineering teams were engaged, the security team moved to active defense. 

Active Defense 

The next major phase in combating such a critical CVE is engaging in active defense. We consider active defense to consist of a few elements: preventions, detections, and honeypots. Luckily Mailgun’s chosen Web Application Firewall (WAF) began deploying detections for this exploit on day one. However, any security practitioner worth their salt will recognize that this is, and was not, good enough. The number of permutations against this vulnerability became nearly impossible to keep up with… 

This is where good logging and writing solid detection logic becomes critical in an incident. Mailgun invests heavily in logging requests sent to our infrastructure, and we were able to see just how targeted this vulnerability became. Overnight, we saw the string `${` skyrocket in our logs from attempts to exploit this vulnerability. Due to the scale of the attack, we recognized that basic IP blocking would not work. Coupled with the attack permutation and barrage of new CVEs, we recognized that we had to throw our efforts into completely mitigating this attack path and deploying ways to cut through the noise. On to the honeypots… 

The next stage in a mature active defense is deploying honeypots. The security community at large came through swimmingly here. Within days of the reported vulnerability, there were numerous published honeypots that could be deployed in minutes. After doing our due diligence against several of these, we chose to deploy a small bash monitoring script that could turn a server into a listening honeypot. Deployed within our internal networks and open to nearly all (inbound) servers in the environment, we would very quickly recognize if a bad actor was working inside of our private networks. 


At Mailgun, we embrace the idea of “when, not if”. We recognize that with this particular vulnerability, we were lucky not to have any exposed external assets or assets that would have been compromised through this exploit without internal access. Tomorrow, we may not be so lucky. But we can continue to prepare and work towards building more secure systems and better capabilities. 

Learn about our security commitment

Mailgun's security commitment

Security has been one of Mailgun’s highest priorities since our founding, and to this day, security informs every design decision in our product. Great email sending starts with top-notch security, and our commitment to your data privacy goes beyond simple encryption.

Related readings

The basics of SPF records

SPF doesn’t refer to how long you can wait before you have to reapply your sunblock (seriously, go get some). Instead, the term “SPF” refers to a security measure that helps...

Read more

The golden age of scammers: AI-powered phishing

Long live the prince of Nigeria, he had a good run. Gone is the age where scammers wield the same mediocre power as a snake oil salesman, reliant on their own persuasion and...

Read more

What are SYN flood attacks and how can you defend against them?

“We’re under attack!” It’s a line that could very well be taken directly from Star Wars or The Matrix, but it’s also a cyber security reality. These attacks are not only sneaky but can be...

Read more

Popular posts

Email inbox.

Build Laravel 10 email authentication with Mailgun and Digital Ocean

When it was first released, Laravel version 5.7 added a new capability to verify user’s emails. If you’ve ever run php artisan make:auth within a Laravel app you’ll know the...

Read more

Mailgun statistics.

Sending email using the Mailgun PHP API

It’s been a while since the Mailgun PHP SDK came around, and we’ve seen lots of changes: new functionalities, new integrations built on top, new API endpoints…yet the core of PHP...

Read more

Statistics on deliverability.

Here’s everything you need to know about DNS blocklists

The word “blocklist” can almost seem like something out of a movie – a little dramatic, silly, and a little unreal. Unfortunately, in the real world, blocklists are definitely something you...

Read more

See what you can accomplish with the world's best email delivery platform. It's easy to get started.Let's get sending
CTA icon