Streamlining bulk email sending with custom queue management using Mailgun’s API

The challenges of sending bulk emails

Rate limits and throttling set by email service providers (ESPs) tend to be the most common culprits behind bulk email deliverability issues. Beyond provider-imposed thresholds, your sender reputation score also plays an important role. Launching a mass email campaign without verifying your score and understanding ESP-specific limits can quickly degrade your reputation.

A declining sender score creates a negative feedback loop, where each bounce or spam folder placement lowers your score, further harming deliverability. And as your score decreases, ESPs are more likely to flag or blocklist your domain(s) or IP.

While this may not be a concern for internal notification systems, deliverability delays (and how you manage them) can disrupt nearly any bulk email use case. Network congestion, aggressive spam filters, and server issues can slow delivery, which is especially problematic for time-sensitive alerts. These delays can also trigger duplicate emails when systems automatically resend undelivered messages, increasing the risk of spam folder placement and further damaging your sender reputation.

Thankfully, you can address many of these issues by enacting more control over how and when your bulk email messages are sent. Rather than blasting your messages randomly, you need a smarter approach that considers key variables. That's where queues come in.

How queues can help

If you're worried about your sender reputation, queues won't directly or immediately amend it, but they can help improve it.

Queues are data structures that allow you to programmatically sort and categorize your messages and then process them in succession. This enables you to coordinate and control the sending rates of your bulk email campaigns.

For instance, you could split and group your bulk emails by mailbox provider, establish a separate queue for each group, and then set rules for how your queues are processed. This can be useful when you need to schedule emails during off-peak hours to increase deliverability.

Queues can also aid in load balancing since they enable you to implement short intervals or pauses between processing each group of emails. You can also temporarily store outgoing messages, giving the server time to process each load.

Queuing systems also make handling undelivered emails more efficient. You can set up smart resend strategies to retry messages that failed due to network or server errors while skipping those with invalid addresses or full inboxes. You can also use your queues to set up a bounce list to filter out unreliable recipient addresses.

Beyond that, queues help optimize bulk email campaigns by simplifying performance monitoring. Because they allow you to control the flow of your outgoing emails, they make it easier to spot bottlenecks, refine your strategy, and troubleshoot issues.

Streamlining bulk email sending with queues and Mailgun

For this guide, let's imagine that you're building a workforce notification service for a large global corporation with staff and branches in different regions all over the world. Some messages will be company-wide, while others will be location- or department-specific.

Select your queuing solution

The first thing you need to do is figure out what tools and components you'll need for this project. You have the following options for queue management:

• RabbitMQ is an open source message broker that has flexible routing and multiple protocols. Thanks to a thriving community, it features client libraries and developer tools for Java, .NET, Erlang, Python, PHP, JavaScript, and Go.

• Amazon SQS is a great option if you're already embedded in the AWS ecosystem or want a fully managed solution. Its biggest selling points are its support for distributed queuing systems and its scalability.

• Apache Kafka is a multipurpose, open source, distributed data streaming and processing platform that can be used as a message broker. Because it was created to handle large volumes of real-time data, it offers superior throughput to most traditional queuing or message brokers. However, configuring and managing Kafka can be challenging due to its complexity.

• Redpanda is a simplified and less resource-intensive alternative to Kafka. It's compatible with Kafka's APIs and supports many of the same implementations, including message brokering.

Since this could be a fairly large project, a managed solution with automatic scaling – like SQS – is preferable. However, if you have the experience and resources to self-manage, an open source solution can cut costs and offer more customization.

But instead of handling email management yourself, a smarter solution is Sinch Mailgun. It not only simplifies email delivery but also lets you create custom queues, giving you full control without the hassle of managing infrastructure.

Architecture overview

The next few sections will show you how to programmatically create custom queues to manage your bulk email campaigns. This example uses RabbitMQ for queue creation and management and the Sinch Mailgun Java SDK to send mail messages asynchronously. Because RabbitMQ and Sinch Mailgun support a wide variety of programming languages and frameworks, you can adapt this example to whatever programming language you prefer.

Along with the HTTP API and the Java SDK, Sinch Mailgun offers language-specific software development kits for Go, Node.js, PHP, and Ruby.

You can use the following architecture diagram (and its subsequent explanation) to better understand the queuing and bulk mailing system's structure and functionality:

This system uses a Sinch Mailgun API (MailgunMailingListApi) to fetch all the mailing lists connected to your account. Mailing lists can be created and populated programmatically from a user registration client/application or manually using the Sinch Mailgun dashboard. Mailing lists allow you to organize and manage your email recipients in a structured manner, and they serve as your system's data source.

Once the system has fetched all mailing lists, a RabbitMQ queue is created for each. The queues are populated with the email contents as well as the name and address of each recipient (member) who will receive the email.

A separate module, containing scheduled executors for each queue, listens for new items on queues at specific intervals. If any of the executors discover new items on the queue, it will begin consuming and processing the queue. In the final step, the system's email processor uses MailgunMessagesApi to send the message.

You can increase the robustness of the system by adding a queue or database to handle messages that may have failed to send and archive successful ones.

Create your queues

To create a queue, you need to configure and run an instance of RabbitMQ using Docker. Once that's done, you can create a new Java project for queue creation. Make sure you add the following libraries to your project's list of dependencies:

• The RabbitMQ amqp-client

• Mailgun

• The Simple Logging Facade for Java

You'll also need to add the following imports to your main class:

Your main method will be populated with code to initialize the Sinch Mailgun Mailing List API, initialize a connection to RabbitMQ, and create a queue for each mailing list:

Once the necessary connections are established, the code generates a queue for each mailing list using its name. A for-each loop fetches every member (MailingListMember) of the corresponding mailing list. Then, the code uses each member's details (address and name) along with the email message's details (subject and message contents) to create and publish a queue entry.

At this point, you could add code to create additional queues for message/subject types. For instance, you could create separate queues for newsletters, notifications, alerts, and other important updates. This would give you more granular control over how your messages are prioritized and sent. However, for simplicity's sake, you can skip this step.

Once the code runs (successfully), you should be able to view and manage your queues from the RabbitMQ dashboard:

Now, you need a client to consume the queues and send your messages.

Consume your queues and send your messages

Just like the previous module, this module requires you to add the RabbitMQ amqp-client, SLF4J, and Mailgun libraries as dependencies. You'll also need to add the latest Feign Core library and the following imports to your new module's main class:

Next, you need to consider how you want your queues to be processed and establish the necessary rules. For instance, if you want each queue to be processed at different hourly intervals, you'll need to set up and create a ScheduledExecutorService:

Remember to add the above code to your new module's main class.

Since the example application has three queues, you'll need to instantiate a ScheduledExecutorService object with a core thread pool size of three. You can then create a task and set a delay interval for each queue.

In this example, each runnable task is named after a queue (mailing list), but you can name your tasks however you wish. The stopProcessing method shuts the scheduler down when you exit the application.

Now, you can create a method to handle your queues:

The consumeEmails method establishes a connection to your queue, parses through it, fetches the details of each entry, then passes them to sendEmail. Once the email is successfully sent, it's labeled as consumed on the queue's registry.

Let's take a closer look at the sendEmail method:

When you use the standard send method (mailgunMessagesApi.sendMessage("<SINCH_MAILGUN_DOMAIN>", message)), Mailgun places your message in a queue for processing. You can bypass the queue by sending the message asynchronously. This requires you to create an AsyncClient to bind to the Mailgun Messages API.

Once that's done, the code builds your message with the details from the RabbitMQ queue and then sends it using the Mailgun Messages API.

To run your application, you need to add a call to the startProcessing() method in your main method and some code to handle application termination:

Conclusion

In this article, you learned how queues can help manage the flow of your mass email campaigns and give you granular control over how your bulk emails are sent. When implemented through a competent queue management system, queues help you mitigate issues like rate limiting, throttling, and network congestion.

Regardless of how you implement and manage your queues, a reliable sending partner like Mailgun can help.a

As an API-first platform, Mailgun provides you with a litany of tools to not just send but programmatically scale, monitor, and track your campaigns. Check out the official Send API documentation and be sure to subscribe to our newsletter for more tutorials like this.