Back to main menu

IT & Engineering

Stress testing HTTP with Twisted Python and Treq

Being in the API business has its challenges and maintaining the robustness of the system during peak hours is one of them. That’s why we do lots of stress testing here at Mailgun.

PUBLISHED ON

PUBLISHED ON

Being in the API business has its challenges and maintaining the robustness of the system during peak hours is one of them. That’s why we do lots of stress testing here at Mailgun.

We have tried many different approaches over time, from simple Apache bench to more complicated custom testing suites. But this post is about a “quick and dirty” yet very flexible stress testing using Python.

When it comes to writing HTTP clients in Python we are fans of the Requests library. This is what we recommend to our API users. Requests is great, but it has one weakness: It’s a blocking one-call-per-thread affair: it’s hard or impossible to generate tens of thousands of requests quickly with it.

Introducing Treq on Twisted

To solve this problem we looked at Treq (Github repository). Treq is an HTTP client library inspired by Requests, but it runs on Twisted and it possesses the typical Twisted powers: it is asynchronous and highly concurrent when it comes to network I/O.

Treq is not specific to stress testing at all: it’s a great tool for writing highly concurrent HTTP clients in general, like web crawlers. Treq is elegant, simple to use and powerful. Here’s an example:

The simple testing script

Below is a simple script which uses Treq to bombard a single URL with maximum possible number of requests.

The output:

The “Generated” ones are the requests that have been prepared, but the Twisted reactor has not sent them yet. This script ignores all errors for simplicity, adding the stats for timeouts is left as an exercise for the reader.

The script can be used as a starting point and improved and extended with your own custom application-specific logic. One suggested improvement would be to use collections.Counter instead of the ugly globals. The script runs on a single thread, and to squeeze the maximum number of requests from a machine something like mulitprocessing can be used.

Happy stress testing!

Cheers, Mailgunners

Sign Up

It's easy to get started. And it's free.

See what you can accomplish with the world’s best email delivery platform.

Related readings

Send email using Python3 and the Mailgun API

If you’re looking to streamline your approach to programmatically sending email, you’re in good hands. Our developers have the experience you need to get started. Read on as we...

Read more

How to send transactional email in a NodeJS app using the Mailgun API

Sending transactional emails is easy regardless of your tools. If you use a NodeJS helper library, this walkthrough will help you get set up in Mailgun. Read more...

Read more

An expanded Mailgun product suite to transform email deliverability

Today marks a special day for Sinch Mailgun. For over a decade, our focus has been to provide the best email experience for businesses all around the world. Now, we take...

Read more

Popular posts

Email inbox.

Build Laravel 10 email authentication with Mailgun and Digital Ocean

When it was first released, Laravel version 5.7 added a new capability to verify user’s emails. If you’ve ever run php artisan make:auth within a Laravel app you’ll know the...

Read more

Mailgun statistics.

Sending email using the Mailgun PHP API

It’s been a while since the Mailgun PHP SDK came around, and we’ve seen lots of changes: new functionalities, new integrations built on top, new API endpoints…yet the core of PHP...

Read more

Statistics on deliverability.

Here’s everything you need to know about DNS blocklists

The word “blocklist” can almost seem like something out of a movie – a little dramatic, silly, and a little unreal. Unfortunately, in the real world, blocklists are definitely something you...

Read more

See what you can accomplish with the world's best email delivery platform. It's easy to get started.Let's get sending
CTA icon