Should you use ChatGPT to automate your email program?

What is ChatGPT

Sometimes, it’s better to hear it straight from the horse’s mouth. To kick off this post, we asked ChatGPT to define itself.

Not bad, but let us dive in with a bit more detail. ChatGPT (Generative Pre-trained Transformer) is a chatbot developed by OpenAI launched in November 2022. It is built from OpenAI's GPT-3 family of large language models. ChatGPT is unique because of the way it uses reinforcement learning from human feedback. This allows it to be more conversational and more “human” in its language sequence patterns, making it possible to influence tone and style more effectively:

Before we jump into concepts like reinforcement learning, cement in your mind the idea that ChatGPT is a language learning model. AI is the umbrella term describing all generative and predictive machine learning tools. Language models specifically are trained on statistics and use machine learning algorithms, probabilities, and sequences to predict words and responses.

This, plus the internal knowledge base it was trained on, is how ChatGPT can provide believable answers to questions without connecting to an internet search engine.

What ChatGPT is not

For our last bit of housekeeping (and then we promise we’ll start dissecting) here are a few of ChatGPT’s limitations. The most important takeaway? ChatGPT is a predictive language model, not an infallible knowledge source.

• ChatGPT is not currently connected to the internet in terms of retrieving information. In fact, all the information it was trained on is from 2021 and prior.

• ChatGPT is not always accurate.

• ChatGPT is not always objective and may produce output with a topic bias.

How does ChatGPT work?

ChatGPT works by predicting responses based on the statistical structure of language. It is the latest language model from OpenAI and represents a significant improvement over its predecessor GPT-3.

In this section on the nuts and bolts of how ChatGPT works, our source information comes from the Training language models to follow instructions with human feedback report that was created for InstructGPT – developed between GPT-3 and ChatGPT – which OpenAI touts as the sibling model to ChatGPT. This report is the most comprehensive document currently available from OpenAI that applies to ChatGPT.

Reinforcement learning from human feedback

Reinforcement learning from human feedback (RLHF) uses humans, referred to as “labelers”, to align the language learning model to user intent. Adding in a human control element generates a more conversational language structure and helps create comparison data to train the language model to better handle these common pitfalls:

• Language models are often not helpful.

• Because they predict text, these models may “make up” facts because the predicted words fit with their understanding of the language structure.

• Responses may be unable to be interpreted.

• Language models may generate output with topic bias based on their datasets.

Here’s how it works:

Humans are hired to be labelers, usually freelancers or subcontractors from sites like Upwork, and they collect human preference data based on OpenAI’s user alignment criteria. The goal is to hire labelers that are empathetic towards different cultures and demographics. Think of human labelers as a jury of peers that evaluates ChatGPT’s output. Labelers interact with the language model at various stages throughout its training and development.

• Phase 1: A pre-trained language model is supplied with a controlled subset of data to learn a supervised policy, the SFT model, which generates output from a set list of prompts.

• Phase 2: The language model mimics human preference based on review performed by the human labelers. The labelers vote on the best output. What results is a new dataset of comparison data called the reward model.

• Phase 3: The reward model is used to improve the original SFT model, and that model is called the policy model.

In practice this looks like this:

OpenAI 2022, Training language models to follow instructions with human feedback - Figure 2

Three AI laws to know before using ChatGPT with your email program

AI isn’t a good thing or a bad thing, it’s just a thing. We’re gearing up to talk about using ChatGPT with your email program. Here’s the knowledge you need to keep top of mind:

1. AI is not a replacement for human creativity. In fact, AI often makes mistakes and is capable of generating content with a bias.

2. AI generates responses only when directed. To integrate with AI you have to create a flow that prompts the tool (ChatGPT) to create a response.

3. AI behaves based on data it is trained on. For the moment, ChatGPT can regenerate responses based on your feedback within its current knowledge bank. It may learn universally from data you input, but only if OpenAI implements that data on the backend.

AI can be used to automate tasks. Automations can trigger AI tools like ChatGPT to create text, but it doesn’t perform autonomously.

How can you use ChatGPT to automate your email program

We asked ChatGPT to write a paragraph telling us how we could use it with Mailgun to automate email sending. This is what it said:

We’re not going to lie, this sounds legit. Not only did it give us the general process for sending an email via API, it told us where and how to use ChatGPT to generate the copy for the emails, and, when prompted, creates email copy like this:

But Step 4 is where it starts to get vague: “You may also need to set up webhooks in order to receive any incoming email and trigger the chatGPT to respond accordingly.”

There are a couple possible ways to integrate ChatGPT with a tool like Mailgun (or any cloud-based tool for that matter). Notably, Zapier has built out a basic Zap for ChatGPT integrations that you can use to connect ChatGPT to other tools and platforms. You can also use webhooks to integrate.

So, we asked ChatGPT to take a stab at writing a webhook to integrate itself with Mailgun:

This code isn’t ready to go right out of the box, though. When we asked ChatGPT to generate the code, we didn’t specify language, so it appears its default is Python – at least in this instance. However, Python isn’t everyone’s go to when writing webhooks.

Our first recommendation when using ChatGPT is to be specific. ChatGPT is capable of generating code and solving mathematical equations, but defining specific parameters for ChatGPT is ridiculously important.

The next callout is to be prepared to take formatting offline. ChatGPT has a copy/paste feature for code but it either isn’t completely accurate, or it doesn’t carry over the formatting to all external readers.

We had to update the code to be bootable:

You’d need to run this generated webapp on either an actual public server or use a service that allows you to expose your application to the internet through a local host tunnel. You’d also likely need to make a few tweaks to the code depending on your tools and environment.

Ok, so where did we land? We love automating things and making processes efficient. Transactional emails that are triggered by user action (password resets, abandoned carts, etc.) are phenomenal for your email program. ChatGPT can help you build that part of your email program from copy to code if you use it as a draft generator and ditch any expectations you have of perfection on the first iteration. We are not at the point yet where AI tools can absorb the complete creation of these processes.

Should you use ChatGPT to automate your email program? 

If this were just a matter of saving time, it would be very easy to recommend integrating a tool like ChatGPT into your email program. But, in the sage wisdom of Vanilla Ice, we can only recommend you “stop, collaborate, and listen” because this brand-new invention isn’t perfection.

Before you fully embrace automation with ChatGPT, make sure you understand how your email program works, and the points where implementing an integration like this makes the most sense.

How using ChatGPT could affect your deliverability

We asked ChatGPT if it could explain our deliverability services – just for fun, in the style of George Carlin. Here’s what it said:

Email addresses verifications, expert support to understand what can land you in the spam folder, and analytics are all part of deliverability, but a lot is missing here (including a likeness to George Carlin). This generated pitch proves our point that deliverability is not only nuanced but varies from sender to sender.

ChatGPT uses RLHF to generate its output and human labelers are tasked with being objective in terms of cultural biases, but the labelers are not tasked (as far as we can tell) with training ChatGPT in depth on different industries, demographics, or communities.

With deliverability, step one is knowing your audience and working to relate to them specifically – even something as simple as the length of the email. If your demographic is mostly Gen Z, then four or five paragraphs of text won’t engage them, which is what ChatGPT will usually produce if you don’t give it a parameter for length.

The same goes for writing code, or error troubleshooting. As our deliverability team tested ChatGPT for various use cases, they found that a lot of review is necessary. This is very obvious in code generation. If we implement scripts from ChatGPT and then try to troubleshoot the subsequent errors, it goes a little something like this:

In general, calling ChatGPT out on errors doesn’t elicit a corrected response, it just reframes the response to restate the error with the information you provided. The takeaway? ChatGPT is a generative response tool, but it doesn’t quite have the ability to troubleshoot, correct, or validate its own responses.

AI and the inbox

As tools like ChatGPT hit the market, and more and more companies use them to generate content like email copy, is there a risk that ISPs will start to crack down on recognizing AI content as spam? Is this even possible? Instead of asking for ChatGPT’s thoughts on this, we talked to Mailgun’s Product Manager, Chris Farmer.

Most mailbox providers rely on user input to identify what is or is not spam content. Users interact with what’s relevant to them, based on what they’ve asked for. So, if ChatGPT is used to generate content for a marketing campaign, and that content contains spammy keywords, content that other users have marked as spam, or content that is not relevant to users based on what they subscribed to receive, then the AI content could easily be marked spam.

In terms of using ChatGPT with your email program, we can see use cases where you could create copy and integrations like:

• Generating email copy for transactional emails or campaigns

• Writing a program to send email through Mailgun

• Writing a program to automatically reply to emails based on an existing integration with your application, such as importing calendar schedules and resource availability to respond to incoming inquiries

But ChatGPT is not a set-it-and-forget-it tool. Anything it creates will need to be edited and adapted.

Final thoughts

Even while writing this post, there have been changes in the world of chat AI tools. Notably Microsoft, who is integrating their Bing search engine and Edge browser with ChatGPT features, and Google who is rapidly developing their own solution called Bard. The results on both fronts are intriguing but definitely not sentient.

On the email front, we’re watching intently with a fresh bowl of popcorn (and one hand on the power cord... just in case), but we aren’t surrendering all our tasks to AI just yet.

We hope you enjoyed this post, but it was just a warm-up. Check out our upcoming webinars and choose your fighter. Marketers vs. Bots, and Email Geeks vs. ChatGPT.