Email marketers wanted to test every version of a message, so I designed an app that also told them when a winner was real.
In 2014 I researched and designed an A/B testing app for email for a marketing language company, built to run inside a major email marketing platform. Marketers could test many versions of an email instead of two, size their test without doing the math, and see plainly when a result was ready to trust.
Client
A marketing language AI company
I changed names and details to respect my nondisclosure agreements.
My role
Product researcher and designer
Focus
A/B testing for email inside a major email marketing platform
Result
Every marketer I tested with said they would use the app
The built-in tool tested two versions of an email at a time.
Email marketers on a major email marketing platform could already split a send between two versions of an email, wait, and send the winner to the rest of the list. My client's business is the language inside those emails. A subject line can be phrased in far more than two ways, and the company wanted a free app in the platform's app marketplace that let marketers test as many versions as they liked.
More versions bring a harder problem. Every version you add needs enough people on the list for its result to mean something. The goal the company set was an app with no learning curve, so the math had to happen in the background while every decision stayed with the marketer. A marketer should never need a statistics course to know whether a result is real.
I started from the tool marketers already knew and tested my wireframes with the people who would use them.
The work moved from research to wireframes to tested wireframes to full high-fidelity screens in early 2014. Click or tap a card to turn it over.
The report says how sure the app is before anyone sends the winner to the whole list.
Every test produces a top number, and a top number does not make a winner. Sending a lucky version to thousands of people can lower a campaign's results without anyone noticing. I designed the report to say plainly, in color and in words, how much the result can be trusted.
Even a clear winner asks for one more confirmation before the email goes out to the rest of the list. That small pause protects the marketer and the people on the list.
- Green means a clear winner at 95% confidence, with a recommendation to send the winner to the rest of the list.
- Amber means a leader at only 90% confidence, with a link that explains what to do next.
- Red means the app does not have enough responses yet to name a winner.
- Gray means the winner was sent, with a link to the send in the email platform.
Click through the app the way a marketer would.
I rebuilt the app as a working prototype from my original 2014 screens in a current style, under the placeholder name Smith Messaging, with AI writing the code from my designs. The data is sample data, and the statistics are calculated live.
Things to try
- Choose View the report, then switch between the four confidence states.
- Choose Confirm before sending and answer the dialog to send the winner.
- Choose New test and press Continue without a title to see the error state.
- On the Sample size step, lower the sample to see the caution carry through to Review.
- On the Versions step, add a version and watch the app warn you when the sample gets too small.
A small set of colors and parts did all the talking in this app.
I carried the parts from my 2014 comps into a current product style: square corners, hairline borders, one violet for action, signal lime as the brand accent, and status colors that always come with words.
Color
- Action violet
#5B3DF5Primary buttons, the current step, the winner's rate bar, and every finished or successful state, always paired with a check icon and a word. - Signal lime
#C9F27ABrand accent used as a fill with ink text, such as the Winner tag, the remainder in the sample bar, and the wordmark square. It is never used for text on white. - Ink
#0B0B12Body text, headings, the dark side navigation, and the selected state in segmented controls. - Caution amber
#F5B83DResults and steps that can run but deserve a second look, always with ink text and the word Caution. - Error red
#B92D45Steps that need a fix and results with too little data, always with an icon and words. - App background
#F7F7F8The cool gray behind every white panel, table, and card. - Hairline
#E5E5E9The 1px borders that separate panels, rows, and table cells.
Type
Page titles
Sample size
Inter
Body and labels
Number of email versions to test
Inter
Labels, numbers, and IDs
New test · Step 3 of 6
JetBrains Mono
Components
Step tabs with status
Six flat tabs sit on a hairline, and each one shows a square check, caution, or error mark, so a marketer can see what is left to do without opening every step.
Confidence banner
A white banner with a thin colored edge names the confidence level in a mono label and in plain words, so the trust question is answered before anyone reads a table.
Sample size bar
A single square bar splits the list into one violet block per version plus the remainder in lime, which turns the sample size math into a picture.
Review table with Fix links
Every part of the test gets one row and a status mark, and each Fix link goes straight to the step that needs the change.
Confirm before sending dialog
A square dialog repeats the version, the date, and the audience before the winner goes out, and adds a warning when the winner is not certain.
Principles
- Say how sure the app is, in words as well as color.
- Let the app do the math and leave every decision with the marketer.
- Mark problems where they live and offer a one-click way to fix them.
- Ask for confirmation before anything reaches a real inbox.
100%
of the email marketers I tested with said they would use the app
Every marketer I tested with said they would use the app.
Marketers who ran A/B tests for a living saw their own work in the flow, and the honest report gave them a reason to trust the numbers. Click or tap a card to turn it over.
I still design for honest confidence.
Telling people plainly how sure a system is, and what to do when the system is unsure, shapes how I design AI products today. An answer from an AI model deserves the same honesty as the result of an A/B test.
There is plenty more where this came from.
Names, brands, and data in my case studies are changed to protect my clients.
Tell me about your project.
Tell me a little about it on my contact form, and a real person will get back to you soon.