When More Is Better
When thinking of a direct response test, the first thing that comes to mind for many of us is a test within a single direct mail, email or paid digital effort. But tests are designed to answer specific questions and not everything we want to “ask” can be answered with the results of a single campaign. Sometimes we have questions with a scope that extends across multiple campaigns or even across our entire program. These are the types of questions that often require a test across more than one campaign to answer.
Since multi-campaign tests might be new territory for some, we’re here to answer your burning questions about when, why and how to set these up!
Image Source: https://www.freepik.com/vectors/banner Banner vector created by katemangostar - www.freepik.com
What are the types of tests that span across multiple campaigns and why?
Sequence Tests – These are tests of a new approach (creative, messaging, offer, ask string, etc.) throughout a set of campaigns. These may be conducted across a series of connected campaigns or across multiple unconnected campaigns. Example testing questions:
o Will a new campaign name for the two-part matching gift campaign series drive stronger response?
o Does an upgraded ask string throughout the Q4 timeframe drive stronger giving without detrimentally impacting response?
o How does adding new event branding within the spring campaigns impact performance?
Cadence Tests – These are tests where you alter the order of campaigns; eliminate or add campaigns within a specific timeframe; or change the date of two or more campaigns across a span of time. Example testing questions:
o Does an additional touchpoint over the summer drive more net revenue from campaigns during that season?
o Does leading with a year-end-themed campaign followed by a holiday-themed campaign, or vice versa, drive more Q4 revenue?
o Does expanding the time between each drop in the three-part Q1 renewal series generate stronger response across all campaigns in the series?
Long-Term Tests – These tests encapsulate the two types of tests above, but these are tests you plan to conduct across several months or even a full year (or more). These tests measure the holistic program impact of making across-the-board shifts in your campaigns. Example testing questions:
o Will retention be detrimentally impacted with fewer annual campaigns?
o Will retention and net revenue be impacted by a fundamental messaging shift across the program?
o Will retention increase while program net revenue decreases with a wholesale switch from a CRE to a BRE?
Are there any special data selection considerations for multi-campaign tests?
Ensure you maintain the same test and control audiences throughout the test for an accurate read.
Determine all special conditions for records which would need to “fall out” of the control and test panels in advance of launching the test. For example, if your test is only to active donors, donors should likely fall out of the test and control panels as they lapse.
Test and control panel sizes can often be smaller for multi-campaign tests, especially long-terms tests. Since audience-level analysis views should aggregate quantity across all campaigns in the test, you can achieve a statistically valid read with fewer test records per campaign. This also offers the advantage of conserving test bandwidth so you can simultaneously be conducting other testing as your multi-campaign test runs.
Are there any special analysis considerations for multi-campaign tests?
Prepare for a “slow burn” analysis. The results should ideally not be read and acted on after just one campaign. You can and should monitor ongoing performance of these tests as results come in, but you should be cautious about when you form insights you can “take to the bank.” If the test spans two campaigns, results from just the first campaign may be very telling, depending on the test. But if the test spans 12 campaigns, there will be no holistic insights from just the first campaign.
Make sure you’re looking at audience-level insights across the full set of campaigns – so the aggregated performance of the test and control audiences – and not simply panel-level insights campaign by campaign. There is value in looking at performance both ways, but often the main through-line insight comes from the aggregate audience analysis.
In the case of a long-term test, especially one that may have the potential to boost one KPI at the expense of depressing another (e.g., a test that lowers investment but has the potential to depress net revenue), you’ll want to monitor for detrimental effects as the test progresses and, if these seem to be notable, weigh the cost-benefit of continuing with the test vs. stopping early.
How do you determine if a long-term test is needed?
You need a long-term test if:
1. You are planning to roll out the winner across all/most campaigns AND
2. There is the possibility that repeated exposure to the test element(s) over the course of several months or more will change donor behavior in a way that may not be apparent in the results of a single test (or even a few tests)
Another way to determine if you need a long-term test is to look at the KPIs you reference as you form your testing questions/hypotheses. If you’re referring to “retention” and “donor value” more than “response rate” and “average gift,” or if you’re talking about program-level (vs. campaign-level) investment and net revenue, this is likely steering you towards the need for a long-term test.
Would your program benefit from a multi-campaign – or even long-term – test? Do you have questions you’ve wanted to “ask” but didn’t think a one-time test could answer? Consider this as you start planning the next tests for your program!
In Part 3 of our Testing Series, we’ll share tips and tricks for where to turn for inspiration when you feel like you’re out of testing ideas.