A/B Testing in Dynamics 365 Real-Time Marketing
As you have likely heard, with the merger of the Customer Insights app with Marketing, Microsoft will be shifting its focus entirely to the Real-Time Marketing module. While Outbound Marketing isn’t going away, Microsoft has recommended that users start thinking about transitioning their users to Real-Time as Outbound will no longer be receiving new updates and features. Most everything should be pretty much the same in terms of functions between what you may be used to with Outbound Marketing and Real-Time Marketing. When it comes to A/B testing, there are a couple differences that are present.
What is A/B Testing and when should you use it?
Before we dive into A/B testing in Real-Time Marketing, let’s go over the basics of what A/B testing is in the first place. At its core, A/B testing is used to test two versions of marketing content to see which performs better with your audience. By creating two versions of the same message and randomly distributing them among our audience, we can get a fair assessment of what our audience is more likely to engage with.
When creating A/B tests, we want to treat it like a science experiment. We should aim to limit our test to one thing at a time and build upon our results. We can take small, incremental steps to adapt our messages. Say for one of our email sends, we try out a different style of subject line to see if it improves open rates. If so, we can start incorporating that in the future. Then later we may want to change the overall style of our email content. That isn’t to say you can’t send two entirely different emails and see which performs better, but you want to make sure that when you are running a test, you have some sort of goal in mind and will be able to interpret the results to gain actionable insight.
Image courtesy of Microsoft.
A/B Testing in Real-Time Marketing
If you’ve done any A/B testing in the Outbound Marketing module, you’ll know that the system is primarily based around setting the tests up at the email-level since that is the primary focus of Outbound Marketing. Real-Time Marketing has a couple additional channel options, such as texting, outside of just sending emails. As such, it pulls the A/B testing back to the Journey-level. Rather than setting up an A/B test on an email and then adding the element to the customer journey, we instead will simply add an action to our journey and select “Test which variation performs better.”
Doing so will then prompt us to select what channel type will be used for versions A and B. This would allow you to experiment with implementing text or other push notifications as an alternative to email messaging. Otherwise, if looking to test variances in an email, we can select email for both versions. Clicking on the step in the journey, we’ll see the details pane on the right where we can edit the details of the test. If it is a segment-based journey where the audience is known prior to the journey starting, we’ll have the option to set up a control group. Adding in a control group will allow us to allocate a percentage of the audience to be divided between the two version—and whichever performs best within our chosen metric will be sent to the remaining members. We can also adjust the distribution of Version A and B among the chosen members, but typically leaving the distribution at 50/50 will produce the most accurate results.
Below all that, we can set the metric we’re testing for, which might be Clickthrough Rate, Open Rate, or the Journey Metric (if we have a goal set for the journey). We can change between ending the test after statistically significant results are shown, or set a specific end date/time, and we can also set the default version if there is no clear winner among the two options.
Image courtesy of Microsoft.
So, while having the A/B tests set up on the journey level gives a bit more options for types of tests, if you are only using emails, it is a little bit less tidy. Outbound marketing had the test on the email level where you could swap between versions A and B at will within the same email canvas. So you’re A/B email versions were contained to a single email within your email list.
With Real-Time, that is not the case. You’ll want to fully set up your version A email, save it, and then use the “Save As” option in the “Save” drop-down menu to create a copy.
The copy will act as Version B, where you can make whatever changes you want, and both emails can then be loaded into the Journey once they are ready and live. This will create additional emails in your email list, so we suggest practicing a naming convention that clearly identifies the email’s content and which version it is, for example:
New Years Sale - Version A
New Years Sale - Version B
A/B testing is one of those things that seems like it’s a lot of work. In reality, it’s really easy to set up and start seeing results right away. Being able to quickly and easily run tests to improve messaging is an extremely important tool for marketers. Be sure to check out all the A/B testing capabilities in Dynamics 365 Customer Insights – Journeys yourself and reach out to us at Syvantis if you have questions or would like to get a demo!