A / B testing is a type of testing that is widely used to test two options. When the A / B test becomes an A / B / C / D test then we are talking about split testing. The basis of A / B testing is to test two products or services, or ad creatives in as similar (ideally the same) conditions as possible.
If you test a candy, you present both candies to the same group of people and ask them for their opinion. The most complicated part of such a test is ensuring the same conditions.
On that example, the goal would be for the test group to try them the same day, to ensure that some previous taste does not affect the results. But still, consuming one affects the rating of the other. The solution for such a problem is that half of the test group first tests candy # 1 and then candy # 2. While the other half will start with candy # 2, then try candy # 1.
The larger the test group, the more accurate the results. And that’s it? We list what people prefer and start production, right? WRONG!
If in a group of 100 people you have a low prevalence, for example, 51/49 or 53/47 then you should expand the test group, perhaps even to limited market entry in more complicated cases.
Even when most potential consumers agree, you need to look at everything from the business and distribution side. Production costs and associated costs are also factors in the decision. How to make such decisions based on information? The answer is … by analyzing and visualizing the data.
Today we will deal with how to visualize the decision (In this case on several sets of Remarketing ads on Google Ads).
What data did we use?
Data is retrieved from Google Ads Reporting capabilities, in .csv form. If you are doing something similar, remember to remove the first two lines that contain the file title and period.
These are 3 ads sets, which started simultaneously on April 2, and one of them was paused on April 24, while the other two continued, and we use the data up to and including May 25.
The ads were created in 5 aspect radios, to increase the number of ad slots where they can appear.
All sets used the same targeting, users who were already on our site, AND showing on the portals index.hr, 24sata.hr, rtl.hr, regionalexpress.hr, slobodnadalmacija.hr, vecernji.hr, jutarnji.hr, bug.hr, poslovni.hr.
Ad Creatives (336 × 280):
Of the three ads sets, two share the same visual, with slightly different text (machine learning, artificial intelligence…,) while the third deals with application development.
When we download data from Google Ads Reporting, and fiddle with it a little regarding data types, we are left with relevant items that you will usually look at in the interface (or at least you should), but this way we can see them divided by days to help us make our decisions.
April 24 was a Friday, and one of the two visually similar ads was scheduled to be paused, so a decision had to be made. But, if someone was to raise the question of how and why this decision was made, it is always good to illustrate/prove it, to explain it to someone through an executive summary.
Relevant variables that we will refer to are:
cl – clicks
im – impressions
ct – cost
ctr – click through rate
cpc – cost per click
Less typing, more graphing!
So let’s compare them one by one, for each of the three ads:
Comparison of key metrics
This is a little messy, let’s fill it out to see more clearly what’s going on:
Click through rate:
Ok, lots of information in a couple of graphs. Then why is the ‘analiza’ paused and ‘napredovanje’ continued?
First, there had to be some, albeit arbitrary deadline, because with such a similar visual and only slightly altered text, it was to be expected that the results of the ads would be similar.
What was the biggest factor in the decision?
Before we get caught up in that, let’s just take a quick look at just those two sets, no ‘aplikacija’ set, just A / B test data.
(when the third set of ‘applications’ is included then we are already talking about the split test)
Analysis of A / B test results
With this, as with producing candy example (or anything you serve to a wider audience), clearly defined priorities are important.
As these two sets of ads went to the same audience, and except for small changes were identical, we needed to decide on priority metrics.
Clicks and impressions give a great advantage to ad set ‘analiza’. But, Nutella produced for the Italian market sells better than that for the Polish market, but if there is no Italian Nutella, people will buy Polish.
The cost is relatively dependent on the number of clicks, so we’ll focus on the last two metrics, CTR (what percentage of people who saw the ad clicked on it) and CPC (how much one click cost us on average).
Here, Ctr is clearly on the ad side of the ‘napredovanje’, while cpc is almost equal (5% difference).
Another thing that has played a role, but I won’t add it here for sake of simplicity is the bounce rate and average time on page (more on plotting those in previous articles) that give advantage to the ‘napredovanje’ ad set.
As you can see, the ad that remained quickly took over the clicks, impressions, and cost from the one that was paused, so we achieved the goal, we know which of those ideas was better. Now there is an ad for applications and an ad that remains, and we are making alternatives for both of remaining ads to continue testing.
ALWAYS BE TESTING!
Visualizations for Executive Summary - CTR
Another thing that is a little harder to look at in Google Ads when you place ads on certain portals, is the daily performance of each ad on each of the placements.
Yes, you can see the average and it will give you a “good enough” overview, but when you need to interpret it all and present it to someone, it’s faster to do it visually.
Let’s then do the following:
The primary thing we are interested in is the performance of our ads, CTR.
We want to know the CTR for each day, for each placement.
If we tried to deliver it in a tabular format, we would have a huge report that would be very difficult to read and understand.
If, on the other hand, we place placements on the X-axis of the graph, days on the Y-axis, we can draw points on the intersections when there were clicks.
his gives us information on which day there were visits from which of the placements.
To expand this a little more, the size of the point will depend on what the CTR was.
This will give us the following information:
• On which portal we appear consistently and on which we have a consistent click through rate.
• Which portal we appear only occasionally (which will mean either that there is more AdSense competition there, or that our users do not visit that portal as often).
This kind of display is useful for presenting the results because we have a lot of dimensions that are readable from one place.
That’s 4-5 metrics on the same graph, which is still intuitive.
And the best part, it can be customized to illustrate almost anything with a time dimension.
We can look at this for each of the sets. The data points are intentionally enlarged to emphasize the data but this can be presented differently depending on what you want to achieve. My data points are big because I want to get you to use your head and know what you want before you do something.
Visualizations for Executive Summary - CPC
And that’s it? Can we summarize it more and make it more pretty?
Of course we can.
If we combine both CPC as the color of the points, and CTR as their size, we can describe everything with one graph, but here we lose part of the intuitiveness of the graph. It all depends on who you present that information to.
So if we put on one graph
Point: Existence of clicks
Point color: CPC
Point size: CTR
ALWAYS BE TESTING!
Whether you’re into digital advertising, making ice cream, or wanting to optimize your sales, A / B testing, or split tests can get you the answers you’re looking for pretty quickly.
Especially if you have to explain and present that decision to someone, visualizing the results will always come across a much better understanding and comprehension than a simple table.
Python, Jupyter, and pandas offer you almost everything you need right out of the box, and then it’s just a matter of asking the right question and knowing how you want to present the answer.
If you need help with complex visualizations you can contact us via the contact page.
Less typing more graphing.