Split testing to validate organic traffic hypotheses

A little bit about how Viva Real and ZAP Imóveis websites run split tests to validate organic traffic hypotheses.

What is the difference between a split test to validate organic traffic hypotheses and an A/B or multivariate test?

A conventional A/B or multivariate test generally splits website traffic into its variations, based on an activation rule.

If you create an A/B that will run on every product page, the A/B will split website traffic between the control group and the B variation as users browse through the site, respecting the samples sizes you have previously set in the A/B tool used for traffic draw.

In order to validate if a hypothesis can impact organic traffic, Google (and other search engines) needs to crawl and index the variant content.

And how to enable this kind of validation?

You have to launch changes to target pages, wait for Google to reindex them and, finally, track data and results to find new learnings.

Split testing to validate organic traffic hypotheses

First of all, you have to be very careful to choose control groups and variation targets. Below, an example of a target choice recently made by the website team, in order to validate a split test that is currently being run.

  • According to Google Analytics, pages targeting the city of Campinas generates an average weekly organic traffic of 25k sessions; the weekly percentual change of organic traffic for these pages is between -1% and +10%;
  • Pages targeting the city of Osasco, 26k sessions; the weekly percentual change, between -2% and +11%;
  • Pages targeting the city of Belo Horizonte, 50k sessions; the weekly percentual change, between -2.5% and +10%;
  • Pages targeting the city of Rio de Janeiro, 60k sessions; the weekly percentual change, between -3% and +12%.

The weekly percentual change of organic traffic is very similar for each chosen target. Each one of them also presents a relevant volume of sessions, that can enable a faster validation.

The conversion rate of traffic generated by control groups and variation targets is very similar to each other as well.

After targets are chosen, you need to decide which cities will be the control groups of your experiment and which ones will present the content variants:

Instead of applying content variants to a partial volume of traffic in the variation pages targeted, content variants must be launched to the whole variation cities. This will enable Google to crawl and index content variants.

Measuring results

Before measuring the split test results, you need to be sure that Google has already crawled your content variants.

At Grupo ZAP, we use Graylog to check this data. The queries used to filter pages that Googlebot crawls are similar to the ones below.

Variation B

application:cloudflare AND ClientRequestUserAgent:/.*(G|g)ooglebot.*/ AND ClientRequestHost:"www.vivareal.com.br" AND ClientRequestURI:/.*\/minas\-gerais\/belo\-horizonte\/.*/ AND CacheResponseStatus:200

Variation C

application:cloudflare AND ClientRequestUserAgent:/.*(G|g)ooglebot.*/ AND ClientRequestHost:"www.vivareal.com.br" AND ClientRequestURI:/.*\/rj\/rio\-de\-janeiro\/.*/ AND CacheResponseStatus:200

The queries above return the following data:

Googlebot crawling data for variation B
Googlebot crawling data for variation C

You can also track organic traffic trend and crawl rate, to ensure Googlebot has crawled all of your brand new content (or most part of it) and how it is impacting the main experiment metric.

Traffic trend for control groups and variations B and C

The lines, in the chart above, represent the traffic trend for each control group or variation. The columns represent how much of content variants have already been crawled by Googlebot.

Important note: crawling data does not indicate that Google has actually reindexed the content variants, however, it is the best proxy metric to that.

Every week after the crawl rate for variation pages reaches ~80%, we use this incredible bayesian calculator to check if there is any statistical significance to the split test.

Bayesian testing calculator by Dynamic Yield

Samples data are extracted from Google Search Console, using the volume of impressions for target pages. Conversions, the metric pursued: organic traffic.

Control group impressions data
Control group impressions data
Variation B impressions data
Variation C impressions data

As soon as the experiment reaches a statistical conclusion, data from two intervals of time is observed:

  • Data before variation pages reach a crawl rate of 80%;
  • Data after variation pages reach a crawl rate of 80%.
Organic traffic for control groups and variations tested
Traffic changes for each control group and variation before and after variation pages reach a crawl rate of 80%; probability to be the best chart for control groups and winner variation between variations
  • The first control group presented an increase of 9,39% on organic traffic;
  • The second control group presented an increase of 9,35% on organic traffic;
  • Variation B, the winner variation, presented an increase of 27,05% on organic traffic;
  • Variation C presented an increase of 16,98% on organic traffic.

Updated on June 9, 2020

Below, a presentation where I talk about the difference between a split test to validate organic traffic hypotheses and an A/B or multivariate test.

If you have any questions, do not hesitate to ask in the responses section ❤️

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Non Binary Fish, by Vini B Peixoto

Non Binary Fish, by Vini B Peixoto

Sharing a little bit about my work as a product manager and also sharing thoughts about my readings, binge-watchings, politics, life, universe and everything