The Zebra

In order for The Zebra to help their users make the best decisions possible, the company needs to run a large number of AB tests on how they present information and guide people through the insurance shopping process. As Principal Data Scientist, Emily Bartha is responsible for ensuring that The Zebra team is making decisions based on true metric impact. 

Challenge

Prior to using Hex and Eppo, it was challenging for The Zebra to evaluate their AB tests consistently. Product managers who were running AB tests had different statistical backgrounds, tests ran different lengths of time, the split in traffic was not always even, and it was difficult to explore oddities in the results of any test.

Solution

To solve this problem, The Zebra searched for tools that would allow them to AB test with confidence without requiring a bunch of modeling.

Centralize testing on the warehouse

Bartha’s team found Eppo, which is the only warehouse-native experimentation platform, and they found Hex, a collaborative platform for data science and analytics. Eppo lets users with any technical background set up statistically significant tests, and Hex lets users of any technical ability explore data with SQL, Python, and no code visualization.

Find insights to inspire AB tests

Hex enabled The Zebra’s analytics team to dig into their web traffic data to find interesting user behaviors. Once they saw drop offs in a funnel or spikes on a particular article they were able to determine if there was an opportunity for an AB experiment to drive core business metrics.

Run statistically significant AB tests

Eppo makes it simple to target tables and columns they had identified in Hex since they both sit on top of the warehouse. Eppo ensures that every test is properly split, run for enough time, and evaluated accurately. Eppo even has a progress bar to let you know how much more data you need to hit statistical significance, automating the complex and manual power analysis process.

Deep dive into results

Eppo automates common segmentation analysis, saving the analytics team a ton of time as PMs inevitably ask for results by key user groups. And when Eppo’s reports and segmentations show a trend worth investigating even more deeply, they can dive back into Hex. Going deeper on an analysis with Hex is easy to do because Eppo allows you to copy the SQL behind each intermediate table with their copy SQL button, this can then be put and queried in a Hex SQL cell.

Hex allows their data team to combine SQL with Python to run additional statistical tests and visualize results to fully vet the data. They can then build reports with the most crucial pieces of information to share with a link back to the product managers. This was all easy, since Hex and Eppo are on the same warehouse, and they can copy over SQL queries and use the same tables for AB testing and exploratory analytics.

Results

The Zebra is now able to generate statistically significant insights regularly for product managers and their analytics team can easily dive deeper into any test. They have saved time in the setup and evaluation process for doing AB tests and they have nearly eliminated false positives in AB testing. Moreover, they found that both tools increase trust in the analytics work getting done at The Zebra.