Implementing a new framework for A/B testing helps team define hypotheses, size out each opportunity and prioritize.

Crunchbase is the leading platform for obtaining company insights from organizations of all sizes.  Over 50 million professionals from around the world trust Crunchbase to inform their business decisions and power their applications.

Crunchbase’s goal was to grow their number of Crunchbase Pro subscribers.  Eager to test theories for improving user conversion to their Pro product, the product team purchased an A/B testing software called Split.  After quickly identifying more than 20 ideas for improving their product UX to drive subscriptions, the team knew that implementing too many tests at once may impair overall results.  How should they structure their new testing program to ensure they see quality results without derailing their current user flows?   How could they perform a pre vs. post analysis with statistically rigorous methods when a true split test is not possible? 

The Crunchbase team reached out to Mammoth Growth seeking their expertise around setting up and executing A/B testing programs.  Mammoth Growth was able to provide a framework for testing that would prevent common mistakes and ensure meaningful outcomes.  The framework began with an exercise to help the product team define each test idea, size out the opportunity and detail how they planned to measure results.  Through this exercise, the team was able to prioritize their testing ideas and create a timeline to follow.

“Mammoth Growth helped push us beyond A/B testing ideation. They helped us systematically size the opportunity of each testing idea, defending each hypothesis with current metrics,” shares Mark Chan, a Product Manager at Crunchbase.  “This really helped our team to align against our core goals, then narrow and prioritize our testing ideas.”

One of the more challenging areas Crunchbase set out to test was a pricing change for their Pro product.  Instead of initiating a traditional A/B test, Mammoth Growth provided a protocol for analyzing pre vs. post experiment results with statistically rigorous methods.  When test results came back, Crunchbase knew they could trust the results and act accordingly because of the forethought put into the testing protocol and the statistical methods applied during analysis.

Powered by A/B testing results, the team has removed friction from their purchase flows, introduced new programs, and identified leading indicators and behaviors correlated with higher Pro subscription rates.  Overall, A/B testing has led the Crunchbase team to improvements that have generated significant growth in Pro subscriptions since their testing program began.