How to improve a metric

The goal here is to increase conversion rate, defined as people who become paying customers divided by people who land on the site. We can break it down into free trial conversion rate * subscription conversion rate, where:

free trial conversion rate = # of people who join the free trial / # of people landing on the site

subscription conversion rate = # people becoming paying customers / # people who join the free trial.

Dropping the CC info requirement for sure will increase free trial conversion rate, but decrease subscription conversion rate. To know exactly how that trade-off works, we would need an A/B test. But the question here is not figuring out which product is better, but if we should test the no CC option. That is, can we find in the data some insights that can lead to making the hypothesis that running that A/B test is a good idea? The insight step is not going to tell us what’s best. We would need causal data for that. But it can give us data-driven ideas for A/B tests.

Whenever you add friction to the funnel, you are making users self-select themselves based on intent. This is true for any kinds of friction. Adding the CC requirement is an obvious example of this, but even something like adding one more page to the conversion funnel will have the same effect. Users with higher intent will be willing to put up with that, while users with lower intent will drop off. You can safely expect that dropping the CC requirement will increase the proportion of low intent users in your free trial.

Therefore, a way to look at this problem is: can you identify in your current product users with low intent when they sign up for the free trial, and see how they behave during the free trial and at the end of it? Having the CC requirement filters out most of users with low intent, but still there must be some of them that make it through. And you can fairly realistically expect that what happens after dropping the CC info requirement is that the proportion of these users will go up.

For instance, you could look at users who sign up for the free trial and shortly after unsubscribe. Essentially, these people put themselves in the same spot as the “no CC sign up” user experience would be. To continue the subscription at the end of the month they will have to take an action (go to the site and manually restart the subscription). How do these people use the product within that month? How many of these users eventually change their mind within the month and decide to become paying customers?

If currently very few free trial users with low intent eventually become paying customers, dropping the CC info requirement doesn’t seem a particularly useful test. There won’t be much to gain from increasing the number of low intent free trial users and the other disadvantages of the no CC model (higher friction at the end of the free trial for all users) will most likely offset whatever little gain you are getting from the low intent users.
If that’s the scenario, before thinking about increasing the low intent user proportion via dropping the CC requirement, you should focus on how to increase subscription conversion rate for low intent users with your current business model. How can you engage them? What is the set of early actions that will make them become paying customers? How can you incentivize those actions? etc. Basically, the usual retention stuff. Once you succeed in that, you can think about increasing the number of low intent users via dropping the CC info requirement.
Obviously, if your low intent users are already becoming paying customers at a relatively high rate, then dropping the CC info requirement is a good thing to try.

Note that this is really the usual product data science stuff. At the highest level, a product data scientist either looks for segments that are already performing well and figures out how to increase their proportion within the current user base (i.e. Germans are converting at a higher rate -> tell marketing team to get more German users) or identifies a segment that’s converting poorly and tries to come up with ideas to fix it (i.e. Germans are converting at a lower rate -> maybe we need a localized version of the site in German).

The CC info scenario is the same thing. If users with low intent when they sign-up for the CC are doing relatively OK by the end of the free trial, it means the product is succeeding in making them change their mind within one month -> drop the CC info requirement so you can get more of them and take advantage of the current product strength. If low intent users are doing very poorly by the end of the month, figure out how to improve the experience for that segment (engage them early on, better content, etc.).

Netflix offers a 30-day free trial. Currently, people need to put their CC info to join the free trial. We are thinking of letting people sign-up for the free trial w/o asking for CC info. How would you figure out if running this test makes sense?

INSIGHTS: Should users be asked for their CC info when starting a free trial?