Introduction
Product Market Fit is just another way to describe whether your product is valued by your users. For the vast majority of SaaS startups, this can be measured by user retention. It is a measure of product health, and in Sequoia’s words, is something that affects every significant metric.
“Before you invest in acquiring new users through marketing or paid channels, you should understand how to strengthen and stabilize retention for early users. If retention is weak, few users will stay long-term, and you will churn through the total addressable market with little to show for it—at great expense.” – Sequoia Capital
Triangle Retention Chart
The most common way to measure retention is the Triangle Retention Chart. Each row of the chart represents a temporal cohort, and each column represents the age of each cohort. The temporal period we choose, whether it’s months, weeks, or days, will depend on our product. For example, where a ride-sharing app might measure on a weekly basis, a social media app would measure on a daily basis.
When analyzing trends in a triangle chart, the key things to look for are:
- Horizontal features show how activity in a certain time period impacted retention for a set of cohorts
- Diagonal features usually reflect changes to product
- Vertical features show trends in subscription models
- Important to track large cohorts that disproportionately impact overall retention numbers
Generally speaking, we’re looking for sustained retention for our oldest users, and improved retention over time in any given column indicating progress towards Product Market Fit. For more detailed information about the Triangle Retention Chart, see Sequoia’s article on Retention.
Figure 1. Cohort Retention Example
Measuring Athenic AI’s Product Market Fit
Figure 2. Athenic AI teams that create dashboards have strong retention.
Cohort retention analysis allows us to measure sustained product value, while segmentation helps us isolate features that resonate particularly well. By measuring our PMF in different feature segments, we can identify which users are getting the most value from our product, and even identify the right pricing model.
Naturally, we start by plotting our overall cohort retention. Because the main way users interact and gain value from Athenic AI is by asking questions, we use questions as our primary usage activity. For a ride-sharing app, usage would be measured as booked rides, and for a social media app, usage could be anything from views to likes to posts.
Initially, we find that retention trends towards zero, though a small percentage of users do retain. This view isn’t all that useful because our Free Tier attracts a great number of people who don’t have a real use case or don’t have real data for our data analytics product. The few percent that do retain are getting washed out in the macro view and obscuring what is the value that our users are getting from the product.
Figure 3. All Athenic AI teams.
It’s important that we segment by our different features to identify what the main drivers of value and retention are.
Figure 4. Teams that have asked >= 5 questions.
Figure 5. Teams that have asked >= 10 questions.
Figure 6. Teams that have asked >= 30 questions start to show good retention.
Figure 7. Teams that have multiple users have great retention.
Figure 8. Teams with Dashboards have great retention.
Key Insights
- There seems to be a threshold for strong retention once users ask ~30 questions. This could be because users who have real data will tend to ask more than 5-10 questions. This could be a good cutoff to push free users to upgrade.
- Users seem to retain very strongly after inviting coworkers to join their teams, and after pinning interesting graphs to their dashboards. There are some obvious ways to increase usage of these features in the user interface.
Most importantly, where the initial “all users” view didn’t give us much useful information, by segmenting, we were able to identify the features that most resonated with our users. This has translated directly to features on our product roadmap, and is helping us strengthen our Product Market Fit and experiment with our pricing model.
Do Your Own Cohort Retention Analysis
Doing Cohort Retention Analysis is easy and straightforward in Athenic AI.
1. Start by connecting your application database.
2. Create a new project using the application database.
3. Select the data that contains usage activity, their timestamps, and any additional data we want to segment on. For Athenic AI, we select questions and question timestamps as our usage activity. Then, we select dashboards and team users as our segmentation data, which are connected to questions via team id.
4. Now, we ask to get data on our usage activity by user by time period: “Show me questions per team id per month.” It’s important that we get the activity timestamp linked to the user. Then, we simply select the “Cohort Retention Heatmap” visualization option.
5. Additionally, we can segment by different features our users interact with: “Show me questions per team id per month, only include teams that have created a dashboard.”
6. Lastly, we can save any of our Cohort Retention Heatmaps to our own dashboard. At Athenic AI, we’ve created a “Measuring PMF” dashboard that contains all of the important user segments, so that we might track and improve them over time.
