Growth experiments to optimise conversions & first-time experience.
Experiment 1: Increase 7 Day Retention
Collect more user data during signup to improve the user's experience and increase 7 day retention.
This experiment went against many traditional optimisation tactics. We looked at how increasing the amount of data collected, during signup, could improve a user's experience and increase 7 day and 14 day retention rates.
With more data about the user, we had much more power to personalise the content they see fromoday 1. Straight away, we're able to increase the relevance of the content they see on the home feed, better content recommendations, as well as better match them with other users, more relevant to their diagnosis, stage of their cancer journey, and other important factors that has been highlighted as relevant, many times during user research and feedback sessions.
To measure the success of this experiment we defined a few different success metrics and compared our regular flow (the control) with the updated flow (variable) through an AB test.
We mixed the quantitative results with qualitative user feedback, by interviewing users in both test groups about their perceived value based on their experience after 7 and 14 days.
The results were outstanding. The drop-offs only increased slightly, retention rates increased much more than we had expected, and user perceived value also increased.
However, we also discovered that user's trust was lowered initially during sign up, since we were asking for a lot of personal information about their cancer diagnosis, before they could directly understand the value. This learning allowed us to plan for a future experiment where we are comparing different ways of asking for user information at more relevant times to increase trust.
Experiment 2: Growth Loops
New growth loops to leverage exclusive app content using deeplinks to increase user acquisitions.
Working closely with the product owner throughout this ongoing process was critical. From understanding our different user behaviours, to defining what the optimal loops and hook models were suitable for our product and user behaviours.
One branch of this experiment was to harness the power of deeplinks to require the receiver of a deeplink to go through our download and signup funnels when our content is accessed and shared externally by a new user. Because we were adding more steps to what is usually a simple tap-and-open-experience, it was critical we didn't add any steps that were unnecessary during the signup experience.
In fact, we were able to cut the average signup time down by 65% for users that entered our app through a deeplink.
We achieved this by completely tearing apart our signup funnel and taking away everything that wasn't necessary, and highlighting the linked content to the user as a direct motivation to complete the process. As they had actively chosen to access the content, the motivation was strong, and not based on any assumptions from our end.
Working closely with the marketing team was critical in this experiment. Our marketing team was developing the exclusive content that was being shared through our external channels (website, socials, partners) that triggered the start of the funnel. It was important that the user not only got to the content quickly, it was also important that we targeted groups that would turn into loyal users and activate other growth loops organically, turning one conversion into an organic growth tree, continuously multiplying.
We challenged our hypotheses through data-driven solutions that balanced business goals with building a great first time experience for our newest members.
What’s a Rich Text element?
Sunheading
Another subheading
The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.
Static and dynamic content editing
A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!
- This is an example bullet point text,
- Here
- And here
- Also this line as well.
How to customize formatting for each rich text
Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.
And here is a really important quote that I'll want to highlight.
• • •
Project objectives...
Being in the transition phase from startup to scale-up, yet still being motivated to satisfy investors while building a valuable product for our members, our objective is to continuously increase all of our North Star metrics.
We had different hypotheses to why our metrics were weak in some areas. Recognising our biases personal, we cross-referenced findings from our most recent user research along with our product roadmap to determine areas that were detracting from our product's potential, and the onboarding and signup experience topped the list.
Optimising this very narrow funnel requires persistence, great understanding of our behavioural metrics and the guts to continuously experiment.
While the onboarding experience isn't the product itself, being an app, it is the only doorway into the product. And when the entranceway to your home isn't inviting, it sets a tone for what's to come.
• • •
Experiment 1: Increase 7 Day Retention
Collect more user data during signup to improve the user's experience and increase 7 day retention.
This experiment went against many traditional optimisation tactics. We looked at how increasing the amount of data collected, during signup, could improve a user's experience and increase 7 day and 14 day retention rates.
With more data about the user, we had much more power to personalise the content they see fromoday 1. Straight away, we're able to increase the relevance of the content they see on the home feed, better content recommendations, as well as better match them with other users, more relevant to their diagnosis, stage of their cancer journey, and other important factors that has been highlighted as relevant, many times during user research and feedback sessions.
To measure the success of this experiment we defined a few different success metrics and compared our regular flow (the control) with the updated flow (variable) through an AB test.
We mixed the quantitative results with qualitative user feedback, by interviewing users in both test groups about their perceived value based on their experience after 7 and 14 days.
The results were outstanding. The drop-offs only increased slightly, retention rates increased much more than we had expected, and user perceived value also increased.
However, we also discovered that user's trust was lowered initially during sign up, since we were asking for a lot of personal information about their cancer diagnosis, before they could directly understand the value. This learning allowed us to plan for a future experiment where we are comparing different ways of asking for user information at more relevant times to increase trust.
Experiment 2: Growth Loops
New growth loops to leverage exclusive app content using deeplinks to increase user acquisitions.
Working closely with the product owner throughout this ongoing process was critical. From understanding our different user behaviours, to defining what the optimal loops and hook models were suitable for our product and user behaviours.
One branch of this experiment was to harness the power of deeplinks to require the receiver of a deeplink to go through our download and signup funnels when our content is accessed and shared externally by a new user. Because we were adding more steps to what is usually a simple tap-and-open-experience, it was critical we didn't add any steps that were unnecessary during the signup experience.
In fact, we were able to cut the average signup time down by 65% for users that entered our app through a deeplink.
We achieved this by completely tearing apart our signup funnel and taking away everything that wasn't necessary, and highlighting the linked content to the user as a direct motivation to complete the process. As they had actively chosen to access the content, the motivation was strong, and not based on any assumptions from our end.
Working closely with the marketing team was critical in this experiment. Our marketing team was developing the exclusive content that was being shared through our external channels (website, socials, partners) that triggered the start of the funnel. It was important that the user not only got to the content quickly, it was also important that we targeted groups that would turn into loyal users and activate other growth loops organically, turning one conversion into an organic growth tree, continuously multiplying.
• • •
Experiment 3: Optimise conversions and improve first-impression
Contrary to the first experiment, we held another experiment to learn more about how we could further optimise our user's first-time experience.
It was important for us to have a clear understanding on how healthy our metrics were in comparison to others, so we compared our analytics with the baseline for conversions in our industry. We also compared ourself to our competitive landscape and leaders in the industry.
While optimising the signup flow was our goal, it required us to weigh up our members time optimisation with building value and understanding of our product. This led us to a secondary goal; understanding the optimal balance between user's time optimisation, user's understanding of the product. This meant many ongoing A/B tests with slight differences in length and content.
After reading recent studies while researching for this experiment, we learnt that including delight factors and reducing friction points increased the amount of time a user was willing to spend during their first-time experience.
Adding delight touch-points to our signup flow such as micro-animations, moving content, and Lottie animations, not only hoped to give our members a more memorable experience and set positive expectations of what's to come, but also increase conversions.
During these experiments we realised the value of working within a diverse team. We vocalised the importance of cultivating constructive disagreements that lead to healthy discussions and learning from each other.
• • •
Applications Used
Applications Used
Miro for product roadmap, North Star vision, visual maps and diagrams
Jira for planning, tracking, and managing the project
Figma for design concepts and prototyping
Slack for team communication, file sharing and more
Mixpanel for product analytics
POEditor for translations
Miro for team retrospective
Apple Notes for ad hoc note-taking