From our experience running growth experiments at multiple companies, we saw early signs that only incentivizing users to perform action ‘X’ or action ‘Y’ is not sustainable. We started exploring ideas that could shape user behavior in a sustainable manner while also engaging them as they perform core actions in the app.

As we consulted companies to help solve their growth problems, most were also looking to leverage gamification to move their metrics, but did not know how to get started or had previously set up gamification experiments that did not yield any results hence causing apprehension.

That’s when Vamsi and I decided to join forces and design a framework to explore game led experiments based on our combined experience running 400+ experiments across industries from early stage startups to Unicorns and public companies.

 and I working on figuring out an easy way for teams across the org and outside to be able to easily shortlist the best mechanics to experiment with irrespective of their exposure to gamification and behavioral science.

In the early days, we realized we had reached "success" when we saw someone post a video of how to “game” one of the experiments we were running. Soon we had a spike in engagement from users who were unconnected to the platform. That’s when we had to introduce fraud governance in our systems to detect and restrict users from gaming the system.

 has found that it's important to create an expected budget for these campaigns based on these probabilities. You can set some limits to ensure you don’t go over budget.

Threedots used a virtual currency that translated into X dollars. 1 million coins had a probability of virtually 0 but it was a great way to get users excited about the chance of winning something big.

While shortlisting experiential rewards, look at rewards that help reduce the friction to transact with you (Ex: Priority pickup or surge protection) or look at rewards that help make their experience memorable or feel special (Ex: Custom app icons or badges that a user can showcase to her peers or in her social circle). If you are considering rewards around transactions(Ex: Surge protection), keep in mind that they do come at a cost as well and are not entirely free for you as a business.

 emphasizes that just because gamification can drive retention, remember that retention is an output metric and gamification is not a silver bullet that will solve your engagement or retention woes altogether. There is a thing as too much gamification and you should be mindful of it!

When considering using this playbook, there are a few items to keep in mind:

- While this is a great template to get started, it assumes a lot of prerequisite knowledge of Psychology and retention.

- You should start with an inspiration document of all gamification campaigns that have worked for your audience/industry handy so that you can come up with creative experiments from that.

- If you plan to build this in-house, iteration cycles between the first and the second experiment can be long and can be a big reason for failure of the campaign. To prevent this, account for levers that you will experiment with in the campaign and include it in the scope of the project. For example, we had a dashboard to manage the probabilities of rewards given out on the ‘spin the wheel’ which did not need an engineer in the loop.

Sumant Subrahmanya

Vamsi Krishna AV

We designed this playbook to share a structured approach to designing gamification experiments that guarantee learnings. We've drawn from Vamsi's experience at Gojek and Sumant's experience working with clients at CustomerGlu.

As growth operators we’ve had to come up with multiple experiments to learn quickly via data and  user behavior and make hypotheses to plan next steps in our product strategy. We’ve collectively run hundreds of experiments around gamification to improve aspects from onboarding, drive repeat purchases and shape user behavior.

While most of these experiments were successful or helped us learn we also had those experiments that never gave any results or were not designed well.

The goal of writing this artifact is to: Share a structured approach to designing a gamification experiment so that you are set up for success. (Success here is ensuring that you’re learning from the experiment or are seeing the results in terms of metrics uplift that you’re aiming for)

A gamification experiment refers to an experiment that is designed to solve a problem by leveraging gamification constructs (Ex: Spin the wheel, Streaks, Leaderboards etc)

The outline or process to setup the experiment or series of experiments can be broadly divided into 5 steps.

Let’s get into the details.  As we explain each step, we will illustrate 3 examples of companies we’ve worked in or worked with to solve their growth problems. They are:

 - Gojek is a super-app in South-East Asia that provides services to users ranging from mobility (Booking a car) ordering food,  and payments

Threedots - Threedots is a trading app that helps connect early traders with investment experts using a social platform and becomes a one stop shop for all their investment needs.

Zalora - An international fashion eCommerce conglomerate that operates in South-East Asia (SEA) that sells curation of top international and local brands with over 3000 brands across a variety of categories.

Like every feature that you launch or an idea that you come up with, an experiment needs a clear goal too. A goal or insight that you’re looking to learn via the experiment. This is a crucial step because the type of problem will heavily influence the type of gamification construct you should choose.

​​Understand who you’d like to shape behavior for (Ex: Newly signed up users), what situations they are in (Ex: For Gojek when a user wants to order food, she’s likely to be in a situation where she’s hungry or is scheduling an order to arrive by the time she gets hungry) and the activities that we want them to do (Ex: Complete signing up or place an order etc).

One of the other goals is to understand what are the leading activities/actions that drive the metric. Here are some ways you can do this:

Analyzing good users for the business and the actions they took before becoming good users

Finding drop-off points in the journey and also points after which retention is good. Example: We found that after signing up, most people who churn do not come back after their first session. The goal shifted to How might we make users to do core activities in the first session that would keep them on the platform

The approach to gamification varies by geography, largely due to cultural differences, local preferences, and regulatory environments. These factors can impact how games are designed, the types of rewards or incentives that are offered, and how these incentives are distributed, too.

: Cultural factors such as societal values, attitudes, and behaviors significantly affect the design and implementation of gamification strategies. For example, countries with a high uncertainty avoidance index (like Japan) may respond better to games with clear rules and predictable outcomes, while countries with a lower index (like the United States) might be more open to games with elements of risk and uncertainty.

: Preferences for certain types of games or rewards can also vary regionally. For instance, competitive games may be more popular in individualistic societies (e.g., the United States), while cooperative games might be more successful in collectivist societies (e.g., Denmark, Indonesia). Similarly, the type of rewards that motivate people can differ. In some regions, tangible rewards like discounts or merchandise may be preferred, while in others, intangible rewards like status or recognition may be more effective.

: When it comes to implementing gamification experiences, legal and regulatory considerations are critical. Countries have different laws regarding games of chance, data privacy, advertising, etc. For instance, in the U.S, there are strict regulations governing sweepstakes and contests, which require clear rules, ways to enter without purchase, and fair selection processes. In the European Union, data privacy regulations under GDPR can affect how user data is collected and used in gamification strategies. With GDPR the implication is on gamification strategies that require personalisation.

 like Wish.com/Shein run Spin the Wheel/Raffles but fintech companies like Robinhood can't even use 

 in their experience. It's interesting to note that Robinhood's referral program "

However, there are some universal principles of behavioral science that apply across cultures, such as the effectiveness of immediate feedback, social influence, and progress tracking in motivating behavior. These principles can be incorporated into gamification strategies in any geography.

Here, the most appropriate game mechanics are chosen based on the goal alignment and audience appeal. This involves considering factors such as the complexity of the game mechanic, the level of user involvement required, and how well the mechanic can be integrated into the existing user experience.

Furthermore, different constructs appeal to different type of users. For example, a leaderboard might work well for competitive users, while a "streaks" system might work better for users who are motivated by maintaining consistency.

Another example is, mechanics such as spin the wheel are based on unpredictability and can create excitement in the moment while progress meters help people feel accomplished which is great when there are a series of steps that need to be completed.

Mechanics can be categorized into 3 types. They are:

At threedots, we identified 3 mechanics we wanted to leverage for different user goals in their journey from sign up to activation based on audience insights.

At Gojek, the goal we set out to achieve was to improve the conversion rate from new users to mature users. What this essentially meant from a user’s perspective was that they had to transact more often and at different restaurants as well on the app (Essentially a change in how they usually behaved) . After assessing multiple game mechanics we decided to go with missions for the following reasons:

Missions give a sense of progress to the user as and when she places an order for food

It doesn’t force the user to be too consistent and gives them just enough flexibility to place a food order whenever they need to. ( Streaks demand higher consistency from users whereas in missions you can make it more long-drawn)

Through missions you can also experiment with the reward frequency. Ex: If you have 5 steps to complete a mission you can structure it in multiple combinations:

Complete step - Complete Step - Reward - Complete Step - Complete Step - Reward

Complete step - Reward - Complete Step - Reward - Complete Step - Reward - Complete Step - Reward - Complete Step - Reward

Complete Step - Complete Step - Complete Step - Complete Step - Complete Step - Reward and so on…

Missions is a game mechanic that rewards a user for doing something on the app.

As you design the gamification experiment, the experience that you intend to provide the user from discovery of this mechanic up until the point of redemption is important. Experience here can be classified into 2 sub-categories. They are:

Discovery/Entry points : Where will the users discover this mechanic and at what point in their journey? For example, at Zalora we used a floating button to drive adoption of the campaigns

Construct detailing : How will this mechanic be constructed ? (Ex: A spin the wheel can have 5 options or can have 7 options too)

Customers can “game” the system and you’ll need to have easy ways to detect and block fraudulent users.

 For example, we found that a user was referring a lot of fake accounts and selling the stock as soon as the referral was complete.

The time the user completes the game mechanic you’ve designed as part of the experiment

The way she’s given the reward and how she can access it for redemption (Ex: Give a voucher directly or add it behind a scratch card to make it more engaging)

After redemption of the reward, what needs to happen for the user

Here’s an example experience of a Daily Spin the wheel that was used by threedots.

Rewards are the end goal or outcome that a user unlocks/achieves after going through the gamification experience that you have designed for her. They can be broadly classified into:

 include discount vouchers, cashback, and promotions. These rewards are more commonly used in gamification experiments to incentivize and delight users. (Ex: A scratch card in google pay(India) is issued to every user after she makes a payment which in turn reveals a discount voucher from a brand or a cashback)

However, these rewards need to be allocated judiciously as this could drive your incentive spends significantly if you’re not careful. Alternatively, you can also experiment with variable rewards when it comes to monetary rewards to have more control over your budget spends while retaining engagement. Variable rewards are a great way to motivate customers with the highest rewards while disbursing rewards based on probability in the 

. Here’s a example for the Spin the wheel that we used at Threedots.

Threedots used a virtual currency that translated into X dollars

 are rewards that help your users get an unfair advantage on your app without any monetary value attached to it.

Ex: Priority driver allocation and surge protection can be examples of an experiential reward for on-demand delivery and mobility apps such as Gojek, Doordash, Uber, Lyft etc

In Gojek’s example, the reward was intentionally monetary (A set of GoFood vouchers)  as this would reduce the friction for the new user to place their next order once the mission is completed.

For every experiment that you setup, measuring results or learning from the experiment is a must. You can measure results by:

Comparing your control user segment with the treatment and measure uplift or change in your target metric

If you’ve met the desired result for which you set up the experiment, conduct a retro with the team and understand what hypotheses were correct as you set up your experiment and what actually worked. Was it the game mechanic in itself? Was the reward enticing? Or was it a mixture of both? Doing a retro here and setting up followup experiments will help you tweak this further and eventually replicate it multiple times for different contexts

These are successful results!

Experiments can either be successful or there might be a lot of learnings from it. In either case, as product operators one aspect around experimentation is to ensure you iterate and learn so that you can make better decisions for the business and most importantly, your users.

If the experiment is not yielding the desired results during your checkpoint, here are some ways that you can iterate before you stop the experiment.

Tweak the difficulty of the game mechanic

Play with probability if you’re relying on variable rewards as your means to disburse rewards to users

Gamification Experiment Playbook from Gojek and CustomerGlu

Setup Gamification Experiments

Background

Outline

Sign up to see more