5 or 5000: How many users should you test with?
The approach of validating a new idea with users before going to production isn’t new. Companies of all sizes, early-stage startups to large enterprises, rely on user testing. But how many users should you test with?
The honest, though vague, answer is: It depends. You could test with as few as 5 users and as many as 5000 users, depending on how much clarity you have.
Are you testing with users to check if the broad idea makes sense? Do you want to figure out what features your product will need? Or are you looking to learn where all your product can fall short? We’ve found it helpful to think about testing in these three stages. Let’s dive in!
Does the broad idea make sense?
Say it’s 2008 and getting food delivered requires calling a restaurant to place your order and waiting for the delivery with a vague estimate of when the food will arrive. You have a product idea—a mobile app for food delivery. But you’re wondering if people will find it valuable and if online booking will take off.
At this stage, you need to test your idea with just five users. Look for people who’d become fans of the product if it worked (in this example, people who order from restaurants everyday) and show them a prototype. If the idea doesn’t resonate with potential fans, it’s unlikely that it would resonate with a larger group. The test helps you check if the idea doesn’t work.
When we built the first ever mobile app for Swiggy, we did this. It’s not that we built Swiggy after talking to 5 users, but testing our ideas with them gave us the confidence that there could be a future where people ordered food through their smartphones.
What does the product need for it to work?
Let’s say you know that the product or feature makes sense to the people who you’d call fans. Now you want to see how to make it work for more people so that it can be a viable business.
Back in 2017, Flipkart had been selling refurbished smartphones on their platform and this gave them the confidence that a dedicated app for refurbished electronics could work. But they needed to understand what it would take to move beyond the early adopters. What it would take for a lot of customers to buy refurbished smartphones?
We were working with Flipkart at the time and we got 4-5 cohorts of 5-7 users each to co-create their ideal version of the app. This gave us a strong sense of what features mattered to whom. A student in Bangalore cared a lot more about the tech specs of the camera—he wasn’t buying a refurbished phone, he was buying a better camera at a bargain price. On the other hand, a homemaker in Patna cared about the condition of the phone because she didn’t want anyone to know that she had bought a used phone.
When we finally designed 2Gud by Flipkart, we designed the product detail page in a way that worked for everyone. The app got a million downloads in just 6 months.
To understand your feature set really well, test with 4-5 user cohorts differentiated by relevant attributes like age, geography or socio-economic conditions. Over two rounds of testing, you would have tested with about 50 users and would be able to flesh out the product or feature really well.
Where does the product fall short?
Now you have a good degree of confidence in your features but you still don’t know a few things:
Which features get used the most and which features get used the least?
Does the product work for everyone? Are there edge cases you’ve missed?
This is a good time to test with thousands of users. The only way to do this is to build out the feature or product and design a test where you don’t have to be in the room. When users don’t know they are being observed, their feedback is a lot more objective.
Back in 2019, we were helping Wikipedia figure out how to bring more regional language readers to their platform. We tested prototypes of different rich media formats like stories, reels, timelines and figured out that a rich media format worked better than long form content. But we needed more than 50 users to figure out which of these formats worked the best.
To test this, we designed a microsite with 100 interlinked articles in which all of these formats were included. Over 5000 users visited the website and stories performed the best. Wikipedia built Wiki stories into their mobile experience.
Figure out what you want to evaluate about a feature, and design your tests accordingly. And look out for edge cases that only become evident through real world use.
The right number of users, at the right time
Testing with users, and in fact building with users is a no-brainer. It is the only way to get rid of your own biases and design an intuitive and useful product. But user testing is most effective when it’s the right number of users, in the right place and at the right time in your product journey.
The formula that has worked for us is this:
When you’re testing a broad idea, just five users are enough.
When you want to test features, test with 50 users from different user groups.
And if you need to see where the product falls short, test with 5000 users.