Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor)
Ramesh Johari, a Stanford professor specializing in online marketplaces, discusses how marketplaces reduce transaction friction, the critical role of data science in their success, and common founder pitfalls. He emphasizes prioritizing learning over "wins" in experiments, designing effective rating systems, and the increasing importance of human insight with AI.
Deep Dive Analysis
13 Topic Outline
Defining a Marketplace Business and Its Core Value
The Role of Data Science in Modern Marketplaces
Common Flaws and Misconceptions When Starting a Marketplace
The 'Every Founder is a Marketplace Founder' Mindset
Litmus Test for Scaled Liquidity in Marketplaces
Leveraging Data Scientists for Marketplace Optimization
Distinguishing Prediction from Decision-Making in Data Science
Balancing Incremental Experiments with Big Opportunities
Marketplaces as a 'Whac-A-Mole' Game: Winners and Losers
Shifting Organizational Focus from Impact to Learning
The Cost of Learning and Experimentation
Designing Effective Rating Systems for Marketplaces
Impact of Large Language Models on Data Science
7 Key Concepts
Transaction Costs
These are the frictions involved in finding a place to stay, a driver, or any other service in a market. Marketplaces like Uber and Airbnb fundamentally sell the reduction or elimination of these transaction costs, acting as intermediaries that connect supply and demand efficiently.
Marketplace Data Science Flywheel
This describes the cyclical process in marketplaces involving three key stages: finding potential matches, making the actual match, and then learning from those matches (e.g., through ratings or passive data). This learned information then feeds back into improving future matching and finding processes.
Scaled Liquidity
This refers to having a significant number of both buyers and sellers (or demand and supply) on a platform. A business only truly functions as a marketplace when it achieves scaled liquidity on both sides, otherwise, it's primarily focused on scaling one side or a different value proposition.
Correlation vs. Causation
Prediction models in data science identify correlations or patterns in past data. However, making effective business decisions requires understanding causation – whether a specific action or intervention *causes* a desired outcome, rather than just being associated with it. This distinction is crucial for moving from merely predicting to actively improving outcomes.
Fat Tails in A/B Testing
This concept applies to businesses where experiments can potentially yield very large positive or negative outcomes, beyond typical expectations. It suggests that companies should be willing to try more risky, non-incremental experiments and not run them for excessively long durations to discover these 'fat tail' opportunities.
Rating Inflation
This is an observed phenomenon in many marketplaces where average ratings tend to increase over time. Factors like reciprocity (people don't want to be mean) and norming (what constitutes a 'good' rating shifts) contribute to this, making it harder to distinguish true quality over time.
Bayesian A/B Testing
Unlike frequentist statistics, which analyzes each experiment in isolation, Bayesian A/B testing incorporates prior beliefs (based on past experiments or knowledge) with new experiment data to form a conclusion. This approach allows for continuous learning and updates to understanding, even from experiments that don't yield a 'win'.
9 Questions Answered
Marketplaces fundamentally sell the reduction of 'transaction costs' or friction, making it easier for buyers and sellers to find each other and complete transactions. They are paid for taking away the difficulty of finding a place to stay, a driver, or other services.
Data science is central because it underpins the ability to architect and re-architect the marketplace on the fly, enabling the three core functions: finding potential matches, making matches, and learning from those matches to improve future interactions.
Many founders think too much about being a marketplace before they actually have the necessary scale. A marketplace business rarely starts as one; it must first offer a valuable solution to a specific friction point, even without scaled liquidity on both sides.
A litmus test is to ask if you have 'scaled liquidity' on both sides of your platform – meaning a lot of buyers and a lot of sellers. If you don't have both, you are not yet a marketplace, and your focus should be on scaling one side or a different core value proposition.
Data scientists should always be thinking that their goal is to help the business make decisions, recognizing the critical distinction between correlation (prediction) and causation. The aim is to understand how actions *cause* changes in business value, not just to predict patterns.
Companies often become risk-averse, testing only incremental changes and running experiments for too long. To find big opportunities, they should foster a culture where 'learning is a win' (even from 'failed' experiments), try riskier ideas, and increase experimentation velocity by not running every test for an extended period.
Learning comes at a cost, especially in experimentation. When you run an A/B test, you allocate resources (e.g., users, ad spend) to a control group or a 'losing' treatment, which could have been used for a 'winning' option. This cost is often overlooked if the culture only rewards 'wins' rather than the insights gained from all experiments.
Marketplace founders should be aware of 'rating inflation' and consider re-norming rating labels (e.g., using 'exceeded expectations' for top ratings). They should also be careful about simple averaging of ratings, as it can disproportionately harm new participants with early negative reviews, and explore methods like incorporating prior beliefs to ensure distributional fairness.
LLMs and AI are massively expanding the frontier of possible hypotheses and ideas, making it easier to generate code and visualizations. This increased capability puts *more* pressure on humans to be in the loop, focusing on funneling down the vast possibilities to identify what truly matters for business decisions, rather than automating humans out.
28 Actionable Insights
1. Serve Both Sides as Customers
Recognize that both buyers (e.g., riders, guests) and sellers (e.g., drivers, hosts) are customers of your platform, as the marketplace’s core value is removing friction for both parties.
2. Start with Non-Marketplace Value
When launching a business, focus on a clear value proposition that solves a specific friction point, rather than immediately trying to be a scaled marketplace, as you won’t have the necessary liquidity on both sides.
3. Focus Data Science on Decisions & Causation
Direct data scientists to help the business make better decisions by understanding causal impact, rather than just predicting correlations from past data, as prediction is not the same as making decisions.
4. Prioritize Learning Over Wins in Experiments
Shift the organizational culture to view experiments as opportunities for learning about your business and testing hypotheses, even if they don’t result in an immediate “win” or positive metric change.
5. Recognize Learning Has a Cost
Understand that running experiments and gaining insights incurs a cost (e.g., lost potential revenue from control groups or “failing” treatments), but this investment is necessary to learn and make better future decisions.
6. Evaluate Algorithms by Business Impact
When comparing different algorithms (e.g., ranking systems), assess them based on their direct impact on core business metrics like bookings and revenue, not solely on their ability to predict past outcomes.
7. Integrate Business Understanding with Experiment Results
Do not discard your deep knowledge of the business when interpreting experiment results; use it to contextualize findings, especially when data is sparse or short-term metrics are flat.
8. Be Quantified, Not Just Data-Driven
When full measurement isn’t possible, process experiment results by quantifying and considering competing beliefs or hypotheses from leadership, combining data with informed judgment.
9. Leaders: Expect More from Data Scientists
Encourage data scientists to go beyond delivering statistically rigorous results by also articulating what they’ve learned about business flows, funnels, and customer preferences in their reports.
10. Define Tests by Learning Hypotheses
Structure experiments not just to identify winners and losers, but to test specific hypotheses about business processes, customer behavior, or market dynamics, fostering deeper organizational learning.
11. Avoid Early Overcommitment
Be cautious about making early commitments, such as specific pricing schemes or monetization models, that could limit future flexibility and lead to issues like disintermediation as your platform matures.
12. Prioritize Winners Over Losers (Marketplace Changes)
Acknowledge that many consequential marketplace changes will create both winners and losers; the key is to recognize if the value generated for winners outweighs the negative impact on losers for your business.
13. Implement Data Science Flywheel
Continuously improve your marketplace by cycling through three core data science problems: finding potential matches, making the best matches, and learning from those matches (via ratings, feedback, and passive data).
14. Assess Scaled Liquidity
Objectively determine if your platform has “scaled liquidity” (many buyers and sellers); if not, you are not yet a true marketplace, regardless of your long-term vision.
15. Leverage One-Sided Liquidity
If you have scaled liquidity on only one side of your marketplace, strategically use that strength (e.g., subsidizing one side) to attract and grow the other side.
16. Focus on Scaling One Side First
If you lack scaled liquidity on both sides, concentrate on scaling one side of your business first, treating it as a general startup rather than immediately trying to build a marketplace.
17. Be Humble About Marketplace Status
Let go of the ego of being a “marketplace founder” if you haven’t achieved scaled liquidity on both sides; focus on general startup growth and problem-solving, as virtually every business will have the option to become a platform eventually.
18. Don’t Over-rely on Experiments
Recognize that while experimentation is crucial, it cannot solve every strategic challenge; balance it with other forms of insight and decision-making, as you can’t experiment your way out of everything.
19. Avoid Incrementalism & Long Experiments
Challenge the tendency to only test incremental changes and run experiments for excessively long durations, as this can hinder velocity and the discovery of larger, more transformative opportunities.
20. Use Bayesian A/B Testing
Employ Bayesian A/B testing methods to incorporate past learning and prior beliefs into experiment analysis, which can help reward contributions of information and improve future decision-making by connecting current data with historical knowledge.
21. Focus on Learning from Matches & Rating Systems
Dedicate significant attention to understanding what is learned from successful and unsuccessful matches, and how rating systems capture and reflect this information, as this is a core differentiator of modern marketplaces.
22. Re-Norm Rating System Labels
Design rating systems to counteract rating inflation by re-norming labels (e.g., “exceeded expectations” instead of just “excellent”) or asking users to compare experiences to past highly-rated interactions.
23. Beware of Rating Averaging Issues
Be aware that simply averaging ratings can disproportionately harm new participants with few reviews, especially if their initial ratings are negative, potentially leading to early exit from the platform.
24. Use Priors for Rating Fairness
Implement prior beliefs in rating calculations to provide “distributional fairness,” helping new participants overcome initial negative reviews by averaging with a more balanced prior and giving them a chance to succeed.
25. Analyze “Sound of Silence” in Ratings
Extract valuable information from “ratings not left,” as the absence of a review can be a significant indicator of user experience and future performance, rather than just focusing on explicit feedback.
26. AI Increases Human Importance in Data Science
Understand that AI and LLMs expand the frontier of hypotheses and ideas, increasing the need for human judgment and interaction to funnel down to what truly matters in data science, rather than automating humans out of the loop.
27. Slow Down for Deeper Thinking
Cultivate a habit of slowing down to develop meaningful mental models of your business and its users, rather than rushing through problem-solving, to uncover structural insights and make better decisions.
28. Prioritize Data Literacy
Foster data literacy across your organization, enabling all individuals to better interact with data tools, interpret results, and make informed decisions, as this is crucial for navigating the modern tech-enabled economy.
5 Key Quotes
Many of the changes that are most consequential create winners and losers. And rolling with those changes is about recognizing whether the winners you've created are more important to your business than the losers you've created in the process.
Servaes Tholen (quoted by Ramesh Johari)
The marketplace's customers aren't just the people buying the rides or buying the listings. Actually, the hosts are Airbnb's customers. And the drivers are also Uber's customers, right? So both sides of the marketplace are the customers of the platform.
Ramesh Johari
Prediction is inherently about correlation. But when we ask people to make decisions, we're asking them to think about causation.
Ramesh Johari
Experimentation is always very hypothesis driven. It's about what are you learning? And that's really an important distinction because what it means is if I go with something big, risky, and it quote unquote fails, meaning that doesn't win, I nevertheless, if I was being rigorous about what hypothesis that's testing about my business, I'm potentially learning a lot.
Ramesh Johari
If you don't have either side, don't worry about it, don't worry about being a marketplace, worry about scaling one side and in that world, it's opened up, it opens your visibility up completely into the advice of many, many, you know, startup advisors, right? People who have advice not so much about scaling a marketplace, but about scaling a startup.
Ramesh Johari