Lessons from scaling Uber and Opendoor | Brian Tolkin (Head of Product at Opendoor, ex-Uber)
Brian Tolkien, Head of Product & Design at Opendoor and former Uber product leader, shares lessons on building products with heavy operational components. He discusses product-ops synergy, effective product reviews, applying the Jobs-to-be-Done framework, and staying calm under pressure.
Deep Dive Analysis
17 Topic Outline
Introduction and Brian's Background
Starting in Operations at Uber
Ops Experience as a Foundation for Product Leadership
Fostering Synergy Between Product and Operations Teams
Evolution of Uber's Surge Pricing System
Wild Stories from Early Uber Days
Opendoor's Business Model and COVID-19 Adaptation
Running Effective Product Reviews
Implementing the Jobs to Be Done Framework
Experimentation Challenges with Low Sample Sizes
Increasing Conviction Beyond A/B Testing
When to Trust Intuition in Product Decisions
Opendoor's Partnership with Zillow and Market Focus
Developing Calmness Under Pressure as a Leader
Identifying the 'Kernel of Truth' in Product Management
Lessons from Early Uber Pool Launch Failure
Final Thoughts and Lightning Round
4 Key Concepts
Product Operations (Product Ops)
A function that acts as a bi-directional feedback loop between a centralized product team and globally distributed operations teams. Its purpose is to effectively launch new features in global markets and gather input from those markets to build better features.
Twin Turbine Jet Plane Mentality
A mental model for product and operations teams, suggesting that while a business can operate on one engine (either product or ops) for a short period, it runs most efficiently and effectively when both functions work together in harmony.
Jobs to Be Done (JTBD)
A framework that encourages product teams to deeply empathize with customers by understanding the context in which they are operating and what they are truly trying to accomplish, rather than just focusing on features or immediate product interactions.
Kernel of Truth
In product management, this refers to the core job of understanding what truly matters amidst a multitude of signals and feedback from various sources. It involves discerning noise from valuable insights that will genuinely move the customer and product forward.
10 Questions Answered
Starting in operations provides a deep understanding of how the business actually works, including day-to-day operations and direct customer interaction, which forms a strong foundation for building scalable technology solutions.
Product teams often miss that operations teams can iterate faster, scale customer interactions more efficiently, and provide great qualitative insights from the field, which are crucial for building better products.
Initially, surge pricing was a manual system where city GMs controlled parameters like when and where surge could activate, allowing for local knowledge of events, before evolving into a fully automated, dynamic system.
Opendoor temporarily halted its core business of buying homes and used those months to virtualize its entire home buying and selling process, allowing them to resume operations without physical entry into homes.
The two main goals are accountability and informing an audience about product progress, but most importantly, to help make the product better by fostering intellectual conversation and helping teams think through problems without feeling like a 'firing squad'.
Acknowledge the problem by running power analyses to understand detectable impact and runtime, consider longer runtimes for important experiments, and use alternative methods like customer interviews, observational data, diff-in-diff analysis, segmenting by geo, or reducing confidence levels.
When canonical A/B testing is not feasible due to low sample sizes or other constraints, and other conviction-building techniques have been exhausted, it's sometimes necessary to trust intuition, especially if it's high conviction, and ensure a reasonable feedback loop is in place to understand the outcome.
The home buying and selling business is complex and challenging, requiring expertise in pricing, product, operations, risk management, and capital markets, all integrated vertically. Zillow, being primarily a software-driven company, likely underestimated the need for this deep vertical integration and operational complexity.
Leaders can develop this by understanding that reflecting stress onto teams is counterproductive, adopting an even-keel demeanor, gaining perspective from repeated challenging situations, and learning from the experiences of others through stories and biographies.
It's the ability to discern what truly matters and will move the customer forward from the vast amount of feedback and signals received from various sources, requiring the discipline to say no to good ideas that aren't core to the main objective.
37 Actionable Insights
1. Gain Deep Business & Customer Understanding
Product leaders should gain deep understanding of how the business actually works by being close to day-to-day operations and customers (e.g., talking to them, handling support), as this foundation informs what scalable technology to build.
2. Prioritize Tech Investment for Leverage
For ops-heavy tech companies, especially in early days with limited technical resources, identify where technology provides the most leverage (e.g., dispatching, pricing) and explicitly prioritize investment there, transparently deciding not to invest in other areas for now.
3. Product & Operations as Twin Turbines
Foster a harmonious relationship between product and operations teams, viewing them as ’twin turbines’ that operate most efficiently and effectively when working together, rather than in competition.
4. Stay Calm Under Pressure
As a leader, stay calm under pressure because reflecting stress onto your teams causes them to tense up, which counterintuitively does not produce better outcomes.
5. Find the Kernel of Truth
The core job of product management is to find the ‘kernel of truth’ – what truly matters and will move the customer forward – amidst a sea of ambiguous signals and inputs from various sources.
6. Automate Operations with Software
View operations as potential inefficiencies and continuously work to automate them with software and product solutions, freeing up operational capacity for new challenges and experimentation.
7. Build Products with Real-World Flex
Design products with flexibility and fail-safes, acknowledging that the real world is messy and humans are unpredictable, unlike deterministic computers, to account for issues like missed appointments or cancellations.
8. Define Clear Product Review Goals
Clearly define the goals of product reviews as both accountability/information sharing and, primarily, to help teams make the product better by fostering intellectual conversation and problem-solving.
9. Lead Product Reviews Collaboratively
During product reviews, senior leaders should act as facilitators, probing with questions and offering ideas as suggestions (not mandates), and providing missing context, rather than creating a ‘firing squad’ atmosphere, to empower the team closest to the problem.
10. Apply Jobs-to-be-Done for Empathy
Use the Jobs-to-be-Done framework to deeply understand customer needs and context, fostering empathy by forcing product teams to consider the user’s perspective, especially when they are not typical users of the product.
11. Conduct Power Analysis for AB Tests
Before AB testing, always conduct a power analysis to determine if you can get statistically significant results, what effect size you can detect, and the required runtime, being honest about whether a long runtime is acceptable for important experiments.
12. Increase Conviction Beyond AB Tests
When canonical AB tests aren’t feasible due to low sample sizes, focus on alternative methods to increase conviction in solutions, such as talking to more customers, using observational data, diff-in-diff analysis, or segmenting by geo.
13. Trust Intuition When Data Lacking
If statistical significance is unattainable and no other conviction-building techniques are available, trust your intuition and ship the product, avoiding the pursuit of false precision.
14. Be Market-Focused, Not Competition-Focused
Be aware of competitors but remain primarily focused on the vast market opportunity and serving your customers well, rather than getting distracted by direct competition, especially when the market is not saturated.
15. Scale What Works
Start by doing things that don’t scale (e.g., manual onboarding) to learn and iterate, then identify successful processes and leverage technology to scale them.
16. Leverage Ops for Faster Iteration
Recognize that operations teams, especially local ones, can iterate faster and scale customer interactions more efficiently, providing valuable qualitative insights that help build better products.
17. Establish Product Operations Function
Create a product operations function to strengthen the bi-directional feedback loop between centralized product teams and globally distributed operations teams, ensuring effective feature deployment and valuable market input.
18. Foster Mutual Respect Between Teams
Ensure mutual respect between product and operations teams, acknowledging that both functions have distinct skillsets, time, and place, which is crucial for building large, successful businesses.
19. Communicate Growth Opportunities to Ops
Assure operations teams that automating their current tasks creates new opportunities for them to tackle different challenges, experiment, and ‘do things that don’t scale’ in other areas, preventing job insecurity.
20. Keep Product Reviews Small
To foster the best conversations, keep product review meetings relatively small, ideally under 10 attendees, while distributing artifacts (documents, recordings) widely for broader awareness.
21. Use Product Review Artifacts for Onboarding
Create and store artifacts (documents, recordings) from product reviews, as they are powerful for team understanding and serve as an excellent resource for new hires to quickly grasp ongoing work and context.
22. Implement Flexible Product Review Cadence
Establish a flexible product review cadence (e.g., two slots per week for teams to sign up) and proactively invite teams if their work hasn’t been reviewed recently, aiming for a quarterly cycle to ensure consistent oversight.
23. Use Frameworks Selectively
Treat frameworks like ‘jobs to be done’ as tools in a toolbox, understanding when and how to apply them effectively rather than forcing every problem into a single framework.
24. Consider Customer Context with JTBD
When applying Jobs-to-be-Done, think broadly about the full context in which a user operates, including external factors and their multi-week or multi-month journeys, not just interactions within your product.
25. Use JTBD in Product Review Templates
Integrate Jobs-to-be-Done into product review templates, requiring teams to pre-fill sections on problem statements and the customer’s jobs to be done, ensuring consistent application and discussion.
26. Foster Cultural Internalization of JTBD
Move beyond mere template adherence to foster a cultural internalization of Jobs-to-be-Done, encouraging deep discussions about whether a stated ‘job’ truly reflects the customer’s underlying need and context.
27. Test Assumptions with Humility
Always test assumptions and hypotheses when feasible, acknowledging with humility that intuition can be wrong, but if testing isn’t possible, don’t pretend it is.
28. Validate Low-Conviction Decisions
For decisions of consequence with low or medium conviction, talk to more customers and gut-check with others to build higher conviction before proceeding.
29. Establish Feedback Loops for Intuitive Ships
When shipping based on intuition due to constraints, establish rigorous feedback loops (e.g., customer support tickets, adoption rates) to validate hypotheses and understand if the decision was correct.
30. Prioritize Customer Focus
Maintain a desperate focus on understanding and serving your customer’s needs, as this provides confidence and direction, especially in competitive or large markets.
31. Maintain Even-Keeled Demeanor
Adopt an even-keeled demeanor, remembering that outcomes are rarely as good or bad as they seem, to maintain a clear head and think more clearly under pressure.
32. Learn from Stressful Experiences
Actively expose yourself to stressful situations, learn from them, and reflect on the tools used, so that future challenges feel less daunting and you can navigate them with experience.
33. Learn from Others’ Journeys
Expose yourself to stories of other entrepreneurs and leaders (e.g., podcasts, books) to understand that career paths are not linear and learn how others navigate challenges.
34. Say No to Good Ideas
Be disciplined in saying no to good ideas that don’t align with the ‘kernel of truth’ or the most impactful customer needs, focusing only on what truly matters to advance the product.
35. Document All Feedback & Ideas
Document all incoming ideas and feedback, not only for future reference but also to ensure that those who provided the input feel heard and respected, knowing their contributions were considered.
36. Match PM Skillset to Problem
When building a product team, focus on matching a PM’s unique skillset and context to the specific problem type that needs to be solved, rather than just hiring generalists.
37. Stay Curious
Cultivate a mindset of continuous curiosity in all aspects of work and life.
6 Key Quotes
Uber always had this mentality and Open Door does too, of kind of like a twin turbine jet plane, where you can like fly the plane on one engine for a little bit if you need to, but it's operating most efficiently and effectively if both are working together.
Brian Tolkin
When you reflect the stress onto your teams, everybody tenses up. It counterintuitively doesn't produce better outcomes.
Brian Tolkin
The real world has entropy and it's hard and it's messy. Computers are deterministic, but humans aren't. And so building products that have a little bit more flex or a little bit more fail safes in case those things happen, uh, it becomes a little bit more of a paramount.
Brian Tolkin
Experimentation is all about increasing your conviction in the problem or the solution.
Brian Tolkin
You're never as good as you think you are. You're never as bad as you think you are.
Brian Tolkin
The core job is to understand what really matters, right? Like what is noise? What is a good idea? What is a suggestion? What is, uh, and what is, uh, you know, back to the jobs to be done for like, what is really going to move the customer forward?
Brian Tolkin
2 Protocols
Effective Product Review Structure
Brian Tolkin- Define clear goals: accountability/information and, most importantly, making the product better.
- Foster a culture of intellectual conversation, not a 'firing squad'.
- Leaders should probe with questions and offer ideas as suggestions, not mandates, providing missing context.
- Keep the conversation relatively small (under 10 people) for best results.
- Distribute artifacts (documents or recordings) widely for broader understanding and onboarding.
- Operate on a sign-up cadence (e.g., two slots a week) and ensure work cycles through quarterly.
Increasing Conviction in Solutions with Low Sample Sizes
Brian Tolkin- Acknowledge when canonical A/B testing is not viable due to low sample sizes.
- Run power analyses to determine detectable impact and acceptable runtime for experiments.
- Consider longer runtimes (e.g., 6 months) for critical experiments if necessary, setting and forgetting them for future planning.
- Talk to more customers to gather qualitative insights.
- Utilize statistical techniques like observational data, diff-in-diff analysis, or segmenting by geo.
- Reduce statistical power (e.g., 80% confidence instead of 95%) if the trade-off is acceptable.
- If all other methods are exhausted, trust intuition and ship the solution, ensuring a reasonable feedback loop to understand its impact.