First interview with Scale AI’s CEO: $14B Meta deal, what’s working in enterprise AI, and what frontier labs are building next | Jason Droege
Jason Droege, CEO of Scale AI, discusses the evolution of AI models from knowing to doing, emphasizing the critical role of expert data labeling and human judgment in making AI smarter. He also shares product and entrepreneurship lessons from his career, including building Uber Eats.
Deep Dive Analysis
16 Topic Outline
Early Career Lessons from Scour and Legal Challenges
Scale AI's Current State and Relationship with Meta
The Evolution to Expert-Driven AI Data Labeling
Strategies for Recruiting AI Experts
Reinforcement Learning and AI Agent Environments
The Future of Human Involvement in AI Development
Understanding the Role of Evals in AI Training
AI Models Shifting from Knowing to Doing
Building Uber Eats: Customer Obsession and Unit Economics
The Importance of Independent Thinking in Business
Evaluating New Business Ideas and Market Selection
The Uber Eats McDonald's Partnership Story
Gross Margins as a Filter for Business Feasibility
"Not Losing is a Precursor to Winning" Philosophy
Hiring for Adaptability and Team Composition
Practical AI Uses: Tutoring and Document Summarization
5 Key Concepts
Expert Data Labeling
This is the process where highly skilled professionals, such as PhDs, web developers, or doctors, contribute their specialized knowledge to train AI models. It involves performing complex tasks, explaining nuanced topics, or building entire websites for models to learn from, evolving from earlier generalist labeling as AI models became more sophisticated.
RL Environments (Reinforcement Learning Environments)
These are sandboxes or simulated systems where AI agents can interact and learn to accomplish specific goals. They are used to train agents to navigate complex software systems (like Salesforce), recognize data, understand configurations, and achieve business processes with high reliability, including knowing when to escalate to a human.
Evals (Evaluations)
Evals are benchmarks or comprehensive sets of criteria established by human experts to define 'what good looks like' for an AI system's performance. They help models understand the desired quality and accuracy for specific tasks, especially in nuanced or mission-critical use cases within enterprises.
Digitizing Human Judgment
This concept refers to capturing and encoding the nuanced decision-making, deep expertise, and contextual understanding of human subject matter experts into AI models. It's becoming a bottleneck because off-the-shelf models often struggle with the specific cultural, objective, and incentive-driven judgments unique to individual enterprises.
Not Losing as a Precursor to Winning
This philosophy emphasizes managing risk and ensuring survival, particularly in entrepreneurial ventures. It suggests that while calculated risks are important, avoiding decisions that could compromise the entire enterprise is crucial for long-term perseverance and eventually thriving, as success often requires enduring through difficult periods.
12 Questions Answered
The main lesson was that everything in business and startups is negotiable, and there isn't one fixed way to do things; success often comes from effectively negotiating and aligning incentives.
Scale AI remains a fully independent company, with Meta making a non-voting stock investment and Alex Wang transitioning to Meta, while Scale continues to grow its two major businesses, each generating hundreds of millions in revenue.
Data labeling has shifted significantly from basic tasks performed by generalists to sophisticated tasks requiring highly specialized experts (e.g., PhDs, web developers, doctors) to provide nuanced feedback and build complex examples for AI models.
Scale AI finds experts primarily through referrals from existing contributors, campus programs where they engage with professors and students, and traditional channels like LinkedIn, by providing a great experience and an outlet for their expertise.
Jason Droege believes humans will always be in the loop because as models advance, they will continually need new human skills and knowledge to be incorporated, and the idea of AI not needing any external human data is 'unfathomable'.
Many pilots fail because while it's easy to get AI systems to 60-70% accuracy, achieving the high reliability (e.g., 98%+) required for mission-critical enterprise processes takes significant time (6-12 months), investment, and addresses complex issues like legal, policy, and change management.
The general trend is moving from AI models primarily 'knowing things' to 'doing things,' meaning they will increasingly act as agents capable of navigating complex environments and making decisions on behalf of users.
Since restaurant owners were hesitant to share detailed unit economics, Jason's team independently researched by ordering food, weighing ingredients, and matching them with supplier catalogs to triangulate ingredient and labor costs, gaining an insight into incremental gross margins.
Independent thinking helps founders identify unique insights (alpha) in a crowded market, rather than being solely influenced by prevailing narratives, and fosters the ability to constantly question one's own ideas for the sake of the mission.
Good business models often feature high gross margins, recurring revenue, stickiness, and network effects, becoming more valuable at scale and making it harder for competitors to erode margins.
He focuses on hiring curious problem-solvers who can articulate their thoughts, are humble enough to collaborate, and demonstrate good leadership, prioritizing adaptability and the ability for team members to compensate for each other's strengths and weaknesses over just experience at scale.
He uses AI as a tutor to learn new concepts in the rapidly evolving AI space, especially for technical nuances, and also uses it to summarize internal documents, asking 'what's the most important thing in this document?'
22 Actionable Insights
1. Cultivate Independent Entrepreneurial Insight
As an entrepreneur, seek ‘alpha’ by cultivating independent insights that others miss, questioning conventional wisdom, and avoiding being overly influenced by prevailing narratives to find unique opportunities.
2. Prioritize Survival as Precursor to Winning
In entrepreneurship, prioritize survival and avoid decisions that could compromise the enterprise, understanding that perseverance through challenges is essential for long-term success, even in high-risk environments.
3. Commit to Problems You’re Passionate About
Choose to work on problems you have a burning desire to solve for 5 to 10 years, rather than just solving a customer’s stated problem, as sustained passion is crucial for navigating the challenges of building a business.
4. Understand Customer Incentives Deeply
When getting close to customers, don’t just take their words literally; instead, analyze their underlying incentives (financial, ego, career growth) to truly understand their needs and how your product can unlock opportunities for them.
5. Address Buyer’s Daily Urgency
When building new products, ensure they address the customer’s most urgent daily problems, as even valuable solutions will struggle if they are not top-of-mind for busy buyers.
6. Form Independent Customer View
To gain true insight, independently research and triangulate customer economics (e.g., ingredient costs, labor costs) rather than solely relying on what customers tell you, which can help you identify unique value propositions.
7. Filter for Strong Business Models
When evaluating new business ideas, apply filters for strong business models (e.g., marketplaces, SaaS, recurring revenue, sticky, network effects) that are more valuable at scale and have a chance of becoming very large.
8. Use Gross Margins to Assess Value
Use gross margins as a quick filter to assess if your business idea adds enough value and is sufficiently differentiated; if margins are low, question why customers would pay and how you’ll avoid margin compression.
9. Negotiate Everything in Business
Realize that in business and startups, there’s no fixed way to do things; success often comes from negotiating your way through situations, as terms and expectations are often flexible.
10. Maintain Wide Aperture for Ideas
Keep a very wide aperture on ideas for as long as possible, even those that seem bad initially, and keep digging to verify if your initial assessment is correct or incorrect.
11. Hire Curious, Collaborative Problem-Solvers
When hiring, prioritize candidates who are curious problem-solvers, humble enough to work across teams, and good leaders, as adaptability and strong interpersonal skills are crucial in a rapidly changing world.
12. Build Teams with Complementary Strengths
When building a management team, compose an ‘organism of strengths’ where members know each other’s strengths and weaknesses and can compensate for each other, valuing this over classic experience for every role.
13. Plan 6-12 Months for Robust AI
Understand that implementing robust AI solutions for important processes typically takes 6 to 12 months, requiring time for legal, policy, regulatory approval, change management, and achieving comfortable accuracy levels.
14. Apply AI to Low-Accuracy Processes
Deploy AI in human processes that are currently 10-20% accurate, as AI can significantly improve accuracy to 50-80%, delivering substantial value even if human intervention is still needed for the remainder.
15. Digitize Human Judgment for AI
Recognize that digitizing human judgment and deep subject matter expertise is a critical bottleneck for AI systems to achieve mission-critical accuracy, especially in nuanced enterprise contexts where culture and objectives vary.
16. Establish Clear AI Benchmarks (Evals)
For AI systems, establish ’evals’ (evaluations) to define what ‘good looks like’ and set comprehensive benchmarks, which is crucial for training models and ensuring they meet necessary accuracy levels.
17. Prioritize Generalizable AI Data
When collecting data for AI, focus on data that is generalizable across a broad spectrum of use cases to avoid needing trillions of specific combinations, thereby increasing the data’s value for model builders.
18. Train AI in RL Environments
To make AI agents useful for specific goals, train them in ‘RL environments’ (sandboxes) that simulate real-world scenarios, allowing them to learn how to navigate complex systems and accomplish tasks reliably.
19. Contribute Expertise to AI
If you possess deep expertise, consider contributing to AI model training, as models increasingly require PhDs and professionals to provide nuanced data and correct understanding. This can be a paid outlet for your knowledge.
20. Maintain Human Oversight in AI
Expect humans to always be in the loop for AI systems, especially for critical decisions, as these systems need to work for us and require ongoing human involvement to ensure they align with our needs and values.
21. Use AI as a Personal Tutor
Leverage AI as a personal tutor to quickly learn new concepts and stay updated, especially in fast-moving fields, by asking it questions or using voice mode during commutes.
22. Summarize Documents with AI
Use AI to quickly identify the most important information in internal documents, saving time and ensuring you focus on critical details amidst a large volume of information.
7 Key Quotes
The general trend right now is going from models knowing things to models doing things.
Jason Droege
If you can imagine it and it makes sense and you can align incentives, then it can happen. But there is no like way.
Jason Droege
If you have a human process that is like 10 or 20% accurate or 10 or 20% light, AI is awesome because it can get you, if you get to 50, 60, 70, 80% accurate, you're in the money.
Jason Droege
You can build something that provides a lot of value, but if it's not the top thing that the customer is thinking about in their busy days, then you're just going to have a long road to a small town.
Jason Droege
Show me the incentive and I'll show you the outcome.
Jason Droege
Luck is part of the game. That's the other thing that's important to realize. Luck is part of the game. So do not begrudge people for luck. Like this industry is hard. All these things we're doing are really, really hard. Luck is just part of the game.
Jason Droege
The end is never the end.
Jason Droege