OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more | Kevin Weil (CPO at OpenAI, ex-Instagram, Twitter)
Kevin Wheel, Chief Product Officer at OpenAI, discusses the rapid pace of AI development, the importance of evals for product building, and how AI will transform work and creativity. He also shares insights on where startups can thrive in the AI ecosystem.
Deep Dive Analysis
15 Topic Outline
The Ever-Evolving Nature of AI Capabilities
OpenAI's Viral Image Generation Model
The Role of Chief Product Officer at OpenAI
Kevin Weil's Recruitment Story at OpenAI
The Importance of Evals in AI Product Development
OpenAI's Approach to Shipping Products Quickly and Iteratively
Chat as a Versatile Interface for AI
Collaboration Between Research and Product Teams
Hiring and Operating Product Teams at OpenAI
Leveraging AI in Product Workflows: Vibe Coding
The Future of Product Teams and Fine-Tuned Models
AI in Education and Personalized Tutoring
Optimism and Concerns About AI's Future Impact
The Next Big Leap in AI-Assisted Creativity
Reflections on the Libra/Novi Project
6 Key Concepts
Evals (AI Evaluation Tests)
Evals are like quizzes or tests for AI models, designed to gauge how well a model understands specific subject material or responds to certain questions. They act as benchmarks for a model's intelligence and capabilities, helping product teams understand model performance for different use cases and continuously improve them through fine-tuning.
Iterative Deployment
This philosophy at OpenAI involves shipping products early and often, even when not fully polished, to learn and co-evolve with users and society. It acknowledges that the understanding of AI models' full capabilities is a continuous learning process, making public iteration more valuable than prolonged internal development.
Model Maximalism
This mindset suggests that product builders should focus on pushing the envelope with current AI capabilities, even if models are not perfect, rather than spending excessive time building scaffolding around current limitations. The expectation is that AI models will rapidly improve, making products that barely work today 'sing' in a few months.
Vibe Coding
Vibe coding is a method of rapidly generating code using AI tools like Cursor or GitHub Copilot, where the developer provides high-level prompts and largely lets the model generate the code, tapping through suggestions. It's particularly useful for quickly creating proofs of concept or demos, allowing for rapid iteration and exploration of ideas.
Model Ensembles
This refers to using multiple specialized AI models together to solve a complex problem, much like a company of humans with different skills. Different models, potentially fine-tuned or of varying sizes, are used for specific tasks based on latency, cost, or capability requirements, and their outputs are integrated to provide a comprehensive solution.
Fine-Tuned Models
Fine-tuned models are base AI models that have been further trained on specific datasets to perform better on particular use cases. By providing thousands of examples of problems and good answers, a model can be customized to excel at industry-specific or company-specific tasks, becoming a core part of future product development workflows.
7 Questions Answered
It's often effective to reason about how an AI product should work, or why an AI behavior occurs, by thinking about how a human or a group of humans would behave in a similar situation.
Chat is an incredibly versatile and universal interface because it mirrors how humans naturally communicate, allowing for unstructured, open-ended, and flexible interaction across a wide range of intelligence levels and topics, which aligns perfectly with the power of LLMs.
OpenAI seeks product managers with high agency, comfort with ambiguity, a readiness to execute quickly, and strong emotional intelligence to lead through influence, especially given the collaborative nature with self-directed research teams.
Product teams will likely integrate researchers and machine learning engineers more deeply, as fine-tuning models will become a core workflow for building most products, requiring specific expertise to customize models for particular use cases.
Parents should focus on fostering curiosity, independence, self-confidence, and critical thinking skills, as these fundamental abilities will remain important regardless of how technology evolves.
The next big leap will involve AI enabling greater exploration and better final results in creative fields like film and writing, allowing creators to rapidly generate and iterate on numerous variations of ideas, leading to more creative and refined outcomes.
Libra, intended to allow free and instant money transfers via WhatsApp, ultimately failed to launch due to trying to introduce too many new things at once (a new blockchain, basket of currencies, WhatsApp integration) and Facebook's reputation at the time, despite the underlying technology living on in other projects.
18 Actionable Insights
1. Mindset for AI Builders
Adopt a mindset that current AI models are the worst you’ll ever use, as capabilities advance every two months. This encourages continuous innovation and rethinking product strategies, as technology changes underneath you so quickly.
2. Build on the Edge
If your product is currently pushing the limits of AI models, continue building. Models improve so rapidly that what barely works today will “sing” in a few months, allowing you to innovate ahead of the curve.
3. Master Writing Evals
Learn to write “evals” (tests or quizzes for AI models) to understand model capabilities and guide product development. This skill is crucial for knowing when a model is 60% vs. 99.5% accurate for a use case and for fine-tuning models to improve performance on specific use cases.
4. Leverage AI for Prototyping
Use AI tools like Cursor or Windsurf for “vibe coding” to rapidly create prototypes and proofs of concept instead of traditional design tools. This allows for quick exploration of ideas and demos, even if the code isn’t production-ready.
5. Build AI Startups in Verticals
Focus on building AI-based products in specific industries and verticals, leveraging company-specific or use-case-specific data. Foundational model companies like OpenAI won’t pursue most of these niche applications, creating immense opportunities for startups.
6. Integrate Researchers into Teams
Build product teams that include researchers or machine learning engineers, especially for fine-tuning models. This collaborative approach, where research, engineering, and product work as a single unit, leads to more novel and effective AI products.
7. Design AI Like Humans
When designing user interfaces and experiences for AI, consider how humans interact with each other. For example, when an AI needs to “think,” it can provide small updates or “babble” like a human, rather than just pausing silently.
8. Prioritize Iterative Deployment
Adopt an “iterative deployment” philosophy by shipping products early and often, even when not fully polished. This allows for co-evolution with users and society, gathering feedback and iterating in public to learn about model capabilities together.
9. Embrace Model Maximalism
Don’t over-engineer scaffolding around AI models to fix minor imperfections. Instead, embrace “model maximalism” by building for the capabilities that are almost there, trusting that future model improvements will quickly address current limitations.
10. Structure Teams PM-Light
Aim for a PM-light organizational structure where PMs work with slightly too many engineers. This empowers engineers to make decisions, fosters product-focused engineering, and encourages rapid movement by preventing micromanagement.
11. Hire for High Agency
When hiring for AI product roles (especially PMs), prioritize candidates with “high agency” who proactively solve problems without waiting for permission, and who are comfortable with massive ambiguity due to the rapidly changing nature of AI.
12. Be Decisive as Leader
As a product manager or leader, cultivate decisiveness. Understand when to empower your team to innovate and when to step in to make a clear call on ambiguous issues to ensure progress and avoid stagnation.
13. Utilize Fine-Tuned Models
Break down complex problems into specific tasks and use fine-tuned models or ensembles of different models for each. This allows for specialized performance, cost optimization, and better overall problem-solving, similar to how human teams work.
14. Teach Kids Core Skills
Focus on teaching children fundamental skills like curiosity, independence, and self-confidence, along with how to think. These qualities are considered essential for navigating an unpredictable future shaped by rapidly advancing AI.
15. Use Examples in Prompts
Improve AI output by providing examples within your prompts, effectively performing “poor man’s fine-tuning.” Include “Here’s an example, here’s a good answer” multiple times before asking the model to solve your problem.
16. Frame AI Prompts Effectively
Enhance AI responses by framing your prompts with role-playing (e.g., “You are the world’s greatest marketer”) or by emphasizing the importance of the task (e.g., “This is very, very important to my career”). This can shift the model into a more effective mindset.
17. Practice Consistent Good Work
Adopt the philosophy that consistent, good work over a long period, rather than seeking a “silver bullet,” leads to compounding gains and significant excellence. This applies to personal growth and professional achievement.
18. Provide Feedback to Developers
Actively provide feedback to AI developers about what works well and what fails in their products. This helps them understand user needs and improve models and features.
7 Key Quotes
The AI models that you're using today is the worst AI model you will ever use for the rest of your life. And when you actually get that in your head, it's kind of wild.
Kevin Weil
Every two months, computers can do something they've never been able to do before, and you need to completely think differently about what you're doing.
Kevin Weil
Our general mindset is in two months, there's going to be a better model and it's going to blow away whatever the current set of limitations are. And we say this to developers too. If you're building and the product that you're building is kind of right on the edge of the capabilities of the models, keep going because you're doing something right. Give it another couple months and the models are going to be great. And suddenly the product that you have that just barely worked is really going to sing.
Kevin Weil
Libra is probably the biggest disappointment of my career. It fundamentally disappoints me that this doesn't exist in the world today because the world would be a better place if we'd been able to ship that product.
Kevin Weil
Plans are useless. Planning is helpful.
Kevin Weil (quoting Dwight D. Eisenhower)
No matter how big your company gets, no matter how like incredible the people are, there are way more smart people outside your walls than there are inside your walls.
Kevin Weil (quoting Ev Williams)
Sometimes it's not any one thing. It's just good work consistently over a long period of time.
Kevin Weil (quoting Mark Zuckerberg)