How to measure AI developer productivity in 2025 | Nicole Forsgren
Nicole Forsgren, creator of Dora and Space frameworks, discusses how AI impacts developer productivity and experience (DevEx). She shares insights on measuring gains, common mistakes, and a seven-step process to reduce friction and improve engineering team performance in the age of AI.
Deep Dive Analysis
17 Topic Outline
Defining Developer Experience (DevEx)
Flow State and Cognitive Load in the Age of AI
Challenges in Measuring Productivity with AI
Importance of Developer Experience for Business Value
Common Issues and Solutions in Developer Experience
Signs Your Engineering Team is Moving Too Slowly
How AI is Improving Developer Productivity
Real Examples of AI-Driven Productivity Improvements
Introducing the Frictionless Book and DevEx Framework
Getting Started with a DevEx Team
Measuring the Impact of DevEx Teams
Measuring the Impact of AI Tools on Productivity
Effective Survey Design for Developer Experience
Applying a Product Mindset to DevEx Improvements
AI Corner: Personal Use of AI for Home Design
Lightning Round: Book Recommendations and Life Motto
Nicole Forsgren's New Role at Google
5 Key Concepts
Developer Experience (DevEx)
DevEx refers to what it's like for a developer to build software day-to-day, encompassing the friction they face, workflows, and support. It's crucial because poor DevEx hinders the effectiveness of even the best processes and tools, impacting overall productivity and innovation.
Flow State (in coding)
A state of deep concentration and enjoyment developers enter when coding, crucial for happiness and productivity. AI can disrupt this by introducing interruptions for prompt creation and code review, but advanced workflows can also help maintain flow by offloading details to agents and providing context.
Cognitive Load
The mental effort required to perform a task. High cognitive load, often from focusing on 'plumbing' or mechanics, leaves less brain space for innovative solutions. Optimizing DevEx aims to reduce unnecessary cognitive load, freeing developers for more creative problem-solving.
SPACE Framework
An acronym-based framework for measuring developer productivity that considers multiple dimensions: Satisfaction, Performance, Activity, Communication & Collaboration, and Efficiency & Flow. It's designed to be flexible and applicable to new contexts like AI, unlike prescriptive metrics.
DORA Metrics
A set of four prescriptive metrics (Deployment Frequency, Lead Time for Changes, Mean Time to Recover, Change Failure Rate) used to assess the speed and stability of software delivery pipelines. While still useful for pipeline assessment, they are not sufficient on their own to measure AI's impact on developer productivity due to earlier feedback loops.
8 Questions Answered
DevEx is what it's like to build software day-to-day for a developer, including friction, workflows, and support. It's important because poor DevEx prevents even the best processes and tools from being effective, ultimately hindering software creation and business value.
AI can interrupt flow by requiring developers to prompt, review, and integrate generated code, but it can also contribute to flow by offloading detailed writing, reminding of context, and generating system diagrams, allowing developers to focus on higher-level goals.
Metrics like 'lines of code' are easily gamed by AI, which can generate verbose code and comments, leading to inflated numbers without actual productivity gains and potentially introducing technical debt. AI changes the nature of work, making these metrics less relevant for true output or quality.
Signs include consistently broken builds, flaky tests, overly long processes, difficulty requesting new systems or provisioning environments, and high switching costs that make it hard for engineers to move between tasks or projects within the organization.
The best first step is to talk to developers and listen to their experiences, asking about delightful and difficult points, frustrations, and areas of friction. This often surfaces low-lift improvements or unnecessarily complex processes that can be easily changed without significant engineering effort.
Impact can be measured by time savings, cost reductions (e.g., cloud costs from optimized builds), increased speed to value/market, risk reduction, and improved focus time for developers. For leadership, framing these in terms of accelerated revenue, profit margins, or market share is often most compelling.
The best approach depends on what leadership values most (e.g., market share, profit margin, velocity). Focus on measuring broad metrics like 'idea to customer' or 'idea to experiment' time, or quantify cost savings from reduced toil or optimized cloud spend, disclosing that both AI tools and DevEx improvements contribute.
Happiness is influenced by too many factors (work, family, hobbies) to be a useful metric for specific DevEx improvements. Instead, focusing on 'satisfaction' with specific tools, processes, or aspects of the job provides more actionable signal, as job satisfaction contributes to overall happiness.
17 Actionable Insights
1. Conduct Developer Listening Tour
Before implementing any tools or automation, talk to developers and listen to their daily experiences. Ask them to walk you through their day, identifying delightful points, difficulties, frustrations, slowdowns, and friction to uncover high-impact improvement areas.
2. Prioritize Process Improvement
Look for unnecessarily complex or slow processes within your team or organization. Often, simple changes, like replacing a multi-step manual approval with an email, can significantly reduce friction without requiring engineering effort.
3. Rethink Workflows with AI
Structure your days and work to leverage AI’s ability to handle context and generate diagrams, potentially making shorter work blocks (e.g., 45 minutes) more useful for deep work by offloading flow initiation to the machine.
4. Plan AI-Assisted Development Upfront
When using AI agents for coding, spend more time upfront planning the architectural components, stack, and general workflow. This systematic approach allows AI to work in parallel on pieces, leading to more production-ready code and better outcomes.
5. Evaluate AI-Generated Code Critically
Do not blindly accept AI-generated code. Actively evaluate it for hallucinations, reliability, and adherence to typical style and conventions, as AI models are non-deterministic and can introduce complexity or technical debt.
6. Measure AI Impact Strategically
Tailor your AI productivity measurement to what your leadership cares about most (e.g., market share, profit margin, velocity). Frame your metrics in terms of their priorities, such as tracking ‘idea to customer’ speed for market share or cloud cost savings for profit margin.
7. Use Surveys for DevEx Baseline
If you’re just starting to measure developer experience, use well-designed surveys to quickly get an overall view of challenges. Ask specific, single-focus questions, have developers prioritize their top three issues, and inquire about the frequency of impact.
8. Focus on Developer Satisfaction
Instead of broad ‘happiness’ surveys, measure developer satisfaction with specific tools, processes, and their job. High satisfaction contributes to better work and team collaboration, which is a more actionable metric.
9. Clean Up Test and Build Suites
Address breaking builds and flaky tests, as these are clear signs of friction. Cleaning up test and build suites can save significant developer time, reduce toil, and lower cloud costs by eliminating wasted compute cycles.
10. Provide Organizational Support for DevEx
As a business leader, provide structure and support for DevEx initiatives. Communicate priorities, celebrate wins, and ensure these efforts are not isolated projects, as they have huge potential returns for the business.
11. Improve Documentation for AI Tools
Actively work on writing and cleaning up documentation and code comments. AI agents rely on good data for training and grounding, so better documentation leads to better performance from your AI tools.
12. Bring Product Mindset to DevEx
Approach DevEx improvements and metrics with a product mindset. Identify problems for users, create MVPs, iterate rapidly with feedback, define a clear strategy, understand your ‘market,’ and establish success metrics and communication plans.
13. Leverage AI for Strategic Refinement
Utilize AI tools to refine your product strategy, messaging, experimentation methods, and total addressable market analysis. This allows for more informed discussions with key stakeholders and faster progress.
14. Partner with Data Science for Experiments
Before running large-scale A/B tests or experiments, partner with your data science team. Use AI to inform initial plans, then consult experts to ensure proper instrumentation, avoid privacy/security issues, and guarantee usable data.
15. Recognize Signs of Team Friction
Be alert for signs that your engineering team could move faster, such as constant complaints about breaking builds, flaky tests, overly long processes, difficulty switching tasks or projects, or general dissatisfaction with the system.
16. Use AI for Home Design Visualization
Employ AI tools like ChatGPT or Gemini to render home design ideas. Provide floor plans, existing room photos, and images of desired items, then prompt the AI to change elements like walls or furniture layout to visualize concepts.
17. Explore Cloud Code for Non-Coding Tasks
Investigate Cloud Code for various non-engineering use cases, such as cleaning up laptop storage. It can act as a personal assistant, performing diverse tasks on your computer beyond just code generation.
5 Key Quotes
Most productivity metrics are a lie.
Nicole Forsgren
We can ship trash faster every single day. We need strategy and really smart decisions to know what to ship.
Nicole Forsgren
So much of the time is now going to be spent reviewing code versus writing code.
Nicole Forsgren
Happy devs make happy code. They write better programs. They do better work. They're better team members and collaborators.
Nicole Forsgren
Hindsight is 20-20, but it's also really dumb.
Nicole Forsgren
2 Protocols
Seven-Step Process for Frictionless DevEx
Nicole Forsgren and Abhinoda (from the book 'Frictionless')- Start the journey: Conduct a listening tour, talk to people, synthesize learnings, and visualize current workflows and tools.
- Get a quick win: Start small, pick impactful projects, and share the results to build momentum.
- Use data to optimize work: Establish a data foundation, collect new data, and use surveys for fast insights.
- Decide strategy and priority: Use evaluation frameworks to determine which remaining issues to address next.
- Sell your strategy: Get feedback and clearly communicate why the chosen strategy is the right one.
- Drive change at your scale: Implement changes, leveraging either local grassroots efforts or global top-down strategies depending on your scope of control.
- Evaluate your progress and show value: Measure the impact of changes and loop back to reassess and continue improvements.
Designing Effective Developer Experience Surveys
Nicole Forsgren- Ask about satisfaction with specific tools or processes, rather than general happiness.
- Ask respondents to prioritize a limited number of top issues (e.g., three) to avoid messy data.
- For each prioritized issue, ask about its frequency (e.g., hourly, daily, weekly, quarterly) to understand its impact.
- Include an open-text option for additional feedback.
- Ensure each survey question is singular and clear, avoiding asking multiple questions in one (e.g., 'slow or complicated').
- Consult with survey design experts or use AI tools to refine questions and ensure they yield actionable data.