What are developer experience metrics?

Good software development teams are focused on outputs, and can bring key metrics to bear that illustrate just what the engineering organization is building on a daily, monthly and yearly basis. Developer productivity is often assessed retrospectively: if the team is hitting DORA metrics, we assume everything in the lifecycle before production is sound. But the best teams dig deeper, and aim to solve the problem backwards as well as forwards by looking at the process as well as the results. These teams invest time and resources in measuring and improving developer experience.

What is developer experience?

So what is Developer Experience, or DevEx? Origins of the term aren’t completely agreed upon, but one of the earliest known publications of the term was via Jeremiah Lee in a 2011 article in UX magazine. More recently, in a paper from the authors who brought us the SPACE framework, defined as focusing on “the lived experience of developers and the points of friction they encounter in their everyday work.”

These definitions are somewhat vague, often necessarily so, since developer experience is subjective to the individuals, teams, environments and companies involved. It might be useful then, to consider it in context with another more familiar practice, DevOps. DevOps focuses more on the outcomes, whereas DevEx focuses on the work it takes to deliver said outcomes. If DevEx is how vital signs are captured—the comfort of the blood pressure cuff, the invasiveness of the thermometer, how the paperwork is laid out and the results are displayed... then DevOps relates to the vitals captured, who is keeping that sensitive information safe, where that data is stored, and even the broader health management, insurance processing, etc. While it’s necessary for both to be defined in somewhat subjective terms, the importance of DevEx in the development process is clear and objective.

What are developer experience metrics?

It’s helpful to think about DevEx metrics in terms of what they are not. These metrics are not focused on developer productivity for its own sake, or based around outputs and results as a means to measure overall health. Where DORA metrics offer data on software delivery, DevEx metrics look at the process by which it is delivered.

Traditional metrics like code quality and performance are important, but they don’t capture the full development experience. Hitting numbers in the short-term could represent a highly efficient engineering team running smoothly, or one staffed by disillusioned developers considering leaving the organization due to burnout.

DevEx metrics focus on the daily experiences of the developer. Is high developer productivity due to short-term targets or high morale? Does the development process foster a sense of ownership and mission, or is it frustrating with a high cognitive load? Does management have feedback loops in place to understand how to get the most out of the development team?

Why developer experience are metrics important

By asking questions focused on "how?", “who?” and “why?”, DevEx metrics can get to the root of how a team operates and what makes them tick. Tracking the right data helps get to a place where team members feel valued at their work and have the confidence to innovate and be creative. Paradoxically, removing productivity as the sole focus actually increases developer productivity significantly, while reducing churn.

The benefits of this approach are first felt at the individual and team level, before gradually filtering up to make a difference across the entire organization. Better DevEx means happier and more productive employees, who will want to act as ambassadors for the organization and help with recruitment and retention.

Better work environment

While there have been a few ill-constructed (and ill-fated) studies on measuring developer experience, a recent study we do recommend focuses more on the difference just making the investment can have in improving outlook for the team. A better work environment doesn’t need to mean coming up with all the answers, but showing real attention to the problem.

Cost reduction

Getting DevEx right means cutting lead times, reducing cognitive load for developers and ensuring they have a developer environment optimized for deep work and problem solving rather than administration and bureaucracy. This increases output per individual and emphasizes team members’ wellbeing, reducing costs on wasted time while minimizing churn.

Optimizing resource allocation

Anyone who has worked with good developers knows that a happy dev is a productive dev. Programming attracts people who crave intellectual challenge, and a good developer experience involves solving complex problems rather than waiting for tickets and approval. Solving for DevEx means getting your programmers working on the most important challenges, which helps with resource optimization and keeps your productivity high.

Supporting accountability

An efficient organization with good DevEx means there is nowhere to hide. Accountability is clearly defined and devs take pride in their work, so anyone coasting or contributing negative work will be quickly identified. This keeps standards high and helps to weed out poor performers. Engineering leadership should be cautioned against individual stack ranking by inaccurate metrics, however, and should observe metrics at the team or organizational level. Work such as optimization of dev tooling or supporting other team members is often not captured, which at scale can wrongly identify team members as unproductive.  

Key components of positive developer experience

Anyone working in software should be comfortable with data, so gathering and tracking metrics through surveys or monitoring is a straightforward step. These KPIs should be carefully considered and regularly reviewed. Because you are looking to measure a subjective quality, there is no single way to go about it. Consider capturing this data during a workflow step that already requires input from a developer, such as a pull request or commit, to avoid creating additional interruptions.

Metrics should be quantitative and qualitative, touching on tangible qualities such as usability, efficiency and effectiveness, as well as more intangible considerations around satisfaction and engagement.

What are the key components of a positive developer experience?

Best practice DevEx is about creating a culture and working environment rooted in technical excellence and empathy. This ranges from high standards of documentation and robust tooling, to a strong sense of mission, good relations with the developer community and processes for optimizing flow states.

It's important to bear in mind that good DevEx has a flywheel effect. Developers who feel valued will help to optimize processes, while those who consistently achieve flow state will take greater pride in their work and contribute more to a healthy culture.

There is a subjective element to selecting DevEx metrics based on your team's individual needs, but the following list includes some metrics to consider.

What are some common developer experience metrics?

Common developer experience metrics include:

1. Time to First Hello, World: getting to the first "aha!" moment quickly demonstrates effective onboarding.

2. Developer Satisfaction Score (DSS): survey data demonstrating how satisfied developers are with a platform, tool or service can act as a useful approximation for measuring DevEx in real-time.

3. Time to Onboard: quicker onboarding isn't always better, but it is useful to pair onboarding time with DSS.

4. API Response Time: slow responses increase frustration and reduce meaningful work.

5. Documentation Quality/Completeness: best practice documentation resolves issues faster and increases efficiency.

6. SDK/Library Usability: efficient use of SDKs and libraries reduces cognitive load considerably and speeds up code review.

7. Tooling Availability: getting tooling right means that developers can solve problems quickly without facing the paradox of choice or contributing to shelfware.

8. Platform Stability: high platform uptime and reliability allows developers to spend more time in flow state and focus on solving problems.

9. Developer Productivity Engineering (DPE): metrics such as build and test cycle durations and test set flakiness.

10. IDE Integration: quick and easy integration of tools into an IDE provides a smooth experience for developers.

11. Community Support/Engagement: knowledge work is collaborative, and strong community engagement helps share best practice and foster a sense of mission.

12. Code Readability and Reusability: good quality code increases efficiency and reduces cognitive load for developers.

13. Time to debug/resolve issues: while debugging is crucial to a developer's role, making the process as efficient as possible improves cycle time and deployment frequency. This increases developer morale.

14. Error rate/error frequency: like debugging, errors will never be eliminated, but if error rates are trending down then DevEx should be trending up as technical debt is kept to a minimum.

15. API usage volume: efficient APIs make for happy developers.

16. Feature adoption rate: developers take pride in good work, and a quick adoption rate for users demonstrates good functionality and a high quality developer experience.

17. Feedback response time: quick responses improves user experience and customer satisfaction, ultimately contributing to a better DevEx by minimizing toil from tickets and requests.

18. Time to first contribution of new hire: quick contributions indicate best practice onboarding of developers, which is a crucial component of developer experience.

Best practice for implementing and using developer experience metrics

Gathering metrics on developer experience is an important step, but this is only useful if the data is regularly reviewed and used as a catalyst for continuous improvement. These metrics should inform decisions that enhance tooling, streamline processes and remove bottlenecks and roadblocks to a productive and satisfying developer experience.

• Define clear objectives: because DevEx includes subjective elements, it's important to start by defining objectives that align with the culture and goals of your organization. This should follow your specific needs, whether that is reducing cognitive load, increasing retention or building a culture of ownership. You can't build a better experience until you know what that looks like within your organization.
• Choose the right metrics: the data you collect should flow from your objectives. It helps to communicate extensively with your team before defining these and ensure you have their buy-in. You should also consider some existing frameworks (see here) when deciding on metrics.
• Establish benchmarks: once you have a feel for your goals and how to reach them, use benchmarks to define what good looks like. These might include references to external market data, but should be primarily based on best practice as defined by your own circumstances. Developers are the primary stakeholders here, but consider the needs of management and investors, as well as user experience, while laying out targets.
• Implement measurement tools: DevEx metrics exist within software development and it is crucial that they are embedded in this wider universe. Measurement and review of this data needs to take place within existing development workflows. It should be seen as an extension of development, and the tools chosen should support this.
• Encourage feedback: buy-in from developers is crucial to getting DevEx right. If you're not getting appropriate feedback, it represents a DevEx red flag in itself. Provide the culture and incentives to get regular feedback from developers and ensure the metrics and objectives you choose have popular support.
• Focus on continuous, iterative improvement: Improving DevEx is a continuous process and not a project that can be shelved after it achieves short-term results. If you try to dramatically overhaul processes and operations you risk creating pushback from the stakeholders you're trying to reach. It's better to look at DevEx as something that is improved gradually and continuously.

What role can internal developer portals play?

The risk with addressing developer experience badly is that it can create additional cognitive load, and worsen the problem it sets out to solve.

Adopting an internal developer portal (IDP) allows you to build a system of record that enables best practice implementation of tools and platforms, catalogs and libraries, scorecards and initiatives, and much more. This reduces cognitive load significantly when building the codebase, which solves for one of the central challenges to building a good DevEx.

How can Cortex help?

Cortex is an IDP that can track and analyze DevEx metrics in the context of software development and productivity more broadly. It is built to offload toil, busywork and bureaucracy, and in doing so to reduce cognitive load for development teams. These use cases ultimately help to enable flow states and enhance collaboration, based on each engineering team's individual needs.

Our IDPs provide golden paths to developers, standardizing processes and using automation so that developers are spending their time solving problems rather than fighting fires.

Integrations and plugins for DevEx

Embedding DevEx in existing workflows means using tools and plug-ins effectively. Our IDP is differentiated from developer portals because we put an emphasis on accessing real-time, custom data - the kind that powers subjective metrics related to developer experience.

Once you have decided your DevEx goals and built out associated metrics, you’re left with the technical challenge of gathering and analyzing this data in the context of continuous improvement. With Cortex, you can sync this data using any of our out-of-the-box integrations, or use plug-ins to build homegrown apps and embed them within the IDP’s UI.

Our Eng Intelligence product is particularly relevant as it surfaces data from across the SDLC, highlighting data that is particularly relevant for DevEx. This allows you to spot bottlenecks and investigate the causes of problems, not just symptoms and outcomes.

Cortex can also help you to build DevEx goals into continuous improvement through the use of custom scorecards that can track progress for relevant experience metrics. This offers a tangible way to demonstrate and report on improving this crucial target.

Finally, our IDP communicates priorities the way you want. The developer homepage can use Slack and Teams notifications to communicate priorities, so developer experience can get the priority that it deserves within the engineering organization.

‍Book a demo today to learn more about how Cortex can help you improve your Developer Experience!