Thursday, April 18, 2024
HomeSoftware DevelopmentMeasuring Developer Productiveness through People

Measuring Developer Productiveness through People

Someplace, proper now, a expertise government tells their administrators: “we
want a option to measure the productiveness of our engineering groups.” A working
group assembles to discover potential options, and weeks later, proposes
implementing the metrics: lead time, deployment frequency, and variety of
pull requests created per engineer.

Quickly after, senior engineering leaders meet to evaluation their newly created
dashboards. Instantly, questions and doubts are raised. One chief says:
“Our lead time is 2 days which is ‘low performing’ in keeping with these
benchmarks – however is there truly an issue?”. One other chief says: “it’s
unsurprising to see that a few of our groups are deploying much less typically than
others. However I’m undecided if this spells a chance for enchancment.”

If this story arc is acquainted to you, don’t fear – it is acquainted to
most, together with among the largest tech corporations on the planet. It isn’t unusual
for measurement packages to fall quick when metrics like DORA fail to supply
the insights leaders had hoped for.

There’s, nevertheless, a greater strategy. An strategy that focuses on
capturing insights from builders themselves, moderately than solely counting on
primary measures of velocity and output. We’ve helped many organizations make the
leap to this human-centered strategy. And we’ve seen firsthand the
dramatically improved understanding of developer productiveness that it

What we’re referring to right here is qualitative measurement. On this
article, we offer a primer on this strategy derived from our expertise
serving to many organizations on this journey. We start with a definition of
qualitative metrics and learn how to advocate for them. We comply with with sensible
steering on learn how to seize, observe, and make the most of this information.

In the present day, developer productiveness is a crucial concern for companies amid
the backdrop of fiscal tightening and transformational applied sciences similar to
AI. As well as, developer expertise and platform engineering are garnering
elevated consideration as enterprises look past Agile and DevOps
transformation. What all these considerations share is a reliance on measurement
to assist information selections and observe progress. And for this, qualitative
measurement is vital.

Be aware: once we say “developer productiveness”, we imply the diploma to which
builders’ can do their work in a frictionless method – not the person
efficiency of builders. Some organizations discover “developer productiveness”
to be a problematic time period due to the best way it may be misinterpreted by
builders. We suggest that organizations use the time period “developer
expertise,” which has extra optimistic connotations for builders.

What’s a qualitative metric?

We outline a qualitative metric as a measurement comprised of information
supplied by people. It is a sensible definition – we haven’t discovered a
singular definition inside the social sciences, and the choice
definitions we’ve seen have flaws that we talk about later on this

Determine 1: Qualitative metrics are measurements derived from people

The definition of the phrase “metric” is unambiguous. The time period
“qualitative,” nevertheless, has no authoritative definition as famous within the
2019 journal paper What’s Qualitative in
Qualitative Analysis

There are numerous definitions of qualitative analysis, but when we search for
a definition that addresses its distinctive function of being
“qualitative,” the literature throughout the broad subject of social science is
meager. The principle cause behind this text lies within the paradox, which, to
put it bluntly, is that researchers act as in the event that they know what it’s, however
they can’t formulate a coherent definition.

An alternate definition we’ve heard is that qualitative metrics measure
high quality, whereas quantitative metrics measure amount. We’ve discovered this
definition problematic for 2 causes: first, the time period “qualitative
metric” contains the time period metric, which suggests that the output is a
amount (i.e., a measurement). Second, high quality is often measured
by ordinal scales which can be translated into numerical values and
scores – which once more, contradicts the definition.

One other argument we’ve heard is that the output of sentiment evaluation
is quantitative as a result of the evaluation ends in numbers. Whereas we agree
that the information ensuing from sentiment evaluation is quantitative, based mostly on
our authentic definition that is nonetheless a qualitative metric (i.e., a amount
produced qualitatively) except one have been to take the place that
“qualitative metric” is altogether an oxymoron.

Other than the issue of defining what a qualitative metric is, we’ve
additionally encountered problematic colloquialisms. One instance is the time period “gentle
metric”. We warning towards this phrase as a result of it harmfully and
incorrectly implies that information collected from people is weaker than “onerous
metrics” collected from methods. We additionally discourage the time period “subjective
metrics” as a result of it misconstrues the truth that information collected from people
could be both goal or subjective – as we talk about within the subsequent

Qualitative metrics: Measurements derived from people
Kind Definition Instance
Attitudinal metrics Subjective emotions, opinions, or attitudes towards a selected topic. How glad are you together with your IDE, on a scale of 1–10?
Behavioral metrics Goal details or occasions pertaining to a person’s work expertise. How lengthy does it take so that you can deploy a change to manufacturing?

Later on this article we offer steering on learn how to gather and use
these measurements, however first we’ll present a real-world instance of this
strategy put to apply

Peloton is an American expertise firm
whose developer productiveness measurement technique facilities round
qualitative metrics. To gather qualitative metrics, their group
runs a semi-annual developer expertise survey led by their Tech
Enablement & Developer Expertise crew, which is a part of their Product
Operations group.

Thansha Sadacharam, head of tech studying and insights, explains: “I
very strongly imagine, and I feel quite a lot of our engineers additionally actually
recognize this, that engineers aren’t robots, they’re people. And simply
primary numbers would not drive the entire story. So for us, having
a very complete survey that helped us perceive that total
developer expertise was actually necessary.”

Every survey is shipped to
a random pattern of roughly half of their builders. With this strategy,
particular person builders solely have to take part in a single survey per yr,
minimizing the general time spent on filling out surveys whereas nonetheless
offering a statistically important consultant set of information outcomes.
The Tech Enablement & Developer Expertise crew can be chargeable for
analyzing and sharing the findings from their surveys with leaders throughout
the group.

For extra on Peloton’s developer expertise survey, hearken to this

with Thansha Sadacharam.

Advocating for qualitative metrics

Executives are sometimes skeptical concerning the reliability or usefulness of
qualitative metrics. Even extremely scientific organizations like Google have
needed to overcome these biases. Engineering leaders are inclined towards
system metrics since they’re accustomed to working with telemetry information
for inspecting methods. Nevertheless, we can’t depend on this similar strategy for
measuring individuals.

Keep away from pitting qualitative and quantitative metrics towards one another.

We’ve seen some organizations get into an inner “battle of the
metrics” which isn’t a superb use of time or vitality. Our recommendation for
champions is to keep away from pitting qualitative and quantitative metrics towards
one another as an both/or. It’s higher to make the argument that they’re
complementary instruments – as we cowl on the finish of this text.

We’ve discovered that the underlying explanation for opposition to qualitative information
are misconceptions which we handle under. Later on this article, we
define the distinct advantages of self-reported information similar to its capacity to
measure intangibles and floor crucial context.

False impression: Qualitative information is just subjective

Conventional office surveys sometimes deal with the subjective
opinions and emotions of their workers. Thus many engineering leaders
intuitively imagine that surveys can solely gather subjective information from

As we describe within the following part, surveys may also seize
goal details about details or occasions. Google’s DevOps Analysis and
Evaluation (DORA)
program is a wonderful concrete

Some examples of goal survey questions:

  • How lengthy does it take to go from code dedicated to code efficiently
    operating in manufacturing?
  • How typically does your group deploy code to manufacturing or
    launch it to finish customers?

False impression: Qualitative information is unreliable

One problem of surveys is that individuals with all method of backgrounds
write survey questions with no particular coaching. In consequence, many
office surveys don’t meet the minimal requirements wanted to supply
dependable or legitimate measures. Effectively designed surveys, nevertheless, produce
correct and dependable information (we offer steering on how to do that later in
the article).

Some organizations have considerations that individuals could lie in surveys. Which
can occur in conditions the place there may be concern round how the information shall be
used. In our expertise, when surveys are deployed as a device to assist
perceive and enhance bottlenecks affecting builders, there isn’t a
incentive for respondents to lie or sport the system.

Whereas it’s true that survey information isn’t all the time 100% correct, we regularly
remind leaders that system metrics are sometimes imperfect too. For instance,
many organizations try to measure CI construct occasions utilizing information aggregated
from their pipelines, solely to seek out that it requires important effort to
clear the information (e.g. excluding background jobs, accounting for parallel
jobs) to supply an correct end result

We’re releasing this text in installments. Future installments will
describe the 2 forms of qualitative metrics, clarify their advantages,
and go into element on learn how to seize them.

To seek out out once we publish the following installment subscribe to the
web site’s
RSS feed, Martin’s
Mastodon feed, or
X (Twitter) stream.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments