Here’s the good news: at least you’re not hearing voices. The bad news: there’s a swarm of numbers in your head telling you what to do. I’m talking about app development metrics and you—like so many others—are possessed by what’s “on paper” about your apps.
In many companies, app development leadership devoutly obey their metrics. For these teams, productivity measurements guide all app development decisions and are the most critical key performance indicators (KPIs) for developers. But this practice harbors major doubts that should be questioned:
- Is it wise to unquestionably trust productivity metrics?
- Are there side effects to using production metrics to evaluate team performance?
- What is lost when metrical data—not concrete reality—dominates development?
When does managing by the metrics stop being productive, and start being a straitjacket?
The metrics made me do it! An over-reliance on app development metrics causes unintended consequences and restrains app developer productivity. App development projects that get blinded by benchmarks pose a risk to your small business.
Recommendation: It is critical that your small business not only identifies important metrics, but ensures that these metrics are being used wisely to have their intended result: empower app development teams and boost productivity. In this article we will identify how an injudicious use of metrics does not lead to project greatness, but in fact, has the opposite effect: damages team morale, leads to perverse incentives and behaviors, and creates artificial, inaccurate data.
Here’s what we’ll cover:
Managing by the metrics is a straitjacket on app development teams
Metrics used as a bludgeon at review time tank team morale
Developers game the system or lie in the face of punishing metrics
Best practices are avoided in fear of ugly numbers
Align metrics with business goals for better projects and team performance
When metrics take the wheel and drive your project forward, it’s like cruising the highway affixed to your GPS or speedometer but never bothering to look up and watch the road. The problem with metrics is that they don’t always tell the full story.
The other problem with metrics is that—good or bad—they have a predictable effect on teams.
“People’s behaviors change in predictable ways to provide the answers management asks for when metrics are applied.” Hoffman said.
Developers change their tact to satisfy the metrics. They change their behavior to succeed under the new parameters of success and create passing data. This can lead to false information and bad practices. Ultimately, metrics lose all meaning when teams adapt to game the system.
Relying too much on metrics is a problem
So what if your cyclomatic score is out of this world when the reality is that most people are turned off by your lackluster UI. By the numbers, everyone should love your app—but what if they don’t? Does it still make sense to use metrics as a straitjacket on app development teams?
“When you try to manage by the numbers, be they test scores, sales activity, or productivity measures, you drive towards mediocrity. Quantitative (numerical) measurements alone will never make an organization great, because it is the qualitative (non numerical) elements of performance that achieve greatness.” McLeod said.
To explain this better, here are three popular metrics and how overdosing on them can lead to project failure.
What is it: Velocity is how fast an application or capability is delivered in a certain period of time. Velocity is measured by the amount of work (e.g., projects complete, tasks, or point systems) over a preset interval, and compared relatively to the speed at which similar work is/was completed.
The metrics made me do it! When management teams misuse velocity as a success metric, according to Gartner Research VP, Sean Kenefick, it becomes a bludgeon at review time. Punitive metrics beat down team morale and developers respond negatively in one or more of the following ways:
- The acceptance criteria is changed to allow project completion at a higher velocity—but the overall product is of lesser quality and may not be ready for delivery.
- Teams alter past results that are used as reference for future velocity target goals. They then favorably race against/lap old times but do the same amount of work.
- Developers scramble to complete projects to meet velocity deadlines, get overloaded, and burnout.
Recommendation: Velocity should only be used as a tool to plan iterations and give estimates, never as a performance indicator for teams, or worse, individuals.
Metric: Defect escape rate
What is it: Defect escape rate is calculated as a ratio of how many bugs are tracked during pre-production testing vs post production. Defect escape rate is the degree of bugs or glitches that avoid being stomped out in testing, but instead survive into the later stages of app development.
The metrics made me do it! Bugs are the bane of app developers, and in organizations where defect escape rate is heavily penalized, limiting glitches results in illicit behavior. For example, to reduce the number of defects, developers may count bugs as duplicate or report them only after the problem was already fixed.
According to Douglas Hoffman, in his report, The Darker Side of Metrics, Hoffman witnessed collusion between Q/A testers and developers, and cites this as a common side effect of punishing bug defect escape metrics. Hoffman observed the following behavior:
“Testers withheld defect reports to befriend developers who were under pressure to get the open defect count down.”
“In another case the testers would record defects when the developers were ready with a fix to reduce the apparent time required to fix problems.”
“A test group took heat for not having found the problems sooner (to give the developers more time to fix the problems).”
Recommendation: Do not use bug tracking for punishment or reward. It is better to think on a grander scale than spot check: if defect escape rate is higher than expected, this may be an indication you’re app development is moving too fast or perhaps testing needs to be improved.
Metric: Q/A tests passed
What is it: Q/A or quality assurance is important step in app development. Software goes through a litany of tests to evaluate its readiness for delivery. Software must pass several of the following tests to be checked complete:
- Satisfies design requirements set at outset
- Responds correctly to inputs and, when applicable, functions under stress testing
- Performs its functions within an acceptable timeframe
- Reaches usability standards.
Q/A tests passed is recorded as either a number of tests passed, or a percentage of tests running vs tests passed.
The metrics made me do it! Many development teams lack pre-defined testing. What constitutes an agreed test, pass, fail, or proper procedure to run a test is often debatable. Most times, tests are created dynamically, and this makes Q/A tests passed metrics vulnerable to manipulation.
For example, developers may redefine what a “test” is in order to increase the percentage of passing. In pursuit of better numbers, managers may apply more liberal or conservative definitions of pass/fail, number of tests, or types of tests needed before release—as it suits them.
Recommendation: Prohibit your development teams from singling out favorable metrics or avoiding certain metrics to pad results. According to Gartner Research VP, Sean Kenefick, metrics should be used to rate productivity not progress:
“When metrics are used only to measure productivity, the staff members try to cover themselves by only recording measurements that show them in a good light. When these types of metrics cherry-picking occur, teams greatly curtail improvement possibilities and fail to shine light on problem root causes,” Kenefick said.
To avoid making this misstep, use metrics only as a means for continuously improving and not “as a bludgeon at review time.”
1. Forget your metrics for a moment. What is it that you’re trying to achieve with your app development projects? Redefine or identify your business goals. Dig deeper than “make a popular and profitable app,” and think about specific aims like improving UX, customer service, or operational efficiency.
2. Metrics can be a powerful tool to gain insight and assess trends in your app development. This information can be used to improve team performance and create higher quality products. However, it is crucial to use the right metrics. This means avoid using metrics punitively, or else risk running-over team morale and causing evasive maneuvers that harm the quality of deliverables.
3. Be alert and be careful. Consider whether the metrics that your app development teams rely on are achieving the desired meanings, and account for any unintended side effects. Your metrics may be doing more harm than good.