Why Choosing the Right Measurement Frequency is Crucial

VPs of Engineering often make decisions on various vectors such as business alignment, engineering execution, customer satisfaction, developer happiness, and outsourcing projects to vendors. Highly regarded VPs of Engineering seldom make these decisions based on gut and intuition. They must rely on accurate data and metrics. 

Key questions that leaders need to ask include how often to measure, what are the right inferences to draw then, when to respond to changes in data, and when to take action. These are no simple answers but a good starting point for these is to have a well thought-out strategy on the metrics to measure. Equally important is the measurement frequency and knowing exactly when to take action in response to the metrics.

It’s important to differentiate between metrics that have a natural frequency and ones that don't. Metrics such as sprint metrics and release metrics have a natural frequency. For example, if your metrics relate to sprints, then its natural frequency is the sprint length (2, 3, or 4 weeks depending on the organization or product). Likewise, a release may have a natural frequency of 1, 2, or X months depending on the organization and product.

However, an engineering leader must also monitor several other metrics that do not have a natural frequency. How often should one measure developer productivity or developer happiness? Is it appropriate to measure it weekly, monthly, or quarterly? In such situations, choosing the measurement frequency is extremely critical because based on how frequently a metric is consumed and acted upon could lead to varying conclusions, actions, and results.

Two obvious pitfalls in choosing the frequency of measurement are either the metric is measured and acted upon too soon, or we let too much time pass before we measure and act upon the metric. Both can be problematic. 

Measuring and acting too soon 

• Not enough time is given for the impact of decisions to percolate or for a meaningful conclusion to be reached.
• We start reacting to near-term variations and make decisions with insufficient data or overcorrecting for situations too soon.

Too much time between measurements and subsequent action

• Increases the risk that the measure is influenced by many factors and you cannot attribute results to specific actions you’ve taken to improve the metric
• A second problem of waiting too long is that we have allowed the problems to creep and build, thereby wasting valuable time and resources. If we had measured it sooner then problems could be detected sooner and we could have taken corrective steps. This is the bane of today’s systems such as Jira Dashboards or Tableau dashboards - by the time you can measure a metric and act, the value of action is already lost.

Ironically, the answer is it does not matter what metric you choose, there isn’t one ideal frequency. There are actually TWO frequencies for most metrics.

We suggest that you need both a short-term measurement and a medium-term measurement. This might seem counterintuitive or a letdown but there are very sound reasons and value in using both a short- and medium-term measurement for a metric. Here are the benefits of each.

Benefits of Short-Term Measurement

• Quick pulse check on the decision or action
• Minimizes the impact of concomitant variables. The longer our measurement frequency, the more concomitant variables come into play. Thus making it hard to understand the cause and effect

Benefits of Medium-Term Measurement

• True measure of the impact of an action/decision. Measures whether the changes mattered to the business
• Regardless of the impact of other variables, how are we doing on the target metric. Aka the “no-excuses” state

Then what is short term and medium term? Giventhat we need to measure in short term and medium term, the definition of short and medium term varies based on the metric being measured. This means we need great flexibility in how we slice data, view, aggregate, and respond to insights. Engineering leaders need a system that satisfies these criteria.

• Correlates data from diverse DevOps tools in a unified manner. I.e. it is not just about Git, Jira or some other siloed system. You also need the ability to correlate data from systems such as CI-CD, Security, Customer Support into a unified view because the metrics you need to measure across multiple DevOps tools.
• Offers the ability to choose arbitrary time frames. Since the frequency of insights and metrics vary widely based on the issue being addressed, it stands to reason that we must be able to choose different time frames and immediately view the data and insights based on our chosen timeframe. In other words, systems that present aggregated data based on fixed timeframes are of little use, in fact detrimental to the decisions you make, 
• Provides context. Data devoid of context is highly suspect and leads to sub-optimal outcomes. You need the ability to slice the data based on your organization, division, teams, sprints, releases, products or any other segmentation that are relevant to your business.
• Allows ad-hoc exploration and what-if analysis by executives themselves. This is often ignored but is extremely important because (a) it cuts down weeks of cycle time between a question and answer (b) ensures that the secondary set of questions and answers can be quickly obtained in seconds rather than days or weeks. This enables a deep contextual analysis of the data, what-if analysis, and leads to well-thought out decisions.

To put it bluntly, solutions or products that provide pre-canned metrics based on fixed time frames and with minimal context are actually misleading you into decisions that are sub-optimal, perhaps even detrimental.

Utilizing a platform such as the Harness Software Engineering Insights can provide support for the diverse DevOps toolsets along with the flexibility to adapt, combine and formulate analytics appropriate to each given audience. Harness offers out-of-the-box dashboards for standardized software metrics such as DORA, and it also provides the flexibility and adaptability to not only visualize the metrics required in your organizations but also provide predictive analytics and orchestration to take action upon those analytics.

Check out a live demo of how we can help you transform your engineering organization!