Confession; I find measuring stuff a fascinating challenge. Sometimes measuring is straight forward, like the fuel gauge on your car, but often times it’s more complex. The volt meter in your car quietly drains the battery while measuring its health. The motivation survey in your inbox will quietly change your motivation.
Its termed the observer effect, where the act of measuring affects the thing you’re trying to measure. Measuring, or even just assessing, the output of groups is similarly taxing, even the act of posing a question can project your own biases. Last year I got interested in measuring the progress of steps towards DevOps culture. At Nokia Entertainment’s MixRadio development emporium we’d had good continuous delivery tools in place for months, but weren’t certain our culture continued to improve. Complacency was a risk, but we couldn’t tell how large. It seems one of the hardest parts of change is keeping things going in the period between that initial burst of enthusiasm and when practices truly take root as habit.
So, I shared my thoughts at DevOps Days London, and received some really useful feedback from the crowd there. I’ll let you into a secret though: I wasn’t happy, it just wasn’t rigorous enough. Paul Swartout and I created metrics focused on adoption, we wanted simple, no cost methods that anyone could use, without needing a big budget or corperate sponsorship. We called them ‘Vital Signs‘ and they comprised; Cycle Time, Shared Purpose, Motivation, Collaboration and Effectiveness.
The main aim was to benchmark, ready to see which way our desired ways of working were trending. However I wanted to capture elusive things: just how ready was the team to ignore organizational setup and work together? We also didn’t want to bias for DevOps, Scrum, Kanban or any of our other preferred methods, if someone found a better way, we wanted to learn.
The art was to find metrics in which these desirable behaviours surface, and of them only cycle time was measurable with any consistency. We learnt an awful lot from the other metrics, particularly free form comments. The problem was all that prose was impossible to graph, impossible to track.
Frustratingly those things that are hard to measure are amongst the most critical. They often indicate how long you can sustain a pace or practice. It is very easy to focus exclusively on productivity, but you might be slowly killing your workforce, as Amazon recently discovered. In general engineering teams aren’t a temporary construct, they need to be looked after for longer than the holiday season. Engagement and well-being over time are going to drive quality and productivity as much as anything else. (Pseudo science here).
So why raise this now? Well, I was enjoying a coffee at the DevOps Café , and was interested to hear a side remark about metrics by the ever eloquent Damon Edwards and John Willis. They described the following as their ideal set of metrics:
- Cycle Time – From customer report to change in production.
- Mean Time To Detect (an issue)
- Mean Time To Repair (or make a change)
- Quality at source – or escape distance, how far do errors get before they are noticed? Worst case: customer.
- Repetition Rate – Does the same issue keep happening, or are we learning?
Used together, these are just genius, because it’s very hard to achieve good results without a healthy productive relationship between teams. Furthermore it doesn’t matter how you describe what you’re doing: DevOps, OpsDev, Agile, Scrum or one great big group hug – those metrics don’t test adherence to a methodology. I guess mavens like these will often drive common practice, and these metrics are very much evident in Puppet Lab’s excellent surveys, (some words on the magic here) (for DevOps archaeologists* early John Allspaw thoughts here).
But there is one metric which went un-mentioned, my old measurement nemesis; engagement. I suspect that you could be proudly watching all the above metrics trending positive, and be rudely awoken by burn out or a rash of exit interviews. To avoid surprises shouldn’t the impact of change on key people be monitored too? Retention is a favourite for this. A good indicator, but actually people leave for a lot of other reasons. If someone departs for a role closer to the best trails it should not be seen as the first sign of DevOps culture crumbing into nothingness.
So while it seems DevOps operational metrics are mature, there is more work to be done to understand if we’re getting results and simultaneously creating a healthy, sustainable, culture. That suggests three dimensions to measure for DevOps, and other flavours of adoption.
- Efficiency – Our key measures like cycle time, and mean time to XYZ, are they improving?
- Effectiveness – Is the right kind of work being done, and steering the team towards success in their organization’s field?
- Engagement – Have we created an environment for people to be at their best? Are we making the most of our talent?
Of course, these three need to be balanced – focus on one could easily be to the detriment of the others. Measuring engagement, culture change and people things will always be hard, and methods flawed, but we should avoid measurement inversion, and strive to measure things not just because they are simple, but because they are valuable.