What do points mean?

www.flickr.com/photos/vpickering/14669824798/

I’d like to think that my previous posts have provided an informative and well balanced, ego free commentary on pertinent topics such as learning, change and even improv in agile.  Well, this one is different; it’s good old fashioned, vitriolic rant.  It’s about velocity, or at least what people do to velocity, a well meaning, innocent and largely defenseless concept.  The other thing this post isn’t is a comparison or comment on the value of velocity or estimating in general.  You’ll find plenty of good #NoEstimates conversation elsewhere.

Time and time again I see velocity abuse:
– Equating velocity to volume of output (more points equates to more productivity)
– Using velocity as a target, linked to incentives (The Scrum master shouts: “what do points mean? – prizes” )
– Assuming that velocity is a real number, double the engineers, double the velocity, right?

There are plenty of good velocity explanations around, the way I see it velocity is part of a system for:
a) Improving group estimation ability (mostly by encouraging exploration of work, and an appreciated of other’s roles)
b) Forecasting the rate work will be carried out by the team.

 As such, velocity needs to be honest and without interference, otherwise the outputs of neither estimation or forecasting activity can be trusted.  

Increase your velocity; Go big or go home!
It seems there is an obsession with velocity in volume, if a team knocks over 20 points one iteration, it should better itself in the next, 25 points next time anyone?

Consider however, a situation where a team doubles its velocity compared to the previous two iterations, does that really indicate double productivity?  There are numerous factors which could contribute, perhaps they are not working in a sustainable fashion, or corners are being cut.  Was a large amount of unfinished work rolled over from the previous iteration?  In these circumstances we often have what looks like high volume output, but is it to the detriment of other (typically operational) concerns?  What if a team halved its velocity in the last iteration?  Is the team failing, or is the organization not providing what they need to succeed?  Did they change the value of points to align with demand?  Is the team hitting its predicted point every single sprint? Suspicious to say the least.

The key point is that a team’s velocity fluctuates over time, and there is often considerable variation as a team forms, and starts to understand it’s work, constraints and processes.

Velocity with benefits
To some extent velocity is a target for the team, but one that is most effective when set and owned by the group, rather than linked to incentives.  Pushing velocity onto a team, demanding or targeting an increase is a dangerous, counterproductive practice. Strongly linking velocity to incentives compromises its primary value as an estimation tool. Organisations generally hire the smartest people available, do they somehow think those people won’t be smart enough to game velocity given even the smallest incentive?

“The moment a measure becomes a target, it ceases to be a measure” – Goodheart’s Law

A further reason not to link velocity to incentives is the frame of mind it encourages during an iteration. It seems preferable to focus on an iteration goal as opposed to churning point earning stories.  Strong focus on an iteration goal invites creative thinking and awareness of user goals – if we find a new way to achieve an iteration goal with the side effect of throwing away the remaining stories, we should get on with it, and not mourn lost points and old stories.

Once the rough quantity of ‘stuff’ to be delivered is agreed, it is all about execution, as such focus should shift away from what was estimated and towards what should be achieved.  Velocity is like your last order for team pizza, use it to inform quantity, once the mountain of pizza arrives you and your colleagues just have to deal it, constant reference to the original estimate serves little value.

Comparing Velocity
I’ve observed something of an obsession with velocity as a quantity, the cause of much sniggering and derision during Scrum of Scrums meetings.  A team with a velocity of 100 must be better than a team with a velocity of 20, right?  This reminds me of futile attempts to compare different people’s number of steps walked in all but the most sophisticated fitness apps. My Fitbit tells me I average 10,000 steps each day.  I happen to know that some of those ‘steps’ are activities like chopping firewood, lifting heavy coffee cups, and gesturing wildly at whiteboards. I can use this figure to compare how active I’ve been across different days, but to compare my number of ‘steps’ to another person’s would be fruitless. Just like an agile team sizing and executing its own stories, my context is unique.

I also see many attempts to compare, or harmonise different team’s velocities.  We can certainly compare estimating ability with some confidence.  Standard deviation of predicted velocity against actual velocity can be useful for both forecasting, and prompting improvement conversations.  For instance, a team with a standard deviation of +/-5% on estimates is more predictable than one with a deviation of +/- 20%.  This of course says nothing about the team’s actual achievements against their potential, but this predictability is a solid foundation for improvement. 

Often though the conversation is about stack ranking teams, and proving who does most.  A flaw in only considering velocity is that it is solely and indicator of a team’s output of stories, stories are by no means the only kind of value that teams add to organisation.  A ‘slow’ team might be the one which fixes more defects, handles more support calls, is more active in recruitment or assisting other teams.  Velocity gives no guarantee of completeness, and could simply be the rate at which bug riddled code is being unleashed on unsuspecting downstream teams.  This tends to indicate the need for a more varied and balanced set of measures.

…and further more
A big problem here is that first impressions last, velocity it is such a convenient, accessible term that people intuitively grasp it; incorrectly. This first impression may be very difficult to unlearn, corrupted understanding spreads, particularly when carried by people with influence and a penchant for direction. Thanks to these unstructors, before you know it you’re being asked why your velocity isn’t 400 like Team Over-achievers in the corner. Analogous to technical debt, this is a kind of methodology debt, as unproductive habits become calcified and baked into culture, it is just as hard to pay down.

Ultimately I have a simple plea: think before you use velocity, consider the side effects and take time to educate stakeholders. Ideally bring other perspectives on progress and discourage the comfort often brought by over simplification. This is especially true if you publicise raw velocity values outside the team, it is a fragile concept, and if miss-used it could leave the team with compromised estimates and a painful legacy.

 

 

 

The Final Countdown; Adjourning Agile Teams

www.flickr.com/photos/mbiddulph/5310895988/

When introducing agile  I’m sometimes asked to assist with the creation of teams, I’ll be asked questions like:  how many testers do we need?  Is ten people enough?  Who manages performance?  Should they wear shorts?  These are predominantly valuable questions, lucky covered at length elsewhere.  Something that’s very seldom planned is what happens when these teams disband.  And yet, without due care team members may end up demotivated, disappointed or feeling unappreciated.  In the age where everyone is fighting to retain smart people, and to transform their organisations, at the expense of a little forethought, it seems a high risk to take.

New teams are recognised  to move through similar phases, regardless of their domain.  Tuckman’s stages of group development is the ubiquitous cycle, suggesting that teams move through the following stages as they gel and become productive:
Forming – When a team gets together there is a buzz, there is expectation, and caution as they figure out their mission and their colleagues.
Storming – The team realise what they are up against, both from their mission, and each other, there is vying for position, conflict and resistance.
Norming – The team start to behave as more collaboratively, making progress towards goals, and developing team relationships, they are increasingly effective as a group.
Performing – The team is stable and performing its best, there is respect, understanding and a strong sense of shared purpose.

The sequence provides a useful heuristic for what to expect as a new team gets together.  Of course progress isn’t strictly linear, and teams iterate through these stages.  Something as simple a desk swap may prompt a little storming, with care though the overall trend remains towards performing.

Having studied and developed the concept, a decade later Tuckman added a fifth stage.  It is sometimes referred to as ‘mourning’ although I suspect that’s largely due to rhyme compatibility with the other stages, Tuckman named it ‘adjourning’.

Imagine this:  you go on a sunny holiday with a bunch of  new friends, sure the journey was a bit fraught, it took a while to get used to the new place, pace and lifestyle, there were some heated words along the way, in the end everyone is getting along, doing their thing and generally having a good time.  The holiday peaks when everyone gets together, sharing experiences and ideas, the energy is tangible.  Then one day you notice someone has left the group; “needed elsewhere” apparently.  Next day the hotel barman disappears, then you notice no-one is organising activities anymore.  The swimming pool is turning a curious green hue.  Two people wonder off because ‘there’s a more interesting holiday going on over there’.  The kitchen runs out of food – the chef is only there once a week, mumbling about “other priorities”.  The manager who used to stroll over enthusiastically and ask; “how can I help today?” now seems afraid to look you in the eye.  Slowly people drift off until the sense of fun and bonhomie are lost.  Triumphs forgotten, you kill time until the holiday reaches its end date.  A sense of loss and a slow fade to boredom becomes the overriding memory.

The adjourning phase in teams is important because it sets up the attitude, enthusiasm and levels of energy taken to the next team.  What is carried forward is largely based on the emotional response someone has looking back at the project.  The final weeks are particularly important because we are recent creatures, more recent negative experiences will replace older positive feelings.

Sensitivity to the adjourning phase is especially important during an agile transformation, and when introducing change in general.  In those early days a change initiative needs allies and evangelists to support and promote it.  Peer to peer recommendations are particularly respected, and people who have enjoyed a project help form a cohort of change agents.  The opposite is also true, word of a poorly handled team will soon spread, and be seen as part of the ‘new way’.

So you need to make sure that a team has a positive emotional response when they think about the project otherwise, regardless of what ‘facts’ or ‘reasons’ are given, it’s the emotional side that will determine whether a similar initiative is supported, or resisted.  I’d suggest the following:

Mark The Occasion – Lunch, cakes, a flaming aquatic Viking burial for the team board, anything that underscores that the project is done.  No need for speeches, but be sure to say thanks, recognise achievements, and just mark the last time the team exists in this form.  Often it feels more useful to do when the whole team can attend, rather than the calendar close date.

Close it down – Agree what work should be completed, rather than allowing that nagging unfinished, lost opportunity feeling.  Consider the tasks that will make it possible for other teams to work with product when handed over.

Retrospect – There are two motivations for this, firstly to gain learning and insight for future projects, secondly so team members feel like they’ve been heard and that things will improve in future projects.  A good option is to hold a ‘futureospective’ focus the retro on the future, asking each participant to choose a couple of initiatives they’d introduce to their new team.

Communicate – Often team changes are requested by outside influences,  it is good to soften the feeling that team changes are being ‘done to them’, especially in an agile environment which encourages self organisation and team responsibility.  If the team faces a slow wind down with people moving over a few weeks, explain why, ask for input into how the team’s remaining commitments and assets should be managed.  Again you’ll uncover solid ideas and increase engagement.

In the aspect of adjourning agile teams are no different to any other, except perhaps that they are expected to move through the stages of group development more often and more rapidly than their counter parts.  We should be mindful that our search of agility does not lead to disenfranchised groups and teams that never truly form due to prior poor experiences when disbanding.

Compared to the effort we put into forming a team the effort required for a successful adjournment is small, and the rewards are high; raised enthusiasm, engagement and even increased support for transformation.  So let’s not short change our teams, lets facilitate the closing phase of team life with as much thought and attention as the beginning.

INVEST a little more in your stories

https://www.flickr.com/photos/jakerust/16207445813/

I’m a big fan of the INVEST mnemonic, which encourages agile story authors to make their stories Independent, Negotiable, Valuable, Estimable, Small and Testable. There are plenty of great sources for more detail, so I won’t rehash them here. Having introduced a number of teams to INVEST I often find I need to follow up with three more items.  Leading to the notion of INVESTpul, and yes, before you say it, I really should figure out a better backronym:

Provoking – A good story provokes conversation, it is a much an expression of wish as a challenge, how can this goal best be achieved given the constraints the team operates within?  All too often this element is lost, and work becomes about churning stories, regardless of their value, with the dangerous assumption that all the quality thinking has been done already. Opportunities present themselves in different ways throughout an iteration, and we should remain alert and ready to seize them, to pivot based on new information.

Ubiquitous – The language of stories should be readily understood by the team, its stakeholders and drive-by observers. Drive-by observers are often influential, and may even be the ultimate sponsors of the team, packing up late I often see evening board walking by the leadership team. Their support of the team, and agile, may be effected by how well they understand stories, and the impression they get from the board carrying them. Ubiquitous language discourages technical terms, the presence of which often indicates that a solution to the story has already been agreed, diminishing connection to the user and narrowing the team’s potential to find other ways of achieving the same aims.

Legible – It seems incredible that this needs saying, but it does. It should be possible to read stories easily, this doesn’t just apply to handwriting, it means using sensible fonts when printing cards, and avoiding cramming information into small spaces. The ideal story card passes a three foot test – it can be read by anyone participating in a board based discussion, like a stand up, which necessitates some members standing about 3ft away

The Six Foot Test
While we’re on the subject, another distance themed test I like relates to the overall board. The six foot test is simple; from standing that far away what could someone learn about the team’s work?  That distance is deliberately chosen because generally you can’t read individual story cards, putting focus firmly on flow and the system of work. It is particularly useful to forget what you know and don a stakeholder hat for this exercise.  At a minimum I would hope to be able to determine the following:

• The team name
• Their high level goal
• Amount of work in progress
• Phases or stages work passes through.
• Who is doing what
• Which work is blocked, or needs help

From the appointed distance I’d also look around the board, the presence of artifacts like definition of done, column policies, burn down charts and metrics are positive indicators.  Another interesting aspect is the presence, or otherwise, of playful elements, often these are reflections of trust and safety within the team.

So that’s how to write INVESTpul stories, along with some bonus musings on good board practice. The author Antoine de Saint-Exupery once said; “Perfection is Achieved Not When There Is Nothing More to Add, But When There Is Nothing Left to Take Away”.  So tell me, what would you add or remove from INVEST?

Why is a Spike called a Spike?

http://www.flickr.com/photos/tulanesally/5244470514/

Unusually for an agile practice, it appears practitioners largely agree on what the term ‘spike’ or Spike Story describes:  a brief, focused effort to answer a question, explore a concept or investigate an issue.  Fortunately the unsettling feeling of general agreement is quickly displaced when we try to agree the origin of the term.

Kent Beck is generally recognized for introducing ‘spike’ to software development parlance, as part of the XP movement. However, he has not been forth coming on why that particular word was chosen, at one stage even avoiding the term because it was too nuanced:

Because people variously associate “spike” with volleyball, railroads, or dogs, I have begun using “architectural prototype” to describe this implementation.
– Guide to Better Smalltalk:

It is interesting too that Cynefin, much of whom’s focus is on experimentation chooses not to borrow the term spike.  Understanding the source of the term is useful, and potentially interesting, given how often the concept is illustrated by real word example.  Listed below are some of the definitions I frequently hear, do let me let me know if you’ve heard others!

The Mountain Climber – A spike is another term for piton, used to anchor oneself to a rock face.  If the piton in secure it is safe to take that route and hang more gear (and actually yourself) from it, if not it’s time to find another path.

The Railroader – Even within this single domain there are at least two derivations.  A spike is a huge nail used to secure track to sleepers, hence ‘spiking a track’ – in order for a train to move safely over newly laid rail.  Secondly a spike, or similar shaped wedge may be used to hold open a point (switch) and set the direction of subsequent traffic.

The Geologist – A spike is a kind of probe used to assess the layers beneath by inserting a hollow core, which fills with material, withdrawing the core, and inspecting the resulting sample.  From this a geologist may determine if there is firm foundation to build upon, or if there is anything of value to extract.

The Sci-Fi Fan – Named for the Buffy The Vampire Slayer character, a Spike disregards team norms and provides an excuse to go it alone with a focus on what is cool, often leaving behind a trail of destruction for others to deal with.  Actually, I just made this definition up, but all too often I see these behaviors.

The Builder – This definition, from Ron Jeffries, appeared during a debate on Stack Overflow: “Spike” is an Extreme Programming term meaning “experiment”. We use the word because we think of a spike has a quick, almost brute-force experiment aimed at learning just one thing. think of driving a big nail through a board.

The Electrical Engineer – This one I remember from my brief time as an Electrical Engineer, in the days when it was far more necessary to tinker with hardware to support our software habit.  We used to spike relays to hold them on or off, isolating part of the circuit as a temporary measure to assess behaviour and assist problem solving.

The Statistician – I’m sure you’re familiar with this one; when charting a values over time a spike is a sudden sharp increase in those values, often of short duration.  When monitoring production systems this sharp rise frequently correlates with increased shouting and coffee consumption.  In terms of a spike story then, this is a burst of effort towards a goal, a sprint within a sprint.

Of course, there is another option, one that perhaps we’d rather not consider.  Thanks to a cognitive bias we all have (yes, even you) called the Halo Effect, we are prone to liking or believing things from certain sources.  Simply because we believe the term ‘spike’ was coined by a person we admire and respect we believe it must have a respectable and admirable definition.  I’m afraid it is entirely possible that little thought was given to the term, the wrong word was used, it is an in joke, or that the reference is something only the author would understand.  Luckily for agile and XP practioners everywhere, we’ll probably never know.

 

 

There’s no script for agile – what can we learn from Improv theatre?

It’s always good to hear a new perspective on a seemingly familiar subject, it often provokes thought, inspires action, and encourages change.  Paul Goddard has introduced one such perspective on coaching teams in his book ‘Improving Agile Teams’.  There are plenty of books with similar titles, but the clue here is ‘Improv’ – the book shows readers how improv theatre techniques and games are of benefit to agile teams, both aspiring and experienced.  

I’m confident improv needs no introduction, the likes of Paul Merton, Josie Laurence and more recently, The Mighty Boosch have brought the art to the mainstream. Something you quickly realise reading the book is just how spontaneous improvisers are.  There are subtle cues, and openings the players use to both support, promote, and challenge each other.  Take a look at the expressions on the faces of Ryan Stiles and Jeff B Davis in the first minutes of this Improv-a-ganza clip, you’ll see reactions to unexpected events, and rapid adaptation to changing circumstances.

The idea of using improv techniques to improve team collaboration is an appealing one, the principles of agile and improv seem closely aligned, success in entertaining an audience and working effectively on complex, creative projects requires trust, collaboration, respect and a sense of play.

The book draws out five principles;
Safety – concerned with creating an environment in which teams (players in improv speak) can trust and rely on each other, supporting, encouraging, accepting failure and learning together.
Spontaneity – the business of developing an open mind, to increase creativity and receptiveness to others ideas.
Storytelling – as a method of engaging, inspiring and developing empathy, not just applicable to conversation, but also in user stories.
Status – identifying status, and manipulating it in a safe way to identify issues and further creativity and collaboration.
Sensitivity – the ability to sense, or listen and respond appropriately to others, in order to work effectively with them.

These are novel choices for what must, by extension, be agile principles, but the book explains the thinking behind these, and grounds them in current theory. For instance, in safety pointing out that absence of trust is one of the five dysfunctions of a team .  It’s nice to see Mihaly’s flow model get a mention too.

What I really like is that these principles are used to introduce a improv games and tips designed to further the maturity of teams.  These range from simple, everyday exercises that most teams would be comfortable with, to techniques that may challenge both team and coach.  It’s interesting to run through the techniques, and think “Could I run this with my team?”  if the response is ‘no’ then it’s likely attention to one of the five principles is warranted.

There are fifty or so techniques in the book, some quick and simple and great for livening up stale meetings (like conducting a stand up without using the letter ‘S’) while others require more time, and set to explore a particular area with a team.

One technique is ‘The physic Stand Up’, when team members aim to give someone else’s daily report.  I’ve seen this in action, and it’s fascinating, especially when vociferous people give reports on behalf of more timid people, there’s a strange mix of relief (that’s what I’ve been wanting to say) and shock (did they really just say that?).

I found the status techniques are particularly interesting, especially as it’s an area less explored by agile.  The Dinner Party game is aimed at understanding and recognising status, and the responses it elicits.  In brief, players are invited to an imaginary dinner party, split into two groups and asked to show either high or low status behaviors.  The groups then switch for the same amount of time before retrospecting on the experience.  

Sounds simple?  I’d suggest reserving judgement until you’ve tried some of the techniques, either in a safe environment or on the ‘stage’ of the shop floor.  There are great benefits to be found, experimenting with improv is both challenging and thrilling, the learning is definitely in the doing.

P.S.
In a similar vein, I’d highly recommend the book Artful Making, an unlikely collaboration been a Harvard Professor and a director/playwright.  It draws agile and leadership lessons from the world of makers, particularly experiences running a busy theatre and working with groups of actors.

Five Gators Of The Apocalypse?

I generally dislike war and military metaphors for team and making activities.   Admittedly IT has a lot to learn from the military in terms of teams and scale, but in the wrong hands these metaphors seem to encourage unproductive conflict and counter-collaborative behaviours. This strikes me as odd because although prepared for conflict, the military spend much of their time avoiding or minimising it.  However, I do need to call upon a slightly violent metaphor to describe the relationship between constraints encountered when building a continuous delivery capability in an organisation.

The process of change reminds me of the nineties arcade game Whacky Gators, where a number of cheeky gators poke their heads out of caves, and you biff them with a large hammer, hands, or other appendage depending on personal preference. You never know which gator will appear, or when, and more than one might show up at once.

When encouraging continuous delivery (and by extension DevOps) those gators might be named: Culture, Tools, Organisation, Process, and Architecture.

These five are interdependent constraints, each affecting the other.  However, while inside Whacky Gators is a fairly simple randomiser determining which gator will surface, behind the scenes our organisations look more like a hideous HR Giger meets Heath Robinson mash up.  We can’t readily inspect them to determine what to change.

My theory is that when one constraint is eased it will reveal a new constraint in a different area. This is a tenet of most agile and learning methods – surface a significant issue, deal with it, see what surfaces next.  Often a method, and our expertise, focuses on just a couple of areas, we’re well versed at solving problems with technical solutions, or just improving our own team’s capability in isolation.

A good continuous delivery capability involves the whole engineering organisation (a great one involves the entire organisation). This means it is crucial to consider all five constraints, and when there is a problem be ready to shift focus and look for the solution in one of the other areas.  In fact, this simple shifting may lead to the root of a problem.  Do reams of process indicate a risk adverse culture?  The solution may not be more process, but a different attitude.  Are those tools covering up or compensating for some thorny, unspoken issue no one dared to face?  When trying to improve delivery capability there may be a temptation to replace an old tool with an improved version, maybe the need for that tool (and associated overheads) can disappear with an ounce of process and a healthy dollop of collaboration?

Returning to our Whacky Gators metaphor, the big question is how are you playing?  Do you simply wait for that same familiar gator to return?  The one you’re most comfortable dealing with?  Do you hover where you are comfortable while other opportunities pass by or, are you nimble, and brave enough to tackle each constraint as it appears?

Footnote:
While I was looking up Whacky Gators, I couldn’t resist a peak in the machine service manual, there I found this uncanny quote on success, as applicable in the game, as it is in change:
“The player does not score extra points by hitting harder; a light touch will activate the circuits and will lead to higher scores.”

 

Measure for Measure – exploring DevOps adoption metrics.

Confession; I find measuring stuff a fascinating challenge.  Sometimes measuring is straight forward, like the fuel gauge on your car, but often times it’s more complex.  The volt meter in your car quietly drains the battery while measuring its health.  The motivation survey in your inbox will quietly change your motivation.

Its termed the observer effect, where the act of measuring affects the thing you’re trying to measure.  Measuring, or even just assessing, the output of groups is similarly taxing, even the act of posing a question can project your own biases.  Last year I got interested in measuring the progress of steps towards DevOps culture. At Nokia Entertainment’s MixRadio development emporium we’d had good continuous delivery tools in place for months, but weren’t certain our culture continued to improve.  Complacency was a risk, but we couldn’t tell how large.  It seems one of the hardest parts of change is keeping things going in the period between that initial burst of enthusiasm and when practices truly take root as habit. 

So, I shared my thoughts at DevOps Days London, and received some really useful feedback from the crowd there.  I’ll let you into a secret though: I wasn’t happy, it just wasn’t rigorous enough. Paul Swartout and I created metrics focused on adoption, we wanted simple, no cost methods that anyone could use, without needing a big budget or corperate sponsorship. We called them ‘Vital Signs‘  and they comprised; Cycle Time, Shared Purpose, Motivation, Collaboration and Effectiveness.

The main aim was to benchmark, ready to see which way our desired ways of working were trending. However I wanted to capture elusive things: just how ready was the team to ignore organizational setup and work together?  We also didn’t want to bias for DevOps, Scrum, Kanban or any of our other preferred methods,  if someone found a better way, we wanted to learn.

The art was to find metrics in which these desirable behaviours surface, and of them only cycle time was measurable with any consistency.  We learnt an awful lot from the other metrics, particularly free form comments.  The problem was all that prose was impossible to graph, impossible to track.

Frustratingly those things that are hard to measure are amongst the most critical. They often indicate how long you can sustain a pace or practice.  It is very easy to focus exclusively on productivity, but you might be slowly killing your workforce, as Amazon recently discovered.  In general engineering teams aren’t a temporary construct, they need to be looked after for longer than the holiday season.  Engagement and well-being over time are going to drive quality and productivity as much as anything else. (Pseudo science here).

So why raise this now?  Well, I was enjoying a coffee at the DevOps Café , and was interested to hear a side remark about metrics by the ever eloquent Damon Edwards and John Willis.  They described the following as their ideal set of metrics:

• Cycle Time – From customer report to change in production.
• Mean Time To Detect (an issue)
• Mean Time To Repair (or make a change)
• Quality at source – or escape distance, how far do errors get before they are noticed?  Worst case: customer.
• Repetition Rate – Does the same issue keep happening, or are we learning?

Used together, these are just genius, because it’s very hard to achieve good results without a healthy productive relationship between teams.  Furthermore it doesn’t matter how you describe what you’re doing: DevOps, OpsDev, Agile, Scrum or one great big group hug – those metrics don’t test adherence to a methodology.  I guess mavens like these will often drive common practice, and these metrics are very much evident in Puppet Lab’s excellent surveys, (some words on the magic here) (for DevOps archaeologists* early John Allspaw thoughts here).

But there is one metric which went un-mentioned, my old measurement nemesis; engagement.  I suspect that you could be proudly watching all the above metrics trending positive, and be rudely awoken by burn out or a rash of exit interviews.  To avoid surprises shouldn’t the impact of change on key people be monitored too?  Retention is a favourite for this.  A good indicator, but actually people leave for a lot of other reasons.  If someone departs for a role closer to the best trails it should not be seen as the first sign of DevOps culture crumbing into nothingness.

So while it seems DevOps operational metrics are mature, there is more work to be done to understand if we’re getting results and simultaneously creating a healthy, sustainable, culture.  That suggests three dimensions to measure for DevOps, and other flavours of adoption.

• Efficiency – Our key measures like cycle time, and mean time to XYZ, are they improving?
• Effectiveness – Is the right kind of work being done, and steering the team towards success in their organization’s field?
• Engagement – Have we created an environment for people to be at their best?  Are we making the most of our talent?

Of course, these three need to be balanced – focus on one could easily be to the detriment of the others.  Measuring engagement, culture change and people things will always be hard, and methods flawed, but we should avoid measurement inversion, and strive to measure things not just because they are simple, but because they are valuable.

* Note to recruitment agents:  DevOps Archaeologist is not a real role, don’t go there.

.