A Tale of DevOps and Daring Do

http://www.flickr.com/photos/chodhound/7178026433

I recently read Phil Thompson’s great post around the value of story telling ‘Story telling in a business context’.  Reading it called to mind my favorite DevOps leadership story, but in a cautionary note Phil also talks about the dilemma of creating a tale to suit one’s own point.  He recommends a subtle caveat such as “It’s like that story where….”.  As it is so hard to authenticate tales from the past I’ll take that recommendation:  It’s like a story of daring do in the hay day of the industrial revolution…

The Avon Gorge in Bristol is a wide one and half-mile long valley cutting through craggy limestone on the edge of the city, in places it is almost a hundred metres deep. In the 1800’s engineer Isambard Kingdom Brunel was commissioned to build a suspension bridge across it, the chosen location was no less than seventy five metres high. Already the commission doubted Brunel’s calculations, forcing him to build two massive piers to narrow the span to two hundred and fourteen metres. Huge towers where constructed on either side of the river then ropes were strung across the gorge, followed by cables. After building sufficient strength, the plan was simple; a basket could be hung from an iron bar, allowing workers to cross with more cables and equipment. This approach would save long, expensive boat and land trips from side to side.

Everything went to plan until someone was needed to pilot the basket across the windy gorge. This was a time when trades had little protection, there were high injury rates, and workers were almost considered expendable. So when the work force were concerned about the safety of the basket, who took the job? It was Brunel himself. He made the first crossing, showing utmost faith in the design and trades that built it. In so doing Brunel set an example, demonstrated commitment, and earned the trust of the people he worked with.

I like this story, because it demonstrates the bold steps leaders (by which I mean anyone wishing to inspire or encourage, regardless of role or title) may take to gain support for new ideas and ways of working. This is particularly true of DevOps, Agile, Lean and similar learning methods – at first they might look different, maybe even as precarious as Brunel’s basket, but there are significant benefits to be found. Teams united by a purpose, and principles, are likely to discover those benefits, and learn, quicker than teams that are divided or fearful of change. Visible acts of leadership, and the stories that follow them, are as powerful now as they were hundreds of years ago.

This post was first published on the Kainos blog.

Anti Pigeon Holing or Why Leaders Should Consider Capabilities, Not Job Titles

Often during a DevOps or agile transformation, we demonstrate the potential of fresh ways of working with a single, pioneering team. Generally these teams produce solid results and there is a strong desire to scale the approach to more teams. This moment is something of a tipping point for the department, successful scaling leads to successful teams, leading to successful projects. How people are picked for those teams is crucial. A team’s make up is just as important as the practices it adopts; personalities, skills, experience and enthusiasm will all determine the drive, output and diligence of the team.

So how are teams created in your organisation? The easy way is to ‘do what we always do’: gather people finishing a project, or copy the template from the last team created, or maybe teams don’t change at all! Slightly more adventurous is to replicate the list of job titles in the pioneering team. These cookie cutter approaches may be successful, but only if the department is already populated by talented flexible individuals. However a little consideration will often yield better results, not just for the business but for the individuals in the team.

…for the rest of  post see the Skelton Thatcher Blog

The Things I Learnt about DevOps When My Car Was Engulfed by Flames

This is a true story, based on a talk from DevOps Days London 2016:

It was a gorgeous sunny spring day, my family and I were driving through my home town of Bristol, ready for another weekend adventure. We were cruising along when my wife said quietly: “I can smell smoke”. Now, I’m a good mechanic, I used to restore classic cars. The car we were driving was modern and recently serviced. I checked the instruments, everything normal. “It must be outside”, I declared confidently. Two minutes later tentacles of smoke were curling around my ankles and my shins were getting remarkably warm.

We can learn something relevant to DevOps, and many other disciplines, from this experience… read the rest on InfoQ

 

What do points mean?

www.flickr.com/photos/vpickering/14669824798/

I’d like to think that my previous posts have provided an informative and well balanced, ego free commentary on pertinent topics such as learning, change and even improv in agile.  Well, this one is different; it’s good old fashioned, vitriolic rant.  It’s about velocity, or at least what people do to velocity, a well meaning, innocent and largely defenseless concept.  The other thing this post isn’t is a comparison or comment on the value of velocity or estimating in general.  You’ll find plenty of good #NoEstimates conversation elsewhere.

Time and time again I see velocity abuse:
– Equating velocity to volume of output (more points equates to more productivity)
– Using velocity as a target, linked to incentives (The Scrum master shouts: “what do points mean? – prizes” )
– Assuming that velocity is a real number, double the engineers, double the velocity, right?

There are plenty of good velocity explanations around, the way I see it velocity is part of a system for:
a) Improving group estimation ability (mostly by encouraging exploration of work, and an appreciated of other’s roles)
b) Forecasting the rate work will be carried out by the team.

 As such, velocity needs to be honest and without interference, otherwise the outputs of neither estimation or forecasting activity can be trusted.  

Increase your velocity; Go big or go home!
It seems there is an obsession with velocity in volume, if a team knocks over 20 points one iteration, it should better itself in the next, 25 points next time anyone?

Consider however, a situation where a team doubles its velocity compared to the previous two iterations, does that really indicate double productivity?  There are numerous factors which could contribute, perhaps they are not working in a sustainable fashion, or corners are being cut.  Was a large amount of unfinished work rolled over from the previous iteration?  In these circumstances we often have what looks like high volume output, but is it to the detriment of other (typically operational) concerns?  What if a team halved its velocity in the last iteration?  Is the team failing, or is the organization not providing what they need to succeed?  Did they change the value of points to align with demand?  Is the team hitting its predicted point every single sprint? Suspicious to say the least.

The key point is that a team’s velocity fluctuates over time, and there is often considerable variation as a team forms, and starts to understand it’s work, constraints and processes.

Velocity with benefits
To some extent velocity is a target for the team, but one that is most effective when set and owned by the group, rather than linked to incentives.  Pushing velocity onto a team, demanding or targeting an increase is a dangerous, counterproductive practice. Strongly linking velocity to incentives compromises its primary value as an estimation tool. Organisations generally hire the smartest people available, do they somehow think those people won’t be smart enough to game velocity given even the smallest incentive?

“The moment a measure becomes a target, it ceases to be a measure” – Goodheart’s Law

A further reason not to link velocity to incentives is the frame of mind it encourages during an iteration. It seems preferable to focus on an iteration goal as opposed to churning point earning stories.  Strong focus on an iteration goal invites creative thinking and awareness of user goals – if we find a new way to achieve an iteration goal with the side effect of throwing away the remaining stories, we should get on with it, and not mourn lost points and old stories.

Once the rough quantity of ‘stuff’ to be delivered is agreed, it is all about execution, as such focus should shift away from what was estimated and towards what should be achieved.  Velocity is like your last order for team pizza, use it to inform quantity, once the mountain of pizza arrives you and your colleagues just have to deal it, constant reference to the original estimate serves little value.

Comparing Velocity
I’ve observed something of an obsession with velocity as a quantity, the cause of much sniggering and derision during Scrum of Scrums meetings.  A team with a velocity of 100 must be better than a team with a velocity of 20, right?  This reminds me of futile attempts to compare different people’s number of steps walked in all but the most sophisticated fitness apps. My Fitbit tells me I average 10,000 steps each day.  I happen to know that some of those ‘steps’ are activities like chopping firewood, lifting heavy coffee cups, and gesturing wildly at whiteboards. I can use this figure to compare how active I’ve been across different days, but to compare my number of ‘steps’ to another person’s would be fruitless. Just like an agile team sizing and executing its own stories, my context is unique.

I also see many attempts to compare, or harmonise different team’s velocities.  We can certainly compare estimating ability with some confidence.  Standard deviation of predicted velocity against actual velocity can be useful for both forecasting, and prompting improvement conversations.  For instance, a team with a standard deviation of +/-5% on estimates is more predictable than one with a deviation of +/- 20%.  This of course says nothing about the team’s actual achievements against their potential, but this predictability is a solid foundation for improvement. 

Often though the conversation is about stack ranking teams, and proving who does most.  A flaw in only considering velocity is that it is solely and indicator of a team’s output of stories, stories are by no means the only kind of value that teams add to organisation.  A ‘slow’ team might be the one which fixes more defects, handles more support calls, is more active in recruitment or assisting other teams.  Velocity gives no guarantee of completeness, and could simply be the rate at which bug riddled code is being unleashed on unsuspecting downstream teams.  This tends to indicate the need for a more varied and balanced set of measures.

…and further more
A big problem here is that first impressions last, velocity it is such a convenient, accessible term that people intuitively grasp it; incorrectly. This first impression may be very difficult to unlearn, corrupted understanding spreads, particularly when carried by people with influence and a penchant for direction. Thanks to these unstructors, before you know it you’re being asked why your velocity isn’t 400 like Team Over-achievers in the corner. Analogous to technical debt, this is a kind of methodology debt, as unproductive habits become calcified and baked into culture, it is just as hard to pay down.

Ultimately I have a simple plea: think before you use velocity, consider the side effects and take time to educate stakeholders. Ideally bring other perspectives on progress and discourage the comfort often brought by over simplification. This is especially true if you publicise raw velocity values outside the team, it is a fragile concept, and if miss-used it could leave the team with compromised estimates and a painful legacy.

 

 

 

The Entirely Random DevOps Days London Write Up

There are some great posts out there on the superlative DevOpsDays London 2016 Conference. Of particular note are Manual Pais, DevOpsGuys  and Helen Beal’s DevOpsDays London: Making Happy. These are well structured, balanced pieces which neatly represent each talk along with insight from experienced authors.  I hate to disappoint, but this post is not one of those. It’s a random collection of the things that peaked my interest at DevOps Days London.

What is Legacy?
There was lots of talk of legacy, the BiModal debate rolled on. Much like the sporadic agile wars many of the detractors use out dated definitions as convenient ammunition, Gartner’s understanding of the challenges seems to have evolved away from its original strategy towards an exploit and explore approach for recent and legacy systems respectively.  Defining legacy is challenging, with considerations beyond just the age of the software or system. There were a few definitions I really liked:
“Legacy is code you can’t iterate on as quickly as you need to” – Casey West
“Legacy is code you don’t have automated tests for. – Micheal Feathers
“Legacy is where your customer money lives” –  Bridget Kromhout

The merits of really reading
I keep having the same conversation, like being in a endless loop, it goes like this:

Person: “Yeah, I know all about Conway’s law, it is super insightful”
Me: “Exactly, that example about teams building compilers, that’s a light bulb moment”
Person: “….the compilers?”
Me: (Thinks) “Have you really taken time to understand what you’re advising people to do, or are you just reciting tweets?”
Me: (Says) “ Check out the article, there’s lots more in it”

It is like this with so many topics, REST, OODA, Learning. That’s why I was so pleased to hear the ever-eloquent Gareth Rushgrove call out the value of reading academic papers in his talk.  These days we are so prone to snacking on sound bites we seldom get the satisfaction of a full reading meal, yet our brains cry out for this kind of nourishment. I believe papers, and source material in general, are the best way to gain a firm understanding of a topic, particularly because they build a picture of what motivated the author, not just what they did.  Much like software patterns, understanding the intent and motivation is key to successful application.

Burnout
A couple of talks touched on burnout, as did a well attended and lively open space. What surprised me was how many people had direct experience of it, it remains an issue in the industry despite raised awareness and talk of sustainable, humane ways of working. Oliver Wood talked about his experiences working so many hours that he slipped a disk. Keen that others may avoid the same fate, he created GoodCoderBadPosture.  During his ignite he reminded us “you are ephemeral , you are not highly available“.  My talk (Things I learnt About DevOps When My Car Caught Fire) used the analogy of looking through the windscreen of my burnt out car, all the instruments you normally rely upon to sense the world are warped and confused, your view is fogged and distorted. If only we could see metal strain as readily as we can bad posture.

I noticed much of the burnout open space was concerned with what management and organisations can do prevent burnout, and recognise it’s occurrence. This is a reasonable standpoint given that our behaviours are generally shaped by the systems we work within. However In the spirit of DevOps we should also note that it isn’t a problem for effected individuals and their mangers to tackle alone.  While organisations take action, we might also ask ourselves:

1. “How would I know if one of my colleagues was suffering from burnout?”
2. “How would I help someone I thought on the verge of burnout?”
3. “How do my own behaviours effect the likelihood of burnout in my colleagues?”

There was a nice note in Jeromy Carriere’s talk, and a potential answer to question 3: “Work hard to make every alert exciting” this has implications for burnout, exciting alerts implies only being disturbed for hard problems, not simple switch flicking exercises.

Change
There was plenty of talk of change, particularly the danger of not evolving and experimenting.  Change strategies were discussed, including the value of heading into conversations well armed with data.  It was clear from their talk that Microsoft are changing in places, for instance setting up open team rooms or neighborhoods.  I rate this approach, it appears to balance team privacy, open communications and the distractions of full open plan.

“It is not necessary to change, survival is not mandatory” – Deming (Who wasn’t present!)
“The riskiest thing is not to change” – Joanne Molesky

The change theme included the importance of investing time in the most valuable activities, and how to discover them. It highlighted that many of those valuable activities are operational features, not just shiny new toys to please users. If you’re in the mood for self reflection you might give some thought to this quote from Bridget Kromhout:
“When evaluating yourself don’t forget to look at the value you are adding”

A conference, with a culture
The thing I love about DevOpsDays, and the way it’s organised, is that it still feels like a community event, sure it’s scaled, but the level of friendliness, inclusion and support are almost as the first time I spoke in Goteborg 2011. The story of this scaling and principles behind it were told by Kris Buytaert, it’s surprising how many of the early adopters are still active. The conference manages to short circuit a lot of anchoring and group think by giving almost 50% of the time to open spaces. Taking responsibility for the schedule out of the committee’s hands into the delegate’s ensures that topics are relevant to attendees, right then and there. The willingness of speakers to stay and participate in these sessions is key to their success and makes for some great learning.  Not bad for a movement that still can’t agree what its about.

Inflicting Trust

I often introduce teams to the notion of ‘inflicting help’, those well-meaning activities which deprive the recipient of an opportunity to learn, or practice.  For example, if I always helpfully facilitate retrospectives for a team, they will miss the experience of running their own. If someone helpfully handles all the build work for the team, ‘because they are best at it’ the team will not learn how to deploy for themselves.  This notion is of particular importance for agile and DevOps teams, as they cross skill, and share responsibility.

I also believe it is possible to ‘inflict trust’, that is give so much trust that it is detrimental to the recipient.  This idea may not be popular, how can it be plausible when the agile and DevOps communities talk so much about building trust?

Consider the following example.  It is day one for a new employee in the fictional organisation Great Western Widgets. At Great Western Widgets we deploy a few times each day, and we chase the Continuous Delivery Nirvana of making deployments boring. The new employee makes some code changes and is then is asked to deploy to production, following steps in the build book. “You’ll be fine, I trust you” says the senior engineer. Except things don’t go fine; the load balancer doesn’t route traffic away from the node that’s being deployed to, alerts don’t fire and eventually end users report multiple errors.
The new start suffers a massive dent in their confidence.  They are now far less trusting of their mentors, and apprehensive about battling a reputation for being ‘the one who broke production on their first day’.

In this instance the senior engineer was too trusting of the new start, they inflicted trust. In other, more severe situations, you may recognise this lack of support as a form of negligence. As leaders and mentors that’s something we should be wary of.  In addition to good intentions, it is often convenient (time wise or politically) to use the mantra of ‘high trust’ to expect others to do things, perhaps even things we wouldn’t risk doing ourselves.  Being ready to support, and if necessary rescue, those we are placing trust in is critical to creating an environment of safety, in which people are willing to challenge themselves. It is this feeling of safety that makes teams comfortable taking calculated risks, going fast and innovating. These traits are often seen to lead to high individual and team performance, not to mention a more pleasant work environment.

When encouraging learning and granting more trust, it’s often useful to consider various likely outcomes, if, when and how to step in. Doing the thing for the person is not an option, being ready, and available, to avert to disaster is mandatory.

One example is helping a child learn to carry a tray of drinks to the table, at some point they are going to just have to get on and try it, unless their parents want to follow them to every dining hall, café and pub they visit in their lifetime. So in that moment some steps are taken, almost without thinking; don’t use the best China, don’t over load the tray (you’ve just reduced the consequences of failure) and be ready with a tissue and some wise words if anything does spill (to encourage reflection and restore confidence). The converse, ‘inflicting trust’ would be to load up the tray with boiling hot drinks, fragile, expensive crockery put them into the hands of the wide eyed child and say “Off you go, I trust you”.

The key question to ask yourself is:  When you suggest someone tries something for the first time, which style are you using, supporting or inflicting trust?

The Final Countdown; Adjourning Agile Teams

www.flickr.com/photos/mbiddulph/5310895988/

When introducing agile  I’m sometimes asked to assist with the creation of teams, I’ll be asked questions like:  how many testers do we need?  Is ten people enough?  Who manages performance?  Should they wear shorts?  These are predominantly valuable questions, lucky covered at length elsewhere.  Something that’s very seldom planned is what happens when these teams disband.  And yet, without due care team members may end up demotivated, disappointed or feeling unappreciated.  In the age where everyone is fighting to retain smart people, and to transform their organisations, at the expense of a little forethought, it seems a high risk to take.

New teams are recognised  to move through similar phases, regardless of their domain.  Tuckman’s stages of group development is the ubiquitous cycle, suggesting that teams move through the following stages as they gel and become productive:
Forming – When a team gets together there is a buzz, there is expectation, and caution as they figure out their mission and their colleagues.
Storming – The team realise what they are up against, both from their mission, and each other, there is vying for position, conflict and resistance.
Norming – The team start to behave as more collaboratively, making progress towards goals, and developing team relationships, they are increasingly effective as a group.
Performing – The team is stable and performing its best, there is respect, understanding and a strong sense of shared purpose.

The sequence provides a useful heuristic for what to expect as a new team gets together.  Of course progress isn’t strictly linear, and teams iterate through these stages.  Something as simple a desk swap may prompt a little storming, with care though the overall trend remains towards performing.

Having studied and developed the concept, a decade later Tuckman added a fifth stage.  It is sometimes referred to as ‘mourning’ although I suspect that’s largely due to rhyme compatibility with the other stages, Tuckman named it ‘adjourning’.

Imagine this:  you go on a sunny holiday with a bunch of  new friends, sure the journey was a bit fraught, it took a while to get used to the new place, pace and lifestyle, there were some heated words along the way, in the end everyone is getting along, doing their thing and generally having a good time.  The holiday peaks when everyone gets together, sharing experiences and ideas, the energy is tangible.  Then one day you notice someone has left the group; “needed elsewhere” apparently.  Next day the hotel barman disappears, then you notice no-one is organising activities anymore.  The swimming pool is turning a curious green hue.  Two people wonder off because ‘there’s a more interesting holiday going on over there’.  The kitchen runs out of food – the chef is only there once a week, mumbling about “other priorities”.  The manager who used to stroll over enthusiastically and ask; “how can I help today?” now seems afraid to look you in the eye.  Slowly people drift off until the sense of fun and bonhomie are lost.  Triumphs forgotten, you kill time until the holiday reaches its end date.  A sense of loss and a slow fade to boredom becomes the overriding memory.

The adjourning phase in teams is important because it sets up the attitude, enthusiasm and levels of energy taken to the next team.  What is carried forward is largely based on the emotional response someone has looking back at the project.  The final weeks are particularly important because we are recent creatures, more recent negative experiences will replace older positive feelings.

Sensitivity to the adjourning phase is especially important during an agile transformation, and when introducing change in general.  In those early days a change initiative needs allies and evangelists to support and promote it.  Peer to peer recommendations are particularly respected, and people who have enjoyed a project help form a cohort of change agents.  The opposite is also true, word of a poorly handled team will soon spread, and be seen as part of the ‘new way’.

So you need to make sure that a team has a positive emotional response when they think about the project otherwise, regardless of what ‘facts’ or ‘reasons’ are given, it’s the emotional side that will determine whether a similar initiative is supported, or resisted.  I’d suggest the following:

Mark The Occasion – Lunch, cakes, a flaming aquatic Viking burial for the team board, anything that underscores that the project is done.  No need for speeches, but be sure to say thanks, recognise achievements, and just mark the last time the team exists in this form.  Often it feels more useful to do when the whole team can attend, rather than the calendar close date.

Close it down – Agree what work should be completed, rather than allowing that nagging unfinished, lost opportunity feeling.  Consider the tasks that will make it possible for other teams to work with product when handed over.

Retrospect – There are two motivations for this, firstly to gain learning and insight for future projects, secondly so team members feel like they’ve been heard and that things will improve in future projects.  A good option is to hold a ‘futureospective’ focus the retro on the future, asking each participant to choose a couple of initiatives they’d introduce to their new team.

Communicate – Often team changes are requested by outside influences,  it is good to soften the feeling that team changes are being ‘done to them’, especially in an agile environment which encourages self organisation and team responsibility.  If the team faces a slow wind down with people moving over a few weeks, explain why, ask for input into how the team’s remaining commitments and assets should be managed.  Again you’ll uncover solid ideas and increase engagement.

In the aspect of adjourning agile teams are no different to any other, except perhaps that they are expected to move through the stages of group development more often and more rapidly than their counter parts.  We should be mindful that our search of agility does not lead to disenfranchised groups and teams that never truly form due to prior poor experiences when disbanding.

Compared to the effort we put into forming a team the effort required for a successful adjournment is small, and the rewards are high; raised enthusiasm, engagement and even increased support for transformation.  So let’s not short change our teams, lets facilitate the closing phase of team life with as much thought and attention as the beginning.

INVEST a little more in your stories

https://www.flickr.com/photos/jakerust/16207445813/

I’m a big fan of the INVEST mnemonic, which encourages agile story authors to make their stories Independent, Negotiable, Valuable, Estimable, Small and Testable. There are plenty of great sources for more detail, so I won’t rehash them here. Having introduced a number of teams to INVEST I often find I need to follow up with three more items.  Leading to the notion of INVESTpul, and yes, before you say it, I really should figure out a better backronym:

Provoking – A good story provokes conversation, it is a much an expression of wish as a challenge, how can this goal best be achieved given the constraints the team operates within?  All too often this element is lost, and work becomes about churning stories, regardless of their value, with the dangerous assumption that all the quality thinking has been done already. Opportunities present themselves in different ways throughout an iteration, and we should remain alert and ready to seize them, to pivot based on new information.

Ubiquitous – The language of stories should be readily understood by the team, its stakeholders and drive-by observers. Drive-by observers are often influential, and may even be the ultimate sponsors of the team, packing up late I often see evening board walking by the leadership team. Their support of the team, and agile, may be effected by how well they understand stories, and the impression they get from the board carrying them. Ubiquitous language discourages technical terms, the presence of which often indicates that a solution to the story has already been agreed, diminishing connection to the user and narrowing the team’s potential to find other ways of achieving the same aims.

Legible – It seems incredible that this needs saying, but it does. It should be possible to read stories easily, this doesn’t just apply to handwriting, it means using sensible fonts when printing cards, and avoiding cramming information into small spaces. The ideal story card passes a three foot test – it can be read by anyone participating in a board based discussion, like a stand up, which necessitates some members standing about 3ft away

The Six Foot Test
While we’re on the subject, another distance themed test I like relates to the overall board. The six foot test is simple; from standing that far away what could someone learn about the team’s work?  That distance is deliberately chosen because generally you can’t read individual story cards, putting focus firmly on flow and the system of work. It is particularly useful to forget what you know and don a stakeholder hat for this exercise.  At a minimum I would hope to be able to determine the following:

• The team name
• Their high level goal
• Amount of work in progress
• Phases or stages work passes through.
• Who is doing what
• Which work is blocked, or needs help

From the appointed distance I’d also look around the board, the presence of artifacts like definition of done, column policies, burn down charts and metrics are positive indicators.  Another interesting aspect is the presence, or otherwise, of playful elements, often these are reflections of trust and safety within the team.

So that’s how to write INVESTpul stories, along with some bonus musings on good board practice. The author Antoine de Saint-Exupery once said; “Perfection is Achieved Not When There Is Nothing More to Add, But When There Is Nothing Left to Take Away”.  So tell me, what would you add or remove from INVEST?

Why is a Spike called a Spike?

http://www.flickr.com/photos/tulanesally/5244470514/

Unusually for an agile practice, it appears practitioners largely agree on what the term ‘spike’ or Spike Story describes:  a brief, focused effort to answer a question, explore a concept or investigate an issue.  Fortunately the unsettling feeling of general agreement is quickly displaced when we try to agree the origin of the term.

Kent Beck is generally recognized for introducing ‘spike’ to software development parlance, as part of the XP movement. However, he has not been forth coming on why that particular word was chosen, at one stage even avoiding the term because it was too nuanced:

Because people variously associate “spike” with volleyball, railroads, or dogs, I have begun using “architectural prototype” to describe this implementation.
– Guide to Better Smalltalk:

It is interesting too that Cynefin, much of whom’s focus is on experimentation chooses not to borrow the term spike.  Understanding the source of the term is useful, and potentially interesting, given how often the concept is illustrated by real word example.  Listed below are some of the definitions I frequently hear, do let me let me know if you’ve heard others!

The Mountain Climber – A spike is another term for piton, used to anchor oneself to a rock face.  If the piton in secure it is safe to take that route and hang more gear (and actually yourself) from it, if not it’s time to find another path.

The Railroader – Even within this single domain there are at least two derivations.  A spike is a huge nail used to secure track to sleepers, hence ‘spiking a track’ – in order for a train to move safely over newly laid rail.  Secondly a spike, or similar shaped wedge may be used to hold open a point (switch) and set the direction of subsequent traffic.

The Geologist – A spike is a kind of probe used to assess the layers beneath by inserting a hollow core, which fills with material, withdrawing the core, and inspecting the resulting sample.  From this a geologist may determine if there is firm foundation to build upon, or if there is anything of value to extract.

The Sci-Fi Fan – Named for the Buffy The Vampire Slayer character, a Spike disregards team norms and provides an excuse to go it alone with a focus on what is cool, often leaving behind a trail of destruction for others to deal with.  Actually, I just made this definition up, but all too often I see these behaviors.

The Builder – This definition, from Ron Jeffries, appeared during a debate on Stack Overflow: “Spike” is an Extreme Programming term meaning “experiment”. We use the word because we think of a spike has a quick, almost brute-force experiment aimed at learning just one thing. think of driving a big nail through a board.

The Electrical Engineer – This one I remember from my brief time as an Electrical Engineer, in the days when it was far more necessary to tinker with hardware to support our software habit.  We used to spike relays to hold them on or off, isolating part of the circuit as a temporary measure to assess behaviour and assist problem solving.

The Statistician – I’m sure you’re familiar with this one; when charting a values over time a spike is a sudden sharp increase in those values, often of short duration.  When monitoring production systems this sharp rise frequently correlates with increased shouting and coffee consumption.  In terms of a spike story then, this is a burst of effort towards a goal, a sprint within a sprint.

Of course, there is another option, one that perhaps we’d rather not consider.  Thanks to a cognitive bias we all have (yes, even you) called the Halo Effect, we are prone to liking or believing things from certain sources.  Simply because we believe the term ‘spike’ was coined by a person we admire and respect we believe it must have a respectable and admirable definition.  I’m afraid it is entirely possible that little thought was given to the term, the wrong word was used, it is an in joke, or that the reference is something only the author would understand.  Luckily for agile and XP practioners everywhere, we’ll probably never know.

 

 

An introduction to ChatOps

I first heard of something resembling ChatOps about five years ago when I had the good fortune to share a beer with Scott Chacon, one of GitHub’s co-founders, while I ranted about Deming, he talked enthusiastically about their fledgling organization.  Surprisingly, one of the things he talked about with most passion was Hubot a sort of robot butler who hung around in Github chat rooms serving useful data and whimsical content with equal aplomb.  It seemed a great concept, simple and powerful, it improved operability whist increasing knowledge sharing and encouraging collaboration.

I often wonder why chatOps doesn’t garner much attention, especially as it appears to have played an important part in GitHub’s success.  Perhaps that’s because everyone is gazing adoringly at Docker, or perhaps because ChatOps sits discretely and indistinctly on the boundary between Culture and Tools.

By way of introduction ChatOps combines three key technologies: Asynchronous Chat, Robot Assistants and Automation; let’s spend a moment looking into each.  (The pictorially minded may prefer to spin through my spring DevOps Summit talk where ChatOps was one of my ‘Collaboration Catalysts’.

Asynchronous Chat
Asynchronous Chat needs no introduction, it allows people to congregate in a virtual space to view and post messages and media.  These apps are a good way for a distributed team to collaborate, but there are more subtle advantages – chats can be saved enabling a searching and reference.  Chats allow broadcast, without the publisher having to manage their audience.  You’ll understand the value of this if you have ever been trying to chase down a gnarly production issue with your manager over one shoulder and Project Manager on the other asking for updates.  Oddly, the speed of work does not increase with the frequency of update requests, quite the opposite in fact.
In this situation chat could be used to broadcast progress, without having to manage a distribution list, when people monitor chat, the originator doesn’t get distracted, and may even get proactive support.

In the context of ChatOps it is chat apps which can be readily extended that really matter.  That’s because many of the operations performed will be specific to an organisations and it’s systems, processes and integration requirements.  HipChat , Flowdock, Slack and Campfire are popular options, and choices are often driven by the lingua franca of the development team.

Robot Assistants
Robot assistants lurk in chat rooms waiting to do the user’s bidding.  They may wait to be summoned by a specific command, or step in when they think it’s needed.  Assistants may grab things, like logs from production, or find out who is on call.  This reduces the interruption cost for a user, who is already thinking and collaborating in chat.

A good bot also recognizes the value of play, amusing features are almost mandatory, from adding a mustache to a photo, meme generation to playing tunes.  A useful side effect of this is it encourages folks to hangout in chat rooms, humor keeps people engaged, and generally engaged people are more productive and ready to innovative.  Notable bots include Lita, Hubot,  Err and Stack Storm.  Iron Man’s J.A.R.V.I.S is similar in concept, but somewhat less likely to inundate you with pictures of small miserable faced dogs.

By way of an aside, Terri Winnograd, who later went on to mentor one Larry Page, pondered the utility of robot assistants as early as 1970.  Perhaps he had a premonition of clippy when he wrote:

“I should reiterate that good programming systems do not demand a super-intelligent program. We can get by with a moderately stupid assistant as long as he doesn’t make mistakes. The degree of Al needed is much less than that needed for a full-fledged natural language or vision system”

Automate, mate.
The third component is Automation.  Hooking the bot up to automation, and other deployment and operations tools, is where things get really interesting.  If a bot can integrate with search engines and meme generators, why not link it to development environments, perhaps even production?  Then, if people are discussing a thorny deployment problem they can call in logs, graphs and pertinent data.  The chat room, becomes the war-room; distributed, observable and documented for later learning.

Perhaps the pinnacle of ChatOps is allowing deployment orchestration through chat.   As Jessie Newland describes it succinctly in his highly recommended ChatOps at GitHub talk “Chat becomes the primary control surface for ops”  not only is it is convenient, but a chat client is more portable.  Chat can also serve as a layer of abstraction over the under laying tech, enabling it to change and evolve independently of the commands driving it.   This abstraction opens an opportunity for training, enabling production commands to be executed against a sandbox.  Of course, there is some risk to be considered, and it is possible to restrict commands to people or rooms.

Still not impressed?  In the same talk Jessie outlines a scenario where he makes a deployment, observes a problem, orchestrates load balancers, fixes and redeploys.  Impressively, it all happens in chat, all while keeping his team updated, and leaving a record, with minimum extra effort.

More than just tools?
Looking beyond tools, ChatOps brings more to teams than mere efficiencies. ChatOps liberates Institutionalized knowledge once locked in the heads of key, time challenged, individuals.  Once in the open, ways of doing things can be inspected and built upon.  This isn’t necessarily a threat to those people; often freeing them up to tackle more challenging problems.

ChatOps can be an excellent training tool.  Like the gallery trainee doctors use to observe a surgeon at work, chats can be reviewed and replayed for education.  Need to know how something is done?  Check the archives, and look at the commands used last time, or ask in Chat, someone can demonstrate directly, and show everyone else at the same time.

Reflections
Having written this, I realize I have to some extent answered my own question: Why don’t we hear more about ChatOps?

Effective ChatOps requires maturity of culture and tools.  Even small things, like knowing more senior or experienced people are able to see, and potentially respond to, every comment, takes some getting used to for both parties.  The organisation’s culture must encourage the openness which allows productivity to thrive in the chat.  As such, striving towards ChatOps may provide a useful mechanism to highlight organisational and cultural impediments.  To make operational features available in chat requires not just trust, but investment in tech, safely connecting all those moving parts is not trivial.  To the many organizations who struggle to deploy once a month, ChatOps must seem like a distant Nirvana.

Despite the necessary investment ChatOps can bring many benefits, and can do it unobtrusively, at a pace of change that suits the community.  Using Chat as gateway to operations, adding capabilities when it is considered safe to do so, is an excellent way to introduce and observe new ideas.  ChatOps invites collaboration, and not just because it’s novel.  If all the engineers, regardless of title, hang out and work in the same space it helps build an appreciation of other’s challenges and responsibilities, not to mention attitude and sense of humour.