Mike MacDonagh's Blog

Somewhere in the overlap between software development, process improvement and psychology

Tag Archives: kanban

How to make real progress in a software project

I’ve not been blogging much this year because I’ve been crazy busy writing software (more on that soon I hope). As you might expect I’ve changed my ways of working more often than process improvement consultants change the name of the process they’re selling but one key thing always seems to drop out whether I’m working on my own, with a team, a team of teams or a globally distributed team.

Do the top thing on the list, whatever it is.

Don’t do anything else until it’s done.

Or in process speak: “Reduce the WIP and clarify the Definition of Done“.

Too often I fall into the trap of doing the most interesting thing from the list, which normally means the most complex. And far too often I get carried away with the next interesting thing before finishing something. Both of these lead me to have far too many things in various states of progress all interfering with each other and me with no idea of how well I’m doing. And that’s assuming I actually know what “finished” means.

Software is complex enough without us making it harder with processes, remember the KISS principle?

Advertisements

Reduce waste: Visualise the value stream

A big thing in software process improvement (urgh) is reducing waste.  A great idea, but how do you identify waste?

One powerful method that I use is value stream visualisation. There’s a number of ways of getting information about the value stream ranging from physically walking the chain to simply asking people.

Sounds simple doesn’t it? But one of the best sources of information about wasteful processes and ways of working in a team is simply asking the team what they feel they shouldn’t be spending time on. By freeing up people to actually do their job they can reduce waste and improve productivity. One method I use is to ask each team member to identify a list of n items that they have to spend time on that they think are either entirely or partially wasteful. Basically a list of their impediments. I then anonymize the list look for commonality and then the top few are a reasonable target for improvement.

Alternatively actually walking the path of a piece of work through an organisation can really open the eyes to the sheer number of people and unnecessary waiting involved in some wasteful processes…

Anyway, having got some data I find it useful to visualise it in terms of waiting time and doing time. In Lean-speak cycle time is the time it takes to actually do something, lead time is the time it takes from the request being made to it finally being delivered. In many tasks the lead time far outstrips the cycle time, visualising it can make that very clear.

Low Lead time For example, a task with has a lead time of 5 days might actually be made up of 2 days waiting time before it actually gets picked up and the cycle time of 3 days starts.

Large lead timeIf you look at all of the tasks in your value chain you might find that many of them look more like this, with very large wasteful waiting times. There are many reasons why there might be a large waiting time, typically the team doing the tasks is over-subscribed, or a team might be protecting it’s SLAs, KPIs or other TLAs. These problems can have significant effects when looking at the system as a whole.

Even more interesting is looking at how these tasks might fit together in a value stream. If we imagine a business function that involves a request moving through a bunch of tasks across a number of teams that have different waiting and doing times a simple sequence of tasks, with a little buffering between them from the Project Manager to allow for schedule variance naturally, might end up looking like this:

 sequential_value_chain_arrows

Here we have 26 days of actual work spread over almost 60 days (12 weeks) of elapsed time = 40% waste. Even if the person planning the work is willing to accept some risk and try to optimise their workflow a bit without improving the waste inherent in the tasks involved there’s still a lot of waste in the system.

parallel_value_chain_arrows

By visualising the value stream in this way we can see straight away (apologies to red/green colour blind folks) that there’s a lot of red, a lot of waste. In many cases planners aren’t willing to accept the risks inherent in overlapping activities as shown here, or aren’t even aware that they can leading to the more sequential path shown above. The result is a minimum time that it takes a request to get done, based on the impedence of the current value chain of, in this case, 38 days before we even start thinking about actually doing any work.

Surely that’s not right.

Understand the software value chain by walking it

There’s no better way to understand something than by actually giving it a go. If you want to know what writing a particular programming language is like you need to try it. If you want to know what kung fu is like watching a video won’t help, you need to try it.

If you want to really understand the software value chain in your organisation drawing diagrams in an ivory tower isn’t going to help, you need to get out there and “Go and see”.

In Steve Handy’s blog on Scaling Software Agility he talks about applying some of the thinking from the Lean school of thought to “go and see”, observing the value chain by actually physically following it.

  • Identify a new piece of work and follow it through the delivery chain
  • Attend every meeting, track the activities and note how long it’s static.
  • Don’t attempt to fix it during this period of observation, don’t criticise, just watch and learn.

I think there’s some real value here, there’s something quite simple and powerful about physically walking the walk of a piece of work through the software value chain. Doing so will make some problems, blockages and issues blindingly obvious – processes and organisational structures that seemed to make sense on someone’s bit of paper will just not make sense.

Experiencing the value chain in a very physical way promotes professional mindfulness and will clearly identify waste. I think this is a great way of putting common sense back in to software development.

I call this Go See, and it’s the most effective form of measurement for software development.

 

I’m coming out as not-Agile and not post-Agile

Big A vs. Little a

Huh? What? I’ve written a fair bit on this blog about agile topics, but I always try to write about agility with a small “a”. I’m not really into Agile with a big “A” though – I’m not into doing things according to a set of rules and having arguments about whether I’m doing it right or not. I’m not anti-agile, but I’m increasingly anti-Agile.

To me, the ideological arguments, definitions of everything, frameworks, money-spinning certifications and money-spinning tooling are what’s wrong with doing “Agile”. Being empirical, reflecting and adapting, honestly communicating and putting people first as an approach is being “agile”.

I don’t really like the term “post-Agile” either though as it comes with a bunch of baggage and is easily misinterpreted – and I still see benefits in adopting agile practices. I don’t want to see another specific set of rules or a statement or beliefs with elitist signatures. For me what’s next after agile is about dropping the ideology in software process, destroying the ivory tower of trademarks, formal definitions, money spinning tools and money-spinning certification programmes.

We need to get rid of the League of Agile Methodology Experts and if anyone says “Ah, but this is different” when showing a website of process content then you have my permission to hit them with a stick.

So what does the future look like?

Software development is a complex social activity involving teams of people forming and self-organising to work together, sometimes in teams of teams which is even harder. As technology is increasingly abstracting up and raising in maturity so is the way that developers, managers and organisations think about software and doing software. I think the problem is getting increasingly social, and the solutions will start looking increasingly social using more “soft practices”.

Software process improvement agents/consultants/coaches/mentors (including myself) need to take a long hard look at themselves. Are they telling people how to do it right when they can’t even write HelloWorld in a modern language? I’ve said that to some acquaintances in the industry, generously qualifying “modern language” as something significantly commercially used in the last 10 years and seen them look quite offended at my insulting affront to their professional integrity. I’ll go out on a limb and say you can’t coach a software development team if you don’t know how to write software.

So… software process improvement?

TOverlapping concerns in process improvementhe world runs on software, it’s everywhere and it’s critical. Getting better at doing software, improving the software value chain, is a noble aim and will continue to be as it means getting better at doing our businesses and our lives.

For me process improvement (dislike that phrase as well) is going to be more about combining psychology based business change practices with the best bits of a wide variety of ways of working (agile, lean, Scrum, what you currently do, RUP, Kanban, various workflow patters etc.) with technical software development practices like continuous integration, continuous delivery.

We need to work together, not as “leaders and followers” or “consultants and clients” but as collaborative peers working together to apply specialist skills and experience so that we can all improve. Smart people should be treated as smart people, we have much to learn from them and should be thinking in terms of fellowship rather than leadership.

I’m calling this overlap “soft practices”* because the term is evocative of:

  • The practice of doing software
  • The practices involved in doing software
  • Being able to deal with people is sometimes called  having “soft skills
  • Soft power

What do you think about “post-Agile”?

* I’ve even set up a company called Soft Practice to do this stuff, that’s why I’ve not been blogging much recently, who knew there’d be so many forms to fill in!

Edit 14/8/13: Seems others are now talking about the same things: Ken Schwaber, Dave Anderson, Aterny

Process vs. Tools

What’s more important: Process or tools?

I’ve sat through numerous arguments with people trying to work out whether process or tools are more important to their software development effort. The arguments go along the lines of:

Process Geek: Tools are just the things you use to do the process, the value is in the process. Let’s do a process first roll out.

Tools Geek: Tools are the things people actually use. Process is academic and is constrained by the tool features anyway. Let’s do a tools based roll out.

Sadly both of these are real life quotes from people I’ve worked around in the past. I won’t embarrass them by naming them because they’re both really wrong. For me the real value is not in the tools or the process but in the people. Process and tools are something that you add if the people need them.

Minimal Tooling

For me the best planning tool is a whiteboard. The best architecture and design tool is also a whiteboard. I like to capture some info about what I’m doing and how close I am to achieving my goals, some really simple measures. The best tool for that is probably Excel, despite my love of Open Source. Other than that I need to manage some lists of things like requirements (stories/use cases whatever), defects and whatever other bits the team thinks are important. They might fit on a whiteboard but typically we want to have some attributes on them, be able to sort them and link them. The one downside of a white board is the lack of drag and drop. I’ve used a few different types of smart board and active touch screens but none of them have really given me the smooth experience I want.

The one bit of tooling I really need is my IDE + SCM + Continuous Integration.

Minimal Process

This is potentially a can of worms if I go into things like process kernels and ideological arguments about different agile approaches and practice collections. There’s no such thing as a process silver bullet so for me though it’s all about the team and being effective. Teams need to:

  • Have Clear Obligations
  • Have Clear Business Priorities
  • Seize their autonomy
  • Have the relevant technical skills and resource availability
  • Be active in actually doing their work (seems obvious but it’s something a lot of processes seem to hinder rather than help)
  • Demonstrate visible progress (via working software not a tracking gantt chart)
  • Continuously improve themselves (because no way of working is perfect)

This list based on the 7 Habits of Agile Teams by Roly Stimson

So why go more complicated?

Unfortunately life is rarely simple. Many teams work in complex environments where a number of factors might make them choose to need more than this kind of minimal set including things like physical distribution; audit, regulatory or security constraints; cross-team interaction in system of systems environments, more people etc.

When faced with increased complexity it might be appropriate to add more powerful tooling or development practices. The more people involved in a project the harder it is to herd all of those socially complex cats into working together.

This is why ALM suites exist ranging from plugging together some open source bits like git, SimpleGit, jenkins, agilefant, bugzilla etc. to commercial vendor products like IBM Rational RTC (which does agile planning, work item management, scm and build in one tool) and other IBM Rational Jazz tools – it’s worth noting that you can also get RTC for free for just 10 users or for academic/research use on jazzhub.

Alternatively there’s Microsoft TFS for source control, work item tracking etc. tying together existing Microsoft stuff like MS Office, Visual Studio and sharepoint if you’re a more Microsoft focussed development house.

Of course you can also mix and match this stuff to a greater or lesser degree with various integrations. No toolset is a silver bullet however and each have their own problems.

So what’s my point?

My point is that process and tools need to reinforce each other, that both should be selected based on the needs of the team and the environment the team has to operate in. In the process vs. tools argument both lose out to the people. People in a development organisation create value, they just use tools and process to do it.

What does “Agile at Scale” mean?

There’s a lot of talk in the process improvement industry about the meaning of “agile at scale”, devops, lean and agile for a number of reasons. One reason that I’ve seen in successful agile organisations is that as development maturity increases with true agile adoption bigger problems present themselves. This is the natural progression of the science of development, what used to be considered complex (like Object Orientation) is now normal, a commodity. Innovation in ways of working is happening at the organisational, cross-team, cross-product level.

For me agile at scale (I’ve got fed up of the quotes already) means a couple of different things:

  • Repeating agile successes embodied in a team across an organisation (scaling out?)
  • Applying agile thinking to cross-product projects
  • Applying agile and lean thinking to development organisations
  • Applying agile and lean thinking to high assurance environments like medical, security, financial, safety critical, audited, regulated businesses.

Agile and lean? Yep, both with lower case letters. I’m not particularly interested in ideological approaches to software development, I believe strongly in taking the best bits of whatever processes, techniques, practices etc. you find externally, mixing them up with internal practices and ways of doing things to develop simple, pragmatic approaches to ways of working. Both agile and lean schools of thought promote minimising unnecessary work, shorter delivery cycles and higher quality, continuously learning lessons and empirical decision making.

The agile manifesto gave us:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.

Some great practices have evolved for applying agile and lean thinking like scrum, kanban etc. However all of the complex organisations I’ve worked with have found that there’s still space for more thinking in terms of how to run a software business, how to deal with big complex system-of-system problems, multiple competing stakeholder sets, programme and portfolio management etc. Not surprising really because the agile movement wasn’t about trying to do any of that stuff.

However organisations who are successful with agile transformations want to apply the successful open and honest philosophy behind the agile manifesto to other parts of their business as well as bigger and bigger projects and programmes because the results of “doing agile” (I promise I’ll stop with the quotes soon!) are so attractive when it’s done well, namely:

  • Shorter delivery cycles, higher quality
  • Deep engagement between customers and development teams leading to respect, collaboration and better morale
  • Quick identification of when things are starting to go wrong

Consider the following model, not uncommon amongst large organisations:

Diagram of nominal large inter-dependant organistion structure

This represents a typical software department or vertical section of a software department with a portfolio that provides the funding for work. Big portfolio’s are normally broken down into a number of programmes which in turn may levy high level requirements onto organisations (organisational sub-divisions that own multiple products) which may affect both legacy and new product development. Often within a vertical section of a business there will be many cross-dependencies in terms of requirements, technical dependencies etc. For many large businesses this picture is overly simplistic, indeed I’ve not included things like projects, component teams and a variety of business organisation constructs like product centres, feature teams etc.  So how do you apply agile and lean philosophy to this lot and more?

You can’t simply repeat the same practices recursively throughout an organisation to deal with larger scale complexity.  Imagine a chain of scrum-of-scrums, daily stand-ups at every level (at the same time, or staggered to allow representation up the chain?), sprint plans at programme level etc. What about if the business is regulated, audited, security focussed, high risk financial, safety critical, etc.

Ok, so what’s agile at scale then?

Agility at Scale is applying the spirit of agility and lean thinking if not the letter to these bigger problems. It’s about:

  • Valuing individuals and interactions, encouraging collaboration, reducing layers of communication over processes, tools and hierarchy
  • Valuing working software in the form of quality releases from short development cycles over comprehensive documentation, business analysis, enterprise architecture documentation
  • Valuing customer, business, developer and operations (see DevOps) collaboration over contract negotiation
  • Valuing good governance, transparency and honesty in progress, plans, costs and impediments over regular reporting
  • Valuing responding to change over following a plan at all levels of the business

(Borrowed from the Holistic Software Manifesto)

Agility at scale is focussed simply on reducing unnecessary bureaucracy, reducing time to market and improving value.

So how do you achieve it?

The application of:

Of course each of those (and more!) is a complex can of worms in itself. A lot of these higher scale practices are only just emerging from high maturity complex (post-)agile organisations but in time more of those things will turn into links.

A good example of  “Agile at Scale” in action is the Project Forum practice

As always this blog is  a stream of consciousness so please, let me know your opinion?

%d bloggers like this: