Mike MacDonagh's Blog

Somewhere in the overlap between software development, process improvement and psychology

And about me

I’m some way between a tools geek, process geek, (post-)agile geek, psychology geek and programming geek.

I work with both private and public sector organisations worldwide to help improve the way software is done.

I’m a UK based consultant with over 10 years experience in the software industry enabling teams in the adoption of various software engineering practices and associated toolsets – providing training, consultancy and support in both small teams and entire development organisations (1000s) – scaling agility and lean concepts at the organisational level and de-conflicting them with portfolio, programme and project management.

I specialise agile, lean and iterative methods and practices ranging through the practical implementation of scrum, kanban and RUP, business and system analysis covering both use cases and user stories (often together), requirements management, architecture and design as a sketch, iterative and continuous flow patterns and continuous integration practices. I don’t believe in slavishly following a process or meta-model but in collaboratively improving the software value stream.

I’m a co-Author and co-Founder of Holistic Software Engineering which brings together all of these things in one simple picture.

I’m the Technical Director of Holistic Software Consulting Ltd and also Soft Practice Ltd.

I’ve got deep expertise in various development tools such as (git, jenkins etc.) as well as commercial tools IBM (Rational toolset – RTC, RRC, RQM, RSA etc., Microsoft TFS) – I also make my own.

I’m interested in new technology, psychology, kung fu, music and guitar playing, travelling, philosophy, linguistics, close up magic, biology, maths, physics (especially astronomy and quantum physics) etc…

Launch: Holistic Software Engineering

How do we join up business strategy to agile development? Is program management relevant? Where do project managers fit in? What about architecture?

Holistic Software Engineering (HSE) answers all of these questions – for free.

Agile and continuous flow are great for small teams or a small number of inter-related small teams working on exploratory or maintenance work. But what if we’re spending 100s of millions on an IT strategy of inter-related products that need to work together to deliver business value. What is business value anyway?

H-Model To answer these questions (and more) my friend Steve Handy and I have distilled our collective 30+ years of software experience in a single, cohesive model of software development. We’ve developed the H model that moves on from the v-model and it’s siblings by combining:

…all elegantly combined and de-conflicted by release planning.

We’ve not invented very much, we’ve simply put a lot of good ideas from others together into a cohesive framework. We’ve drawn it all as a big picture and developed a website that shows how to get value from all of this stuff. Everything is clickable, everything has content.

The best bit: it’s free! There’s no paywall, there’s no private “full” version, you can just use it or not as you like.

We don’t believe in process zealotry, or putting academic concerns above clarity and usefulness. HSE is indicative, not prescriptive. You’re already doing it and if you use the big picture to draw red blobs on the bits that aren’t working well, or missing, in your organisation then you can use the model to make tangible improvements – immediately.

Using HSE doesn’t replace any of your existing processes, it provides the glue that joins them all up together in one simple, elegant and cohesive model.

Holistic Software Engineering
is for large/complex software organizations
who need to understand modern software engineering in a business context
our solution is to present a big picture of software business backed up with practical detail avoiding academic or heavyweight process documentation
that covers people issues, business strategy, portfolio, programme and project management as well as architecture, technical delivery and integration
unlike simple small team based processes such as RUP, Scrum etc.
The big picture of software engineering

Holistic Software Engineering

And if it’s too big, we’ve got a small picture, which is essentially “normal” agile development.

Please share HSE with your friends and colleagues.

No more Project Managers, bring in the Movie Producers

I was reading some course material recently that was trying to teach people something to do with software development and it was using the same old tired “ATM machine” example. I’ve worked with hundreds of projects, many in the finance sector and none of them are anything like an ATM machine.  One of the reasons that example is soooo tired is that it’s describing commoditised software development, it’s something I’d expect to buy off a shelf or go to a specialised vendor for. It’s not something I’d put my team of uber |33t  haxorz on.

Developing software products is a balance between a creative and scientific pursuit, so it’s a little hard to describe, especially when that software can range from a tiny smartphone app, to a website, to router firmware, to an enterprise hr system (urgh!), to a big data processing system of systems etc. You get the gist.

The things these types of system have in common with each other is how different they are. And yet the traditional approach to managing these diverse kinds of work has been classical project management with a one size fits all process (I’m looking at Prince2, RUP, Iterative, Agile etc.) or worse hiring a large company full of body-shop project managers. For me this is one of the root causes behind large scale IT disasters.

I once had a discussion with the leader of a PMO in a large organisation of thousands of people about the nature of Project Management for software and he assured me that “someone who knows how to manage a bit of work can manage software, it’s just a different type of work” and indeed I should “stop thinking of software as special”. I’ve seen this attitude resonate in many organisations and is to me best summed up by the same Head of PMOs statement “from a project management point of view building software is the same as building a bridge”.

Now then. I know a little about software development, but not that much about bridge building (if you exclude my illustrious Lego engineer career when I was 7). If I managed the building of a bridge next door to one build by someone with a track record of bridge building who’s bridge would you drive your family on? Not mine if you’ve got any sense.

There are so many problems in the software development industry caused by people not understanding it and applying bad metaphors. When people who do understand it start to get the point across (e.g. the early agile movement) they often get misunderstood by the old guard who dress up their normal ways of working in new language.

Many of our “best” metaphors come from manufacturing lines where the same thing is made over and over again. That’s nothing like software.

To me a software project is more similar to making a movie than anything else:

  • It’s a unique one off product (or a remake of a previous version) with new parts working together in new ways
  • Once made we’ll often duplicate and digitally distribute
  • It can cost a lot of money because it can be very complex and needs lots of specialist skills
  • You need a high level plan for how you’re going to make it
  • There’s no guarantee on return
  • What’s written down originally bares little resemblance to the finished product
  • We know when it’s good or bad but it’s hard to objectively quantify
  • There’s lots of different types that need different collections of specialist skills
  • Both involve people and time and so are subject to change, they have to be adaptive
  • You can tell when the people making it had fun
  • It’s not feasible that any team member can fit in any role
  • There’s almost always going to be some quality problems
  • You wouldn’t get a movie maker to build your bridge
  • You wouldn’t get a bridge builder to make your movie
  • You don’t make a movie faster by telling the actors to act faster

So, I don’t want a Project Managers that know how to work a gannt chart. I want movie producers that know how to work with a team holistically to get the best out of them both technically and artistically.

 

Agile contracts? What about Weighted Fixed Price?

MichaelangeloAgile contracts are tricky. In fact Software Development contracts are tricky. Software development is a non-deterministic creative endeavour, you’d be better off trying to nail down the acceptance criteria for one of Michelangelo‘s sculptures than a typical software project. Software is complex.

A few years ago I was involved in putting together a contract model called “Weighted Fixed Price” and recently it occurred to me that I’ve never blogged about it, so I’ve dragged myself away from the source to do so.

What’s wrong with the normal models?

So, fixed price is bad because it only works when you can really accurately define the requirements, acceptance criteria and the potential risk exposure is tiny. Otherwise you get schedule and cost over-runs, formal change control etc. etc. Of course supplier’s always build in their risk buffer into the fixed price so it tends to cost a little more than honest T&M would. Software work is rarely this predictable which is why there are so many high profile failures.

Time and Materials is bad because it’s hard to manage across commercial boundaries and customers can feel that they’re paying for nothing, endlessly. Even defining waypoints is difficult due to changes in requirements and over time. Unscrupulous suppliers could just sit back taking the money, delivering as little as possible to avoid gross negligence but nothing more. Scrupulous suppliers could be working hard but hit by the many and varied risks arising from such complexity and yet perceived by their customers as the unscrupulous.

Of course there’s variations like Capped T&M, Phased Fixed, Phased Capped T&M etc. but they all start getting a bit complicated and hard to manage, not to mention open to gaming.

Then there’s models like pay-per-point and other “incentive” schemes that raise over-complication, missing the point and gamability to an art form. Just say “no”.

Competition

The real incentive to suppliers to do a good job is follow on work. That means that as a customer you need your suppliers to be in a competitive market, being locked in to individual suppliers leads to captive market behaviours.

Weighted Fixed Price?

In an attempt to put something in the middle between Fixed Price for low risk exposure and T&M/internal for high exposure I put together “Weighted Fixed Price”. The idea here is to to basically use a Fixed Price model but have suppliers weighted by previous performance affecting whether they get the business or not.

So if a supplier 1 delivers 10% late on Project A then when we’re looking to place Project B for 500k and we’re looking at each supplier’s estimates then we’ll add 10% to supplier 1’s estimate.  That means all over things being equal we’ll choose the other supplier’s who typically come in closest to, or even under, their previous estimates. Weighting is transparent to each supplier but not across suppliers.

This provides the following pressures:

  • Suppliers don’t want to underestimate, they’ll come in over their initial estimates and adversely affect their weighting and therefore their chance for follow on work
  • Suppliers don’t want to overestimate to artificially lower their weighting or build in risk buffers, they won’t get the work because they’ll cost too much
  • Suppliers don’t want to take on complex, high risk work because they can’t predict the effect on their weighting (more below)

Out-sourcing doesn’t transfer risks to suppliers, because the customer is still the one needing the business benefit and paying for it, especially when things go wrong. The Weighted Fixed Price model is intended to genuinely transfer the complexity risk to suppliers, if they’re willing to take it.

What happens when the supplier finishes early?

They deliver early, watch their weighting affect their potential for winning future work increase massively and enjoy their fixed price money meaning that they’ve increased their profit margin on this job.

What happens when they deliver late?

They deliver late, their weighting increases and they’re less likely to win against other suppliers next time. However, if they could predict that they’re going to deliver late early in the project then they could renegotiate with the customer in light of what everyone’s learned about the work.

What happens when suppliers don’t want to bid for work because it’s too risky/complex?

Excellent, this means that they’re honestly telling us that the work is too risky to predict. This means we shouldn’t be out-sourcing it in the first place, time to stand up an internal team or supplement with some time-hire contracts if we need extra capacity.

Conclusion

The Weighted Fixed Price model is an evidence based contract model can be used in competitive supplier markets for reasonably well understood work with low risk exposure. It incentivises suppliers to estimate accurately and deliver to their estimates as that affects their opportunities for follow on work. It incentives customers not to out-source overly complex risky work.

It’s not perfect because we’re still trying to contractually define how a sculpture should make us feel but it’s something you can add to the mix of contract models you use.

How to make real progress in a software project

I’ve not been blogging much this year because I’ve been crazy busy writing software (more on that soon I hope). As you might expect I’ve changed my ways of working more often than process improvement consultants change the name of the process they’re selling but one key thing always seems to drop out whether I’m working on my own, with a team, a team of teams or a globally distributed team.

Do the top thing on the list, whatever it is.

Don’t do anything else until it’s done.

Or in process speak: “Reduce the WIP and clarify the Definition of Done“.

Too often I fall into the trap of doing the most interesting thing from the list, which normally means the most complex. And far too often I get carried away with the next interesting thing before finishing something. Both of these lead me to have far too many things in various states of progress all interfering with each other and me with no idea of how well I’m doing. And that’s assuming I actually know what “finished” means.

Software is complex enough without us making it harder with processes, remember the KISS principle?

Reduce waste: Visualise the value stream

A big thing in software process improvement (urgh) is reducing waste.  A great idea, but how do you identify waste?

One powerful method that I use is value stream visualisation. There’s a number of ways of getting information about the value stream ranging from physically walking the chain to simply asking people.

Sounds simple doesn’t it? But one of the best sources of information about wasteful processes and ways of working in a team is simply asking the team what they feel they shouldn’t be spending time on. By freeing up people to actually do their job they can reduce waste and improve productivity. One method I use is to ask each team member to identify a list of n items that they have to spend time on that they think are either entirely or partially wasteful. Basically a list of their impediments. I then anonymize the list look for commonality and then the top few are a reasonable target for improvement.

Alternatively actually walking the path of a piece of work through an organisation can really open the eyes to the sheer number of people and unnecessary waiting involved in some wasteful processes…

Anyway, having got some data I find it useful to visualise it in terms of waiting time and doing time. In Lean-speak cycle time is the time it takes to actually do something, lead time is the time it takes from the request being made to it finally being delivered. In many tasks the lead time far outstrips the cycle time, visualising it can make that very clear.

Low Lead time For example, a task with has a lead time of 5 days might actually be made up of 2 days waiting time before it actually gets picked up and the cycle time of 3 days starts.

Large lead timeIf you look at all of the tasks in your value chain you might find that many of them look more like this, with very large wasteful waiting times. There are many reasons why there might be a large waiting time, typically the team doing the tasks is over-subscribed, or a team might be protecting it’s SLAs, KPIs or other TLAs. These problems can have significant effects when looking at the system as a whole.

Even more interesting is looking at how these tasks might fit together in a value stream. If we imagine a business function that involves a request moving through a bunch of tasks across a number of teams that have different waiting and doing times a simple sequence of tasks, with a little buffering between them from the Project Manager to allow for schedule variance naturally, might end up looking like this:

 sequential_value_chain_arrows

Here we have 26 days of actual work spread over almost 60 days (12 weeks) of elapsed time = 40% waste. Even if the person planning the work is willing to accept some risk and try to optimise their workflow a bit without improving the waste inherent in the tasks involved there’s still a lot of waste in the system.

parallel_value_chain_arrows

By visualising the value stream in this way we can see straight away (apologies to red/green colour blind folks) that there’s a lot of red, a lot of waste. In many cases planners aren’t willing to accept the risks inherent in overlapping activities as shown here, or aren’t even aware that they can leading to the more sequential path shown above. The result is a minimum time that it takes a request to get done, based on the impedence of the current value chain of, in this case, 38 days before we even start thinking about actually doing any work.

Surely that’s not right.

What is “good” software architecture or design?

Architecture is a fine balance between a subtle science and exact art that combines cognitive problem solving, technical direction and expressing abstract views to aid common understanding. Design is similarly an inexact science which is inextricably linked to Architecture. As a result it’s quite hard to define what makes a “good” architecture or design.

There are high numbers of design and architecture practices ranging from the purely transformational to the evolutionary and emergent (both manual and automated) but there seems to be some heuristic consistency between these various practices and the approaches of the traditional, agile and post-agile movements.

Initially I saw the simple descriptions of that makes for “good” design from from Ron Jeffries and Kent Beck in Xtreme Programming. I like the style and direction of this kind of description so here’s my take on the characteristics that make for “good” design or architecture:

  • Intentional structure and behaviour
  • Highly modular: consisting of separate services, components, classes, objects or modules
    • Elements are highly cohesive
    • Elements are loosely coupled
    • Elements have low algorithmic complexity
  • Avoids duplication
  • Well described elements: modular elements have simple expressive meaningful name enabling readers to quickly understand their purpose, similar to code cleanliness we should strive for design cleanliness
  • Runs and passes all defined tests or acceptance criteria
  • Lightweight documentation

Intentional vs. Emergent Architecture

I’ve been thinking about architecture a lot recently but one thing that I often discuss but have never blogged about for some odd reason is intentional vs. emergent software architecture. Some old fashioned software methods such as waterfall led people into doing a lot of up front architecture work, they analysed and designed away for ages producing huge reams of UML and documentation that no one could ever squeeze into their heads if they had the patience to read it. This is an example of an intentional architecture – the architecture was intended, planned and deliberate.

Lots of folks hated that way of doing things as it meant people were writing docs and drawing diagrams instead of making working software, not to mention an all to frequent tendency to over-engineer architecture past the point of usefulness. This led to some people saying that we’re better off not trying to do any architecture and just letting it emerge from the work we do developing small little customer focused requirements (like user stories or similar).  Ok, so there’d be some rework along the way as we encounter a deeper understanding of the system and refactor our emergent architecture but it’d still be better than the old way of doing large upfront architecture.

So, there seem to be two opposed viewpoints: intentional architecture is best, emergent architecture is best.

For me, neither is true. I’ve seen some really terrible examples of badly over-engineered architectures that crippled a project and projects that never got past their over-reaching architectural analysis. Equally I’ve seen products with emergent architecture that had to be entirely re-architected as time went pay because their emergent architecture was so wrong it was comical (imagine a software management tool that only supports a single project, and that concept being deeply embedded in the architecture).

There’s a scale with intentional architecture on one side and emergent architecture on the other.

Intentional vs. Emergent ArchitectureVarious factors might push us one way or another… The second I listed on the right is interesting as if you’ve got a well known technology and problem domain you can get away with emergent architecture, but similarly if you have a totally unknown technology and problem domain it can be very effective to evolve towards a solution and architecture rather than crystal ball gaze by creating a (probably wrong) intentional architecture.

Which rather sums up the point I’m trying to make. The purpose of architecture is to shape the solution and address technical risks. Solving the big problems, creating common ways of doing something (lightweight architectural mechanisms) are all good architectural goals but only if we’re sure of the solution. If we’re not we’re better off evolving an emergent architecture, at least initially.

I think that the extremes at either end of the scale, as with most extremes, are a bad idea at best and impossible at worst. If you gather a group of people together and tell them to create a web app given a backlog but they’re not allowed to think or communicate about the architecture up front then you’ll find they all start dividing the problem in different ways and writing in different languages for different server and client frameworks. Hardly a good idea. Of course on the other end of the scale, believing that we can foresee all of the technical issues, all of the technology and requirements changes that might happen is as likely as a 12 month project plan Gantt chart being correct after a few levels of cumulative error margin have been combined.

For more on architecture see:

Understand the software value chain by walking it

There’s no better way to understand something than by actually giving it a go. If you want to know what writing a particular programming language is like you need to try it. If you want to know what kung fu is like watching a video won’t help, you need to try it.

If you want to really understand the software value chain in your organisation drawing diagrams in an ivory tower isn’t going to help, you need to get out there and “Go and see”.

In Steve Handy’s blog on Scaling Software Agility he talks about applying some of the thinking from the Lean school of thought to “go and see”, observing the value chain by actually physically following it.

  • Identify a new piece of work and follow it through the delivery chain
  • Attend every meeting, track the activities and note how long it’s static.
  • Don’t attempt to fix it during this period of observation, don’t criticise, just watch and learn.

I think there’s some real value here, there’s something quite simple and powerful about physically walking the walk of a piece of work through the software value chain. Doing so will make some problems, blockages and issues blindingly obvious – processes and organisational structures that seemed to make sense on someone’s bit of paper will just not make sense.

Experiencing the value chain in a very physical way promotes professional mindfulness and will clearly identify waste. I think this is a great way of putting common sense back in to software development.

I call this Go See, and it’s the most effective form of measurement for software development.

 

A cross-project Release Burnup?

Huh?

Points are abstract and relative, comparing them across projects is like comparing apples and elephants.

Releases only make sense in themselves, let alone across project time lines so why would I want to look at a cross-project burnup? Such a thing is foolishness surely…

Well maybe not, it might be useful as an agile at scale measure, as a way of looking at the work churn in a department or other high level unit in an organisation. I’ve been pondering if there’s a way of aggregating burnup information and point burnage across teams (with distinct disjoint timelines) and thought that there might be a way.

Ideally I want to be able to show the amount of work planned within an organisational group and progress towards that scope, showing when scope goes up and down (does that ever happen?). Then I want to show progress towards that scope overall, the angle of the progress line could give me an organisational velocity – perhaps I could even add an ideal velocity that would indicate what perfect robots would to if real life never intervened (although that could be dangerous).

Aggregate burnup

First I need a way of normalising points and understanding what 100% of scope means when it can be incorporating many projects at different points in their lifecycles. Perhaps a way of doing it is considering everything in terms of percentage, after all that’s an easy thing for people to consume. To define 100% of a scope of various contributing projects is tricky since it’ll change and be dependant on their releases, continuous flows, changing scope.

A simplistic approach to this is to use a moving baseline, perhaps we can determine 100% of a projects scope as whatever it thinks it would deliver within the time area being considered (the scope of the x-axis) at 15% of it’s timeline (or whatever).

In the example above this tells me that work is consistently overplanned not just in terms of actual velocity, but in terms of idealised capacity aswell – the demand is higher than the supply. I think this could be useful for “agiley” portfolio management.

Perhaps I could start establishing a budget cycle velocity, and start limiting work planned based on empirical evidence. Ok, so no project is the same and points aren’t comparable but the Law of Large Numbers is on my side.

What do you think? Is this bonkers?

Follow

Get every new post delivered to your Inbox.

Join 388 other followers

%d bloggers like this: