How do we join up business strategy to agile development? Is program management relevant? Where do project managers fit in? What about architecture?
Holistic Software Engineering (HSE) answers all of these questions – for free.
Agile and continuous flow are great for small teams or a small number of inter-related small teams working on exploratory or maintenance work. But what if we’re spending 100s of millions on an IT strategy of inter-related products that need to work together to deliver business value. What is business value anyway?
To answer these questions (and more) my friend Steve Handy and I have distilled our collective 30+ years of software experience in a single, cohesive model of software development. We’ve developed the H model that moves on from the v-model and it’s siblings by combining:
- Business strategy
- People and team issues
- Iterative and feedback loops
- Lightweight requirements and architecture
- Lean portfolio and program management
- Agile and continuous product delivery
- A focus on integration, quality and business value
…all elegantly combined and de-conflicted by release planning.
We’ve not invented very much, we’ve simply put a lot of good ideas from others together into a cohesive framework. We’ve drawn it all as a big picture and developed a website that shows how to get value from all of this stuff. Everything is clickable, everything has content.
The best bit: it’s free! There’s no paywall, there’s no private “full” version, you can just use it or not as you like.
We don’t believe in process zealotry, or putting academic concerns above clarity and usefulness. HSE is indicative, not prescriptive. You’re already doing it and if you use the big picture to draw red blobs on the bits that aren’t working well, or missing, in your organisation then you can use the model to make tangible improvements – immediately.
Using HSE doesn’t replace any of your existing processes, it provides the glue that joins them all up together in one simple, elegant and cohesive model.
And if it’s too big, we’ve got a small picture, which is essentially “normal” agile development.
Please share HSE with your friends and colleagues.
I was reading some course material recently that was trying to teach people something to do with software development and it was using the same old tired “ATM machine” example. I’ve worked with hundreds of projects, many in the finance sector and none of them are anything like an ATM machine. One of the reasons that example is soooo tired is that it’s describing commoditised software development, it’s something I’d expect to buy off a shelf or go to a specialised vendor for. It’s not something I’d put my team of uber |33t haxorz on.
Developing software products is a balance between a creative and scientific pursuit, so it’s a little hard to describe, especially when that software can range from a tiny smartphone app, to a website, to router firmware, to an enterprise hr system (urgh!), to a big data processing system of systems etc. You get the gist.
The things these types of system have in common with each other is how different they are. And yet the traditional approach to managing these diverse kinds of work has been classical project management with a one size fits all process (I’m looking at Prince2, RUP, Iterative, Agile etc.) or worse hiring a large company full of body-shop project managers. For me this is one of the root causes behind large scale IT disasters.
I once had a discussion with the leader of a PMO in a large organisation of thousands of people about the nature of Project Management for software and he assured me that “someone who knows how to manage a bit of work can manage software, it’s just a different type of work” and indeed I should “stop thinking of software as special”. I’ve seen this attitude resonate in many organisations and is to me best summed up by the same Head of PMOs statement “from a project management point of view building software is the same as building a bridge”.
Now then. I know a little about software development, but not that much about bridge building (if you exclude my illustrious Lego engineer career when I was 7). If I managed the building of a bridge next door to one build by someone with a track record of bridge building who’s bridge would you drive your family on? Not mine if you’ve got any sense.
There are so many problems in the software development industry caused by people not understanding it and applying bad metaphors. When people who do understand it start to get the point across (e.g. the early agile movement) they often get misunderstood by the old guard who dress up their normal ways of working in new language.
Many of our “best” metaphors come from manufacturing lines where the same thing is made over and over again. That’s nothing like software.
To me a software project is more similar to making a movie than anything else:
- It’s a unique one off product (or a remake of a previous version) with new parts working together in new ways
- Once made we’ll often duplicate and digitally distribute
- It can cost a lot of money because it can be very complex and needs lots of specialist skills
- You need a high level plan for how you’re going to make it
- There’s no guarantee on return
- What’s written down originally bares little resemblance to the finished product
- We know when it’s good or bad but it’s hard to objectively quantify
- There’s lots of different types that need different collections of specialist skills
- Both involve people and time and so are subject to change, they have to be adaptive
- You can tell when the people making it had fun
- It’s not feasible that any team member can fit in any role
- There’s almost always going to be some quality problems
- You wouldn’t get a movie maker to build your bridge
- You wouldn’t get a bridge builder to make your movie
- You don’t make a movie faster by telling the actors to act faster
So, I don’t want a Project Managers that know how to work a gannt chart. I want movie producers that know how to work with a team holistically to get the best out of them both technically and artistically.
When I started doing software I was a simple developer interested in elegant code and shiny things. I worked in a small software house that taught me many bad practices in terms of configuration management, change control, estimation, management etc. It was an incredibly valuable experience for me in learning how not to do things and led me to strive for something better even though at the time I didn’t really know what it might be.
That led me through a path of process definition and documentation (RUP) and a rather limited following of an iterative lifecycle. I remember a conversation I once had with one of my project managers (when I was an architect) asking if we were going to do “Use Case Analysis” on a particular project or just skip straight into “Use Case Design”. The question puzzled me then because it felt like a trap and I didn’t really understand what I was being asked.
It puzzles me even more now as I know understand “analysis” to be a stop-and-think-for-a-moment activity and I’d always do a bit of analysis of my requirements, however these days it’s extremely unlikely that it’d be an analysis UML model, instead it might be a sketch, a conversation and a bit of thinking before another conversation.
So I moved from process prescription to process understanding, applying the spirit of doing things well rather than the letter of whatever current process law was in fashion. Following on from that I’m not really interested now in the details of what a book says someone’s role should be or how people should interact instead I think the real challenges in software development are social, not collections of technical practices (although there is value in evolving better practices and tooling).
A colleague of mine commented recently that 20 years ago when he was doing software it was possible to understand everything from the metal all the way to the blinking lights on his bit of hardware. Software development in just 20 years has progressed incredibly and it’s just not that simple any more. We can understand the basics all the way through the stack but not all of the details. There are so many bits interacting that it’s just too complex. As a result the problems, technology and team working are all abstracting away from the tangible mechanistic past.
Developers are on a path from technical skill to mastery, as they begin to understand the kung fu of software engineering they can apply experience and deep understanding to solving the technical problems, doing away with formal process and just using the practices that they intuitively need, happily breaking the “rules” to get the job done in an efficient high quality way. I’m not entirely sure where brogrammers are on this evolutionary path, I’ll leave that up to you.
The problem, if it even is a problem, is that each person in an organisation is somewhere different along this path and even if they’re at the same point in the same dimension they might not be aligned in their interests and motivations. This makes team working amongst inherently complex social creatures a tricky proposition. The sweet spot is a team of fully enlightened software kung fu masters but that’s a really hard target to meet for a number of reasons. Consider the flower of team working evolution weirdness, which area is your team in?
One reason that this sweet spot is difficult to hit is because in any organisation half of the developers are below average in technical and social skills. Sounds horrible but is obviously true and the larger the organisation the more likely it is to be the industry average, not the organisational average.
This means that the centre of skill gravity for any team is unlikely to be on the w00t side of the scale.
In most large organisations the teams are more likely to be flattened pancakes across this bell curve taking in a reasonable representation of the organisation as a whole (especially as the highest skilled are often distributed amongst an organisation to attempt to bring up other teams.
This isn’t necessarily a problem though as there is value in diversity and individuals will each have different ways of thinking about things that can bring value to teams and organisations.
I’m beginning to think that the purpose of classic software process, and classic software process improvement is to try and move people along this scale from basic developer to software ninja, helping them to gain mastery through experience and feeding in the experience of others. When a team moves along this scale they begin to not need their process mentors any more and will seize their autonomy.
Technology is increasingly commoditised by innovation, as is software development and software process. I used to spend a lot of time teaching people Object Orientation but these days everyone just seems to know it, it’s nothing special or new, it’s just what people do. Similarly people are increasingly aware of the value of iteration, limiting work in progress, continuous integration and delivery. They need less process and less instruction as this stuff is becoming business as usual at least in more mature organisations.
As these problems are being solved I think that bigger problems are coming to the fore as they’re less hidden by low level development issues. The questions I see a lot of clients wrestling with are things like:
- How can we foster the right kind of organisational culture?
- How do we deal with requirements, architecture, releases, resources etc. in a complex system of systems environment?
- How do we bring together multiple divergent interpretations of “agile”?
- How do we manage outsourcing contracts in high speed agile projects?
- How do we motivate and engage the business and technical communities?
- etc. etc.
Solving these problems is less about technical skills and technical process content and more about social skills, psychology and understanding. Both from a coaching and business perspective.
So what’s my point?
My point is that software process improvement needs to focus less on individuals and more on teams, and teams of teams. That we should avoid ideology and take the best bits of knowledge and experience wherever we find them, growing our teams and individuals.
Also that we should apply more psychology to software process and business change finding socially resonant patterns for how we do things. True mastery involves not worrying about “breaking the rules” and from the outside can easily be confused with ineptitude.
There’s a lot of talk in the process improvement industry about the meaning of “agile at scale”, devops, lean and agile for a number of reasons. One reason that I’ve seen in successful agile organisations is that as development maturity increases with true agile adoption bigger problems present themselves. This is the natural progression of the science of development, what used to be considered complex (like Object Orientation) is now normal, a commodity. Innovation in ways of working is happening at the organisational, cross-team, cross-product level.
For me agile at scale (I’ve got fed up of the quotes already) means a couple of different things:
- Repeating agile successes embodied in a team across an organisation (scaling out?)
- Applying agile thinking to cross-product projects
- Applying agile and lean thinking to development organisations
- Applying agile and lean thinking to high assurance environments like medical, security, financial, safety critical, audited, regulated businesses.
Agile and lean? Yep, both with lower case letters. I’m not particularly interested in ideological approaches to software development, I believe strongly in taking the best bits of whatever processes, techniques, practices etc. you find externally, mixing them up with internal practices and ways of doing things to develop simple, pragmatic approaches to ways of working. Both agile and lean schools of thought promote minimising unnecessary work, shorter delivery cycles and higher quality, continuously learning lessons and empirical decision making.
The agile manifesto gave us:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.
Some great practices have evolved for applying agile and lean thinking like scrum, kanban etc. However all of the complex organisations I’ve worked with have found that there’s still space for more thinking in terms of how to run a software business, how to deal with big complex system-of-system problems, multiple competing stakeholder sets, programme and portfolio management etc. Not surprising really because the agile movement wasn’t about trying to do any of that stuff.
However organisations who are successful with agile transformations want to apply the successful open and honest philosophy behind the agile manifesto to other parts of their business as well as bigger and bigger projects and programmes because the results of “doing agile” (I promise I’ll stop with the quotes soon!) are so attractive when it’s done well, namely:
- Shorter delivery cycles, higher quality
- Deep engagement between customers and development teams leading to respect, collaboration and better morale
- Quick identification of when things are starting to go wrong
Consider the following model, not uncommon amongst large organisations:
This represents a typical software department or vertical section of a software department with a portfolio that provides the funding for work. Big portfolio’s are normally broken down into a number of programmes which in turn may levy high level requirements onto organisations (organisational sub-divisions that own multiple products) which may affect both legacy and new product development. Often within a vertical section of a business there will be many cross-dependencies in terms of requirements, technical dependencies etc. For many large businesses this picture is overly simplistic, indeed I’ve not included things like projects, component teams and a variety of business organisation constructs like product centres, feature teams etc. So how do you apply agile and lean philosophy to this lot and more?
You can’t simply repeat the same practices recursively throughout an organisation to deal with larger scale complexity. Imagine a chain of scrum-of-scrums, daily stand-ups at every level (at the same time, or staggered to allow representation up the chain?), sprint plans at programme level etc. What about if the business is regulated, audited, security focussed, high risk financial, safety critical, etc.
Ok, so what’s agile at scale then?
Agility at Scale is applying the spirit of agility and lean thinking if not the letter to these bigger problems. It’s about:
- Valuing individuals and interactions, encouraging collaboration, reducing layers of communication over processes, tools and hierarchy
- Valuing working software in the form of quality releases from short development cycles over comprehensive documentation, business analysis, enterprise architecture documentation
- Valuing customer, business, developer and operations (see DevOps) collaboration over contract negotiation
- Valuing good governance, transparency and honesty in progress, plans, costs and impediments over regular reporting
- Valuing responding to change over following a plan at all levels of the business
(Borrowed from the Holistic Software Manifesto)
Agility at scale is focussed simply on reducing unnecessary bureaucracy, reducing time to market and improving value.
So how do you achieve it?
The application of:
- Agile at scale practices
- soft practices
- technical practices
Of course each of those (and more!) is a complex can of worms in itself. A lot of these higher scale practices are only just emerging from high maturity complex (post-)agile organisations but in time more of those things will turn into links.
A good example of “Agile at Scale” in action is the Project Forum practice
As always this blog is a stream of consciousness so please, let me know your opinion?
The idea of a self-organising team is pretty established in agile software development, as a (post-)agile coach/mentor I sometimes find myself working with teams who are trying to self-organise. Here’s my approach for transitioning a team from being externally organised to becoming autonomous and self-organising.
The first thing to point out is that I believe that the result of a self-organising team practice is an organisation. I don’t know what the the organisation will be ahead of time, that’s what the team work out. What I do (and I think others should do) as an agile mentor/coach is:
- Bring to the team methods and techniques they can use to self-organise
- Information of self-organising and how to interact with other teams in an organisation
- Experience of different structures and behaviours
- Facilitation and support of the organisation
It’s not my job to organise a team, it’s my job to help them organise themselves.
Being an architect/designer at heart I take a simple approach to this kind of thing and think about it in terms of mission, structure and behaviour.
Team’s need to have a purpose. I believe in team’s seizing autonomy and declaring their purpose, even in a complex bureaucratic organisation. So long as senior management are willing to let people they employ to do a complex technical job actually do a complex technical job then there tends not to be a problem. Unfortunately it sometimes takes enlightened leaders to do this but I’ve seen more and more of them in both the private and public sector. A team needs to know it’s purpose and agree on it.
Structurally there are some nice ways of working out who should be in a team and what their role is. I recommend using competencies rather than roles. Competencies are a description of the skills required to deliver the mission of the team and come in levels ranging from the most basic to the most advanced form of e.g. Tester. Either grab some from a suitable process.
Get the team together and get them to score themselves on the range of competencies required. This achieves a couple of things:
- Builds mutual understanding between team members of where the various skills lie and what level they’re at leading to mutual respect and avoiding misunderstandings around what people are “meant” to do
- Establishes whether the team has enough coverage of the required skills for the job, without this knowledge how can a team commit to anything?
Team’s need to know how they’re going to work so they need to understand their functions. As a starter I’d suggest that software development teams have the following functions:
- Production (delivery of working product)
- Communication (both internally with customers and with external stakeholders)
- Decision making
- Way of Working
- Buy-in to the mission and direction
- High level scope
- Changes to agreed scope
A true agile team does all of these things as it includes all of the people and skills necessary (from devs, testers, users, customers, ops people etc.) to achieve them. The team is the vehicle for doing these things.
Each of these things is the tip of a fairly large iceberg. But the team need to work on understanding their Way of Working to do the production job. The team need to agree how they’ll communicate with each other and externally. Both of these are Ways of Working decisions. All function in a team comes down to decision making. Ways of working, communication techniques, scope management etc. need to be constantly refined which means you need constant decision making within a team.
Even buy-in to the team mission and direction (planning and/or technical) is a decision making process. It often happens implicitly with someone charismatic selling a vision and everyone kind-of nodding but that’s not the only way of doing it.
Different decision types deserve different methods
Just because a team is groovy and agile doesn’t mean that every decision should be fully democratic. For example when doing a customer demo to see if the product is on track it’s not normally open to a general vote on whether the product meets the customer’s needs or not. Usually the customer tells you and you have to put up with it.
– identify the different categories of decisions the team needs to make
– identify (using a variation of the Vroom-Jago model) the mechanisms the team will use to make each decision type
– facilitate the decison making process reminding the team of their chosen mechanisms
The output of a self-organising practice is an organisation. Since I believe in honesty and transparency I think that the best way to define and communicate this stuff is in a Team Charter. A Team Charter can simply be:
Some blurb about what the team does – Mission statement
Assertion of rights the team assumes – Autonomy
Who the team is
Who’s in the team, what’s their ability, how do they contribute to the required competencies – Structure
How the team works
A little bit about the practices the team uses. Such as using a continuous flow model, an iterative approach etc. Including the decisions the team makes and how they make them – Decision Making
If you’d like me to help your team self-organise get in touch. Do you have any other ideas on self-organising? Do you use a different approach?