When I started doing software I was a simple developer interested in elegant code and shiny things. I worked in a small software house that taught me many bad practices in terms of configuration management, change control, estimation, management etc. It was an incredibly valuable experience for me in learning how not to do things and led me to strive for something better even though at the time I didn’t really know what it might be.
That led me through a path of process definition and documentation (RUP) and a rather limited following of an iterative lifecycle. I remember a conversation I once had with one of my project managers (when I was an architect) asking if we were going to do “Use Case Analysis” on a particular project or just skip straight into “Use Case Design”. The question puzzled me then because it felt like a trap and I didn’t really understand what I was being asked.
It puzzles me even more now as I know understand “analysis” to be a stop-and-think-for-a-moment activity and I’d always do a bit of analysis of my requirements, however these days it’s extremely unlikely that it’d be an analysis UML model, instead it might be a sketch, a conversation and a bit of thinking before another conversation.
So I moved from process prescription to process understanding, applying the spirit of doing things well rather than the letter of whatever current process law was in fashion. Following on from that I’m not really interested now in the details of what a book says someone’s role should be or how people should interact instead I think the real challenges in software development are social, not collections of technical practices (although there is value in evolving better practices and tooling).
A colleague of mine commented recently that 20 years ago when he was doing software it was possible to understand everything from the metal all the way to the blinking lights on his bit of hardware. Software development in just 20 years has progressed incredibly and it’s just not that simple any more. We can understand the basics all the way through the stack but not all of the details. There are so many bits interacting that it’s just too complex. As a result the problems, technology and team working are all abstracting away from the tangible mechanistic past.
Developers are on a path from technical skill to mastery, as they begin to understand the kung fu of software engineering they can apply experience and deep understanding to solving the technical problems, doing away with formal process and just using the practices that they intuitively need, happily breaking the “rules” to get the job done in an efficient high quality way. I’m not entirely sure where brogrammers are on this evolutionary path, I’ll leave that up to you.
The problem, if it even is a problem, is that each person in an organisation is somewhere different along this path and even if they’re at the same point in the same dimension they might not be aligned in their interests and motivations. This makes team working amongst inherently complex social creatures a tricky proposition. The sweet spot is a team of fully enlightened software kung fu masters but that’s a really hard target to meet for a number of reasons. Consider the flower of team working evolution weirdness, which area is your team in?
One reason that this sweet spot is difficult to hit is because in any organisation half of the developers are below average in technical and social skills. Sounds horrible but is obviously true and the larger the organisation the more likely it is to be the industry average, not the organisational average.
This means that the centre of skill gravity for any team is unlikely to be on the w00t side of the scale.
In most large organisations the teams are more likely to be flattened pancakes across this bell curve taking in a reasonable representation of the organisation as a whole (especially as the highest skilled are often distributed amongst an organisation to attempt to bring up other teams.
This isn’t necessarily a problem though as there is value in diversity and individuals will each have different ways of thinking about things that can bring value to teams and organisations.
I’m beginning to think that the purpose of classic software process, and classic software process improvement is to try and move people along this scale from basic developer to software ninja, helping them to gain mastery through experience and feeding in the experience of others. When a team moves along this scale they begin to not need their process mentors any more and will seize their autonomy.
Technology is increasingly commoditised by innovation, as is software development and software process. I used to spend a lot of time teaching people Object Orientation but these days everyone just seems to know it, it’s nothing special or new, it’s just what people do. Similarly people are increasingly aware of the value of iteration, limiting work in progress, continuous integration and delivery. They need less process and less instruction as this stuff is becoming business as usual at least in more mature organisations.
As these problems are being solved I think that bigger problems are coming to the fore as they’re less hidden by low level development issues. The questions I see a lot of clients wrestling with are things like:
- How can we foster the right kind of organisational culture?
- How do we deal with requirements, architecture, releases, resources etc. in a complex system of systems environment?
- How do we bring together multiple divergent interpretations of “agile”?
- How do we manage outsourcing contracts in high speed agile projects?
- How do we motivate and engage the business and technical communities?
- etc. etc.
For me doing these things in a social collaborative way, that values individuals and teams, is Enlightened Software Engineering and encompasses the buzzphrase of the moment: Agile at Scale.
Solving these problems is less about technical skills and technical process content and more about social skills, psychology and understanding. Both from a coaching and business perspective.
So what’s my point?
My point is that software process improvement needs to focus less on individuals and more on teams, and teams of teams. That we should avoid ideology and take the best bits of knowledge and experience wherever we find them, growing our teams and individuals.
Also that we should apply more psychology to software process and business change finding socially resonant patterns for how we do things. True mastery involves not worrying about “breaking the rules” and from the outside can easily be confused with ineptitude.