Mike MacDonagh's Blog

Somewhere in the overlap between software development, process improvement and psychology

Tag Archives: business change

Holistic Software Development – the book

The HSD book is currently in it’s final editing phase. If you’d like to purchase the book then please bookmark this page and come back soon.

What’s inside?HSD Book

The book covers a detailed introduction to Holistic Software Development and it’s principles. Covering the H-Model and how it us used to map levels of decomposition to recomposition and integration the book then follows a structured path through HSD material.

We discuss Strategy and Holistic Portfolio Management to help guide organisations in adopting post-agile flexibility at the top level. We cover the HSD requirements model and how it supports strategic intent without falling into the Big Up Front Requirements trap.

A detailed chapter on architecture follows covering enterprise, solution and system architecture and its representation through over sketches and architectural mechanisms.

We cover Planning and how to balance the needs of business planning with agile just in time planning. We deconflict project management and agile ways of working showing how they can work together in a holistic organisaiton.

Releases and Integration is the heart of the H-Model and so is where the focus of HSD comes together. We discuss the dual nature of releases, and how they are best set up across teams of teams in both a technical configuration management context and a planning context.

We cover the HSD approach to Quality and Operations before covering a detailed view of People structures and organisational design. Based in psychology we look at motivation, teaming and large scale structuring covering the HSD Hybrid Dynamic Model in depth.

Finally we look at Governance and HSD variants for simple organisations and COTS before briefly covering Adoption practices.

 

We’re working through the last few edits of version 1.0 currently and hope to have the book published in 2016, you can now sign up to be notified when it’s published on the HSD site.

Holistic Software Development @AgileInABag London 12 June 2015

Agile In a Bag London 2015

I’ll be presenting with my friend and colleague Steve Handy about Holistic Software Development at the Agile in a Bag conference in London on the 12th June 2015.

If you can make it, do come along to this great 1 day conference to have some fun, share some thoughts and ideas and meet some interesting people 🙂 Here’s the blurb for our talk first thing (9am) in the morning:


 

These are not the practices you are looking for

Is your time to market quick enough for your business to respond?
Is your Programme and Project Management smoothly joined up with your agile teams?
Does everyone understand how their work contributes to business value?
Do you know what end-to-end scenarios will be delivered in the next programme release?

Come to this talk to hear how these common problems are easily solved, even in highly complex system-of-systems organizations by using the freely available “Holistic Software Development” (HSD) framework which aligns agile philosophy throughout the enterprise, at-scale engineering, architecture, strategy, portfolio and programme management. Using a novel but lightweight end-to-end scope level requirement type we iteratively drive the development of well understood Releases and customer feedback loops.

Combined with layered iterative or continuous integration and clearly understood progressive Definitions of Done we can simplify the complex resulting in a perfect join up between customers, requirements, planning, delivery teams and governance. HSD is the glue that plugs together your existing agile team practices (e.g. Scrum) with PPM practices (e.g. Prince2) and business strategy providing people focussed, lightweight guidance firmly based in evidence, experience and excellence.

And here’s the learning outcomes we’re aiming at:

  • Understanding how to de-conflict agile and enterprise portfolio and programme management
  • How to unify agile teams, governance, customers and executives in a simple model
  • The “H-Model” of software development
  • How to join up different schools of thought/processes into a cohesive way of working
  • How a strong approach to understanding business value enables effective software businesses

 

You may detect a slight Yoda-like grammar to the title… Come or come not, there is no spoiler.

How to write a good strategy

Strategy is a plan that provides overall direction for an organization and a framework for decision making.

Strategy is the instrument by which executive decision makers communicate and drive the future direction of their organization, without clear well-communicated strategy the organization becomes directionless. The purpose of strategy is to set the overall direction that an organization will take; it is the expression of executive intent and is effectively a contract between the directing layer of an organization and its sponsors (shareholders or elected representatives for public sector concerns). Strategy, specifically in the form of Strategic Goals, forms the very top of the requirements stack driving the portfolio of work and shaping enterprise architecture.

Recommendations for effective strategy

Articulate how Business Value is understood and measured – this helps everyone understand the business purpose and the primary measurement of success helping avoid implementation of intermediary measures that are mis-aligned to strategy

Minimize the number of organizational layers – this avoids unnecessary translation of information across layers and unnecessary middle-management resourcing. See information on Resourcing Levels and Workforce Shaping for more.

Make bold strategic changes – if change is required then sometimes drastic change is required rather than a series of small changes, it might be painful but it’s likely to be more cost-effective to make a bold change.

Push tactical decision making as low as possible in the organization – this empowers individuals to use their skills properly within the organization and prevents stragic decision makers from getting swamped by minor issues. Business Leaders must back up tactical decisions made by their staff for this to work.

Continuously make small tactical improvements – don’t wait for a large change in a year to improve something small that can be fixed today. Make the change and measure the improvement to validate the change and ensure systemic improvement.

Forget the past – Business situations involving large numbers of people are extremely complex and so similar situations from the past are not good indicators of present or future events. Correlation does not equal causation.

 

Tests for a good strategy

Overall
Does the strategy communicate a clear change?
Is it engaging, inspirational and compelling?
Is it clear and unambiguous?

Why?
Does it describe the rationale for change?
Is the Business Strategy achievable? or worth risk of failure?
It it relevant to the prevailing business environment?

When?
Do the business goals provide value to the sponsors?
Does it contain a roadmap/plan? Or can one be easily derived?
Does it contain short, medium and long-term goals?

What?
Are strategic goals identifiable, measurable, tangible business state-change milestones?
Do strategic goals follow a logical sequence to a desired end-state?

How?
Does it consider resourcing necessary to realise the goal?
Is it executable?

 

Extract from Holistic Software Development – Strategic Direction

The Hybrid Dynamic Model

The Hybrid Dynamic Model is a modern approach to structuring an organisation’s portfolio that allows for multiple ways of working to co-exist from innovation to utility work and for that work to change which processes, cultures and practices it uses over time.

An “Operating Model” describes how an organisation does its business in terms of both its structure and its behaviour – it is the business architecture. An “Operating Model” exists in every organisation, whether it is explicitly written down or simply a collection of processes and practices. The Operating Model includes how strategy flows through planning and delivery, how money is spent and controlled, how governance works, how people are organised, how quality is assured etc. If the Operating Model is inefficient or unable to react well to change, everything else suffers.

Many organisations evolve an Operating Model that fits their stable way of working, allowing for a focus on efficiency and improvement, driving out variation. This approach can work well in periods of stability but makes organisations brittle and unable to react well to change because the only way the organisation knows how to deal with things is to apply its tried and tested approaches. When the environment is less stable, or the variety of work types increases a Hyrbid Dynamic Model is more appropriate.

A useful mechanism for understanding different types of work, and the different approaches that may be applicable to delivering them within a portfolio is the “Commoditization Scale”. The Commoditization Scale describes a range of types of work from innovation (undefined, rare, speculative and high risk) to more commodity/utility (well defined, ubiquitous, and less differentiated with lower risk).

These different types of work require different ways of working, contractual models, technologies, supply chains, cultures, processes etc. More well understood work is more predictable and manageable by its nature, whereas the opposite is true for less well-understood work.

Commoditization scale Commoditization scale inspired by Simon Wardley – CSC Leading Edge Forum

As organisations begin to understand their work in terms of a hybrid model they can then look to actively apply a commoditization pressure where possible moving work across this scale. Because commodity solutions (often bought in) are typically cheaper than in-house development, maintenance and hosting commoditizing work earns a “Commoditization Dividend” which can then be used to reinvest into the Innovation parts of a business.

As mentioned in Build or Buy we generally recommend against building utility or commodity components, systems or product (unless disrupting a service industry) – these kind of products are best bought or provided by open source alternatives (COTS). However in terms of inventing/innovating and specialist products different processes, working practices, teams and cultures may be necessary. Innovation work is typically riskier and speculative than more mainstream specialist products and so will be less tolerant of up front planning. “Failing fast” is particularly important in innovative/inventive work and this type of work is typically very unpredictable. As a result building significant structure (in terms of project, programme and Solution Architecture) around innovation work is unlikely to be productive – this work may also be unconstrained by Enterprise Architecture (and indeed “standard” ways of working).

Unless explicitly necessary we recommend that innovative work is treated as self-organized Product Delivery teams that are directly part of the Portfolio, with no programme structure. This (Small Picture) model is preferred for small, simple projects or organizations generally. Small organizations can often outperform large monolithic organizations due to their lack of process, management and architectural inertia – they are free to innovate. Since the relentless commoditization of technology forces all organizations to innovate to survive we strongly recommend freeing people to innovate in whatever manner they need.

There’s nothing worse for innovation than an ideas process

Similarly, and perhaps counter intuitively, there is also less structure required in the Utility and Commodity areas of the commoditisation scale (because the products built are often well-understood, predictable, related to system integration and frequently COTS). This type of very predictable work often benefits from workflow management techniques such as Lean, Kanban or service management processes.processes and commoditisation

Products that are more Specialist may need a “bigger engineering” approach depending on their scale or risk factors. Many large organizations will adopt more formal processes in this area as they seek to find economies of scale, especially around reuse. However over time reusable platforms and architectures can become a hindrance rather than a help. Technically skilled people will notice this point and are encouraged to Bubble Up their concerns. Organizations with significant effort in more than one of these areas of the scale above tend to gravitate towards a hybrid model with a structure embedded in the Specialist, Commodity or Utility part of the business. Applying the same structures, architectures, processes and working practices across all of these work types can cause problems so a hybrid model is likely to be required for large/complex organizations.

We recommend using a hybrid, dynamic model for organising a large/complex portfolio. Large, complex organisations have significant effort and coverage across the commoditisation scale. Implementation of a hybrid model caters for this variance using unstructured and structured models where necessary. Structure is introduced only by exception when explicitly needed to orchestrate effort across divergent teams and portfolio(s) not as the default approach. Importantly as pieces of work become more understood, or product sets evolve they will likely begin to move on the commoditisation scale and so may need to dynamically change their ways of working.

The Hybrid Dynamic Model allows for significantly different ways of working, within the cohesive governance of the H-Model of Holistic Software Development. Using the hybrid dynamic model organisations can have multiple ways if working, cultures and practices co-existing peacefully within a single enterprise portfolio. Using different ways of working, practices and techniques across the portfolio helps bring together the otherwise extremely different extremes of innovation and research work with predictable project delivery. Other business models, such as Incubation, can also be used to help bridge the gaps between areas such as innovation and mainstream engineering practices.

Hybrid Dynamic Model

Many organisations evolve a Portfolio → Programme → Project → Team model which is then applied to all work in the portfolio due to its past success at delivering large predictable engineering projects. However, this model is wasteful when complexity is low and can actually inhibit communication and feedback cycles in more speculative work.

Programme and project structures cause layers of separation, which result in emergent transactional behaviour, effort duplication and confusion over roles and responsibilities. As a result programmes should only be used when there is a genuine requirement for co-ordinating multiple projects that must be integrated for a cohesive product family or business capability. In other cases programme structures should be condensed – removing programmes that are simply funding lines; Portfolio Management can and should manage funding lines and stand-alone projects can simply be part of the portfolio.

Reduction in structural complexity will reduce costs, increase flexibility and adaptability while simultaneously improving an organisations ability to react in a rapidly changing environment. Additionally, simplifying structure will reduce the sheer numbers of people involved in project management and definition work, removing unnecessary barriers between delivery teams and their customers.

This kind of hybrid structure is similar to using the Small Picture of HSD for the Commodity Portfolio, the Big Picture for the Specialist Portfolio and the Small Picture again for the Research and Innovation portfolio – naturally sharing the same Strategy and Governance layer.

COTS implementation may be a signficant piece of work by itself. The COTS Implementation article describes a variant of the Big Picture for use with COTS.

A man’s perspective on feminism in the technology industry

I normally avoid posting on controversial things but this is a topic that shouldn’t even be controversial.

The world is full of a marvellous variety of people and they come in all shapes and sizes, with a staggering range of ways of thinking. People are different due to things like gender, race, ability, sexual orientation, cognitive difference and all sorts of other things that affect identity. Importantly people are never just one of these things, real people exist at and between all of the intersections.

In the software industry we often rally behind people based statements such as “Individuals and interactions over processes and tools” and yet discrimination and unfairness are rife in the technology industry with “brogrammer culture”, sexist pay  situations and the infamous glass ceiling. And yet we talk about being open and inclusive in how we do software in teams? And wonder why there aren’t more women taking technology courses at universities?

Feminism is a collection of movements and ideologies that share a common goal: to define, establish, and achieve equal political, economic, cultural, personal, and social rights for women. This includes seeking to establish equal opportunities for women in education and employment.

Women aren’t less than men. Men aren’t better than women. Instead people are just people, and each individual isn’t just a woman or man, they will be different in many other ways, expressing their identity, individualism and indeed diversity in each of them. There is value in diversity, and as people that can understand recursion we should be able to grasp that concept. The more ways we have of looking at a problem the better our solutions will be.

I believe that everyone should be treated fairly, that equal opportunities should exist for individuals, that if we need to differentiate between people (e.g. in terms of who to hire) that we should do it on merit, on skill, attitude and behaviours not on what makes a person different. I guess that makes me a feminist.

I’m a feminist because I believe in equality. Gender is one axis of difference that is used to discriminate against people and women are generally treated less fairly in the tech industry than men and so that’s why I’m a feminist not an “equallist”. I realise that this is an unpopular stance to take amongst some groups, especially those who are uncomfortable with a man using the word “feminist” but I think those people should take a good long hard look a themselves. If you’re not a feminist, you should probably go and explain why to your mother rather than argue with me about it.

Men, and women, and in-betweens, need to stand up for equality, need to challenge unfairness when it raises it’s ugly guise. The amount of times I’ve seen sexist treatment, casual racist and homophobic language in the tech and gaming industry is shocking and it needs to change. Otherwise we can hardly call ourselves civilised. Of course there are always weird situations where it’s not clear whether something is discriminatory or friendly banter – exploring those situations in an open and honest way, discovering if behaviour, language or tone has crossed a line helps us all understand each other better. Conflict is best avoided through clarity of understanding between people. Ultimately though the meaning of any communication is that which is received, regardless of the intentions. If you accidentally offend someone it’s still you who are offensive, not the victim who needs to grow thicker skin.

As men, we are morally bankrupt if we leave standing up for equality to those who are marginalised and discriminated against. Equality requires us to collectively act.

 

What is HSD? #hsd

HSD stands for Holistic Software Development which is a process framework that extends the principles of agile philosophy throughout the entire software development enterprise. Similar to “agile at scale” ideas HSD goes a few steps further integrating software development concerns from business strategy to continuous development, structural organisation to governance on the H-Model of software development.

H-Model

HSD provides the “big picture” of software development providing a glue that combines other techniques and practices. It doesn’t replace what you currently do (e.g. Scrum, XP, Lean, Prince2, whatever) instead it provides a simple way to join them all up together, without introducing any new TLAs (Three letter acronyms)!

Big picture of agile at scale

Holistic Software Development (HSD) – Big Picture

HSD is free to use, free to access and free to implement!

#Devops – A sticking plaster for a bigger problem

Why not StratDev, BusOps, etc.?

Holistic Communication

The term “DevOps” refers to tight integration between the Software Development and Operations parts of an organization. As I mentioned in my article on Conway’s Law the reinforcement of there being two separate parts “Dev” and “Ops” causes a separation in the way we think about these problems and then how we interact within an organisation. If the industry truly believed in DevOps then “dev” and “ops” would no longer exist. I realise that some people intend “DevOps” to be a verb, but it sounds like a noun.

Of course the idea behind of DevOps, of bringing together stakeholders involved in creating, hosting, running and maintaining is a great idea – focussing on these things can make it easier to deploy products and get user feedback. However we might just end up building the wrong products quickly. We must make sure that the things we build are aligned to strategy and deliver business value. To do that we need a holistic approach, that involves all stakeholders (not just the dev and ops folks).

Operations stakeholders should be involved in the Portfolio Build/Selection practice to ensure that the impact of new product development or maintenance decisions on current Operational positions are understood. Significant changes to current Operational positions are raised as Portfolio Requests so they can be properly costed and scoped.

Work that is approached holistically must by definition include addressing the needs of all stakeholders from any part of the business. If a divide exists between parts of the business then the dysfunctional relationship needs addressing rather than process and tooling being introduced. Processes and Tools are helpful to gain agreement on a common approach and automate ways of working but not to solve dysfunctional relationships.

HSE promotes regular builds and releases in either iterative/agile or continuous flow models Operations stakeholders must be involved early and often in software development product delivery as a product is likely to be delivered frequently, or even continuously, and so will need transitioning activities performed repeatedly.

So, I like the idea of joining up dev and ops stakeholders and work. But I only like it in the context of also joining up business strategy, requirements, architecture and quality with dev and ops – all flowing business value to customers. On it’s own, DevOps isn’t enough and worse could just be a management fad.

Launch: Holistic Software Engineering

How do we join up business strategy to agile development? Is program management relevant? Where do project managers fit in? What about architecture?

Holistic Software Engineering (HSE) answers all of these questions – for free.

Agile and continuous flow are great for small teams or a small number of inter-related small teams working on exploratory or maintenance work. But what if we’re spending 100s of millions on an IT strategy of inter-related products that need to work together to deliver business value. What is business value anyway?

H-Model To answer these questions (and more) my friend Steve Handy and I have distilled our collective 30+ years of software experience in a single, cohesive model of software development. We’ve developed the H model that moves on from the v-model and it’s siblings by combining:

…all elegantly combined and de-conflicted by release planning.

We’ve not invented very much, we’ve simply put a lot of good ideas from others together into a cohesive framework. We’ve drawn it all as a big picture and developed a website that shows how to get value from all of this stuff. Everything is clickable, everything has content.

The best bit: it’s free! There’s no paywall, there’s no private “full” version, you can just use it or not as you like.

We don’t believe in process zealotry, or putting academic concerns above clarity and usefulness. HSE is indicative, not prescriptive. You’re already doing it and if you use the big picture to draw red blobs on the bits that aren’t working well, or missing, in your organisation then you can use the model to make tangible improvements – immediately.

Using HSE doesn’t replace any of your existing processes, it provides the glue that joins them all up together in one simple, elegant and cohesive model.

Holistic Software Engineering
is for large/complex software organizations
who need to understand modern software engineering in a business context
our solution is to present a big picture of software business backed up with practical detail avoiding academic or heavyweight process documentation
that covers people issues, business strategy, portfolio, programme and project management as well as architecture, technical delivery and integration
unlike simple small team based processes such as RUP, Scrum etc.
The big picture of software engineering

Holistic Software Engineering

And if it’s too big, we’ve got a small picture, which is essentially “normal” agile development.

Please share HSE with your friends and colleagues.

No more Project Managers, bring in the Movie Producers

I was reading some course material recently that was trying to teach people something to do with software development and it was using the same old tired “ATM machine” example. I’ve worked with hundreds of projects, many in the finance sector and none of them are anything like an ATM machine.  One of the reasons that example is soooo tired is that it’s describing commoditised software development, it’s something I’d expect to buy off a shelf or go to a specialised vendor for. It’s not something I’d put my team of uber |33t  haxorz on.

Developing software products is a balance between a creative and scientific pursuit, so it’s a little hard to describe, especially when that software can range from a tiny smartphone app, to a website, to router firmware, to an enterprise hr system (urgh!), to a big data processing system of systems etc. You get the gist.

The things these types of system have in common with each other is how different they are. And yet the traditional approach to managing these diverse kinds of work has been classical project management with a one size fits all process (I’m looking at Prince2, RUP, Iterative, Agile etc.) or worse hiring a large company full of body-shop project managers. For me this is one of the root causes behind large scale IT disasters.

I once had a discussion with the leader of a PMO in a large organisation of thousands of people about the nature of Project Management for software and he assured me that “someone who knows how to manage a bit of work can manage software, it’s just a different type of work” and indeed I should “stop thinking of software as special”. I’ve seen this attitude resonate in many organisations and is to me best summed up by the same Head of PMOs statement “from a project management point of view building software is the same as building a bridge”.

Now then. I know a little about software development, but not that much about bridge building (if you exclude my illustrious Lego engineer career when I was 7). If I managed the building of a bridge next door to one build by someone with a track record of bridge building who’s bridge would you drive your family on? Not mine if you’ve got any sense.

There are so many problems in the software development industry caused by people not understanding it and applying bad metaphors. When people who do understand it start to get the point across (e.g. the early agile movement) they often get misunderstood by the old guard who dress up their normal ways of working in new language.

Many of our “best” metaphors come from manufacturing lines where the same thing is made over and over again. That’s nothing like software.

To me a software project is more similar to making a movie than anything else:

  • It’s a unique one off product (or a remake of a previous version) with new parts working together in new ways
  • Once made we’ll often duplicate and digitally distribute
  • It can cost a lot of money because it can be very complex and needs lots of specialist skills
  • You need a high level plan for how you’re going to make it
  • There’s no guarantee on return
  • What’s written down originally bares little resemblance to the finished product
  • We know when it’s good or bad but it’s hard to objectively quantify
  • There’s lots of different types that need different collections of specialist skills
  • Both involve people and time and so are subject to change, they have to be adaptive
  • You can tell when the people making it had fun
  • It’s not feasible that any team member can fit in any role
  • There’s almost always going to be some quality problems
  • You wouldn’t get a movie maker to build your bridge
  • You wouldn’t get a bridge builder to make your movie
  • You don’t make a movie faster by telling the actors to act faster

So, I don’t want a Project Managers that know how to work a gannt chart. I want movie producers that know how to work with a team holistically to get the best out of them both technically and artistically.

 

Reduce waste: Visualise the value stream

A big thing in software process improvement (urgh) is reducing waste.  A great idea, but how do you identify waste?

One powerful method that I use is value stream visualisation. There’s a number of ways of getting information about the value stream ranging from physically walking the chain to simply asking people.

Sounds simple doesn’t it? But one of the best sources of information about wasteful processes and ways of working in a team is simply asking the team what they feel they shouldn’t be spending time on. By freeing up people to actually do their job they can reduce waste and improve productivity. One method I use is to ask each team member to identify a list of n items that they have to spend time on that they think are either entirely or partially wasteful. Basically a list of their impediments. I then anonymize the list look for commonality and then the top few are a reasonable target for improvement.

Alternatively actually walking the path of a piece of work through an organisation can really open the eyes to the sheer number of people and unnecessary waiting involved in some wasteful processes…

Anyway, having got some data I find it useful to visualise it in terms of waiting time and doing time. In Lean-speak cycle time is the time it takes to actually do something, lead time is the time it takes from the request being made to it finally being delivered. In many tasks the lead time far outstrips the cycle time, visualising it can make that very clear.

Low Lead time For example, a task with has a lead time of 5 days might actually be made up of 2 days waiting time before it actually gets picked up and the cycle time of 3 days starts.

Large lead timeIf you look at all of the tasks in your value chain you might find that many of them look more like this, with very large wasteful waiting times. There are many reasons why there might be a large waiting time, typically the team doing the tasks is over-subscribed, or a team might be protecting it’s SLAs, KPIs or other TLAs. These problems can have significant effects when looking at the system as a whole.

Even more interesting is looking at how these tasks might fit together in a value stream. If we imagine a business function that involves a request moving through a bunch of tasks across a number of teams that have different waiting and doing times a simple sequence of tasks, with a little buffering between them from the Project Manager to allow for schedule variance naturally, might end up looking like this:

 sequential_value_chain_arrows

Here we have 26 days of actual work spread over almost 60 days (12 weeks) of elapsed time = 40% waste. Even if the person planning the work is willing to accept some risk and try to optimise their workflow a bit without improving the waste inherent in the tasks involved there’s still a lot of waste in the system.

parallel_value_chain_arrows

By visualising the value stream in this way we can see straight away (apologies to red/green colour blind folks) that there’s a lot of red, a lot of waste. In many cases planners aren’t willing to accept the risks inherent in overlapping activities as shown here, or aren’t even aware that they can leading to the more sequential path shown above. The result is a minimum time that it takes a request to get done, based on the impedence of the current value chain of, in this case, 38 days before we even start thinking about actually doing any work.

Surely that’s not right.

%d bloggers like this: