Mike MacDonagh's Blog

Somewhere in the overlap between software development, process improvement and psychology

Tag Archives: lean

Holistic Software Development @AgileInABag London 12 June 2015

Agile In a Bag London 2015

I’ll be presenting with my friend and colleague Steve Handy about Holistic Software Development at the Agile in a Bag conference in London on the 12th June 2015.

If you can make it, do come along to this great 1 day conference to have some fun, share some thoughts and ideas and meet some interesting people 🙂 Here’s the blurb for our talk first thing (9am) in the morning:


 

These are not the practices you are looking for

Is your time to market quick enough for your business to respond?
Is your Programme and Project Management smoothly joined up with your agile teams?
Does everyone understand how their work contributes to business value?
Do you know what end-to-end scenarios will be delivered in the next programme release?

Come to this talk to hear how these common problems are easily solved, even in highly complex system-of-systems organizations by using the freely available “Holistic Software Development” (HSD) framework which aligns agile philosophy throughout the enterprise, at-scale engineering, architecture, strategy, portfolio and programme management. Using a novel but lightweight end-to-end scope level requirement type we iteratively drive the development of well understood Releases and customer feedback loops.

Combined with layered iterative or continuous integration and clearly understood progressive Definitions of Done we can simplify the complex resulting in a perfect join up between customers, requirements, planning, delivery teams and governance. HSD is the glue that plugs together your existing agile team practices (e.g. Scrum) with PPM practices (e.g. Prince2) and business strategy providing people focussed, lightweight guidance firmly based in evidence, experience and excellence.

And here’s the learning outcomes we’re aiming at:

  • Understanding how to de-conflict agile and enterprise portfolio and programme management
  • How to unify agile teams, governance, customers and executives in a simple model
  • The “H-Model” of software development
  • How to join up different schools of thought/processes into a cohesive way of working
  • How a strong approach to understanding business value enables effective software businesses

 

You may detect a slight Yoda-like grammar to the title… Come or come not, there is no spoiler.

Advertisements

What is HSD? #hsd

HSD stands for Holistic Software Development which is a process framework that extends the principles of agile philosophy throughout the entire software development enterprise. Similar to “agile at scale” ideas HSD goes a few steps further integrating software development concerns from business strategy to continuous development, structural organisation to governance on the H-Model of software development.

H-Model

HSD provides the “big picture” of software development providing a glue that combines other techniques and practices. It doesn’t replace what you currently do (e.g. Scrum, XP, Lean, Prince2, whatever) instead it provides a simple way to join them all up together, without introducing any new TLAs (Three letter acronyms)!

Big picture of agile at scale

Holistic Software Development (HSD) – Big Picture

HSD is free to use, free to access and free to implement!

Video: holistic approach to scaling agile in 46 seconds

What is agile architecture?

Agile ArchitectureArchitecture is a high level view of a system in the context of its environment, dependencies, technology, structure and behavior.

Architecture must be solid, useful and beautiful.

Software architecture is typically hard to define as the term software architecture is used to describe many facets of software structure, behaviour, design, activity and documentation. The concept of software architecture is also relevant at different levels of a system as the components of one system become a reusable platform for the next system. Ultimately the emergent behavior caused by layers of architectural dependencies and reuse can be difficult to predict.

Architecture is a simplification of the system, used to communicate the nature of that system and guide its development. However, architecture is more than just a diagram, it is grounded in technical context describing technology stacks, standards adopted, common ways of doing things (mechanisms), how non-functional requirements are met and the elements that will meet functional requirements. Architecture provides a shape and look and feel to the internals of a system that are the foundation for the ultimate external behavior.Da Vinci's Virtuvian Man

The idea of an “Architect” role with such a broad focus is analogous to that of a structural architect concerned with buildings. Indeed in 25 BCE Vitruvius (a well known Roman writer and architect who inspired Leonardo Da Vinci, hence the Vitruvian Man ) described an architect as:

The ideal architect should be a man of letters, a mathematician, familiar with historical studies, a diligent of philosophy, acquainted with music, not ignorant of medicine, learned in the responses of jurisconsults, familiar with astronomy and astronomical calculations.

Virtuvius is famous for stating that architecture must be solid, useful, beautiful (De architectura). The same three qualities relate to software architecture. Despite architecture being a fine balance between a subtle science and an exact art we must realise it is only useful if it is aligned to requirements and becomes executable.

Architecture, not documentation

In line with one of the founding Principles of Holistic Software Architecture I:

Value working software in the form of quality releases from short development cycles over comprehensive documentation, business analysis and enterprise architecture documentation.

Choosing how much architectural work is necessary up-front and especially how much architectural documentation is necessary is difficult.

Traditional document focused development methods promoted large up-front effort to detail, in document or model form, the architecture and system design prior to development. As well as the inherent risk extension issues in waterfall processes this could often cause “analysis paralysis” where architecture work was seen as an endless diagram drawing exercise.

Agile methods focus on working software over documents and designs, however this doesn’t mean that no documents or designs should be produced. It is almost always useful to have some description of an architecture before development even if it’s just a statement that the team’s usual architecture will be used as otherwise team members won’t even know which language to start writing in.

Good architecture addresses how we’ll resolve the major technical risks, communicates between the team the overall structure of the solution and works out how our solution will meet the functional and non-functional requirements. Knowing when we’ve worked out enough architecture so that we’ve reduced the complexity of the problem into manageable sizes for the team is a difficult challenge.

Levels of architectural formality

Doing too much architectural analysis or elaboration, either through abstract design and modelling or practical spiking (writing small amounts of throwaway code to demonstrate the feasibility of a technical approach or architectural idea) will slow down a development project and increase the risk of wasted work. Value is only achieved once working systems are in the hands of the customers/users.

Alternatively not doing enough architectural analysis can lead to significant architectural refactoring during a projects lifetime, as key requirements can’t be met based on the current architecture, again leading to wasted work and slowing down the project.

When doing up front architecture, especially in the form of documents and models, we need to be careful to consider architectural work in the context of the team’s definition of done. Typical levels of done don’t normally include “analysed”, “architected” or “designed”. Although these terms might be meaningful to describe internal team development states they do not constitute working software and are only a step on the way to creating value.

Guidelines for good architecture and design

Having a common understood architecture, regardless of the format of that understanding (documents, models, sketches, whiteboards, implicit team knowledge), is described as “Intentional Architecture”.

Holistic Software Engineering described “good” architecture and design as having the following characteristics:

  • Intentional structure and behaviour
  • Highly modular: consisting of separate services, components, classes, objects or modules
    • Elements are highly cohesive
    • Elements are loosely coupled
    • Elements have low algorithmic complexity
  • Avoids duplication
  • Well described elements: modular elements have simple expressive meaningful name enabling readers to quickly understand their purpose, similar to code cleanliness we should strive for design cleanliness
  • Runs and passes all defined tests or acceptance criteria
  • Lightweight documentation

#Devops – A sticking plaster for a bigger problem

Why not StratDev, BusOps, etc.?

Holistic Communication

The term “DevOps” refers to tight integration between the Software Development and Operations parts of an organization. As I mentioned in my article on Conway’s Law the reinforcement of there being two separate parts “Dev” and “Ops” causes a separation in the way we think about these problems and then how we interact within an organisation. If the industry truly believed in DevOps then “dev” and “ops” would no longer exist. I realise that some people intend “DevOps” to be a verb, but it sounds like a noun.

Of course the idea behind of DevOps, of bringing together stakeholders involved in creating, hosting, running and maintaining is a great idea – focussing on these things can make it easier to deploy products and get user feedback. However we might just end up building the wrong products quickly. We must make sure that the things we build are aligned to strategy and deliver business value. To do that we need a holistic approach, that involves all stakeholders (not just the dev and ops folks).

Operations stakeholders should be involved in the Portfolio Build/Selection practice to ensure that the impact of new product development or maintenance decisions on current Operational positions are understood. Significant changes to current Operational positions are raised as Portfolio Requests so they can be properly costed and scoped.

Work that is approached holistically must by definition include addressing the needs of all stakeholders from any part of the business. If a divide exists between parts of the business then the dysfunctional relationship needs addressing rather than process and tooling being introduced. Processes and Tools are helpful to gain agreement on a common approach and automate ways of working but not to solve dysfunctional relationships.

HSE promotes regular builds and releases in either iterative/agile or continuous flow models Operations stakeholders must be involved early and often in software development product delivery as a product is likely to be delivered frequently, or even continuously, and so will need transitioning activities performed repeatedly.

So, I like the idea of joining up dev and ops stakeholders and work. But I only like it in the context of also joining up business strategy, requirements, architecture and quality with dev and ops – all flowing business value to customers. On it’s own, DevOps isn’t enough and worse could just be a management fad.

Launch: Holistic Software Engineering

How do we join up business strategy to agile development? Is program management relevant? Where do project managers fit in? What about architecture?

Holistic Software Engineering (HSE) answers all of these questions – for free.

Agile and continuous flow are great for small teams or a small number of inter-related small teams working on exploratory or maintenance work. But what if we’re spending 100s of millions on an IT strategy of inter-related products that need to work together to deliver business value. What is business value anyway?

H-Model To answer these questions (and more) my friend Steve Handy and I have distilled our collective 30+ years of software experience in a single, cohesive model of software development. We’ve developed the H model that moves on from the v-model and it’s siblings by combining:

…all elegantly combined and de-conflicted by release planning.

We’ve not invented very much, we’ve simply put a lot of good ideas from others together into a cohesive framework. We’ve drawn it all as a big picture and developed a website that shows how to get value from all of this stuff. Everything is clickable, everything has content.

The best bit: it’s free! There’s no paywall, there’s no private “full” version, you can just use it or not as you like.

We don’t believe in process zealotry, or putting academic concerns above clarity and usefulness. HSE is indicative, not prescriptive. You’re already doing it and if you use the big picture to draw red blobs on the bits that aren’t working well, or missing, in your organisation then you can use the model to make tangible improvements – immediately.

Using HSE doesn’t replace any of your existing processes, it provides the glue that joins them all up together in one simple, elegant and cohesive model.

Holistic Software Engineering
is for large/complex software organizations
who need to understand modern software engineering in a business context
our solution is to present a big picture of software business backed up with practical detail avoiding academic or heavyweight process documentation
that covers people issues, business strategy, portfolio, programme and project management as well as architecture, technical delivery and integration
unlike simple small team based processes such as RUP, Scrum etc.
The big picture of software engineering

Holistic Software Engineering

And if it’s too big, we’ve got a small picture, which is essentially “normal” agile development.

Please share HSE with your friends and colleagues.

How to make real progress in a software project

I’ve not been blogging much this year because I’ve been crazy busy writing software (more on that soon I hope). As you might expect I’ve changed my ways of working more often than process improvement consultants change the name of the process they’re selling but one key thing always seems to drop out whether I’m working on my own, with a team, a team of teams or a globally distributed team.

Do the top thing on the list, whatever it is.

Don’t do anything else until it’s done.

Or in process speak: “Reduce the WIP and clarify the Definition of Done“.

Too often I fall into the trap of doing the most interesting thing from the list, which normally means the most complex. And far too often I get carried away with the next interesting thing before finishing something. Both of these lead me to have far too many things in various states of progress all interfering with each other and me with no idea of how well I’m doing. And that’s assuming I actually know what “finished” means.

Software is complex enough without us making it harder with processes, remember the KISS principle?

Reduce waste: Visualise the value stream

A big thing in software process improvement (urgh) is reducing waste.  A great idea, but how do you identify waste?

One powerful method that I use is value stream visualisation. There’s a number of ways of getting information about the value stream ranging from physically walking the chain to simply asking people.

Sounds simple doesn’t it? But one of the best sources of information about wasteful processes and ways of working in a team is simply asking the team what they feel they shouldn’t be spending time on. By freeing up people to actually do their job they can reduce waste and improve productivity. One method I use is to ask each team member to identify a list of n items that they have to spend time on that they think are either entirely or partially wasteful. Basically a list of their impediments. I then anonymize the list look for commonality and then the top few are a reasonable target for improvement.

Alternatively actually walking the path of a piece of work through an organisation can really open the eyes to the sheer number of people and unnecessary waiting involved in some wasteful processes…

Anyway, having got some data I find it useful to visualise it in terms of waiting time and doing time. In Lean-speak cycle time is the time it takes to actually do something, lead time is the time it takes from the request being made to it finally being delivered. In many tasks the lead time far outstrips the cycle time, visualising it can make that very clear.

Low Lead time For example, a task with has a lead time of 5 days might actually be made up of 2 days waiting time before it actually gets picked up and the cycle time of 3 days starts.

Large lead timeIf you look at all of the tasks in your value chain you might find that many of them look more like this, with very large wasteful waiting times. There are many reasons why there might be a large waiting time, typically the team doing the tasks is over-subscribed, or a team might be protecting it’s SLAs, KPIs or other TLAs. These problems can have significant effects when looking at the system as a whole.

Even more interesting is looking at how these tasks might fit together in a value stream. If we imagine a business function that involves a request moving through a bunch of tasks across a number of teams that have different waiting and doing times a simple sequence of tasks, with a little buffering between them from the Project Manager to allow for schedule variance naturally, might end up looking like this:

 sequential_value_chain_arrows

Here we have 26 days of actual work spread over almost 60 days (12 weeks) of elapsed time = 40% waste. Even if the person planning the work is willing to accept some risk and try to optimise their workflow a bit without improving the waste inherent in the tasks involved there’s still a lot of waste in the system.

parallel_value_chain_arrows

By visualising the value stream in this way we can see straight away (apologies to red/green colour blind folks) that there’s a lot of red, a lot of waste. In many cases planners aren’t willing to accept the risks inherent in overlapping activities as shown here, or aren’t even aware that they can leading to the more sequential path shown above. The result is a minimum time that it takes a request to get done, based on the impedence of the current value chain of, in this case, 38 days before we even start thinking about actually doing any work.

Surely that’s not right.

Understand the software value chain by walking it

There’s no better way to understand something than by actually giving it a go. If you want to know what writing a particular programming language is like you need to try it. If you want to know what kung fu is like watching a video won’t help, you need to try it.

If you want to really understand the software value chain in your organisation drawing diagrams in an ivory tower isn’t going to help, you need to get out there and “Go and see”.

In Steve Handy’s blog on Scaling Software Agility he talks about applying some of the thinking from the Lean school of thought to “go and see”, observing the value chain by actually physically following it.

  • Identify a new piece of work and follow it through the delivery chain
  • Attend every meeting, track the activities and note how long it’s static.
  • Don’t attempt to fix it during this period of observation, don’t criticise, just watch and learn.

I think there’s some real value here, there’s something quite simple and powerful about physically walking the walk of a piece of work through the software value chain. Doing so will make some problems, blockages and issues blindingly obvious – processes and organisational structures that seemed to make sense on someone’s bit of paper will just not make sense.

Experiencing the value chain in a very physical way promotes professional mindfulness and will clearly identify waste. I think this is a great way of putting common sense back in to software development.

I call this Go See, and it’s the most effective form of measurement for software development.

 

%d bloggers like this: