Mike MacDonagh's Blog

Somewhere in the overlap between software development, process improvement and psychology

Category Archives: software development, methods and tools

Holistic Software Development – the book

The HSD book is currently in it’s final editing phase. If you’d like to purchase the book then please bookmark this page and come back soon.

What’s inside?HSD Book

The book covers a detailed introduction to Holistic Software Development and it’s principles. Covering the H-Model and how it us used to map levels of decomposition to recomposition and integration the book then follows a structured path through HSD material.

We discuss Strategy and Holistic Portfolio Management to help guide organisations in adopting post-agile flexibility at the top level. We cover the HSD requirements model and how it supports strategic intent without falling into the Big Up Front Requirements trap.

A detailed chapter on architecture follows covering enterprise, solution and system architecture and its representation through over sketches and architectural mechanisms.

We cover Planning and how to balance the needs of business planning with agile just in time planning. We deconflict project management and agile ways of working showing how they can work together in a holistic organisaiton.

Releases and Integration is the heart of the H-Model and so is where the focus of HSD comes together. We discuss the dual nature of releases, and how they are best set up across teams of teams in both a technical configuration management context and a planning context.

We cover the HSD approach to Quality and Operations before covering a detailed view of People structures and organisational design. Based in psychology we look at motivation, teaming and large scale structuring covering the HSD Hybrid Dynamic Model in depth.

Finally we look at Governance and HSD variants for simple organisations and COTS before briefly covering Adoption practices.

 

We’re working through the last few edits of version 1.0 currently and hope to have the book published in 2016, you can now sign up to be notified when it’s published on the HSD site.

Advertisements

Holistic Software Development @AgileInABag London 12 June 2015

Agile In a Bag London 2015

I’ll be presenting with my friend and colleague Steve Handy about Holistic Software Development at the Agile in a Bag conference in London on the 12th June 2015.

If you can make it, do come along to this great 1 day conference to have some fun, share some thoughts and ideas and meet some interesting people 🙂 Here’s the blurb for our talk first thing (9am) in the morning:


 

These are not the practices you are looking for

Is your time to market quick enough for your business to respond?
Is your Programme and Project Management smoothly joined up with your agile teams?
Does everyone understand how their work contributes to business value?
Do you know what end-to-end scenarios will be delivered in the next programme release?

Come to this talk to hear how these common problems are easily solved, even in highly complex system-of-systems organizations by using the freely available “Holistic Software Development” (HSD) framework which aligns agile philosophy throughout the enterprise, at-scale engineering, architecture, strategy, portfolio and programme management. Using a novel but lightweight end-to-end scope level requirement type we iteratively drive the development of well understood Releases and customer feedback loops.

Combined with layered iterative or continuous integration and clearly understood progressive Definitions of Done we can simplify the complex resulting in a perfect join up between customers, requirements, planning, delivery teams and governance. HSD is the glue that plugs together your existing agile team practices (e.g. Scrum) with PPM practices (e.g. Prince2) and business strategy providing people focussed, lightweight guidance firmly based in evidence, experience and excellence.

And here’s the learning outcomes we’re aiming at:

  • Understanding how to de-conflict agile and enterprise portfolio and programme management
  • How to unify agile teams, governance, customers and executives in a simple model
  • The “H-Model” of software development
  • How to join up different schools of thought/processes into a cohesive way of working
  • How a strong approach to understanding business value enables effective software businesses

 

You may detect a slight Yoda-like grammar to the title… Come or come not, there is no spoiler.

The rise of the Chief Software Architect

ArchitectureSoftware is increasingly important to everyone, it’s everywhere. It’s in our phones, runs our cars, our cities, our healthcare, entertainment and utilities. There are few businesses without a software element and many that are critically software dependent. What these organisations are finally beginning to understand is that the business of doing software is difficult, in fact it’s complex. It needs C-suite level direction and coverage.

Being good at software isn’t a matter of hiring the good coders, buying the right workflow management tool, using the right Configuration Management system, hiring the right management consultants or using the correct technology. These things all have a part to play but they’re supporting parts to the most important factor: culture.

Software Development is a team activity and the team is bigger than we used to think it was (a few developers, testers, direct customers, managers, analysts etc.). A good set of development practices at team level isn’t enough to be effective because software development is a business enabling activity and so it touches all parts of an organisation (development teams, hr, procurement, security, business change, legal, financial, business services, operations, support etc.).

As organisations evolve to make the maximum use of technology they are making increasingly complex software products, often through diverse technology stacks, using a variety of in-, near-, and out- sourcing partners. Delivering systems of systems across teams of teams is not simple and so organisations are now looking to embrace the importance of software, technology and ways of working to their businesses at board level.

What is a Chief Software Architect?

Whether a “Chief of Software“, a “Chief Software Officer“, a “Chief Digital Officer” or “Chief Architect” the role tends to include:

  • Leading and encouraging a positive software development culture
    • Where delivery of business value is the primary measure of success and decisions are based on business value
    • Where failing fast is a cause for celebration not a career ending issue
    • Where software is designed for users/customers needs
    • Where continuous/iterative development integration and deployment are the norm
    • Where teams collaborate and co-create with each other and their customers to produce the best products
    • Where thinking all day to write a perfect algorithm is worth more than pushing out hundreds of lines of code quickly
    • Where professionals are trusted to do their jobs properly
    • Where organisational impediments to smooth software development are removed, from board level if necessary (due to Conway’s Law transformational change is often impossible without such top-level leadership)
  • Providing top level architectural direction and governance (sometimes called Enterprise Architecture)  that strikes the balance between directing technical decisions towards alignment but not imposing restrictive unchanging standards on development teams
    • Architecture is understood cohesively at Enterprise, Solution and System levels
    • Architecture exists in the form of directed (but community driven) standards, off-the-shelf mechanisms and principles
    • Architecture exists primarily as executable software although appropriate weight documentation, models, examples and usage guides describe it
    • Architecture provides a context for making technical decisions in normal development, makes technical decisions on commoditized platform/software/operations choices but does not constrain innovation and research (architecture has a different impact in different parts of a hybrid dynamic model)
    • Architecture enables control of whole-lifecycle costs by smoothing the flow of work from innovation and research into mainstream development and support
  • Ensuring that teams can use the tools they want and need to use, not those that had the best sales pitch to a non-technical board (hint: these are normally small, simple and/or open source, not those provided by large vendors) balanced by creating a cohesive community that isn’t unnecessarily fragmented by every team needing to set up different tools environments
    • Tool chains take the pain out of development, testing, build and deployment – automated wherever possible
    • Development environments are appropriate for actually developing, building, testing and deploying
  • Ensuring, and embodying, the principles of agility, devops, and agility-at-scale at the organisational level – promoting the evolution of working practices, operating models and ways of working while avoiding endless management fads
    • Building and championing the needs of the development community, attending to their needs
    • Ensuring that portfolio selection takes into account technical issues, shaping it in line with architecture and in turn using business priorities to shape architectural choices

Why “Architect”?

The Vitruvian Man - Leonardo Da Vinci

The Vitruvian Man – Leonardo Da Vinci

At a business level Architecture exists to guide organisations through the business, information, process and technology changes necessary to execute the organisation’s strategy. Because “architecture” has always been more than just the technology bit, it also focusses on the non-functional aspects necessary to make the technology work terming a “Chief Software Officer” an Architect is an organisation recognising that to be successful at software a holistic approach is required that combines business concerns with technical concerns.

The idea of an “Architect” role with such a broad focus is analogous to that of a structural architect concerned with buildings. Indeed in 25 BCE Vitruvius (a well known Roman writer and architect who inspired Leonardo Da Vinci, hence the Vitruvian Man ) described an architect as:

The ideal architect should be a man of letters, a mathematician, familiar with historical studies, a diligent of philosophy, acquainted with music, not ignorant of medicine, learned in the responses of jurisconsults, familiar with astronomy and astronomical calculations.

Virtuvius is famous for stating that architecture must be solid, useful, beautiful (De architectura). The same three qualities relate to software architecture. Architecture is a fine balance between a subtle science and an exact art – as such it is the magic sauce that turns a technology strategy into business benefits.

If your business is reliant on technology and/or software and you don’t have a Chief Architecture, you should promote/get one before you become technically irrelevant, encumbered by lumbering traditional delivery approaches based on manufacturing and negative development cultures.

It’s time for businesses to recognise the important of software, and architecture, before they’re outmanoeuvred by those who have.


 

For more on Architecture see:Architecutural assets

 

 

How to write a good strategy

Strategy is a plan that provides overall direction for an organization and a framework for decision making.

Strategy is the instrument by which executive decision makers communicate and drive the future direction of their organization, without clear well-communicated strategy the organization becomes directionless. The purpose of strategy is to set the overall direction that an organization will take; it is the expression of executive intent and is effectively a contract between the directing layer of an organization and its sponsors (shareholders or elected representatives for public sector concerns). Strategy, specifically in the form of Strategic Goals, forms the very top of the requirements stack driving the portfolio of work and shaping enterprise architecture.

Recommendations for effective strategy

Articulate how Business Value is understood and measured – this helps everyone understand the business purpose and the primary measurement of success helping avoid implementation of intermediary measures that are mis-aligned to strategy

Minimize the number of organizational layers – this avoids unnecessary translation of information across layers and unnecessary middle-management resourcing. See information on Resourcing Levels and Workforce Shaping for more.

Make bold strategic changes – if change is required then sometimes drastic change is required rather than a series of small changes, it might be painful but it’s likely to be more cost-effective to make a bold change.

Push tactical decision making as low as possible in the organization – this empowers individuals to use their skills properly within the organization and prevents stragic decision makers from getting swamped by minor issues. Business Leaders must back up tactical decisions made by their staff for this to work.

Continuously make small tactical improvements – don’t wait for a large change in a year to improve something small that can be fixed today. Make the change and measure the improvement to validate the change and ensure systemic improvement.

Forget the past – Business situations involving large numbers of people are extremely complex and so similar situations from the past are not good indicators of present or future events. Correlation does not equal causation.

 

Tests for a good strategy

Overall
Does the strategy communicate a clear change?
Is it engaging, inspirational and compelling?
Is it clear and unambiguous?

Why?
Does it describe the rationale for change?
Is the Business Strategy achievable? or worth risk of failure?
It it relevant to the prevailing business environment?

When?
Do the business goals provide value to the sponsors?
Does it contain a roadmap/plan? Or can one be easily derived?
Does it contain short, medium and long-term goals?

What?
Are strategic goals identifiable, measurable, tangible business state-change milestones?
Do strategic goals follow a logical sequence to a desired end-state?

How?
Does it consider resourcing necessary to realise the goal?
Is it executable?

 

Extract from Holistic Software Development – Strategic Direction

What is HSD? #hsd

HSD stands for Holistic Software Development which is a process framework that extends the principles of agile philosophy throughout the entire software development enterprise. Similar to “agile at scale” ideas HSD goes a few steps further integrating software development concerns from business strategy to continuous development, structural organisation to governance on the H-Model of software development.

H-Model

HSD provides the “big picture” of software development providing a glue that combines other techniques and practices. It doesn’t replace what you currently do (e.g. Scrum, XP, Lean, Prince2, whatever) instead it provides a simple way to join them all up together, without introducing any new TLAs (Three letter acronyms)!

Big picture of agile at scale

Holistic Software Development (HSD) – Big Picture

HSD is free to use, free to access and free to implement!

What’s new in HSE 1.3

Video: holistic approach to scaling agile in 46 seconds

What is agile architecture?

Agile ArchitectureArchitecture is a high level view of a system in the context of its environment, dependencies, technology, structure and behavior.

Architecture must be solid, useful and beautiful.

Software architecture is typically hard to define as the term software architecture is used to describe many facets of software structure, behaviour, design, activity and documentation. The concept of software architecture is also relevant at different levels of a system as the components of one system become a reusable platform for the next system. Ultimately the emergent behavior caused by layers of architectural dependencies and reuse can be difficult to predict.

Architecture is a simplification of the system, used to communicate the nature of that system and guide its development. However, architecture is more than just a diagram, it is grounded in technical context describing technology stacks, standards adopted, common ways of doing things (mechanisms), how non-functional requirements are met and the elements that will meet functional requirements. Architecture provides a shape and look and feel to the internals of a system that are the foundation for the ultimate external behavior.Da Vinci's Virtuvian Man

The idea of an “Architect” role with such a broad focus is analogous to that of a structural architect concerned with buildings. Indeed in 25 BCE Vitruvius (a well known Roman writer and architect who inspired Leonardo Da Vinci, hence the Vitruvian Man ) described an architect as:

The ideal architect should be a man of letters, a mathematician, familiar with historical studies, a diligent of philosophy, acquainted with music, not ignorant of medicine, learned in the responses of jurisconsults, familiar with astronomy and astronomical calculations.

Virtuvius is famous for stating that architecture must be solid, useful, beautiful (De architectura). The same three qualities relate to software architecture. Despite architecture being a fine balance between a subtle science and an exact art we must realise it is only useful if it is aligned to requirements and becomes executable.

Architecture, not documentation

In line with one of the founding Principles of Holistic Software Architecture I:

Value working software in the form of quality releases from short development cycles over comprehensive documentation, business analysis and enterprise architecture documentation.

Choosing how much architectural work is necessary up-front and especially how much architectural documentation is necessary is difficult.

Traditional document focused development methods promoted large up-front effort to detail, in document or model form, the architecture and system design prior to development. As well as the inherent risk extension issues in waterfall processes this could often cause “analysis paralysis” where architecture work was seen as an endless diagram drawing exercise.

Agile methods focus on working software over documents and designs, however this doesn’t mean that no documents or designs should be produced. It is almost always useful to have some description of an architecture before development even if it’s just a statement that the team’s usual architecture will be used as otherwise team members won’t even know which language to start writing in.

Good architecture addresses how we’ll resolve the major technical risks, communicates between the team the overall structure of the solution and works out how our solution will meet the functional and non-functional requirements. Knowing when we’ve worked out enough architecture so that we’ve reduced the complexity of the problem into manageable sizes for the team is a difficult challenge.

Levels of architectural formality

Doing too much architectural analysis or elaboration, either through abstract design and modelling or practical spiking (writing small amounts of throwaway code to demonstrate the feasibility of a technical approach or architectural idea) will slow down a development project and increase the risk of wasted work. Value is only achieved once working systems are in the hands of the customers/users.

Alternatively not doing enough architectural analysis can lead to significant architectural refactoring during a projects lifetime, as key requirements can’t be met based on the current architecture, again leading to wasted work and slowing down the project.

When doing up front architecture, especially in the form of documents and models, we need to be careful to consider architectural work in the context of the team’s definition of done. Typical levels of done don’t normally include “analysed”, “architected” or “designed”. Although these terms might be meaningful to describe internal team development states they do not constitute working software and are only a step on the way to creating value.

Guidelines for good architecture and design

Having a common understood architecture, regardless of the format of that understanding (documents, models, sketches, whiteboards, implicit team knowledge), is described as “Intentional Architecture”.

Holistic Software Engineering described “good” architecture and design as having the following characteristics:

  • Intentional structure and behaviour
  • Highly modular: consisting of separate services, components, classes, objects or modules
    • Elements are highly cohesive
    • Elements are loosely coupled
    • Elements have low algorithmic complexity
  • Avoids duplication
  • Well described elements: modular elements have simple expressive meaningful name enabling readers to quickly understand their purpose, similar to code cleanliness we should strive for design cleanliness
  • Runs and passes all defined tests or acceptance criteria
  • Lightweight documentation

Scaling agile is the wrong approach

The heart of agile software development is feedback loops. Doing a bit of software, looking at it, at how we did it and then improving things as we do another bit of software. The “things” that can be improved can be quality, scope, usability, performance etc. etc. Perhaps most importantly, relationships and ways of working can also be improved.

This rapid feedback cycle ensures that product development is aligned to customer needs, that progress against guestimates is understood (so that plans can be refined) and that teams which include customers find problems early, and so can fix them, as quickly as possible. It’s a quicker, better, cheaper and happier way of working.

What about that needs to scale? Empirical feedback is just fine at any scale. Across 100s of teams or teams-of-teams and entire organisations short term feedback cycles work perfectly well.

The problem – Portfolio Management?

The problem is that there’s other stuff that goes on too. Product Development in big or complex organisations is part of an investment portfolio and costs a lot of money. So people want to track it and check that work that they’re paying for is delivering, and is aligned to strategy.

The answer… short feedback cycles. Feedback cycles let you measure what you’re spending and you can go and have a look at the output to check it’s delivering business value.

The problem – Programmes?

Sometimes software solutions aren’t simple products or mashups. Sometimes they’re big/complex combinations of things developed internally, externally or collaboratively with partners. In these cases we sometimes need to align teams towards common scope and architecture so that products can work together to deliver business value. Collections of Projects working together  = Programmes.

This means we need an understanding of the stories across products (I call these Integration Scenarios) and the common product family architecture (Solution Architecture).

The answer… short feedback cycles. Feedback cycles at product level work as described above. At Product Family level contributing product releases can be brought together to form a Product Family Release – as often as possible to get feedback working. (Hint: 10 weeks is too long, especially if integration fails because 20 is definitely too long). Continuous, event-driven integration streams enable flexible and recursive short feedback cycles.

The problem – Support Costs?

If teams are chasing the latest shiny technology all the time then a large organisation can quickly end up in a support hell of mixed platforms, technologies, middlewares, deployment stacks, licensing and support costs. Needing to recruit specialists in everything and duplicating operational environments.

This one can be partly solved by moving the external complexity to external clouds (where the diversity is embraced and normal) and also by (lightweight!) Enterprise Architecture. Lightweight Enterprise Architecture makes technology choices for the organisation (for it’s mainstream development, it shouldn’t constrain research & innovation) to prevent this fragmentation in the first place.

This can also be partly solved by getting IT Operations stakeholders involved early and often as part of the holistic team (DevOps) along with the other stakeholder sets.

Of course it needs checking to make sure it’s still a good idea and that it’s aligned to strategy. Probably should do that frequently and empirically… short feedback cycles.

Don’t scale the practices, just be agile!

Often proponents of “Agile at scale” or “scaled Agile” will talk about scaling Scrum (or similar). Using practices that work really well at team iteration level to run a business strategy, or a team of teams of teams (at programme level). In my experience that’s not a great idea, just because you’ve got a hammer doesn’t mean every problem in an organisation is a nail.

Staged daily standups or scrum-of-scrums are just painful to behold. Execs don’t want a strategy backlog, nor do organisations want architecture to emerge with no regard for support costs, recruitment or cross-project dependencies.

If I was invited to a 2 day planning session I’d probably walk out of the door never to return. I’d rather watch animated musicals on repeat.

We don’t need to scale agile, we just need to be agile… with a little “a”. Short feedback cycles, a focus on people and relationships with tight customer collaboration. Open, transparent and empirical collaboration between people – that’s the right approach. One that work at every level of an organisation.

Recursive feedback cycles enable scaled agile

Feedback cycles scale to all levels of an enterprise

 

%d bloggers like this: