Mike MacDonagh's Blog

Somewhere in the overlap between software development, process improvement and psychology

Tag Archives: governance

Holistic Software Development – the book

The HSD book is currently in it’s final editing phase. If you’d like to purchase the book then please bookmark this page and come back soon.

What’s inside?HSD Book

The book covers a detailed introduction to Holistic Software Development and it’s principles. Covering the H-Model and how it us used to map levels of decomposition to recomposition and integration the book then follows a structured path through HSD material.

We discuss Strategy and Holistic Portfolio Management to help guide organisations in adopting post-agile flexibility at the top level. We cover the HSD requirements model and how it supports strategic intent without falling into the Big Up Front Requirements trap.

A detailed chapter on architecture follows covering enterprise, solution and system architecture and its representation through over sketches and architectural mechanisms.

We cover Planning and how to balance the needs of business planning with agile just in time planning. We deconflict project management and agile ways of working showing how they can work together in a holistic organisaiton.

Releases and Integration is the heart of the H-Model and so is where the focus of HSD comes together. We discuss the dual nature of releases, and how they are best set up across teams of teams in both a technical configuration management context and a planning context.

We cover the HSD approach to Quality and Operations before covering a detailed view of People structures and organisational design. Based in psychology we look at motivation, teaming and large scale structuring covering the HSD Hybrid Dynamic Model in depth.

Finally we look at Governance and HSD variants for simple organisations and COTS before briefly covering Adoption practices.

 

We’re working through the last few edits of version 1.0 currently and hope to have the book published in 2016, you can now sign up to be notified when it’s published on the HSD site.

Advertisements

Holistic Software Development @AgileInABag London 12 June 2015

Agile In a Bag London 2015

I’ll be presenting with my friend and colleague Steve Handy about Holistic Software Development at the Agile in a Bag conference in London on the 12th June 2015.

If you can make it, do come along to this great 1 day conference to have some fun, share some thoughts and ideas and meet some interesting people 🙂 Here’s the blurb for our talk first thing (9am) in the morning:


 

These are not the practices you are looking for

Is your time to market quick enough for your business to respond?
Is your Programme and Project Management smoothly joined up with your agile teams?
Does everyone understand how their work contributes to business value?
Do you know what end-to-end scenarios will be delivered in the next programme release?

Come to this talk to hear how these common problems are easily solved, even in highly complex system-of-systems organizations by using the freely available “Holistic Software Development” (HSD) framework which aligns agile philosophy throughout the enterprise, at-scale engineering, architecture, strategy, portfolio and programme management. Using a novel but lightweight end-to-end scope level requirement type we iteratively drive the development of well understood Releases and customer feedback loops.

Combined with layered iterative or continuous integration and clearly understood progressive Definitions of Done we can simplify the complex resulting in a perfect join up between customers, requirements, planning, delivery teams and governance. HSD is the glue that plugs together your existing agile team practices (e.g. Scrum) with PPM practices (e.g. Prince2) and business strategy providing people focussed, lightweight guidance firmly based in evidence, experience and excellence.

And here’s the learning outcomes we’re aiming at:

  • Understanding how to de-conflict agile and enterprise portfolio and programme management
  • How to unify agile teams, governance, customers and executives in a simple model
  • The “H-Model” of software development
  • How to join up different schools of thought/processes into a cohesive way of working
  • How a strong approach to understanding business value enables effective software businesses

 

You may detect a slight Yoda-like grammar to the title… Come or come not, there is no spoiler.

The rise of the Chief Software Architect

ArchitectureSoftware is increasingly important to everyone, it’s everywhere. It’s in our phones, runs our cars, our cities, our healthcare, entertainment and utilities. There are few businesses without a software element and many that are critically software dependent. What these organisations are finally beginning to understand is that the business of doing software is difficult, in fact it’s complex. It needs C-suite level direction and coverage.

Being good at software isn’t a matter of hiring the good coders, buying the right workflow management tool, using the right Configuration Management system, hiring the right management consultants or using the correct technology. These things all have a part to play but they’re supporting parts to the most important factor: culture.

Software Development is a team activity and the team is bigger than we used to think it was (a few developers, testers, direct customers, managers, analysts etc.). A good set of development practices at team level isn’t enough to be effective because software development is a business enabling activity and so it touches all parts of an organisation (development teams, hr, procurement, security, business change, legal, financial, business services, operations, support etc.).

As organisations evolve to make the maximum use of technology they are making increasingly complex software products, often through diverse technology stacks, using a variety of in-, near-, and out- sourcing partners. Delivering systems of systems across teams of teams is not simple and so organisations are now looking to embrace the importance of software, technology and ways of working to their businesses at board level.

What is a Chief Software Architect?

Whether a “Chief of Software“, a “Chief Software Officer“, a “Chief Digital Officer” or “Chief Architect” the role tends to include:

  • Leading and encouraging a positive software development culture
    • Where delivery of business value is the primary measure of success and decisions are based on business value
    • Where failing fast is a cause for celebration not a career ending issue
    • Where software is designed for users/customers needs
    • Where continuous/iterative development integration and deployment are the norm
    • Where teams collaborate and co-create with each other and their customers to produce the best products
    • Where thinking all day to write a perfect algorithm is worth more than pushing out hundreds of lines of code quickly
    • Where professionals are trusted to do their jobs properly
    • Where organisational impediments to smooth software development are removed, from board level if necessary (due to Conway’s Law transformational change is often impossible without such top-level leadership)
  • Providing top level architectural direction and governance (sometimes called Enterprise Architecture)  that strikes the balance between directing technical decisions towards alignment but not imposing restrictive unchanging standards on development teams
    • Architecture is understood cohesively at Enterprise, Solution and System levels
    • Architecture exists in the form of directed (but community driven) standards, off-the-shelf mechanisms and principles
    • Architecture exists primarily as executable software although appropriate weight documentation, models, examples and usage guides describe it
    • Architecture provides a context for making technical decisions in normal development, makes technical decisions on commoditized platform/software/operations choices but does not constrain innovation and research (architecture has a different impact in different parts of a hybrid dynamic model)
    • Architecture enables control of whole-lifecycle costs by smoothing the flow of work from innovation and research into mainstream development and support
  • Ensuring that teams can use the tools they want and need to use, not those that had the best sales pitch to a non-technical board (hint: these are normally small, simple and/or open source, not those provided by large vendors) balanced by creating a cohesive community that isn’t unnecessarily fragmented by every team needing to set up different tools environments
    • Tool chains take the pain out of development, testing, build and deployment – automated wherever possible
    • Development environments are appropriate for actually developing, building, testing and deploying
  • Ensuring, and embodying, the principles of agility, devops, and agility-at-scale at the organisational level – promoting the evolution of working practices, operating models and ways of working while avoiding endless management fads
    • Building and championing the needs of the development community, attending to their needs
    • Ensuring that portfolio selection takes into account technical issues, shaping it in line with architecture and in turn using business priorities to shape architectural choices

Why “Architect”?

The Vitruvian Man - Leonardo Da Vinci

The Vitruvian Man – Leonardo Da Vinci

At a business level Architecture exists to guide organisations through the business, information, process and technology changes necessary to execute the organisation’s strategy. Because “architecture” has always been more than just the technology bit, it also focusses on the non-functional aspects necessary to make the technology work terming a “Chief Software Officer” an Architect is an organisation recognising that to be successful at software a holistic approach is required that combines business concerns with technical concerns.

The idea of an “Architect” role with such a broad focus is analogous to that of a structural architect concerned with buildings. Indeed in 25 BCE Vitruvius (a well known Roman writer and architect who inspired Leonardo Da Vinci, hence the Vitruvian Man ) described an architect as:

The ideal architect should be a man of letters, a mathematician, familiar with historical studies, a diligent of philosophy, acquainted with music, not ignorant of medicine, learned in the responses of jurisconsults, familiar with astronomy and astronomical calculations.

Virtuvius is famous for stating that architecture must be solid, useful, beautiful (De architectura). The same three qualities relate to software architecture. Architecture is a fine balance between a subtle science and an exact art – as such it is the magic sauce that turns a technology strategy into business benefits.

If your business is reliant on technology and/or software and you don’t have a Chief Architecture, you should promote/get one before you become technically irrelevant, encumbered by lumbering traditional delivery approaches based on manufacturing and negative development cultures.

It’s time for businesses to recognise the important of software, and architecture, before they’re outmanoeuvred by those who have.


 

For more on Architecture see:Architecutural assets

 

 

The Hybrid Dynamic Model

The Hybrid Dynamic Model is a modern approach to structuring an organisation’s portfolio that allows for multiple ways of working to co-exist from innovation to utility work and for that work to change which processes, cultures and practices it uses over time.

An “Operating Model” describes how an organisation does its business in terms of both its structure and its behaviour – it is the business architecture. An “Operating Model” exists in every organisation, whether it is explicitly written down or simply a collection of processes and practices. The Operating Model includes how strategy flows through planning and delivery, how money is spent and controlled, how governance works, how people are organised, how quality is assured etc. If the Operating Model is inefficient or unable to react well to change, everything else suffers.

Many organisations evolve an Operating Model that fits their stable way of working, allowing for a focus on efficiency and improvement, driving out variation. This approach can work well in periods of stability but makes organisations brittle and unable to react well to change because the only way the organisation knows how to deal with things is to apply its tried and tested approaches. When the environment is less stable, or the variety of work types increases a Hyrbid Dynamic Model is more appropriate.

A useful mechanism for understanding different types of work, and the different approaches that may be applicable to delivering them within a portfolio is the “Commoditization Scale”. The Commoditization Scale describes a range of types of work from innovation (undefined, rare, speculative and high risk) to more commodity/utility (well defined, ubiquitous, and less differentiated with lower risk).

These different types of work require different ways of working, contractual models, technologies, supply chains, cultures, processes etc. More well understood work is more predictable and manageable by its nature, whereas the opposite is true for less well-understood work.

Commoditization scale Commoditization scale inspired by Simon Wardley – CSC Leading Edge Forum

As organisations begin to understand their work in terms of a hybrid model they can then look to actively apply a commoditization pressure where possible moving work across this scale. Because commodity solutions (often bought in) are typically cheaper than in-house development, maintenance and hosting commoditizing work earns a “Commoditization Dividend” which can then be used to reinvest into the Innovation parts of a business.

As mentioned in Build or Buy we generally recommend against building utility or commodity components, systems or product (unless disrupting a service industry) – these kind of products are best bought or provided by open source alternatives (COTS). However in terms of inventing/innovating and specialist products different processes, working practices, teams and cultures may be necessary. Innovation work is typically riskier and speculative than more mainstream specialist products and so will be less tolerant of up front planning. “Failing fast” is particularly important in innovative/inventive work and this type of work is typically very unpredictable. As a result building significant structure (in terms of project, programme and Solution Architecture) around innovation work is unlikely to be productive – this work may also be unconstrained by Enterprise Architecture (and indeed “standard” ways of working).

Unless explicitly necessary we recommend that innovative work is treated as self-organized Product Delivery teams that are directly part of the Portfolio, with no programme structure. This (Small Picture) model is preferred for small, simple projects or organizations generally. Small organizations can often outperform large monolithic organizations due to their lack of process, management and architectural inertia – they are free to innovate. Since the relentless commoditization of technology forces all organizations to innovate to survive we strongly recommend freeing people to innovate in whatever manner they need.

There’s nothing worse for innovation than an ideas process

Similarly, and perhaps counter intuitively, there is also less structure required in the Utility and Commodity areas of the commoditisation scale (because the products built are often well-understood, predictable, related to system integration and frequently COTS). This type of very predictable work often benefits from workflow management techniques such as Lean, Kanban or service management processes.processes and commoditisation

Products that are more Specialist may need a “bigger engineering” approach depending on their scale or risk factors. Many large organizations will adopt more formal processes in this area as they seek to find economies of scale, especially around reuse. However over time reusable platforms and architectures can become a hindrance rather than a help. Technically skilled people will notice this point and are encouraged to Bubble Up their concerns. Organizations with significant effort in more than one of these areas of the scale above tend to gravitate towards a hybrid model with a structure embedded in the Specialist, Commodity or Utility part of the business. Applying the same structures, architectures, processes and working practices across all of these work types can cause problems so a hybrid model is likely to be required for large/complex organizations.

We recommend using a hybrid, dynamic model for organising a large/complex portfolio. Large, complex organisations have significant effort and coverage across the commoditisation scale. Implementation of a hybrid model caters for this variance using unstructured and structured models where necessary. Structure is introduced only by exception when explicitly needed to orchestrate effort across divergent teams and portfolio(s) not as the default approach. Importantly as pieces of work become more understood, or product sets evolve they will likely begin to move on the commoditisation scale and so may need to dynamically change their ways of working.

The Hybrid Dynamic Model allows for significantly different ways of working, within the cohesive governance of the H-Model of Holistic Software Development. Using the hybrid dynamic model organisations can have multiple ways if working, cultures and practices co-existing peacefully within a single enterprise portfolio. Using different ways of working, practices and techniques across the portfolio helps bring together the otherwise extremely different extremes of innovation and research work with predictable project delivery. Other business models, such as Incubation, can also be used to help bridge the gaps between areas such as innovation and mainstream engineering practices.

Hybrid Dynamic Model

Many organisations evolve a Portfolio → Programme → Project → Team model which is then applied to all work in the portfolio due to its past success at delivering large predictable engineering projects. However, this model is wasteful when complexity is low and can actually inhibit communication and feedback cycles in more speculative work.

Programme and project structures cause layers of separation, which result in emergent transactional behaviour, effort duplication and confusion over roles and responsibilities. As a result programmes should only be used when there is a genuine requirement for co-ordinating multiple projects that must be integrated for a cohesive product family or business capability. In other cases programme structures should be condensed – removing programmes that are simply funding lines; Portfolio Management can and should manage funding lines and stand-alone projects can simply be part of the portfolio.

Reduction in structural complexity will reduce costs, increase flexibility and adaptability while simultaneously improving an organisations ability to react in a rapidly changing environment. Additionally, simplifying structure will reduce the sheer numbers of people involved in project management and definition work, removing unnecessary barriers between delivery teams and their customers.

This kind of hybrid structure is similar to using the Small Picture of HSD for the Commodity Portfolio, the Big Picture for the Specialist Portfolio and the Small Picture again for the Research and Innovation portfolio – naturally sharing the same Strategy and Governance layer.

COTS implementation may be a signficant piece of work by itself. The COTS Implementation article describes a variant of the Big Picture for use with COTS.

Agile contracts? What about Weighted Fixed Price?

MichaelangeloAgile contracts are tricky. In fact Software Development contracts are tricky. Software development is a non-deterministic creative endeavour, you’d be better off trying to nail down the acceptance criteria for one of Michelangelo‘s sculptures than a typical software project. Software is complex.

A few years ago I was involved in putting together a contract model called “Weighted Fixed Price” and recently it occurred to me that I’ve never blogged about it, so I’ve dragged myself away from the source to do so.

What’s wrong with the normal models?

So, fixed price is bad because it only works when you can really accurately define the requirements, acceptance criteria and the potential risk exposure is tiny. Otherwise you get schedule and cost over-runs, formal change control etc. etc. Of course supplier’s always build in their risk buffer into the fixed price so it tends to cost a little more than honest T&M would. Software work is rarely this predictable which is why there are so many high profile failures.

Time and Materials is bad because it’s hard to manage across commercial boundaries and customers can feel that they’re paying for nothing, endlessly. Even defining waypoints is difficult due to changes in requirements and over time. Unscrupulous suppliers could just sit back taking the money, delivering as little as possible to avoid gross negligence but nothing more. Scrupulous suppliers could be working hard but hit by the many and varied risks arising from such complexity and yet perceived by their customers as the unscrupulous.

Of course there’s variations like Capped T&M, Phased Fixed, Phased Capped T&M etc. but they all start getting a bit complicated and hard to manage, not to mention open to gaming.

Then there’s models like pay-per-point and other “incentive” schemes that raise over-complication, missing the point and gamability to an art form. Just say “no”.

Competition

The real incentive to suppliers to do a good job is follow on work. That means that as a customer you need your suppliers to be in a competitive market, being locked in to individual suppliers leads to captive market behaviours.

Weighted Fixed Price?

In an attempt to put something in the middle between Fixed Price for low risk exposure and T&M/internal for high exposure I put together “Weighted Fixed Price”. The idea here is to to basically use a Fixed Price model but have suppliers weighted by previous performance affecting whether they get the business or not.

So if a supplier 1 delivers 10% late on Project A then when we’re looking to place Project B for 500k and we’re looking at each supplier’s estimates then we’ll add 10% to supplier 1’s estimate.  That means all over things being equal we’ll choose the other supplier’s who typically come in closest to, or even under, their previous estimates. Weighting is transparent to each supplier but not across suppliers.

This provides the following pressures:

  • Suppliers don’t want to underestimate, they’ll come in over their initial estimates and adversely affect their weighting and therefore their chance for follow on work
  • Suppliers don’t want to overestimate to artificially lower their weighting or build in risk buffers, they won’t get the work because they’ll cost too much
  • Suppliers don’t want to take on complex, high risk work because they can’t predict the effect on their weighting (more below)

Out-sourcing doesn’t transfer risks to suppliers, because the customer is still the one needing the business benefit and paying for it, especially when things go wrong. The Weighted Fixed Price model is intended to genuinely transfer the complexity risk to suppliers, if they’re willing to take it.

What happens when the supplier finishes early?

They deliver early, watch their weighting affect their potential for winning future work increase massively and enjoy their fixed price money meaning that they’ve increased their profit margin on this job.

What happens when they deliver late?

They deliver late, their weighting increases and they’re less likely to win against other suppliers next time. However, if they could predict that they’re going to deliver late early in the project then they could renegotiate with the customer in light of what everyone’s learned about the work.

What happens when suppliers don’t want to bid for work because it’s too risky/complex?

Excellent, this means that they’re honestly telling us that the work is too risky to predict. This means we shouldn’t be out-sourcing it in the first place, time to stand up an internal team or supplement with some time-hire contracts if we need extra capacity.

Conclusion

The Weighted Fixed Price model is an evidence based contract model can be used in competitive supplier markets for reasonably well understood work with low risk exposure. It incentivises suppliers to estimate accurately and deliver to their estimates as that affects their opportunities for follow on work. It incentives customers not to out-source overly complex risky work.

It’s not perfect because we’re still trying to contractually define how a sculpture should make us feel but it’s something you can add to the mix of contract models you use.

Intentional vs. Emergent Architecture

I’ve been thinking about architecture a lot recently but one thing that I often discuss but have never blogged about for some odd reason is intentional vs. emergent software architecture. Some old fashioned software methods such as waterfall led people into doing a lot of up front architecture work, they analysed and designed away for ages producing huge reams of UML and documentation that no one could ever squeeze into their heads if they had the patience to read it. This is an example of an intentional architecture – the architecture was intended, planned and deliberate.

Lots of folks hated that way of doing things as it meant people were writing docs and drawing diagrams instead of making working software, not to mention an all to frequent tendency to over-engineer architecture past the point of usefulness. This led to some people saying that we’re better off not trying to do any architecture and just letting it emerge from the work we do developing small little customer focused requirements (like user stories or similar).  Ok, so there’d be some rework along the way as we encounter a deeper understanding of the system and refactor our emergent architecture but it’d still be better than the old way of doing large upfront architecture.

So, there seem to be two opposed viewpoints: intentional architecture is best, emergent architecture is best.

For me, neither is true. I’ve seen some really terrible examples of badly over-engineered architectures that crippled a project and projects that never got past their over-reaching architectural analysis. Equally I’ve seen products with emergent architecture that had to be entirely re-architected as time went pay because their emergent architecture was so wrong it was comical (imagine a software management tool that only supports a single project, and that concept being deeply embedded in the architecture).

There’s a scale with intentional architecture on one side and emergent architecture on the other.

Intentional vs. Emergent ArchitectureVarious factors might push us one way or another… The second I listed on the right is interesting as if you’ve got a well known technology and problem domain you can get away with emergent architecture, but similarly if you have a totally unknown technology and problem domain it can be very effective to evolve towards a solution and architecture rather than crystal ball gaze by creating a (probably wrong) intentional architecture.

Which rather sums up the point I’m trying to make. The purpose of architecture is to shape the solution and address technical risks. Solving the big problems, creating common ways of doing something (lightweight architectural mechanisms) are all good architectural goals but only if we’re sure of the solution. If we’re not we’re better off evolving an emergent architecture, at least initially.

I think that the extremes at either end of the scale, as with most extremes, are a bad idea at best and impossible at worst. If you gather a group of people together and tell them to create a web app given a backlog but they’re not allowed to think or communicate about the architecture up front then you’ll find they all start dividing the problem in different ways and writing in different languages for different server and client frameworks. Hardly a good idea. Of course on the other end of the scale, believing that we can foresee all of the technical issues, all of the technology and requirements changes that might happen is as likely as a 12 month project plan Gantt chart being correct after a few levels of cumulative error margin have been combined.

For more on architecture see:

Scaled Agility: The Project Forum

This blog is an extract from the Project Forum practice: Holistic Software EngineeringThe Project Forum

When it might be appropriate

  • In situations where multiple competing stakeholder groups with different agendas are required to work together
  • In situations where multiple product groups need to collaborate on a bigger outcome
  • Where there is a conflict in direction, resource management/ownership or scope between collaborating groups
  • System of systems development

What is it?

The Project Forum is an application of agile philosophy to large project structures. Rather than impose a hierarchy of decision making from the Project Manager downwards the Project Forum is a virtual team in the middle of all stakeholders.

The Project Forum is a self-organising democratic group that balances competing voices and concerns, owns high level scope and architecture, runs the high level release train and performs integration activities for the product.

Use of the Project Forum practice does not prevent any communication directly between contributing groups it only provides a vehicle for that conversation when it’s relevant for the wider project.

From Traditional to Agile at ScaleThe Project Forum practice is an example of Agile at Scale combining social business practices, technical software practices and ways of working to make a simple way of doing big complicated bits of work.

Read more of this post

Simple software project measures

I’m not a big fan of metrics, measures, charts, reporting and data collection. I’m not terribly impressed by dashboards with 20 little graphs on showing loads of detailed information. When I’m involved in projects I want to know 3 simple things:

  • How quick are we doing stuff?
  • Are we on track or not?
  • Is the stuff good enough quality?

There can be some deep science behind the answers to those questions but at the surface that’s all I want to see.

Organisations need to know that teams are delivering quality products at the right pace to fit the business need. To achieve this goal teams need to be able to demonstrate that their product is of sufficient quality and that they can commit to delivering the required scope within the business time scales. If the project goal may not be achieved then the business or the team need to change something (such as scope, resources or time scales). This feedback mechanism and the open transparent communication of this knowledge is key to the success of agile delivery.

The goal of delivering quality products at the right pace can be measured in many complex ways however, when designing the Project Forum agile at scale practice we looked at just 3 measures. In fact I should probably call them 2.5 measures as the throughput/release burnup can be considered mutually exclusive (if you’re continuous flow or iterative). The most important measure is people’s opinions when you go and talk to your team.

Simple Measures Dashboard

Note: in the measures section I often refer to “requirements” as a simple number, this could be a count, a normalised count, magnitude, points, etc. it doesn’t matter what’s used so long as it’s consistent.

Read more of this post

How to avoid Fragile Agile, Flexibility in Context

EAC = Estimated cost at Completion

One of the figures often included on project reports, and present in many PM tools, is the EAC and EAC variance. EAC stands for “Estimated Cost at Completion” and is equal to the actual cost of work performed so far + the remaining estimate to complete. Expressed as a formula of confusing ‘initialisms‘ this means that EAC = ACWP + ETC.

For projects  the EAC represents the total projected cost of the project. At the very beginning of a project when there hasn’t been any spend,  the EAC is equal to the Estimate to Complete (ETC) and this is the first project baseline. As a project executes changes happen and projects may start over-spending or under-spending. In these case the EAC will start to vary from the baselined value as the actual costs + the remaining estimates are either up or down from the original baseline. This variance is expressed as EAC variance. A negative EAC variance indicates a project that looks like it is going to come in under the original budget baseline figure. A positive EAC variance indicates that a project is likely to overspend relative to the baseline figure.

EAC may be expressed as a financial figure or as an effort (in hours) value. EAC can also be combined with agile burnup measures across projects.

%d bloggers like this: