Mike MacDonagh's Blog

Somewhere in the overlap between software development, process improvement and psychology

What’s new in HSE 1.3

We’ve released an update for Holistic Software Engineering (www.holistic-software.com) which:

 

Collective nouns for software development roles

An embarrassment of Project Managers

An impasse of Architects

A confusion of Business Analysts

A mob of Developers

A silo of Testers

A brethren of Scrum Coaches

A waste of Lean Consultants

A conspiracy of Process Improvement Consultants

People are important, not roles! But recently I saw a weekly team meeting for a project that had 12 project managers! I thought it was a joke at first.

Can you think of any others?

Video: holistic approach to scaling agile in 46 seconds

What is agile architecture?

Agile ArchitectureArchitecture is a high level view of a system in the context of its environment, dependencies, technology, structure and behavior.

Architecture must be solid, useful and beautiful.

Software architecture is typically hard to define as the term software architecture is used to describe many facets of software structure, behaviour, design, activity and documentation. The concept of software architecture is also relevant at different levels of a system as the components of one system become a reusable platform for the next system. Ultimately the emergent behavior caused by layers of architectural dependencies and reuse can be difficult to predict.

Architecture is a simplification of the system, used to communicate the nature of that system and guide its development. However, architecture is more than just a diagram, it is grounded in technical context describing technology stacks, standards adopted, common ways of doing things (mechanisms), how non-functional requirements are met and the elements that will meet functional requirements. Architecture provides a shape and look and feel to the internals of a system that are the foundation for the ultimate external behavior.Da Vinci's Virtuvian Man

The idea of an “Architect” role with such a broad focus is analogous to that of a structural architect concerned with buildings. Indeed in 25 BCE Vitruvius (a well known Roman writer and architect who inspired Leonardo Da Vinci, hence the Vitruvian Man ) described an architect as:

The ideal architect should be a man of letters, a mathematician, familiar with historical studies, a diligent of philosophy, acquainted with music, not ignorant of medicine, learned in the responses of jurisconsults, familiar with astronomy and astronomical calculations.

Virtuvius is famous for stating that architecture must be solid, useful, beautiful (De architectura). The same three qualities relate to software architecture. Despite architecture being a fine balance between a subtle science and an exact art we must realise it is only useful if it is aligned to requirements and becomes executable.

Architecture, not documentation

In line with one of the founding Principles of Holistic Software Architecture I:

Value working software in the form of quality releases from short development cycles over comprehensive documentation, business analysis and enterprise architecture documentation.

Choosing how much architectural work is necessary up-front and especially how much architectural documentation is necessary is difficult.

Traditional document focused development methods promoted large up-front effort to detail, in document or model form, the architecture and system design prior to development. As well as the inherent risk extension issues in waterfall processes this could often cause “analysis paralysis” where architecture work was seen as an endless diagram drawing exercise.

Agile methods focus on working software over documents and designs, however this doesn’t mean that no documents or designs should be produced. It is almost always useful to have some description of an architecture before development even if it’s just a statement that the team’s usual architecture will be used as otherwise team members won’t even know which language to start writing in.

Good architecture addresses how we’ll resolve the major technical risks, communicates between the team the overall structure of the solution and works out how our solution will meet the functional and non-functional requirements. Knowing when we’ve worked out enough architecture so that we’ve reduced the complexity of the problem into manageable sizes for the team is a difficult challenge.

Levels of architectural formality

Doing too much architectural analysis or elaboration, either through abstract design and modelling or practical spiking (writing small amounts of throwaway code to demonstrate the feasibility of a technical approach or architectural idea) will slow down a development project and increase the risk of wasted work. Value is only achieved once working systems are in the hands of the customers/users.

Alternatively not doing enough architectural analysis can lead to significant architectural refactoring during a projects lifetime, as key requirements can’t be met based on the current architecture, again leading to wasted work and slowing down the project.

When doing up front architecture, especially in the form of documents and models, we need to be careful to consider architectural work in the context of the team’s definition of done. Typical levels of done don’t normally include “analysed”, “architected” or “designed”. Although these terms might be meaningful to describe internal team development states they do not constitute working software and are only a step on the way to creating value.

Guidelines for good architecture and design

Having a common understood architecture, regardless of the format of that understanding (documents, models, sketches, whiteboards, implicit team knowledge), is described as “Intentional Architecture”.

Holistic Software Engineering described “good” architecture and design as having the following characteristics:

  • Intentional structure and behaviour
  • Highly modular: consisting of separate services, components, classes, objects or modules
    • Elements are highly cohesive
    • Elements are loosely coupled
    • Elements have low algorithmic complexity
  • Avoids duplication
  • Well described elements: modular elements have simple expressive meaningful name enabling readers to quickly understand their purpose, similar to code cleanliness we should strive for design cleanliness
  • Runs and passes all defined tests or acceptance criteria
  • Lightweight documentation

Scaling agile is the wrong approach

The heart of agile software development is feedback loops. Doing a bit of software, looking at it, at how we did it and then improving things as we do another bit of software. The “things” that can be improved can be quality, scope, usability, performance etc. etc. Perhaps most importantly, relationships and ways of working can also be improved.

This rapid feedback cycle ensures that product development is aligned to customer needs, that progress against guestimates is understood (so that plans can be refined) and that teams which include customers find problems early, and so can fix them, as quickly as possible. It’s a quicker, better, cheaper and happier way of working.

What about that needs to scale? Empirical feedback is just fine at any scale. Across 100s of teams or teams-of-teams and entire organisations short term feedback cycles work perfectly well.

The problem – Portfolio Management?

The problem is that there’s other stuff that goes on too. Product Development in big or complex organisations is part of an investment portfolio and costs a lot of money. So people want to track it and check that work that they’re paying for is delivering, and is aligned to strategy.

The answer… short feedback cycles. Feedback cycles let you measure what you’re spending and you can go and have a look at the output to check it’s delivering business value.

The problem – Programmes?

Sometimes software solutions aren’t simple products or mashups. Sometimes they’re big/complex combinations of things developed internally, externally or collaboratively with partners. In these cases we sometimes need to align teams towards common scope and architecture so that products can work together to deliver business value. Collections of Projects working together  = Programmes.

This means we need an understanding of the stories across products (I call these Integration Scenarios) and the common product family architecture (Solution Architecture).

The answer… short feedback cycles. Feedback cycles at product level work as described above. At Product Family level contributing product releases can be brought together to form a Product Family Release – as often as possible to get feedback working. (Hint: 10 weeks is too long, especially if integration fails because 20 is definitely too long). Continuous, event-driven integration streams enable flexible and recursive short feedback cycles.

The problem – Support Costs?

If teams are chasing the latest shiny technology all the time then a large organisation can quickly end up in a support hell of mixed platforms, technologies, middlewares, deployment stacks, licensing and support costs. Needing to recruit specialists in everything and duplicating operational environments.

This one can be partly solved by moving the external complexity to external clouds (where the diversity is embraced and normal) and also by (lightweight!) Enterprise Architecture. Lightweight Enterprise Architecture makes technology choices for the organisation (for it’s mainstream development, it shouldn’t constrain research & innovation) to prevent this fragmentation in the first place.

This can also be partly solved by getting IT Operations stakeholders involved early and often as part of the holistic team (DevOps) along with the other stakeholder sets.

Of course it needs checking to make sure it’s still a good idea and that it’s aligned to strategy. Probably should do that frequently and empirically… short feedback cycles.

Don’t scale the practices, just be agile!

Often proponents of “Agile at scale” or “scaled Agile” will talk about scaling Scrum (or similar). Using practices that work really well at team iteration level to run a business strategy, or a team of teams of teams (at programme level). In my experience that’s not a great idea, just because you’ve got a hammer doesn’t mean every problem in an organisation is a nail.

Staged daily standups or scrum-of-scrums are just painful to behold. Execs don’t want a strategy backlog, nor do organisations want architecture to emerge with no regard for support costs, recruitment or cross-project dependencies.

If I was invited to a 2 day planning session I’d probably walk out of the door never to return. I’d rather watch animated musicals on repeat.

We don’t need to scale agile, we just need to be agile… with a little “a”. Short feedback cycles, a focus on people and relationships with tight customer collaboration. Open, transparent and empirical collaboration between people – that’s the right approach. One that work at every level of an organisation.

Recursive feedback cycles enable scaled agile

Feedback cycles scale to all levels of an enterprise

 

#Devops – A sticking plaster for a bigger problem

Why not StratDev, BusOps, etc.?

Holistic Communication

The term “DevOps” refers to tight integration between the Software Development and Operations parts of an organization. As I mentioned in my article on Conway’s Law the reinforcement of there being two separate parts “Dev” and “Ops” causes a separation in the way we think about these problems and then how we interact within an organisation. If the industry truly believed in DevOps then “dev” and “ops” would no longer exist. I realise that some people intend “DevOps” to be a verb, but it sounds like a noun.

Of course the idea behind of DevOps, of bringing together stakeholders involved in creating, hosting, running and maintaining is a great idea – focussing on these things can make it easier to deploy products and get user feedback. However we might just end up building the wrong products quickly. We must make sure that the things we build are aligned to strategy and deliver business value. To do that we need a holistic approach, that involves all stakeholders (not just the dev and ops folks).

Operations stakeholders should be involved in the Portfolio Build/Selection practice to ensure that the impact of new product development or maintenance decisions on current Operational positions are understood. Significant changes to current Operational positions are raised as Portfolio Requests so they can be properly costed and scoped.

Work that is approached holistically must by definition include addressing the needs of all stakeholders from any part of the business. If a divide exists between parts of the business then the dysfunctional relationship needs addressing rather than process and tooling being introduced. Processes and Tools are helpful to gain agreement on a common approach and automate ways of working but not to solve dysfunctional relationships.

HSE promotes regular builds and releases in either iterative/agile or continuous flow models Operations stakeholders must be involved early and often in software development product delivery as a product is likely to be delivered frequently, or even continuously, and so will need transitioning activities performed repeatedly.

So, I like the idea of joining up dev and ops stakeholders and work. But I only like it in the context of also joining up business strategy, requirements, architecture and quality with dev and ops – all flowing business value to customers. On it’s own, DevOps isn’t enough and worse could just be a management fad.

And about me

I’m some way between a tools geek, process geek, (post-)agile geek, psychology geek and programming geek.

I work with both private and public sector organisations worldwide to help improve the way software is done.

I’m a UK based consultant with over 10 years experience in the software industry enabling teams in the adoption of various software engineering practices and associated toolsets – providing training, consultancy and support in both small teams and entire development organisations (1000s) – scaling agility and lean concepts at the organisational level and de-conflicting them with portfolio, programme and project management.

I specialise agile, lean and iterative methods and practices ranging through the practical implementation of scrum, kanban and RUP, business and system analysis covering both use cases and user stories (often together), requirements management, architecture and design as a sketch, iterative and continuous flow patterns and continuous integration practices. I don’t believe in slavishly following a process or meta-model but in collaboratively improving the software value stream.

I’m a co-Author and co-Founder of Holistic Software Engineering which brings together all of these things in one simple picture.

I’m the Technical Director of Holistic Software Consulting Ltd and also Soft Practice Ltd.

I’ve got deep expertise in various development tools such as (git, jenkins etc.) as well as commercial tools IBM (Rational toolset – RTC, RRC, RQM, RSA etc., Microsoft TFS) – I also make my own.

I’m interested in new technology, psychology, kung fu, music and guitar playing, travelling, philosophy, linguistics, close up magic, biology, maths, physics (especially astronomy and quantum physics) etc…

Launch: Holistic Software Engineering

How do we join up business strategy to agile development? Is program management relevant? Where do project managers fit in? What about architecture?

Holistic Software Engineering (HSE) answers all of these questions – for free.

Agile and continuous flow are great for small teams or a small number of inter-related small teams working on exploratory or maintenance work. But what if we’re spending 100s of millions on an IT strategy of inter-related products that need to work together to deliver business value. What is business value anyway?

H-Model To answer these questions (and more) my friend Steve Handy and I have distilled our collective 30+ years of software experience in a single, cohesive model of software development. We’ve developed the H model that moves on from the v-model and it’s siblings by combining:

…all elegantly combined and de-conflicted by release planning.

We’ve not invented very much, we’ve simply put a lot of good ideas from others together into a cohesive framework. We’ve drawn it all as a big picture and developed a website that shows how to get value from all of this stuff. Everything is clickable, everything has content.

The best bit: it’s free! There’s no paywall, there’s no private “full” version, you can just use it or not as you like.

We don’t believe in process zealotry, or putting academic concerns above clarity and usefulness. HSE is indicative, not prescriptive. You’re already doing it and if you use the big picture to draw red blobs on the bits that aren’t working well, or missing, in your organisation then you can use the model to make tangible improvements – immediately.

Using HSE doesn’t replace any of your existing processes, it provides the glue that joins them all up together in one simple, elegant and cohesive model.

Holistic Software Engineering
is for large/complex software organizations
who need to understand modern software engineering in a business context
our solution is to present a big picture of software business backed up with practical detail avoiding academic or heavyweight process documentation
that covers people issues, business strategy, portfolio, programme and project management as well as architecture, technical delivery and integration
unlike simple small team based processes such as RUP, Scrum etc.
The big picture of software engineering

Holistic Software Engineering

And if it’s too big, we’ve got a small picture, which is essentially “normal” agile development.

Please share HSE with your friends and colleagues.

No more Project Managers, bring in the Movie Producers

I was reading some course material recently that was trying to teach people something to do with software development and it was using the same old tired “ATM machine” example. I’ve worked with hundreds of projects, many in the finance sector and none of them are anything like an ATM machine.  One of the reasons that example is soooo tired is that it’s describing commoditised software development, it’s something I’d expect to buy off a shelf or go to a specialised vendor for. It’s not something I’d put my team of uber |33t  haxorz on.

Developing software products is a balance between a creative and scientific pursuit, so it’s a little hard to describe, especially when that software can range from a tiny smartphone app, to a website, to router firmware, to an enterprise hr system (urgh!), to a big data processing system of systems etc. You get the gist.

The things these types of system have in common with each other is how different they are. And yet the traditional approach to managing these diverse kinds of work has been classical project management with a one size fits all process (I’m looking at Prince2, RUP, Iterative, Agile etc.) or worse hiring a large company full of body-shop project managers. For me this is one of the root causes behind large scale IT disasters.

I once had a discussion with the leader of a PMO in a large organisation of thousands of people about the nature of Project Management for software and he assured me that “someone who knows how to manage a bit of work can manage software, it’s just a different type of work” and indeed I should “stop thinking of software as special”. I’ve seen this attitude resonate in many organisations and is to me best summed up by the same Head of PMOs statement “from a project management point of view building software is the same as building a bridge”.

Now then. I know a little about software development, but not that much about bridge building (if you exclude my illustrious Lego engineer career when I was 7). If I managed the building of a bridge next door to one build by someone with a track record of bridge building who’s bridge would you drive your family on? Not mine if you’ve got any sense.

There are so many problems in the software development industry caused by people not understanding it and applying bad metaphors. When people who do understand it start to get the point across (e.g. the early agile movement) they often get misunderstood by the old guard who dress up their normal ways of working in new language.

Many of our “best” metaphors come from manufacturing lines where the same thing is made over and over again. That’s nothing like software.

To me a software project is more similar to making a movie than anything else:

  • It’s a unique one off product (or a remake of a previous version) with new parts working together in new ways
  • Once made we’ll often duplicate and digitally distribute
  • It can cost a lot of money because it can be very complex and needs lots of specialist skills
  • You need a high level plan for how you’re going to make it
  • There’s no guarantee on return
  • What’s written down originally bares little resemblance to the finished product
  • We know when it’s good or bad but it’s hard to objectively quantify
  • There’s lots of different types that need different collections of specialist skills
  • Both involve people and time and so are subject to change, they have to be adaptive
  • You can tell when the people making it had fun
  • It’s not feasible that any team member can fit in any role
  • There’s almost always going to be some quality problems
  • You wouldn’t get a movie maker to build your bridge
  • You wouldn’t get a bridge builder to make your movie
  • You don’t make a movie faster by telling the actors to act faster

So, I don’t want a Project Managers that know how to work a gannt chart. I want movie producers that know how to work with a team holistically to get the best out of them both technically and artistically.

 

Follow

Get every new post delivered to your Inbox.

Join 392 other followers

%d bloggers like this: