Software is increasingly important to everyone, it’s everywhere. It’s in our phones, runs our cars, our cities, our healthcare, entertainment and utilities. There are few businesses without a software element and many that are critically software dependent. What these organisations are finally beginning to understand is that the business of doing software is difficult, in fact it’s complex. It needs C-suite level direction and coverage.
Being good at software isn’t a matter of hiring the good coders, buying the right workflow management tool, using the right Configuration Management system, hiring the right management consultants or using the correct technology. These things all have a part to play but they’re supporting parts to the most important factor: culture.
Software Development is a team activity and the team is bigger than we used to think it was (a few developers, testers, direct customers, managers, analysts etc.). A good set of development practices at team level isn’t enough to be effective because software development is a business enabling activity and so it touches all parts of an organisation (development teams, hr, procurement, security, business change, legal, financial, business services, operations, support etc.).
As organisations evolve to make the maximum use of technology they are making increasingly complex software products, often through diverse technology stacks, using a variety of in-, near-, and out- sourcing partners. Delivering systems of systems across teams of teams is not simple and so organisations are now looking to embrace the importance of software, technology and ways of working to their businesses at board level.
What is a Chief Software Architect?
Whether a “Chief of Software“, a “Chief Software Officer“, a “Chief Digital Officer” or “Chief Architect” the role tends to include:
- Leading and encouraging a positive software development culture
- Where delivery of business value is the primary measure of success and decisions are based on business value
- Where failing fast is a cause for celebration not a career ending issue
- Where software is designed for users/customers needs
- Where continuous/iterative development integration and deployment are the norm
- Where teams collaborate and co-create with each other and their customers to produce the best products
- Where thinking all day to write a perfect algorithm is worth more than pushing out hundreds of lines of code quickly
- Where professionals are trusted to do their jobs properly
- Where organisational impediments to smooth software development are removed, from board level if necessary (due to Conway’s Law transformational change is often impossible without such top-level leadership)
- Providing top level architectural direction and governance (sometimes called Enterprise Architecture) that strikes the balance between directing technical decisions towards alignment but not imposing restrictive unchanging standards on development teams
- Architecture is understood cohesively at Enterprise, Solution and System levels
- Architecture exists in the form of directed (but community driven) standards, off-the-shelf mechanisms and principles
- Architecture exists primarily as executable software although appropriate weight documentation, models, examples and usage guides describe it
- Architecture provides a context for making technical decisions in normal development, makes technical decisions on commoditized platform/software/operations choices but does not constrain innovation and research (architecture has a different impact in different parts of a hybrid dynamic model)
- Architecture enables control of whole-lifecycle costs by smoothing the flow of work from innovation and research into mainstream development and support
- Ensuring that teams can use the tools they want and need to use, not those that had the best sales pitch to a non-technical board (hint: these are normally small, simple and/or open source, not those provided by large vendors) balanced by creating a cohesive community that isn’t unnecessarily fragmented by every team needing to set up different tools environments
- Tool chains take the pain out of development, testing, build and deployment – automated wherever possible
- Development environments are appropriate for actually developing, building, testing and deploying
- Ensuring, and embodying, the principles of agility, devops, and agility-at-scale at the organisational level – promoting the evolution of working practices, operating models and ways of working while avoiding endless management fads
- Building and championing the needs of the development community, attending to their needs
- Ensuring that portfolio selection takes into account technical issues, shaping it in line with architecture and in turn using business priorities to shape architectural choices
At a business level Architecture exists to guide organisations through the business, information, process and technology changes necessary to execute the organisation’s strategy. Because “architecture” has always been more than just the technology bit, it also focusses on the non-functional aspects necessary to make the technology work terming a “Chief Software Officer” an Architect is an organisation recognising that to be successful at software a holistic approach is required that combines business concerns with technical concerns.
The idea of an “Architect” role with such a broad focus is analogous to that of a structural architect concerned with buildings. Indeed in 25 BCE Vitruvius (a well known Roman writer and architect who inspired Leonardo Da Vinci, hence the Vitruvian Man ) described an architect as:
The ideal architect should be a man of letters, a mathematician, familiar with historical studies, a diligent of philosophy, acquainted with music, not ignorant of medicine, learned in the responses of jurisconsults, familiar with astronomy and astronomical calculations.
Virtuvius is famous for stating that architecture must be solid, useful, beautiful (De architectura). The same three qualities relate to software architecture. Architecture is a fine balance between a subtle science and an exact art – as such it is the magic sauce that turns a technology strategy into business benefits.
If your business is reliant on technology and/or software and you don’t have a Chief Architecture, you should promote/get one before you become technically irrelevant, encumbered by lumbering traditional delivery approaches based on manufacturing and negative development cultures.
It’s time for businesses to recognise the important of software, and architecture, before they’re outmanoeuvred by those who have.
For more on Architecture see:
- An overview of architectural purpose and concerns, including detail on different levels and approaches: Holistic Architecture
- Architectural Profiling
- Agile Architecture
- What makes “good” software architecture or design
- Intentional vs. Emergent Architecture
- How to write a good strategy
- Hybrid Dynamic Model
HSD stands for Holistic Software Development which is a process framework that extends the principles of agile philosophy throughout the entire software development enterprise. Similar to “agile at scale” ideas HSD goes a few steps further integrating software development concerns from business strategy to continuous development, structural organisation to governance on the H-Model of software development.
HSD provides the “big picture” of software development providing a glue that combines other techniques and practices. It doesn’t replace what you currently do (e.g. Scrum, XP, Lean, Prince2, whatever) instead it provides a simple way to join them all up together, without introducing any new TLAs (Three letter acronyms)!
HSD is free to use, free to access and free to implement!
Many teams I’ve worked with struggle to make sense of their huge backlogs of stories, struggle to explain the scope of a release and current work – and most importantly, struggle to explain how the stories they’re working on contribute to business value!
All of these problems are killers when it comes to joining up what a development team is doing to a corporate portfolio, governance and milestones. Features and business benefits seem to be too high level, stories seem to be to low level.
A common approach to trying to solve these problems is to use “epics”. Originally intended to just be another type of story (one that was too big for a sprint) “epics” are now commonly used in a requirements hierarchy by teams and organizations. I’ve seen loads of backlogs where every story had an epic “parent” – sometimes even a 1-1 mapping from epics to stories(!!!!). I know that some process frameworks recommend this approach, but I don’t and here’s why:
Stories can just be on a backlog
Stories don’t have to have a parent. A backlog made up of stories is just great, it doesn’t need a hierarchy that’s a Work Breakdown Structure using some agile-y sounding words to hide the fact. A backlog isn’t a gantt chart, stop trying to turn it into one by stealth!
Actual epics are useful for teams
“Epics” in their original form were simply stories too big for a sprint, sometimes that just creeps up on you so it’s useful to be able to tear up a big story and replace it with some smaller ones. By formalizing a Feature -> Epic -> Story hierarchy as an “agile at scale” process I’ve seen corporate processes become protectionist over epics, trying to use them to fix scope, counting and metricating them to the extent that teams can’t simply discover some of their stories are epics.
Bigger stories need more than a sentence
I know that a story can have more elaboration than the standard form of “as a user I want to…” but few stories in the wild have more than a sentence (you’re often lucky if you find some with acceptance criteria). I find that describing how various stories contribute together (especially when across several teams and/or systems) to deliver business value I need more than a sentence. I often need a diagram or a paragraph or two.
“Epics” sound silly
What’s in a name? Would a rose by any other name smell so sweet? Probably. Names shouldn’t be that important and yet they are, what we call things has a significant impact on how people represent them. “Epics” are part of the money spinning agile movement’s language. They either sound silly to businesses or as a way to pretend they’re doing “agile at scale” by adopting the term for their portfolio requests.
I like Integration Scenarios more
Ok, so this is clearly subjective and personal but that’s kinda my point. I don’t use process elements because someone told me to, and I don’t think that something like Integration Scenarios are always better than epics. I prefer to pick and choose different tools for different jobs. In my experience Integration scenarios are frequently a better tool for scoping, elaborating and agreeing end-to-end threads across a system or system-of-systems so they’re one of my “go to” tools.
So what are Integration Scenarios?
Integration Scenarios describe an end to end workflow across a Product or Product Family. Integration scenarios are threads across a number of low level requirements such as user stories that collectively deliver Business Value. They are ideally suited to describing the scope of Releases and providing the input for integration testing necessary to achieve higher levels of acceptance.
Integration Scenarios provide the context for discussions around User Stories
Integration Scenarios can be described in a number of ways, ranging from textual to graphical depending on the nature of the system and the best way to achieve clarity. In all of these examples we recommend using a meaningful title, not “Integration Scenario”!
One of our favorite ways is to draw an activity diagram of user interactions using GUI mockups in the nodes with references to features/use cases/stories as appropriate. We do not recommend strictly following a diagram format but using the visual or textual elements that will communicate the scenario as effectively as possible. Story-boarding can also be an effective way of describing Integration Scenarios.
When using a graphical view we recommend including links to stories or systems that are covered by the scenario and adding rich graphics to standard modeling notation to increase communication.
Integration Scenarios using textual formats are best described as a narrative with links to contributing stories or systems.
They should ideally be no more than a couple of paragraphs that describe interactions between actors across a number of stories. Sometimes adding extra alternative or exceptional flows can be useful, in the style popularized by use cases, but if there’s a lot of “flow logic” involved then we recommend using a diagrammatic form (e.g. workflow) to avoid convoluted textual documents. Similarly adding a non-functional section indicating non-functional requirements or constraints that apply across the stories making up the scenario can be useful.
Integration Scenarios describe the dynamics between stories and how they collectively contribute to delivering business value. To that end we recommend highlighting and directly linking to the stories that make up the integration scenario. In this way Integration Scenarios add context to the mass of user stories and provide a sound basis for scoping a system, describing a release, functional architectural mechanisms, integration testing, and product acceptance.
Regardless of format we recommend that Integration Scenarios are described from the user perspective and focus on achieving business value. The choice of formats should be driven by the type of Product being developed (e.g. graphical/user facing vs. back end service) and further influenced by team experience and available tooling. We strongly advise not being concerned with the constraints imposed by formal modeling languages, the communication is important not the notation.
Sometimes we wish to use Integration Scenarios as part of diagrams, even workflow/activity diagrams (e.g. for Value Stream Analysis) in which case the full text version might be a bit big. In these situations we recommend using a “zoomed out” view which just lists a title and the related stories. In fact this “elided” form of an Integration Scenario is sometimes useful for simply gathering up a collection of stories into a larger chunk for planning purposes. Story lists in these diagrams don’t have to be exhaustive, just indicative.
Textual Integration Scenarios can be crafted in and decent rich text editor that allows for hyperlinks. We recommend trying to keep them short since the detail of the requirements should be in lower level stories or UX designs/storyboards. Instead the Integration Scenario focuses on the dynamics between requirements. A good rule of thumb is that they should be only a couple of paragraphs, and at their biggest if alternates/exceptions and nonfunctionals are included they should only be 2 or 3 pages. Creating a large number of fully documented Integration Scenarios prior to any development effort is a very bad idea as it will introduce the problems inherent in waterfall development, however elaborating the Integration Scenarios for a Release prior to development iterations/sprints or continuous flow development (with customer representation and team representation) is an excellent way to align collective understanding of scope and requirements.
Each Release should have no more than a couple of Integration Scenarios in it, if there’s more and they are not too interdependent then they can be delivered in separate Releases enabling earlier feedback cycles, (even if they are internal Releases rather than operationally deployed customer release.
Links to other elements
Integration Scenarios provide a mid-level requirement type, higher than User Stories but lower than Portfolio Requests containing a lot more information than a Feature. As a result Integration Scenarios provide a unit of incremental business value which can be used for Release Planning, tying in milestones for programme/portfolio tracking and governance.
Because Integration Scenarios are constructed in a narrative form that describes a scenario, tried and tested scenario based testing methods can be applied for Integration testing.
Integration Scenarios also provide a useful starting point for understanding Solution Architecture, in the case of Product Families or systems of systems, and even System Architecture in terms of indicating structure and identification of mechanisms. It is possible to resonate architectural structure with Integration Scenario structure although there is a risk of inappropriately introducing a Homomorphic force to the architecture.
By aligning Integration Scenarios with Integration Streams, Releases and Milestones we can focus the various views of Portfolio Management, Programme Management, Project Management, Architecture, Delivery, Quality, Deployment, Acceptance and and Business Change onto Releases at the center of the H-Model de-conflicting business management and technical delivery. Providing a context for requirements, planning and delivery Integration Stories are a common currency for the entire IT organization to work with.
Integration Scenarios can also be used as part of Integration Modeling (creating scope diagrams) to indicate which actors/systems sets of requirements interact with, helping to clarify overall scope and identify potential gaps in scope.
If using Integration Modeling we recommend color coding Integration Scenarios based on priority (e.g. Musts, Shoulds etc.).
A Scoping Diagram is used to draw a picture of the primary requirements and users/interacting systems.
What are your experiences of “Epics”, “Integration Scenarios”, “Scoping Use Cases” or other alternatives?
An embarrassment of Project Managers
An impasse of Architects
A confusion of Business Analysts
A mob of Developers
A silo of Testers
A brethren of Scrum Coaches
A waste of Lean Consultants
A conspiracy of Process Improvement Consultants
People are important, not roles! But recently I saw a weekly team meeting for a project that had 12 project managers! I thought it was a joke at first.
Can you think of any others?
The heart of agile software development is feedback loops. Doing a bit of software, looking at it, at how we did it and then improving things as we do another bit of software. The “things” that can be improved can be quality, scope, usability, performance etc. etc. Perhaps most importantly, relationships and ways of working can also be improved.
This rapid feedback cycle ensures that product development is aligned to customer needs, that progress against guestimates is understood (so that plans can be refined) and that teams which include customers find problems early, and so can fix them, as quickly as possible. It’s a quicker, better, cheaper and happier way of working.
What about that needs to scale? Empirical feedback is just fine at any scale. Across 100s of teams or teams-of-teams and entire organisations short term feedback cycles work perfectly well.
The problem – Portfolio Management?
The problem is that there’s other stuff that goes on too. Product Development in big or complex organisations is part of an investment portfolio and costs a lot of money. So people want to track it and check that work that they’re paying for is delivering, and is aligned to strategy.
The answer… short feedback cycles. Feedback cycles let you measure what you’re spending and you can go and have a look at the output to check it’s delivering business value.
The problem – Programmes?
Sometimes software solutions aren’t simple products or mashups. Sometimes they’re big/complex combinations of things developed internally, externally or collaboratively with partners. In these cases we sometimes need to align teams towards common scope and architecture so that products can work together to deliver business value. Collections of Projects working together = Programmes.
The answer… short feedback cycles. Feedback cycles at product level work as described above. At Product Family level contributing product releases can be brought together to form a Product Family Release – as often as possible to get feedback working. (Hint: 10 weeks is too long, especially if integration fails because 20 is definitely too long). Continuous, event-driven integration streams enable flexible and recursive short feedback cycles.
The problem – Support Costs?
If teams are chasing the latest shiny technology all the time then a large organisation can quickly end up in a support hell of mixed platforms, technologies, middlewares, deployment stacks, licensing and support costs. Needing to recruit specialists in everything and duplicating operational environments.
This one can be partly solved by moving the external complexity to external clouds (where the diversity is embraced and normal) and also by (lightweight!) Enterprise Architecture. Lightweight Enterprise Architecture makes technology choices for the organisation (for it’s mainstream development, it shouldn’t constrain research & innovation) to prevent this fragmentation in the first place.
This can also be partly solved by getting IT Operations stakeholders involved early and often as part of the holistic team (DevOps) along with the other stakeholder sets.
Of course it needs checking to make sure it’s still a good idea and that it’s aligned to strategy. Probably should do that frequently and empirically… short feedback cycles.
Don’t scale the practices, just be agile!
Often proponents of “Agile at scale” or “scaled Agile” will talk about scaling Scrum (or similar). Using practices that work really well at team iteration level to run a business strategy, or a team of teams of teams (at programme level). In my experience that’s not a great idea, just because you’ve got a hammer doesn’t mean every problem in an organisation is a nail.
Staged daily standups or scrum-of-scrums are just painful to behold. Execs don’t want a strategy backlog, nor do organisations want architecture to emerge with no regard for support costs, recruitment or cross-project dependencies.
If I was invited to a 2 day planning session I’d probably walk out of the door never to return. I’d rather watch animated musicals on repeat.
We don’t need to scale agile, we just need to be agile… with a little “a”. Short feedback cycles, a focus on people and relationships with tight customer collaboration. Open, transparent and empirical collaboration between people – that’s the right approach. One that work at every level of an organisation.
Why not StratDev, BusOps, etc.?
The term “DevOps” refers to tight integration between the Software Development and Operations parts of an organization. As I mentioned in my article on Conway’s Law the reinforcement of there being two separate parts “Dev” and “Ops” causes a separation in the way we think about these problems and then how we interact within an organisation. If the industry truly believed in DevOps then “dev” and “ops” would no longer exist. I realise that some people intend “DevOps” to be a verb, but it sounds like a noun.
Of course the idea behind of DevOps, of bringing together stakeholders involved in creating, hosting, running and maintaining is a great idea – focussing on these things can make it easier to deploy products and get user feedback. However we might just end up building the wrong products quickly. We must make sure that the things we build are aligned to strategy and deliver business value. To do that we need a holistic approach, that involves all stakeholders (not just the dev and ops folks).
Operations stakeholders should be involved in the Portfolio Build/Selection practice to ensure that the impact of new product development or maintenance decisions on current Operational positions are understood. Significant changes to current Operational positions are raised as Portfolio Requests so they can be properly costed and scoped.
Work that is approached holistically must by definition include addressing the needs of all stakeholders from any part of the business. If a divide exists between parts of the business then the dysfunctional relationship needs addressing rather than process and tooling being introduced. Processes and Tools are helpful to gain agreement on a common approach and automate ways of working but not to solve dysfunctional relationships.
HSE promotes regular builds and releases in either iterative/agile or continuous flow models Operations stakeholders must be involved early and often in software development product delivery as a product is likely to be delivered frequently, or even continuously, and so will need transitioning activities performed repeatedly.
So, I like the idea of joining up dev and ops stakeholders and work. But I only like it in the context of also joining up business strategy, requirements, architecture and quality with dev and ops – all flowing business value to customers. On it’s own, DevOps isn’t enough and worse could just be a management fad.
How do we join up business strategy to agile development? Is program management relevant? Where do project managers fit in? What about architecture?
Holistic Software Engineering (HSE) answers all of these questions – for free.
Agile and continuous flow are great for small teams or a small number of inter-related small teams working on exploratory or maintenance work. But what if we’re spending 100s of millions on an IT strategy of inter-related products that need to work together to deliver business value. What is business value anyway?
To answer these questions (and more) my friend Steve Handy and I have distilled our collective 30+ years of software experience in a single, cohesive model of software development. We’ve developed the H model that moves on from the v-model and it’s siblings by combining:
- Business strategy
- People and team issues
- Iterative and feedback loops
- Lightweight requirements and architecture
- Lean portfolio and program management
- Agile and continuous product delivery
- A focus on integration, quality and business value
…all elegantly combined and de-conflicted by release planning.
We’ve not invented very much, we’ve simply put a lot of good ideas from others together into a cohesive framework. We’ve drawn it all as a big picture and developed a website that shows how to get value from all of this stuff. Everything is clickable, everything has content.
The best bit: it’s free! There’s no paywall, there’s no private “full” version, you can just use it or not as you like.
We don’t believe in process zealotry, or putting academic concerns above clarity and usefulness. HSE is indicative, not prescriptive. You’re already doing it and if you use the big picture to draw red blobs on the bits that aren’t working well, or missing, in your organisation then you can use the model to make tangible improvements – immediately.
Using HSE doesn’t replace any of your existing processes, it provides the glue that joins them all up together in one simple, elegant and cohesive model.
And if it’s too big, we’ve got a small picture, which is essentially “normal” agile development.
Please share HSE with your friends and colleagues.