Intentional vs. Emergent Architecture

I’ve been thinking about architecture a lot recently but one thing that I often discuss but have never blogged about for some odd reason is intentional vs. emergent software architecture. Some old fashioned software methods such as waterfall led people into doing a lot of up front architecture work, they analysed and designed away for ages producing huge reams of UML and documentation that no one could ever squeeze into their heads if they had the patience to read it. This is an example of an intentional architecture – the architecture was intended, planned and deliberate.

Lots of folks hated that way of doing things as it meant people were writing docs and drawing diagrams instead of making working software, not to mention an all to frequent tendency to over-engineer architecture past the point of usefulness. This led to some people saying that we’re better off not trying to do any architecture and just letting it emerge from the work we do developing small little customer focused requirements (like user stories or similar).  Ok, so there’d be some rework along the way as we encounter a deeper understanding of the system and refactor our emergent architecture but it’d still be better than the old way of doing large upfront architecture.

So, there seem to be two opposed viewpoints: intentional architecture is best, emergent architecture is best.

For me, neither is true. I’ve seen some really terrible examples of badly over-engineered architectures that crippled a project and projects that never got past their over-reaching architectural analysis. Equally I’ve seen products with emergent architecture that had to be entirely re-architected as time went pay because their emergent architecture was so wrong it was comical (imagine a software management tool that only supports a single project, and that concept being deeply embedded in the architecture).

There’s a scale with intentional architecture on one side and emergent architecture on the other.

Various factors might push us one way or another… The second I listed on the right is interesting as if you’ve got a well known technology and problem domain you can get away with emergent architecture, but similarly if you have a totally unknown technology and problem domain it can be very effective to evolve towards a solution and architecture rather than crystal ball gaze by creating a (probably wrong) intentional architecture.

Which rather sums up the point I’m trying to make. The purpose of architecture is to shape the solution and address technical risks. Solving the big problems, creating common ways of doing something (lightweight architectural mechanisms) are all good architectural goals but only if we’re sure of the solution. If we’re not we’re better off evolving an emergent architecture, at least initially.

I think that the extremes at either end of the scale, as with most extremes, are a bad idea at best and impossible at worst. If you gather a group of people together and tell them to create a web app given a backlog but they’re not allowed to think or communicate about the architecture up front then you’ll find they all start dividing the problem in different ways and writing in different languages for different server and client frameworks. Hardly a good idea. Of course on the other end of the scale, believing that we can foresee all of the technical issues, all of the technology and requirements changes that might happen is as likely as a 12 month project plan Gantt chart being correct after a few levels of cumulative error margin have been combined.

For more on architecture see:

• Architectural profiling
• Architecture sketching
• Lightweight Mechanisms
• What is enough agile architecture?

Scaled Agility: The Project Forum

Name: Project Forum (Middle-out management structure) – Agile at Scale practice

When it might be appropriate

• In situations where multiple competing stakeholder groups with different agendas are required to work together
• In situations where multiple product groups need to collaborate on a bigger outcome
• Where there is a conflict in direction, resource management/ownership or scope between collaborating groups
• System of systems development

What is it?

The Project Forum is an application of agile philosophy to large project structures. Rather than impose a hierarchy of decision making from the Project Manager downwards the Project Forum is a virtual team in the middle of all stakeholders.

The Project Forum is a self-organising democratic group that balances competing voices and concerns, owns high level scope and architecture, runs the high level release train and performs integration activities for the product.

Use of the Project Forum practice does not prevent any communication directly between contributing groups it only provides a vehicle for that conversation when it’s relevant for the wider project.

The Project Forum practice is an example of Agile at Scale combining social business practices, technical software practices and ways of working to make a simple way of doing big complicated bits of work.

Simple software project measures

I’m not a big fan of metrics, measures, charts, reporting and data collection. I’m not terribly impressed by dashboards with 20 little graphs on showing loads of detailed information. When I’m involved in projects I want to know 3 simple things:

• How quick are we doing stuff?
• Are we on track or not?
• Is the stuff good enough quality?

There can be some deep science behind the answers to those questions but at the surface that’s all I want to see.

Organisations need to know that teams are delivering quality products at the right pace to fit the business need. To achieve this goal teams need to be able to demonstrate that their product is of sufficient quality and that they can commit to delivering the required scope within the business time scales. If the project goal may not be achieved then the business or the team need to change something (such as scope, resources or time scales). This feedback mechanism and the open transparent communication of this knowledge is key to the success of agile delivery.

The goal of delivering quality products at the right pace can be measured in many complex ways however, when designing the Project Forum agile at scale practice we looked at just 3 measures. In fact I should probably call them 2.5 measures as the throughput/release burnup can be considered mutually exclusive (if you’re continuous flow or iterative). The most important measure is people’s opinions when you go and talk to your team.

Note: in the measures section I often refer to “requirements” as a simple number, this could be a count, a normalised count, magnitude, points, etc. it doesn’t matter what’s used so long as it’s consistent.

How to avoid Fragile Agile, Flexibility in Context

A presentation I gave at the UK RUG Annual Meeting on what can make agile development fragile and how to avoid that fragility.

Unfortunately animations don’t work and slideshare’s screwed up the agenda slides but you can still follow it

EAC = Estimated cost at Completion

One of the figures often included on project reports, and present in many PM tools, is the EAC and EAC variance. EAC stands for “Estimated Cost at Completion” and is equal to the actual cost of work performed so far + the remaining estimate to complete. Expressed as a formula of confusing ‘initialisms‘ this means that EAC = ACWP + ETC.

For projects  the EAC represents the total projected cost of the project. At the very beginning of a project when there hasn’t been any spend,  the EAC is equal to the Estimate to Complete (ETC) and this is the first project baseline. As a project executes changes happen and projects may start over-spending or under-spending. In these case the EAC will start to vary from the baselined value as the actual costs + the remaining estimates are either up or down from the original baseline. This variance is expressed as EAC variance. A negative EAC variance indicates a project that looks like it is going to come in under the original budget baseline figure. A positive EAC variance indicates that a project is likely to overspend relative to the baseline figure.

EAC may be expressed as a financial figure or as an effort (in hours) value. EAC can also be combined with agile burnup measures across projects.

Knowledge Centered Support in my project

Kelly Drahzal recently published this great presentation on Knowledge Centered Support which made me think a bit on the nature of support mechanisms. I’m currently engaged in rolling out a large and complex enterprise tool (Rational Portfolio Manager) and associated governance, portfolio management and project management practices in a large and complex client.

One of the things we need to do to get these pracitices and the tool embedded in an organisation is manage support. Our support takes two forms, tools support and process support. Normally when a person thinks they’re asking for one of them they’re actually asking for the other One of the interesting things about the support that my rollout team provides to the practitioners is that ultimately it’s a transient function – we won’t be the long term support team on this product, in fact support will be handed over to the centralised support function and the rollout team (comprised of external consultants (some IJIers, an IBMer and some independents) and contractors) will dissapate into the ether from whence it came. So obviously, as per Kelly’s presentation we’re very keen on knowledge centered support – we don’t want to waste our time, effort and brain power by re-recreating the answers to people’s problems.

So what do we actually do to try and avoid some of these problems and do some knowledge based support? We’re a transient support function so we don’t have and super tools or even specialist knowledge base management skills. What we do have is a highly skilled team and a number of communication channels.

We capture all support requests in a humble excel spreadsheet, regardless of their communication channel and categorise the requests into a number of categories. (Of course this gave me an excuse to write some cunning macros to keep everything updated automatically).

As well as providing lovely graphs the spreadsheet captures the issue and the response. As a result the team can all see who had what problem and how it was resolved. As problems are solved knowledge is created, capturing it in a spreadsheet is all well and good, and can be searched on by the support team but it’s not great in terms of sharing that knowledge broadly. (As it happens the support spreadsheet is publicly accessible via a guest account on our config management repository – but that doesn’t mean anyone is looking!)

To share the knowledge we communicate it through many channels. Sometimes it’s apparent that our education has been lacking some good guidance so we update the education programme (training courses, open surgeries, lunch ‘n’ learns). We have a wiki where we can post new bits of information, a message board/forum, emailing lists, laminated desk drops, a FAQ on the wiki and also some mentoring guides. One of the functions of our team is to mentor practitioners in the adoption of practices and tools and to do that we have a number of mentoring packages that we give to adopting teams. Ensuring that the mentors are all saying the same thing, giving the same solution to the same problem is important. One of the best ways of doing this it to get the mentors together to talk to each other, run through scenarios and gain consensus on the common answers. We also document these scenarios, sometimes in the practitioner facing User Guide and sometimes through mentor guides.

What is Governance?

I’ve recently been writing an article on Governance in IT organisations and thought I’d publish the introduction here. The article goes on to discuss Strategic Frameworks and Processes, Governance Functions, Roles and Rights, Data Analysis and Remediation Mechanisms and will probably be made public sooner or later when I finish writing it!

---

“Governance” is defined as the means by which the leading authority guides and monitors the values and goals of an organization.

Because this definition uses some very broad terms such as “leading authority” and “organization” it is possible to apply the concept of governance recursively in IT organizations. There is governance in teams, projects, programmes, portfolios, departments and any other organizational unit.

Governance strategies are most effective when they are designed and explicit. In the absence of cohesive integrated governance between these various organizational layers the values and goals of an organization can become emergent due to the lack of a clear connection between the executive strategy and the work being executed at the coal face. This situation is typified by project practitioners being unaware of how their work relates and contributes to the global strategy and layers in between.

The “means” referred to in the definition above refer to practices, processes, procedures, standards, rules and criteria related to the execution of business decisions executed by real people within their tool environments.

Effective governance relies on known standards for strategic management, programme and portfolio management as well as project management and execution. This enables individuals in an organization to make the right decisions at the right time based on accurate data. Governance is often described as a mechanism for empowering individuals with the rights they should have and a clear delineation and integration of these rights from the rights exercised by other roles. Because governance strategies can involve a lot of data process management based on data analysis there is a strong case for automating parts of a governance strategy with tools.

---

Thoughts?

RSDC 2008 Day 2: Jazz Planning, Governance, Steak

No big product announcements today but a lot of deeper dives into each of the new tools I blogged about before. It’s my mission to grok Jazz and so I’m looking forward to a meeting with some of the Jazz team tomorrow after Sir Shat does the keynote. It’s great to have William Shatner as a warm up guy for my two talks tomorrow

There are 6 new Jazz based tools, significant updates to five existing Rational tools that add Jazz capabilities and “plans for certified add-in products from 11 Rational partners”.

One particular session that stood out for me today was on agile planning with Rational Team Concert. There were lots of questions around planning details but what interested me were the implications outside of projects. A lot of the problems that I see in organisations are caused by issues with strategic management, linkage to tactical management, governance and portfolio & programme management. You can get really good at doing projects but if they’re the wrong projects and can’t be resourced decently there’s little point. With that in mind I think I may raise some change requests as work items at jazz.net since little issues like not being able to assign a work item to a pair can become big issues when aggregated up across projects to see where the time/money is being spent.

Today we gave out around 3000 IvarMan masks to the RSDC attendees in keeping with the superhero theme this year:

At the end of the day we got a bunch of foam missiles from another stand at the exhibition centre and had a small war with the bar staff opposite our stand It was a long day and a quiet 10 minutes

I’m taking some of these things home for my eldest son! Finally to finish off the night we went to Shula’s most excellent steak restaurant. We declined to try and eat the 48 ounce steak and settled for some more reasonable chunks of perfectly cooked meat. Now it’s time for an early night in preparation for William Shatner’s guest speaking slot and my two talks tomorrow.

• PPM 14: Case Study: CUPID – Implementing the IBM(R) Rational Unified Process(R) at PricewaterhouseCoopers – 11:15 am – 12:45 pm
• PPM27: Case Study: Large-Scale Portfolio Management Experiences at PricewaterhouseCoopers – 4:15 pm – 5:45 pm

RSDC 2008 Day 1: Jazz, Sushi, Wallflowers

Today was day 1 of the RSDC 2008. I’ve already been busy for a couple of days but today was the proper first day. Unfortunately I spent the beginning a little hung over. I turned up to the keynote presentation at 8am local time with a bit of a headache – after watching the acrobatics and suffering the loud noise I had an even worse headache – oops With only myself to blame I soldiered on to experience many cool things today:

Lots of Jazz stuff

Today IBM Rational officially announced both Rational Requirements Composer and Rational Quality Manager. Also Rational Team Concert has got a lot of press. I blogged a while ago about RRC and RQM (here) so it was good to finally see these products.

Rational Composer is especially interesting to me as it a new Jazz based tool that allows you to manage requirements, create process flow diagrams, GUI mock-ups, manage glossary terms, create traceability and even create Use Case Docs in a single environment. Of course there is integration with Requisite Pro but it seems clear to me that in the long-ish term the products will converge until ReqPro is no longer necessary. I’m looking forward to the GA release of Requirements Composer, and especially the future release that is fully Jazz enabled, making use of Jazz SCM for requirements management and integrating deeply into the other Jazz products. I said previously on my blog that Reqiurements Composer was due for open beta today, but the word is that it will be next week! I’m impressed by this product and I think it will really help people that want to elicit requirements – all the way from organisational business analysts to project business analysts and system analysts. It’s a cool product.

I also saw today the Microsoft Visual Studio client for Rational Team Concert. As a .Net developer myself and someone that works with clients that have heterogeneous environments this is particularly important to me. So far there are some VS native windows for Jazz views such as Team Artifacts and Work Items but I’ve not really seen what the relationship to TFS/VSTS is. Also at the moment some of the views (particularly of work item details) are through the web interface in a html pane inside Visual Studio rather than in a native VS plugin interface – although I’m assured this is to come soon. It’s good to see IBM Rational focussing on integrating with other platform and vendors. As most users would probably say though – I want it now!

Other interesting elements included the news about Rational Project Management and Rational Enterprise Reporting, not to mention the new governance tools from IBM Research including

• IBM Rational Financier – gives project and program managers insight into the financial value of one ore more projects to help identify and manage risks
• IBM Rational Governor – helps IT organisations manage project roles and associates decision rights including managing the polices that constrain decisions and promote compliance with processes
• IBM Rational Tempo – lets project managers understand and mange the variability of schedule overruns, a key source of risk in software development projects
• IBM Rational Ensemble – reduces risks incurred by communication failures by promoting communication between developers doing related work

All of the above are native Jazz tools and indicate the bright shiny new Jazz future. It seems obvious to me that these tools and will be the future path in the long term for Rational Portfolio Manager

Rational Team Concert will be the first generally available product and will be released towards the end of this month – here’s a screenshot of IJI EssWork in Team Concert just because I’ve been playing with it:

(cliccy piccy)

Many other things caught my eye today – including an excellent presentation on RPM by Scott Craig that happened to include some ideas that I think will seed solutions for some of the problems faced by my own RPM implementation at the moment at my favourite client.

I also liked the look of IBM Rational Self Check as it’s a good tool for supporting what I and my team have been doing manually for years. In fact the Self Check screenshots look remarkably like the measurements slides in my RUP implementation case study! If only this tool had been available 4 years ago! Rational Self Check is part of the focus on measuring practice adoption and the IBM Rational Measured Capability Improvement Framework (MCIF) – of course practice adoption and measurement is something that we know rather a lot about at IJI

The hat

image shamefully nicked from kelly

Met Kelly and Ferdy

It was cool to meet up with some folks who I’ve previously only interacted with online. I think we need to setup a group photo of us all

The Wallflowers gig

More gratuitous photo stealing from kelly:

Ian (Spence) had to point out to me that the lead singer (Jakob Dylan) was the son of Bob Dylan – which once he’d told me it was actually pretty obvious. Clearly I need to be more in touch! There were sparkly cups at the wallflowers gig too, obviously I grabbed a couple to take home for my boys (clicky piccy):

Sushi with the Wallflowers

After the exhibition centre and after the wallflowers gig we went over to Kimonos in the Swan for some Sushi and cocktails, the Wallflowers took the table next to us and it wasn’t long before our group and there’s were intermingled. They were kind enough, after getting hassled by Bryon Baker, to sign some autographs for Garth’s kids 8)

Bryon (and others) at karaoke

Speaking of Bryon, he did a funnygreat rendition of Coca Cabana with Gina, I got the vid on my phone so I’ll upload it and post it tomorrow

The ribbons

And here’s an updated pic of the copious number of ribbons I have this year. They’re a good conversation starter though