One powerful method that I use is value stream visualisation. There’s a number of ways of getting information about the value stream ranging from physically walking the chain to simply asking people.

Sounds simple doesn’t it? But one of the best sources of information about wasteful processes and ways of working in a team is simply asking the team what they feel they shouldn’t be spending time on. By freeing up people to actually do their job they can reduce waste and improve productivity. One method I use is to ask each team member to identify a list of n items that they have to spend time on that they think are either entirely or partially wasteful. Basically a list of their impediments. I then anonymize the list look for commonality and then the top few are a reasonable target for improvement.

Alternatively actually walking the path of a piece of work through an organisation can really open the eyes to the sheer number of people and unnecessary waiting involved in some wasteful processes…

Anyway, having got some data I find it useful to visualise it in terms of waiting time and doing time. In Lean-speak cycle time is the time it takes to actually do something, lead time is the time it takes from the request being made to it finally being delivered. In many tasks the lead time far outstrips the cycle time, visualising it can make that very clear.

For example, a task with has a lead time of 5 days might actually be made up of 2 days waiting time before it actually gets picked up and the cycle time of 3 days starts.

If you look at all of the tasks in your value chain you might find that many of them look more like this, with very large wasteful waiting times. There are many reasons why there might be a large waiting time, typically the team doing the tasks is over-subscribed, or a team might be protecting it’s SLAs, KPIs or other TLAs. These problems can have significant effects when looking at the system as a whole.

Even more interesting is looking at how these tasks might fit together in a value stream. If we imagine a business function that involves a request moving through a bunch of tasks across a number of teams that have different waiting and doing times a simple sequence of tasks, with a little buffering between them from the Project Manager to allow for schedule variance naturally, might end up looking like this:

Here we have 26 days of actual work spread over almost 60 days (12 weeks) of elapsed time = 40% waste. Even if the person planning the work is willing to accept some risk and try to optimise their workflow a bit without improving the waste inherent in the tasks involved there’s still a lot of waste in the system.

By visualising the value stream in this way we can see straight away (apologies to red/green colour blind folks) that there’s a lot of red, a lot of waste. In many cases planners aren’t willing to accept the risks inherent in overlapping activities as shown here, or aren’t even aware that they can leading to the more sequential path shown above. The result is a minimum time that it takes a request to get done, based on the impedence of the current value chain of, in this case, 38 days before we even start thinking about actually doing any work.

Surely that’s not right.

There are high numbers of design and architecture practices ranging from the purely transformational to the evolutionary and emergent (both manual and automated) but there seems to be some heuristic consistency between these various practices and the approaches of the traditional, agile and post-agile movements.

Initially I saw the simple descriptions of that makes for “good” design from from Ron Jeffries and Kent Beck in Xtreme Programming. I like the style and direction of this kind of description so here’s my take on the characteristics that make for “good” design or architecture:

• Intentional structure and behaviour
• Highly modular: consisting of separate services, components, classes, objects or modules
  • Elements are highly cohesive
  • Elements are loosely coupled
  • Elements have low algorithmic complexity
• Avoids duplication
• Well described elements: modular elements have simple expressive meaningful name enabling readers to quickly understand their purpose, similar to code cleanliness we should strive for design cleanliness
• Runs and passes all defined tests or acceptance criteria
• Lightweight documentation

Lots of folks hated that way of doing things as it meant people were writing docs and drawing diagrams instead of making working software, not to mention an all to frequent tendency to over-engineer architecture past the point of usefulness. This led to some people saying that we’re better off not trying to do any architecture and just letting it emerge from the work we do developing small little customer focused requirements (like user stories or similar).  Ok, so there’d be some rework along the way as we encounter a deeper understanding of the system and refactor our emergent architecture but it’d still be better than the old way of doing large upfront architecture.

So, there seem to be two opposed viewpoints: intentional architecture is best, emergent architecture is best.

For me, neither is true. I’ve seen some really terrible examples of badly over-engineered architectures that crippled a project and projects that never got past their over-reaching architectural analysis. Equally I’ve seen products with emergent architecture that had to be entirely re-architected as time went pay because their emergent architecture was so wrong it was comical (imagine a software management tool that only supports a single project, and that concept being deeply embedded in the architecture).

There’s a scale with intentional architecture on one side and emergent architecture on the other.

Various factors might push us one way or another… The second I listed on the right is interesting as if you’ve got a well known technology and problem domain you can get away with emergent architecture, but similarly if you have a totally unknown technology and problem domain it can be very effective to evolve towards a solution and architecture rather than crystal ball gaze by creating a (probably wrong) intentional architecture.

Which rather sums up the point I’m trying to make. The purpose of architecture is to shape the solution and address technical risks. Solving the big problems, creating common ways of doing something (lightweight architectural mechanisms) are all good architectural goals but only if we’re sure of the solution. If we’re not we’re better off evolving an emergent architecture, at least initially.

I think that the extremes at either end of the scale, as with most extremes, are a bad idea at best and impossible at worst. If you gather a group of people together and tell them to create a web app given a backlog but they’re not allowed to think or communicate about the architecture up front then you’ll find they all start dividing the problem in different ways and writing in different languages for different server and client frameworks. Hardly a good idea. Of course on the other end of the scale, believing that we can foresee all of the technical issues, all of the technology and requirements changes that might happen is as likely as a 12 month project plan Gantt chart being correct after a few levels of cumulative error margin have been combined.

For more on architecture see:

• Architectural profiling
• Architecture sketching
• Lightweight Mechanisms
• What is enough agile architecture?

If you want to really understand the software value chain in your organisation drawing diagrams in an ivory tower isn’t going to help, you need to get out there and “Go and see”.

In Steve Handy’s blog on Scaling Software Agility he talks about applying some of the thinking from the Lean school of thought to “go and see”, observing the value chain by actually physically following it.

• Identify a new piece of work and follow it through the delivery chain
• Attend every meeting, track the activities and note how long it’s static.
• Don’t attempt to fix it during this period of observation, don’t criticise, just watch and learn.

I think there’s some real value here, there’s something quite simple and powerful about physically walking the walk of a piece of work through the software value chain. Doing so will make some problems, blockages and issues blindingly obvious – processes and organisational structures that seemed to make sense on someone’s bit of paper will just not make sense.

Experiencing the value chain in a very physical way promotes professional mindfulness and will clearly identify waste. I think this is a great way of putting common sense back in to software development.

Huh?

Points are abstract and relative, comparing them across projects is like comparing apples and elephants.

Releases only make sense in themselves, let alone across project time lines so why would I want to look at a cross-project burnup? Such a thing is foolishness surely…

Well maybe not, it might be useful as an agile at scale measure, as a way of looking at the work churn in a department or other high level unit in an organisation. I’ve been pondering if there’s a way of aggregating burnup information and point burnage across teams (with distinct disjoint timelines) and thought that there might be a way.

Ideally I want to be able to show the amount of work planned within an organisational group and progress towards that scope, showing when scope goes up and down (does that ever happen?). Then I want to show progress towards that scope overall, the angle of the progress line could give me an organisational velocity – perhaps I could even add an ideal velocity that would indicate what perfect robots would to if real life never intervened (although that could be dangerous).

First I need a way of normalising points and understanding what 100% of scope means when it can be incorporating many projects at different points in their lifecycles. Perhaps a way of doing it is considering everything in terms of percentage, after all that’s an easy thing for people to consume. To define 100% of a scope of various contributing projects is tricky since it’ll change and be dependant on their releases, continuous flows, changing scope.

A simplistic approach to this is to use a moving baseline, perhaps we can determine 100% of a projects scope as whatever it thinks it would deliver within the time area being considered (the scope of the x-axis) at 15% of it’s timeline (or whatever).

In the example above this tells me that work is consistently overplanned not just in terms of actual velocity, but in terms of idealised capacity aswell – the demand is higher than the supply. I think this could be useful for “agiley” portfolio management.

Perhaps I could start establishing a budget cycle velocity, and start limiting work planned based on empirical evidence. Ok, so no project is the same and points aren’t comparable but the Law of Large Numbers is on my side.

What do you think? Is this bonkers?

Installation

It’s still not exactly shiny but, and it’s a huge but, it is simple and it works! To install on Linux I had to download 3 .deb files (1 for Lazarus, 1 for the compiler, 1 for the source) and then I could install them by simply doing:

doing sudo dpkg -i *.deb

It’s worth tidying up any old Lazarus installs first which you can do by running my script here.

Downloading *.deb files and installing them is pretty normal for linux users so although it’s not a shiny wizard installer or single command it is simple and most importantly it just works! I’m going to upgrade this from 1/19 to 8/10

First Impressions

A lot of the little niggles have gone away and it’s a much more stable, solid feeling environment. It does still load up with a million windows (Delphi style) and feel a little old fashioned (a lack of wizzy GUI controls and UI customisation, single window mode etc.) but being open source and written in itself I can change these things if I want to reasonably easily. In fact loading up the AnchorDockingDesign package makes the IDE a docked single window affair and there’s plenty of 3rd part controls to download and play with.

There’s still no multi-project support and recompiling the IDE to get a new component on the toolbar feels a bit weird even if it does work perfectly well. I’m going to stick with 6/10 for those reasons.

The GUI designer and Code Editor haven’t really changed since my last review and so they stay happily at 9/10.

Language features

The handling of generics has improved since I last looked at Lazarus and all seems to work pretty well now

Otherwise nothing much has changed, no  multicast events or garbage collection but those things don’t really slow me down much. That’s why I gave it 6/10.

Despite the lack of some of these things the language has always been and still is quite elegant (other than the nasty globabls), it’s got an excellent simple OO implementation and it’s really easy to quickly put an app together – oh yeah and it’s cross platform.

Cross-platform and backwards compatible

Lazarus works on windows, linux and mac. You can write a bit of code and natively compile it on each platform making for lightening fast code with no external dependencies. Write once, compile anywhere. You can already (partially) compile for android and other platforms are possible – anywhere the fpc compliler can be ported your code could work. That’s pretty impressive and it just works brilliantly.

Similarly all that ancient Delphi code can be loaded up edited and compiled and largely just works. Brilliant again. For that reason I’m going to bump up the language features to 8/10. (9 when android is working well).

I’ve not tried the feedback process again since v1 so I’m going to leave it at 7/10.

Conclusion

Despite it’s old fashioned feel in some places it’s simplicity is elegant and actually quite powerful. If you know the Object Pascal/Delphi language then it’s so fast to create good looking cross platform apps that it’ll knock your socks off.

When I pick up a new language I tend to do a challenge to load an xml file into a multi-column listview and create a details form to edit the selected row. In Java, something so apparently simple is a massive pain due to the imposition of the MVC pattern on everything. In C# it’s pretty easy, I can go MVC or not. In mono it’s reasonably easy too, although not the same easy and not as easy to get away from MVC.

In Lazarus it’s really easy, and it works fast and well on 3 major platforms. That’s a killer feature.

In my previous review I said I couldn’t really recommend it, and that made me sad. Now I can recommend it, in fact it’s quickly becoming my technology of choice for linux gui development, because it’s quick to put things together and they look good. Just maybe Lazarus is living up to it’s vision and bringing Delphi back to life, but bigger and better than it was before!

Of course in an agile, or indeed post-agile world, we don’t want to spend forever document and designing stuff in analysis paralysis. I’ve worked in projects where I had to draw every class in detail in a formal UML tool before I could go and code it. I’m pretty sure this halved my development speed without adding any real value. But I’ve also worked on projects where we’ve drawn some UML on a whiteboard while discussing what we were going to do and how we were going to do it – and that was really valuable.

This makes an architect’s job difficult. Of course, it’s always been hard:

The ideal architect should be a man of letters, a mathematician, familiar with historical studies, a diligent of philosophy, acquainted with music, not ignorant of medicine, learned in the responses of jurisconsults,  familiar with astronomy and astronomical calculations.

Vitruvius ~ 25 BCE

But as well as being a bit of a Renaissance man an architect also needs to know when enough is enough. I’ve found that I’ve done enough architecture with the team (not to the team) when we collectively feel like we understand the proposed solution, how it’s going to hang together, how it will address the risky bits and meet it’s requirements.
To do that, we tend to draw a few diagrams and write some words.

First, an architectural profile that gives us an idea of where the complexity is and therefore where the technical and quality risks are.

Second an overview sketch that shows the overall structure, maybe technology, target deployment platforms and major bits.

Third a set of lightweight mechanisms that cover the common ways of doing things or address particularly knotty problems and address some of those risks. These tend to describe the architecture (or mechanism flows) by example rather than aiming for total coverage.

I might add some other stuff to this if the project calls for it, like maybe a data model, a GUI mockup but generally that’s it

When it might be appropriate

• In situations where a lightweight approach to intentional architecture is required
• In situations where high design formality isn’t required
• When a simple approach to architecture analysis is required at a team of teams level before more analysis in contributing teams
• Where a team wants to cut wasteful requirements and architectural “analysis paralysis” without throwing out ignoring technical risks
• System of systems development

What is it?

When I look at a potential (or existing) system I think of it in terms of it’s complexity in terms of a few dimensions, they’re not set in stone and I might add or remove dimensions as the mood, and context, takes me.  Doing this early on gives me a feel for the shape of a project’s requirements, architecture and solution. In fact it also means I can short cut writing a whole bunch of requirements, acceptance tests, designs and even code and tests.

Here’s an example of thinking about a simple-ish app that does some fairly hefty data processing, needs to do it reasonably quickly but not excessively and has got to do some pretty visualisation stuff. Other than that it’s reasonably straight forward.

You might notice that the x-axis is pretty much FURPS with a couple of extras bolted on (I’ll come back to the carefully avoided first dimension in a minute).

The y-axis ranges from no complexity to lots of complexity but is deliberately not labelled (normally one of my pet hates) so we can focus on the relative complexity of these dimensions of the requirements, quality,  architecture and therefore solution.

The height of one of these bars helps me shape the architectural approach I’ll take to the project, and which bits and bobs I can reuse from my super bag of reuse.

For example, let’s take the security dimension. If it’s empty I know I don’t need users, audit logs etc. but if it’s medium, like it is in this example, I know that I’m probably going to need Use Cases and/or User Stories around things like:

• logging in
• changing password
• managing users
• managing permissions
• checking permissions

These are things I’ve done loads of times before so why bother writing lots of requirements for them? Why bother doing detailed design? Why bother writing the code and testing, instead let’s reuse some requirements, components and tests and let the reuse do it for us. We’ll still to end to end testing across our set of requirements, I’m not advocating ignoring all of the quality risks simply avoiding them in low complexity situations.

If the bar is higher up I might need things like LDAP integration, OAuth integration, encryption, auditing, digital signing, federated user data stores, legal compliance and other security related shenanigans.

The other dimensions I tend to look at other than the standard URPS are “Data Processing” by which I mean the number of entities I’ve got to deal with and the complexity of their relationships, finally this dimension is pushed up if I’ve got to do algorithmic stuff across that data, and it’s maxed out if I’ve got big data and massively parallelized processing. However for the lower levels of complexity I’d never write a CRUD use case or a bunch of storied that batch up to mean the same thing. Or write endless design diagrams covering the CRUD of each entity. A simple persistency mechanism (if anything other than obvious) and a good data model are all I need here.

Reporting and Visualisation means mucking around with complicated and often interactive displays of the aforementioned data.

The dimension I’ve ignored so far is “Differentiating Functionality” as this is the set of functional requirements that make your project unique. If this is empty you should probably be just buying something or slotting together some off the shelf components. Of course there are types of projects which might be low in functionality but high on performance and scale… that might be it’s differentiator.

This dimension is the stuff that adds business value and is often the most important stuff that the (maybe unenlightened) customer  might prioritise above everything else. So if you backlog is only stuff related to this category then it’s probably not a very good reflection of reality. I make it a different colour to make all of these points and to help balance it’s content with the other important dimensions.

Technical risks, and likely quality risks, will be hiding in any dimension with high complexity, in testing terms high complexity areas are fertile ground when hunting for fringe cases.

This is how I’ve been doing architecture for over 10 years but I’ve never written it down before! I’m writing it down now so that collectively “architectural profiling”,”architectural overview sketching” and “lightweight mechanisms” can join up with the Project Forum practice for “just-enough” architectural approach to agile at scale.

Mechanisms are little snippets of the architecture that address an important problem, provide a common way of doing something or are good examples of how the architecture hangs together.

Mechanisms exist within the context of an architecture, which provides overall structure for the mechanisms. I tend to use a simple architectural overview sketch to do that and then further refine the architecture, if necessary, in terms of mechanisms according to the architectural profile and (during development) the needs of my team.

Sometimes during a project the team will comment on the need to have a common way of doing something, or that they’ve uncovered something tricky that we need to consider as a more significant part of the system than our early analysis showed. In these cases it’s time to create a mechanism.

Mechanisms are great, but you don’t want to many of them, or to document and detail them too much, just enough to communicate the architecture and support maintenance efforts. Indeed writing too much actually makes it harder to communicate.

Mechanisms are best expressed in terms of their structure and behaviour, I tend to use a simple class diagram for the first and whatever seems appropriate for the second. This might be a UML sequence diagram, but I don’t really like those, instead I might use a good old fashioned activity diagram, or a flowchart with GUI mockups in the nodes. Either way I recommend limited the documentation and description, just because one flow is worth writing down to explain it the others might not be. In this way I do architectural specification by example. Once I’ve written enough about a mechanism that the rest can be inferred I stop.

The words aren’t important in this example but you can see that I try to fit the description into a fairly small concise area – that helps me focus on just the really important stuff. In the top left there’s a list titled “Appropriate for stories like:” which is an indicative list of a few things to which the mechanism is appropriate.  Next to it is some blurb that says what it’s for and the main scenarios it covers, so in the case of persistency it’s the normal query, create, edit, save & delete. There might be some notes around important constraints or whatever else is important.

I’ll then describe each important scenario in terms of it’s behaviour in whatever language or visual form makes sense. Sometimes  this is a photo of a whiteboard Sometimes it’s text, sometimes it’s a combination of those things.

The flip side

Just like stories having a flip side which contains their acceptance tests I also like to put acceptance tests on the flip side of my mechanisms. Although many are easy to frame in terms of customer acceptance tests (e.g. Search Mechanism will have performance, consistency and accuracy acceptance criteria) some are a little harder to frame. Technical mechanism formed to provide a common way of doing something in an architecture or to express the shape and aesthetics of an architecture may feel like they only make sense in terms of the development team’s acceptance criteria, however I always make sure they relate back to a story if this is the case, otherwise I could be needlessly gold plating.

Mechanisms are best found by understanding the architectural profile initially and then by actually building the system. If the customer doesn’t have a story that will be satisfied in part by a mechanism then it probably shouldn’t be there. Even if it is shiny.

I quite like UML, it’s useful to be able to draw a symbol and others know what it is without me having to explain to everyone what I mean, but I don’t like the way it has so many restrictive rules that stop me from making a nice sketch to explain what I mean, also everyone else doesn’t seem to know the language to the same degree, I need something a bit lighter.

I’ve always done a high level diagram that shows the overall pattern for my architecture, something like layers or pipes and filters or whatever. I’ve also always done a breakdown of the important stuff within each layer but I’ve had the best success (in terms of communicating with others) when I’ve mixed both, with elements of the target deployment and actor interaction.

Being terrible at naming things I call this marvellous diagram the “Architectural Overview Sketch”. Here’s an example:

It  expresses all of the structural things that I care about. It expresses the:

• the shape and feel of the system
• high level layers
• primary interfaces between subsystems
• target client platforms
• User – GUI interaction paradigm
• important classes in each layer and major layer package structure
• critical data schema
• interaction with external APIs
• the middleware and database hosting and distribution

I might have more diagrams to explain more structure in part of this if it’s really important, but I don’t want to have every class in my system on a diagram somewhere. I’m using my diagrams to communicate, not specify. I’m broken a bunch of UML rules of course, and there’s a lot of implied stuff but adding those details makes it harder to explain what I really want. One thing I really like about it is how it shows the important detailed parts of a design in the context of the bigger architecture.

Architecture, like any design, is best expressed in terms of both structure and behaviour. So far this is just structure, there’s some hints at behaviours but nothing terribly useful. My next post will be about how I minimally specify the important bits of an architectures behaviour – the mechanisms.