Mike MacDonagh's Blog

Somewhere in the overlap between software development, process improvement and psychology

How to make real progress in a software project

I’ve not been blogging much this year because I’ve been crazy busy writing software (more on that soon I hope). As you might expect I’ve changed my ways of working more often than process improvement consultants change the name of the process they’re selling but one key thing always seems to drop out whether I’m working on my own, with a team, a team of teams or a globally distributed team.

Do the top thing on the list, whatever it is.

Don’t do anything else until it’s done.

Or in process speak: “Reduce the WIP and clarify the Definition of Done“.

Too often I fall into the trap of doing the most interesting thing from the list, which normally means the most complex. And far too often I get carried away with the next interesting thing before finishing something. Both of these lead me to have far too many things in various states of progress all interfering with each other and me with no idea of how well I’m doing. And that’s assuming I actually know what “finished” means.

Software is complex enough without us making it harder with processes, remember the KISS principle?

Reduce waste: Visualise the value stream

A big thing in software process improvement (urgh) is reducing waste.  A great idea, but how do you identify waste?

One powerful method that I use is value stream visualisation. There’s a number of ways of getting information about the value stream ranging from physically walking the chain to simply asking people.

Sounds simple doesn’t it? But one of the best sources of information about wasteful processes and ways of working in a team is simply asking the team what they feel they shouldn’t be spending time on. By freeing up people to actually do their job they can reduce waste and improve productivity. One method I use is to ask each team member to identify a list of n items that they have to spend time on that they think are either entirely or partially wasteful. Basically a list of their impediments. I then anonymize the list look for commonality and then the top few are a reasonable target for improvement.

Alternatively actually walking the path of a piece of work through an organisation can really open the eyes to the sheer number of people and unnecessary waiting involved in some wasteful processes…

Anyway, having got some data I find it useful to visualise it in terms of waiting time and doing time. In Lean-speak cycle time is the time it takes to actually do something, lead time is the time it takes from the request being made to it finally being delivered. In many tasks the lead time far outstrips the cycle time, visualising it can make that very clear.

Low Lead time For example, a task with has a lead time of 5 days might actually be made up of 2 days waiting time before it actually gets picked up and the cycle time of 3 days starts.

Large lead timeIf you look at all of the tasks in your value chain you might find that many of them look more like this, with very large wasteful waiting times. There are many reasons why there might be a large waiting time, typically the team doing the tasks is over-subscribed, or a team might be protecting it’s SLAs, KPIs or other TLAs. These problems can have significant effects when looking at the system as a whole.

Even more interesting is looking at how these tasks might fit together in a value stream. If we imagine a business function that involves a request moving through a bunch of tasks across a number of teams that have different waiting and doing times a simple sequence of tasks, with a little buffering between them from the Project Manager to allow for schedule variance naturally, might end up looking like this:

 sequential_value_chain_arrows

Here we have 26 days of actual work spread over almost 60 days (12 weeks) of elapsed time = 40% waste. Even if the person planning the work is willing to accept some risk and try to optimise their workflow a bit without improving the waste inherent in the tasks involved there’s still a lot of waste in the system.

parallel_value_chain_arrows

By visualising the value stream in this way we can see straight away (apologies to red/green colour blind folks) that there’s a lot of red, a lot of waste. In many cases planners aren’t willing to accept the risks inherent in overlapping activities as shown here, or aren’t even aware that they can leading to the more sequential path shown above. The result is a minimum time that it takes a request to get done, based on the impedence of the current value chain of, in this case, 38 days before we even start thinking about actually doing any work.

Surely that’s not right.

What is “good” software architecture or design?

Architecture is a fine balance between a subtle science and exact art that combines cognitive problem solving, technical direction and expressing abstract views to aid common understanding. Design is similarly an inexact science which is inextricably linked to Architecture. As a result it’s quite hard to define what makes a “good” architecture or design.

There are high numbers of design and architecture practices ranging from the purely transformational to the evolutionary and emergent (both manual and automated) but there seems to be some heuristic consistency between these various practices and the approaches of the traditional, agile and post-agile movements.

Initially I saw the simple descriptions of that makes for “good” design from from Ron Jeffries and Kent Beck in Xtreme Programming. I like the style and direction of this kind of description so here’s my take on the characteristics that make for “good” design or architecture:

  • Intentional structure and behaviour
  • Highly modular: consisting of separate services, components, classes, objects or modules
    • Elements are highly cohesive
    • Elements are loosely coupled
    • Elements have low algorithmic complexity
  • Avoids duplication
  • Well described elements: modular elements have simple expressive meaningful name enabling readers to quickly understand their purpose, similar to code cleanliness we should strive for design cleanliness
  • Runs and passes all defined tests or acceptance criteria
  • Lightweight documentation

Intentional vs. Emergent Architecture

I’ve been thinking about architecture a lot recently but one thing that I often discuss but have never blogged about for some odd reason is intentional vs. emergent software architecture. Some old fashioned software methods such as waterfall led people into doing a lot of up front architecture work, they analysed and designed away for ages producing huge reams of UML and documentation that no one could ever squeeze into their heads if they had the patience to read it. This is an example of an intentional architecture – the architecture was intended, planned and deliberate.

Lots of folks hated that way of doing things as it meant people were writing docs and drawing diagrams instead of making working software, not to mention an all to frequent tendency to over-engineer architecture past the point of usefulness. This led to some people saying that we’re better off not trying to do any architecture and just letting it emerge from the work we do developing small little customer focused requirements (like user stories or similar).  Ok, so there’d be some rework along the way as we encounter a deeper understanding of the system and refactor our emergent architecture but it’d still be better than the old way of doing large upfront architecture.

So, there seem to be two opposed viewpoints: intentional architecture is best, emergent architecture is best.

For me, neither is true. I’ve seen some really terrible examples of badly over-engineered architectures that crippled a project and projects that never got past their over-reaching architectural analysis. Equally I’ve seen products with emergent architecture that had to be entirely re-architected as time went pay because their emergent architecture was so wrong it was comical (imagine a software management tool that only supports a single project, and that concept being deeply embedded in the architecture).

There’s a scale with intentional architecture on one side and emergent architecture on the other.

Intentional vs. Emergent ArchitectureVarious factors might push us one way or another… The second I listed on the right is interesting as if you’ve got a well known technology and problem domain you can get away with emergent architecture, but similarly if you have a totally unknown technology and problem domain it can be very effective to evolve towards a solution and architecture rather than crystal ball gaze by creating a (probably wrong) intentional architecture.

Which rather sums up the point I’m trying to make. The purpose of architecture is to shape the solution and address technical risks. Solving the big problems, creating common ways of doing something (lightweight architectural mechanisms) are all good architectural goals but only if we’re sure of the solution. If we’re not we’re better off evolving an emergent architecture, at least initially.

I think that the extremes at either end of the scale, as with most extremes, are a bad idea at best and impossible at worst. If you gather a group of people together and tell them to create a web app given a backlog but they’re not allowed to think or communicate about the architecture up front then you’ll find they all start dividing the problem in different ways and writing in different languages for different server and client frameworks. Hardly a good idea. Of course on the other end of the scale, believing that we can foresee all of the technical issues, all of the technology and requirements changes that might happen is as likely as a 12 month project plan Gantt chart being correct after a few levels of cumulative error margin have been combined.

For more on architecture see:

Understand the software value chain by walking it

There’s no better way to understand something than by actually giving it a go. If you want to know what writing a particular programming language is like you need to try it. If you want to know what kung fu is like watching a video won’t help, you need to try it.

If you want to really understand the software value chain in your organisation drawing diagrams in an ivory tower isn’t going to help, you need to get out there and “Go and see”.

In Steve Handy’s blog on Scaling Software Agility he talks about applying some of the thinking from the Lean school of thought to “go and see”, observing the value chain by actually physically following it.

  • Identify a new piece of work and follow it through the delivery chain
  • Attend every meeting, track the activities and note how long it’s static.
  • Don’t attempt to fix it during this period of observation, don’t criticise, just watch and learn.

I think there’s some real value here, there’s something quite simple and powerful about physically walking the walk of a piece of work through the software value chain. Doing so will make some problems, blockages and issues blindingly obvious – processes and organisational structures that seemed to make sense on someone’s bit of paper will just not make sense.

Experiencing the value chain in a very physical way promotes professional mindfulness and will clearly identify waste. I think this is a great way of putting common sense back in to software development.

I call this Go See, and it’s the most effective form of measurement for software development.

 

A cross-project Release Burnup?

Huh?

Points are abstract and relative, comparing them across projects is like comparing apples and elephants.

Releases only make sense in themselves, let alone across project time lines so why would I want to look at a cross-project burnup? Such a thing is foolishness surely…

Well maybe not, it might be useful as an agile at scale measure, as a way of looking at the work churn in a department or other high level unit in an organisation. I’ve been pondering if there’s a way of aggregating burnup information and point burnage across teams (with distinct disjoint timelines) and thought that there might be a way.

Ideally I want to be able to show the amount of work planned within an organisational group and progress towards that scope, showing when scope goes up and down (does that ever happen?). Then I want to show progress towards that scope overall, the angle of the progress line could give me an organisational velocity – perhaps I could even add an ideal velocity that would indicate what perfect robots would to if real life never intervened (although that could be dangerous).

Aggregate burnup

First I need a way of normalising points and understanding what 100% of scope means when it can be incorporating many projects at different points in their lifecycles. Perhaps a way of doing it is considering everything in terms of percentage, after all that’s an easy thing for people to consume. To define 100% of a scope of various contributing projects is tricky since it’ll change and be dependant on their releases, continuous flows, changing scope.

A simplistic approach to this is to use a moving baseline, perhaps we can determine 100% of a projects scope as whatever it thinks it would deliver within the time area being considered (the scope of the x-axis) at 15% of it’s timeline (or whatever).

In the example above this tells me that work is consistently overplanned not just in terms of actual velocity, but in terms of idealised capacity aswell – the demand is higher than the supply. I think this could be useful for “agiley” portfolio management.

Perhaps I could start establishing a budget cycle velocity, and start limiting work planned based on empirical evidence. Ok, so no project is the same and points aren’t comparable but the Law of Large Numbers is on my side.

What do you think? Is this bonkers?

Linux GUI Development: Lazarus 1.0 Review update

A while ago I wrote a review of Lazarus 0.9.30 and it came out with the 63% mainly because the installation process was so horrible. Well now it’s almost a year later and v1.0 has been released so it’s only fair I update my comments…

Installation

It’s still not exactly shiny but, and it’s a huge but, it is simple and it works! :) To install on Linux I had to download 3 .deb files (1 for Lazarus, 1 for the compiler, 1 for the source) and then I could install them by simply doing:

doing sudo dpkg -i *.deb

It’s worth tidying up any old Lazarus installs first which you can do by running my script here.

Downloading *.deb files and installing them is pretty normal for linux users so although it’s not a shiny wizard installer or single command it is simple and most importantly it just works! I’m going to upgrade this from 1/19 to 8/10

First Impressions

A lot of the little niggles have gone away and it’s a much more stable, solid feeling environment. It does still load up with a million windows (Delphi style) and feel a little old fashioned (a lack of wizzy GUI controls and UI customisation, single window mode etc.) but being open source and written in itself I can change these things if I want to reasonably easily. In fact loading up the AnchorDockingDesign package makes the IDE a docked single window affair and there’s plenty of 3rd part controls to download and play with.

Normal multi-window interface for Lazarus
Lazarus as a single window IDE

There’s still no multi-project support and recompiling the IDE to get a new component on the toolbar feels a bit weird even if it does work perfectly well. I’m going to stick with 6/10 for those reasons.

The GUI designer and Code Editor haven’t really changed since my last review and so they stay happily at 9/10.

Language features

The handling of generics has improved since I last looked at Lazarus and all seems to work pretty well now :)

Otherwise nothing much has changed, no  multicast events or garbage collection but those things don’t really slow me down much. That’s why I gave it 6/10.

Despite the lack of some of these things the language has always been and still is quite elegant (other than the nasty globals), it’s got an excellent simple OO implementation and it’s really easy to quickly put an app together – oh yeah and it’s cross platform.

Cross-platform and backwards compatible

Lazarus works on windows, linux and mac. You can write a bit of code and natively compile it on each platform making for lightening fast code with no external dependencies. Write once, compile anywhere. You can already (partially) compile for android and other platforms are possible – anywhere the fpc compliler can be ported your code could work. That’s pretty impressive and it just works brilliantly.

Similarly all that ancient Delphi code can be loaded up edited and compiled and largely just works. Brilliant again. For that reason I’m going to bump up the language features to 8/10. (9 when android is working well).

I’ve not tried the feedback process again since v1 so I’m going to leave it at 7/10.

Conclusion

Despite it’s old fashioned feel in some places it’s simplicity is elegant and actually quite powerful. If you know the Object Pascal/Delphi language then it’s so fast to create good looking cross platform apps that it’ll knock your socks off.

When I pick up a new language I tend to do a challenge to load an xml file into a multi-column listview and create a details form to edit the selected row. In Java, something so apparently simple is a massive pain due to the imposition of the MVC pattern on everything. In C# it’s pretty easy, I can go MVC or not. In mono it’s reasonably easy too, although not the same easy and not as easy to get away from MVC.

In Lazarus it’s really easy, and it works fast and well on 3 major platforms. That’s a killer feature.

Category Score
Installation 8/10
First Impressions 6/10
GUI Designer 9/10
Code Editor 9/10
Language Features 8/10
Feedback process 7/10
Cross-Platform 10/10
Overall 81% – Easy to use and powerful

In my previous review I said I couldn’t really recommend it, and that made me sad. Now I can recommend it, in fact it’s quickly becoming my technology of choice for linux gui development, because it’s quick to put things together and they look good. Just maybe Lazarus is living up to it’s vision and bringing Delphi back to life, but bigger and better than it was before!

SimpleGit is developed in Lazarus

What is enough agile architecure?

I wasn’t planning on writing a “how long is a piece of string?” post but it’s a question I often get, and something that I’ve played with a bit. The point of architecture is to address the aesthetics of a system, to consider its reusable bits or common forms, the overall shape and nature, the technology it’ll use, the distribution pattern and how it will meet its functional and non-functional requirements.

Of course in an agile, or indeed post-agile world, we don’t want to spend forever document and designing stuff in analysis paralysis. I’ve worked in projects where I had to draw every class in detail in a formal UML tool before I could go and code it. I’m pretty sure this halved my development speed without adding any real value. But I’ve also worked on projects where we’ve drawn some UML on a whiteboard while discussing what we were going to do and how we were going to do it – and that was really valuable.

This makes an architect’s job difficult. Of course, it’s always been hard:

The ideal architect should be a man of letters, a mathematician, familiar with historical studies, a diligent of philosophy, acquainted with music, not ignorant of medicine, learned in the responses of jurisconsults,  familiar with astronomy and astronomical calculations.

Vitruvius ~ 25 BCE

But as well as being a bit of a Renaissance man an architect also needs to know when enough is enough. I’ve found that I’ve done enough architecture with the team (not to the team) when we collectively feel like we understand the proposed solution, how it’s going to hang together, how it will address the risky bits and meet it’s requirements.
To do that, we tend to draw a few diagrams and write some words.

First, an architectural profile that gives us an idea of where the complexity is and therefore where the technical and quality risks are.

Second an overview sketch that shows the overall structure, maybe technology, target deployment platforms and major bits.

Third a set of lightweight mechanisms that cover the common ways of doing things or address particularly knotty problems and address some of those risks. These tend to describe the architecture (or mechanism flows) by example rather than aiming for total coverage.

I might add some other stuff to this if the project calls for it, like maybe a data model, a GUI mockup but generally that’s it :)

This post is an extract from the Agile Architecture content from Holistic Software Engineering

Scaled Agility: Architectural profiling

Architectural Profiling is borrowed from Holistic Software Engineering

When it might be appropriate

  • In situations where a lightweight approach to intentional architecture is required
  • In situations where high design formality isn’t required
  • When a simple approach to architecture analysis is required at a team of teams level before more analysis in contributing teams
  • Where a team wants to cut wasteful requirements and architectural “analysis paralysis” without throwing out ignoring technical risks
  • System of systems development

What is it?

When I look at a potential (or existing) system I think of it in terms of it’s complexity in terms of a few dimensions, they’re not set in stone and I might add or remove dimensions as the mood, and context, takes me.  Doing this early on gives me a feel for the shape of a project’s requirements, architecture and solution. In fact it also means I can short cut writing a whole bunch of requirements, acceptance tests, designs and even code and tests.

Here’s an example of thinking about a simple-ish app that does some fairly hefty data processing, needs to do it reasonably quickly but not excessively and has got to do some pretty visualisation stuff. Other than that it’s reasonably straight forward.

You might notice that the x-axis is pretty much FURPS with a couple of extras bolted on (I’ll come back to the carefully avoided first dimension in a minute).

The y-axis ranges from no complexity to lots of complexity but is deliberately not labelled (normally one of my pet hates) so we can focus on the relative complexity of these dimensions of the requirements, quality,  architecture and therefore solution.

The height of one of these bars helps me shape the architectural approach I’ll take to the project, and which bits and bobs I can reuse from my super bag of reuse.

Read more of this post

Lightweight architectural mechanisms – specification by example

In my previous post I talked about using a sketch to describe architectural structure, but the other part of a useful architectural description is it’s dynamics, best expressed as architectural mechanisms.

Mechanisms are little snippets of the architecture that address an important problem, provide a common way of doing something or are good examples of how the architecture hangs together.

Mechanisms exist within the context of an architecture, which provides overall structure for the mechanisms. I tend to use a simple architectural overview sketch to do that and then further refine the architecture, if necessary, in terms of mechanisms according to the architectural profile and (during development) the needs of my team.

Sometimes during a project the team will comment on the need to have a common way of doing something, or that they’ve uncovered something tricky that we need to consider as a more significant part of the system than our early analysis showed. In these cases it’s time to create a mechanism.

Mechanisms are great, but you don’t want to many of them, or to document and detail them too much, just enough to communicate the architecture and support maintenance efforts. Indeed writing too much actually makes it harder to communicate.

Mechanisms are best expressed in terms of their structure and behaviour, I tend to use a simple class diagram for the first and whatever seems appropriate for the second. This might be a UML sequence diagram, but I don’t really like those, instead I might use a good old fashioned activity diagram, or a flowchart with GUI mockups in the nodes. Either way I recommend limited the documentation and description, just because one flow is worth writing down to explain it the others might not be. In this way I do architectural specification by example. Once I’ve written enough about a mechanism that the rest can be inferred I stop.

The words aren’t important in this example but you can see that I try to fit the description into a fairly small concise area – that helps me focus on just the really important stuff. In the top left there’s a list titled “Appropriate for stories like:” which is an indicative list of a few things to which the mechanism is appropriate.  Next to it is some blurb that says what it’s for and the main scenarios it covers, so in the case of persistency it’s the normal query, create, edit, save & delete. There might be some notes around important constraints or whatever else is important.

I’ll then describe each important scenario in terms of it’s behaviour in whatever language or visual form makes sense. Sometimes  this is a photo of a whiteboard :) Sometimes it’s text, sometimes it’s a combination of those things.

The flip side

Just like stories having a flip side which contains their acceptance tests I also like to put acceptance tests on the flip side of my mechanisms. Although many are easy to frame in terms of customer acceptance tests (e.g. Search Mechanism will have performance, consistency and accuracy acceptance criteria) some are a little harder to frame. Technical mechanism formed to provide a common way of doing something in an architecture or to express the shape and aesthetics of an architecture may feel like they only make sense in terms of the development team’s acceptance criteria, however I always make sure they relate back to a story if this is the case, otherwise I could be needlessly gold plating.

Mechanisms are best found by understanding the architectural profile initially and then by actually building the system. If the customer doesn’t have a story that will be satisfied in part by a mechanism then it probably shouldn’t be there. Even if it is shiny.

Follow

Get every new post delivered to your Inbox.

Join 383 other followers

%d bloggers like this: