Quality Confidence: a lead measure for software quality

For a long time I’ve wanted to be able to express the quality of my current software release in a simple intuitive way. I don’t want a page full of graphs and charts I just want a simple visualisation that works at every level of requirements to verification (up and down a decomposition/recomposition stack if you’ve got such a beasty). My answer to that is Quality Confidence.

What it is

QC combines a number of standard questions together in a single simple measure of the current quality of the release, so instead of going to each team/test manager/project manager and asking them the same questions and trying to balance the answers in my head I can get a simple measure that I can use to quantitively determine whether my teams are meeting our required “level of done“. The QC measure combines:

• how much test coverage have we got?
• what’s the current pass rate like?
• how stable are the test results?

We can represent QC as a single value or show it changing over time as shown below.

How to calculate it

Quality Confidence is 100% if all of the in scope requirements have got test coverage and all of those tests have passed for the last few test runs.

To calculate QC we track the requirements in a project and when they’re planned for/delivered. This is to limit the QC to only take into account delivered requirements. There’s no point in saying we’ve only got 10% quality of the current release because it only passes some of the tests because the rest having been delivered yet.

We also track all of the tests related to each requirement, and their results for each test run. We need to assert when a requirement has “enough coverage” so we know whether to include it or not – the reason for this is that if I say a requirement has been delivered but doesn’t yet have enough test coverage then even if all of it’s testing has passed and been stable then I don’t want it adding to the 100% of potential quality confidence. The assertion that coverage isn’t enough means that we aren’t confident in the quality of that requirement.

So 100% quality for a single requirement that’s in scope for the current release is when all the tests for that requirement have been run and passed (not just this time but for the last few runs) and that the requirement has enough coverage. For multiple requirements we simply average (or maybe weighted average) the results across the requirements set.

If we don’t run all the tests each during each test run then we can interpolate the quality for each requirement but I suggest decreasing the confidence for a requirement (maybe by 0.8) for each missing run. After all just because a test passed previously doesn’t mean it’s going to still pass now. We also decrease the influence of each test run on the QC of a requirement based on it’s age so that if 5 tests ago the test failed it has less impact on the QC that the most recent test run. Past 5 or so (depending on test cycle frequency) test runs we decrease the influence to zero. More info on calculation here.

So… how much coverage is enough?

Enough coverage for a requirement is an interesting question… For some it’s when they’ve covered enough lines of code, for others the cyclomatic complexity has an impact, or the number of paths through the requirements/scenarios/stories/use cases etc. For me, a requirement has enough test coverage when we feel we’ve covered the quality risks. I focus my automated testing on basic and normal flows and my human testing on the fringe cases. Either way, you need to make the decision of when enough is enough for your project.

To help calibrate this you can correlate the QC with the lag measure of escaped defects.

Measurement driven behaviour

The QC measure is quite intuitive and tends to be congruent with people’s gut feel of how the project/release is going, especially when shown over time, however there’s simply no substitute for going and talking to people. QC is a useful indicator but not something that you can use in favour of real communication and interaction.

The measurement driven behaviour for QC is interesting as you can only calculate it if you track requirements (of some form) related to tests and their results. You can push it up by adding more tests and testing more often Or by asserting you have enough coverage when you don’t However correlation to escaped defects would highlight that false assertion.

If you’ve got a requirements stack ranging from some form of high level requirements with acceptance tests to low level stories and system tests you can implement the QC measure at each level and even use it as a quality gate prior to allowing entry to a team of teams release train.

Unfortunately, because me and my mate Ray came up with this in a pub there aren’t any tools (other than an Excel spreadsheet) that automatically calculate QC yet. If you’re a tool writer and would like then please do, I’ll send you the maths!

Rational Quality Manager Beta 2

I hear that there will be a Rational Quality Manager Beta 2 released in about a week and half. Here’s my review of Beta 1

More info

See my first look at IBM Rational Requirements Composer here

See more stuff about Jazz in general here: What’s all that Jazz?

First Look: IBM Rational Quality Manager

Note this is from 2008, for a review of RQM in 2011 see CLM 2011 review

Yet again hats off the Jazz folks for making something consumable. With no effort involved I was able to download and install a Beta 2 of Rational Quality Manager (RQM) and login and get a default empty dashboard with no configuration at all 8)

RQM is a web based tool “for test planning, workflow control, tracking and metrics reporting capable of quantifying how project decisions and deliverables impact and align with business objectives.” It’s also Jazz based. A couple of weeks ago I did a First Look analysis of Rational Requirements Composer (here) so I thought I’d have a look at RQM today. I should caveat this by saying I’m not a QA/Test expert!

This being a Beta there are some bugs hanging around like the horizontal tabs having a higher z-order than the pop-up menus from the left navigation bar so you’ve got to be precise with the mouse to open the “My Test Plans” tab at the moment and it’s a little hard to create a new test plan :S If only the left menu bar auto scrolled down like the “related sites” links on the far right! Eventually I worked around this by constructing the URI to create a new test plan – https://localhost:9443/jazz/web/console/QM#action= com.ibm.rqm.planning.home.actionDispatcher&subAction=newTestPlan and just typing it into the browser address bar. I had this problem in FF and IE, anyway I’ve got a work around so off I go.

Here’s the blank empty GUI:

Once I got into the Test Plan editor I was able to start creating a structured Test Plan. I like this idea, as instead of a huge Word document with no structure here there are discreet sections with potentially seperate ownership. Obviously in my demo repository there are no requirements to hook into a test plan so I clicked the button to create one, this led me into the Rational Team Concert web interface 8)

Unfortunately I got quite a few errors in trying to save sections of my test plan, and also none of the editors for the various sections would let me type in them no matter how many times I hit the “Edit” link. The idea of structured test assets continues in the Test Case editor, although I had the same problems as with Test Plans here in terms of editing some of the content. I was however able to generate work items (which naturally I did twice) based on my test case and configured environments using the excellent lab management interface

Upon executing my generated work item the problem I found was that my test failed. There were no doughnuts to be had!

This being based on the Jazz platform there is of course a lot of reporting available (I shifted over to the sample Java PetStore test plan since there wasn’t enough data in my dummy project).

Conclusions

RQM gives you a fully web based quality management portal. The structured nature to managing test assets is a major improvement, and since this is Jazz based I’m sure it’s all deeply configurable for different ways of working and approaches to quality management and testing.

The integration of information from Team members, Test Plans, Test Cases, Test Scripts, Execution, Reports and Defects is seamless and intuitive, with requirements and defects being managed through the Rational Team Concert interface and of course the data is easily reported on.

I’d like to dig more into the integration with other tools such as Rational Team Concert, possibilities with Rational Requirements Composer and the relationship between process templates in these various tools. Not to mention migration paths from other QM/Test tools like the “classic” Rational tools and those from other vendors.

All in all it looks like an impressive, intuitive and powerful product but it’s an early access Beta and has quite a few bugs that can make playing with it a bit frustrating at the moment.

More info

See my first look at IBM Rational Requirements Composer here

See more stuff about Jazz in general here: What’s all that Jazz?