Mike MacDonagh's Blog

Somewhere in the overlap between software development, process improvement and psychology

What is Fringe Testing?

Fringe testing is testing the unusual cases, the things that weren’t written down, weren’t predictable and weren’t obvious. In short it’s the kind of thing inventive, creative humans are good at.
Use Case basic flowWhen we’ve got requirements they’re often written in a form that states the normal way something will go, like a basic flow in a use case (or similar). Testing this might be useful but if it doesn’t work then the developer who wrote it clearly owes everyone a doughnut because after writing it, or finishing writing it, the very first thing that should be tried is going through the basic flow. In fact it’s a good candidate for automation and testing it should be considered part of the development team’s lowest levels of their definition of done.

Use Case Alternate FlowsUse Cases are normally elaborated in terms of more than just a basic flow with a set of alternative or exceptional cases or scenarios. You might consider each of these scenarios as Use Stories – I quite like this way of working with both Use Cases to make a high level scope diagram and User Stories for actually elaborating the requirements and acceptance criteria (see Use Case vs. User Story). The thing is, if it’s been written down in the requirements then the team should be testing it as a matter of course, the probability of finding defects in these flows is again pretty low. If they’re written as stories and implemented incrementally with acceptance criteria then it’s even more unlikely as they’re not much different from the basic flow. Just another story that should be developed and tested as part of the lowest levels of getting things done.

Use Case FringesThe fringe cases are the weird things that weren’t written down, that are odd ways through the requirements and are the places where the quality risks probably lie. Coming up with these is what real professional testers are good at and what proper test techniques are good for. Testing the basic paths of everything is less useful.

I’ve often thought that one way to simulate a user using my app is to take a scatter gun approach to clicking and pressing buttons in an app, because I think algorithmically, users often don’t. Many “normal” usages of your software may actually be fringe cases because things aren’t necessarily always used as designed. Of course that makes for a good argument for simpler interfaces (both GUIs and APIs).

So “Fringe Testing” is simply testing the unusual paths through your software, the places that most likely are the highest quality risks. Of course the most “fringey” cases are often the cross requirement paths, the end to end scenarios, that users take through your software or set of integrating components. As for traditional testing… I think it’s dead.

Quality ConfidenceI’ve previously blogged on the Quality Confidence metric being a useful lead indicator for software quality. One common question I’ve had on that is “Are you expecting us to fully test every requirement?”. The simple answer is “no”, testing should be focussed on the fringe cases to put an expensive resource where it’s most valuable. The Quality Confidence measure requires you to assert when you have enough coverage, for me that’s when you feel you’ve got a handle on the quality risks by testing a mix of important flows, risky stories and weird fringe cases – the rest I cover with simple professional development which always covers verification as part of development.

Advertisements

2 responses to “What is Fringe Testing?

  1. theendofsilence1 September 22, 2012 at 9:42 am

    In all my MOD acceptance testing sceanarios, we always included periods of unscripted testing where the user would be given free reign – freeing them from the contraints of “stories” or “process”. While this was often difficult to agree commercially with suppliers, it often produced some of the more interesting failure modes as users tried to use the new technology in more imaginative ways.

    I still recall an Admiral doing a test on a touchscreen shipbourne command and control system “as a matelot would do”. Dispensing with the stylus he touched the screen with his hand, noting from the less polite error message “ah, I see you’ve thought about that one already” :o)

    • mikemacd September 22, 2012 at 5:29 pm

      That’s brilliant, thanks for sharing 🙂

      Reminds me of the CEO of an insurance company in London I talked to. He didn’t want his underwriters wasting their time by pressing buttons on computers so he issued them all with dictaphones and had the secretary pool use the software overnight! I thought he was being incredibly old fashioned but he justified it by not trusting them to be able to use the software properly!

What do you think?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: