Mike MacDonagh's Blog

Somewhere in the overlap between software development, process improvement and psychology

Tag Archives: requirements

What is Fringe Testing?

Fringe testing is testing the unusual cases, the things that weren’t written down, weren’t predictable and weren’t obvious. In short it’s the kind of thing inventive, creative humans are good at.
Use Case basic flowWhen we’ve got requirements they’re often written in a form that states the normal way something will go, like a basic flow in a use case (or similar). Testing this might be useful but if it doesn’t work then the developer who wrote it clearly owes everyone a doughnut because after writing it, or finishing writing it, the very first thing that should be tried is going through the basic flow. In fact it’s a good candidate for automation and testing it should be considered part of the development team’s lowest levels of their definition of done.

Use Case Alternate FlowsUse Cases are normally elaborated in terms of more than just a basic flow with a set of alternative or exceptional cases or scenarios. You might consider each of these scenarios as Use Stories – I quite like this way of working with both Use Cases to make a high level scope diagram and User Stories for actually elaborating the requirements and acceptance criteria (see Use Case vs. User Story). The thing is, if it’s been written down in the requirements then the team should be testing it as a matter of course, the probability of finding defects in these flows is again pretty low. If they’re written as stories and implemented incrementally with acceptance criteria then it’s even more unlikely as they’re not much different from the basic flow. Just another story that should be developed and tested as part of the lowest levels of getting things done.

Use Case FringesThe fringe cases are the weird things that weren’t written down, that are odd ways through the requirements and are the places where the quality risks probably lie. Coming up with these is what real professional testers are good at and what proper test techniques are good for. Testing the basic paths of everything is less useful.

I’ve often thought that one way to simulate a user using my app is to take a scatter gun approach to clicking and pressing buttons in an app, because I think algorithmically, users often don’t. Many “normal” usages of your software may actually be fringe cases because things aren’t necessarily always used as designed. Of course that makes for a good argument for simpler interfaces (both GUIs and APIs).

So “Fringe Testing” is simply testing the unusual paths through your software, the places that most likely are the highest quality risks. Of course the most “fringey” cases are often the cross requirement paths, the end to end scenarios, that users take through your software or set of integrating components. As for traditional testing… I think it’s dead.

Quality ConfidenceI’ve previously blogged on the Quality Confidence metric being a useful lead indicator for software quality. One common question I’ve had on that is “Are you expecting us to fully test every requirement?”. The simple answer is “no”, testing should be focussed on the fringe cases to put an expensive resource where it’s most valuable. The Quality Confidence measure requires you to assert when you have enough coverage, for me that’s when you feel you’ve got a handle on the quality risks by testing a mix of important flows, risky stories and weird fringe cases – the rest I cover with simple professional development which always covers verification as part of development.

Scaled Agility: The Project Forum

Name: Project Forum (Middle-out management structure) – Agile at Scale practice

When it might be appropriate

  • In situations where multiple competing stakeholder groups with different agendas are required to work together
  • In situations where multiple product groups need to collaborate on a bigger outcome
  • Where there is a conflict in direction, resource management/ownership or scope between collaborating groups
  • System of systems development

What is it?

The Project Forum is an application of agile philosophy to large project structures. Rather than impose a hierarchy of decision making from the Project Manager downwards the Project Forum is a virtual team in the middle of all stakeholders.

The Project Forum is a self-organising democratic group that balances competing voices and concerns, owns high level scope and architecture, runs the high level release train and performs integration activities for the product.

Use of the Project Forum practice does not prevent any communication directly between contributing groups it only provides a vehicle for that conversation when it’s relevant for the wider project.

From Traditional to Agile at ScaleThe Project Forum practice is an example of Agile at Scale combining social business practices, technical software practices and ways of working to make a simple way of doing big complicated bits of work.

Read more of this post

Simple software project measures

I’m not a big fan of metrics, measures, charts, reporting and data collection. I’m not terribly impressed by dashboards with 20 little graphs on showing loads of detailed information. When I’m involved in projects I want to know 3 simple things:

  • How quick are we doing stuff?
  • Are we on track or not?
  • Is the stuff good enough quality?

There can be some deep science behind the answers to those questions but at the surface that’s all I want to see.

Organisations need to know that teams are delivering quality products at the right pace to fit the business need. To achieve this goal teams need to be able to demonstrate that their product is of sufficient quality and that they can commit to delivering the required scope within the business time scales. If the project goal may not be achieved then the business or the team need to change something (such as scope, resources or time scales). This feedback mechanism and the open transparent communication of this knowledge is key to the success of agile delivery.

The goal of delivering quality products at the right pace can be measured in many complex ways however, when designing the Project Forum agile at scale practice we looked at just 3 measures. In fact I should probably call them 2.5 measures as the throughput/release burnup can be considered mutually exclusive (if you’re continuous flow or iterative). The most important measure is people’s opinions when you go and talk to your team.

Simple Measures Dashboard

Note: in the measures section I often refer to “requirements” as a simple number, this could be a count, a normalised count, magnitude, points, etc. it doesn’t matter what’s used so long as it’s consistent.

Read more of this post

What does “Agile at Scale” mean?

There’s a lot of talk in the process improvement industry about the meaning of “agile at scale”, devops, lean and agile for a number of reasons. One reason that I’ve seen in successful agile organisations is that as development maturity increases with true agile adoption bigger problems present themselves. This is the natural progression of the science of development, what used to be considered complex (like Object Orientation) is now normal, a commodity. Innovation in ways of working is happening at the organisational, cross-team, cross-product level.

For me agile at scale (I’ve got fed up of the quotes already) means a couple of different things:

  • Repeating agile successes embodied in a team across an organisation (scaling out?)
  • Applying agile thinking to cross-product projects
  • Applying agile and lean thinking to development organisations
  • Applying agile and lean thinking to high assurance environments like medical, security, financial, safety critical, audited, regulated businesses.

Agile and lean? Yep, both with lower case letters. I’m not particularly interested in ideological approaches to software development, I believe strongly in taking the best bits of whatever processes, techniques, practices etc. you find externally, mixing them up with internal practices and ways of doing things to develop simple, pragmatic approaches to ways of working. Both agile and lean schools of thought promote minimising unnecessary work, shorter delivery cycles and higher quality, continuously learning lessons and empirical decision making.

The agile manifesto gave us:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.

Some great practices have evolved for applying agile and lean thinking like scrum, kanban etc. However all of the complex organisations I’ve worked with have found that there’s still space for more thinking in terms of how to run a software business, how to deal with big complex system-of-system problems, multiple competing stakeholder sets, programme and portfolio management etc. Not surprising really because the agile movement wasn’t about trying to do any of that stuff.

However organisations who are successful with agile transformations want to apply the successful open and honest philosophy behind the agile manifesto to other parts of their business as well as bigger and bigger projects and programmes because the results of “doing agile” (I promise I’ll stop with the quotes soon!) are so attractive when it’s done well, namely:

  • Shorter delivery cycles, higher quality
  • Deep engagement between customers and development teams leading to respect, collaboration and better morale
  • Quick identification of when things are starting to go wrong

Consider the following model, not uncommon amongst large organisations:

Diagram of nominal large inter-dependant organistion structure

This represents a typical software department or vertical section of a software department with a portfolio that provides the funding for work. Big portfolio’s are normally broken down into a number of programmes which in turn may levy high level requirements onto organisations (organisational sub-divisions that own multiple products) which may affect both legacy and new product development. Often within a vertical section of a business there will be many cross-dependencies in terms of requirements, technical dependencies etc. For many large businesses this picture is overly simplistic, indeed I’ve not included things like projects, component teams and a variety of business organisation constructs like product centres, feature teams etc.  So how do you apply agile and lean philosophy to this lot and more?

You can’t simply repeat the same practices recursively throughout an organisation to deal with larger scale complexity.  Imagine a chain of scrum-of-scrums, daily stand-ups at every level (at the same time, or staggered to allow representation up the chain?), sprint plans at programme level etc. What about if the business is regulated, audited, security focussed, high risk financial, safety critical, etc.

Ok, so what’s agile at scale then?

Agility at Scale is applying the spirit of agility and lean thinking if not the letter to these bigger problems. It’s about:

  • Valuing individuals and interactions, encouraging collaboration, reducing layers of communication over processes, tools and hierarchy
  • Valuing working software in the form of quality releases from short development cycles over comprehensive documentation, business analysis, enterprise architecture documentation
  • Valuing customer, business, developer and operations (see DevOps) collaboration over contract negotiation
  • Valuing good governance, transparency and honesty in progress, plans, costs and impediments over regular reporting
  • Valuing responding to change over following a plan at all levels of the business

Agility at scale is focussed simply on reducing unnecessary bureaucracy, reducing time to market and improving value.

So how do you achieve it?

The application of:

Of course each of those (and more!) is a complex can of worms in itself. A lot of these higher scale practices are only just emerging from high maturity complex (post-)agile organisations but in time more of those things will turn into links.

A good example of  “Agile at Scale” in action is the Project Forum practice

As always this blog is  a stream of consciousness so please, let me know your opinion?

Use Case vs. User Story

Use Cases are too big to fit into a sprint/iteration! User Stories are so fine grained there’s too many too keep track of! Where’s the big picture? How to we define releases? Argh!!! I don’t know which to use!

Personally I tend to use both. I don’t think there’s any conflict between Use Cases and User Stories, in fact they’re rather complementary. Here’s how and why….

A few years ago I delivered a presentation at an agile software development conference entitled “Do requirements still matter?”. The short answer was “yes”. Most descriptions of agile iteration start with having a backlog, prioritising etc. etc. But where does the backlog come from? My answer: the Backlog Fairy; obviously.

The Backlog Fairy

Of course requirements still matter, they’re how we get a common understanding between customers and development teams of what we need to do. They’re the units by which we can incrementally build systems. In the rush to adopt agile processes a lot of teams have forgotten that they still need to start up with some lightweight analysis of scope and risk.

One of the things Use Cases are great at is capturing the big picture, showing a simple diagram of stick people and blobs that people who’ve never heard of UML can happily discuss in terms of what big bits of functionality are in and out of the system and who’s going to do them. The humble Use Case diagram. Not so easy to do that with a massive list of small stories.

However Use Cases can often be a bit too big to fit into a single development cycle but since they’re made up of a lot of different scenarios they’re easy to slice up into smaller bits. This is where I often use stories.

The advantage of Use Cases defining the scope is they’re nice chunky things to estimate and prioritise in a first pass. Of course as we get going we’ll estimate and prioritise stories for sprints/iterations but Use Cases help us prioritise requirements into Releases.

So I use a Use Case diagram to describe the high level scope of the thing to be developed. This normally takes about 5minutes to sketch. Then I use the Use Cases identified by the diagram (not documents, just the ovals on the diagram) to focus discussion on deriving stories (or use case slices) for development and testing within a sprint/iteration.

Sometimes stories turn up that don’t really fit into the early Use Case model, this is a rather good thing as it lets us challenge the understanding of scope. Does the story not fit in because it’s not really in line with the customers priorities and needs, it’s a cross-cutting or architectural concern or because we’ve missed some important part of the scope? All are important things to understand.

First Look: CLM 2011 RTC + RRC + RQM 3.0.1 review and screenshots

What is it?

CLM 2011 is a suite of tools making up IBM Rational Team Concert, Rational Requirements Composer and Rational Quality Manager properly integrated for what is arguably the first time. In my opinion this release is the closest to my interpretation of the original Jazz vision that IBM have delivered yet. It’s not perfect but I like the direction it’s heading in.

It’s taken me a while to write this, mainly because I’ve been very busy but also because it seemed to take some deep magic to get all of this stuff installed and running properly. But I’ve done it a couple of times now so I thought I’d post some screenshots and comments. Of course if you look back on my posts of the original RTC review you’ll see that life is harder these days already. With RTC 1 and 2 you could just download a zip, extract, run a script and you were off and away. Installation isn’t too bad… it’s not great and simple but this is a big complex suite of tools so I don’t really expect it to be trivial.

Lifecycle Project Administration

To start with, the management of projects across these separate tools has been significantly improved by the introduction of Lifecycle Project Administration which allows you to create an integrated cross tool project in one step and simply manage members and roles across them. This is a big step forward although there are still some problems in that it’s not easy to do these kind of things at scale. For example if I want to add 3 people with a set of roles to 40 projects I’m going to be doing a lot of manual clicking. In fact generally it’s not so easy to deal with project areas at scale in the Jazz toolset and that hasn’t significantly improved yet although Lifecycle Project Administraton is an important step in that direction.

Creating a new Lifecycle Project

Creating a new Lifecycle Project

A Lifecycle Project

A Lifecycle Project

Editing roles accross tool project areas

Editing roles accross tool project areas

I’m not a big fan of the navigation between project areas (linked or unlinked) as the user needs to understand the architectural relationship between personal dashboards, Change and Configuration Management (which has the RTC bits like plans, work items, source control, builds etc.), Quality Management (which has the RQM bits like test plans, cases, scripts etc.) and Requirements Management (which has the RRC bits like diagrams, UI sketches, docs, requirements etc.) to navigate the stuff in their single project. I think it’s a mistake to build the UI navigation around the architecture, I would prefer to see a unified project interface and navigation structure with the extra products adding to the project mass like aspects on a foundation. As before this becomes even more of an issue when you scale up to hundreds of projects. Incidentally, aspect orientation is how we apply practices to process kernels while still providing a seamless user experience of composed practices.

Cross (linked) project navigation

Cross (linked) project navigation

So although the three products are closer together than ever before, sharing a UI and a common architecture they still are separate products and that’s clear in terms of navigation, UI differences  and linking items between them. This is a shame for many reasons but one of the most important is that it’s still providing niche tools for separate roles, building walls between people in development teams and making cross-functional teams harder to create as individuals have to learn specific skills. These differences are not a strength, they make the whole game of software development harder.

To be fair though I’m yearning for an ideal perfect solution, CLM 2011 isn’t that idealised perfect solution but it’s a lot closer to it than anything else I’ve used!

Let’s start with IBM Rational Requirements Composer

RRC 3.0.1 is a totally different beast than RRC 1 (see my original 2008 review here) and shouldn’t be thought of in the same light as RRC1. The old version was eclipse only, badly integrated, dependent on IE and full of bugs – this version is entirely web based, deeply integrated into the Jazz platform and not dependent on IE! As for bugs, I don’t know yet but it’s actually usable which v1 wasn’t!

What it does:

  • Tracks requirements of different types with different attributes and links
  • Web based diagram editing (via browser plugin for FireFox or MSIE ), although someone used Microsoft CAB technology inside the FireFox xpi so don’t expect to do any diagram editing from linux or a mac :(
  • Ability to produce use case diagrams, UI prototypes, UI storyboards

For me this is the first time RRC has lived up to some of my early hopes for it. I see this as a replacement for ReqPro, and indeed a replacement for DOORS in time.

Creating Requirements Artifacts

Creating Requirements Artifacts

Linking a requirement to a new implementation story

Linking a requirement to a new implementation story

Unfortunately I only use linux at home so I couldn’t take screenshots of the RRC web editor, even in a virtual machine running windows on an explicitly supported browser I can’t get the plugin to work. When I’ve used it my office environment I’ve really quite liked it though, although it’s got quite a lot of basic usability problems. I’m looking forward to when it’s more mature.

There is also a dependency/traceability tree viewer but that’s a separate download at the moment.

Next Implement something in IBM Rational Team Concert

RTC hasn’t changed that much from a user perspective, it’s still a great tool for tracking work items, making atomic multi-file change sets in parallel with continuous integration and build management. I’ve been using RTC in anger for 3 years now and still like it and think it can help drive agile benefits in development teams. Granted it’s got some issues like any tool, enterprise scalability issues and annoyances the biggest of which is the lack of cross-project queries which is practically unforgivable from my perspective. See this work item on jazz.net and demand cross-project queries from IBM!

With that said, working in Eclipse from workitems, fully using Jazz SCM and continuous build is awesome.

Edit a work item in RTC

Edit a work item in RTC

Sprint plan in RTC

Sprint Plan in RTC

Making code changes in RTC Eclipse

Making code changes in RTC Eclipse

Looking at build results in Eclipse

Looking at build results in Eclipse

Looking at Build results in the web

Looking at Build results in the web

Then Test via IBM Rational Quality Manager

I’ll admit it, I’m not a cross-functional ideal. I’m a typical developer, I’m not a tester, nor am I a test manager so I’ll leave the finer points of testing theory to others. However I do have some things to say about RQM from a cohesive logical process perspective.

In this example I’ve created a Test Case based on the pizza requirement I entered in RRC (in exactly the same way that I created the implementation story). At this point frankly I’m a little disappointed because my Test Case has been created and is linked to the requirement (good) but it has no knowledge of the implementation story that I also created from RRC.

Test Case in RQM

Test Case in RQM

Test Case link back to requirement

Test Case link back to requirement

No pre-existing link to development items

No pre-existing link to development items

The Missing Link

For me this is the missing link. I realise it’s not as a simple as their always being a related triangle of 1-1 associations between these different types of artifacts, but the link between the test and the development items should at least be suggested as the odds are fairly strong that if a requirement is developed by an bit of code the test for the requirement is likely to need to test the aforementioned bit of code. Obviously I can manually create this link but that’s not the point.

In fact this is symptomatic of the fact that these CLM tools are still separate and then integrated. There is a separation between requirements in RRC, development plan items and tasks in RTC and test plan items and tests in RQM. I have to create work items /artifacts  in each tool to represent these different things and link them all together. Which is not really what I want to do.

I don’t want to spend a lot of my time creating at least 2 items for every requirement in my product (1 dev story+tasks and 1 test case+scripts).

What I want to do is elaborate a set of requirements in a simplistic user friendly UI with nice diagrams (RRC can do this) then view that candidate backlog immediately in my planning tool and further refine it – not copy it (even in bulk) to RTC but to actually manipulate the same work items, albeit a different aspect of them, in RTC. I want testing to be an inherent part of development with quality management and assurance being just another aspect of the system. Every requirement should have quality dimensions and be a test case although I might want to add more of course.

I want to have requirements with dev tasks and tests hanging off them.

Basically I want to define something that needs to be done, plan it, do it and test it all as one high level thing. I might drop off loads of tasks, different types of tests, supporting sketches etc. but I want a holistic understanding of a development item with the ability to project different views to different types of stakeholders (customer requirements views, test professional views, management views, development views).

Some conclusions

CLM2011 is a serious step in the right direction from IBM. In my opinion Jazz always has been but it’s been suffering from too many silly little problems and a lack of meaningful deep integration (ironic considering the original mission). If I had a magic wand I would do the following feats of greatness to the Jazz platform which I believe are all necessary to turn what is a good suite of tools into a true killer software development environment.

  • all jazz tools to apply new aspects to projects rather than creating seperate project areas which then need linking and administration via LPA
  • all artifacts and workitems to be viewable as native items in any tool interface
  • all artifacts and workitems created by all three tools (and other future ones) be instantiated and linked with truly consistent UI, all taggable and commentable
  • all artifacts, scm/build elements and work items queryable via a single query engine (across linked and un-linked project areas)
  • the ability to self serve projects (without granting massive impractical rights), finer grained security and permissions control
  • parent-child change sets
  • smoother flow of work items between tool aspects rather than creating new ones with links between them
  • make RRC diagram editing work on linux – things that only work on Windows are not enterprise deployable, and if they’re browser based and only work on windows someone needs shouting at. Even a much maligned school boy shouldn’t be making this error
  • a decent reporting solution and technology

Finally , being able to capture and elaborate requirements, plan iteratively, develop code, continuously integrate with unit tests, system test and manage test plans (amongst other things!) in what feels close to a single tool is extremely powerful. If only navigation wasn’t architecturally focused this strength would be much stronger and be a killer feature for IBM and Jazz.

If they sort out cross-project querying.

First Look: IBM Rational Requirements Composer

Note this is from 2008, for a review of RRC in 2011 see CLM 2011 review

So I downloaded and installed IBM Rational Requirements Composer (RRC) today. I’m not very good at reading instructions so typically I didn’t read them but I still managed to set up RRC server and connect a client within an hour :D Excellent job yet again Jazz people, in the past with the “classic” tools this sort of thing wouldn’t have been possible in such a short time. It even co-exists (but isn’t integrated) with my Rational Team Concert installation. At the moment I’ve got two Jazz server instances which is a shame, but this is only a Beta.

Anyway, I used the configuration utility and with only referring to the instructions once or twice I quickly got RRC setup and working. Having said that the config utility uses an embedded IE instance to access the Jazz Admin console and for me that wasn’t working so I gave up on the config utility and just used trusty FireFox.

The client is Eclipse based but isn’t shell sharing with my other eclipse shell at the moment. I’ve created a Test Project and thought about creating some artifacts to go along with it. I can’t see where to edit templates but since this is Jazz based I’m sure that everything is customisable. I’ve got a bunch of errors showing in my logs and in the Jazz admin web UI so I’m not sure if I’m seeing everything anyway. Perhaps reading the instructions is a good idea!

Overview

It’s clear that it’s an early Beta as there’s still a lot of simple UI bugs but the point of these releases is not to provide a finished product but to give people that are interested a chance to get to grips with the functionality and look and feel. So here’s some of my thoughts and screenshots (clicky piccies):

I set about creating a process diagram

Then a glossary that supported some of the terms that I identified in thinking about the business process

I also played around with creating a Use Case diagram

Because I’ve got a software development background I immediately decided to mock up a UI and screen flow rather than consider any of those pesky requirement things :P

At this point it seemed like a good idea to think about writing an initial Use Case specification, this was cool because I could integrate the various things I’ve done already such as embed the UI mockup, link to the business process and have glossary management done for me too :)

All of which left me with some cross linked integrated stuff to do with capturing my requirements as regards eating doughnuts

Conclusions

It’s quite easy to start setting up a set of integrated stuff including storyboards, process diagrams, use cases, UI mockups etc. and is very non-technical to use. Personally I found the UI mockup functionality to be limited, I’d prefer more free form drawing capability when I create a “sketch” it would take me longer to mock up a UI here than it would for me to build it in Visual Studio – but then again this is aimed at analysts that may not be able to use IDEs. The UI seems very windows based as well, what about trusty web widgets!

I’d like to get at the project template and see what can be done in terms of the elements and strucutre of the project, not to mention document templates for things like Use Cases.

Does this replace RequisitePro? No. Although it’s got requiremetns authoring, marking and linking RRC doesn’t yet provide full traceability management and (at least at the moment) I can’t see where I’d go about attributing and managing requirements attributes. That’s why RRC has integration into ReqPro to provide these things.

Personally I’d like to see versionable requirements artifacts, more flexible UI sketching, traceability management, attribute management and more integration into other Rational tools such as Rational Software Architect and Team Concert.

This is a good start in terms of providing a single tool to support requirements elicitation and elaboration, all the diagrams and docs in one place, easily distributed and collaborated on. I’ll look forward to seeing more of it as time goes by :)

Follow

Get every new post delivered to your Inbox.

Join 345 other followers

%d bloggers like this: