Advanced RTC SCM Streaming – Part 1 Development, Integration and Release Streams

Basic usage of RTC Jazz SCM is easy enough but what about more advanced usage? Using streams to isolate development from integration? Cross-project component reuse? Parallel branched development? This is the guide for that stuff.

Feel free to contact me with questions or just use the comments section below – I can also offer RTC Masterclasses covering this stuff and more.

This blog is in three parts:

1. Development, Integration and Release Streams
2. Streams for parallel branched development
3. Streams for component reuse

Part 1 also covers the introduction and overall conclusions

Use Case vs. User Story

Use Cases are too big to fit into a sprint/iteration! User Stories are so fine grained there’s too many too keep track of! Where’s the big picture? How to we define releases? Argh!!! I don’t know which to use!

Personally I tend to use both. I don’t think there’s any conflict between Use Cases and User Stories, in fact they’re rather complementary. Here’s how and why….

A few years ago I delivered a presentation at an agile software development conference entitled “Do requirements still matter?”. The short answer was “yes”. Most descriptions of agile iteration start with having a backlog, prioritising etc. etc. But where does the backlog come from? My answer: the Backlog Fairy; obviously.

Of course requirements still matter, they’re how we get a common understanding between customers and development teams of what we need to do. They’re the units by which we can incrementally build systems. In the rush to adopt agile processes a lot of teams have forgotten that they still need to start up with some lightweight analysis of scope and risk.

One of the things Use Cases are great at is capturing the big picture, showing a simple diagram of stick people and blobs that people who’ve never heard of UML can happily discuss in terms of what big bits of functionality are in and out of the system and who’s going to do them. The humble Use Case diagram. Not so easy to do that with a massive list of small stories.

However Use Cases can often be a bit too big to fit into a single development cycle but since they’re made up of a lot of different scenarios they’re easy to slice up into smaller bits. This is where I often use stories.

The advantage of Use Cases defining the scope is they’re nice chunky things to estimate and prioritise in a first pass. Of course as we get going we’ll estimate and prioritise stories for sprints/iterations but Use Cases help us prioritise requirements into Releases.

So I use a Use Case diagram to describe the high level scope of the thing to be developed. This normally takes about 5minutes to sketch. Then I use the Use Cases identified by the diagram (not documents, just the ovals on the diagram) to focus discussion on deriving stories (or use case slices) for development and testing within a sprint/iteration.

Sometimes stories turn up that don’t really fit into the early Use Case model, this is a rather good thing as it lets us challenge the understanding of scope. Does the story not fit in because it’s not really in line with the customers priorities and needs, it’s a cross-cutting or architectural concern or because we’ve missed some important part of the scope? All are important things to understand.

Working with Maven multi-module projects in RTC Jazz SCM with m2

I’ve been working with Maven projects with m2 for a while in RTC Jazz SCM so thought I’d post some notes. For reference this post covers using RTC v3.x things may change in the future. If you’re not using m2 then I’d suggest you read this blog by Philippe Krief.

First of all it’s important to understand that RTC v3 has limited support for Maven. You can setup a build definition which can call a maven goal (such as a local install) easily enough but that’s about it. The RTC build mechanism is basically a cron job with a nice web (like Hudson/Jenkins without the plugins) and Eclipse front end, it doesn’t have a deep understanding of Maven.

The whole point of Maven over Ant is that it does a lot by convention rather than specification, however in most advanced Maven project I’ve seen pom files are as big as equivalent Ant + Ivy files. Regardless I still think Maven with the excellent Sonatype Nexus is awesome (although I also like the structured bash/batch like nature of Ant as well).

Most serious Maven projects are multi-module. This is because component based development is a good thing (shameless employer plug, but it’s true, it is a good thing). Most (tending to all) multi-module projects have the following structure where “O” is a folder and “-” is a file, considering a hypothetical ProductX:

- pom.xml (the parent pom for the master/uber/product build, lists child modules)

- various files at the root of ProductX

O ModuleA

- pom.xml (for ModuleA)

- various ModuleA root files

O various ModuleA folders of stuff (such as src)

O Module B

- pom.xml (for ModuleB)

- various ModuleB root files

O various ModuleB folders of stuff

If you want to work with this stuff in RTC Jazz SCM then you need to balance the constraints of Maven m2, RTC and Eclipse. If you’re considering doing this then you’ll probably be considering moving existing Maven projects to RTC, in which case you may find a previous blog on a manual migration script useful. The only edit to this I make for Maven is to separately import the parent from the children if flattening the structure (although in most cases I wouldn’t recommend flattening).

The constraints playing against each other are:

• Maven and plugins expect sub-modules to be in child directories as in the tree above
• RTC doesn’t allow both an Eclipse project and root files to be at the root of a component
• The simple case of Eclipse and RTC SCM interacting during a load from the server restricts Eclipse to only seeing projects which are represented as folders at the root of the local Eclipse workspace containing .project files.
• Eclipse doesn’t really support parent and child projects.

I’ve found there are a number of approaches, none are perfect:

1. Flatten the structure (my opinion the worst option)

You can edit the parent pom.xml to reference the sub-modules as ../ModuleA etc. although the Maven release plugin (amongst others) don’t play well with this. Also, and somewhat fundamentally, you shouldn’t have to restructure your code based on your SCM tool.

2. Only load the relevant bits (my opinion is it’s sometimes ok)

You could just load the child modules you’re currently working on and then unload (remember to delete local files or you hit a bug) and then reload the parent when you want to do a “big” parent build. Frankly this is unacceptable from a developers perspective and will fail anyway if there’s a dependency on the parent from the children. Although in theory this may be frowned upon it’s quite common and so needs supporting by the scm tool.

3. Load then import, with auto-refresh (my preferred option)

This one is going to be more than one paragraph….

Prediction fulfilled ;p Anyway… Currently your developers may be used to a scm system where they load/copy/access files from the scm system on the local filesystem and then do an Eclipse -> Import Existing Maven projects to get all of the various goodies.

If you import the multi-module project as above but with a root folder (as an eclipse project) so it has the following structure:

O Product X

.project

- pom.xml modules)

- various files at the root of ProductX

O ModuleA

- pom.xml (for ModuleA)

- various ModuleA root files

O various ModuleA folders of stuff

O Module B

- pom.xml (for ModuleB)

- various ModuleB root files

O various ModuleB folders of stuff

Then when you do a load from Jazz SCM you’ll get in the Eclipse Project Explorer (or equivalent) ProductX correctly detected as a Maven project (or an Eclipse project with a Maven nature for the pedantic amongst you).

You can then do a normal Eclipse -> Import Existing Maven Projects based on your workspace/ProductX/… (taking care not to select “Copy projects into workspace”) You’ll then get in Project Explorer (or whatever):

ProductX (info about rtc scm links)

ModuleA

ModuleB

In this view ModuleA and ModuleB exist in two places: as projects referenced in the workspace but apparently outside of RTC SCM control; as a folder structure under ProductX/ModuleA and ProductX/ModuleB.

This has the following advantages:

• Cross-project dependencies resolve
• You can run the parent/uber pom goals and the child pom goals without reloading anything
• You can shift focus between the child-modules and parent structure without reloading anything

If you edit files in ModuleA and ModuleB under the folder hierarchy of ProductX then Pending Changes will keep track in the normal way but if you edit a file under the apparent “root level” ModuleA and ModuleB then Pending Changes won’t unless you do a “deep refresh”.

Alternatively just go to Eclipse -> Window -> Preferences ->  General -> Workspaces -> Refresh automatically (er… I think, I’m doing this from memory, hence the lack of screenshots). You can then add “edit files tracked by Pending Changes at either Product or child module level” to the advantages above.

So this option sounds great so far, what are the downsides?

This solution maintains the hierarchical structure between ProductX and it’s child modules and as such constrains everything from ProductX downwards to be in the same Jazz SCM component as eclipse projects can only be seen (without far too much fiddling) if they’re root folders in components. This may not seem like such a big deal but components are the lowest level of baseline (or label) in Jazz SCM. In a tightly coupled cohesive single architecture this might not be a problem but if you want to reuse any of these modules (at a code level) or have a separate ownership it will be a problem. It’s also very common to have dependencies from child modules to parent projects (which may initially seem to resolve if they’re in the local ~/.m2 repository).

Having said that source code dependencies between components lead to brittle architectures, binary dependencies are far superior, and made much easier by the use of things like Ivy/Maven and Nexus.

Other considerations

The point above above binary vs. code dependencies can be argued until the cows come home and go out again (an extended metaphor), especially if you consider project/product/team/release boundaries. I’m not going to pretend to answer those with an academic silver bullet here, the important thing I hope you take away from this blog is that there are pros and cons to all of these approaches, balancing those with the needs of your team/architecture/project/organisation are not easy. Shameless plug: Give me a call if you need help

Eclipse refactoring isn’t dealt with well by Jazz SCM, especially between Eclipse projects in and “out” of source control (advocated by option 3 above). Expect a lot of “remove/add” combinations if you are doing anything more than a trivial set of file changes. This is also a problem for the ClearCase <-> RTC automatic synchroniser developed by the excellent Samecs however I know that I, others and Samecs are pushing for improvements on this from IBM.

This post has been very scm focussed but what about build and release? The RTC build engine understands maven to a limited degree and can easily invoke maven builds for information on integating the maven release process see How to do maven releases with Jazz SCM

An agile approach to innovation in a small company

I recently spent a little while thinking about an open, transparent and honest way to deal with innovation/staff objectives/supporting work in a small company. Note that this isn’t what my current employer does, just an idea for how these things can be done.

Firstly people need time to do this stuff, if it’s valuable for your organisation to spend time doing supporting work, intellectual property development or general innovation then they need the time to do it. People need the Google 20%. I’m very fortunate that in my job I get that time, or close to it normally.

New Ideas:

Create a wiki page or similar somewhere (that everyone can see and everyone can edit) – a bit like the Google ideas wall. Start it with a freeform/categorised ideas list in the form, this is a bit like a backlog, once an ideas done/delivered/published it drops off this list

Ideas – Owner – State:

Idea for doing something – Mike – Identified

Idea for taking over the world – Unassigned – Identified

If someone like one of these ideas they can run with it, recruit a team of colleagues if necessary and get on with it. Once I’ve asked someone to help with my idea I can be fairly sure they’ll ask for reciprocal help on something. This improves teaming in distributed companies and provides a channel for autonomous action, critical for motivation.

In Progress Ideas:

When ideas are in progress they’ll have their own tracking and information in the appropriate technology. For example a blog idea probably goes from being picked up to being published without any tracking. Maybe there’s a review process if it’s a big position piece so it might go through Identified to In Review on the list above but that’s it. Another idea might be for the development of some software, in which case it’ll take some time and probably have a project web and bunch of tools and info to communicate.

Complete bits of work:

Once something has been done the owner can add it to the achievements table, a column for each person where they can list what they’ve done with links to the various bits of output.

Person A	Person B	Person C
Blah Practice published	Blog series on X
	Paper published on Y

All bits of achievement aren’t the same size, but this allows people to see the creativity and production of their colleagues and find all of the useful things people have been doing. It also applies some peer pressure to Person C who on the face of it doesn’t appear to be adding much, although there could be a good reason for that.

Why did I say all this was agile?

Individuals and interactions over processes and tools

Individuals ideas are valued and encouraged, as are their achievements. People are encouraged to help each other and work together on ideas. There’s very little process and tooling involved, just a list on a wiki. If I had a co-located team I’d just use some whiteboard paint on a wall or some post-its.

Working software over comprehensive documentation

Well we’re not talking about software here, but achieving ideas. The emphasis is on the achievements table, not the detail of how people go there. In progress ideas can be tracked and reported using whatever info is relevant for the type of thing being done, which can be determined by the people doing it.

Customer collaboration over contract negotiation

This is an open and transparent process. Anyone can comment on the ideas on the list have a look at in progress ideas or question published/achieved goals.

Responding to change over following a plan

All this stuff is just written down, it would be easy to rewrite it if things changed. People, ideas, the environment etc. will all change so we need to be responsive to too and not plan the next 50 ideas to the hour.

Other thoughts

Since the list at the top is a bit like a backlog, it could be managed with input from management to direct the backlog, prioritise and agree. But I’d be wary of applying that kind of governance to creative/innovative ideas. I’m not proposing this as a general work management process, just an ideas/ancillary objectives process. Remember, bad management kills great ideas and worse this kind of management intervention stifles bad ideas, which are necessary and valuable.

Also, there are few things worse in life than an “Ideas Process”

RSC 2009 – What to expect

I’m not attending the IBM Rational Software Conference (what used to be RSDC, RUC etc. etc.) this year however since it started today for partners and tomorrow for delegates so I thought I’d cast my opinion on what to expect this year. Of course I could be wrong since it’s just assumption but I suspect the main themes will be:

A vision of one notion of process for all Rational tools, made explicit in the relevant tools, surfaced through process rules and defined in practice libraries allowing easily composable and customisable processes based on the Jazz Process architectural elements.

So to tie  it all together the Process Library (which of course is practice based probably driven by Rational Asset Manager) feeds definition of processes in RMC, which is measured by MCIF and delivered by the Rational Software Development Platform. So what does that really mean:

Rational Team Concert 2 beta – because it’s a lot better than RTC 1 which was pretty good to begin with!

Rational Tara -  a Jazz based PPM (Project Portfolio Management) tool. Think MS Project by IBM on Jazz with better Resource Management stuff. It’s part of the replacement suite for Rational Portfolio Manager (RPM).  Should be interesting to see how it ties into the other products (inc. some of the Telelogic ones). I expect this one to be big news – it should be. Gantts, bubble charts, resource utilisation views, agile dashboards etc. will abound. (Now known as Focal Point for Project Management)

Rational Insight – used to be codenamed Vega and is a data warehousing and RTL solution on top of Jazz. Cross project reporting goodness. Of course I’m sure there will also be integration through to the “classic” rational tools. Interestingly I read a bit of blurb recently from somwhere that referred to their old tools as the “classic rational toolset” I wonder if they borrowed that from me

MCIF – Measured Capability Improvement Framework – It’s been around for a while but with more emphasis from IBM now on practice based process the story between a more practice orientated Rational Method Composer, Health Assessment tools and Self Check tools is a bit tighter.

Telelogic – I expect there to be some announcements about the integration of the Rational and Telelogic brands. Especially around FocalPoint.

I’d also expect some stuff on other Jazz based tools like Requirements Composer, Quality Manager, Ensemble, Governor and stuff like that.

As I said due to other commitments I’m not there myself, but if you are there then please stop by the Ivar Jacobson International stand in the Exhibit Hall downstairs when they let you in. Grab a free beer and talk to my colleagues about how to achieve tangible process improvement with the use of effective practices in a proven framework. Also talk to them about achieving “Agile for Grown Ups” (TM of the MacDaddy) in your organisation. Ivar, Ian Spence, Kurt Bittner are all speaking at the conference and a bunch of other IJIers will be there so go along and say hi.

And ask for a free bag, they’re great for carrying around your exhibit hall swag

Simple Rules for web site design

• Frequency of web content change is inversely proportional to publishing control
• Frequency of web content change is proportional to number of people able to publish
• Content quality is inversely proportional to frequency of change
• Engagement is proportional to ability to comment and track replies

Thoughts?

How to avoid Fragile Agile, Flexibility in Context

A presentation I gave at the UK RUG Annual Meeting on what can make agile development fragile and how to avoid that fragility.

Unfortunately animations don’t work and slideshare’s screwed up the agenda slides but you can still follow it

How to avoid Fragile Agile

So I’ve not blogged for a while. It’s twitter’s fault. Well that and I’ve been super busy with some other projects and work stuff that isn’t blog worthy.

So what is blog worthy?  I’m giving a talk at the UK RUG next week called “How to avoid Fragile Agile” subtitled “Flexibility in Context”. I can’t take credit for the super title, one of my colleagues came up with the name, but it should be a good talk. It includes content from various IJI folk so come along and say hi if you’re there.

http://www.rational-ug.org/groups.php?groupid=20

Here come the slides from my presentation

Why do people still waterfall their development process?

Generally the software development industry has caught on to the fact that iterative development is a good thing. Only people that don’t believe in common sense think it’s a great idea to leave all of your testing to the end and hope for the best. With that in mind then why do people still have a waterfall mentality to their software development processes?

At the beginning of a project you can’t possibly know everything about it, that’s one of the many reasons it makes sense to iterate, so you can do some work and then refine your plans based on your work. For some odd reason though a lot of projects, even though they might develop iteratively, seem to think they should define their process at the beginning of a project in great detail and then stick to it regardless of the facts due to some misguided need for process alignment or similar.

That attitude just doesn’t make sense. A view of what practices will be followed is necessary at the beginning so that everyone’s on the same page and now what they’re meant to be doing, but even the formally defined iterative processes like RUP say that the process should be examined and refined each iteration.

In my experience an extremely detailed formal process is difficult to change. If you’ve got a 50 page development case. huge stack of xml files in your favourite process authoring tool or similar it’s very hard to make useful changes to a process. Of course you can change the review requirements of the Use Case Specifications from Formal-Internal to Formal-External, but it’s not so easy to scale down the formality of business modelling while adding a practice to address user experience issues.

Software development process can (and should) be thought of as a collection of practices. Each practice should address a particular aspect of software development in a complete fashion. Every small technique is not a practice (in coding terms every idiom is not a pattern). Using Use Cases, being Iterative,  using Component Based Development are all good examples of practices. They can be added or removed to your project’s way of working and the level of formality can be scaled up or down on each practice. Practices are the natural unit of adoption for processes.

It’s always best to start small and simple. At the beginning of a project just do what you can see. It might initially make sense to think about using Use Cases as a common way of managing requirements for your project and it clearly makes sense to iterate in most cases. Then as the project progresses and more understanding is reached more practices can be added as necessary. This is an iterative approach to software development process, it’s just common sense.

At IJI we’ve been doing this sort of practice based way or working for ages – in fact it’s what the company was created around as the fine brains we have created this approach and brought it to life in EssWork and EssUP which is a kernel and a collection of practices.

Anyway, enough of the company blurb, more about practices. I think everyone can agree that this is a sensible approach to software development as it applies the lessons the industry has learnt about software development back onto the problem. It doesn’t matter which vendors tools are used, or what process frameworks are used this is just a sensible approach, and the best thing is it is compatible with whatever you’re already doing that is successful or necessary in your organisation. I work with organisations helping them to change and adopt their processes and in every organisation I’ve been to is doing at least one thing well. Using practices to work with process improvement means that we can just add to what’s already working, incorporate the necessary governance or operational activities in an organisation, add SCRUM if that’s what’s wanted and have it all work together in a cohesive way of working.

To achieve this adding together of practices in a way that works you need to have a foundation that provides a common understanding of the basic concepts of software development so that practices can fit together neatly. This is intellectually quite tricky to achieve but necessary. It should also be practically invisible to the people doing projects – they don’t want to think about process geeky things like the process architecture that can support practice composition in a seamless way. At IJI we call this the kernel – which I think is a suitably geeky term. We have also built it both conceptually and in software (that’s what EssWork is) along with what we think are the “essential practices” (that’s what EssUP is) and now the big vendors are following suit and packaging their processes this way.

Maybe one day we can get away from branded processes by vendors and just have a software development community using, sharing and collaborating on practices.

Oh yeah, and they should be in tools not websites, and free!

Microsoft VSTS, Camano and UML

I recently had the pleasure of going to visit the Microsoft Technology Centre to learn about all things MS for software engineering. Having long been a Visual Studio user and a VSTS (Visual Studio Team System) user since it turned up I was really pleased to see VSTS 2008 and some interesting stuff coming in 2010.

VSTS seems to be really maturing and now has a nive new non-sharepoint web UI. It will be interesting to see what happens in terms of Microsoft VSTS vs. IBM Jazz. I’ve written about Jazz before on this blog and haven’t gotten around to a detailed comparison yet but I was excited to see how mature VSTS is now looking and how nice it is to use. It’s designed to make developing easier and use the tools people already have in many cases. For example, using MS Project, Excel, Word, SharePoint etc.

Things of special note were the automated testing support (including coming soon recording and playback of GUIs other than web), the upcoming cool manual test runner, Camano for test management and test execution by proper testers not just developers doing unit tests etc. etc. Also checking out the project + project server for portfolio and project management views was interesting.

What I really really liked are the VSTS 2010 architecture tools. Microsoft has seen the light and embraced UML + DSLs (or DSLs + UML if you prefer) as the pragmatic modelling solution. I really like the idea of being able to do just enough  modelling, using the most appropriate language such as a high level architecture diagram that then imposes constraints on the code through things like check in policies!

I think this stuff is going to be really powerful and I can’t wait to get my hands on it properly (and somehow conjure the time from somewhere) to do some detailed analysis and also a comparison against competing products/toolsets.

Of course if you are using Visual Studio (or Eclipse) you can still use EssWork to use practices and processes directly inside your IDE

EssWork in Visual Studio