Mike MacDonagh's Blog

Somewhere in the overlap between software development, process improvement and psychology

Tag Archives: Jazz

scm checkin and deliver script for IBM Jazz RTC SCM

Following on from a previous post where I covered how to load some content from a Rational Team Concert server in a single command here’s a script I use to checkin and deliver in one command back to the stream. As before this is just a wrapper around the standard command line and maybe more appropriate for the more occasional user rather than a cli loving geek but I’m both so here it is!

Also available on github easy cloning with SimpleGit

#!/bin/bash

echo ""
echo "Checks in and delivers current workspace changes against a workitem"
echo "Usage: scmdeliver <workitem_number>"
echo ""

workitem=$1
error=0

if [ -z "$1" ]
then
  echo "Error: Please specify a workitem id"
  error=1
fi

if [ $error -eq 0 ]
then
  echo "Checking in..."
  checkin_str=`scm checkin * | grep "<No comment>"`
  change_id=`echo $checkin_str | cut -d "(" -f2 | cut -d ")" -f1`
  echo "Checked local changes into changeset $change_id"
  echo "Associating with work item $workitem"
  scm changeset associate $change_id $workitem
  echo "Delivering..."
  scm deliver
fi

So using this script and the scmload script in the previous post you can do this to hack some code:

scmload "Stream A" "Component A"

# Make a bunch of changes

scmdeliver 1234

Simplez!

Note this script assumes a clean workspace before you start making edits, if you want it to work with a previously created changeset you’ll need to tweak it.

Unfortunately you can’t find out your workitems from the command line but that’s a problem for IBM to fix!
Checkout my RTC Masterclasses and my advanced streaming guide:

  1. Development, Integration and Release Streams
  2. Streams for parallel branched development
  3. Streams for component reuse

Process vs. Tools

What’s more important: Process or tools?

I’ve sat through numerous arguments with people trying to work out whether process or tools are more important to their software development effort. The arguments go along the lines of:

Process Geek: Tools are just the things you use to do the process, the value is in the process. Let’s do a process first roll out.

Tools Geek: Tools are the things people actually use. Process is academic and is constrained by the tool features anyway. Let’s do a tools based roll out.

Sadly both of these are real life quotes from people I’ve worked around in the past. I won’t embarrass them by naming them because they’re both really wrong. For me the real value is not in the tools or the process but in the people. Process and tools are something that you add if the people need them.

Minimal Tooling

For me the best planning tool is a whiteboard. The best architecture and design tool is also a whiteboard. I like to capture some info about what I’m doing and how close I am to achieving my goals, some really simple measures. The best tool for that is probably Excel, despite my love of Open Source. Other than that I need to manage some lists of things like requirements (stories/use cases whatever), defects and whatever other bits the team thinks are important. They might fit on a whiteboard but typically we want to have some attributes on them, be able to sort them and link them. The one downside of a white board is the lack of drag and drop. I’ve used a few different types of smart board and active touch screens but none of them have really given me the smooth experience I want.

The one bit of tooling I really need is my IDE + SCM + Continuous Integration.

Minimal Process

This is potentially a can of worms if I go into things like process kernels and ideological arguments about different agile approaches and practice collections. There’s no such thing as a process silver bullet so for me though it’s all about the team and being effective. Teams need to:

  • Have Clear Obligations
  • Have Clear Business Priorities
  • Seize their autonomy
  • Have the relevant technical skills and resource availability
  • Be active in actually doing their work (seems obvious but it’s something a lot of processes seem to hinder rather than help)
  • Demonstrate visible progress (via working software not a tracking gantt chart)
  • Continuously improve themselves (because no way of working is perfect)

This list based on the 7 Habits of Agile Teams by Roly Stimson

So why go more complicated?

Unfortunately life is rarely simple. Many teams work in complex environments where a number of factors might make them choose to need more than this kind of minimal set including things like physical distribution; audit, regulatory or security constraints; cross-team interaction in system of systems environments, more people etc.

When faced with increased complexity it might be appropriate to add more powerful tooling or development practices. The more people involved in a project the harder it is to herd all of those socially complex cats into working together.

This is why ALM suites exist ranging from plugging together some open source bits like git, SimpleGit, jenkins, agilefant, bugzilla etc. to commercial vendor products like IBM Rational RTC (which does agile planning, work item management, scm and build in one tool) and other IBM Rational Jazz tools – it’s worth noting that you can also get RTC for free for just 10 users or for academic/research use on jazzhub.

Alternatively there’s Microsoft TFS for source control, work item tracking etc. tying together existing Microsoft stuff like MS Office, Visual Studio and sharepoint if you’re a more Microsoft focussed development house.

Of course you can also mix and match this stuff to a greater or lesser degree with various integrations. No toolset is a silver bullet however and each have their own problems.

So what’s my point?

My point is that process and tools need to reinforce each other, that both should be selected based on the needs of the team and the environment the team has to operate in. In the process vs. tools argument both lose out to the people. People in a development organisation create value, they just use tools and process to do it.

How to do maven releases with Jazz SCM

I’ve previously blogged on structuring maven multi-module projects to work well with m2, eclipse and Jazz SCM. In this post I’m going to write about how to do releases because it’s not as straight forward as you might hope and expect. This blog is explicitly about using maven 2.2 and IBM Rational Team Concert 3.0.1.2 although it will apply to various other previous versions. Later versions of RTC should (I hope) alleviate some of the problems and the Jazz SCM Maaven plugin is being actively work on.

Overview

During development or maven projects we start with a label such as 1.0.0-SNAPSHOT which reflects the current development build. Over time the development snapshot might be occasionally pushed to a maven repository as a “snapshot release”  if it’s roughly stable so that dependant teams can get the dev version if they really want it. In time development gets stable enough to release the definitive 1.0.0 so the -SNAPSHOT is stripped from the label and the version is pushed to the repository as a proper “release” for other teams to depend on. At this point a new number is selected and development continues on the new label such as 1.0.1-SNAPSHOT which might be occasionally pushed etc.

Diagram showing how releases and version numbers work with maven

The problem

This process works pretty smoothly in a lot of contexts as the release is done by issuing a command line as simple as mvn release and maven’s smart enough to work out the rest. The problem comes in the context of Jazz SCM for a couple of reasons:

  1. The Jazz Maven SCM plugin is still in early development so doesn’t do everything well and some things not at all
  2. The Jazz Maven SCM plugin incompatible with the maven release plugin because it requires the pom scm tags to be written in a way that the release plugin can’t read
  3. The jazz scm plugin insists on outputting your password in cleartext all over the logs so you can’t do it as part of an RTC build without exposing the build user password (which depending on your organisation might not matter or could be a huge deal)
  4. Both maven and rtc use the word snapshot for different purposes and other terms like “baseline” and “tag” are synonymous

The solution

Write a little script to work around the problems with the following requirements:

  • All published version numbers in the maven repository must be reflected by RTC scm snapshots
  • The process of releasing is simple to use

To make this work I’m actually dumping the use of the Jazz Maven SCM plugin as it’s not yet developed enough to be useful. Instead since every build happens in RTC occurs in a snapshot here’s the process:

  1. Green Build in RTC (optionally promoted to RTC Release) with a known snapshot name e.g. v1.0.1 release
  2. Integrator/Releaser/Committer/Nominated dev runs custom mvn-release script on their local machine
  3. Sits back and watches, feels good

Here’s the pseudo-code for how to do it all. One of my clients is developing this currently and I’m reasonably optimistic that they might open source (or allow me to) the full script. However if you’ve got a bash monkey working for you it shouldn’t be hard to develop this up pretty quickly.

Currently this process needs some parameters which could be passed in or the user could be prompted. I prefer to pass in and prompt so that the script can be invoked from a remote process (like a build server) passing in properties or can be interactive if a user invokes it. Parameters required at least somewhere along the way are:

  • rtc user name and password
  • is this a snapshot release or proper release
  • rtc build snapshot
  • new version number to base label (if doing a proper release)

CURRENT_VERSION = getCurrentVersion(pom.xml) //1.0.0-SNAPSHOT

//If just doing a basic snapshot release
scm create snapshot "CURRENT_VERSION"
scm checkin & deliver
mvn deploy

//If doing a full release
PROJECT_NAME = getCurrentProjectName() //basename of current directory
NEW_WKS_NAME = create new rtc workspace in /tmp/rtc/workspace/RTC_SNAPSHOT_NAME/
load temp workspace with contents of RTC_SNAPSHOT_NAME
cd /tmp/rtc/workspace/RTC_SNAPSHOT_NAME/PROJECT_NAME
//Generate Release
RELEASE_VERSION = StripSnapshot(CURRENT_VERSION) //1.0.0
mvn versions:set -DnewVersion="RELEASE_VERSION"
scm create snapshot "RELEASE_VERSION Release"
scm checkin and deliver
mvn deploy

//Generate new development starting point
NEW_VERSION = AskUser(); //1.0.1-SNAPSHOT
mvn versions:set -DnewVersion="NEW_VERSION"
scm create snapshot "NEW_VERSION Initial baseline"
scm checkin and deliver
//don't do a mvn deploy as that would be the user choosing to run this script choosing the snapshot-release option

//Tidy up
scm delete workspace NEW_WKS_NAME

The advantage of running this script creating a temporary local workspace and running from there is that you can do this from the command line without closing eclipse (or invoke from Eclipse as an external tool configuration). Also it isolates the release process from any possible local changes in the /Releaser/Committer/Nominated dev’s workspace.

Until the Maven SCM plugin comes of age I hope this helps.

Advanced RTC SCM Streaming – Part 3 Component reuse

Basic usage of RTC Jazz SCM is easy enough but there’s what about more advanced usage? Using streams to isolate development from integration? Cross-project component reuse? Parallel branched development? This is the guide for that stuff.

Feel free to contact me with questions or just use the comments section below – I can also offer RTC Masterclasses covering this stuff and more.

This blog is in three parts:

  1. Development, Integration and Release Streams
  2. Streams for parallel branched development
  3. Streams for component reuse

Part 1 also covers the introduction and overall conclusions

Working with components across multiple RTC projects can present a bit of a challenge. It’s simple enough to create a stream with any baseline of any component in your repository but dealing with changes to those common components is not simple or obvious, especially if you need different visibility controls and access rights for different teams on different components.

Read more of this post

Advanced RTC SCM Streaming – Part 2 Parallel branched development

Basic usage of RTC Jazz SCM is easy enough but there’s what about more advanced usage? Using streams to isolate development from integration? Cross-project component reuse? Parallel branched development? This is the guide for that stuff.

Feel free to contact me with questions or just use the comments section below – I can also offer RTC Masterclasses covering this stuff and more.

This blog is in three parts:

  1. Development, Integration and Release Streams
  2. Streams for parallel branched development
  3. Streams for component reuse

Part 1 also covers the introduction and overall conclusions

Change set based SCM systems like git (SimpleGit)and RTC do away with the need for a lot of branching. You don’t need to branch to get developer isolation, to gather file changes together to understand them as an atomic set of changes related to a bit of work or to manage concurrent development of the same files on a small scale. You get all of that for free by using change sets. Developers are always in their own private personal branch separated from the rest of the team but unified by a common stream or repository. However sometimes it’s still appropriate to branch, this blog covers some of the situations when it’s appropriate and how to do it effectively in RTC Jazz SCM.

Read more of this post

Advanced RTC SCM Streaming – Part 1 Development, Integration and Release Streams

Basic usage of RTC Jazz SCM is easy enough but what about more advanced usage? Using streams to isolate development from integration? Cross-project component reuse? Parallel branched development? This is the guide for that stuff.

Feel free to contact me with questions or just use the comments section below – I can also offer RTC Masterclasses covering this stuff and more.

This blog is in three parts:

  1. Development, Integration and Release Streams
  2. Streams for parallel branched development
  3. Streams for component reuse

Part 1 also covers the introduction and overall conclusions

Read more of this post

Working with Maven multi-module projects in RTC Jazz SCM with m2

I’ve been working with Maven projects with m2 for a while in RTC Jazz SCM so thought I’d post some notes. For reference this post covers using RTC v3.x things may change in the future. If you’re not using m2 then I’d suggest you read this blog by Philippe Krief.

First of all it’s important to understand that RTC v3 has limited support for Maven. You can setup a build definition which can call a maven goal (such as a local install) easily enough but that’s about it. The RTC build mechanism is basically a cron job with a nice web (like Hudson/Jenkins without the plugins) and Eclipse front end, it doesn’t have a deep understanding of Maven.

The whole point of Maven over Ant is that it does a lot by convention rather than specification, however in most advanced Maven project I’ve seen pom files are as big as equivalent Ant + Ivy files. Regardless I still think Maven with the excellent Sonatype Nexus is awesome (although I also like the structured bash/batch like nature of Ant as well).

Most serious Maven projects are multi-module. This is because component based development is a good thing (shameless employer plug, but it’s true, it is a good thing). Most (tending to all) multi-module projects have the following structure where “O” is a folder and “-” is a file, considering a hypothetical ProductX:

- pom.xml (the parent pom for the master/uber/product build, lists child modules)

- various files at the root of ProductX

O ModuleA

- pom.xml (for ModuleA)

- various ModuleA root files

O various ModuleA folders of stuff (such as src)

O Module B

- pom.xml (for ModuleB)

- various ModuleB root files

O various ModuleB folders of stuff

If you want to work with this stuff in RTC Jazz SCM then you need to balance the constraints of Maven m2, RTC and Eclipse. If you’re considering doing this then you’ll probably be considering moving existing Maven projects to RTC, in which case you may find a previous blog on a manual migration script useful. The only edit to this I make for Maven is to separately import the parent from the children if flattening the structure (although in most cases I wouldn’t recommend flattening).

The constraints playing against each other are:

  • Maven and plugins expect sub-modules to be in child directories as in the tree above
  • RTC doesn’t allow both an Eclipse project and root files to be at the root of a component
  • The simple case of Eclipse and RTC SCM interacting during a load from the server restricts Eclipse to only seeing projects which are represented as folders at the root of the local Eclipse workspace containing .project files.
  • Eclipse doesn’t really support parent and child projects.

I’ve found there are a number of approaches, none are perfect:

1. Flatten the structure (my opinion the worst option)

You can edit the parent pom.xml to reference the sub-modules as ../ModuleA etc. although the Maven release plugin (amongst others) don’t play well with this. Also, and somewhat fundamentally, you shouldn’t have to restructure your code based on your SCM tool.

2. Only load the relevant bits (my opinion is it’s sometimes ok)

You could just load the child modules you’re currently working on and then unload (remember to delete local files or you hit a bug) and then reload the parent when you want to do a “big” parent build. Frankly this is unacceptable from a developers perspective and will fail anyway if there’s a dependency on the parent from the children. Although in theory this may be frowned upon it’s quite common and so needs supporting by the scm tool.

3. Load then import, with auto-refresh (my preferred option)

This one is going to be more than one paragraph….

Prediction fulfilled ;p Anyway… Currently your developers may be used to a scm system where they load/copy/access files from the scm system on the local filesystem and then do an Eclipse -> Import Existing Maven projects to get all of the various goodies.

If you import the multi-module project as above but with a root folder (as an eclipse project) so it has the following structure:

O Product X

.project

- pom.xml modules)

- various files at the root of ProductX

O ModuleA

- pom.xml (for ModuleA)

- various ModuleA root files

O various ModuleA folders of stuff

O Module B

- pom.xml (for ModuleB)

- various ModuleB root files

O various ModuleB folders of stuff

Then when you do a load from Jazz SCM you’ll get in the Eclipse Project Explorer (or equivalent) ProductX correctly detected as a Maven project (or an Eclipse project with a Maven nature for the pedantic amongst you).

You can then do a normal Eclipse -> Import Existing Maven Projects based on your workspace/ProductX/… (taking care not to select “Copy projects into workspace”) You’ll then get in Project Explorer (or whatever):

ProductX (info about rtc scm links)

ModuleA

ModuleB

In this view ModuleA and ModuleB exist in two places: as projects referenced in the workspace but apparently outside of RTC SCM control; as a folder structure under ProductX/ModuleA and ProductX/ModuleB.

This has the following advantages:

  • Cross-project dependencies resolve
  • You can run the parent/uber pom goals and the child pom goals without reloading anything
  • You can shift focus between the child-modules and parent structure without reloading anything

If you edit files in ModuleA and ModuleB under the folder hierarchy of ProductX then Pending Changes will keep track in the normal way but if you edit a file under the apparent “root level” ModuleA and ModuleB then Pending Changes won’t unless you do a “deep refresh”.

Alternatively just go to Eclipse -> Window -> Preferences ->  General -> Workspaces -> Refresh automatically (er… I think, I’m doing this from memory, hence the lack of screenshots). You can then add “edit files tracked by Pending Changes at either Product or child module level” to the advantages above.

So this option sounds great so far, what are the downsides?

This solution maintains the hierarchical structure between ProductX and it’s child modules and as such constrains everything from ProductX downwards to be in the same Jazz SCM component as eclipse projects can only be seen (without far too much fiddling) if they’re root folders in components. This may not seem like such a big deal but components are the lowest level of baseline (or label) in Jazz SCM. In a tightly coupled cohesive single architecture this might not be a problem but if you want to reuse any of these modules (at a code level) or have a separate ownership it will be a problem. It’s also very common to have dependencies from child modules to parent projects (which may initially seem to resolve if they’re in the local ~/.m2 repository).

Having said that source code dependencies between components lead to brittle architectures, binary dependencies are far superior, and made much easier by the use of things like Ivy/Maven and Nexus.

Other considerations

The point above above binary vs. code dependencies can be argued until the cows come home and go out again (an extended metaphor), especially if you consider project/product/team/release boundaries. I’m not going to pretend to answer those with an academic silver bullet here, the important thing I hope you take away from this blog is that there are pros and cons to all of these approaches, balancing those with the needs of your team/architecture/project/organisation are not easy. Shameless plug: Give me a call if you need help

Eclipse refactoring isn’t dealt with well by Jazz SCM, especially between Eclipse projects in and “out” of source control (advocated by option 3 above). Expect a lot of “remove/add” combinations if you are doing anything more than a trivial set of file changes. This is also a problem for the ClearCase <-> RTC automatic synchroniser developed by the excellent Samecs however I know that I, others and Samecs are pushing for improvements on this from IBM.

This post has been very scm focussed but what about build and release? The RTC build engine understands maven to a limited degree and can easily invoke maven builds for information on integating the maven release process see How to do maven releases with Jazz SCM

scmload script for IBM Jazz SCM

A common complaint by command line users of the IBM Rational Team Concert (RTC) Software Configuration Management (SCM) command line is that it’s too complicated to just load some stuff. And they’re right, especially as compared to something like git. To work around this I wrote a little bash script which you’re welcome to use/edit/abuse to load stuff in a single command.

Note that this is for the occasional user, if you’re a command line user by nature then just use the normal scm command line, it’s quite easy when you’re used to it. Using this script will involve the creation of an RTC repository workspace, since at the time of writing this can’t be removed using the command line you may wish to tidy them up using the GUI. The follow on to this is my scmdeliver script.

The script (available via github) easy cloning with SimpleGit:

#!/bin/bash

rtc_server="https://your_server:9443/ccm"

echo ""
echo "Loads the contents of a component from a Jazz SCM stream to the current directory"
echo 'Usage: scmload "&lt;stream_name&gt;" "&lt;component_name&gt;"'
echo ""

stream=$1
comp=$2
error=0

if [ -z "$1" ]
then
echo "Error: Please specify a stream"
error=1
fi

if [ -z "$2" ]
then
echo "Error: Please specify a component"
error=1
fi

#generate a hopefully unique workspace name
workspace=my_cmdline_wks_$stream

if [ $error -eq 0 ]
then
echo "Loading stream \"$stream\" component \"$comp\" into local workspace"
scm login -c -r $rtc_server -n rtclive
wks_created_str='scm create workspace -s "$stream" "$workspace" -r rtclive'
echo $wks_created_str
wks_success='echo "$wks_created_str" | grep "successfully created"'
if [ -n "$wks_success" ]
then
wks_id='echo $wks_created_str | cut -d "(" -f2 | cut -d ")" -f1'
scm load $wks_id "$comp" -r rtclive
fi
fi

echo ""

Like I said this only automates a fairly simple set of command, but for a novice user it might be helpful as it takes parameters and wotnot. Here’s what it would look like using the plain scm cli:

scm login -c -r https://your_server:9443/ccm -n rtclive
scm create workspace -s "Stream A" "My New Workspace" -r rtclive
scm load "My New Workspace" "Component A" -r rtclive

Using this script you can load Component A from Stream A like this:

scmload "Stream A" "Component A"

Simplez!

Checkout my RTC Masterclasses and my advanced streaming guide:

  1. Development, Integration and Release Streams
  2. Streams for parallel branched development
  3. Streams for component reuse

RTC: How to write cross project queries

Ok, so I’ve previously said that you can’t write cross-project work item queries in IBM Rational Team Concert but that’s not entirely true… You can, but it’s a bit “hacky” and isn’t very supportable. In Eclipse go to:

Team Artifacts -> My Team Areas node, select a user -> right click -> Show Recent Work

This seems to create a work item query that has the user as a top level scoping element rather than the Project Area, you can even add the Project Area column to the result set :) You can edit this query in the normal way but it seems to save itself to a project area selected by rolling a dice and there’s no way to control which one as far as I can see. Also, remember that it’s top level scope is the user that the query was generated from, that may have interesting effects.

Simples.

Don’t forget to demand proper cross-project querying from IBM on jazz.net

How to migrate source to Jazz RTC SCM

First of all, although it might seem tempting to seperate scm and build, especially from a business change and planning perspective you just can’t (unless you’re writing scripting languages!). Software is just useless text if it can’t be built/run and a software team can’t be expected to move where they store their source without ensuring that build doesn’t get broken. In a perfect world build scripts would be generic and work regardless of the scm system they’re coming from but that generally isn’t the case due to this not being a perfect world:

  • different scm systems/ides imposing different structural constraints
  • absolute paths to files
  • dodgy use of symlinks
  • lack of parameterization of build scripts
  • complexity of numerous build technologies

This means that when migrating a team into Jazz SCM you need to at least ensure that their current builds work in their current normal way. Ideally it would be great to (re)engineer them to word with RTC Build but that isn’t always practical. This build refactoring shouldn’t be underestimated as it can take a while because of general build complexity, project complexity and technology complexity. It’s not unusual to find in just one organisation the use of: ant, maven, junit, ivy, make, clearmake, msbuild sometimes all on the same “project”. Add unit testing, style checking, static analysis and other things that people tend to add into the build process and you’re looking at a large amount of technical skill needed to even begin to make sense of what’s currently going on.

For this reason when moving a team from cm system X to RTC Jazz SCM I find it necessary to build the stream in 2 parts, which means you need a repeatable migration script that doesn’t get tripped up by things like permission or line end delimiter platform differences.

My general migration script (assuming not synchronizing or bridging):

  1.     Label/Baseline/Shapshot source files in old scm system
  2.     Using linux copy source files from existing scm/view/folder to a holding area
  3.     chown if necessary and chmod files to u:rw(x),o—,g—
  4.     I tend to do a dos2unix conversion on appropriate files to normalise mixed mode files
  5.     Share to RTC
  6.     Baseline/Snapshot in RTC
  7.     Make builds work (generally a process of changing absolute path references to local relative paths – often “/view/blah/…/proj” to “../” (here be technical complexity)
  8.     Baseline/Snapshot in RTC (“Initial scm migration with working build”)

There is a great advantage in following this process (which is why I’ve scripted steps 1-4 although unfortunately I can’t share that) somewhat slavishly in that it is repeatable to update a file set as a change set. If you do this to create an example stream for a team to experiment and gain confidence in while they’re still developing in their existing system, then when they choose to adopt you can simply update their stream(s) by:

  1.     [Optionally] revert to working build snapshot “Initial scm migration with working build” from step 8 above
  2.     Repeat steps 1-4 above
  3.     Copy files in holding area to RTC workspace
  4.     In RTC Pending Changes view drop-down the refresh button to select “Refresh Sandbox and Remote Changes”

You then get a change set showing the file changes since the experimental migration and the present time, simply undo or merge changes to build files to retain local workspace references and then deliver to update RTC using an atomic change set(s) :)

Follow

Get every new post delivered to your Inbox.

Join 345 other followers

%d bloggers like this: