Automatic bidirectional synchronisation between RTC and ClearCase
I’ve previously blogged on a manual ClearCase to RTC migration script I use during migration from IBM Rational ClearCase to IBM RTC. However sometimes it’s necessary to consider automatic synchronisation for the following reasons:
- Partial team adoption in large teams/organisations
- There are code dependencies between teams (you’d be better off breaking them to binary dependencies and using something like maven or ivy though…)
- Support teams on RTC SCM while interacting with content from ClearCase multisite locations
- Maintaining complex builds that are dependant on ClearCase (i.e. using ClearMake).
- Allowing teams to pilot RTC SCM without working about branching between tools
- …. and many more…
The off the shelf offerings to bridge between RTC/ClearCase are a bit limited due to the relationship between eclipse projects, rtc components and streams. It’s necessary to have eclipse projects at the root of a component for them to load smoothly as proper projects in eclipse with their correct natures, so any serious synchronisation solution needs to allow you to pick and choose parts of a folder structure in ClearCase and sync them to projects in components in RTC. The excellent open source RTC ClearCase workspace synchroniser from samecs does exactly this.
I personally use a combination of the manual migration script (mentioned at the top) and the RTC ClearCase workspace synchroniser to deal with team migrations from ClearCase to RTC. Generally my preference is to do it manually as synchronisation is a tactical solution during rollout and introduces a fair amount of technical complexity, but in cases with the features mentioned above sometimes you need a powerful and flexible synchronisation solution.
How to migrate source to Jazz RTC SCM
First of all, although it might seem tempting to seperate scm and build, especially from a business change and planning perspective you just can’t (unless you’re writing scripting languages!). Software is just useless text if it can’t be built/run and a software team can’t be expected to move where they store their source without ensuring that build doesn’t get broken. In a perfect world build scripts would be generic and work regardless of the scm system they’re coming from but that generally isn’t the case due to this not being a perfect world:
- different scm systems/ides imposing different structural constraints
- absolute paths to files
- dodgy use of symlinks
- lack of parameterization of build scripts
- complexity of numerous build technologies
This means that when migrating a team into Jazz SCM you need to at least ensure that their current builds work in their current normal way. Ideally it would be great to (re)engineer them to word with RTC Build but that isn’t always practical. This build refactoring shouldn’t be underestimated as it can take a while because of general build complexity, project complexity and technology complexity. It’s not unusual to find in just one organisation the use of: ant, maven, junit, ivy, make, clearmake, msbuild sometimes all on the same “project”. Add unit testing, style checking, static analysis and other things that people tend to add into the build process and you’re looking at a large amount of technical skill needed to even begin to make sense of what’s currently going on.
For this reason when moving a team from cm system X to RTC Jazz SCM I find it necessary to build the stream in 2 parts, which means you need a repeatable migration script that doesn’t get tripped up by things like permission or line end delimiter platform differences.
My general migration script (assuming not synchronizing or bridging):
- Label/Baseline/Shapshot source files in old scm system
- Using linux copy source files from existing scm/view/folder to a holding area
- chown if necessary and chmod files to u:rw(x),o—,g—
- I tend to do a dos2unix conversion on appropriate files to normalise mixed mode files
- Share to RTC
- Baseline/Snapshot in RTC
- Make builds work (generally a process of changing absolute path references to local relative paths – often “/view/blah/…/proj” to “../” (here be technical complexity)
- Baseline/Snapshot in RTC (“Initial scm migration with working build”)
There is a great advantage in following this process (which is why I’ve scripted steps 1-4 although unfortunately I can’t share that) somewhat slavishly in that it is repeatable to update a file set as a change set. If you do this to create an example stream for a team to experiment and gain confidence in while they’re still developing in their existing system, then when they choose to adopt you can simply update their stream(s) by:
- [Optionally] revert to working build snapshot “Initial scm migration with working build” from step 8 above
- Repeat steps 1-4 above
- Copy files in holding area to RTC workspace
- In RTC Pending Changes view drop-down the refresh button to select “Refresh Sandbox and Remote Changes”
You then get a change set showing the file changes since the experimental migration and the present time, simply undo or merge changes to build files to retain local workspace references and then deliver to update RTC using an atomic change set(s) 🙂
How to deal with line feeds in Jazz RTC SCM
I’ve found that there’s sometimes a bit of confusion over how Eclipse & RTC deal with line feeds, especially when migrating files from ClearCase so thought I’d post about it.
By default RTC tries to be helpful when opening text files on windows and linux platforms by rewriting the text file contents to either windows standard or linux standard depending on which platform you’re currently using. This is generally quite helpful as it means that when you load a workspace on whatever platform and use an external editor to mess around with text files they’re in the right format to make sense to your editor.
There are some cases when this behaviour isn’t quite so helpful however which can often come up during migration from ClearCase or other SCM tools. During migration you tend to copy a set of files from a location, change permissions on them and then stick them into a workspace for sharing into Jazz SCM. If you do this on windows then your files will be converted to windows format. As I said in the previous paragraph this is generally helpful but in some cases isn’t, as if the team generally develop on linux then when they load the files they’ll convert the format again making a change set which can interfere in a simple understanding of the history of files or when doing a file contents comparison.
Similarly if you build on windows and then deploy to linux and run some bash scripts or similar in your set of files you can hit problems as the files loaded on windows will be converted to windows format, if they’re copied to a server without running a dos2unix (or similar) conversion on them then the scripts won’t run on the target server.
There are a couple of features that you can use to prevent these problems, which although minor and easily corrected, can sometimes dent confidence in adopting teams. I’ve heard teams say they want exactly the same files (in binary terms) that they had in ClearCase and they don’t want RTC to rewrite to platform… that’s easily achievable you just need to fiddle with some settings. Of course other teams simply don’t notice as the apparent contents stay the same whatever platform you’re on.
My advice for avoiding these issues is:
- Set a sensible set of default file format options in Eclipse before starting (Window -> Preferences -> Team -> File Content) including setting to “binary” any file formats you don’t want to change such as .sh (unfortunately this doesn’t have a */other files setting)
- Do the migration on the normal development platform the team will use with a bias towards linux – My process for doing migrations
- Ensure the adopting team is aware of the difference between line feed settings on different platforms and knows how to change them in their project.
RTC provides features to work around line feed formats, if you right click on a project in project explorer you can select Team -> Change File Properties to change the properties of the files in your project. The setting text/plain : platform indicates that RTC should rewrite files to the current platform, so setting .sh files delimiter to LF (Unix) will prevent problems in the previous scenario even if the project is built and deployed on Windows. Choosing next on this wizard allows you to set general preferences for types as well.
Mike MacDonagh's Blog 