Toward an ideal workflow with ClearCase and Git

Question

Introduction

This is more than a fact, using ClearCase (not UCM) as main SCM for large projects maintained by few people is a pretty not efficient solution. When it's Corporate Standard, we are stuck with it and we need to find an efficient workaround.

The usual workflow with ClearCase consists of a master branch, a develop branch and several new feature branches.

        o--------o (feature)
       /          \
  ----o--o---o-----o----o (develop)
 /        \         \
*----------o---------o (main)

Example

Actually, what I call a feature could be also a simple refactoring, such as a massive renaming inside the project. In this example, and because ClearCase is file-centric, we always need a temporary branch (no atomic checkin in non-UCM CC). Creating a new branch is painful and having the right config-spec is a struggling task. From here, I see two solutions:

1. Be corporate and start the massive checkout of all the files. Because the SCM environment does not reside on the same site as the ClearCase servers, everything gets slower. I count 2s per file and 8min for 1k files. After the first coffee break, we do the work and we check in all the files (another 8min wasted). Some tests, a new massive checkout, a bugfix, a massive checkin, the merge to the develop branch and eventually we delete the feature branch, which is no longer useful.

   In this solution, everything's slow, a lot of caffeine gets consumed and the workflow is pretty inefficient. I don't think this is a good solution.

2. Because we would like to keep a trace of our changes, and we do not want to waste time checkin/out all the project's files, we init a Git repository from a snapshot view. Actually, the Git repository could be located anywhere else ouside ClearCase. The changes are made locally with the help of Git and, once everthing's done, the project is pushed back on ClearCase with clearfsimport. After this, we still need to merge the feature branch. This is done in CC. The feature branch can then be deleted.

   This solution requires lots of manipulations. Also, clearfsimport can be very dangerous if we misspell the target view or if we forget to remove all the temporary files. Last but not least, the final merge has to be done on CC.

Snapshot Views

In my examples, I did not mentioned the Snapshot Views because they are not compatible with Git. A hijacked file identified based on its timestamp. If I manually modify a file and restore its original modification date, ClearCase won't see any changes. This could be very dangerous as the following example proves it.

If you don't believe me, you can try this:

stat -c 'touch --no-create -d "%y" "%n"' foo > restore_timestamp
echo "ClearCase will not see this" >> foo
source restore_timestamp
rm restore_timestamp

With this mechanism, no Git repository can reside inside a ClearCase Snapshot View.

Separated Git repository

I doubt we can find any workaround to the requirement for a temporary branch creation. The merge has to be done on ClearCase even though Git is holding everything behind.

I have tried to use the Copy/Paste extensively to synchronize a completely separated Git repository with ClearCase. Right before the final merge, I copy/paste the current state of the develop branch in a new Git branch and do the merge as quickly as possible. At the end, I used clearfsimport to push the modifications back to the develop branch.

This operation could be very dangerous if someone wants to access the project during the merge process. For this reason, the development branch has to be locked or reserved during the operation. Unfortunately, this additional operation is very time consuming on ClearCase.

The ClearCase "git checkout -b feature" equivalent

In Git when I want to create a new branch I simply type:

 git checkout -b feature

That's all, and I can work on my new feature right away. On ClearCase, things are a bit different.

First, I need to place a label where I want to create my branch. From Windows with Cygwin, I can do this:

LABEL=my_temporary_label
VOB=vob_name
cleartool mklbtype -global -nc lbtype:${LABEL}@\\${VOB}
cleartool mklabel -replace ${LABEL} `cleartool find . -cview -print -nxn | awk '{printf "%s ", $0}'`
cleartool find . -cview -type l -exec 'cleartool ls %CLEARCASE_XPN%' | \
   perl -p -e 's/\\/\//g' | \
   perl -p -e 's/\r//g' | \
   perl -e 'while(<>) {s@([.]/)(.*/)(.*-->\s*)@$2@;print;}' | \
   xargs -n1 cleartool mklabel ${LABEL}

However, we need to be careful because symbolic links are not expanded.

Then, a branch has to be created:

mkbrtype –c "Temporary Branch" my_temporary_branch 

To work on this branch, a view need to be created:

cleartool mkview -tag my_temporary_view \\shared\path\to\viewStorage\my_temporary_view.vws

The config-spec has to be edited:

element * CHECKEDOUT
element  .../lost+found -none

element * .../develop_branch/LATEST

mkbranch develop_branch
   element * /main/A_LABEL_WHERE_THE_BRANCH_IS_BRANCHED_ON
   element * /main/LATEST

end mkbranch develop_branch

Now, the branch is created. We can work on our feature. Easy, huh?

In Git, I usually create 3-5 branches a day. I don't think I can do the same with ClearCase. Am I wrong?

Discussion

The two proposed solutions are far from good because they all require a lot of time consuming operations on ClearCase (create a branch, set a view, checkin, checkout, do the merge, delete the branch).

I am looking for a solution that does not involve complex manipulations. And I still believe Git could be a good ally, but how can ClearCase and Git work together?

I think we can use some scripting. I recently found git-cc which could be a good start. Unfortunately, this project is not mature yet.

I haven't investigated this approach, but I think there is one solution where we can use a ClearCase Snapshot View with Git. The timestamp of each file has to be saved before making any changes:

 find . -type f -exec stat -c 'touch--no-create -d "%y" "%n"' {} \; > restore

From there, we can work on Git and once it is time to push the changes on ClearCase or just pull the changes from from the development branch, the original timestamp can be restored on all files but the modified files by since the last synchronization:

 source ./restore
 git diff --name-only SHA1 SHA2 | xargs touch

The Question

On StackOverflow, people like precise questions, not primary opinion-based questions. Thus, after this pretty long introduction I eventually can ask my question:

I have tried many different approaches to improve my workflow with ClearCase but working on large projects with a file-centric SCM is not straightforward. I believe Git could help a lot, but I need to find a workflow that allows the synchronization of a local git repository with ClearCase and, if possible, without requiring a temporary branch.

Can that be done?

Answer 1

A possible solution from VonC

In VonC's answer, he suggests to not use an intermediate branch and manage everything with Git.

Through an example I would like to show his method. We start with this configuration on ClearCase:

  o------o----o (develop)
 /        \ 
*----------o (main)

The idea is to use Git to facilitate the development of a new feature that will eventually be merged to the develop branch.

We first copy the ClearCase project into a local folder and init a Git repository (in the case the Git repository doesn't already exist).

WORKSPACE=~/foo
cp `cleartool ls -r | grep '@@' | sed 's/@@.*$//'` $WORKSPACE
cd $WORKSPACE
git init
git add .
git commit -m "Initial commit"
git checkout -b feature

We spend some time developing the feature on its own local Git branch:

                  x----x--x---x----x (feature on Git)
                 /
                x---------- (master on Git)
               /
  o------o----o------o----o (develop)
 /        \ 
*----------o (main)

At the end of the day it is time to sync-down the possible changes from ClearCase:

git checkout master
git --ls-files | xargs rm
cd $CCVIEW
cleartool ls -r | grep '@@' | sed 's/@@.*$//' > $WORKSPACE/ccview
cd $WORKSPACE
cat ccview | xargs -n1 cp {} $WORKSPACE    
cat ccview | xargs git add 
git commit -m "Imported from CC"

So now we have made several commit on the feature branch and the master Git branch was synchronized with ClearCase.

                  x----x--x---x----x (feature on Git)
                 /
                x-----------o (master on Git)
               /           /
  o------o----o------o----o (develop)
 /        \ 
*----------o (main)

We must not to forget to lock the ClearCase View during the entire merge process. This is required to prevent other developers to see their own changes erased by clearfsimport. To lock a ClearCase branch it is easy:

cleartool lock brtype:$BR_NAME 

Then the merge can be done on Git:

git checkout master
git merge feature

The feature Git branch was merged with the master.

                  x----x--x---x----x (feature on Git)
                 /                  \
                x-----------o--------o (master on Git)
               /           /
  o------o----o------o----o (develop)
 /        \ 
*----------o (main)

The modifications can be pushed back to ClearCase

OUT="$(mktemp -d)"
cp -v --parents `git ls-files | sed 's/[^ ]*\.gitignore//g'` $OUT
clearfsimport -comment 'clearfsimport' -rec -unco -nset $OUT $CVIEW    
rm -rf $OUT

And the lock can be removed to re-authorize the changes on the branch

cleartool unlock brtype:$BR_NAME

.

                  x----x--x---x----x (feature on Git)
                 /                  \
                x-----------o--------o (master on Git)
               /           /          \
  o------o----o------o----o------------o (develop)
 /        \ 
*----------o (main)

The Git repository and the local workspace may be removed unless we need to go on.

  o------o----o------o----o------------o (develop)
 /        \ 
*----------o (main)

In this solution, we did not use an intermediate branch on ClearCase and all the merge process occurred on Git. The ClearCase history is preserved. The only bad point is the need to lock the development branch for the final merge.

@VonC, feel free to modify my answer if I am wrong.

Answer 2

Actually, what I call a feature could be also a simple refactoring, such as a massive renaming inside the project. In this example, and because ClearCase is file-centric, we always need a temporary branch (no atomic checkin in non-UCM CC).
Creating a new branch is painful and having the right config-spec is a struggling task

So... don't create a temporary branch?
If this is to be used in collaboration with Git, only create that feature branch in the Git repo, do the final merge in the Git Repo, and then clearfsimport the result in the main ClearCase view.