Pages

Wednesday, November 07, 2007

Comparing JUnit test results between CDK trunk/ and a branch

I have started using branches for non-trivial patches, like removing the HückelAromaticityDetector, in favor of the new CDKHückelAromaticityDetector. I am doing this in my personal remove-non-cdkatomtype-code branch, where I can quietly work on the patch until I am happy about it. I make sure to keep it synchronized with trunk with regular svn merge commands.

Now, the goal is that my branch only fixed failing JUnit tests, not that it creates new regressions. To compare the results between two versions of the CDK, I use these commands:

$ cd cdk/trunk/cdk
$ ant -lib develjar/junit-4.3.1.jar -logfile ant.log test-all
$ cd ../../branches/egonw/remove-non-cdkatomtype-code/
$ ant -lib develjar/junit-4.3.1.jar -logfile ant.log test-all
$ cd ../../..
$ grep Testcase branches/egonw/remove-non-cdkatomtype-code/reports/*.txt | cut -d':' -f2,3 > branch.results
$ grep Testcase trunk/cdk/reports/*.txt | cut -d':' -f2,3 > trunk.results
$ diff -u trunk.results branch.results

The last diff commands gives me a quick overview of what has changed. See get the statistics, I can do:

$ diff -u trunk.results branch.results | grep "^-Testcase" | wc -l
$ diff -u trunk.results branch.results | grep "^+Testcase" | wc -l

The first gives me the number of JUnit tests which are now no longer failing, while the second
gives me the number of tests which are new fails. Ideally, the second is zero. Unfortunately, not yet the case :)

3 comments:

  1. You know about svnmerge.py right? It's takes the pain out of managing branches, and can handle bidirectional merges...

    ReplyDelete
  2. Noel, that seems very useful indeed! I have, so far, managed to do this by hand... but it seems to be able a good tool to show which patches are available... Thanx for the suggestions.

    Here's a wiki page with info on it:

    http://www.orcaware.com/svn/wiki/Svnmerge.py

    ReplyDelete
  3. The nightly build page now shows the number of failures fixed and number of new failure since the last build

    ReplyDelete