Jenkins and JUnit - How to make it work for you

Jenkins is a widely used CI (Continuous Integration) tool.
Its fairly easy to create your own Jenkins Server on a decent machine or even virtual machine.
Usually, you create some tests.
Then you require that data to be presented in a graph showing successes and failures.
Jenkins will do that for you, there is no need for you to write your own code to create the renderings and graphs etc.
Jenkins just requires an xml file which contains the results of your test suite.
The problem is that the xml does not have a fixed format, it can contain very few items, or a lot of items; yet there is no standardization.

So, what do you do now?
First things first.
Install the JUnit plugin on Jenkins.
Then have your test script/framework generate an xml. There are a lot of custom tools, scripts, plugins and libraries which can be used to generate xmls from ready data.

However, how do you know that your xml has everything that Jenkins needs to publish that beautiful graph?
The JUnit plugin requires a very simple xml format.
Just match that and voila!! You have a graph that is plotting your test failures and successes.
Here is an example of a very simple xml that Jenkins can process into the graph.

=============== BEGIN XML SNIPPET ================
<testsuite name="Sample tests" tests="4">
    <testcase name="test1">
        <property name="Passed"/>
    </testcase>
    <testcase name="test2">
        <property name="Passed"/>
    </testcase>
    <testcase name="test3">
        <error name="Some Error message."/>
    </testcase>
    <testcase name="test4">
        <failure message="Some Failure message."/>
    </testcase>
</testsuite>
=============== END XML SNIPPET ================

The JUnit plugin does not differentiate between errors and failures. (Ref - https://issues.jenkins-ci.org/browse/JENKINS-4951 )

Next, add a build step in your Jenkins job as "Execute Shell".
JUnit plugin does not fail or pass your builds. You need to do that yourself.
You need to mark you build as failed in case there is some failure or error.
One thing to note, the "Execute Shell" step will fail the build for any non-zero return codes.
We will make use of this for our own good. ;)

Here is a script that I use -
=============== BEGIN SHELL SNIPPET ================
grep -Ei 'failure|error' result.xml
res=`echo $?`
if [ $res -eq 0 ]; # Grep found at least one match
then
    echo "Some tests failed or had some error. Failing the build..." ;
    exit 1;
elif [ $res -ne 0 ];
then
    echo "Pass"
fi
=============== END SHELL SNIPPET ================

I myself like the "no tolerance" approach, so I fail my builds if there is even one error or failure, but you can write your own custom logic and parsing using percentages or something else.


Now you will get a nice graph which tracks the test failures and successes AND the appropriate build status.
Without this shell script, the builds will be marked as success or as unstable.
The JUnit only looks into the xml to find failures and errors, but it does NOT control the status of the build.