Local build OK but fails on Jenkins -- NoSuchMethodException

A couple of integration tests run well on my local but Jenkins is not happy and gives this error:

[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.583 s <<< FAILURE! - in au.gov.nla.ums.keycloak.integration.ITKeyCloakIntegrationTest
[ERROR] getUserInfoFromUsernameAndPassword(au.gov.nla.ums.keycloak.integration.ITKeyCloakIntegrationTest)  Time elapsed: 0.488 s  <<< ERROR!
java.lang.NoSuchMethodError: org.apache.commons.io.output.DeferredFileOutputStream.<init>(ILjava/lang/String;Ljava/lang/String;Ljava/io/File;)V
 at au.gov.nla.ums.keycloak.integration.ITKeyCloakIntegrationTest.getUserInfoFromUsernameAndPassword(ITKeyCloakIntegrationTest.java:65)

[ERROR] getRolesAndGroupsFromClientCredentials(au.gov.nla.ums.keycloak.integration.ITKeyCloakIntegrationTest)  Time elapsed: 0.088 s  <<< ERROR!
javax.ws.rs.ProcessingException: java.lang.NoSuchMethodError: org.apache.commons.io.output.DeferredFileOutputStream.<init>(ILjava/lang/String;Ljava/lang/String;Ljava/io/File;)V
 at au.gov.nla.ums.keycloak.integration.ITKeyCloakIntegrationTest.getRolesAndGroupsFromClientCredentials(ITKeyCloakIntegrationTest.java:46)
Caused by: java.lang.NoSuchMethodError: org.apache.commons.io.output.DeferredFileOutputStream.<init>(ILjava/lang/String;Ljava/lang/String;Ljava/io/File;)V
 at au.gov.nla.ums.keycloak.integration.ITKeyCloakIntegrationTest.getRolesAndGroupsFromClientCredentials(ITKeyCloakIntegrationTest.java:46)

DeferredFileOutputStream comes from commons-io.jar. Version 1.3 doesn't have such constructor while version 2.5 does. OK, It should be an easy fix, I think, I just have to explicitly set the version of this jar by:

<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.5</version>
    <scope>test</scope>
</dependency>

But it doesn't work.

Then I use reflection to print out all the constructor signature in the hope to ascertain the version of the jar.

Class<?> cl = Class.forName("org.apache.commons.io.output.DeferredFileOutputStream");
Constructor<?>[] cons = cl.getConstructors();
System.out.println("Constructors:---");
for (Constructor<?> con : cons) {
    System.out.println(con);
}

Here is the output:

Constructors:---
public org.apache.commons.io.output.DeferredFileOutputStream(int,java.io.File)

OK, now I am pretty certain that version 1.3 is at play. But the question is which jar is dependent upon commons-io and brings it on our classpath. More importantly, why can't the version of it be overwriten?

Eclipse's Dependency Hierarchy shows it comes indirectly from keycloak-modal-jpa. It says 'omitted for conflict with 2.5', and that's why local is working. How to get Jenkins show something like this?

A maven command comes to rescue:

mvn dependency:tree

The result shows commons-io is not being omitted

Compared to the result of my local run, everthing is the same except the local result doesn't have commons-io included in the tree.

Then I try excluding common-io from the keycloak-model-jar:

<dependency>
    <groupId>org.keycloak</groupId>
    <artifactId>keycloak-model-jpa</artifactId>
    <scope>provided</scope>
    <version>${keycloak.version}</version>
    <exclusions>
        <exclusion>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Unfortunately, Jenkins shows exactly the same dependency tree.

Oh, wait a second. The group id in the dependency tree is org.apache.common, but the group id of what I exclude is commons-io.

Changing the group id to org.apache.commons:

<dependency>
    <groupId>org.keycloak</groupId>
    <artifactId>keycloak-model-jpa</artifactId>
    <scope>provided</scope>
    <version>${keycloak.version}</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-io</artifactId>
        </exclusion>
    </exclusions>
</dependency>

And finally it's working!

Upon closely inspecting the pom.xml of openshift-restclient-jar, which is directly dependent on commons-io, I make some surprising discovery. Both org.apache.commons.commons-io and commons-io.commons-io are on its dependency list.

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-io</artifactId>
    <version>1.3.2</version>
</dependency>
<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.1</version>
</dependency>

It looks like a bug to me. Having said that, it remains a mystery to me as to why Jenkins sovles dependency differently from my local maven build. Could it be due to the different version of Maven?

NCM Investigation

Story 1: Why the deployment didn't pick up my changes?

I created a new jvmctl node of NCM on Hammer. I changed the port number and specified the git branch, and the deployment was successful. However, the system didn't seem to pick up my changes.

After making sure all my code changes were indeed deployed on Hammer (by checking the /apps folder), initially I thought the JSP might be cached, but then I realised not only JSP changes were not picked up, but none of the other changes were effective either.

I suspected, albeit a long shot, that it could be an issue of environmental nature. Then I repeated the same steps on Hoist, only to prove my theory wrong – it had exactly the same symptom. I went so far as to rename start.jar to start2.jar and yet the server still managed to start successfully. How was that even possible?

Upon carefully inspecting the jcmctl node, I finally found the culprit.

APP_OPTS=-Denv=dev -Dconfig=/apps/ncm/jetty/ncm/webapps/ndp/ -Dndpqa=yes -Djetty.http.port=9900 -Dorg.eclipse.jetty.server.Request.maxFormContentSize=20000000 -Dorg.eclipse.jetty.util.FileResource.checkAliases=false -Djetty.home=/apps/ncm/jetty -Djetty.base=/apps/ncm/jetty/ncm -jar /apps/ncm/jetty/start.jar --lib=/apps/ncm/jetty/ncm/lib/nla.jar /apps/ncm/jetty/ncm/etc/jetty.xml

There are numerous references made to the folder /apps/ncm, while it should be /apps/! 

I did, more than ten times, have looked at this line of configuration. It had continued eluding me because the word 'ncm' looked so innocuous sitting there. The eureka moment only hit me when I was checking the contents under /apps/ncm/jetty, fully expecting to see start2.jar. Of course start2.jar wouldn't be there. The sight of start.jar made me mumble "of course, of course, that's it!" and I felt a great sense of satisfaction.

Story 2: Why changing JspWrapper has no effect?

Once I deployed my code changes on hammer, I noticed the changes made on JspWrapper.java had no effect. However, a new class added to the same package which JspWrapper belonged to was clearly functioning.

Considering JspWrapper was accessed in a less-than-usual fashion (by a JSP file), I wondered if the compiled JSP was cached, so I renamed JspWrapper to JspWrapper2, and changed other places accordingly. Yep, it worked. Then I changed it back to JspWrapper, much to my surprise, only to see it fail again.

I suspected there was a rogue JspWrapper class lurking somewhere. To prove it, I changed JspWrapper to JspWrapper2 again, only this time, without changing the places that referenced to it. And the system could still find this class.

Now how to locate this rogue class, which resided in one of jar files? I used this command:

find . -name *.jar

I went through the list of jars carefully, and one jar caught my eye: nla-backup.jar. What the hell was that? Oh, I remembered. Before I had made the changes to the infrastructure java classes, I backed up the old nla.jar by renaming it to nla-blackup.jar just in case something went wrong.

I had committed the jar into github, not unintentionally, because I thought storing the file on github was safer than keeping it locally. Obviously it backfired now. There were two JspWrapper classes (and many other duplicate classes) on classpath. Which one would be loaded by JVM was nondeterministic. In my local environment, it happened to be the newer class. However, on hammer, it was the old one that got loaded.

Separate unit test from integration test

This is the template of pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.smt</groupId>
    <artifactId>smt</artifactId>
    <version>0.1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <profiles>
        <!-- The Configuration of the development profile -->
        <profile>
            <id>dev</id>
            <activation>
                <activeByDefault>true</activeByDefault>
            </activation>
            <properties>
                <build.profile.id>dev</build.profile.id>
                <skip.integration.tests>true</skip.integration.tests>
                <skip.unit.tests>false</skip.unit.tests>
            </properties>
        </profile>
        <!-- The Configuration of the integration-test profile -->
        <profile>
            <id>integration-test</id>
            <properties>
                <build.profile.id>integration-test</build.profile.id>
                <skip.integration.tests>false</skip.integration.tests>
                <skip.unit.tests>true</skip.unit.tests>
            </properties>
        </profile>
    </profiles>
    
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.2</version>
                <configuration>
                    <!-- Skips unit tests if the value of skip.unit.tests property is true -->
                    <skipTests>${skip.unit.tests}</skipTests>
                    <!-- Excludes integration tests when unit tests are run -->
                    <excludes>
                        <exclude>**/IT*.java</exclude>
                    </excludes>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>2.22.2</version>
                <executions>
                    <!-- Invokes both the integration-test and the verify goals of the Failsafe Maven plugin -->
                    <execution>
                        <id>integration-tests</id>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                        <configuration>
                            <!-- Skips integration tests if the value of skip.integration.tests property is true -->
                            <skipTests>${skip.integration.tests}</skipTests>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

To run unit test

mvn clean test

To run integration test

mvn clean verify -P integration-test