Selenium + JUnit 5 maven dependency and sample test case

pom.xml

<!-- Allows the legacy unit tests code under JUnit 4 still to run -->
<dependency>
    <groupId>org.junit.vintage</groupId>
    <artifactId>junit-vintage-engine</artifactId>
    <version>5.4.2</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-api</artifactId>
    <version>5.4.2</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.4.2</version>
    <scope>test</scope>
</dependency>
<dependency> 
    <groupId>org.junit.platform</groupId> 
    <artifactId>junit-platform-launcher</artifactId> 
    <version>1.5.0-M1</version> 
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>3.141.59</version>
</dependency>

Sample test case:

import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;

public class SeleniumTest {

    private WebDriver browser;
    
    @BeforeAll
    public static void init(){
        System.setProperty("webdriver.chrome.driver", "chromedriver");
    }
    
    @BeforeEach
    protected void setUp() throws Exception {
        browser = new ChromeDriver();
    }

    @Test
    public void browseGoogle() {
        browser.get("https://www.google.com.au");
    }
    
    @AfterEach
    protected void tearDown() throws Exception {
        browser.close();
    }

}

Selenium issue: java.lang.NoClassDefFoundError: okhttp3/WebSocketListener

A simple selenium test case yields the following error.

java.lang.NoClassDefFoundError: okhttp3/WebSocketListener
 at org.openqa.selenium.remote.internal.OkHttpClient.openSocket(OkHttpClient.java:75)
 at org.openqa.selenium.devtools.Connection.<init>(Connection.java:58)
 at org.openqa.selenium.chrome.ChromeDevToolsLocator.getChromeConnector(ChromeDevToolsLocator.java:81)
 at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:196)
 at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:176)
 at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:131)
 at au.gov.nla.ums.keycloak.selenium.TroveTest.test(TroveTest.java:21)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 ....

It's caused by the version of selenium-java:

<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>4.0.0-alpha-1</version>
</dependency>

Downgrading it to 3.141.59 solves the error.

<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>3.141.59</version>
</dependency>

No tests found with test runner 'Junit 5'

When I execute a junit 5 test case, I get this error.

This is part of my pom.xml that is relevant.

<!-- Allows the legacy unit tests code under JUnit 4 still to run -->
<dependency>
    <groupId>org.junit.vintage</groupId>
    <artifactId>junit-vintage-engine</artifactId>
    <version>5.4.2</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-api</artifactId>
    <version>5.4.2</version>
    <scope>test</scope>
</dependency>
<dependency> 
    <groupId>org.junit.platform</groupId> 
    <artifactId>junit-platform-launcher</artifactId> 
    <version>1.5.0-M1</version> 
    <scope>test</scope>
</dependency>

I try various suggestions on https://stackoverflow.com/questions/46717693/eclipse-no-tests-found-using-junit-5-caused-by-noclassdeffounderror, and the solution that works for me is adding another dependency of junit-jupiter-engine.

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.4.2</version>
    <scope>test</scope>
</dependency>

It is this article that gives me the decisive hint.

Local build OK but fails on Jenkins -- NoSuchMethodException

A couple of integration tests run well on my local but Jenkins is not happy and gives this error:

[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.583 s <<< FAILURE! - in au.gov.nla.ums.keycloak.integration.ITKeyCloakIntegrationTest
[ERROR] getUserInfoFromUsernameAndPassword(au.gov.nla.ums.keycloak.integration.ITKeyCloakIntegrationTest)  Time elapsed: 0.488 s  <<< ERROR!
java.lang.NoSuchMethodError: org.apache.commons.io.output.DeferredFileOutputStream.<init>(ILjava/lang/String;Ljava/lang/String;Ljava/io/File;)V
 at au.gov.nla.ums.keycloak.integration.ITKeyCloakIntegrationTest.getUserInfoFromUsernameAndPassword(ITKeyCloakIntegrationTest.java:65)

[ERROR] getRolesAndGroupsFromClientCredentials(au.gov.nla.ums.keycloak.integration.ITKeyCloakIntegrationTest)  Time elapsed: 0.088 s  <<< ERROR!
javax.ws.rs.ProcessingException: java.lang.NoSuchMethodError: org.apache.commons.io.output.DeferredFileOutputStream.<init>(ILjava/lang/String;Ljava/lang/String;Ljava/io/File;)V
 at au.gov.nla.ums.keycloak.integration.ITKeyCloakIntegrationTest.getRolesAndGroupsFromClientCredentials(ITKeyCloakIntegrationTest.java:46)
Caused by: java.lang.NoSuchMethodError: org.apache.commons.io.output.DeferredFileOutputStream.<init>(ILjava/lang/String;Ljava/lang/String;Ljava/io/File;)V
 at au.gov.nla.ums.keycloak.integration.ITKeyCloakIntegrationTest.getRolesAndGroupsFromClientCredentials(ITKeyCloakIntegrationTest.java:46)

DeferredFileOutputStream comes from commons-io.jar. Version 1.3 doesn't have such constructor while version 2.5 does. OK, It should be an easy fix, I think, I just have to explicitly set the version of this jar by:

<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.5</version>
    <scope>test</scope>
</dependency>

But it doesn't work.

Then I use reflection to print out all the constructor signature in the hope to ascertain the version of the jar.

Class<?> cl = Class.forName("org.apache.commons.io.output.DeferredFileOutputStream");
Constructor<?>[] cons = cl.getConstructors();
System.out.println("Constructors:---");
for (Constructor<?> con : cons) {
    System.out.println(con);
}

Here is the output:

Constructors:---
public org.apache.commons.io.output.DeferredFileOutputStream(int,java.io.File)

OK, now I am pretty certain that version 1.3 is at play. But the question is which jar is dependent upon commons-io and brings it on our classpath. More importantly, why can't the version of it be overwriten?

Eclipse's Dependency Hierarchy shows it comes indirectly from keycloak-modal-jpa. It says 'omitted for conflict with 2.5', and that's why local is working. How to get Jenkins show something like this?

A maven command comes to rescue:

mvn dependency:tree

The result shows commons-io is not being omitted

Compared to the result of my local run, everthing is the same except the local result doesn't have commons-io included in the tree.

Then I try excluding common-io from the keycloak-model-jar:

<dependency>
    <groupId>org.keycloak</groupId>
    <artifactId>keycloak-model-jpa</artifactId>
    <scope>provided</scope>
    <version>${keycloak.version}</version>
    <exclusions>
        <exclusion>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Unfortunately, Jenkins shows exactly the same dependency tree.

Oh, wait a second. The group id in the dependency tree is org.apache.common, but the group id of what I exclude is commons-io.

Changing the group id to org.apache.commons:

<dependency>
    <groupId>org.keycloak</groupId>
    <artifactId>keycloak-model-jpa</artifactId>
    <scope>provided</scope>
    <version>${keycloak.version}</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-io</artifactId>
        </exclusion>
    </exclusions>
</dependency>

And finally it's working!

Upon closely inspecting the pom.xml of openshift-restclient-jar, which is directly dependent on commons-io, I make some surprising discovery. Both org.apache.commons.commons-io and commons-io.commons-io are on its dependency list.

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-io</artifactId>
    <version>1.3.2</version>
</dependency>
<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.1</version>
</dependency>

It looks like a bug to me. Having said that, it remains a mystery to me as to why Jenkins sovles dependency differently from my local maven build. Could it be due to the different version of Maven?

NCM Investigation

Story 1: Why the deployment didn't pick up my changes?

I created a new jvmctl node of NCM on Hammer. I changed the port number and specified the git branch, and the deployment was successful. However, the system didn't seem to pick up my changes.

After making sure all my code changes were indeed deployed on Hammer (by checking the /apps folder), initially I thought the JSP might be cached, but then I realised not only JSP changes were not picked up, but none of the other changes were effective either.

I suspected, albeit a long shot, that it could be an issue of environmental nature. Then I repeated the same steps on Hoist, only to prove my theory wrong – it had exactly the same symptom. I went so far as to rename start.jar to start2.jar and yet the server still managed to start successfully. How was that even possible?

Upon carefully inspecting the jcmctl node, I finally found the culprit.

APP_OPTS=-Denv=dev -Dconfig=/apps/ncm/jetty/ncm/webapps/ndp/ -Dndpqa=yes -Djetty.http.port=9900 -Dorg.eclipse.jetty.server.Request.maxFormContentSize=20000000 -Dorg.eclipse.jetty.util.FileResource.checkAliases=false -Djetty.home=/apps/ncm/jetty -Djetty.base=/apps/ncm/jetty/ncm -jar /apps/ncm/jetty/start.jar --lib=/apps/ncm/jetty/ncm/lib/nla.jar /apps/ncm/jetty/ncm/etc/jetty.xml

There are numerous references made to the folder /apps/ncm, while it should be /apps/! 

I did, more than ten times, have looked at this line of configuration. It had continued eluding me because the word 'ncm' looked so innocuous sitting there. The eureka moment only hit me when I was checking the contents under /apps/ncm/jetty, fully expecting to see start2.jar. Of course start2.jar wouldn't be there. The sight of start.jar made me mumble "of course, of course, that's it!" and I felt a great sense of satisfaction.

Story 2: Why changing JspWrapper has no effect?

Once I deployed my code changes on hammer, I noticed the changes made on JspWrapper.java had no effect. However, a new class added to the same package which JspWrapper belonged to was clearly functioning.

Considering JspWrapper was accessed in a less-than-usual fashion (by a JSP file), I wondered if the compiled JSP was cached, so I renamed JspWrapper to JspWrapper2, and changed other places accordingly. Yep, it worked. Then I changed it back to JspWrapper, much to my surprise, only to see it fail again.

I suspected there was a rogue JspWrapper class lurking somewhere. To prove it, I changed JspWrapper to JspWrapper2 again, only this time, without changing the places that referenced to it. And the system could still find this class.

Now how to locate this rogue class, which resided in one of jar files? I used this command:

find . -name *.jar

I went through the list of jars carefully, and one jar caught my eye: nla-backup.jar. What the hell was that? Oh, I remembered. Before I had made the changes to the infrastructure java classes, I backed up the old nla.jar by renaming it to nla-blackup.jar just in case something went wrong.

I had committed the jar into github, not unintentionally, because I thought storing the file on github was safer than keeping it locally. Obviously it backfired now. There were two JspWrapper classes (and many other duplicate classes) on classpath. Which one would be loaded by JVM was nondeterministic. In my local environment, it happened to be the newer class. However, on hammer, it was the old one that got loaded.

Separate unit test from integration test

This is the template of pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.smt</groupId>
    <artifactId>smt</artifactId>
    <version>0.1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <profiles>
        <!-- The Configuration of the development profile -->
        <profile>
            <id>dev</id>
            <activation>
                <activeByDefault>true</activeByDefault>
            </activation>
            <properties>
                <build.profile.id>dev</build.profile.id>
                <skip.integration.tests>true</skip.integration.tests>
                <skip.unit.tests>false</skip.unit.tests>
            </properties>
        </profile>
        <!-- The Configuration of the integration-test profile -->
        <profile>
            <id>integration-test</id>
            <properties>
                <build.profile.id>integration-test</build.profile.id>
                <skip.integration.tests>false</skip.integration.tests>
                <skip.unit.tests>true</skip.unit.tests>
            </properties>
        </profile>
    </profiles>
    
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.2</version>
                <configuration>
                    <!-- Skips unit tests if the value of skip.unit.tests property is true -->
                    <skipTests>${skip.unit.tests}</skipTests>
                    <!-- Excludes integration tests when unit tests are run -->
                    <excludes>
                        <exclude>**/IT*.java</exclude>
                    </excludes>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>2.22.2</version>
                <executions>
                    <!-- Invokes both the integration-test and the verify goals of the Failsafe Maven plugin -->
                    <execution>
                        <id>integration-tests</id>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                        <configuration>
                            <!-- Skips integration tests if the value of skip.integration.tests property is true -->
                            <skipTests>${skip.integration.tests}</skipTests>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

To run unit test

mvn clean test

To run integration test

mvn clean verify -P integration-test

Project Euler 146 - Investigating a Prime Pattern

Investigating a Prime Pattern

Problem 146

The smallest positive integer n for which the numbers n²+1, n²+3, n²+7, n²+9, n²+13, and n²+27 are consecutive primes is 10. The sum of all such integers n below one-million is 1242490.
What is the sum of all such integers n below 150 million?

---

Well, the key was to find a good algorithm to test prime number.

import time
import sympy

now = time.time()
result = []
for n in range (10, 150000001, 10):
    if n ** 2 % 3 == 1 and n ** 2 % 7 == 2 and n ** 2 % 13 != 0 and sympy.isprime(n ** 2 + 1) and sympy.isprime(n ** 2 + 3) and sympy.isprime(n ** 2 + 7) and sympy.isprime(n ** 2 + 9) and sympy.isprime(n ** 2 + 13) and sympy.isprime(n ** 2 + 27) \
            and not sympy.isprime(n ** 2 + 23) and not sympy.isprime(n ** 2 + 21) and not sympy.isprime(n ** 2 + 19) and not sympy.isprime(n ** 2 + 17):
        result.append(n)
        print(n)
print ('result=', sum(result))
print('time spent is {}'.format(time.time() - now))