java.security.cert.CertPathValidatorException: Certificate chaining error seen with IBM Business Process Manager v8

In this post, I describe how we identified, and resolved, an issue with SSL certificates, seen IBM Business Process Manager (BPM) version 8 ( Standard ).

Having built out an IBM BPM environment, with a topology spanning two WAS cells ( Process Center and Process Server ), we were experiencing a strange issue when attempting to install a process from the Process Center cell to the Process Server cell.

Whilst the Process Server always appeared correctly within the Process Center environment ( Process Server - PS - broadcasts it's presence to Process Center - PS ), we saw: -

Caused by: com.lombardisoftware.core.TeamWorksException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: 
java.security.cert.CertPathValidatorException: The certificate issued by OU=CustomerName CA, O=CustomerName CAs, O=1359.2.1, C=gb is not trusted; internal cause is: 
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.lombardisoftware.core.TeamWorksException.asTeamWorksException(TeamWorksException.java:136)
at com.lombardisoftware.core.TWHttpClient.login(TWHttpClient.java:144)
at com.lombardisoftware.server.ejb.repositoryservices.DeployToServerSupport.deploySnapshot(DeployToServerSupport.java:223)
... 101 more
Caused by: javax.net.ssl.SSLHandshakeException: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: 
java.security.cert.CertPathValidatorException: The certificate issued by OU=CustomerName CA, O=CustomerName CAs, O=1359.2.1, C=gb is not trusted; internal cause is: 
java.security.cert.CertPathValidatorException: Certificate chaining error
at com.ibm.jsse2.o.a(o.java:22)

Having seen this kind of issue before, I was convinced that the solution was to ensure that the client's CA-generated root and intermediate certificates were present in the WAS cell-level trust store, and proceeded to use WAS' "Retrieve from Port" function to import the certificate chain ( from the load-balanced hostname of the IBM HTTP Servers into which these two certificates had already been installed ).

When this didn't fix the problem, I went one step further and, using openSSL to extract the "device-level" certificate from the IHS key store - keystore.kdb, I imported the certificate, as a plain ASCII file, again into the cell-level trust store.

Having gone around this loop over and over again, I was beginning to doubt my sanity.

Talking with a friend about the problem, he mentioned the key ( pardon the pun  ) phrase CACerts.

Having come across this before in the context of WAS, I asked him what he meant.

He directed me at a file - cacerts - located here : -

/opt/IBM/WebSphere80/AppServer/java/jre/lib/security

which contains the WAS JVM's default trust store.

Using the ikeycmd command that ships with IBM HTTP Server, we explorer this trust store: -

$ /opt/IBM/HTTPServer80/java/jre/bin/ikeycmd -cert -list -db /opt/IBM/WebSphere80/AppServer/java/jre/lib/security/cacerts -pw changeit -type jks

Whilst there were 77 individual labels within the store, all of which are provided by default with WAS, we did NOT have anything that tied back to the client's CA-generated certificates.

Once we imported ONLY the root and intermediate certificates, and restarted the Process Center and Process Server JVMs, our ability to publish processes improved 100% :-)

The moral of the story ?

If the error message about chaining suggests that you don't have the certificate chain within WAS, then you probably don't :-)

One more for the book ....