The article describes some of the TLS/SSL handshake issues and how to debug it in Integration Server.
Product: Integration Server
Version: 9.0 onwards
When a SSL connection between a client and server fails, the first thing one should do is to enable the SSL debugging. This will help to find whether it is SSL connection issue or an application issue. Most of the times TLS/SSL connection problems are because of not doing the TLS/SSL configuration or setup properly.
TLS/SSL Protocol Support
Integration server supports TLS/SSL through Entrust IAIK library and Java JSSE library. The TLS/SSL protocol versions supported in Integration Sever are:
- SSL 3.0 & TLS 1.0 by Entrust IAIK
- SSL 3.0, TLS1.0, TLS 1.1 & TLS 1.2. by Java JSSE (Based on Java version, SSL 3.0 & TLS 1.0 version might be deprecated or disabled by default.)
Enabling IAIK and JSSE Debugging
To enable IAIK debugging, set the watt property "watt.ssl.iaik.debug" to "true".
To enable JSSE debugging, set the Java System property "javax.net.debug" to "ssl".
- Set the system property in the
IS profiles c ustom_wrapper.conf file (e.g. wrapper.java.additional.204=-Djavax.net.debug=ssl) Or Set watt property "watt.config.systemProperties" (e.g. watt.config.systemProperties=javax.net.debug=ssl)
Understanding SSL Handshake
Client and Server negotiates on the SSL protocol version and cipher suites and compression methods at the start of the handshake. Server will always sends its certificate to Client and Client will verifies it against the trusted certificates stored in client's truststore. Sever generally also sends a list of certificates (CA certificates), it trust. These CA certificates are stored in the Server's truststore. If Server requires Client to be authenticated, then Client will send it's certificate. But client will send it's certificate only if the clients's certificate is issued by one of the CA certificate from CA certificates list sent by the server. Server will verify client's certificate against the trusted certificates stored in Sever's truststore.
TLS/SSL Handshake Issues & Debugging
For example, below handshake failed as Integration Server using Entrust IAIK connects to a TLS 1.2 enabled server. In this case, change from IAIK to JSSE option and set "watt.net.jsse.client.enabledProtocols=TLSv1.2"
jvm 1 | ssl_debug(3): Starting handshake (iSaSiLk 3.03)...
jvm 1 | ssl_debug(3): Remote client:151.193.164.20:443, Timestamp:Wed Mar 16 18:51:40 IST 2016
jvm 1 | ssl_debug(3): Sending secure renegotiation cipher suite
jvm 1 | ssl_debug(3): Sending v3 client_hello message, requesting version 3.1...
jvm 1 | ssl_debug(3): Received alert message: Alert Fatal: handshake failure
jvm 1 | ssl_debug(3): SSLException while handshaking: Peer sent alert: Alert Fatal: handshake failure
jvm 1 | ssl_debug(3): Shutting down SSL layer...
jvm 1 | ssl_debug(3): Closing transport...
IAIK: watt.net.ssl.server.handshake.minVersion,watt.net.ssl.server.handshake.maxVersion, watt.net.ssl.server.strongcipheronly,watt.net.ssl.server.cipherSuiteList.
JSSE:watt.net.jsse.server.enabledProtocols,watt.net.jsse.server.enabledCipherSuiteList
For outbound SSL connection (i.e. Integration Server is acting as SSL client), the following watt properties control the protocol version and ciphersuites:
IAIK:watt.net.ssl.client.handshake.minVersion,watt.net.ssl.client.handshake.maxVersion, watt.net.ssl.client.strongcipheronly,watt.net.ssl.client.cipherSuiteList
JSSE:watt.net.jsse.client.enabledProtocols,watt.net.jsse.client.enabledCipherSuiteList.
Note: For JSSE, more TLS configurations can be found in jvm/jre/lib/security/java.security file.
Issue 2: Integration Server is not able to connect to SNI enabled server (e.g. IIS 8.0)
Connections fails at the start of the handshake if Integration Server connects to a SNI enabled server. Integration Server doesn't support SNI (Server Name Indication) extension as a Server. As a Client, Integration Server partially supports SNI extension through JSSE library (SNI support is added in Java 7), So user must set either useJSSE=yes in the pub.client:http or set the watt property “watt.net.ssl.client.useJSSE=true” for Integration Server to send the SNI extensions in the ClientHello message during handshake.
If JSSE is used, SNI extension can be seen in the SSL debug message as shown below:
*** ClientHello, TLSv1.2
RandomCookie: GMT: 1441360890 bytes = { 57, 241, 207, 178, 190, ...., 21, 68, 223, 117, 72,166, 105 }
Session ID: {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, ...
Compression Methods: { 0 }
Extension elliptic_curves, curve names: {secp256r1, sect163k1,...sect283r1, ...secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA,..., MD5withRSA
Extension server_name, server_name: [type=host_name (0), value=xxx.yyy.zzz.net]
If the Server certificate is rejected by the Integration Server (acting as a client), then you may see debug message like show below for IAIK:
jvm 1 | ssl_debug(1): Received certificate handshake message with server certificate.
jvm 1 | ssl_debug(1): Server sent a 2048 bit RSA certificate, chain has 2 elements.
jvm 1 | ssl_debug(1): Sending alert: Alert Fatal: bad certificate
jvm 1 | ssl_debug(1): Shutting down SSL layer...
jvm 1 | ssl_debug(1): SSLException while handshaking: Server certificate rejected by ChainVerifier
jvm 1 | ssl_debug(1): Closing transport...
The possible reasons for the exception are:
1. The Server's CA/ROOT certificate is not present in the Integration Server (acting as client) truststore that is configured at "Security > Certificates" section and hence not able to verify the Server certificate. Add Server CA/ROOT certificate to the truststore specified in the "Security > Certificate" section.
2. If there is no truststore specified in the "Security > Certificates", then watt property "watt.security.cert.wmChainVerifier.trustByDefault" value could be "false". Set the watt property value to "true".
3. Server certificate might have expired. To ignore expired certificate, set the watt property "watt.security.ssl.ignoreExpiredChains" to "true".
jvm 1 | ssl_debug(4): Received certificate_request handshake message.
jvm 1 | ssl_debug(4): Accepted certificate types: RSA, DSS
jvm 1 | ssl_debug(4): Accepted certificate authorities:
jvm 1 | ssl_debug(4): cn=ca,ou=ca,o=ca,l=BLR,st=KA,c=IN
jvm 1 | ssl_debug(4): cn=leaf,ou=leaf,o=leaf,l=BLR,st=KA,c=IN
jvm 1 | ssl_debug(4): Received server_hello_done handshake message.
jvm 1 | ssl_debug(4): No client certificate available, sending empty certificate message...
jvm 1 | ssl_debug(4): Sending client_key_exchange handshake message (2048 bit)...
jvm 1 | ssl_debug(5): Received certificate handshake message with client certificate.
jvm 1 | ssl_debug(5): Certificate message is empty, client has no certificate.
jvm 1 | ssl_debug(5): Sending alert: Alert Fatal: bad certificate
jvm 1 | ssl_debug(5): Shutting down SSL layer...
In JSSE debug mode, the Certificate request along with Certificate authorities list can be seen as below:
*** CertificateRequest
Cert Types: RSA, DSS,
Cert Authorities:
CN=Duke, OU=Java Software, O="Sun Microsystems, Inc.",
L=Cupertino, ST=CA, C=US>
*** ServerHelloDone
If Integration Server (acting as Client) is not sending it's certificate then possible reasons are:
1. Ceriticates are not configured properly for the Integration Server. Set the Certificates to be used for outbound connection at "SSL Key" fields in "Security > Certificates" section. If pub.security:setKeyAndChain is used, then check whether it is using the correct keystore or not.
2. Integration Server's certificate is not issued by any CA certificates send by the server during the handshake. Add Integration Server's CA/ROOT certificates in the Sever's truststore.
3. If the Certificate Authorities list sent by the Server during the handshake is empty, then also Integration Server will not send it's certificate unless the watt property "watt.security.ssl.client.ignoreEmptyAuthoritiesList" is set to "true".
If the Server doesn't trust the client's certificate, then the handshake will fail with exception saying "bad certificate". This is similar to issue 3 and in this case, the Integration Server (acting as client) CA/ROOT certificates are not present in the Sever's truststore.
Add Integration Server's CA/ROOT certificates in the Server's truststore.
Issue 6: Certificates version is X509 ver 1
Handshake sometimes fails if the Certificates are of old version (i.e X509 version 1). X509 v1 certificate is similar to X509 v3 certificate except that it doesn't have the extensions. Nowadays, very few server issues X509 v1 certificates, but some servers still do.