The article describes some of the TLS/SSL handshake issues and how to debug it in Integration Server.
Product: Integration Server
Version: 9.0 onwards
When a SSL connection between a client and server fails, the first thing one should do is to enable the SSL debugging. This will help to find whether it is SSL connection issue or an application issue. Most of the times TLS/SSL connection problems are because of not doing the TLS/SSL configuration or setup properly.
TLS/SSL Protocol Support
Integration server supports TLS/SSL through Entrust IAIK library and Java JSSE library. The TLS/SSL protocol versions supported in Integration Sever are:
- SSL 3.0 & TLS 1.0 by Entrust IAIK
- SSL 3.0, TLS1.0, TLS 1.1 & TLS 1.2. by Java JSSE (Based on Java version, SSL 3.0 & TLS 1.0 version might be deprecated or disabled by default.)
Enabling IAIK and JSSE Debugging
To enable IAIK debugging, set the watt property "watt.ssl.iaik.debug" to "true".
To enable JSSE debugging, set the Java System property "javax.net.debug" to "ssl".
- Set the system property in the IS profiles custom_wrapper.conf file (e.g. wrapper.java.additional.204=-Djavax.net.debug=ssl)
- Or Set watt property "watt.config.systemProperties" (e.g. watt.config.systemProperties=javax.net.debug=ssl)
Both settings requires Integration Server to be restarted and the TLS/SSL debug messages will be written to the IS profile wrapper.log file.
Understanding SSL Handshake
During SSL handshake , a client and server secures the socket connection by exchanging digital certificates and agreeing on the cipher suite and other algorithms. Details of the SSL handshake protocol can be found in the respective RFCs.
The key things that one needs to understand in the SSL handshake for debugging purpose are:
- Client and Server negotiates on the SSL protocol version and cipher suites and compression methods at the start of the handshake.
- Server will always sends its certificate to Client and Client will verifies it against the trusted certificates stored in client's truststore.
- Sever generally also sends a list of certificates (CA certificates), it trust. These CA certificates are stored in the Server's truststore.
- If Server requires Client to be authenticated, then Client will send it's certificate. But client will send it's certificate only if the clients's certificate is issued by one of the CA certificate from CA certificates list sent by the server.
- Server will verify client's certificate against the trusted certificates stored in Sever's truststore.
TLS/SSL Handshake Issues & Debugging
Issue 1: Client and Server fails to negotiate on protocol version/cipher suites/compression method
If the Server doesn't support the protocol version/cipher suites that client sends in the ClientHello message, then the handshake fails before any certificates are exchanged.
For example, below handshake failed as Integration Server using Entrust IAIK connects to a TLS 1.2 enabled server. In this case, change from IAIK to JSSE option and set "watt.net.jsse.client.enabledProtocols=TLSv1.2"
jvm 1 | ssl_debug(3): Starting handshake (iSaSiLk 3.03)...
jvm 1 | ssl_debug(3): Remote client:151.193.164.20:443, Timestamp:Wed Mar 16 18:51:40 IST 2016
jvm 1 | ssl_debug(3): Sending secure renegotiation cipher suite
jvm 1 | ssl_debug(3): Sending v3 client_hello message, requesting version 3.1...
jvm 1 | ssl_debug(3): Received alert message: Alert Fatal: handshake failure
jvm 1 | ssl_debug(3): SSLException while handshaking: Peer sent alert: Alert Fatal: handshake failure
jvm 1 | ssl_debug(3): Shutting down SSL layer...
jvm 1 | ssl_debug(3): Closing transport...
For Inbound SSL connection (i.e. Integration Server is acting as a SSL server) , the following watt properties control the protocol version and ciphersuites:
IAIK: watt.net.ssl.server.handshake.minVersion,watt.net.ssl.server.handshake.maxVersion, watt.net.ssl.server.strongcipheronly,watt.net.ssl.server.cipherSuiteList.
JSSE:watt.net.jsse.server.enabledProtocols,watt.net.jsse.server.enabledCipherSuiteList
For outbound SSL connection (i.e. Integration Server is acting as SSL client), the following watt properties control the protocol version and ciphersuites:
IAIK:watt.net.ssl.client.handshake.minVersion,watt.net.ssl.client.handshake.maxVersion, watt.net.ssl.client.strongcipheronly,watt.net.ssl.client.cipherSuiteList
JSSE:watt.net.jsse.client.enabledProtocols,watt.net.jsse.client.enabledCipherSuiteList.
Note: For JSSE, more TLS configurations can be found in jvm/jre/lib/security/java.security file.
Issue 2: Integration Server is not able to connect to SNI enabled server (e.g. IIS 8.0)
Connections fails at the start of the handshake if Integration Server connects to a SNI enabled server. Integration Server doesn't support SNI (Server Name Indication) extension as a Server. As a Client, Integration Server partially supports SNI extension through JSSE library (SNI support is added in Java 7), So user must set either useJSSE=yes in the pub.client:http or set the watt property “watt.net.ssl.client.useJSSE=true” for Integration Server to send the SNI extensions in the ClientHello message during handshake.
If JSSE is used, SNI extension can be seen in the SSL debug message as shown below:
*** ClientHello, TLSv1.2
RandomCookie: GMT: 1441360890 bytes = { 57, 241, 207, 178, 190, ...., 21, 68, 223, 117, 72,166, 105 }
Session ID: {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, ...
Compression Methods: { 0 }
Extension elliptic_curves, curve names: {secp256r1, sect163k1,...sect283r1, ...secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA,..., MD5withRSA
Extension server_name, server_name: [type=host_name (0), value=xxx.yyy.zzz.net]
Issue 3: Server certificate rejected by the client
If the Server certificate is rejected by the Integration Server (acting as a client), then you may see debug message like show below for IAIK:
jvm 1 | ssl_debug(1): Received certificate handshake message with server certificate.
jvm 1 | ssl_debug(1): Server sent a 2048 bit RSA certificate, chain has 2 elements.
jvm 1 | ssl_debug(1): Sending alert: Alert Fatal: bad certificate
jvm 1 | ssl_debug(1): Shutting down SSL layer...
jvm 1 | ssl_debug(1): SSLException while handshaking: Server certificate rejected by ChainVerifier
jvm 1 | ssl_debug(1): Closing transport...
The possible reasons for the exception are:
1. The Server's CA/ROOT certificate is not present in the Integration Server (acting as client) truststore that is configured at "Security > Certificates" section and hence not able to verify the Server certificate. Add Server CA/ROOT certificate to the truststore specified in the "Security > Certificate" section.
2. If there is no truststore specified in the "Security > Certificates", then watt property "watt.security.cert.wmChainVerifier.trustByDefault" value could be "false". Set the watt property value to "true".
3. Server certificate might have expired. To ignore expired certificate, set the watt property "watt.security.ssl.ignoreExpiredChains" to "true".
Issue 4: Client is not sending it's certificate
When Clients needs to be authenticated, then Server sends CertificateRequest message during SSL handshake and also server sends a list of CA certificatesduring handshake. If Client doesn't send its certificate, then handshake fails with exception like show below:
jvm 1 | ssl_debug(4): Received certificate_request handshake message.
jvm 1 | ssl_debug(4): Accepted certificate types: RSA, DSS
jvm 1 | ssl_debug(4): Accepted certificate authorities:
jvm 1 | ssl_debug(4): cn=ca,ou=ca,o=ca,l=BLR,st=KA,c=IN
jvm 1 | ssl_debug(4): cn=leaf,ou=leaf,o=leaf,l=BLR,st=KA,c=IN
jvm 1 | ssl_debug(4): Received server_hello_done handshake message.
jvm 1 | ssl_debug(4): No client certificate available, sending empty certificate message...
jvm 1 | ssl_debug(4): Sending client_key_exchange handshake message (2048 bit)...
jvm 1 | ssl_debug(5): Received certificate handshake message with client certificate.
jvm 1 | ssl_debug(5): Certificate message is empty, client has no certificate.
jvm 1 | ssl_debug(5): Sending alert: Alert Fatal: bad certificate
jvm 1 | ssl_debug(5): Shutting down SSL layer...
In JSSE debug mode, the Certificate request along with Certificate authorities list can be seen as below:
*** CertificateRequest
Cert Types: RSA, DSS,
Cert Authorities:
CN=Duke, OU=Java Software, O="Sun Microsystems, Inc.",
L=Cupertino, ST=CA, C=US>
*** ServerHelloDone
If Integration Server (acting as Client) is not sending it's certificate then possible reasons are:
1. Ceriticates are not configured properly for the Integration Server. Set the Certificates to be used for outbound connection at "SSL Key" fields in "Security > Certificates" section. If pub.security:setKeyAndChain is used, then check whether it is using the correct keystore or not.
2. Integration Server's certificate is not issued by any CA certificates send by the server during the handshake. Add Integration Server's CA/ROOT certificates in the Sever's truststore.
3. If the Certificate Authorities list sent by the Server during the handshake is empty, then also Integration Server will not send it's certificate unless the watt property "watt.security.ssl.client.ignoreEmptyAuthoritiesList" is set to "true".
Issue 5: Server is rejecting the client certificate
If the Server doesn't trust the client's certificate, then the handshake will fail with exception saying "bad certificate". This is similar to issue 3 and in this case, the Integration Server (acting as client) CA/ROOT certificates are not present in the Sever's truststore.
Add Integration Server's CA/ROOT certificates in the Server's truststore.
Issue 6: Certificates version is X509 ver 1
Handshake sometimes fails if the Certificates are of old version (i.e X509 version 1). X509 v1 certificate is similar to X509 v3 certificate except that it doesn't have the extensions. Nowadays, very few server issues X509 v1 certificates, but some servers still do.