Troubleshooting and FAQs for SUM Connectivity Security Proxies and other related settings

This technical article is targeting anybody who has intermediate technical knowledge about SAG Update Manager (SUM) and some base knowledge about Java certificates and truststores.

Here we cover a collection of the frequently asked questions about Software AG Update Manager (SUM) related to network, connectivity, proxiesm, and other configurations and troubleshooting.

Part One - How to do proper diagnostics of Connectivity Issues. Solutions

Question: Which are the Connectivity Diagnostic steps to follow?

Also: How to do the proper connectivity issues diagnostics and check if we have a SUM-related issue? Check the connectivity to the servers which are part of the SUM infrastructure (i.e. SDC server).

Steps to follow here:

Try to reach out the SDC (Software Download Center) server over another, an independent third-party tool like a web browser, cURL, Postman or another.

You can test against any arbitrary server you are interested in.

Example with SDC server:

You can try to ping SDC with a browser or use cURL:

curl -k https://sdc.softwareag.com/sumv2/api/version

with both HTTP and HTTPS. It should return a version like “10.0.0.0623-0623”, which is the current version of the SUM server.
As for specific ports, SUM uses the standard HTTP and HTTPS ports, so there should not be any need to open ports.

curl -k https://sdc.softwareag.com/sumv2/api/version

If you CAN NOT establish the connection, then the Connectivity Issue is not SUM specific one. In this situation you should contact your System Administrator.

If the connection to be diagnosed is over HTTPS you should be aware that a Software AG certificate is used.

This certificate should not be added manually to the browser truststore list. This is because on your local IE, Firefox and Chrome browsers this CA should be part of the trusted CAs list. Thus the Software AG certificate should be trusted out of the box.

If the above is working and there are network glitches you can try to mimic what SUM fails to do by using this command to download a component e.g.:

curl -Lk https://sdc.softwareag.com/sumv2/api/core/download/GA_Fix_Repo?update=java-W64_10.1.0.0000-0033 --output jvm.zip

This should download the image component that SUM fails to download.

Sticking to the above example after downloading jvm.zip it should be ~38.2MB and have a proper sha256 checksum that can be checked from SUM V2 Server.

To verify that jvm.zip is downloaded correctly run:

sha265sum jvm.zip

and it should return the same checksum as set at the server for the given component.

For https://empower.softwareag.com EQUAL TO sdc.softwareag.com URL:

The new certificate is now signed by ’ GlobalSign ', while the previous was signed by ’ Thawte '.

Certificates issued by any vendor , which is a root CA should be trusted by the JVM and should be found into the default jvm/jre/lib/security/cacerts truststore . Otherwise this should be manually added.
Do not use this truststore to store custom (e.g. proxy) certificates. If you add a certificate in this truststore it will be deleted when the JVM is updated. SUM exposes a better way to do that. Please, refer the documentation you can find below.

Note: Previously it was signed by: CA – Thawte RSA CA and Verified by DigiCert Inc

Warning: Unfortunately, HP Java does not ship any Root-CAs of ’ GlobalSign ’ yet, so this leads to TLS Handshake issue (e.g. CertificateException). That is why SUM Bootstrappers/Installer are shipped with a modified HP-UX java.

Anyway, there are deviations between different browsers and there are cases when a given browser vendor does not trust by default a given CA .

You should check if the Thawte RSA CA is listed as “Trusted Root Certification Authorities” .

If not trusted then the certificate should be added manually.

Anyway, if a Proxy is used, then some extra configurations should be done by your local System Administrator and the Proxy should be set up to trust the certificate.

Servers certificate chain

Regarding rfc5246 - RFC 5246: The Transport Layer Security (TLS) Protocol Version 1.2 server must send complete certificate chain.

Аlthough all three certificates are in the output of “keytool -list …”, e.g.:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 3 entries

Alias name: idata

Creation date: 9 Nov 2021

Entry type: PrivateKeyEntry

Certificate chain length: 1

Certificate[1]:

.......

server does not provide certificate chain.

You can verify it with:

openssl s_client -showcerts -connect <SERVER_TO_SSL>:443

Adding intermediate certificate to the client cacerts file is the workaround you can use in this case.

SSL and Connection Diagnostic Tools:

SSL connection diagnostics can be done with SSLPoke Attlassian’s tool. More you can read here:

https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-779355358.html

https://confluence.atlassian.com/kb/ssl-tls-troubleshooting-755140932.html

TL;TR

Here are examples of how you can test if given Java and its certificate truststore can establish SSL connection with a given server.

SSLPoke examples:

/usr/java/jdk1.8.0_60/bin/java SSLPoke <SERVER_TO_SSL> 443

/usr/java/jdk1.7.0_45/bin/java SSLPoke <SERVER_TO_SSL> 443

Successfully connected

/usr/java/jdk1.7.0_45/bin/java -Djavax.net.ssl.trustStore=/my/custom/truststore SSLPoke <SERVER_TO_SSL> 443

Successfully connected

To diagnose a connection another very useful set of tools are Postman, Fiddler and Wireshark .

Question: How I could configure my truststore (certificate chain)?

Relates to: exceptions like CertPathBuilderException, CertificateException, SSLHandshakeException.

Let us brief the theory a bit. An SSL connection succeeds only if the client can trust the server.

When we establish a connection over HTTPS, the web server will respond by providing its site and intermediate certificates. It is then up to the client to complete the chain by having the root certificate. This chain validation is necessary for the client to trust the site.

Self-signed Certificates

Certificates not issued by known CA but rather by the server hosting the certificate are called self-signed.

These are often used in internal development environments that are not customer facing.

!This is the preferred solution!
Thus the customer keeps the configuration separate from the standard JVM truststore and thus the trust store is not overridden during JVM update (e.g. throughout SUM update)!

You can add it to a custom store and then use java system property:

-Djavax.net.ssl.trustStore=<path_to_the_custom_truststore>

More information here: Debugging SSL/TLS connections

Warning: This parameter is a Java one. You can NOT provide it directly to the SUM Installer script.

Warning: If you do not have direct access to the java home to be used (e.g. when executing a SUM installer script that contains the Java into it) then you should set the parameter into the JAVA_OPTIONS environment variable. For example in Linux:

export JAVA_OPTIONS="$JAVA_OPTIONS -Djavax.net.ssl.trustStore=<path_to_the_custom_truststore>"

SUM V11

Since SUM V11 there is a dedicated JAVA_OPTIONS_SUM_V11 . Alternative to the classical JAVA_OPTIONS environment variable. The idea behind JAVA_OPTIONS_SUM_V11 is that set globally it will not interfere with other products/services/processes that make use of JAVA_OPTIONS one.

Thus, no side effects and trusts store shadowing/masking will be possible. E.g. :

export JAVA_OPTIONS_SUM_V11="$JAVA_OPTIONS_SUM_V11 -Djavax.net.ssl.trustStore=<path_to_the_custom_truststore>"

export JAVA_OPTIONS_SUM_V11 ="$JAVA_OPTIONS_SUM_V11 -[Djavax.net](http://Djavax.net).debug=ssl:handshake

SUM V10

Since SUM V10 with Launcher 10.1.0.0000-0217 there is a dedicated JAVA_OPTIONS_SUM_V10 , too. Alternative to the classical JAVA_OPTIONS environment variable. The idea behind JAVA_OPTIONS_SUM_V10 is that set globally it will not interfere with other products/services/processes that make use of JAVA_OPTIONS one.

Thus, no side effects and trusts store shadowing/masking will be possible.

Warning:
It is highly recommended to use JAVA_OPTIONS_SUM_V10 .

Please, reconfigure your custom trust store when you update to Launcher 10.1.0.0000-0217 or later!

E.g.:

export JAVA_OPTIONS_SUM_V10="$JAVA_OPTIONS_SUM_V10 -Djavax.net.ssl.trustStore=<path_to_the_custom_truststore>"

export JAVA_OPTIONS_SUM_V10=$JAVA_OPTIONS_SUM_V10 -[Djavax.net](http://Djavax.net).debug=ssl:handshake

Important:
Since the keystore only gets read once when the JVM is initialized, please restart the source application service after importing the new certificate(s).

When CCE is used to call SUM

If SUM is used in conjunction with CCE

After installation you might set com.softwareag.plm.sum.cc.java.truststore =<location_to_trustore> system property in SPM (and restart it).

Using this property you can change the default java truststore used by Update Manager.
It is recommended to point this location to the installation java directory where the certificates were already imported. For example:

com.softwareag.plm.sum.cc.java.truststore=<CCE_installDir>/jvm/jvm/jre/lib/security/cacerts

where <CCE_installDir> is the full path to the installation directory.

Thus, whenever there is a call to SUM from CCE this truststore location will be considered by SUM for the establishment of any SSL connection.

Anyway, there are several approaches on how to configure and consume a custom truststore:

  1. You can provide it throughout the following parameter [Djavax.net](http://djavax.net/).ssl.trustStore=<path_to_the_truststore> when you run SUM Client.

-Djavax.net.ssl.trustStore=<path_to_the_custom_custom_truststore>

  1. You can set/export (windows/linux) it into JAVA_OPTIONS/JAVA_OPTIONS_SUM_V11/JAVA_OPTIONS_SUM_V10 variable for the current SHELL only – not that much benefit here…

  2. Set JAVA_OPTIONS/JAVA_OPTIONS_SUM_V11/JAVA_OPTIONS_SUM_V10 permanently , or system wide (all users and processes) . This is well documented for both Windows and Linux (search the net for something like “How do I add environment variables?” ). Not a good solution, as this may interfere with a process that needs another custom truststore.

Option 1

You can create an alternate default file called jssecacerts in the same location as the cacerts file.

https://www.cloudera.com/documentation/enterprise/5-2-x/topics/cm_sg_create_key_trust.html

Warning: Will be reset during Java update. So java restart will be needed.

Option 2

Add it to your Java JRE truststore, usually available at the following path - $JAVA_HOME/jre/lib/security/cacerts file. This contains the default CA information shipped with the JDK. Thus it will be effective for whatever application run with this JRE.

Add the missing keystore/certificate to your certificate truststore:

Warning: Will be reset during Java update. So java restart will be needed.

Option 3

From within your browser, e.g. Chrome, you could download certificate in CER format.

From the image bellow you could see the certificate in Chrome and we download it with the following button “Copy to File”

Then you have to follow these steps to include certificate in Java JRE truststore:
a) Download and Install KeyStore Explorer - KeyStore Explorer

There is two ways to continue - GUI or CMD steps:
b) GUI

  • start KeyStore Explorer and open Java cacerts file with the password " changeit ". For Example:
    C:\Program Files\Java\jre1.8.0_65\lib\security\cacerts

  • use the “red ribbon” icon in the middle of the taskbar - “Import Trusted Certificate” to import CER file that is already downloaded.

  • save and exit
    c) CMD

  • execute the following command in Command prompt as Administrator:
    <Path to JAVA>\bin\keytool.exe -import -file <File Path to CER file> -keystore <Path to JAVA cacert file> -alias <alias of certificate>

  • enter password " changeit " for following the message “Enter keystore password:”

  • enter " YES " for the following message “Trust this certificate? [no]:”

Question: I am running SUM throughout Proxy and experience TLS and certificate issues?

Relates to errors like: java.security.cert.CertificateException , javax.net.ssl.SSLHandshakeException etc. What to do?

The possible problems could be:

  1. Customer Infrastructure team should add an exception to SSL decryption (for sdc.softwareag.com) into the Network.

  2. The Proxy certificate should be added into the truststore of the client.

This link shows the proper way this should be done with SUM: Configure My Certificates Truststores (in the current document - Question: How I could configure my trust store (certificate chain), so it is considered when SUM establishes TLS/SSL communication? Thus avoiding exceptions like CertPathBuilderException, CertificateException, SSLHandshakeException.)

  1. The client may have missed to add their Proxy certificate into the truststore of the Proxy.

How to check if the Proxy certificate is into the truststore? :

This can be detected if you rum SUM with “-Djavax.net.debug=ssl:handshake” java parameter.

Then, looking into the SUM console log and read from:

"init truststore

adding as trusted cert:"

up-to

"keyStore is :"

Here, if the Proxy certificate was not added, then you should NOT see the specific “adding as trusted cert:” section that adds the Proxy cert in question.

The customer may have configured its Proxy in order to be able to monitor (decrypt) the TLS traffic. This is done with some truststore configuration techniques.

Fixing this by the customer’s proxy administrator should have the issue solved.

If the issue still persists after fixing it, please, run again with “-Djavax.net.debug=ssl:handshake” and send us the log artifacts/files.

Windows:

set JAVA_OPTIONS=%JAVA_OPTIONS% -Djavax.net.debug=ssl:handshake

Linux:

export JAVA_OPTIONS="$JAVA_OPTIONS -Djavax.net.debug=ssl:handshake"

Then you should reset the JAVA_OPTIONS back, as it could be used by other Java processes of other products.

Question: I am using Firewall/Proxy and experience some Connectivity issues?

Relates to: javax.net.ssl.SSLHandshakeException, “Connection timed out: connect”, java.net.UnknownHostException, Unable to tunnel through proxy. Proxy returns “HTTP/1.1 407 Proxy Authentication Required”

Both sdc.softwareag.com and empower.softwareag.com should be added to the Firewall/Proxy settings.

You should try to connect to these using alternative clients like Postman, Browsers etc.

Warning - Unable to tunnel through proxy. Proxy returns “HTTP/1.1 407 Proxy Authentication Required”:

Initial connectivity check can pass and SUM can go in ONLINE mode . At this stage Empower credentials are also validated. Be aware that ONLINE mode check is a sort of a ping check with the Servers involved.

Anyway, when image is created or product is updated (i.e. Fixes are needed) then there is a file download involved. In this cases it is possible to experience some Proxy issues if the Proxy has some rules set that limit the network communication over, file types, number of connections established etc.

More about how to do diagnostics here:

Question: Connectivity Diagnostic steps. How to do the proper connectivity issues diagnostics if this is a SUM related issue? Check the servers which are part of the SUM infrastructure (i.e. SDC server).

A good troubleshooting step is to change the proxy in use. Usually, proxy and network configurations are causing the connectivity issues.

Another possibility is to have a Firewall that is set to check the certificate to be used over a SSL connection. If your Proxy is using custom certificate you should follow the instructions:

Question: How I could configure my truststore (certificate chain)?

If the problem persists you should contact your system administrator and check the Firewall, Gateway, Proxy logs.

Some issues could be caused by Proxy settings like:

  • The IP/domain or port is incorrect
  • The IP/domain or port (i.e service) is down
  • The IP/domain is taking longer than your default timeout to respond - poor network
  • You have a firewall that is blocking requests or responses on the port you are using
  • You have a firewall that is blocking requests to that particular host - should whitelist, see below
  • Your internet access is down
  • There are specific Proxy rules setup that limit the network communication over, file types, number of connections established etc.
  • There are security policies. (You were correct that is was a proxy issue. Our Networking Team implemented new security policies that prevented the connection to sdc.softwareag.com, empower.softwareag.com and sdc-hq.softwareag.com)
  • There is a content filtering setup.
  • added an exception to SSL decryption (for sdc.softwareag.com) in the Network.

! You should whitelist the connections to sdc.softwareag.com, sdc-hq.softwareag.com and empower.softwareag.com to unblock HTTP/HTTPS.

Please, do not use IP range, as this could change over time. You should use names (i.e. sdc.softwareag.com, empower.softwareag.com and sdc-hq.softwareag.com).

Connectivity issues - Miscellaneous list

Here we will cover some miscellanies connectivity issues caused in more specific scenarios and setups like use of Network Throttling, NTLM proxies etc.

Question: I am using NTLM proxy server over Unix machine and cannot make connection. Why?

(Exception in thread “main” java.lang.RuntimeException: java.util.NoSuchElementException at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1506) )

If you have NTLM proxy setup in your environment and you are using SUM on a UNIX machine, hidden behind the NTLM proxy, you may have connectivity issues with reaching SDC. The root cause is related to some specific behavior of the NTLM proxy on Unix machines. All you should do to resolve this problem is to set the two variables below in the same session, where you are running SUM:

export http_proxy=http://<USERNAME>:<PASSWORD>@<SERVER>:<PORT>/

export https_proxy=https://<USERNAME>:<PASSWORD>@<SERVER>:<PORT>/

This will guarantee that the OS will be notified explicitly for using NTLM proxy and will resolve the connectivity problem.

Question: java.net.SocketException was caught while trying to connect to the SUM server.

If you see that java.net.SocketException is registered into the SUM Launcher log:

2019/11/28 13:12:36 FINEST | The SUM server is sdc.softwareag.com. Checking if it is up...

2019/11/28 13:12:36 FINEST | Checking connectivity to the SUM server: sdc.softwareag.com

2019/11/28 13:12:36 FINEST | The following URL is used: https://sdc.softwareag.com/sumv2/api/version

2019/11/28 13:12:47 FINEST |

2019/11/28 13:12:47 FINEST | java.net.SocketException was caught while trying to connect to the server!

2019/11/28 13:12:47 FINEST | Message: java.net.SocketException: Socket is closed

2019/11/28 13:12:47 FINEST |

2019/11/28 13:12:47 INFO | [WARNING] sdc.softwareag.com is NOT up

2019/11/28 13:12:47 INFO | [WARNING] Update Manager will start without updating its components

There can be a number of reasons for that, for example:

  • network failure
  • firewall timeout
  • account permissions

If there are sporadic log like the one showed up sdc.softwareag.com is NOT up’ mixed with sdc.softwareag.com is up’ entries:

2019/11/28 13:10:18 FINEST | The SUM server is sdc.softwareag.com. Checking if it is up...

2019/11/28 13:10:18 FINEST | Checking connectivity to the SUM server: sdc.softwareag.com

2019/11/28 13:10:18 FINEST | The following URL is used: https://sdc.softwareag.com/sumv2/api/version

2019/11/28 13:10:19 FINEST |

2019/11/28 13:10:19 FINEST | sdc.softwareag.com is up

2019/11/28 13:10:19 FINEST |

Then it is almost certain that there is a network issue and you should check with your local administrator. Your network, proxy, firewall etc. should be checked.

Useful links

Secure Sockets Layer (SSL) Overview

SSL TLS troubleshooting

Debugging SSL/TLS Connections

Java ConnectException

Atlassian SSLPoke Tool

Most Common Java Keytool keystore commands

Connecting to SSL services | Atlassian Support | Atlassian Documentation

2 Likes