SUM Part One: SUM Troubleshooting and FAQs. SUM Connectivity, Security, Proxies and other related settings

Dobri_Kitipov · April 6, 2022, 7:08am

This technical article is targeting anybody who has intermediate technical knowledge about SAG Update Manager (SUM) and some base knowledge about Java certificates and truststores.

Here we cover a collection of the frequently asked questions about Software AG Update Manager (SUM) related to network, connectivity, proxies, and other configurations and troubleshooting.

Part One - How to do proper diagnostics of Connectivity Issues. Solutions

This is part one of a SUM article series:
SUM Part Two

Question: Which are the Connectivity Diagnostic steps to follow?

Also: How to do the proper connectivity issues diagnostics and check if we have a SUM-related issue? Check the connectivity to the servers which are part of the SUM infrastructure (i.e. SDC server).

Steps to follow here:

Try to reach out the SDC (Software Download Center) server over another, an independent third-party tool like a web browser, cURL, Postman or another.

You can test against any arbitrary server you are interested in.

Example with SDC server:

You can try to ping SDC with a browser or use cURL:

curl -k https://sdc.softwareag.com/sumv2/api/version

with both HTTP and HTTPS. It should return a version like “10.0.0.0623-0623”, which is the current version of the SUM server.
As for specific ports, SUM uses the standard HTTP and HTTPS ports, so there should not be any need to open ports.

curl -k https://sdc.softwareag.com/sumv2/api/version

If you CAN NOT establish the connection, then the Connectivity Issue is not SUM specific one. In this situation you should contact your System Administrator.

If the connection to be diagnosed is over HTTPS you should be aware that a Software AG certificate is used.

This certificate should not be added manually to the browser truststore list. This is because on your local IE, Firefox and Chrome browsers this CA should be part of the trusted CAs list. Thus the Software AG certificate should be trusted out of the box.

If the above is working and there are network glitches you can try to mimic what SUM fails to do by using this command to download a component e.g.:

curl -Lk https://sdc.softwareag.com/sumv2/api/core/download/GA_Fix_Repo?update=java-W64_10.1.0.0000-0033 --output jvm.zip

This should download the image component that SUM fails to download.

Sticking to the above example after downloading jvm.zip it should be ~38.2MB and have a proper sha256 checksum that can be checked from SUM V2 Server.

To verify that jvm.zip is downloaded correctly run:

sha265sum jvm.zip

and it should return the same checksum as set at the server for the given component.

For https://empower.softwareag.com EQUAL TO sdc.softwareag.com URL:

The new certificate is now signed by ’ GlobalSign ', while the previous was signed by ’ Thawte '.

Certificates issued by any vendor , which is a root CA should be trusted by the JVM and should be found into the default jvm/jre/lib/security/cacerts truststore . Otherwise this should be manually added.
Do not use this truststore to store custom (e.g. proxy) certificates. If you add a certificate in this truststore it will be deleted when the JVM is updated. SUM exposes a better way to do that. Please, refer the documentation you can find below.

Note: Previously it was signed by: CA – Thawte RSA CA and Verified by DigiCert Inc

Warning: Unfortunately, HP Java does not ship any Root-CAs of ’ GlobalSign ’ yet, so this leads to TLS Handshake issue (e.g. CertificateException). That is why SUM Bootstrappers/Installer are shipped with a modified HP-UX java.

Anyway, there are deviations between different browsers and there are cases when a given browser vendor does not trust by default a given CA .

You should check if the Thawte RSA CA is listed as “Trusted Root Certification Authorities” .

If not trusted then the certificate should be added manually.

Anyway, if a Proxy is used, then some extra configurations should be done by your local System Administrator and the Proxy should be set up to trust the certificate.

Servers certificate chain

Regarding rfc5246 - RFC 5246: The Transport Layer Security (TLS) Protocol Version 1.2 server must send complete certificate chain.

Аlthough all three certificates are in the output of “keytool -list …”, e.g.:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 3 entries

Alias name: idata

Creation date: 9 Nov 2021

Entry type: PrivateKeyEntry

Certificate chain length: 1

Certificate[1]:

.......

server does not provide certificate chain.

You can verify it with:

openssl s_client -showcerts -connect <SERVER_TO_SSL>:443

Adding intermediate certificate to the client cacerts file is the workaround you can use in this case.

SSL and Connection Diagnostic Tools:

SSL connection diagnostics can be done with SSLPoke Attlassian’s tool. More you can read here:

https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-779355358.html

https://confluence.atlassian.com/kb/ssl-tls-troubleshooting-755140932.html

TL;TR

Here are examples of how you can test if given Java and its certificate truststore can establish SSL connection with a given server.

SSLPoke examples:

/usr/java/jdk1.8.0_60/bin/java SSLPoke <SERVER_TO_SSL> 443

/usr/java/jdk1.7.0_45/bin/java SSLPoke <SERVER_TO_SSL> 443

Successfully connected

/usr/java/jdk1.7.0_45/bin/java -Djavax.net.ssl.trustStore=/my/custom/truststore SSLPoke <SERVER_TO_SSL> 443

Successfully connected

To diagnose a connection another very useful set of tools are Postman, Fiddler and Wireshark .

Question: How I could configure my truststore (certificate chain)?

Relates to exceptions like: CertPathBuilderException, CertificateException, SSLHandshakeException.

Root vs. Intermediate Certificate

When importing a certificate into the JVM trust store, it is important to import the root CA certificate, not the intermediate one. This is because the root CA certificate is the top-level certificate in the certificate chain, and it is used to establish trust for all certificates issued by the CA.

The intermediate CA certificate, on the other hand, is issued by the root CA and is used to issue end-entity certificates (e.g., server certificates). The intermediate CA is not trusted directly by the JVM and must be presented along with the end-entity certificate to form a chain of trust.

So, when you import an intermediate CA certificate into the JVM trust store, it will not be recognized as a trusted root CA, and any certificates issued by it may still be rejected by the JVM. In other words, importing an intermediate CA certificate into the trust store is not enough to establish trust for the server’s end-entity certificate.

Therefore, it is essential to import the root CA certificate into the JVM trust store to establish trust for all certificates issued by the CA, including the server’s end-entity certificate. If the root CA certificate is not already in the trust store, you will need to obtain it from the CA and import it using the keytool command-line tool as described in the previous answer.

Let us brief the theory a bit. An SSL connection succeeds only if the client can trust the server.

When we establish a connection over HTTPS, the web server will respond by providing its site and intermediate certificates. It is then up to the client to complete the chain by having the root certificate. This chain validation is necessary for the client to trust the site.

Self-signed Certificates

Certificates not issued by known CA but rather by the server hosting the certificate are called self-signed.

These are often used in internal development environments that are not customer facing.

General Step-by-step guide to import a certificate into the JVM trust store

Check the SSL certificate of the server: Verify that the server certificate is valid and issued by a trusted certificate >authority. You can use a tool like OpenSSL to connect to the server and check the certificate details.

Check the SSL configuration of the client: Verify that the client’s SSL configuration is correct and matches the server’s >SSL configuration. Make sure that the client is using the correct protocol version, cipher suite, and certificate.

Check the trust store: Verify that the trust store contains the root certificate of the server’s certificate chain. The trust >store is a repository of trusted certificates that the JVM uses to verify the identity of remote servers. By default, the JVM >trust store is located at $JAVA_HOME/lib/security/cacerts.

Import the root certificate: If the root certificate is not already in the trust store, you will need to import it. You can use >the keytool command-line tool to import the certificate. The command to import a certificate is as follows:

keytool -import -alias <alias> -file <certificate_file> -keystore <trust_store_file>

Where <alias> is a name you choose to identify the certificate, <certificate_file> is the file containing the root >certificate, and <trust_store_file> is the path to the trust store file. You will be prompted to enter the keystore password.

Verify the import: Use the keytool command-line tool to verify that the certificate was imported correctly. The command to list the certificates in the trust store is as follows:

keytool -list -keystore <trust_store_file>

Restart the JVM: If you have made any changes to the trust store, you will need to restart the JVM for the changes to take effect.

Test the connection: Test the connection to the server again to verify that the SSL and certificate issues have been resolved.

!Impportant!
Best to keep the configuration separate from the standard JVM truststore and thus the trust store is not overridden during JVM update (e.g. throughout SUM update)!

You can add it to a custom store and then use java system property:

-Djavax.net.ssl.trustStore=<path_to_the_custom_truststore>

More information here: Debugging SSL/TLS connections

Warning: This parameter is a Java one. You can NOT provide it directly to the SUM Installer script.

Warning: If you do not have direct access to the java home to be used (e.g. when executing a SUM installer script that contains the Java into it) then you should set the parameter into the JAVA_OPTIONS environment variable. For example in Linux:

export JAVA_OPTIONS="$JAVA_OPTIONS -Djavax.net.ssl.trustStore=<path_to_the_custom_truststore>"

SUM V11

Since SUM V11 there is a dedicated JAVA_OPTIONS_SUM_V11 . Alternative to the classical JAVA_OPTIONS environment variable. The idea behind JAVA_OPTIONS_SUM_V11 is that set globally it will not interfere with other products/services/processes that make use of JAVA_OPTIONS one.

Thus, no side effects and trusts store shadowing/masking will be possible. E.g. :

export JAVA_OPTIONS_SUM_V11="$JAVA_OPTIONS_SUM_V11 -Djavax.net.ssl.trustStore=<path_to_the_custom_truststore>"

export JAVA_OPTIONS_SUM_V11 ="$JAVA_OPTIONS_SUM_V11 -[Djavax.net](http://Djavax.net).debug=ssl:handshake

SUM V10

Since SUM V10 with Launcher 10.1.0.0000-0217 there is a dedicated JAVA_OPTIONS_SUM_V10 , too. Alternative to the classical JAVA_OPTIONS environment variable. The idea behind JAVA_OPTIONS_SUM_V10 is that set globally it will not interfere with other products/services/processes that make use of JAVA_OPTIONS one.

Thus, no side effects and trusts store shadowing/masking will be possible.

Warning:
It is highly recommended to use JAVA_OPTIONS_SUM_V10 .

Please, reconfigure your custom trust store when you update to Launcher 10.1.0.0000-0217 or later!

E.g.:

export JAVA_OPTIONS_SUM_V10="$JAVA_OPTIONS_SUM_V10 -Djavax.net.ssl.trustStore=<path_to_the_custom_truststore>"

export JAVA_OPTIONS_SUM_V10=$JAVA_OPTIONS_SUM_V10 -[Djavax.net](http://Djavax.net).debug=ssl:handshake

Important:
Since the keystore only gets read once when the JVM is initialized, please restart the source application service after importing the new certificate(s).

When CCE is used to call SUM

If SUM is used in conjunction with CCE

After installation you might set com.softwareag.plm.sum.cc.java.truststore =<location_to_trustore> system property in SPM (and restart it).

Using this property you can change the default java truststore used by Update Manager.
It is recommended to point this location to the installation java directory where the certificates were already imported. For example:

com.softwareag.plm.sum.cc.java.truststore=<CCE_installDir>/jvm/jvm/jre/lib/security/cacerts

where <CCE_installDir> is the full path to the installation directory.

Thus, whenever there is a call to SUM from CCE this truststore location will be considered by SUM for the establishment of any SSL connection.

Anyway, there are several approaches on how to configure and consume a custom truststore:

You can provide it throughout the following parameter [Djavax.net](http://djavax.net/).ssl.trustStore=<path_to_the_truststore> when you run SUM Client.

-Djavax.net.ssl.trustStore=<path_to_the_custom_custom_truststore>

You can set/export (windows/linux) it into JAVA_OPTIONS/JAVA_OPTIONS_SUM_V11/JAVA_OPTIONS_SUM_V10 variable for the current SHELL only – not that much benefit here…
Set JAVA_OPTIONS/JAVA_OPTIONS_SUM_V11/JAVA_OPTIONS_SUM_V10 permanently , or system wide (all users and processes) . This is well documented for both Windows and Linux (search the net for something like “How do I add environment variables?” ). Not a good solution, as this may interfere with a process that needs another custom truststore.

Option 1

You can create an alternate default file called jssecacerts in the same location as the cacerts file.

https://www.cloudera.com/documentation/enterprise/5-2-x/topics/cm_sg_create_key_trust.html

Warning: Will be reset during Java update. So java restart will be needed.

Option 2

Add it to your Java JRE truststore, usually available at the following path - $JAVA_HOME/jre/lib/security/cacerts file. This contains the default CA information shipped with the JDK. Thus it will be effective for whatever application run with this JRE.

Add the missing keystore/certificate to your certificate truststore:

Warning: Will be reset during Java update. So java restart will be needed.

Option 3

From within your browser, e.g. Chrome, you could download certificate in CER format.

From the image bellow you could see the certificate in Chrome and we download it with the following button “Copy to File”

Then you have to follow these steps to include certificate in Java JRE truststore:
a) Download and Install KeyStore Explorer - KeyStore Explorer

There is two ways to continue - GUI or CMD steps:
b) GUI

start KeyStore Explorer and open Java cacerts file with the password " changeit ". For Example:
C:\Program Files\Java\jre1.8.0_65\lib\security\cacerts
use the “red ribbon” icon in the middle of the taskbar - “Import Trusted Certificate” to import CER file that is already downloaded.
save and exit
c) CMD
execute the following command in Command prompt as Administrator:
<Path to JAVA>\bin\keytool.exe -import -file <File Path to CER file> -keystore <Path to JAVA cacert file> -alias <alias of certificate>
enter password " changeit " for following the message “Enter keystore password:”
enter " YES " for the following message “Trust this certificate? [no]:”

Question: I am running SUM throughout Proxy and experience TLS and certificate issues?

Relates to errors like: java.security.cert.CertificateException , javax.net.ssl.SSLHandshakeException etc. What to do?

Generic steps you can follow to fix TLS and certificate issues when running SUM through a proxy:

Verify that the proxy is configured correctly: Make sure that the proxy settings are configured correctly in the SUM tool. Verify the proxy host, port, and credentials if required.

Check the SSL certificate of the server: Verify that the server’s SSL certificate is valid and issued by a trusted certificate authority. Use a tool like OpenSSL to connect to the server and check the certificate details.

Check the SSL configuration of the client: Verify that the client’s SSL configuration is correct and matches the server’s SSL configuration. Make sure that the client is using the correct protocol version, cipher suite, and certificate.

Import the root CA certificate: If the root CA certificate is not already in the trust store, you will need to import it. Follow the steps I provided earlier to import the certificate into the JVM trust store.

Check the proxy certificate: If the proxy server is using SSL, verify that its SSL certificate is valid and issued by a trusted certificate authority. If it is not trusted, you will need to import the root CA certificate of the proxy server into the >JVM trust store as well.

Use the right TLS version: If the server requires a specific TLS version, make sure that the client is configured to use that version. You can configure the TLS version using the -Dhttps.protocols JVM system property.

Use the right cipher suite: If the server requires a specific cipher suite, make sure that the client is configured to use that cipher suite. You can configure the cipher suite using the -Dhttps.cipherSuites JVM system property.

Disable hostname verification: If the server’s SSL certificate does not match the server’s hostname, you may need to disable hostname verification. You can do this using the -Dcom.sun.net.ssl.checkRevocation=false JVM system property.

Test the connection: Test the connection to the server through the proxy again to verify that the SSL and certificate issues have been resolved.

How to Setup my Proxy self-signed certificate in SUM?

Certificate issues happen when there is a proxy that has a self-singed TLS/SSL certificate that SUM does not trust.

In other, for SUM to connect to this proxy, the customer must provide SUM with the proxy certificate file in a PKCS12 format along with the encrypted version of the password that can unlock the PKCS12 file (.p12). To encrypt the password, they can use the password encrypt utility present in SUM. It can be found on the hamburger menu inside the SUM GUI.

After the password is encrypted, the customer must create a properties file with the following contents:

keystoreFile=<.p12 certificate>
keystorePass=<encrypted password>

Then when starting SUM, the customer must set the following environment variable:

SUM_SSL_CONF=<path to the properties file>

When started, SUM should pick up the certificate and be able to pass through the proxy.

Another more general approach.
You can also add it to a custom store and then use java native system property:

-Djavax.net.ssl.trustStore=<path_to_the_custom_truststore>
and if needed
-Djavax.net.ssl.trustStorePassword=<PASSWORD>

Some summary about possible problems:

Customer Infrastructure team should add an exception to SSL decryption (e.g. sdc.softwareag.com) into the Network.
Full list is:
sdc.softwareag.com
sdc-hq.softwareag.com
cdn.softwareag.com
empower.softwareag.com
The Proxy certificate should be added into the truststore of the client.

This link shows the proper way this should be done with SUM: Configure My Certificates Truststores (in the current document - Question: How I could configure my trust store (certificate chain), so it is considered when SUM establishes TLS/SSL communication? Thus avoiding exceptions like CertPathBuilderException, CertificateException, SSLHandshakeException.)

The client may have missed to add their Proxy certificate into the truststore of the Proxy.

How to check if the Proxy certificate is into the truststore? :

This can be detected if you rum SUM with “-Djavax.net.debug=ssl:handshake” java parameter.

Then, looking into the SUM console log and read from:

"init truststore

adding as trusted cert:"

up-to

"keyStore is :"

Here, if the Proxy certificate was not added, then you should NOT see the specific “adding as trusted cert:” section that adds the Proxy cert in question.

The customer may have configured its Proxy in order to be able to monitor (decrypt) the TLS traffic. This is done with some truststore configuration techniques.

Fixing this by the customer’s proxy administrator should have the issue solved.

If the issue still persists after fixing it, please, run again with “-Djavax.net.debug=ssl:handshake” and send us the log artifacts/files.

Windows:

set JAVA_OPTIONS=%JAVA_OPTIONS% -Djavax.net.debug=ssl:handshake

Linux:

export JAVA_OPTIONS="$JAVA_OPTIONS -Djavax.net.debug=ssl:handshake"

Then you should reset the JAVA_OPTIONS back, as it could be used by other Java processes of other products.

Question: I am using Firewall/Proxy and experience some Connectivity issues?

Relates to: javax.net.ssl.SSLHandshakeException, “Connection timed out: connect”, java.net.UnknownHostException, Unable to tunnel through proxy. Proxy returns “HTTP/1.1 407 Proxy Authentication Required”

Both sdc.softwareag.com and empower.softwareag.com should be added to the Firewall/Proxy settings.

You should try to connect to these using alternative clients like Postman, Browsers etc.

Warning - Unable to tunnel through proxy. Proxy returns “HTTP/1.1 407 Proxy Authentication Required”:

Initial connectivity check can pass and SUM can go in ONLINE mode . At this stage Empower credentials are also validated. Be aware that ONLINE mode check is a sort of a ping check with the Servers involved.

Anyway, when image is created or product is updated (i.e. Fixes are needed) then there is a file download involved. In this cases it is possible to experience some Proxy issues if the Proxy has some rules set that limit the network communication over, file types, number of connections established etc.

More about how to do diagnostics here:

Question: Connectivity Diagnostic steps. How to do the proper connectivity issues diagnostics if this is a SUM related issue? Check the servers which are part of the SUM infrastructure (i.e. SDC server).

A good troubleshooting step is to change the proxy in use. Usually, proxy and network configurations are causing the connectivity issues.

Another possibility is to have a Firewall that is set to check the certificate to be used over a SSL connection. If your Proxy is using custom certificate you should follow the instructions:

Question: How I could configure my truststore (certificate chain)?

If the problem persists you should contact your system administrator and check the Firewall, Gateway, Proxy logs.

Some issues could be caused by Proxy settings like:

The IP/domain or port is incorrect
The IP/domain or port (i.e service) is down
The IP/domain is taking longer than your default timeout to respond - poor network
You have a firewall that is blocking requests or responses on the port you are using
You have a firewall that is blocking requests to that particular host - should whitelist, see below
Your internet access is down
There are specific Proxy rules setup that limit the network communication over, file types, number of connections established etc.
There are security policies. (You were correct that is was a proxy issue. Our Networking Team implemented new security policies that prevented the connection to sdc.softwareag.com, sdc-hq.softwareag.com,empower.softwareag.com, cdn.softwareag.com
There is a content filtering setup.
added an exception to SSL decryption (for sdc.softwareag.com, sdc-hq.softwareag.com,empower.softwareag.com, cdn.softwareag.com) in the Network.

**! You should whitelist the connections to:

sdc.softwareag.com, sdc-hq.softwareag.com,empower.softwareag.com, cdn.softwareag.com to unblock HTTP/HTTPS.**

Please, do not use IP range, as this could change over time. You should use names (i.e. sdc.softwareag.com, empower.softwareag.com and sdc-hq.softwareag.com, , cdn.softwareag.com).

Connectivity issues - Miscellaneous list

Here we will cover some miscellanies connectivity issues caused in more specific scenarios and setups like use of Network Throttling, NTLM proxies etc.

Question: I am using NTLM proxy server over Unix machine and cannot make connection. Why?

(Exception in thread “main” java.lang.RuntimeException: java.util.NoSuchElementException at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1506) )

If you have NTLM proxy setup in your environment and you are using SUM on a UNIX machine, hidden behind the NTLM proxy, you may have connectivity issues with reaching SDC. The root cause is related to some specific behavior of the NTLM proxy on Unix machines. All you should do to resolve this problem is to set the two variables below in the same session, where you are running SUM:

export http_proxy=http://<USERNAME>:<PASSWORD>@<SERVER>:<PORT>/

export https_proxy=https://<USERNAME>:<PASSWORD>@<SERVER>:<PORT>/

This will guarantee that the OS will be notified explicitly for using NTLM proxy and will resolve the connectivity problem.

Question: java.net.SocketException was caught while trying to connect to the SUM server.

If you see that java.net.SocketException is registered into the SUM Launcher log:

2019/11/28 13:12:36 FINEST | The SUM server is sdc.softwareag.com. Checking if it is up...

2019/11/28 13:12:36 FINEST | Checking connectivity to the SUM server: sdc.softwareag.com

2019/11/28 13:12:36 FINEST | The following URL is used: https://sdc.softwareag.com/sumv2/api/version

2019/11/28 13:12:47 FINEST |

2019/11/28 13:12:47 FINEST | java.net.SocketException was caught while trying to connect to the server!

2019/11/28 13:12:47 FINEST | Message: java.net.SocketException: Socket is closed

2019/11/28 13:12:47 FINEST |

2019/11/28 13:12:47 INFO | [WARNING] sdc.softwareag.com is NOT up

2019/11/28 13:12:47 INFO | [WARNING] Update Manager will start without updating its components

There can be a number of reasons for that, for example:

network failure
firewall timeout
account permissions

If there are sporadic log like the one showed up ‘sdc.softwareag.com is NOT up’ mixed with ‘sdc.softwareag.com is up’ entries:

2019/11/28 13:10:18 FINEST | The SUM server is sdc.softwareag.com. Checking if it is up...

2019/11/28 13:10:18 FINEST | Checking connectivity to the SUM server: sdc.softwareag.com

2019/11/28 13:10:18 FINEST | The following URL is used: https://sdc.softwareag.com/sumv2/api/version

2019/11/28 13:10:19 FINEST |

2019/11/28 13:10:19 FINEST | sdc.softwareag.com is up

2019/11/28 13:10:19 FINEST |

Then it is almost certain that there is a network issue and you should check with your local administrator. Your network, proxy, firewall etc. should be checked.