Consumer Web Service Connector authentication issue using private key certificate keyAlias from keyStoreAlias

[StA_A_phane_Desmarais (Stéphane Desmarais)]

Hi,

This is for webMethods Integration server 9.10, but a test today on version 10.5 show same behavior.

I created a Consumer Web Service Descriptor from a WSDL file, as described in 9-10_Web_Services_Developers_Guide.pdf file (version 10.5 document is the same). I called it “testabc”. This generates properly a folder called “testabc_” with 3 folders called “connectors”, “docTypes” and “responseServices” which correspond to the service port defined in my WSDL as well as the data elements I defined in my WSDL.

The input/output of the newly created Web Service Connector are as explained in the Web Services Developers Guide under a section called “Signature for a Web Service Connector”.

If the provider of the web service does not ask for authentication of any sort, I am able to invoke this Web Service Connector by setting the request input (in my case: tns:request), setting up _url to the http or https address of the server which will process the transaction.

My issue is that I need to use a client certificate to authenticate at the transport level to a remote web service server, using https, and this is not working. The consumer Web Service Connector is not sending the client certificate to the remote server.

I added the remote server root CA and intermediate CA certificates in my <IIS install folderr>/common/conf/platform_truststore.jks by using a tool called “KeyStore Explorer”, then I reloaded the “DEFAULT_IS_TRUSTSTORE” using Web Methods admin console -> Security -> Keystore, and for DEFAULT_IS_TRUSTSTORE I clicked on the Reload icon. I’m not sure this was absolutely required to tell Integration Server to trust the certificates issued by this CA, but just in case I did that.

Then, to keep things simple, instead of trying to add a new private key under Security -> Keystore -> Keystore List, I decided to attempt to use the pre-existing default Key Alias key called ssos in Keystore alias DEFAULT_IS_KEYSTORE, as being the client certificate.

So I wrote a little wrapper around the web service connector to set the following inputs:

auth/transport/serverCerts/keyStoreAlias : DEFAULT_IS_KEYSTORE
auth/transport/serverCerts/keyAlias : ssos
_url: https:/my.remote.server.com/the/soap/service (this isn’t the real url)
useJSSE: Yes

Note: using useJSSE=Yes forces webMethods to use the Java security encryption routines instead of webMethods builtin routines, this is required to support TLSv1.1 and TLSv1.2

When I call the service, I get a fault with fault/reasons/reasons[0]/*body set to:
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

I used tcpdump to capture all the packages into a file, and then I used Wireshark to analyze that result, and what I see is that webMethods does not attempt to use any client certificate as can see in the trace below, packet 10 the “Certificate, Client Key Exchange”, in the details of the packet, the “Certificates Length” is set to 0 (no certificate was sent by web methods)

Please note that if I use something else than webMethods to call the same web service, the client cert is send and Certificates Length is then greater than zero.

I then looked at the Web Service Connector, and I see that it uses pub.client:soapClient. What I saw is quite puzzling, the auth parameter of soapClient is not defined the same way as the one from the web service connector. soapClient is using auth/transport/serverCerts/privateKey and auth/transport/serverCerts/certChain instead of auth/transport/serverCerts/keyStoreAlias and auth/transport/serverCerts/keyAlias :

The parameters of soapClient are defined in the webMethods Integration Server Built-In Services Reference and it does mention it should be serverCerts/privateKey and serverCerts/certChain.

How could that be possible to not be the same?

Thinking that maybe I really need to use what soapClient is telling to use, I edited the connector by calling pub.security.keystore:getKeyAndChain with input:
keyStoreAlias: DEFAULT_IS_KEYSTORE
keyAlias: ssos

this produces privateKey and certChain:

Which I then used as input to calling pub.client:soapClient:

When I run this in debug mode, the call to pub.security.keystore:getKeyAndChain returns:

privateKey: sun.security.rsa.RSAPrivateCrtKeyImpl@22eb04
certChain/certChain[0]: [B@c180430

and the call to soapClient returns:
response/fault/reasons/reasons[0]/*body:
com.wm.util.ServerException: WS_CLIENT_INVALID_PRIVATE_KEY

(and no call is made out to the remote server).

I was doing all my tests on WM9.10, but I recently got access to a WM10.5, so I imported my package in WM10.5 and the results are the same.

As anyone used transport level authentication with a client certificate successfully from a consumer web service?

Does anyone has a pointer as to what I am doing wrong?

[Gerardo_Lisboa]

Hi,

Do you need to do all those settings manually?

I usually recommend not to change any of the Cwsd connector properties but instead have a Web Service Consumer Endpoint defined where I set the address, user/pass and certificates required.

All the certificates are handled by the IS keystore and truststore, thus much easier to maintain.

Then I set the Cwsd handler port alias set to it.

The Integration server is then charged with all the communication details.

Also, this is much easier to deploy, as the definitions are centralized in the IS’ web administration pages.

Please check the service development help and IS administration documentation for more details.

Best regards,

[Holger_von_Thomsen]

Hi,

in addition to Gerardos Answer:
I would prefer creating a new Truststore with your custom CA chain as long as it is not available in central cacerts truststore file.
Same applies to the keystore file (create your own additional keystore for your certificate).

When using aliases to assign custom certificates (different from the central default IS Server certificate) you should check for the setKeyAndChain service (instead of the getKeyAndChain service mentioned above) and assign the custom aliases to this service inputs before calling the connector service. The privateKey and CAChain inputs of the connector service remain unmapped in this case.

Avoid to have to modify the content of generated connector service, just map the right values to its inputs. These are then checked inside the connector service and passed to the invocation of the soapClient service.

Regards,
Holger

[StA_A_phane_Desmarais (Stéphane Desmarais)]

Hi Gerardo,

I tried what you said. I used webMethods admin console and went to Settings -> Web Services and clicked on Create Web Service Endpoint Alias and created a consumer transport HTTPS. Here is a screen shot:

I opened up the Consumer web service descriptor. clicked on Binders tab, and set the Port alias to abc:

I attempted to invoke connector with only the request as input. A network capture still shows that Web Methods is not sending any client certificate to the server, because the “Certificate, Client Key Exchange” packet still shows a certificate length of 0.

I was not sure if I had done it the right way, so I edited my Web Service Consumer Endpoint called “abc” and instead of “localhost” I put “badlocalhost”, then submitted a claim. I got a fault :
java.net.UnknownHostException: badlocalhost

so that proves that it is using my Endpoint Alias called abc.

I also tried to not edit the binder, and instead set the endponitAlias service input when calling my connector:

That resulted in the same behavior, webMethods not using any client certificate authentication.

I am not dismissing your suggestion, in fact once I am able to get Web Methods to use a client certificate, I will likely use an Endpoint Alias.

I tried to use the builtin certificate ssos from the DEFAULT_IS_KEYSTORE, and also one I added myself, which is called abc and webMethods does not attempt to authenticate to the remote web service.

Any other idea how I can force webMethods to use a client certificate? Maybe some other configuration missing?

Any debug I can enable in Web Methods to better understand what is happening?

[StA_A_phane_Desmarais (Stéphane Desmarais)]

Hi Holger,

I agree that I should not be modifying the connector service generated by webMethods when I created the consumer Web Service Descriptor from the WSDL file. I was simply attempting to see if the fact that the “auth” parameter of the connector and the “auth” parameter of the soapClient are different could explain the fact that webMethods does not attempt to use a client certificate when establishing the HTTPS session with the remote server.

After trying the solution from Gerardo, using Consumer Web Services Endpoint Alias, I tried to use your solution by calling pub.security.keystore:setKeyAndChain. At first, I used:
keyStoreAlias: DEFAULT_IS_KEYSTORE
keyAlias: ssos

and then I called my WS connector right after.

tcpdump still shows that webMethods is not attempting to use any client certificate.

I then did as you suggested, I created a new keystore. This time I used “keytool” instead of “KeyStore Explorer”

keytool -importkeystore -destkeystore abc.jks -srckeystore my-private-key.pfx -srcstoretype PKCS12

where my-private-key.pfx is my private key in PKCS12 format.

I made sure to use the keytool from the jvm/jvm/bin folder located in the webMethods installation directory, not the one in my operating system default location

This created a file called abc.jks which I added in webMethods Admin Console Security -> Keystore -> Create Keystore Alias

On next screen, I see that it has a key alias called ‘1’. I had to then type in the private key password which protects my private key, and clicked “Save Changes”

This results in Keystore Alias ABC be created, with one key in it (called “1”):

I then updated my call to pub.security.keystore:setKeyAndChain to use:

keyStoreAlias: ABC
keyAlias: 1

I then ran again, setKeyAndChain returns without errors, and then calling my connector is still resulting in the following error:

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

and tcpdump shows WebMethod is not attempting to use any client certificate, the lenght of the cert staying as “0”.

As for the Truststore, I haven’t found how I am supposed to provide this to the Web Service connector. Even when using a Web Services Consumer Endpoints, I don’t see where I could specify the Trusted CA chain.

For now, I added my 2 trusted CA into DEFAULT_IS_TRUSTSTORE:

Any other ideas of things I might have missed, or how to debug this?

[Gerardo_Lisboa]

Hi,

I don’t need any of this if you set the certificate in the Web Service Endpoint alias and call the WSD connector using that alias (or have the WSD binder ‘Port alias’ set).

On the other hand, if the other party’s CA isn’t present in the default truststore of the JVM, the call will be rejected on your side.

You might be mislead by the name “DEFAULT_IS_TRUSTSTORE”; just because it says so, it doesn’t mean it is.

Try setting the Global truststore to one with the needed CAs (Security->Certificates->Trustore).

Best regards,

[Holger_von_Thomsen]

Hi Stéphane,

did you check which other options you have for auth method in the consumer endpoint alias?
Basic refers to user/password, you might have to choose another option for cert auth where you are able to provide the keystore alias to be used with this endpoint alias.

I just checked our implementation for our partnersystem which is using cert auth.
We jsut assign the Keystore alias and the key alias to the inputs of the consumer service for auth/transport/serverCerts and auth/message/serverCerts and this is working for us.
Please note that we are not able to use endpoint aliases in our implementation as we have different URLs (not only host and port) in our DEV/TST and QA/PROD environments. These URLs are currently configured in some properties files.
QA/PROD are using the cert auth via KeyStore while DEV/TST does not require this. Therefore we are checking in our code if the properties for the keystore alias and the key alias are set in the properties file or not.
When they are not set we are using Basic auth via user/password and when they are set, we are using the certificate auth as described above.

Regards,
Holger

[StA_A_phane_Desmarais (Stéphane Desmarais)]

Hi Gerardo,

I’m not exactly sure how to set the Global truststore.

I attempted to set these Web Methods extended settings:

watt.server.ssl.keyStoreAlias=ABC
watt.server.ssl.trustStoreAlias=DEFAULT_IS_TRUSTSTORE

I restarted Integration Server, but still not working.

Here are my Keystore and Truststore:

You also said :

@Gerardo_Lisboa

if the other party’s CA isn’t present in the default truststore of the JVM , the call will be rejected on your side.

but the tcpdump traces I saved to file and loaded in Wireshark show that it is not my side which is rejecting the TLS connection. It is the remote site. This is because the network trace show that webMethods is not sending the client cert.

I tried today using SoapUI, importing same WSDL file and using same client cert, and the network trace show the correct cert being sent by SoapUI.

I tried:

• using endpointAlias when calling the consumer WS connector
• using WSD binder ‘Port alias’
• using auth/transport/serverCerts/keyStoreAlias & auth/transport/serverCerts/keyAlias
• I even tried auth/message/serverCerts/keyStoreAlias & keyAlias (which in my case is not what I need)

There must be something else important I am missing.

[StA_A_phane_Desmarais (Stéphane Desmarais)]

Hi Holger,

auth method can only be:

• Basic
• Digest
• NTLM

My understanding is that these are only significant if you want to use user name & password, which is not my case. I did try all 3, and it does not make any difference, network trace show that Web Methods is not sending a client certificate.

I really grateful for your attempts to help me.

I’ll have to do some more reading in the Web Methods documentation.

[Holger_von_Thomsen]

Hi Stéphane,

are there any error messages in either server.log, error log or wireshark dump why the connection is rejected by the partner system?

The TLS Connection itself is only dealing with server certificates (and their CAs) defined on the HTTPS-Port for securing the transport layer. The Client certificate is part of the message used by the server at the partner to identify who you (not your server, but the user) are and if you are allowed to invoke this webservice or not.

When the transport layer fails, the client certificates does not matter (if it is correct or not).
When the transport layer succeeds, the ApplicationServer on the partner side will evaluate the client certificate if it belongs to a known user and if yes, then checks if this user is in the allow list of the ACL of the WebService being adressed.

Therefore I ask you to check at which point of the Handshake procedure (transport/application) the transaction fails.

Regards,
Holger

[StA_A_phane_Desmarais (Stéphane Desmarais)]

Hi Holger,

TLS Connection connection always includes server certificates, but may also include client certificate portion, if the remote server requests that the client present a certificate. this is described for example on the following web page: https://comodosslstore.com/blog/what-is-ssl-tls-client-authentication-how-does-it-work.html

My host (running Web Methods Integration server) is the client (the “consumer”). The remote server is where I have to connect (the provider). I do not control the remote server, and it is most likely not running Web Methods IS.

I attempted to resend again today, using an endpointAlias parameter (one of the various methods we can used as described earlier in this trail), this is what I see in the server.log file:

2021-01-05 12:49:55 EST [ISC.0077.9998E] Exception --> org.apache.axis2.AxisFault: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

After calling the connector, document transportInfo contains: status = 900 and statusMessage =
"Error occurred while sending SOAP request"

When I look at the Wireshark trace, I can see that the remote server is doing the Certificate and Server Key Exchange, and at the same time doing the Certificate Request:

My undertstanding is the “Certificate Request” is how the remote server (the provider) is asking the client (the consumer, me in this case) to provide authentication using a client certificate.

This should trigger my webMethods consumer program to authenticate using the client certificate, but it does not send it:

(in above trace, Certificates Length: 0 means that Web Methods did not send a client certificate.)

After that, the remote server disconnects with an Handshake Failure, error code 40:

It is normal at this point for the remote server to return this handshake failure with error code 40 because WM did not send a client certificate (length was 0).

I used the same WSDL file in SoapUI to create a test request, and the Certificate, Client Key Exchange shows that SoapUI is sending a certificate of length 1509:

If we want to go deeper, I can show the full Certificate Request portion which shows the types of certificate and signature hash algorithms which the remote server will accept:

I looked at the client certificate which the person in charge of the remote server provided me, and it says for Signature Algorithm:

Signature Algorithm: sha256WithRSAEncryption

It works with SoapUI.

Thanks.

Stéphane

[Holger_von_Thomsen]

Hi Stéphane,

thanks for adding the details.

Do you have a global server certificate (used by your own HTTPS-Ports by default unless specified explicitly in the port itself) configured in your IS?
If yes, this certificate should be send upon TLS-Handshake-Request for certificate unless you specify a different one on the soapClient invocation (or the service which calls the consumer connector) for auth/transport section.
Requesting the client certificate for the user will follow on subsequent handling after the TLS transport could be established.
But from your snippets above it looks like the TLS-transport is failing before the server will start sending out the SOAP message.

You need to distinguish between server client certificate and user client certificate.

• server client certificate is the certificate of the consumer server (your IS in this), which serves as a (technical) client to the partners provider server.
• user client certificate is the certificate to authenticate and authorize the user invoking the webservice on the provider server.
  It is possible to use the same certificate for both roles, but it is not neccessary to do so. These can be two different certificates if the partners agree on that.

Regards,
Holger

[StA_A_phane_Desmarais (Stéphane Desmarais)]

Hi Holger,

I have to disagree on one of your statements.

The server client certificate is not the certificate of the consumer server (my IS). It is the certificate of the provider server (the remote server, somewhere on the Internet, not even running IS). I clearly see that certificate in the Wireshark trace (sorry, I cannot share an image for this one, as you would be able to see who they are).

I agree that the user client certificate is the certificate to authenticate and authorize the user invoking the webservice on the provider server. Just to be clear, the webservice is provided by my customer (somewhere on the Internet). They are the ones who want to authenticate and authorize me. My IS is not sending the client certificate that I am asking it to use. It’s not sending any client certificate. The server certificate and client certificate are different. I do not have the server certificate anyway (only installed on my customer remote server). I do have the trusted Certificate Authority public certificates (root and intermediate CA).

You are also right that the TLS-transport is failing before we have a chance to send a SOAP message.

As for global server certificate on my IS, my understanding is that because my IS is a consumer only of this web service, server certificates on my IS would not be used. It’s my customer who have to use a server certificate on their own server offering this webservice.

Stéphane

[Holger_von_Thomsen]

Hi Stéphane,

then there is some severe misconfiguration for the transport layer security part.

As long as the transport handshaking is not successful we wont get any further in the analysis.

Please check the following:

1. Create a new JKS-Truststore with your own CA and the CAs from the partner and use “-trustcacerts” as an additional parameter during import of the ca certs (root and intermediate). Roots might not be neccessary when already included in the cacerts of the JVM (therefore the trustcacerts parameter).
2. Create a Truststore entry in the IS for this JKS file.
3. Create a PKCS12 file with your own server cert (hopefully signed by our own CA) and PK if you need to provide HTTPS your self.
4. Create a Keystore entry in the IS for this PKCS12 file.
5. Define these two new aliases as the global certificates for the IS as Server-Key and Truststore.
6. Restart IS to make the changes active.

During the transport negotiation your IS server should ask the remote server for its server cert and check if it can validate the CA chain behind this certificate.
Once this is done when the IS is initiating the soap invocation the remote server the remote server should then ask for the user certificate. This user certificate needs to be mapped into the consumer invocation under the auth/transport-section as the auth/message section is meant for WS-A based authentication if I remember right.

See the WS Developers Guide as well as the Built-In-Services reference for further informations.

Regards,
Holger