How to get on-prem IS to upload a file into Azure 'Blob' storage

Hi – I need help having Integration Server 10.1 upload a simple CSV data file to an Azure ‘Blob’ storage container.

That is, I need to make IS upload file.csv to https://<storage-account-name>.blob.core.windows.net/<container-name>. The IS version is 10.1 on-prem (extended support).

I thought it would be easy, but Microsoft offers these two options to upload files to Azure blob storage:

  1. A horribly complicated REST API
    It comes complete with requirements for custom headers with hashed and digitally signed content – see here for a simple script-based query. This seems unworkable.
  2. The uploading client (IS 10.1) uses a Microsoft-authored Java package to upload files
    For this, Microsoft asks we add these entries to pom.xml. I know pom.xml is par for the course for Java ninjas. However, to a webMethods dev like me it’s an Alice in Wonderland-level rabbit hole.

I favor option #2. So I tried something like this:

  1. Identify and download a suitable JAR (is the one below suitable - I have no idea; I just went clicking around?).
    https://repo1.maven.org/maven2/com/azure/azure-storage-blob/12.14.2/azure-storage-blob-12.14.2.jar
  2. Add it to a package’s /code/jars folder (I also added it to /code/classes)
  3. In the new package, add Java code like the following
    (adapted from this Microsoft sample code)
import com.azure.storage.blob.*;
/* Create a new BlobServiceClient using account key */
BlobServiceClient blobServiceClient = new BlobServiceClientBuilder()
    .endpoint("https://<storage-account-name>.blob.core.windows.net")
    .credential (new StorageSharedKeyCredential("<accountName>", "<accountKey>") )
    .buildClient();

/* Create a new container client */
try {
    containerClient = blobServiceClient.createBlobContainer("<container-name>");
} catch (BlobStorageException ex) {
    // The container may already exist, so don't throw an error
    if (!ex.getErrorCode().equals(BlobErrorCode.CONTAINER_ALREADY_EXISTS)) {
        throw ex;
    }
}

/* Upload the file to the container */
BlobClient blobClient = containerClient.getBlobClient("file.csv");
blobClient.uploadFromFile("file.csv");

Unfortunately, the sample Java code fails to compile – I get this error:

Has anyone got IS to upload to Azure ‘Blob’ storage before?

Any thoughts on the error? Does my approach above sound reasonable?

Other options:

  • I heard of some ‘Azure adapter’ for IS, but my site’s not licensed for it (and it may not be available for 10.1)
  • Microsoft has introduced an SFTP transfer option for Azure storage. That would be ideal, but it’s beta and only new subscriptions can use it (so my company’s containers cannot enable SFTP)

Hi Sonam ,

You can use Azure Storage APIs to upload data to Azure Storage from On-Premise .

For Azure Storage API , you need to handle the Authentication properly . There are multiple approach for authentication . For example in our scenario , we are using Authorize with Azure Active Directory via OAuth 2.0 tokens .

We have a registered App on Azure AD associated with a user , which we use to get token . The App has been provided required access on Azure storage to do read/write operations .

If you do not want to use Azure Storage APIs , you can map Azure Storage as SMB/CIFS share and then write files to it .

To achieve above OAuth 2.0 based authentication , we have used flow services with pub.client:http calls .

We are using both the above approaches to read/write files to Azure Storage and working fine for us .

Let me know if further details required .

1 Like

Progress (well, got it compiling at least)!

Initially, I’d downloaded azure-storage-blob-12.14.2.jar. Later, I found its webpage lists four additional JAR package as dependencies. I downloaded these into the /code/jars folder and reloaded the package:

azure-core-1.22.0.jar 
azure-core-http-netty-1.11.2.jar 
azure-storage-common-12.14.1.jar  
azure-storage-internal-avro-12.1.2.jar

…then kept fixing compile errors until I got this code to compile:

import com.azure.storage.blob.models.*;
import com.azure.storage.*;
import com.azure.storage.common.*;
import com.azure.core.exception.*;

...

		/* Create a new BlobServiceClient with a Shared Key */
		BlobServiceClient blobServiceClient = new BlobServiceClientBuilder()
		    .endpoint("https://<storage-account-name>.blob.core.windows.net")
		    .credential (new StorageSharedKeyCredential("<accountName>", "<accountKey>") )
		    .buildClient();    
		 
		BlobContainerClient containerClient =null;  
		/* Create a new container client */
		try {
		    containerClient = blobServiceClient.createBlobContainer("<container-name>");
		} catch (BlobStorageException ex) {
		    // The container may already exist, so don't throw an error
		    if (!ex.getErrorCode().equals(BlobErrorCode.CONTAINER_ALREADY_EXISTS)) {
		        throw ex;
		    }
		}
		
		/* Upload the file to the container */
		BlobClient blobClient = containerClient.getBlobClient("file.csv");
		blobClient.uploadFromFile("file.csv");

Next week I plan to progress further. One essential change I need is to make blobClient accept a stream as input.

Thanks Rakesh – you seem to have this well architected in your infrastructure. My company’s IS servers run on Linux. Does your IS run on Windows? Would that make a difference in approach?

I didn’t see any other option except the REST and Java APIs. When I checked out the REST API, it looked terribly convoluted (e.g., coding up the necessary content signing, etc). That’s why I went down the Java client route. I made some headway (see above) – I intend to find out next week if it actually works.

An SMB/CIFS share sounds like a good third option. Even if an SMB/CIFS share is not directly mounted on Linux, it can be exposed via FTP/SFTP.

Somehow this option never came up in discussions on my side. Looking at the link you share, this refers to ‘Azure Files’. This may be different from ‘Azure Blob storage’ , which is what I am dealing with.

Can Azure Blob storage be exposed over SMB/CIFS? This may not be possible, based on what I read in this link.

Thanks again for your input - it gives me a few things to think about…

Yes , our servers are on Linux Servers as well . Be it Windows or Linux it will not have any difference in approach .
In my view it’s better to use API , also via Java code as well you are going to do the same thing for e.g. passing same set of parameters as required in the API call.

If you go via the App registration approach on Azure AD , you can use it further for any other authentication/authorization request for Any Azure/Microsoft related services .

See below link for registering an app on Azure and getting OAuth authentication token .

The above link has section for Registering App on Azure AD
or

And use the Client Credentials based OAuth grant which is generally preferred to use for any Server to Server authentication .

Yes , via NFS (Network file system) , you can do it for Azure Blob storage as well .

[General Comment] CIFS shares can be directly mounted on Linux servers , we are using it since long for different integration requirements .

This approach should work as well , We try to avoid using Java service unless unavoidable and mostly depend on Flow services , APIs in our landscape .

Also , we have different Microsoft/Azure services in our environment so, we have built Azure OAuth common framework which we use across for authentication/authorization .

Thanks for your guidance Rakesh. Your approach, with its approved app and AD authentication, seem well integrated.

What concerns me about using the REST API is the complexity in constructing the special signature-based Authorization header that MS demands - here someone else built one for a simple query that just list objects in storage (no upload). This is a for simple GET - for a PUT, we’d hash and sign the payload too.

# Build the signature string
canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
canonicalized_resource="/${storage_account}/${container_name}"

string_to_sign="${request_method}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}\ncomp:list\nrestype:container"

# Decode the Base64 encoded access key, convert to Hex.
decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)"

# Create the HMAC signature for the Authorization header
signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary |  base64 -w0)

authorization_header="Authorization: $authorization $storage_account:$signature"
  • I am curious: did you do something equivalent to the above steps in your custom Azure app? Are equivalent HMAC cryptography calls available in WmPublic?

  • Yes, CIFS and SMB should be mountable on Linux. But I doubt Azure Blob storage (which is an object store) exposes its content via CIFS/SMB (block-based storage) - I couldn’t find any documentation for this. I’d be happy to be wrong though.

We didn’t use Shared Access Signature based authorization , but it’s possible .

App Registration step is a which is used as a mechanism to use the authentication and authorization provided by Azure AD Identity platform , other use cases are also there but for this scenario let’s concentrate on Authentication & Authorization.

So, we created a representational App on Azure AD for our webMethods servers , with which we can leverage OAuth features and connect to different Microsoft and Azure services seamlessly .

I have used OAuth 2.0 based authentication with client credentials grant , It’s a different authentication than Shared Access signature and it’s a very simple method. You need to call the Azure AD Oauth 2.0 token endpoint to get the authorization token and then use the token for Azure Storage API Call .

Step by Step which we have followed .

  1. Register Application on Azure AD
  2. Provide the App access on the Azure Storage account , according to the operations you will be performing
  3. Call OAuth 2.0 token endpoint to get the OAuth 2.0 token
  4. Use OAuth token to call the Azure Storage API and do the required operation .

Token API Call .

Once you get the token , you can use it for Azure Storage API Call
Azure Storage API Call [Token from above API call is passed as Bearer Authorization token]





Azure Blob supports NFS , which you can use to mount on Linux servers . link below

Thank you @Rakesh_Kumar3 - This is an excellent illustration and you have been very kind in documenting it so thoroughly.

Summarising my understanding, your technique requires the organisation’s AD admins initially register the integration platform with its Azure AD identity provider. Then at runtime, IS first obtains an OAuth 2.0 token (using its registered identity), and uses the token as ‘Bearer’ Authorization (which simply passes the token) to make blob storage calls.

Thanks also for the NFS3 link!

I plan to discuss your approach with my AD admins next week. I also plan to give my Java code a push to see if it works, then update back here.

Thanks again!

Ok sure , thanks .

keep us posted , how it goes .

Me/Someone else from the community will be happy to help.

Update: while Rakesh’s answer seems like the correct way to do things, I’m proceeding with the Java service I wrote for now. It compiles, but won’t run because the five azure-*.jar JAR files I downloaded into <package>/code/jars are not enough. Here, the code has indirect dependencies on packages like org.reactivestreams.*.

Could not run 'deliverDataToAzureBlobStorageService'
java.lang.reflect.InvocationTargetException: org/reactivestreams/Publish

So it’s not enough for me to go poking Maven Central and manually noting dependencies is not enough - I need to download and use Maven to consolidate all JARs necessary for Azure blob storage operations.

(I’ve made this a new forum question)

I’ve managed to get all required JARs (39 of them!) into the package code/jars folder. But the Java service now fails with the dependency error below. Any ideas how to resolve?

Basically, I need to compel my Java code (and classes it calls) to use the newly packaged Jackson version 2.x JARs (instead of Jackson version 1.x JARs packaged by IS).

Could not run 'deliverDataToAzureBlobStorageService'
java.lang.reflect.InvocationTargetException: Package versions: jackson-annotations=2.10.1, jackson-core=2.10.1, jackson-databind=2.10.1, jackson-dataformat-xml=unknown, jackson-datatype-jsr310=unknown, azure-core=1.22.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot

Below, I outlined the steps taken to package up the JARs.



1. #Install Maven and check version
#Adapted from https://tecadmin.net/install-apache-maven-on-fedora/

wget https://dlcdn.apache.org/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz
sudo tar xzf apache-maven-3.8.4-bin.tar.gz -C /opt
cd /opt && sudo ln -s apache-maven-3.8.4 maven
sudo vi /etc/profile.d/maven.sh
Add this content
--------------------------
export M2_HOME=/opt/maven
export PATH=${M2_HOME}/bin:${PATH}
--------------------------------
source /etc/profile.d/maven.sh
mvn -version


2. # Go to home folder, initialize Maven and have it generate a dummy pom.xml
#Adapted from https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html

cd 
mvn archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=false


3. #Configure Maven to download all required JARs for Azure Blob storage locally
#Adapted from https://technology.amis.nl/software-development/java/download-all-directly-and-indirectly-required-jar-files-using-maven-install-dependencycopy-dependencies/

cd my-app
vi pom.xml
Add the following to the dependencies section:

# Dependency adapted from: https://docs.microsoft.com/en-us/java/api/overview/azure/storage?view=azure-java-stable)
# Only the first dependency is taken. The current version is used 
#If not, you can expect a "Could not resolve dependencies"  error 
-----------------------
<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-storage-blob</artifactId>
    <version>12.4.0</version>
</dependency>
----------------------------
mvn install dependency:copy-dependencies


4. #Maven now downloads 39 JAR files to the 'my-app/target/dependency/' folder
# Move these into the IS <package>/code/jars folder.
#Reload your package
cd /home/<user>/my-app/target/dependency/

5. #At this point, running the service gets past the missing JAR errors, but returns this new dependency error below.
-----------------------------
java.lang.reflect.InvocationTargetException: Package versions: jackson-annotations=2.10.1, jackson-core=2.10.1, jackson-databind=2.10.1, jackson-dataformat-xml=unknown, jackson-datatype-jsr310=unknown, azure-core=1.22.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
-----------------------------

#This is probably because IS 10.1 shows the 1.x version of the Jackson JAR loaded on it's 'About' page, while the JAR version used by Azure is 2.x:
# /opt/SoftwareAG101/IntegrationServer/lib/jars/jackson-coreutils-1.8.jar