Our new delivery channel, webMethods Package Manager

products versions - {webMethods Integration 10.15 and above}

Introduction

We are pleased to announce that we have a new delivery channel for our software, to make it easier for both developers and operations to install and update features into their integration servers.

wpm is a new command line interface tool (CLI) that allows both developers and operations to download and install webMethods packages into either an Integration Server or Microservices runtime. It will allow you to better leverage our base images via our container registry and customise them to include both Software AG modules such as the JDBC adapter, but also to deploy your own custom packages. The only requirement is that you adopt GitHub repositories to version control your packages.

You will no longer need Installer or Update Manager and instead use a simple Dockerfile to build your custom container containing all of the packages and configuration required to implement your service(s) or API. An update to the base image to include a new feature or fix becomes available ? Simply rebuild via your Dockerfile. Want to deploy a new version of your API or service ? Just re-run the Dockerfile again.

wpm is baked into our 11.0 image and can easily be added to our older images. It it is an integral part of our new unified platform and deployment model, see What is Develop Anywhere, Deploy Anywhere?.

wpm also allows developers to collaborate and share packages more easily via a central package registry, either Software AG’s official registry at https://packages.webmethods.io or your own private registry.

In this article we will explain how to use wpm standalone to update your local development environment and also the steps to create a custom image from our base image to include your required packages.

Packages are fundamental to webMethods integration and represent the principal deployment method for both customer code and Software AG licensed modules. You can find out more about packages from this tech article that I wrote in the past “A word about packages”

Pre-requisites

  • wpm.zip cli tools to allow you generate the required Dockerfiles(s).
  • GitHub account to allow you to pull packages without rate limiting.
  • Empower account that allows you to download Software AG licensed packages.
  • Service Designer so that you can develop your own packages.
  • git - Git client of your choice installed locally on your computer.

Steps to follow

The following section will step through the actions to get you up and running with wpm to bundle both Software AG packages and your own into an Integration Server running locally and also how to create a static image for container deployment. However, it doesn’t replace our documentation, nor will it detail all of the available functionality. Please refer to our documentation centre for more detailed information.

Download Service Designer & wpm.

First download Service Designer locally and unzip it, run the setup script and launch it. From the Server menu click on “Run” to startup your local design time server.

Screenshot 2023-11-22 at 10.42.16

It should only take about a 12 seconds to start, refer to the console to check if it has started correctly

You can now connect your Designer to the server via the package navigator and start building packages.

Download wpm.zip (13.4 MB) and unzip somewhere on your machine. I would recommend that you unzip it into you Service Designer folder. To configure it open the setenv.sh or setenv.bat file in your text editor of choice.

Check that the JRE_HOME is not commented out and that it points to a valid jvm directory. You can use the JRE bundled with your Service Designer if using 10.11 or above (In which case replace the ‘opt/softwareag’ with the path to your wMServiceDesigner folder). If using an older version then you will have to install your own JRE as the webMethods version will be too old to be compatible with wpm.

Refer to the page Guide to Download and Install webMethods Service Designer in our tech community for more help about using this tool.

Get to know wpm.

After configuring your environment check that the wpm cli tool works by typing wpm without any arguments on the command line;

$ wpm
(wpm) code: 3 severity: INFO, message: wpm requires at least 1 argument.
Welcome to wpm - the webMethods Package Manager!

A list of commands:

install - download specified packages into the target installation
remove - remove specified packages from the target installation
update - update specified packages to the latest version
clean - clean up specified repositories or all the repositories in the working dir

A list of arguments:

-d - path to the target IS installation - required for install and update commands if it is not defined in the wpm.yml and not the same installation path as where wpm is installed
-u - username for repo connection - required for install and update commands that require authenticated access with basic authentication if the repo and the repo username are not defined in the wpm.yml
-p - password or passphrase for repo connection - required for install and update commands that require authenticated access with basic or private key authentication if the repo and the repo password or passphrase are not defined in the wpm.yml
-k - private key file path - required for install and update commands that require authenticated access with private key authentication if the repo and its private key are not defined in the wpm.yml
-r - Repo URL - required for install and update commands
-j - jwt token - optional, only required for install and update commands with token based authorization
-sb - scan branches - optional, by default is true
-kr - keep the local repository - optional, by default is false
-ws - location of the package registry server, like https://packages.softwareag.com
-wr - name of the registry on the package registry server, e.g. softwareag

The command you will most often use is install, which will allow you to pull a package from a git repository and move it into the package directory of your local Integration Server, which would normally require that you specify a range of switches when entering the command to let wpm know where your target server is located etc.

To simplify this, let’s introduce you to the wpm.yml file in the root folder of wpm. This file allows you to preconfigure the tool to avoid overloading the command every time you type it in. For instance we can use this file to specify where your local IS/MSR resides, rather than using the -d switch every time you enter the command.

Edit the reference to target_installation to point to the directory containing your packages directory, in the case of an Microservices Runtime (MSR) that will be the IntegrationServer folder. If configuring wpm to work with Integration Server reference the correct instance, normally default.

Installing an official Software AG package

First let’s look at how we can install an official Software AG package from the Software AG registry at https://packages.webmethods.io.

Go to the above site and login using your empower credentials. Once logged in you will see a list of the official packages that we have made available and for which you are licensed.

Click on your profile icon in the top right hand corner and click on settings. From here you can generate a token that you can use with the wpm command to download licensed packages. The label will allow you to reference the token if you want to revoke it later or understand the impact when it expires.

You can now use this token with wpm to update the JDBC Adapter in your service Designer to the latest version including all the latest fixes e.g.

$ wpm update -ws https://packages.webmethods.io -wr softwareag -j <PASTE_TOKEN_HERE> WmJDBCAdapter
(wpm) code: 7 severity: INFO, message: wpm update starting
(wpm) code: 9 severity: INFO, message: Update result: Time spent cloning repository wpm-ssh from git@github.com:wm-packages: 5567
(wpm) code: 9 severity: INFO, message: Update result: Time spent determining version to pull (result - 10.3.0.20): 25
(wpm) code: 9 severity: INFO, message: Update result: Time spent checking out version v10.3.0.20: 800
(wpm) code: 9 severity: INFO, message: Update result: Package does not indicate a minimum required runtime version. Proceeding with the operation.
(wpm) code: 9 severity: INFO, message: Update result: Time spent copying repo to /Applications/SoftwareAG/11.0/IntegrationServer/packages/WmJDBCAdapter: 2819
(wpm) code: 9 severity: INFO, message: Update result: Package WmJDBCAdapter successfully updated (version v10.3.0.20).
(wpm) code: 32 severity: INFO, message: cleaning result: Removed working dir of repository wpm successfully.

You will then need to restart your Integration Server in order for the new package to appear.

Version control with wpm

You can specify what version of a package that you want to download via the wpm command, by default it will always try to download the latest available version. Official Software AG packages are versioned according to released features and fixes.

If you need to install or update to a specified version, simply append to the end of the package name separated by a colon ‘:’ e.g.

$ wpm update -ws https://packages.webmethods.io -wr softwareag -j <PASTE_TOKEN_HERE> WmJDBCAdapter:v10.3.0.20

Create a new package and get it into GitHub

Importantly you can use wpm to install or update your own packages, the first step is to get your package into a GitHub repository so that wpm can reference it. To do rthis you will need access to the packages folder of your local development environment and a GitHub account.

First, in GitHub create a new git repository and name it the same as your package and do not initialize it with a README as the repository will need to be empty. After creation, you should see something like the following

You can follow the instructions other than the “add” command, but first make sure that you are at the root level of your package, before running the commands

$ cd <SAG_HOME>/IntegrationServer/packages/MyGreatPackage
$ git init
$ git add .
$ git commit -S -m "Moving my package to a git repo for the first time!"
$ git branch -M main
$ git remote add origin https://github.com/............
$ git push -u origin main

The -S option on the commit command ensures that the update is cryptographically signed, but will require that you have shared a GPG key/pair with your git account (Github docs on signing commits). You can remove it if you haven’t done this, but I would recommend configuring it especially if you want to share the package with the outside world as it will allow you to verify the version and reassure the developers who want to use your package.

Once done reload the web page and you will see that the commands have been replaced with your package contents.

You will notice that in the above screenshot, the package is documented thanks to the included README.md file. As of webMethods 10.15 a README.md file is created automatically when creating a new package and we would recommend that you regularly update this file to help other developers better use your package. For packages created with older versions of webMethods just create the file manually beforehand.

Pull your own custom package directly via git information

Now that you have your package in a git repo you can also install it into another environment via wpm and you do not need to register it with a package registry, as wpm can reference your git details directly e.g.

$ wpm install -u johnpcarter -p <DEVELOPER_ACCESS_TOKEN> -r https://github.com/<GIT_ID> MyGreatPackage

The user and password are your git id and a developer access token that you have generated. The URL should only include the namespace up to your user id or organisation. The package name is still specified as a separate input and will be appended to the URL to locate the correct git repository.

As before if you want to download a specific version then you should either create a new branch or create a tagged release. In either case you can specify the branch or tag by appending it to the package name as before e.g.

$ wpm install -u johnpcarter -p <DEVELOPER_ACCESS_TOKEN> -r https://github.com/<GIT_ID> MyGreatPackage:main

In a future article I will discuss how you can set up your own private package registry to avoid having to reference git specific details to your install you packages.

Using wpm to build an image

One of the main drivers to introduce wpm is to ensure that you can easily leverage our pre-built images that you can find at https://containers.webmethods.io

Our Microservices Runtime container is stripped back to be the bare-bones of our integration runtime. You will need to add both the optional features provided by us as well as your own packages.

Bake wpm into the microservices runtime image

Let’s first add wpm to the base image so that we can then use it from within a Dockerfile to add packages. Copy the following text into a file named Dockerfile.

FROM sagcr.azurecr.io/webmethods-microservicesruntime:10.15

ADD --chown=sagadmin:sagadmin wpm /opt/softwareag/wpm
ENV PATH=/opt/softwareag/wpm/bin:$PATH

Then build your derived image e.g.

$ docker build -t webmethods-microservicesruntime-wpm:10.15 .

Bake your packages in via wpm

Now you can create a Dockerfile that references the above image and use wpm to install packages e.g.

FROM sagcr.azurecr.io/webmethods-microservicesruntime-wpm:10.15

WORKDIR /opt/softwareag/wpm

RUN wpm install -ws https://packages.webmethods.io -wr softwareag -j <WPM_ACCESS_TOKEN> WmJDBCAdapter
RUN wpm install -u johndoe -p <GIT_TOKEN> -r https://github.com/johndoe MyGreatPackage

WORKDIR /

Here we see an example to add both the Software AG database adapter and a custom package to the base image.

Next steps

Useful links | Relevant resources

Guidelines for developing packages - webMethods packages from a developers perspective
Download the latest version of Service Designer - here
Check out our available packages - Software AG package registry
Learn more about version control with git - Git tutorial

11 Likes

Very cool John!

1 Like

Cool, looks very promising! Just a few question:

  1. “wpm is baked into our 11.0 image and can easily be added to our older images”. Does wpm support IS/MSR 10.15?
  2. What is the official supported way to download wpm.zip? I didn’t get a link to official page
  3. How to use mutliple custom packages at once? This is a quite usual use case for any customer project (or is wpm just for presales?)
  4. How to use different wpm-files to configure generic and project specific settings?
  5. How to secure user tokens? Usually this would refer to an external file stored locally or mapped using Secrets in K8S.
  6. Is there any feature to support package dependency checking? You know any package manager offers dependency management 


BR
Marko

1 Like

Hi @Marko_Gorg ,
#1 - yes, any supported version is supported. All versions available via https://containers.softwareag.com/
#2 - it’s included in the container images. More details will be available in the official 11.0 manuals
#3 - You can script it or use Scaffolding structures to orchestrate for details see Patented Super iPaaS: Revolutionizing Enterprise Integration - Software AG | Software AG
#4 - see manuals once available.
#5 - it depends on the use case, e.g. in “Develop Anywhere - Deploy Anywhere” of “Super iPaaS” the tokens are maintained in the HUB and shared with the SPOKE when needed to deploy updates.
#6 - this is part of the “Super iPaaS” offering as well. Overview of Develop Anywhere Deploy Anywhere - webMethods.io Integration

1 Like

Hi Marko,
I’ll add to the response made by @Holm_Steinland3,

in response to (3) you can run the wpm command as many times as you want and automate using a script. For building a container you would just add multiple lines to the Dockerfile, one for each package. The advantage being that each package will have it’s own image layer too, if you want to optimise then you could add a script to group the packages.

(4) Configuration variables are the recommended way to configure your packages, and have been available for a long time. We have added more and more with each passing version, albeit we do have many that are not completely documented and we will need to look into that.

(5) The tokens for downloading SAG packages can be generated after logging into https://packages.softwareag.com → Settings → Access tokens → create

(6) wpm is used by DADA under the hood. However, we wanted to ensure that the underlying technology could be used in other use cases by all our customers and that’s why we created wpm as a separate tool. So, short answer wpm is available to ALL our customers regardless of contract, or deployment.
regards,
John.

2 Likes

@John_Carter4 I haven’t checked yet: will the repository url also take a path to a package (e.g. if you have multiple packages inside a single repository)?

@John_Carter4 the example described here for 10.15 does not work for me. The wpm comes with a newer Java version as the MSRs base image Java version. Is there a compatible version of wpm for 10.15?

0.339 Error: LinkageError occurred while loading main class com.softwareag.is.wpm.cli.Main
0.339 java.lang.UnsupportedClassVersionError: com/softwareag/is/wpm/cli/Main has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0

Meanwhile managed to run wpm on 10.15 with workaround (Multistage Dockerfile):

FROM openjdk:17-alpine3.14 as jdk
FROM webmethods-microservicesruntime-wpm:10.15
COPY --from=jdk /opt/openjdk-17/ /opt/openjdk-17/
WORKDIR /opt/softwareag/wpm
ENV JAVA_HOME=/opt/openjdk-17/
ENV JRE_HOME=/opt/openjdk-17/
ENV JDK_HOME=/opt/openjdk-17/
ENV PATH=/opt/softwareag/wpm/bin:/opt/openjdk-17/bin:/opt/softwareag:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
RUN wpm install -u wpmtests -p <token> -r <repo-url> <package>
ENV PATH=/opt/softwareag/wpm/bin:/opt/softwareag/jvm/jvm/bin:/opt/softwareag:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV JAVA_HOME=/opt/softwareag/jvm/jvm/
ENV JRE_HOME=/opt/softwareag/jvm/jvm/
ENV JDK_HOME=/opt/softwareag/jvm/jvm/
WORKDIR /

Unfortunately wpm does not exit with an error code so you will have to add: --progress=plain to see if something is failing during the installation.

Multiple packages per repository do not seem to work according to my tests - hopefully this will be added soon.

2 Likes

Hi Marc,
I updated the article to include adding the wpm binaries included in this article, which should now be binary compatible with 10.15. Currently there is a bug with wpm in that it does not exit correctly with errors which will be fixed early in the new year.

Thanks for your feedback.
Happy holidays.

1 Like

Hi,
this looks exciting and a big improvement on the way we do things now. However, we use GitLab as our repository and your documentation states that it only works with GitHub. Are you going to be adding more repositories in the future?

2 Likes

Hi,
in my tests on Windows/Rancher Desktop the following line in Dockerfile:
RUN wpm install 

is not working. But it is working like this:
RUN [ “bash”, “-c”, "/opt/softwareag/wpm/bin/wpm.sh install 
 " ]

Hi,

We have a requirement to setup CICD process in customer environment in on-prem. Currently we are giving demo to customer with below process as a demo given by SAG.

[Demo: Continuous Integration with webMethods - YouTube]

(https://www.youtube.com/watch?v=YK5o9rQbgpU&list=LL&index=4&t=1134s)

I see that using WPM we can achieve "Develop anywhere, Deploy anywhere ".

Now many questions are:

  1. Using the below command can we directly install custom package in target server from bitbucket, by replacing GitHub URL with bitbucket?

$ wpm install -u johnpcarter -p <DEVELOPER_ACCESS_TOKEN> -r https://github.com/<GIT_ID> MyGreatPackage:main

  1. Do I need to install WPM and setup all the configurations in each and every target WM Server?

  2. How to deploy multiple/single packages in multiple target servers?

  3. Is WPM officially supported by SAG now?

please correct me if anything I am missing.

If anyone already implemented the steps to pull the code from git/bitbucket and deploy to the target server. If anyone done POC or implemented in any customer landscape, can you please share the steps that would be helpful.

1 Like

Can we use any git repo instead of GitHub? Most companies use something else as git repo. My company uses gitlab for instance. I don’t see anything hardwired to GitHub. In theory it should work with any repo but I was wondering if the package itself requires it somehow.

From discussions we’ve had with @John_Carter4 , I can tell that support of other Git remotes is in the product backlog. It’s just that Github was selected for the v1.

wpm will be shipped in the IS & MSR images starting with v11. For earlier versions, you can get it using the SAG installer.

You would typically use wpm to install packages in your MSR based development environment (like you would do with npm and node.js), and to build your docker images (which embed your integration packages.) In that case, you build your image once and it’s deployed “as is” in all the target environments.

Can you use wpm in traditional webMethods IS deployments? I haven’t tried it, but it should work. You would need to do one deployment for each target runtime. It wouldn’t fully replace the Deployer, as you still need to deal with the configuration aspects (which are totally out of wpm scope.)

So, if you’re still using the traditional IS, it’s probably better to continue using the Deployer for your CICD.
If you’re using the MSR (or possibly an IS with a MSR license) in containerized environments, then wpm should be considered.

1 Like

Does it require an MSR license? My POC environment uses IS container from containers.softwareag.com. I really would like to use WmPM with my new environment but we don’t have an MSR license.

You’re not requested to provide a license for wpm, technically it works with IS or MSR.
I was just mentioning “ideal” patterns for wpm usage.

Thanks for the write up @Stephane.TAILLAND_emp. Just to add that you can use wpm with any variant of webMethods be it Integration Server, Microservices Runtime, or even the Service Designer that you download from tech community.

You can use it deploy any self contained package, but as Stephane pointed out BPMS is not designed with well encapsulated packages in mind and hence we would recommend that you continue using Deployer in the meantime.

Theoretically you can already use wpm with any git compliant provider as it uses a generic git library. However, we haven’t tested so you would need to proceed at your own discretion. However, if you I would love to hear back about whether it worked or if not what errors you faced. That would help me accelerate getting the support for other providers into the tool :wink:

thanks
John.

1 Like

I will test this in my POC environment and update this post. One last question, I am planing to download packages in to a shared persistent storage using WmPM and share the packages directory across the cluster because we have a lot of servers. Is there any technical concerns that I need to be aware of? I will have a dedicated package updater container, the rest of the cluster won’t have write access to packages but only read permission. Once the packages folder is updated, reader pods will be restarted.

Please let me know if this approach doesn’t make sense or anything. I haven’t reached to the point where I can test CI/CD pipelines, so all of these are just thoughts right now.

Is there a way to update JDBC adapter connection strings and credentials?

You can override existing connection strings via configuration variables e.g.

artConnection.EdgeJcHelloWorld.edge.jchelloworld.WmJDBCAdapter.com_wm_adapter_wmjdbc_JDBCAdapter.connections.LOGGER.connectionSettings.password={AES}Y5IgMqjfvkgbg7p5VUZztw==

or you could externalise it as an environment variables that could be set when spinning up the container

artConnection.EdgeJcHelloWorld.edge.jchelloworld.WmJDBCAdapter.com_wm_adapter_wmjdbc_JDBCAdapter.connections.LOGGER.connectionSettings.password=$env{password}

or more secure, use a k8s secret and configure a backend vault

artConnection.EdgeJcHelloWorld.edge.jchelloworld.WmJDBCAdapter.com_wm_adapter_wmjdbc_JDBCAdapter.connections.LOGGER.connectionSettings.password=$secret{k8spassword}

Afaik, we can’t use configuration variables because they require an MSR license. Do these methods require MSR license?