Develop anywhere - A practical guide to using packages with webMethods.io

[John_Carter4 (John Carter)]

products versions - {webMethods Integration 11.0 or above}

Introduction

Develop Anywhere, Deploy Anywhere introduces a new unified integration platform for webMethods and builds on the strengths of our existing products.

In this article, I will show how you can take a webMethods package developed with Service Designer and incorporate it into an Integration project for use and deployment anywhere!

Pre-requisite

You will need the following

• webMethods.io tenant that has these new features enabled
• Service Designer installed locally on your computer if you want to version control your own package.
• GitHub account to allow you to pull packages without rate limiting.

NOTE: You do not need to use Service Designer 11.0 to build packages that can be deployed in webMethods 11.0 and can reuse older packages that you may have developed in the past. But be careful if you have been using undocumented or deprecated functionality, as they might not be available in the most recent release.

Steps to follow

In this section, I will list the actions without necessarily going into lots of detail, for this refer to our online documentation center

Create an integration project

You create a project in webMethods.io Integration and it allows you to package a working integration or API. Everything you need to make your integration work goes into the project, including as-of-now webMethods packages.

So login to your webMethods.io tenant and select Integration. You will then see your projects page, before we begin verify that you see the option “Integration Runtimes” option in the masthead. If you don’t, then I’m afraid you will need to reach out to your Customer Success Manager (CSM) at Software AG or if not raise a support ticket, before you can continue.

We will now create a fresh project to allow us to experiment, just click on the “+” button and name your project.

You will notice a new option that allows you to use your own source control (GitHub only at the moment). This will become really important if you want to be able to build your own edge images for production at a later date. If you have a GitHub account and want to try this out then continue with the next section, otherwise skip to the section on “Create a deploy anywhere flow service”. In which case keep the “Default Server” choice and click the Create button.

Setting up a GitHub account for your project

You need to complete three steps in order to use your own source control;

• Create a GitHub repository - Login to GitHub.com and create a new public or private repository for your project. It must have the same name as your project and also end with the word ‘Project’. Do not initialize the repo, keep it empty.
• Get a developer access token - In GitHub click your account avatar (top right corner) and click on settings. From settings, click on Developer settings (bottom left) and then finally personal access tokens. From this screen, you can generate a new token. You need only select permissions for ‘repo’ access.
• Give access to your repository in webMethods.io - You now need to add an account for your developer access token so that webMethods.io can push updates to your project. Click on the validate button to verify the pasted token. If it’s valid then the ‘add’ button will become active and you will be able to complete the step.

After adding your git account you will be returned to the “New Project” modal and allowed to set the branch name for your initial version. Click Create to create your project, refresh your repo page and you should see the project assets newly created.

The git repository must be empty or it will fail, and the repository cannot be shared we require a unique repo for each project.

Create a deploy anywhere flow service

Firstly we need to explain that we now have two variants to our flow services;

• Flow Service - Pre-existing flow service that you know and love.
• Deploy Anywhere Flow Service - Can be deployed to a runtime of your choice anywhere.

Only deploying anywhere flow services can leverage packages as they run in an isolated spoke or in your own environment and hence the risk presented by 3rd party packages to our infrastructure is mitigated.

This is a good time to remind you that you should also vet any 3rd party packages that you did not develop before introducing them into your integrations.

So from your projects page, click on flow services, then the ‘+’ button, select “deploy anywhere flow service” and add.

The deploy anywhere flow editor is the same as the flow editor and I won’t try to explain the ins and outs of writing flow services here as it is outside the scope of this article. However, I would like to bring to your attention the addition of the runtime chooser to the masthead. This chooser allows you to test/debug your deploy anywhere flow service on a runtime of your choice, by default it preselects the cloud runtime that has been provisioned in your webMethods.io tenant.

It is important to note that this choice is not permanent and the chooser exists only as a convenience. Where this service gets executed is entirely devolved to the orchestration layer, which is implemented using workflows.

So let’s get your service to simply log a hello message to the selected runtime log;

• Name your service - Click on the service name in the top left corner and type ‘helloWorld’
• Add an input - Click in the i/o button add an new input ‘name’
• Add a log step - Click on the first step and start typing ‘log’ and select ‘debugLog’ from the suggestion.
• Set the input - Click on the ‘View/Edit pipeline’ button and double-click on the message input. Toggle the ‘Perform variable substitution’ to true and start typing ‘Hello %’, complete by selecting ‘name’ from the suggestions.

Now click on the run button next to the ‘runtime’ chooser, enter a name when prompted for input and hey presto you have just executed your first deploy anywhere flow service in a dedicated cloud runtime.

Not so excited? Well, we haven’t done anything very different from a typical flow service as of yet. What we need is to be able to see how we can use a custom package and then deploy anywhere.

Adding a 3rd party package

You can add virtually any package that you have developed in the past or present. This is great news as it allows you to share and reuse common logic and services between projects and developers.

The only technical constraint is that the version of your package cannot be higher than the version of the runtime into which you want to install it. Obviously, that will be impossible as of now (when this article was published) because our cloud platform is already at version 11.0 and you will be using either Service Designer 10.15 or our preview version of 11.0, and yes 10.15 is fine by the way. However, there are caveats if you want a project that is reliable and efficient, so please consider the following;

• Small is good - Avoid huge monolithic packages as this will result in fat slow containers. Testing will also be more complicated and change management difficult to judge.
• Deprecation - Generally our platform maintains good backward compatibility, but that doesn’t mean that we will keep deprecated services or assets in the product forever, so be aware that really old packages that haven’t been maintained might run into trouble.
• Stateless - Runtimes are stateless so services that write to the file system to record state risk failing, caches will not be shared across replicas. Packages will have to be refactored if the local state is assumed.
• autonomous - Ensure that the package has everything it needs to work self-contained in the package. For instance, if it is dependent on a configuration file or Java library, ensure that they are both in the package under the config and code/jars folders respectively. resources is another folder that you can make sure of to localize any other type of file.

With that in mind, let’s add a package to our project and see how we can then invoke services in the package. Your packages must be version-controlled in a dedicated GitHub repository. We enforce that to ensure that proper version control and collaboration are respected. To start with we will use one of my packages that are already available via a GitHub repository and after that, I have included a section walking you through the steps to get your own package into GitHub.

OpenWeather client package

The package that we will use here was kindly contributed by @Radhika_Warrier_Mooriyath and @Rahul_Singh11, and it shows how we can leverage the Open Weather service to provide a set of simple services to find out what the weather is like. This is especially useful if you have perhaps created packages for calling back-end for on-premise applications in the past, and now you can bring them into your hybrid integrations without having to rewrite any of your code.

The package leverages our support for importing OpenAPI 3.0 definitions directly into Service Designer that then generates the services for you. For more information refer to our document on webMethods Service Development Help 10.15, chapter 23, page 471.

To add this package from your project page click on the packages tab and you should see something like below

Here we see that we already have a package listed, namely the project package that comprises our deploy anywhere flow services. We will now add our package by clicking on the “Add package” button, then select the GIT option and paste the following string

https://github.com/rahuldemocenter/OpenWeatherClient.git

In the source control chooser, select the same git account that you created earlier. Although the repository is public you still need to provide credentials as this will avoid restrictive download limits depending on your account type. This will be important if you want to deploy the project to many different runtimes later.

It will then ask you to select the branch or tag that you want to clone. This is why we insist on version control as this will allow you to switch different implementations in a controlled manner without sudden inexplicable errors that cannot be controlled. In this case, we only have the main branch to choose from.

Package conflicts

When selecting your version, you may see a hint such as the one below. This is because all of your packages including the project package have to be saved to a design runtime, which currently is the Cloud Runtime provisioned in your tenant. You cannot have different versions of the same package in the same runtime and hence if someone has already imported the same package in a different project you will get this kind of warning.

In this case, it will not replace the existing package with an inferior version and only upload the package if newer than the current version. In which case you may want to inform the owner of the other project as they may need to check that their project is still compatible.

It is good practice to ensure that your packages maintain backward compatibility. Don’t remove or rename services, ensure existing signature remains compatible, and new inputs should always be optional or have an implicit default value. Remember to use deprecation if you need to replace a service where the change cannot be made transparent.

Changing the version or getting updates

If you want to replace the package with a new version, or if working on a main branch then you can pull the latest changes from the package details page as below.

Here you can also see what other projects are referencing the package, view the package contents via ‘assets’, and pull the latest changes or even change the current version.

signup with OpenWeather.org

Before we can continue you will need to register with the OpenWeather org to get an app id so that you are allowed to execute the service. Don’t worry there is a free plan that you can try out after you register here, it’s limited to 60 calls per minute but should suffice for our demo purposes.

You will need to copy the default API key that is automatically generated, either from the email that you receive after registering or via the web portal under your profile, My API keys.

Invoke a service in the package

Okay, now have successfully added our package to the project let’s go back to our deploy anywhere flow service helloWorld, and see how we can invoke a service in our package.

Deploy anywhere flow services introduce a new category when adding a step called “package services”. If you select this, you will then see that it first presents a list of all your imported packages, select our OpenWeather package, and then start typing “Curr” into the service selector and then pick CurrentWeatherData.

From the “edit pipeline” view enter your city of choice into the input and the app id that you obtained earlier. The app id is really important, otherwise you will get an “Authorization required” exception when running the service.

Save and run the service and you should see something like the below after the execution:

Oh dear, it’s raining in Paris!!

Congratulations you have successfully imported and used a webMethods package in your integration project. If you chose to run your service on your own runtime, then the package would automatically be included as part of the deployment.

Extra brownie points or how to version control your own packages

This section is optional, but I thought I would add it for those who would like to know the steps to get their own packages into a Git repository. You can skip this if you wish.

To get your own package into a Git repository you will need access to the packages folder of your local development environment and a GitHub account.

First, in GitHub create a new git repository and name it the same as your package and do not initialize it with a README as the repository will need to be empty. After creation, you should see something like the following

You can follow the instructions other than the “add” command, but first make sure that you are at the root level of your package, before running the commands

$ cd <SAG_HOME>/IntegrationServer/packages/MyGreatPackage
$ git init
$ git add .
$ git commit -S -m "Moving my package to a git repo for the first time!"
$ git branch -M main
$ git remote add origin https://github.com/............
$ git push -u origin main

The -S option on the commit command ensures that the update is cryptographically signed, but will require that you have shared a GPG key/pair with your git account (Github docs on signing commits). You can remove it if you haven’t done this, but I would recommend configuring it especially if you want to share the package with the outside world as it will allow you to verify the version and reassure the developers who want to use your package.

Once done reload the web page and you will see that the commands have been replaced with your package contents.

You will notice that in the above screenshot, the package is documented thanks to the included README.md file. As of webMethods 10.15 a README.md file is created automatically when creating a new package and we would recommend that you regularly update this file to help other developers better use your package. For packages created with older versions of webMethods just create the file manually beforehand.

Make your package deployable

As already mentioned you need to ensure that your package can be easily deployed and without requiring additional steps as that will break the one-click deployment offered by Deploy Anywhere. Keep in mind the following practices.

• Keep packages small and ensure a README.md file is included.
• Implement versioning and ensure that the version and build is updated in the manifest.v3 file.
• Explicitly define dependencies on other packages via the dependencies section in the manifest.v3 file.
• Make sure that your package is autonomous and has no external dependencies.
• Avoid manual steps by automating pre/post actions via startup & shutdown services (keep them short & sweet, otherwise you can adversely affect the startup and shutdown time of your container)

Next steps

• Register your own runtimes in the Integration Runtimes control plane.
• Start orchestrating your services across different runtimes via workflows.
• Share your package with other developers via the Software AG package registry at https://packages.softwareag.com

Useful links | Relevant resources

Overview - What is Develop Anywhere, Deploy Anywhere?
Sign up for a tenant - Sign up - Software AG Cloud
Online documentation - docs.webmethods.io
Check out are YouTube videos on super IPaaS - Youtube - Super IPaaS
Download the latest version of Service Designer here
Check out available packages - Software AG package registry

[reamon]

@John_Carter4

Ensure that the package has everything it needs to work self-contained in the package. For instance, if it is dependent on a configuration file or Java library, ensure that they are both in the package under the config

This is a newbie question, but is there a facility for keeping configuration info outside of the package (edit) in webmethods.io? Our config files often have environment-specific settings within so we do not store them in the package so that repeat deployments of a package don’t overwrite any config changes made in a given environment. (This will likely lead into discussions about repositories, version control and deployment scripts – which we can explore certainly but I’m curious as to options for config.)

[Holger_von_Thomsen]

Hi,

you can create a custom sub directory under the config directory to store your configuration files.
The files can then be read with pub.file:getFile from there: “config//filename”.

This has the advantage that these sub directories & files will be automatically backed up when the IS startup is completed:
“Config directory saved”-Message in the server.log

Regards,
Holger

[reamon]

@Holger_von_Thomsen

you can create a custom sub directory under the config directory

That can be done in webmethods.io?

We used to put configs in and under the IS instance config directory but stopped doing that long ago. The backups have never once come in handy IME–always lose what the previous setting was because invariably the instance gets restarted after someone has made a change. YMMV of course.

[Edit] We place config files (mostly xml with a couple of properties format files) in a directory we defined that is a peer to the IS instance config directory. Putting our files in config or in subdirs sometimes caused confusion with those less familiar with the setup where they did not know which files were defined by SAG/wM and which were “ours.” Our naming convention is the config file shares the same base name as the package that uses it. If multiple packages use it (relatively rare), it has an applicable name that helps identify what it is for.

The key concept is separating the config from the code so that code deployments do not disturb configs in undesirable ways. As with most things, multiple ways to do something.

[jahntech.cj (Christoph Jahn)]

I have spent a little bit of time with this topic, and in the past having a sub-directory within the IS’ ./config directory was my preferred approach.

But with containers (and for other reasons as well) I moved to using dedicated packages for this kind of settings. So if my main packages is MyPackage I also have MyPackageSettings.

Moving slightly in the direction of versioning, deployments, etc., I usually recommend this place for another reason than changes on the system. The scenario I use it for is settings that are supposed to survive package updates. I introduced this many years ago for WxConfig where the most critical of those settings was the environment type. But that is probably a separate discussion …

[John_Carter4 (John Carter)]

@reamon

This is a newbie question, but is there a facility for keeping configuration info outside of the package (edit) in webmethods.io? Our config files often have environment-specific settings within so we do not store them in the package so that repeat deployments of a package don’t overwrite any config changes made in a given environment. (This will likely lead into discussions about repositories, version control and deployment scripts – which we can explore certainly but I’m curious as to options for config.)

Yes, it does. The configuration is contained in what we call the project scaffold and then applied to the runtime via configuration variables.

However, it is always a good idea to ensure that your package has some kind of minimal configuration to ensure that it functions out of the box and your CI/CD pipeline can also merge the configuration by layering configuration on top of the existing packages.

[ma.beaudet]

Hi,

My company does not have access to this feature yet.
As the source control is at the project, are workflows version controlled?
And if they are not, what would you recommend to do it?

Thanks,
Marc

[Sandeep_Deepala (wm developer)]

I don’t see the option “Integration Runtimes” in the masthead of webMethods IO trail version, if it is released in trail version it helps lot of developers to explore. Else it be always under the hood.

[Mike_Ng]

The DADA feature is for “Enterprise Plus” tier customer only.

DADA feature

Important : The Develop Anywhere, Deploy Anywhere and Central Control, Distributed Execution capabilities are currently available only with the Enterprise Plus tier. Contact your Account Executive to discuss upgrading your contract to the Enterprise Plus tier.

[ma.beaudet]

Well too bad
The recommendation would be to use https://github.com/SoftwareAG/webmethods_io_int_cicd?
Is it ok for use in production?
Are there ports for Gitlab?
Would someone be interested to do it with me?

Thanks,
Ma

[jahntech.cj (Christoph Jahn)]

When looking at the presentation in this GutHub project I stumbled over slide 9. This depicts the overall workflow and if I understand it correctly I have some questions.

• How long-lived are feature branches? More than a day and the integration is not really continuous anymore. And merging will become harder and harder.
• It seems to me that no development actually happens on the development environment. It looks more than the first merging stage.
• Is my understanding correct, that merging should also happen in QA and PROD?

At the end of the day my recommendation would be to start by looking at this first from the conceptual side and then dive into the details. Since I don’t know where you are in your process, that may of course already happened, in which case you should ignore my comment .

[jahntech.cj (Christoph Jahn)]

If anyone is interest in a blog article or YouTube video from me about a particular aspect of CI/CD, please let me know.