Introduction
This article is aimed at persons who are interested in administrating a webMethods Integration Server and need to know how to deploy and manage integrations via webMethods packages.
Contents
webMethods packages are the building blocks for integrations and the primary delivery mechanism for provisioning Software AG’s webMethods integration runtime. This is why webMethods Integration Server is ideally situated to responding to the rapidly shifting architectural changes in our production environments, from monolithic high availability clustered servers, to modern micro-services running in Kubernetes. You can define very precise runtime environments for specific needs by simply choosing what packages you want to deploy to your runtime.
The subjects to be covered are;
- What is a webMethods package precisely
- Installing, Archiving and removing packages
- Package life cycle
- Updating & reloading packages in a production environment
- Deployment in a production environment
- The package home page
- Access & Ports
What is a webMethods package
A package is simply a grouping of services and the necessary configuration, code and other stuff that can be installed, archived and removed from your webMethods Integration Runtime. Packages are the smallest deployable unit, and your solutions will normally comprise a number of different packages.
Services cannot be hosted without first defining a package and even Software AG provides about 90% of all the features of webMethods Integration via packages. Our packages are always prefixed with ‘Wm’, albeit you may see some packages provided by our experts in the field with the prefix ‘Wx’ too.
Packages are not binary objects and can be easily browsed via the file system, which also makes it easier to manage them via a version control system such as git. You can find them on the file system of your server or see them via the admin page of your server
Packages are persisted to disk either in;
- MicroServices Runtime - <SAG_HOME>/IntegrationServer/packages
- Integration Server - <SAG_HOME>/IntegrationServer/instances/<instance>/packages
Where <instance> is the name of instance that you have created, which will be ‘default’ 99% of the time.
You can also see if the packages have been successfully loaded or not from the webMethods Integration Server admin portal as show here
Developers should ideally provide a README in the package directory to outline what the package does and any post-installation setup steps that are required.
NOTE: as of 10.15 a README.md file will be created automatically whenever a package is created and developers will be encouraged to document it!
Installing, Archiving and removing packages
Out of the box a webMethods Integration Server is almost an empty box, with most features provided via packages. Software AG packages are primarily installed using the Software AG installer. However, your packages and third party packages will either be managed manually as described here or through some kind of automation, which we describe in the section below ’auto deploying packages’.
Packages can be deployed either via the file system (server will require a restart) or via the webMethods Integration Server admin page (as shown above). Installing the package through the admin portal should allow you to avoid having to restart the server, but not always!
Package management shows a complete list of all the packages that have been found in the aforementioned packages folder. You can choose to enable disable packages without having to remove them from the server. In which case the package will be removed from memory.
Packages can be installed either;
- Copy the package folder into the packages directory and restart the server
- Copy a package archive (effectively a zip) into the replicate/inbound folder and via the package management page, click “Install inbound releases” and then select the package from the pull down list.
The advantage of the latter option is that you should not need to restart the server. The replicate folder is found at the same level as the packages directory
- MicroServices Runtime - <SAG_HOME>/IntegrationServer/replicate
- Integration Server - <SAG_HOME>/IntegrationServer/instances/<instance>/replicate
In the directory you will the find the following folder
- inbound - place package zip here for installation.
- outbound - zip archives are written here when exporting packages.
- archive - copy of installed package zip file including date/time stamp.
- salvage - safely deleted packages, can restore them via “recover packages option”.
- autodeploy - this folder has to be created manually, will discuss in the section below on “auto deployment”
The package should appear immediately after installing a package via the replicate/inbound folder in the list. However check that the package was loaded correctly and that that there are no errors or warnings associated with the package, you can do this by clicking on the package name e.g.
Here you can see an example of a package that has only “partially” loaded and perhaps requires that the server is restarted (This could be the case where a package references a static jar library). Ideally developers should provide packages that do not require restarts and if they do, that it is clearly documented in the associated README.md
Likewise a package may show “No” as the enabled status, which means that the server failed to load the package into memory. This could be for a number of reasons and as stated you should refer to the server log more for more information.
In many cases it will be because the package is dependent on another package that is not yet loaded. You can view the package dependencies via the same package details page as before.
Therefore make sure that you install packages in the correct order. In most cases you will probably want to install packages in an automated manner and we will examine that below.
Archiving packages
Installing a package via the admin portal requires that you first archive the package and this is done via the same page. Each package has an archive icon
Which can be used to create a zip archive, which in turn in written to the replicate/outbound directory of your Integration Server instance.
All the above actions can be automated via administrative APIs. You can download the swagger for these APIs directly from you server with http://localhost:5555/admin/swagger/integrationServer
Package life cycle
Packages are loaded 100% into memory for performance reasons and the files on the disk other than config, are not referenced after that. It is important to understand how packages are loaded and initialised when starting a server, because a badly behaving package can seriously impact startup time or even stop a server from starting.
Firstly a package will only be loaded into memory if it is in the packages folder and has a valid manifest.v3 file.
The ‘enabled’ attribute must be set to ‘true’, otherwise the package will be ignored. Packages are first loaded into memory (alphabetically) and then initialised in order of dependencies.
Huh, what initialisation and what dependencies you may ask ??
Well, a dependent package is simply a package that reference another package. Virtually all package are dependent on the ‘WmPublic’ package as you will certainly have made service calls to it. However, you may have your own packages that have services or references that are used in other packages and they will be included in the “requires” list. Additionally developers can choose to have startup or shutdown services that are automatically executed when a package is loaded or removed. A badly behaving service can directly impact the startup or shutdown of your server either delaying it or blocking it completely. The server will not continue until all services have completed or failed.
You can see if a package is taking too long to startup by examining the server log at startup e.g.
The log shows the package being loaded, followed by the startup services which are executed in the order as dictated by the package dependencies. Triggers and inbound ports are only activated once all packages are loaded and initialised properly.
Disabling packages via the file system
If a package is blocking or stopping your server i.e. the last message in the log say something like ‘JcNastyPackage: Startup service (jc.nasty.tools:startup)’ then you can always disable it manually via the ‘enabled’ attribute in the manifest.v3 file as shown above. However, this in turn could cause other packages to fail if they are in turn dependent on it. Still, this can be your ‘get out of jail’ card if you need to urgently get a server up and running.
Don’t keep all your eggs in one basket
This leads onto an interesting question; “How many packages can be hosted in a single instance ?”
Well, how long is a piece of string ? Seriously there is no fixed answer to this, other than it depends? It also leads into a rather big subject about your chosen architecture and even a debate about the pro’s and con’s of micro-services versus the monolith. However, before we go down that path let’s say it does make sense to breakup your integrations into discrete packages to ensure that your server can startup in a timely manner and that if a server does need maintenance or indeed fails that not all of your integrations will be blocked.
Remember the more packages you host in a single server, the longer the server will take to start and the more memory it will consume and hence have less working memory for your processing.
I don’t want this article to turn into a discussion on tuning, but remember you will improve performance if you fix your min and max memory JVM attributes to be the same. You probably knew that, but did you know that packages are actually loaded initialised in multiple threads in order to improve performance too ? You can change the available number of threads via an extended setting called.
- watt.server.package.parallel.threads
By default it is set to 6, evidently dependent packages are only initialised in the correct order.
Updating & reloading packages
Now let’s look at what happens when we decide to reload a package as it can have an important impact on service availability, with the potential to incur lots of exceptions or even data loss.
First why would we need to reload a package ?
Primarily because we want to be able to update a server without requiring a restart.
It’s important to understand what actually happens when a package is reloaded in order to understand the impact. The steps are as follows if you DON’T have any package dependencies
- Shutdown service of package is executed.
- Package is removed from memory once shutdown has completed.
- Package is read from disk and loaded into memory.
- Startup service is executed.
- Package is now available.
It’s important to appreciate that any invokes to the package during this process will result in a service not found exception (404). In addition if the package has any dependent package(s) then by default these get reloaded too, which you can see if you refer to the server log under admin → logs → server.
As you can see reloading package B, causes package A to also reload. This can cause a cascade of packages to reload. During which time, services in all the related packages will trigger service exceptions.
Therefore we do not recommend that you reload packages manually in a production environment.
Deployment in a production environment
As stated you should not reload a package without understanding the impact in a production environment. However you can update a package without having the service not found exception being triggered.
How ?
By enabling hot deploy in the settings section from the admin page of your server.
The same process as above will trigger when installing a package. However, the server will keep a graph of all the dependent services that are effected and allow incoming calls to be initiated for them. it will then immediately block the threads until all of the relevant packages have finished reloading. After which the threads will then be allowed to proceed and execute the new version of the services.
Once done you can install a new version of a package via the admin portal under packages → manage → “Install inbound release”
You will need to put the zipped version of the package into the replicate/inbound of your webMethods instance in order to see it in the list of packages to install.
The upside is no more “Service not found” exceptions in production. The downside; well potentially there are two issues;
- Client timeout - the client will trigger a timeout if the reload takes too long and so error out. In addition we will still uselessly invoke the service
- Exhaust thread limit - Threads still have to be allocated and hence we could cause the server to max out. This issue can be mitigated by ensuring that you use private thread pools and dedicated ports (see later).
So don’t use this feature to deploy utility packages with lots of dependent packages and certainly not during peak activity. We will be introducing a new package property in 10.15 to allow you to choose whether dependent packages should be reloaded or not. In most cases the reload isn’t required and so we have decided to make it optional. However, you will have to choose to activate it via the package properties when it becomes available in 10.15
NOTE: The hot deploy feature does not work for reload, so you should still avoid reloading packages ini a production environment.
Auto deploying a package
We talked about manually updating a package, but deployment can also be automated via the following techniques;
- Subscribing to packages
Little used feature to allow one server to publish packages to one or more subscribing servers. Personally I have never used this feature, but if you are interested you can refer to our official online document in empower
Copying Packages from One Server to Another
- Deployer
webMethods deployer is Software AG’s solution for packaging complete solutions and deploying them, where a solution comprises multiple package and other assets. You can find more information in our empower documentation centre
- Auto deploy
You need a micro services runtime license to use this feature, but it allows you to drop a package into a directory, where it will be automatically picked up by the server. Refer to our empower documentation centre
- Packages API
webMethods Integration Server has an admin API that allow you to install/archive packages e.g.
- POST - http://localhost:5555/admin/package?action=install&filename=MyPackage.zip&enable=true
This would still use the replicate/inbound directory as the source directory, likewise for archiving e.g. - POST - http://localhost:5555/admin/package/MyPackage?action=archive
You can fetch the swagger definition for the complete Integration Server administrative API from you local webMethods Integration Server with the following link
http://localhost:5555/admin/swagger/integrationServer
All the above techniques leverage hot deployment if it is enabled.
Server clusters
Remember that servers in a cluster should not present different versions of the same package and hence you should upgrade all of the server nodes together. However, this does not mean that you have to bring down your entire cluster, instead you should be able to upgrade one node at a time. Having different versions of the same package in each node should not present an immediate issue. If this is not possible you will have to envisage a blue/green deployment where the entire cluster is cloned and upgraded, before then switching from one to the other.
NOTE: This depends on the quality of the development of your packages and again you will need to refer to the release notes or README provided by the developer of each of the packages to be updated. I will soon post an article on packages aimed at developers where I will discuss how to ensure packages can be updated to ensure that they are backward compatible and capable of running in a mixed environment.
A word on micro-services
Please do not try to deploy packages to directly to runtime containers if you are using a micro-service architecture. The golden rule is to ensure that containers are immutable, which means you should not try modifying them when they are running. Instead employ a build and replace strategy to replace containers of an older version with a new version.
The analogy is that you don’t try to replace the wheels on you car whilst driving. Better to replace your car with a loaner whilst it is updated. In our case the car being an image. I will talk about this in more detail in another tech article at a future date.
The package home page
Did you know that each package has its own home page? You can access it very easily from either package management admin web page or directly by adding the name of the package to the end of the url for your webMethods runtime server.
By default the home page is not very interesting
However, in many cases the home page will be have been modified by the developer to either provide some kind of status page or perhaps to provide online configuration or settings. So make sure to check a package’s home page!
In fact it is perfectly possible to host a full fledged modern React/Angular/javaScript application via a package through the home page!
NOTE: In 10.15 we will be providing a new public service to allow to make your home page much more interesting than above with only a single click. The new service will autogenerate a html page to document all of the API and services (only commented services). This will be of great help to developers who want to use your package and ensure that you can easily keep in sync your development with your documentation. The only requirement on you ? Ensure that you comment your services and APIs in Designer!
Access & Ports
We need to have some kind of access into the server so that your new services or at least the public services can be exploited. This can be the following types.
- Adapter notifications - external applications pushes data into webMethods
- Messaging trigger - UM or other messaging (JMS) system are received and consumed
- File polling - Local directory is polled and files retrieved
- FTP - Integration Server can act as a FTP/SFTP server to receive files.
- HTTP - Integration Server can act as a web server, service or API provider.
- gRCP - Google’s answers to performant APIs.
All of these configurations are provided via packages and you should be sure to make sure that they are properly secure against nefarious access.
This means that you may see additional ports added to your server after installing a package. You can view all of the servers ports via the admin portal under Server → Ports.
This includes file polling and FTP/SFTP, but not messaging or gRPC. There are dedicated web pages for these.
HTTP access
HTTP access is probably the most used protocol and potentially the biggest attack vector to your server, so you should ensure that your ports respects the following best practices.
- Don’t use the admin port for any other use than admin access
A developer should always provide a port via the package if HTTP/HTTPS access is required and not assume use of the admin port. In fact ideally you should block the admin port from external access via your firewall.
The port configuration file can be found in the packages’ config directory and is named ‘*listeners.cnf’
- Secure the port
Ports other than the admin port should be ideally set to “Deny by default”, which will require that the developer has explicitly listed the allowed services in the port configuration. The advantage is that it reduces the attack plane as only these servers can be invoked regardless of the authentication made. Ask questions of any port that is set to “Allow by default”!
- Dedicated thread pool
Ports can be configured to have their own dedicated thread pool and this is our recommendation. The advantage is that the server cannot be brought down by DOS (Denial of Service) attacks on the given port. The port itself will block and refuse requests when its own pool is exhausted. However, the server and all other interfaces will continue to function.
Summary
There you have it, webMethods packages are the building blocks for your integrations. Providing you with a simple means for deploying a well defined set of features, that are self contained and reliable across a number of different runtime environments.
In my next article we will further examine packages through eyes of a developer to ensure that developers provide packages that are robust, reliable and are easy to install and setup
