Packages are critical to development.
As a webMethods developer you perhaps know all about packages, but you still might be able to pick up a few new nuggets of interest in this article about developing packages so that your packages are autonomous, reliable and easy to deploy. If you are new to webMethods and haven’t a clue, refer to my previous article on packages and how to manage them “A word about packages”.
We will cover the following subjects;
What goes inside your package
Hosting web apps via packages
Some recommendation for package layout
How many packages ?
Orchestrate, Don’t chain
Collaborating with other Developers
Versioning packages with GitHub
What’s inside your package
A package is simply a folder in the package directory with a number of files and sub-directories. In reality you can add your own sub-folders, files etc. as you wish. The only critical files are the ‘manifest.v3’, and the ‘code’ and ’ns’ folder.
For now I just want to warn you that the ’ns’ folder is the one folder that you shouldn’t play with. It’s the source of all your services, document types, connections etc., effectively everything that you create in Designer is stored in this directory or below it.
The above screenshot on the left shows the folder structure below ’ns’ to the service ‘jc.API.helloworld_.services.v3:helloworld’ as opposed to the structure that we see in Designer for the same service (see below)
Note that the service is itself a folder with the code represented by a file called flow.xml. Be warned you can edit this file, but be aware that you could corrupt your service, albeit by default there is a backup file alongside. Any changes on the file system will require that you reload the package into memory; which can be done either from the package management page in the admin portal, or if you are using Designer, you can instead right click on the package and select ‘reload package’.
The ‘node.ndf’ file is a non readable blob the collects all of the common properties that services require. However, you can only change these properties from within Designer. The other directories ‘config’, ‘pub’ and ‘resources’ are little used, but seriously cool in ensuring that you can create packages that are easy to configure, setup and understand.
The code directory is where java services are generated and then compiled. Looking closer you will see three sub-folders;
- classes - compiled class files representing your java services.
- jars - place your third party java libraries that you want to reference from within your java services
- source - Generated source files for your java services. Don’t attempt to modify these files as the changes will get overwritten whenever you edit the java services from within Designer.
Hosting modern web applications via your packages
Did you know that each package has its own home page? You can access it very easily from either package management admin web page or directly by adding the name of the package to the end of the url for your webMethods runtime server. However, the default home page is of little interest.
Thus, we recommend that you replace it with your own.
You can either replace or edit the existing ‘index.html’ or if you prefer you can provide dynamic content by replacing it with an ‘index.dsp’, where dsp stands for Dynamic Server Page. a DSP page uses tags enclosed in percent symbols to allow you to invoke services, loop or include optional values via a branch. Refer to the following document on how to write your own DSP pages here Dynamic Server Pages development guide
Alternatively you can deploy a modern javascript applications directly to the package and leverage the power of hosting the required APIs from the same runtime container.
NOTE: In 10.15 we will be providing a new public service to allow to make your home page much more interesting than above with only a single click. The new service will autogenerate a html page to document all of the API and services (only commented services). This will be of great help to developers who want to use your package and ensure that you can easily keep in sync your development with your documentation. The only requirement on you ? Ensure that you comment your services and APIs in Designer!
The package structure
Okay, we have had rather a lot theory, let’s continue this story but perhaps instead with some practical examples and then we can talk about how we should lay out our package. You don’t have to, but it might help you understand better if you create your own package to follow the next section(s).
Create two packages in Designer by ensuring that you are in the service development perspective and connected to your server.
Click on File → New → Package
And name you package A and B, with each package prefixed with your initials e.g.
- JcA
- JcB
A word on package naming. (Best practice Alert!!!)
I’m not going to say you have to use the following naming convention and I should state that “best” practices is perhaps contentious as all practices have pros and cons. I perhaps should rename this as @reamon recommended “common” practices. However, this is a pattern that I have used and by others in Software AG as well as by many webMethods developers around the world. It is loosely based on Camel Case, where each word starts with an upper case letter. However, in addition we recommend that you prefix each package with the initials of your organisation, division or even your own initials. The prefix should start with an upper case letter and the rest in lower case e.g.
- JcMyGreatPackage
- WmPublicTools.
- AcmeTools
Generally we don’t include the version or build numbers in the name as these values can be referenced in the packages’ manifest.v3 file.
Packages cannot be renamed in Designer, but can be renamed in the file system by simply renaming the package folder. However you will then need to restart the server and be careful if you have other packages that are dependent on it as the “requires” attribute in the manifest.v3 file will also need updating too!
NOTE: Another rule about naming is to make it functional rather than technical i.e. which would you prefer as a name ‘JcOracle12xDb’ or ‘JcInvoicesStore’ ? The first is technical and tells me nothing about what it does. Additionally we risk have a very fat package if our DB stores everything. Likewise, I might want to replace the DB one day, whereas I’m pretty sure that we will always need invoices. Well least until the end of money!
Name spaces
Well we now have two packages, but they are empty. We need to create some assets in the package, namely some services. However, we can create anything with out defining at least one folder in each package.
Why ?
It’s all about name space. In order to be able to invoke a service we need to identify it, in which case it has to be unique. Rather than have a stupidly long service name like ‘myReallyUnquiqeServiceByJohnCarterThatJustSaysHi’, which would be difficult to categorise and remember, hence we leverage a namespace.
The cool thing is that name spaces can overlap between packages as the package name is not included; only the folder names. Only the assets themselves cannot overlap, but whilst this is cool it isn’t recommended. That’s why we generally include the name of the package in the namespace.
Package A contains
- jc.a.services:helloWorld
Package B contains - jc.b.services:goodbyeWorld
- jc.b.services:helloWorld
If we had not included the package name then assuming that A is dependent on B then the ‘helloWorld’ service in package B would win and that of package A would not get loaded. In turn a warning would appear in the log and package loaded status would appear as ‘partial’ i.e. “only partially loaded” in package management.
That could only happen if the packages were created in separate environment and then installed into the same server. Designer will not allow you to create a second service in the same server if the name space is already taken.
So back to work, populate the two packages with the following folder hierarchy and services.
Create the folder hierarchy and services in one package and then simply copy the ‘jc’ root into the second package.
You can then rename the folder ‘a’ to ‘b’ in the second package. Not forgetting to then rename the copied folder ‘jc_1’ back to ‘jc’. Remember folders can overlap.Configure the properties of each package so that the relevant startup service is invoked and make sure that package ‘JcA’ is dependent on ‘JcB’
You should have the same structure as above, albeit the services are still empty. However, before continuing lets talk about naming again!
Don’t get lost! (best common practice Alert!!!)
Folder naming is important, especially as packages and projects get more complicated. Therefore try to pick a consistent model across all packages, be descriptive, but also avoid redundancy . Equally important to make a clear distinction between assets that you want to share outside of the package and those that you don’t and ensure that the name space is unique to avoid namespace clashes!
In my case I always start with a root folder representing my organisation i.e. the same as the package prefix (Oh I forgot folders should never use upper case letters, it will become obvious later when you want to start invoking these services from external clients!).
Why ? In many cases you can then use the organisation as a filter for many useful tools such as monitoring, log analysers etc. that way you can distinguish your logging from internal logging. Just filters on ‘jc.*’
The next folder is the name of package in lower case.
Next we have two important folders; namely
-
_priv
Anything that you do not want other developers to use from outside this package goes in here. The underscore is to ensure it stays at the top. Obviously you can create sub folders to group things. I often have a sub-folder for adapters, API resources, mapping and transformers. -
pub
All my public service go in this folder, albeit you are allowed to create sub-folders too if you want. These are services that developers are allowed to reference from other packages.
You may wish to omit the ‘pub’ folder if you don’t have other packages to be dependent on it directly. However that doesn’t mean that the package cannot be invoked, after all hopefully you will be developing APIs for all of your services; in which case you should have something like below;
An ‘API’ folder hosts all of your API descriptors e.g. ‘greeting’ (1)
The rest resources, document types and services are in the ‘_priv’ folder as they don’t need to be referenced directly (2).
The API service (3) itself would then invoke the public service helloworld if requiring the same implementation (4)
The namespace for calling the API would be something like ‘/rad/jc.a.API/greeting/hello/John’
Completing the code
This will be easy as we aren’t here to talk about flow services or APIs. So we will simply get the helloworld service in package to invoke the helloworld service in package B. To do this, double click on the service ‘jc.a.pub:helloWorld’ to open the editor.
Then drag the service ‘jc.b.pub:helloWorld’ into the empty editor.
Then click on the input/output tab of the service and add a string attribute to the inputs and name it name.
Now double click on the helloWorld service in package B and do the same i.e. click on the input/outputs tab and add name to the inputs.
We want B to do something albeit not very much. So click on the ‘Tree’ tab and add an invoke step to the flow. Select the service ‘pub.flow:debug’ from the chooser. Set the function input to something like ‘** DEMO **’ and then set message to ‘Hello %name%. Don’t forget to check the radio button ‘Perform pipeline substitution’ before clicking on “OK”.
Do the same thing for the startup services in each package, i.e. invoke debugLog. Set the messages to ‘Starting up package A’ and ‘Starting up package B’ respectively.
This is just so you can see what happens when packages get loaded.
If you right click on the service helloWorld in package A and then select ‘Run As → Run Flow Service’ you will see a dialog asking for you to input the name. Afterwards if you refer to the server log you should see you message such as ‘** DEMO ** - Hello Bob’
A word on service naming
Remember just like packages make sure that your services have functional descriptive naming that encapsulates without ambiguity the goal of the service e.g.
- jc.ordermgmt.pub:getOrder
- jc.ordermgmt.pub:cancelOrder
However the following would be names to avoid;
- jc.ordermgmt.pub:process - too ambiguous
- jc.ordermgmt.pub:insertIntoOraPL4501Table - too technical, doesn’t describe the “what” and we don’t want to expose implementation details in the naming.
How many packages ?
In my previous article " a word about package" we discussed about how many packages should be hosted by a single server and related that there is no quick answer. The same question arises in relation to a particular integration project, “should a developer develop a single package for a complete project ?”
This is easier to answer and in most cases the answer is ‘no’, you should not create a single fat package to develop your solution. However, there are some rules to make sure that other developers can follow what you are doing, so let’s look at some of them here.
-
Keep packages small and functional
We already mentioned this earlier, a package named ‘JcInvoicesStore’ as opposed to to ‘JcOracle12xDB’ is more descriptive, but will also avoid putting too much in a package. Why do we care? smaller packages will be easier to install and update as they will have less impact on the server, this is explained in the next section. Equally development and maintenance becomes easier as you can have a single developer working on a package as opposed to requiring a team. -
Make package discrete and autonomous
Packages should just work i.e. not require a lot of manual configuration or setup. You can ensure this by using the package’s config directory to store local configuration and using a startup service that then sets the required configuration. For simple configuration think about using global variables and remember virtually all of the admin and features of the platform are available via services and/or APIs. You can even use a startup service to create DB tables or other external resources such as message queues/topics to avoid manual setup. Be careful though to avoid having startup services that take too much time to execute, otherwise your server will be too slow or perhaps get stuck if your startup service is badly coded! -
Manage package dependencies
Making packages autonomous doesn’t preclude allow packages to reference other packages, after all we want to encourage re-use. However, it should be done with discretion. Avoid circular dependencies, try to move shared services into common utility packages and ensure that you explicitly define the dependencies between your packages e.g.
The above diagram shows dependencies to avoid in red and good dependencies in green. Generally you should think of packages in a stack and avoid dependencies within the same layers and allow only downward dependencies from a higher package to a lower package. Even then try to leverage messaging if synchronous calls are not required and if a package needs to invoke leverage another package in the same layer then do this through a public API or again messaging. We can see this in the above example where the order management package may want to leverage the product catalogue to check if a product code is valid.
Defining package dependencies
You have to explicitly set dependencies between packages, it isn’t done dynamically and you will need to do this from within Designer by right clicking on a package and selecting ‘properties’
Then select ‘package dependencies’ and add the package(s) that this package is dependent to the list. I wouldn’t recommend hard-coding the version as it would need to be updated every-time the package versions change. Simply specify ‘.’ for any version. These properties can also be found in the manifest.v3 file in the “requires” asset.
You don’t have to worry about the dependency on the WmPublic package as that is presumed and we always make sure that this package gets loaded first. However, if you have dependencies between two of your own packages; say A invokes B; then you need to make sure that B is loaded and initialised in order for A to work.
Package startup & shutdown
You may have noticed that there is a section called ‘Startup/Shutdown Services’ and this is where you can ensure that your package is properly configured and ready for use when the server is finally flagged as ready. A startup or shutdown service is simply a service that get executed whenever a package is reloaded or reloaded into memory. Likewise a shutdown services gets executed whenever a package is removed from memory i.e. at deactivation, server shutdown, package reload or removal. They are great to ensure that your package is autonomous and doesn’t require manual setup. I often use a startup service to automatically create DB tables required by my JDBC adapter services if they don’t yet exist, or setup a scheduled service etc.
WARNING: startup/shutdown services are brilliant but dangerous and are often the culprits for a server that refused to startup or takes too long to start. Equally package dependencies must be documented, otherwise you risk a startup service failing because a dependent package is itself not yet configured properly. Therefore make sure that the services have known execution durations, are robust and that you have defined your package dependencies properly.
Orchestrate, Don’t chain
Avoid linking one package to another as shown below. If the order management package needs to check if a product is valid this is fine. However, you should avoid linking one service to another in this way.
The above where x calls y calls z is known as chaining and it will be a nightmare to understand and maintain. In which case think about defining an explicit integration layer that acts as orchestrator.
Many customers used our BPMS product in the past to act as the orchestrator. Personally, I was never a fan as it introduces a lot of unnecessary overhead that complicates your integration platform. BPMS is a very powerful tool, way too much just for simple orchestration!
Workflow in webMethods.io or just a flow service.
Today instead you can use webMethods.io workflows to do the same thing, which is seriously cool as it allows you orchestrate complex integrations across different platforms, regions and across multiple hyper-scalars. As you move to the cloud you could use this technique to orchestrate steps between your AWS private cloud, on premise legacy applications and your Salesforce SaaS applications from within in a single view!
Dedicated integration package
Or you can use a good old package and flow service to invoke the necessary services, which again can be done via API calls e.g.
However, try to avoid direct invokes between the orchestrator service and the dependent packages as this will force you to have all the packages in one server. Whilst this might not be a problem early on in your project i.e. a single server suffices for your needs, you need to consider the future where you might want a distributed system to improve performance or to place servers closer to the back end applications; or even revisit your architecture to start moving to the cloud, where you will want to have packages running in different regions or split across cloud and on premise. Adopting the above techniques will make the migration much easier it won’t require you to rewrite too much of your code and make it easier to move a package from one server to another.
Collaborating with other Developers
The last subject for this tech article is to discuss how share our packages with other developers and in turn take advantage of their packages to build our solutions ?
Shared environment, NO!
Traditionally this would be done by leverage a shared webMethods runtime server so that developers can see each other packages and take advantage of them. This is not recommended for the following reasons.
- Developers are reliant on a single runtime and if it becomes unavailable or inaccessible they are stuck.
- Developers may need to work with different versions of the same package, which will cause conflicts.
- Difficult to implement proper version control and even if in place, tends to operation in batch mode where many changes are committed together with no trace of who made the changes or why.
Developer workstation with version control
Today we recommend that each developer has their own dedicated webMethods runtime running locally on their machine. Developers then share packages via an integration version control system. This ensures that developer can work independently, whilst maximising collaboration and proper code control.
You can even download a complete workstation including Designer and local Integration Server via tech community with the following link
Download centre for Service Designer
Unzip it and you have a complete workstation to start developing with at your finger tips!!
We have already removed all of the previous trial restrictions with this version and offer macOs and Linux variants as well as Windows. Our next step will be to get it to accept your license so that you can have access to full support for it without having to install it again via empower (license software users only).
Now you have access to a development environment, how do you share your code and the code of your fellow developers ?
Source code versioning is the answer; It is good practice to version control your packages as you will then have access to the historical code changes, which allows you to track who changed what, when and why. It also allows you to easily setup an environment to a previous released version in order to test for bugs that have occurred in production environment and create fixes/patches quickly. Developers can share packages via the same git VCS system as well.
Versioning packages with GitHub
We will be using github as our preferred versioning system in these examples, but the same process should apply to any VCS system you choose. Service Designer is agnostic about how you choose to position your repositories i.e. do you want a single repository per package or a repository for a collection of packages. My personal recommendation is to use dedicated repositories as it gives you much more control over managing change and versioning each package
If you want to see some example, I have many packages that I distribute via github e.g.
- JcPublicTools - Package with a bunch of useful utility services
- JcAPITools - Set of services to simplify API integration with Software AG’s API Gateway
- c8yPhilipsHueAgent - Example Cumulocity agent for managing for your Philips Hue devices.
etc.
I would recommend that you create a README.md file in the package directory so that developers have some online documentation when referring to the repository in GitHub. In 10.15 we are planning to create a README.md file automatically whenever a package is created. You will still need to add some content to the file though, we aren’t magicians!
So let’s go version control our previous package, in this scenario you will need to register with Github beforehand if you aren’t already. Don’t worry it’s free, no credit card required. You will also need to install the git client on the same machine Git Client Downloads.
Creating the git repository for your package
Once registered it is very easy to create a repository in GitHub, simply click on the repositories tab and then click on the green “New” button. I would recommend naming the repository with the same name as your package, this will ensure that you can later clone it to any server without any additional steps other than the git clone. It can be either public or private, although if you want to make it private you will need a developer access token, which I will describe below. Don’t add any additional resources to the git, we will do that in the local webMethods environment.
Once created you will see something like below.
It very kindly provides us with the necessary commands to run in our local environment so that we can sync our code with the repository, albeit we will want to make a few changes, see below. However, first you need to navigate to the directory where your package is; If using webMethods Service Designer it will be as below where <APP_HOME> is the folder where you installed service designer.
$ cd <APP_HOME>/wmServiceDesigner/IntegrationServer/packages
Enter the following line only if you haven’t already create a README.md
$ echo "# JcPackageA" >> README.md
Create a git ignore file if you don’t want backup files to be uploaded
$ echo .bak > .gitignore
Required on macOS
$ echo .DS_Store >> .gitignore
Create local repo and add all files to it
$ git init
$ git add .
Make your first commit and link it to the remote repository
$ git commit -m "first commit"
$ git branch -M main
$ git remote add origin https://github.com/johnpcarter/JcPackageA.git
This is the command which push your local code to the remote repository
$ git push -u origin main
If using a private git repository
A private repository you will prompt you for your user id and credentials. Do not enter your git password here, it will no longer be accepted. Instead you need to generate a “Personal Access Token” in GitHub. Click on your avatar in the top right hand corner and from the pull down menu select “settings”, then choose “Developer settings”, followed by “Personal Access Tokens”. Click on “generate token”, enter your password when prompted. You will need to name your token and set the access level.
You need only select the first choice ‘repo’. Then scroll to the bottom and click the green button “Generate Token”. Copy the generated token and paste into password prompt. Your local repository is now linked with your GitHub repository. If you refresh the repository page, you will now see something like the following
Thereafter any changes made locally can be synced from the command line using
$ git push
or to pull changes from the repository
$ git pull
Here is a link for a summary of useful git commands.
Using Designer to commit and pull changes
Not forgetting that you can also use Designer to commit or pull changes from your repository. To do this you need to right click on the package and select “Create local service development project”. It will then prompt you to select the proper git repository. It should be already selected so simply click done, but do check beforehand especially if you have multiple git repositories setup.
Thereafter you can right click on any asset in your package and then via the “Team” menu commit or pull changes.
Personally I’m old school and prefer to use the command line. If you prefer to use Designer, make sure that you update your workspace settings to ensure that the file system is synced automatically, otherwise git changes will not be reflected correctly in Designer.
For more details there is also an online video showing how to you use Designer with a VCS system here
Integrating a Version Control System (VCS) with webMethods Service Designer
and another tech community article that covers what we have done here using only Designer
Add a webMethods package to a git repository using Service Designer
A word on Java services
Looking at my example above you may be wondering why I allowed the compiled classes to be committed and did not choose to ignore them. Normally we would not want compiled code to be added to our version control. However, it is important to remember that a package is considered a runtime component and should be viable once installed. If you omit the classes then the java services will not be operable and you will will need to use the ‘jcode.sh’ utility to recompile your packages every time you pull the package. Good luck with that, especially as you will need to make sure that all the dependent jars are included in the class path. Instead I recommend that you include the class files and have packages that are ready to execute out of the box.
Check your Designer settings to ensure that .class file are not in the ignore list under VCS(Teams).
Summary
You can now start building your own API and micro-services now that you know how to use packages and have downloaded and installed your own development environment. My next tech article will revisit package decomposition and dependencies to ensure that you can easily adapt your platform to different architectural patterns be it the classic monolith or micro services.
