The sample code can be found in our Github repository at https://github.com/SoftwareAG/webmethods-integration-examples/tree/master/servicedesigner-csvconverter
Introduction
webMethods Service Designer is a lightweight integrated design and development tool that allows for a quick start in developing services and integrations using the webMethods platform. The combination for graphical coding and mapping side by side with Java coding allows for faster and efficient coding with maximal flexibility.
It includes:
- webMethods Service Development for designing and developing services for the webMethods Microservices Runtime and managing them under version control.
- Unit Test Framework for creating unit tests for the services that you develop and include in Continuous Integration for build verification.
- webMethods Microservices Runtime (referred to as Local Development Server) for deploying and testing the assets.
This example shows you how to use service designer to
- create services for transforming an arbitrary CSV file to xml or json
- build a docker image hosting those services
Import the package - Describes how to get the finished demo package with the working services into your Service Designer. This is the section If you want to build the complete code your own, see below.
Build and run the Docker image - describes how to build a docker image containing the services and run it.
Build the package your own - Describes how to build the complete code on youre own. This is the section if you want to undertsand the complete code. You can also have a look here if you want single parts explained.
- Build CSV2Doc is the main worker (and therefore the longest section). This one converts a csv to an internal document type .
- Build CSV2Json converts a csv to json using CSV2Doc and the built in service pub.json:documentToJSONString.
- Build CSV2XML converts a csv to json using CSV2Doc and converting it's output to XML using a built in service pub.xml:documentToXMLString.
- Expose the service via Rest shows how to make the services accesible via Rest API. We will show how to create a Rest Ressource and then an Rest API Descriptor to which gives us a Swagger descioption of the API.
Disclaimer: Service Designer is intended for rapid prototyping, you need an appropriate license if using generated assets in production environments.
Requirements
- Windows 10 or Windows 2019 Server
- Docker for Windows with Engine 19.03 or newer
Get webMethods Service Designer
Unpack it to a local directory (I'm assuming C:\Programs\wMServiceDesigner in the course of this howto):
Doubleclick ServiceDesigner.exe to start it. Select a workspace:
When asked if you want to start the Local Development Server click yes. Otherwise you can start it later by selecting the menu item Server-Start.
Wait until the server is started. You can either check if the admin page is reachable (http://localhost:5555/) or looking for
ISSERVER|| 2019-11-14 14:03:47 CET [ISS.0014.0002I] Initialization completed in 51 seconds.
ISSERVER|| 2019-11-14 14:03:47 CET [ISS.0025.0016I] Config File Directory Saved 
You can access the admin page of the started Microservice Runtime via http://localhost:5555/.
Username: Administrator
Password: manage
Go to Designer and make sure you use the Service Development Perspective. The local development server is preconfigured. If it does not connect automatically rightclick on the Server icon in the Package Navigator and select Connect to Server.
Import the package
If you just want to view the demo, clone or download the demo repository: https://github.com/SoftwareAG/webmethods-integration-examples/tree/master/servicedesigner-csvconverter
If you want to get the package properly into your workspace using the git integration of designer, please have a look at my article about getting packages from git or github into Service Designer.
If you want to have it quick and easy, copy the directory Demo_CSV_Converter (located inside ISPackages ) inside C:\Programs\wMServiceDesigner\IntegrationServer\packages (This is just for demo purposes and not a recommended way to deploy packages to a Microservice Runtime or Integration Server).
Go to it's admin page (http://localhost:5555/)
On the left side expande the Packages section, click Packages-Management
Click Activate Inactive Packages
Select Demo_CSV_Converter.
Click Activate Package
After the import you can go pack to package management
where the package will be displayed:
For a simple tryout of the services where
* the file is a,b,c;1,2,3;10,20,30
* the lines are delimited by ;
* and the fields by ,
using http just call
while the local development server is running.
Build & run the Docker image
Preparations
Switch windows docker desktop to windows containers.
Open a command prompt. You may need to start cmd as administrator to have all needed write permissions.
Login to docker to be able to pull the windows base image.
docker login -u youruser -p ****
Switch into the docker directory of your installation. If you placed the Service Designer into C:\Programs\wMServiceDesigner, it will be:
cd C:\Programs\wMServiceDesigner\IntegrationServer\dockerProcess of building images
Building an image consits of 2 phases with 2 steps each. The phases are
- Build a Microservice reference image. This contains the Microservice Runtime and in a real life environment would be used as base for multiple different service images. This image will not be startable.
- Build the service image. THis will use the base image and add your specific logic to it.
The steps in each phase are
- Create a dockerfile
- Build the image
Build & run the example
Create the dockerfile for the base image for Microservice Runtime. The following command will create a dockerfile islean in the directory C:\Programs\wMServiceDesigner
is_container.bat createLeanDockerfile -Dfile.name=islean –Dimage.name=mcr.microsoft.com/windows/servercore:ltsc2019
Build the base image
is_container.bat build -Dfile.name=islean -Dimage.name=islean:1.0
Create the dockerfile for the service image. The following command will create a dockerfile CSVConverter in the directory C:\Programs\wMServiceDesigner\IntegrationServer\packages
is_container.bat createPackageDockerfile -Dimage.name=islean:1.0 -Dfile.name=CSVConverter -Dpackage.list=Demo_CSV_Converter
Build the service image.
is_container.bat buildPackage -Dfile.name=CSVConverter -Dimage.name=csvconverter:1.0
Run the service image
docker run --name csvconverter -d -p 15555:5555 -p 19999:9999 csvconverter:1.0
If the container is running, issue one of the following lines in a webbrowser
In the input we assume
- the file is a,b,c;1,2,3;10,20,30
- the lines are delimited by ;
- and the fields by ,
this is kept simple so it fits into the adress line of the browser, when calling with other clients you may pass real files.
Diagnosis
If you want to introspect the running container, you may do so by issuing
docker exec -ti csvconverter cmdThe log of the Microservice Runtime is server.log inside C:\SoftwareAG\IntegrationServer\logs
Build the package your own
Introduction
The services in the sample package take CSV data as input. The data is expected to have
- One specific delimiter between each line
- One specific (different from above) delimiter between each 2 fields inside a line.
- The same line structure for all lines.
- The first line containingthe field names at the exact positions where the corresponding content of the following lines is.
- No empty fields or lines.
For demo purposes we will call the services using plain html, therefoe we will have , and ; as delimiters. But the services can also procees the other usual separators, like carriage return, line feed, tab etc..
Data transportation, transformation and mapping in webMethods Integration uses internal Documents. As webMethods comes with build in services for transforming those form and to JSon and XML, one service transforms the CSV to an internal document and this will be reused in the servcies to create XML or Json.
CSV2Json
This service just consist of 2 steps.
The first invokes CSV2Doc to transform the CSV to an internal document.
The second invokes the build in service pub.json:documentToJSONString located in the package WmPublic to transform the returned document to a JSON String.CSV2XML
This service is similar to CSV2Json, it just invokes pub.xml:documentToXMLString to return a XML instead of JSON.
CSV2Doc
This is the real worker. Here we will parse and split up the file and finally convert it to a document.
Create the package and folders
A package is the delivery unit of webMethods assets. The namespace of the assets is defined by the folder structure below. The package name is not part of the namespace, therefore it is often best practise to reflect the top level folder structure in the name of the package.
The equivalent of a webMethods package in Java is an archive.
The equivalent of a webMethods folder in Java is a package.
In the Package Navigator right click on the server name (Default) from th context menu select New - Package.
Name the package Demo_CSV_Converter. Click finish.
Right click on the new package. From the contect menu select New-Folder.
Name the folder DemoCSVConverter. Add 3 additional folders below by right clicking on the top folder and selecting New-Folder.
Documents are the internal data structures for transporting data. They can be easily converted to and from Json and XML string. We will use a fixed document type for the name value combination in the flow. Right click on the Dcouments folder and select New - Document Type.
Name the document type Field.
Click finish. The editor for the document typer will open. If you want to edit an asset lateron, just doubleclick it in the Package Navigator. Open the palette byclicking the arrow in the upper right corner of the editor window.
From the palette drag 2 String variables to the canvas and name them name and value.
Save your work.
Create CSV2Doc
Right click on the Services folder and select New - Flow Service.
Name the service CSV2Doc.
Make sure you have the Input/Output tab actice in the editor window.
The service will return a a document with several lineitems with a dynamic structure based on the input. Therefore just drag a Document type variable from the palette to the outpur pane and name it lines.
Drag a Document List into the output pane below the lines document.
Having it selected click the Shift Right button in the upper pane to make it a substructure below lines.
Drag three String type variables into the Inpuit pane and name them inCSV, lineDelimiter and fieldDleimiter.
Switch to the Tree tab in the editor. We will use a built in service to separate the input into it's lines (and later also the lines into their fields). In the pacgae explorer open the package WmPublic by clicking the > and navigate down to the folder pub - string. From inside this folder drag the service tokenize to the canvas.
Click the service in the editor pane to make it active (if ot isn't alreday) and select the Pipeline tab in the windows below the editor (Assuming you are using the standard layout of the Service Development perspective).
webMethods flow operates on a concept called pipelinewhich contains all the data present at a specific time during the flow execution. The Pipeline In pane shows the status of the pipeline before the the selected step runs. The Pipeline Out pane shows the status after the execution of the active step. (Usually) each step will do a trnsformation of the pipeline. The built in service tokenize separates a string (inString) by given delimiter (delim). Map the variable inCSV from the Pipeline In pane to inString of the tokenize service and lineDelimiter to delim by clicking and holding them with the mouse and dragging them to the target field. Yopu can also create a mapping by selecting both sides and then clicking the barbell shaped icon in the upper pane of the pipeline window. Drag a String List variable from the palette to the Pipeline Out pane and map valueList from the service ouput to it.
By default webMethods places every value into the piepline. As we do not need the input vaiables of the service itself anymore abd we mapped the ouput to a better name, we will remove them from the pipeline. Select inString, delim and valueList in the Pipeline Out pane and drop them by clicking the downward arrow in the upper pane of the pipeline window (inString and delim may not appear before you selected another step lateron and come back to this step).
We expect the columns names in the first line of the input. We will use the tokenize service again to separate the first line of the file into it's fields. Drag another tokenize service below the first one. Map fieldDelimiter to delim. Map the stringlist CSVLines to inString. To map only the first entry of CSVLines, select the map line by clicking on it. In the property window in the upper tright corner of Desginer click the 3 dots beside Indices. In the window, that pops up, enter 0 into the field Row Index of CSVLines. This will map the first enty of CSVLines to the input string. Click OK.
N.B.: The index used can also be a variable from the pipeline. To adress a pipeline (or global) variable just write the variabel name with %%, like %variablename%.
Although it is not really needed here it is best practise to initialize dynamically generated document structures as empta. This is for the case that lateron, by either reusing the code fragment or refactoring, this part is used in a loop or something similar. Drag a map step from the palette to the flow editor canvas. In the pipeline tab drag a Document List to the Pipeline Output pane and name it lineitem. Select lineitem and click the Set a value ... icon in the upper pane (shaped like (x)= ). Just click OK to set it empty.
Drag a Loop stepü from the palette into the flow editor canvas. Enter /CSVLines into the Input Array feld in the properties window. This will loop over all entries in this list.
Drag a Branch step the he canvas. Set Evaluate labels to true. This setting lets the branch calculate logical expressions from the single branches below.
Drag a Sequence to the canvas below the BRANCH. make sure, it is one level indented, If not, select it and use the Move Right button in the upper pane of the editor window. Sequences are mainly used to group commands together. As the first line contains the column labels we do not want to parse it again. In the Label field in the properties window enter %$iteration%>1. $iteration is a runtime vcariable automatically provided in a loop and holding the cound if the current iteration (starting by 0). You can also see it in the pipeline tables of the later steps.
As we are ignoring the first line we will have to count the lines we are really parsing for data. Drag a map below the Sequence and make sure it is indented again. Drag a String variable to the Pipeline Out pane and name it columnNo. Select columnNo, click the set Value icon ((x)=)and enter 0. Click ok.
We will now separatethe fields of the current line we are looping over. Drag another pub-string-tokenize to the canvas and mapCSVLines to inString and fieldDelimiter to inString. Observe that fieldLines is not a list anymore because we made it the inout list for our current loop. Drag a stringlist to the Piepline Out pane and name it Fieldcontent. Map valueList from the service output to it. Drop inString, delim and valueList from the Pipeline.
Add another loop below and enter /fieldcontent into the Inputr array field and /field into the output array field.
Drag and drop the document field from the Package Explorer to the Pipeline Out pane. This will create a document reference (You can do this also by dragging a document reference from the palette and then selecting the document from the dialogue). Expand the substructure of the document and map Fieldcontent to value and the list Fieldnames to name. Select the map to name by clicking on it and open the index dialogue by clicking on the three dots in the Indices fields. Write %columnNo% into the Row Index of fieldnames to get the name with the number of the current iteration.
Add another map and from the transformers dialogie in the upper pane of the pipeline window select pub.math:addints. From Pipeline In map columnNo in num1, set 1 as num2 and map the output value to columnNo in Pipeline Out.
Below the loop insert pub.document:documentLostToDocument (drag this from the WmPublic package in the package navigator). Make sure it is indented on the same level as the loop, so it is not part of the loop anymore. Map the document list field from Pipeline In to the input variable DocumentList of the service. Set the input variable name to name and value to value, as those are the fieldnames from our input document. Drag a document typre from the canvas to the Pipeline Out pane and map the output document from the Service Out to it. Drop the variables we do not need anymore: documentList, name, value and document.
We use the service appendToDocumentList to collect the object we generated in a list. Drag the service pub.list:appendToDocumentList to the canvas below the last service. From Pipeline In map the list lineitem to toList and object to fromItem. From ServiceOut map toList to lineItem. Drop toList, and fromItem from the Pipeline Out.
Drag a map below the last service and make sure it is not indentet but as lef tas possible in the canvas, so it is not part of the loop anymore. Map the list lineItem to the output variable lines. Drop all variables beside lines and inCSV.
Save the service.
Test it
To test, if it works correctly, rightclick the service in the Package Navigator and select Run As - Run Flow Service.
In the value column of inCSV click the 3 dots to open a bigger editor.
Insert testdata, like
a,b,c
1,2,3
10,20,30
Open the editor for lineDelimiter and just enter [Return]. Make sure to have the same line break like in the input for inCSV, especially if you paste the text from here!
Enter , (or whatever you choose as field delimiter) for fieldDelimiter and click OK. Observe the Results tab. This should show the generated document as wel las the inout paramters.
CSV2Json
Create a new service in the service folder called CSV2Json. Add 3 text fields named inCSV, lineDelimiter and fieldDelimiter to the input pane. Add outJsonString to the service output.
Drag the service CSV2Doc we wrote baove into the canvas and map the input fields from the current service to it's corresponants.
From WmPublix drag the service pub.json:documentToJSONString below and map the lines list from the previous step to the document of the Service In. Map jsonString from the service to outJsonString which we defined as output. Save your work.
CSV2XML
Create CSV2XML identical to CSV2Json but instead of adding pub.json:documentToJSONString use pub.xml:documentToXMLString and name the output variable outXMLString.
Test the services
Beside running the service from designer as we did above with CSV2Doc you can also invoke them via html url. To have this in one line we choose a character as line separator, so if we choose
csv: a,b,c;1,2,3;10,20,30
lineDelimiter: ;
fieldDelimiter,the call can be
for Json
http://localhost:5555/invoke/DemoCSVConverter.Services:CSV2Json?inCSV=a,b,c;1,2,3;10,20,30&lineDelimiter=;&fieldDelimiter=,
for XML
http://localhost:5555/invoke/DemoCSVConverter.Services:CSV2XML?inCSV=a,b,c;1,2,3;10,20,30&lineDelimiter=;&fieldDelimiter=,
Expose the service via Rest
We will first create a rest ressource t oexpose the service DemoCSVConverter.Services:CSV2Json via Rest and then a Rest API Descriptor (RAD), mainly to export the API description easily via swagger.
Create the Rest Ressource
Rightclick on the Rest folder. Select New - REST Ressource.
Select REST V2. Click Next.
Name it CSVParser. Click Next.
Select Empty Rest Ressource. Click Finish.
Click Add.
Click the 2 dots besdie Service Name to select the service CSV2Json and the Post method. Enter /CSV2JsonConverter as URL.
Save your work (by clicking the save symbolin the upper left corner or just [CTRL-S]).
Create the Rest API Descriptor
Right -click the Rest folder and select New - REST API Descriptor.
Name the descriptor CSV2JsonConverter_API. Click Next.
Select Provider an Existing REST - REST V2 Ressource.
Set the Title to CSV2JsonAPI. Select application/json for Consume and Produce. Click Next.
Select our Rest Ressource CSVParser. Click Finish.
Designer open a windows where you can inspect (or change) the AP descriptor.
Click the Swagger tab to inspect the swagger (this one is yaml format).
You can get the ´Swagger definitions direct via link:
http://localhost:5555/rad/DemoCSVConverter.Rest:CSV2JsonConverter_API?swagger.json
http://localhost:5555/rad/DemoCSVConverter.Rest:CSV2JsonConverter_API?swagger.yaml
As we didn't change the security settings you may need to log in (if you didn't already). To test the API open you rest client fo choice and select Import Swagger.
Past the url, e.g. for json:
http://localhost:5555/rad/DemoCSVConverter.Rest:CSV2JsonConverter_API?swagger.json
If prompted for it, enter username and password (Administrator, manage)
In the generated request add basic authentication and enter username and password again.
For the rest call we use a single character as linedelimiter, e.g.
inCSV: a,b,c#1,2,3#10,20,30
lineDelimiter: #
fieldDelimiter: ,
Test the request.
