How to create loop within loop within loop to read document and grab a key to get the value using Webmethods

Hey Reamon,
Sorry for the delayed response. Stuck on new issue now. trying to work through that.

To answer your question hence basically it is a standard, what code is coming from backend it has to be converted into code that is understood by users. Basically they are already defined, when we do conversion, we just set the closest one that almost or nearly matches the backend code or if we don’t have one, we define one. Lets say something like, the most famous one null pointer exception, if we get that from backend, we want to convert that into something that user can understand. Thats just an example or something like bad request.

Also I found out it will be running on-premise server.
I will look into lookup-table on how to best use of that.

Reamon I am working on removing all the hard code values and going for the approach to use properties file like application.properties.
Do you know what I can use/in-built service to call properties and grab values using keys.
Any idea what should be the approach.
I am thinking I will create a property file, I dont know WM supports what type of properties yet. I am doing some research now.
Perhaps if you have any idea, please let me know.
Also I approach will be to grab values based on Environment, like get local value if I am running locally, dev value if I am running in dev, production value if running in production and so on.

Appreciate you for all your buttress. Thx

I found two ways:

first method:(perfect solution for env based values)
image
on IS server
grab values uisng
image

second method:(perfect solution for just local)
image
this help you grab it from your C drive

1 Like

Hi Sufran,

additionally you might want to check out for the Global Variables feature available since late wM 9.x versions.

Regards,
Holger

Thanks for the additional info. I’m still concerned that converting the errors is more than needed. When you say “users” to you mean people using an application/browser UI? If so, are your wM IS-hosted components formally a part of that application? Usually wM IS-host components are not. They are “backend” type things. The application itself should translate as needed to get “user-friendly” error messages, not the wM IS items. The wM IS audience is usually developers – API client. But I think we’ve probably beat this specific item to death. :slight_smile:

For configuration settings:

  • Use the the extended settings in IS extremely sparingly. That is not really for our use. In our environment, we have only a couple of custom items in extended settings. And only because there was no “Global Variables” facility at the time we started. Use Global Variables, not extended settings.
  • Holger noted the Global Variables feature. A good facility for the right types of config. Don’t go crazy with this either since it is global. Easy to get to a point where people are afraid to change anything as they don’t know what it might break. Need naming and usage discipline.
  • External files are the primary way to go. Here is a common convention:
  1. Define the files in a common location. We use /instances//properties.
  2. Do NOT place the files in the package directory. It is very tempting to place the config file in the package config or pub directory but you’ll likely regret doing so later. Make the deployment of packages and config files separate. That way, an updated package deployment won’t clobber any config file changes that may have been made in a specific environment.
  3. The config file content can be whatever you’re most comfortable with. Classic properties file, loadable with java.util.Properties. An XML file, which is what we use. A JSON file. Or possibly an .ini file, which is conceptually a properties file with the added notion of sections (probably not a strong choice). Use a file naming scheme that makes sense for how things are structured. Package based perhaps, depending upon the scope of your packages.
  4. Define a document type that corresponds to the structure/elements in the file.
  5. Define a common service that can be used for any arbitrary config file (for a given format). Various packages can call this to get their config. The service returns a document, with a structure matching the document type defined.

Here is a simplified set of steps for loading an XML file:

image

Basically the same steps when processing any XML. Just loading it from a local file. Be sure to pass in a document type name if the config file will have repeating elements.

For JSON, quite similar but use jsonStringToDocument instead of xmlStringToXMLNode/xmlNodeToDocument.

For properties, create a Java service to read the properties and convert each property to a field in an IS document. Here is an example which supports a caller passing a stream, bytes, string or filename and returns an un-typed document which the caller can map to a document var that has a document type reference. The key for that is the doc type definition and the file content need to match.

public static final void propertiesToDocument(IData pipeline) throws ServiceException {
	IDataCursor idc = pipeline.getCursor();
	java.io.InputStream stream = (java.io.InputStream)IDataUtil.get(idc, "stream");
	byte[] bytes = (byte[])IDataUtil.get(idc, "bytes");
	String s = IDataUtil.getString(idc, "string");
	String filename = IDataUtil.getString(idc, "filename");
	boolean closeStream = false;
	idc.destroy();		
	
	// Figure out which input was passed.
	if(stream == null)
	{
		if(bytes == null)
		{
			if(s == null)
			{
				if(filename != null && filename.length() > 0)
				{
					try {
						stream = new java.io.BufferedInputStream(new java.io.FileInputStream(filename));
						closeStream = true;
					} catch (java.io.FileNotFoundException fnf) {
						throw new ServiceException(fnf);
					}
				} else {
					return; // Nothing to convert; return nothing
				}
			} else {
				stream = new java.io.ByteArrayInputStream(s.getBytes());
			}
		} else {
			stream = new java.io.ByteArrayInputStream(bytes);
		}
	}
	
	// At this point we have a stream (bytes, string converted to a stream)
	java.util.Properties props = new java.util.Properties();
	
	try {
		props.load(stream);
	} catch (java.io.IOException ioe) {
		throw new ServiceException(ioe);      
	} finally {
		if(closeStream)
		{
			try {
				stream.close();
			} catch (java.io.IOException ignored) { }
		}
	}
	
	IData document = IDataFactory.create();
	final IDataCursor dc = document.getCursor();
	props.forEach((key, value) -> {
		IDataUtil.put(dc, (String)key, (String)value);
	});
	dc.destroy();
	
	idc = pipeline.getCursor();
	IDataUtil.put(idc, "document", document);
	idc.destroy();	
}

HTH!

1 Like

Hi Rob,

thanks for your update.

Addendum:
We are using a mix of classic properties files and XML config files, which are located in dedicated subdirectories of IntegrationServer/instances//config folder.
This has the benefit, that these will be backed up during successful IS restart by the internal logic to config/backup subdirectory.

Access to global variables should be able by built-in Services documented in the Built-In-Services Reference similar to pub.utils:getServerProperty.

Regards,
Holger

Hey Holger,
Thanks Holger Global variables eliminated several lines of in built service uses. Worked flawlessly. Thx much

Just in case,
Anyone who wants to know how to create one
Go on IS server and under settings and click
image
image
add one entry
Then you can call it
Double click on any field you want to set the value to
image
check this box image
this tells you have some global variable set to this
image

Hey Reamon,
First thing first, conversions of default error is imperative to provide user friendly code mainly for understanding purpose. As you already can tell these default error code in webMethods are not user friendly at all. From developers perspective not a biggy but from users point of view a little perhaps. So no way out of this but to add some customization to these default errors thrown when failure happens at Schema validation level. Now imagine as I already mention I began to work on the exception thrown part when and if there is any 4xx to 5xx thrown. There are default exception thrown. So I have to customize them as well. Any thoughts on that?
Ton of work to be done here. No time to sleep

Then regarding the UI, I think I misled you a little on that. This orchestration will be build as API/Service rather and will be part of another layer that will be called.
The idea is it will be like a rest service that is internally can call either DB or Web service(Soap/REST). Sky is the limit in this case.

Everything is tentative at this time but approach is just like creating a Spring boot app.

Now going towards using properties like Spring boot we have externalized properties.
So I went with this approach
image
image
image

And then came Global variables and you can see how many lines I have disabled by just using Global variables. You can say I am in love with Global variables now.
But idea behind this is to initialize values at start up, convention or pick up some values down the line when doing further processing in code. FX: DB user/password/driver etc. Global are perfect for these situations. But I do want to use external files in parallel. However there is slight problem I am coming across with this approach.
So check this, I go on IS server
image
set a key value pair
image
Key is something like this
watt.custom.path=C:\Home\users\test.txt
Now what I do is use this
image
in built service. It takes key and return value
image
Then I use this to call the file
image
and you can image the rest.
Now the problem is I will be passing the key as parameter because it will called through rest service endpoint, So I need to use this first
image
Now this gives me parameter directly which is basically my key(watt.custom.path) and I map it to this
image
and I get key back but now the problem is here
I get the path back with double back slashes
C:\\Home\\users\\test.txt
It wasnt happening when I wasn’t using this
image
Its weird.
I tried using this in built service
image
no effect.
This defeats the endeavor here for external files calling.
So I will try second approach you have provided with code
create a Java service to read the properties.
Lets see but I want to use the external file mechanism as well to be more limber. Like you said dont go heavy on Global variables.
I am glad you here mate to help me out.

Much RESPECT.

Hi,

it turned out as a best practice to specify all pathes in wM with simple slashes (‘/’) regardless of OS.

Global Variables are not always the best choice.
They apply only to those keys which are environment specific, but static during runtime.

For more dynamic keys properties files and/or XML-based config files might be the better choice.
If you do not want to read these files from file system every time they are accessed, you can choose the built in caching functionaliy. But make sure to set the cache expiration to a reasonable time so they are refreshed regularly.

Regards,
Holger

1 Like

Thanks Holger. Forward slashes worked even though I have windows 10 OS running. Its weird with when its Linux way.
I will try cache functionality as well. Thx

It depends on the “user” audience. For APIs, the audience is developers, not app users. Messages for app users is the domain of the app, not the API.

For the 4xx, 5xx HTTP status codes, what you do depends upon what the server being called returns. Even for 2xx status you may need to do something more than just check the code. The HTTP body may contain additional information, of course depending upon what the server does in various scenarios. Some will return 200 but indicate an error in the HTTP body.

Those lines should be in a separate service. A service in your “public” package that can be used by any number of integration-specific packages. With that, you’d then have a one-liner in your “do-the-work” services to load configuration for that service. Don’t overuse global vars.

DB user/pw/driver will be configured in the JDBC adapter connection pool, not in any custom config file or global vars. You’re using the JDBC adapter I assume? Hopefully not Java services.

You’ve mentioned pub.flow:getTransportInfo a couple of times. May I ask what you’re using that for? If it is to retrieve the URL parameters, it is not necessary to use that service. The URL parameters are already in the pipeline when the service is invoked via HTTP.

Be careful with this. The caching facility captures the entire pipeline at the time. So if you’re 10-20 steps deep into a service and call a service that has caching, it will capture everything. Then the next run, that cache will overwrite everything in the pipeline – likely causing undesired behavior.

Use caching sparingly (don’t optimize prematurely) and when it is used, use the scope feature to limit what the cache contains – so that it caches only the inputs/outputs of the service being cached.

Hi,

or invoke the service being cached as a transformer as this will only capture the exact input signature of the transformer invocation and not the full outer pipeline.

Regards,
Holger

Hey Reamon,

So what are you recommending, Shall I just keep the default error messages when doing validations on request schema coming in?

On 4/5xx errors, I havent started working on it yet but almost there to start working on them. Because I do have to provide custom response when there is success/failure. Stay tune on that

For Global variables I decided I will be using it mainly in Environment base like Holger mention. JDBC was just an example. However I found another option. What if I want to read some property file, mainly with properties extension
//
String user = null;
String pass = null;

	// pipeline
	IDataCursor pipelineCursor = pipeline.getCursor();
		String	FileName = IDataUtil.getString( pipelineCursor, "FileName" );
		String	getUser = IDataUtil.getString( pipelineCursor, "getUser" );
		String	getPass = IDataUtil.getString( pipelineCursor, "getPass" );
		
				try{
					   // read in the file stream and convert it to a properties object
					          FileInputStream configFileInputStream = new FileInputStream(FileName);
					  
					   // properties object is already defined in the Shared code
					   Properties properties = new Properties();
					   properties.load(configFileInputStream);
					   
					   user = properties.getProperty(getUser);
					   pass = properties.getProperty(getPass);
				
					      }catch ( FileNotFoundException e ){
					          // throw an error
					   throw new ServiceException("ERROR FINDING PROPERTY FILE: "+e);
					      }catch ( IOException e ){
					          //throw an error
					   throw new ServiceException("Error reading "+FileName+" property file: "+e);
					      }catch ( Exception e ){
					          //throw an error
					   throw new ServiceException("Error while registering "+FileName+" property file: "+e);
					      }
				
				
				pipelineCursor.destroy();
		
	pipelineCursor.destroy();
	
	// pipeline
	IDataCursor pipelineCursor_1 = pipeline.getCursor();
	IDataUtil.put( pipelineCursor_1, "user", user );
	IDataUtil.put( pipelineCursor_1, "pass", pass );
	pipelineCursor_1.destroy();

This will help me read properties key/value pair values from .properties file.

Now I am in the process of reading a XML file. Not fairly there yet. Stay tune.
So its either .properties or .xml. There are the two files we mainly read from.

The reason I am using this pub.flow:getTransportInfo is to
get Custom required headers value can be multiple custom header fields
get Authorization field
get JSON request body

These are the three things I retrieve from using this pub.flow:getTransportInfo mainly.

With what you mention about caching, It seems like a problem. I will hold off on that. I think Global variable and the code I mention above to read .properties file will suffice my need for now. Only thing left is XML file which is work in progress.

Thx

I recommend that you don’t do validations and let the “backend” worry about it. :slight_smile: Have wM IS pass the data to the JavaEE components and then return whatever they respond with – data response or errors.

But I suspect you feel the JavaEE components need to be “protected” for whatever reason, so the next best thing would be, yes, just keep the validation messages as they are. It is unlikely that mapping them to other values is a meaningful improvement.

Hmm. Is the code I shared earlier insufficient? It will read any string, byte array, stream or file that contain properties (as defined/supported by java.util.Properties) and return a document containing fields that are named exactly as they exist in the properties file. This is a general purpose way of getting config/properties into an IS document. Other services can call it passing whatever they have (usually a filename) and get back an IS doc that has all the settings. The IS doc can be described by an IS document type - the caller of the service would map the output document to a new var that is a doc reference to the doc type (think of this as type casting) so they can easily use the config fields and values. It is not necessary for the caller to pass the field/properties names. The service I shared gets them all.

Here again, the steps shared earlier do this.

Get headers – good.
Get Authorization field - good (though you should research to see if built-in components can interpret/manage this for you so you don’t have to do anything explicit.
Get JSON request - not so good. You cannot get the HTTP body from there. And you don’t need to. It is already in the pipeline. Also, you can get the raw URL string from there (http/requestUrl) but you don’t need to. The HTTP body (if present) and the URL parameters will already be in the pipeline when your service runs. IS content handlers do the work for you.

For the HTTP body, the variable that is present will depend upon the Content-Type. For JSON, application/json, IS will place a var the name of which depends on the setting of watt.server.http.jsonFormat in IS Administrator | Extended Settings.

When Content-Type=application/json the variable in the pipeline is controlled by two things:
* watt.server.http.jsonFormat setting
* the jsonFormat parameter in the URL (typically will not use this though)
- when jsonFormat is:
* parsed: document with a name matching the name in the JSON – e.g. JSON converted into an IS document
* stream: jsonStream
* bytes: jsonBytes

I recommend setting watt.server.http.jsonFormat to stream so that any integration that may need to handle large JSON payloads can potentially do so. If set to other values, it makes it more difficult to support when a service needs the data in a stream (and not all in memory at once). The var is usable in your service right away. Just define it on the Input tab. At runtime, IS will populate it. Then you can do whatever you need with it – convert it to a JSON string, parse to an IS document, pass it on to something else, etc.

While this covers JSON, the same approach can be used for other types like XML and delimited files. IS can be configured to “help” as much as you want to hand the data to your service.

For the URL parameters, IS parses those and places each in the pipeline. All you need to do is declare them on the input tab of your service so that the FLOW steps “know” what to vars to expect and handle them as desired.

Using getTransportInfo has the drawback of making debugging more difficult. When debugging, it will not return the same elements that are returned when the service is invoked via HTTP. There are techniques to deal with that, but we’ll leave that for another time.

Another side trip – security folks are likely to object to storing passwords in any text file (XML, JSON, properties). You can leverage the services in pub.security.outboundPasswords to store passwords used to access other systems in a secure manner. If you’re really ambitious, you can create an IS Adminsitrator-based facility to allow developers to manage passwords stored there. Then services would use pub.security.outboundPasswords services to get them when needed. Including, possibly, your general-purpose “getProperties” service.

HTH.

Hey Reamon,

Will defiintely use watt.server.http.jsonFormat instead of specifying this Content-Type = application/json in every request as headers
image

Now regarding your code. Its perfect nothing wrong with it but just fail to understand the last part from looking at it

IData document = IDataFactory.create();
** final IDataCursor dc = document.getCursor();**
** props.forEach((key, value) → {**
** IDataUtil.put(dc, (String)key, (String)value);**
** });**

foreach loop?

I wanted to keep this precise and simple and when I say precise just for .properties file reading.
because .properties is current/popular.
XML is becoming stale and antiquated. So I dont want to mix the two.

For XML I am using this approach
image

Then I am creating a custom Java service and grab the whole xml document and grab whatever value I want to grab. An example of xml would be something like Logback in spring boot.

As for what you mention on validation, I think you are right. I am just overdoing things. I will leave the validation on incoming request and expect a perfect data but a small caveat, thats not always going to be the case. We have to be a bit limber and cautious when it comes to malicious data.

Everyday there is a new challenge I am facing. Have to take this one day at a time
Thx

Actually, the Content-Type header must be specified every time. The jsonFormat URL parameter can be used by clients to override the watt.server.http.jsonFormat setting for a given call. Example: https://myserver.abc.com/rest/MyEndpoint/foo?jsonFormat=bytes will cause wM IS to place a byte array named jsonBytes var in the pipeline regardless of the watt.server.http.jsonFormat setting. Whether or not your service supports that is up to you. The Content-Type: application/json header is critical – without it, wM IS won’t provide the JSON vars to your service. (It will provide something else, but that’s a different topic for a different time.)

For the code that loads properties into an IS document: The forEach loop is iterating over all the keys in the properties object, getting the value of each and then using the properties key and value to place in the IS document. Consider this properties example:

website = https://en.wikipedia.org/
language = English
message = Welcome to \
          Wikipedia!

This can be loaded from a file, put into a hard-coded string, etc. Pass any of those to the propertiesToDocument() method I shared earlier and the result will be this in the pipeline (using dots to note they are child elements of document):

document
…website
…language
…message

Then the caller can refer to and use any of those 3 variables in the document directly. Use in a MAP step, pass to a sevice in INVOKE, etc. You can define a document type that has the fields/structure that matches the fields in the properties file.

Why? The steps have converted it to an IS document. You can reference any/all of the fields from the XML in that document. There is no need to use Java to retrieve a particular value. The whole point of converting the XML to an IS document is to be able to easily access those fields. Here are some more details for what to do:

First, an example XML file to use for config.

<?xml version="1.0"?>
<properties>
  <ignoreCredentials>false</ignoreCredentials>
  <skipVerifySCID>000000</skipVerifySCID>
  <issuerCertAliasList>
    <alias>some-issuer</alias>
    <alias>other-issuer_old</alias>
  </issuerCertAliasList>
  <trustStoreAlias>DEVTrustStore</trustStoreAlias>
  <updateIdExpansionPrefix>zOS/RSU/</updateIdExpansionPrefix>
  <updateOrderURITemplate>https://myHost/abcd/UpdateOrder?orderid=%/orderId%</updateOrderURITemplate>
  <orderidPattern>R[A-Z0-9]{9}</orderidPattern>
</properties>

Based upon that, we have defined a document type:

image

With these 2 items, we call the common service we have to load the file the contents of which are now in a document variable. The “getPropertiesEx” is our service that does the getFile, xmlStringToXMLNode, etc. steps.

Later in that service, we simply map the fields that are needed for the processing we want to do:

There are no Java services in any of that. Not needed.

You’d do a similar thing if you want to use a properties file (or JSON) instead of XML. Though for properties you’d use the Java code shared to do the conversion from the properties object to the IS document.

That’s an interesting perception since properties have been around a bit longer than XML. :slight_smile: XML has a variety of things that make it less desirable, but I don’t think age is one of them.

You could also use .ini files if the notion of sections would be useful. The Java code to parse and convert those to an IS document is a bit different but readily doable.

Don’t expect perfect data. The main point I was trying to make is that the J2EE components that you’re passing the data to should already be checking for bad/malicious data – and if they are, you don’t need to do so again.

Hey Reamon,

Let me see if I understood you correctly.
getPropertiesEx it is your own Java custom service where you are performing all these steps in one right?
steps:
getFile → xmlStringToXMLNode → xmlNodeToDocument

I meant XML is the language format of retro but now JSON is the present/now/popular.

Reamon I have another question, that I am working on.
I am using external Jar, That external Jar has some dependency FX:
image
image

Now to use any external Jars it has to be in this path for you to import and compile the code to call the Jar code
C:\SoftwareAG\IntegrationServer\lib\jars\custom

Now the problem came any dependency you have on that Jar, It has to be also included as well inside this path
C:\SoftwareAG\IntegrationServer\lib\jars\custom
otherwise when you run, it throws exception on dependencies.
So that was the solution.
But is there something I can do better. Like a pom file concept.
As you can see I have to look for these Jars on google, Download them/unzip and then copy/paste in this folder.
Now imagine this is just few dependencies I had included in the JAR but what if I had dependencies within dependencies within dependencies. Then I will have to look for all these Jars Download them/unzip and then copy/paste in this folder.
That’s tedious/tiresome work.
Do you have any experience with this.
We will convert all our projects from JAVA to Maven Project in webMethods.
Any thoughts?

Thank you very much

Yes, it is a custom service performing all those steps. It is not Java. It is a FLOW service.

There will always be a “new” and “better” markup language. Don’t be too concerned about that. Use a format for your config files that makes sense for your environment/team. Properties are fine. JSON is fine. XML is fine. .ini files are fine. Storing in a DB is fine. For any of them the concept is the same: store the settings in a file/table(s), load them into an IS document to be used. From a service perspective that uses the config in the IS document, it does not know nor care what the underlying storage format is.

Again, lot’s of things to criticize about XML. Age and “retro” should not be one of them. :slight_smile:

The primary suggestion I have regarding external libraries – be very conservative. Don’t try to use wM IS the same way you would code/use JBoss, WLS, WAS, etc. The run-time environment is Java but this is not a Java development platform. One can “drop down” to Java to create focused utility methods but it is not intended for “significant” Java development. Do that elsewhere if it is needed. Don’t try to load all the typical libraries used when creating typical Java apps. If you keep trying to use wM IS in the same way you do Java app development, you’re going to be disappointed and frustrated.

Review the About page in IS Administrator. You may be pleasantly surprised that the library you want to use is already present. Here is a short list of ours from 10.5 (trimmed to show just the significant part of the end of the path):

common/lib/ext/commons-beanutils.jar
common/lib/ext/commons-codec.jar
common/lib/ext/commons-collections.jar
common/lib/ext/commons-digester.jar
common/lib/ext/commons-discovery.jar
common/lib/ext/commons-fileupload.jar
common/lib/ext/commons-httpclient.jar
common/lib/ext/commons-io.jar
common/lib/ext/commons-jexl3.jar
common/lib/ext/commons-lang.jar
common/lib/ext/commons-lang3.jar
common/lib/ext/commons-logging.jar
common/lib/ext/jackson-annotations.jar
common/lib/ext/jackson-core.jar
common/lib/ext/jackson-databind.jar
common/lib/ext/jackson-dataformat-yaml.jar
common/lib/ext/spring-core.jar
common/lib/ext/swagger-annotations.jar
common/lib/ext/swagger-core.jar
common/lib/ext/swagger-models.jar
common/lib/ext/swagger-parser.jar

That said, there are times when libraries are needed. You’ll want to get the minimum necessary (ideally, 1 for a given function/need) to limit the potentially tedious work of importing a pile of dependencies. If you find a deep list of dependencies that end up “importing the world” you’ll likely want to look for another library with less baggage.

The number of third-party JARs we’ve used over the years is like 2. One for SSH (which has been mostly replaced by built-in facilities for SFTP now) and one for PGP encryption (BouncyCastle – which is also now part of the core set of libs). We considered adding Jackson at some point but decided against it – and now it is part of the core list too. Often times there are useful functions available in Apache Commons. But more often than not, the relatively narrow function we wanted required “import the world” so we either looked elsewhere or created our own using Apache code as a guide. And as can be seen in the above abbreviated list, many of them are now included.

Be cautious here. This indicates a potential mismatch of how to use wM IS as an integration tool (not an app dev tool) vs how Java development is done. Referencing an old post again that cautions about being too Java-focused.

Integration Server and Java - #22 by reamon

1 Like

Wait Reamon I am doing the same thing right?
image

I am confused now?

Yes, those are the right steps for your own getPropertiesEx service (named however you’d like).

My comment was only noting that getPropertiesEx is a FLOW service, not a Java service.