How to return null values as empty strings in integration server designer

[Hamzeh_Muaz]

Product/components used and version/fix level: API GW + Integration server 10.15 + Designer 10.15

Question related to a free trial, or to a production (customer) instance? Production

Detailed explanation of the problem:
I made a flow that sends an API request, and in its response i get some null values. The integration server designer is dropping those values automatically, but I want to keep them as empty string “” or at least as ‘null’ in the json response. The “allow nulls” property is not working for me.

Example:
Original response would be something like > { “field1”: “value”, “field2”: “” }
After using the IS flow, the end response would be > { “field1”: “value” }
the “field2” that is empty is not being returned at all, but i want to keep it.

[srikanth.prathipati1803 (Srikanth Prathipati)]

We had a similar requirement. We used below Java service to address the same.
Below Java service takes inputs string list and returns results string list with null values.

IDataCursor pipelineCursor = pipeline.getCursor();
		String[]	inputs = IDataUtil.getStringArray( pipelineCursor, "inputs" );
		ArrayList<String> outputs = new ArrayList<String>();
		if(inputs == null){
			pipelineCursor.destroy();
			return;
		}
		for (String input: inputs){
			if (input == null || input.isEmpty()){
				outputs.add(null);
			}else{
				outputs.add(input);
			}
		}
		IDataUtil.put( pipelineCursor, "results", outputs.toArray(new String [inputs.length]) );
		pipelineCursor.destroy();

[Hamzeh_Muaz]

Thank you.

I’m going to use this for now.

I want to leave the thread open in case there is a different solution, specifically if it doesn’t use any code.

[reamon]

Can you share the specifics of the FLOW service that is making the call?

• Empty string and null are not the same thing.
• Designer is not dropping anything. Designer is not involved at run-time.
• What leads to the conclusion that the empty string is being dropped? How was it determined that the server is returning an empty string and IS is dropping it?
• Beware of parsing a JSON string and making assumptions about structure. (The Java service raises many questions.)

[Hamzeh_Muaz]

What kind of details would you like to know?

The flow service makes a SOAP request, it returns data such as field1 field2 field3 and so on. After some investigation I found the problem to be as follows, imagine this scenario:
The response is something like:

<xml>
	<field1></field1>
	<field2>asd</field2>
	<objectField>
		<field3></field3>
		<field4>fgh</field4>
	</objectField>
</xml>

I mapped these XML fields from the SOAP response to a JSON in my flow. fields 2 and 4 have no problem because they return data. Fields 1 and 3 are null, they are mapped to “null” in my JSON. such as:

{
	"field1": null,
	"field2": "asd",
	"field3": null,
	"field4": "fgh"
}

These fields are “null” not empty string “”
But that’s okay, for now.
Now if the SOAP XML response doesn’t return the “objectField” element or returns it as null, such as:

<xml>
	<field1></field1>
	<field2>asd</field2>
	<objectField></objectField>
</xml>

fields 3 and 4 are not returned in the JSON output at all, such as:

{
	"field1": null,
	"field2": "asd"
}

As you can see, field3 and field4 are not returned in the JSON response, because their parent object is null.
This causes issues, because the companies who use our APIs expect those fields to be returned, even if null. Causes exceptions on their end.

In conclusion, I want my response to always have the same fields, whether they have values or are null.

[reamon]

This is a good start and provides additional information but you’re still omitting key aspects. The original post didn’t mention XML at all. Now the follow up omits the details about the transition from XML to JSON. I imagine you’re using xmlStringToXMLNode and xmlNodeToDocument. And documentToJSONString. And in between those there is a MAP step or two.

I suspect the issue may be related to the XML that is being received. And I suspect it may be like this:

<xml>
	<field1 xsi:nil="true"></field1>
	<field2>asd</field2>
	<objectField>
		<field3 xsi:nil="true"></field3>
		<field4>fgh</field4>
	</objectField>
</xml>

And at run-time when the XML string is converted to an IS document you get something like this:

xml
..field1
....@nil: true
..field2: asd
..objectField
....field3
......@nil: true
....field4: fgh

In this case, field1 and field3 are documents, not strings. So when the map tries to copy from the document to a string, it cannot. So nothing is mapped.

If that is what is really happening, then the adjustment is relatively easy. But if something else is happening, we’d need to look at the details.

The key here is to share exactly what is being done. Don’t abstract away anything.

Side note: no Java will be needed for this.

[reamon]

@Hamzeh_Muaz

The integration server designer

FYI, there is no such thing as “Integration Server Designer.”. There is “Integration Server” which is the run-time environment (or “application server” if you prefer) that runs the code we put on it. There is “Designer” which is the development tool (it is Eclipse with plug-ins) where we write and debug code. There is no run-time here.

[engin_arlak (Engin SARLAK)]

I wish there were a simple answer and a solution to your problem. First, your assumption on XML and JSON are mutually exclusive is WRONG.

XML doesn’t have a null definition out of the box. In order to define a null in an XML document, you need to add property ‘nill’ to the tag and set its value to true as you can see in reamon’s example, but once you add that value to your tag then your object is no longer a simple type. You cannot map the value of a complex type to another simple type(actually you can but once you do it you will have a complex type, not a simple type hence it won’t be visible in designer unless you debug) and since the types won’t match you won’t see that complex type in your response.

If you inspect your nullable field from designer, you will see

supposed_simple_type
|- @nill
|- data

in your supposed simple string type object. In order to copy the value of this field you need to map the data field, but in order to do it your document must be generated properly (preferably from an XSD file) and it will still not be set to null even when your nill atribute is true.

As reamon said, if you include business logic/requirement for this task, we may recommend a better solution. Right now you are experiencing an xy problem.

As a starting point, assume everything should be a document in Integration Server. So If you want to generate an XML or JSON string, you need to have a document first. This document can be generated from a JSON or XML string as well as other types like flat file. But this doesn’t mean that you will get the same document from the JSON and XML object as long as you generate them from the same initial object.

If a field in a document has a null value, the string object you create from that document will have different values according to the type you are converting that in to. For JSON strings you will have a proper null values, for xml strings its highly likely that you won’t have nulls but you will have empty strings in your XML string after you convert. This is because by definition XML doesn’t have a null and it does not support it. Both of the solutions are workaround (one of them is nill attribute, the other is empty tag like </nullableField>.

tl;dr

XML object’s don’t support null values by definition hence we don’t have a way to interpret null values in XML files. If you have issues with null values when dealing with XML files, it’s so much more easier to not use them at all then implementing a workaround.

Try not to convert JSON to XML and vice versa. If you need to generate both as output, do it from the document(not from the generic one, define one if not defined), not from the string itself.

[Hamzeh_Muaz]

Honestly, i didn’t think its such a complex issue. I just want to assert an output field to always exist in the output even if its not given a value.

Here’s my full flow:

First, the “FlowStaticValues” is to map input fields to static values that I always use, such as credentials, URLs, and so on. Consider them as environment variables.
The “Map” afterwards is just to group the input fields so I can insert them as parameters in the next step.
The “InvokeGetPersonal” invokes a SOAP descriptor API. This calls an external API that retrieves information. This is a SOAP API that returns XML values. Its response is “MapSoapResponse”.
If the response HTTP status is 200, and the response exists, I go to the next step “MapGetPersonal”
“MapGetPersonal” is where I map the field from the XML to JSON. The input is a document reference to the WSDL soap descriptor. The output are the fields that I map to, which are strings and documents.

The image shows the mapping.
The field “GOVR_C” on the left is mapped to “cardGovCode” on the right.

“GOVR_C” string field on the left is inside an object element also called “GOVR_C”. If this object ‘parent’ field “GOVR_C” is null, then “cardGovCode” doesn’t show up in the output

[reamon]

@Hamzeh_Muaz

Honestly, i didn’t think its such a complex issue.

It isn’t complex really. But there are details to understand. wM IS does a lot of things for us but we still have to understand how IS parses XML to create the corresponding document var.

The screen shot is nice for showing what steps you’ve defined. What is not shown, and what may be the underlying issue, is what exactly is happening at run-time. The MAP step you shared shows what you’re expecting the document in the pipeline to look like. But the suspicion is that it does not look like that during execution as @engin_arlak and I have noted. If the response from the called endpoint is indicating that the q1:GOVR_C value is null by using the xsi:nil=“true” attribute, q1:GOVR_C will be a document, not a string.

Use the debugger to step through this and share a screen shot of what the document looks like in the pipeline. You might also be able to see/capture what the XML string that is returned from the SOAP call, though that may be hidden from view.

Side note: the use of clearPipeline in multiple places is a little suspect. Not a real problem per se but something to consider avoiding. Search for “clearPipeline” in the forums and you’ll find various discussions and viewpoints, including Use of ClearPipeline in the flow services. The guideline we follow is to explcitly drop vars as soon as they are no longer needed. Use clearPipeline judiciously and only to help protect against “bad” services that create “pipeline litter” – vars that are not declared as output.

[engin_arlak (Engin SARLAK)]

In addition to Reamon

@Hamzeh_Muaz

“MapGetPersonal” is where I map the field from the XML to JSON

Where do you do xml to json conversion? I don’t see anything related to that in your flow.

What you do in mapping service is map from document to document. There is nothing about XMLs or JSONs. You receive your data as XML but once you parse that XML, then it is no longer an XML, it is document.

As seen from the screenshot, you cannot get a null value for q1:GOVR_C because you don’t have data and @nil attributes in your field. This means the XSD (probably from WSDL) which you used to import this document is not defined properly. If it was nillable, it would have @nill attribute. But even if you had @nill attribute, you still wouldn’t have a null value at cardGovCode. You had to map conditionally using that @nill attribute to genearate the null value on the right side. If you don’t do this, then all you will have on the right side will be the same document which is an Object(will have @nill and data in the document, therefore when you convert that document to json, you will have attributes instead of a simple string) not a string.

Like I said, there is no easy way to handle nulls and persist them across different object notations. I would recommend setting everything to optional if they are null-able and establishing it as a ground rule across all of the integrations. Implementing any logic to null and making it a mandatory field doesn’t make much sense anyway. If it can be null, it can be missing or empty string as well; can’t it?

[Hamzeh_Muaz]

Solved it by giving the field an empty string as default value.
I “set value” to nothing and uncheck overwrite pipeline.

[reamon]

Reasonable approach, as long as empty string is okay/expected and the recipient is not expecting null. As noted, empty string.and null are not the same thing.