Receiving a json file as input. My requirement is s to convert all the field values to upper case. Any suggestions to do this at json string level or after converting to a document?
Hi @Warrior_S ,
There is no public service available to do this out of the box, you might want to write a custom java service and do the necessary conversion. And if the input is a json string, you could use the toUpper() to achieve this.
Thanks,
Sree
And if the input is a json string, you could use the toUpper() to achieve this.
@Sreekanth_Siddapur_Channakeshava But here the field names also will get converted to upper which I don’t want to. Any alternatives?
Hi
Can you clarify more on the input json ? I’m assuming that your json is mostly like a properties format with a key and a value, but what happens when the input json is of the format
{
"a":"b",
"c":
{
"c1" : "c2"
}
}
which is also a perfectly valid json. Is the requirement to find the recursive and find the last node, “b” and “c2” in this case?
-NP
I recommend doing this after the IS document is created from the JSON string. One should avoid manipulating incoming JSON strings (or XML, or delimited, etc.) directly, as a general rule.
As @Sreekanth_Siddapur_Channakeshava noted you can create a Java service to do this in a general way. Indeed, we have a service we named “normalizeDocument” that traverses a document and performs one or more changes to the document fields and values. In ours these changes are supported:
- Convert all field names to uppercase (could easily change values too–we just have never needed that)
- Convert non-string entities to strings; e.g. Long, Integer, Boolean etc. to string
- Remove all fields with a null value, including empty arrays
- Trim all string fields, and if the result ends with an empty string, remove it if the previous option is desired (except if in a string list)
- Convert child documents with @xsi:nul=“true” to a null object (instead of a document with the @xsi var)
The general approach is create a recursive method that iterates over every field in the document and do whatever tests apply. Here is some pseudo-code:
Get IDataCursor for the document
Get first field
while(more)
get field value as an Object
// order of tests is important
if value is null, remove field if desired
if instanceOf IData`[]`, loop over the list, recursively calling this method
if instanceOf IData, recursively call this method
if instanceOf String`[]`, loop over the array to trim, upper, lower, whatever
if instanceOf String, trim, upper, lower, whatever
if value is not null, convert object to string if desired
get next field
You’ll want to determine if this can be done “in place” or if the document should be cloned first.
We use this service quite a bit for JSON interactions since the current behavior of json*ToDocument is to create objects instead of strings – and we never want objects. :
HTH!
(EDIT) I’ll see if it will be okay if I share the Java service we have for this. It’s not very big nor complex
1 Like
@reamon, that’s why we will be introducing a new jsonToDocument service in 10.15, which will no longer generate objects by default, but assume everything is a string, unless a document type/schema says otherwise.
regards,
John.
Thanks @reamon can you please share the code.
@John_Carter4 when will be the 10.15 release date?
1 Like
@Warrior_S 10.15 is scheduled to be released by Oct-2022.
Thanks,
Sree
1 Like
@John_Carter4 - thanks for the reminder about this coming feature, as you noted in another thread. It will be a welcome one.
@Warrior_S as noted, will do so if I can.
Wanted to note too that the feature in 10.15 that @John_Carter4 mentioned doesn’t apply to your scenario – it pertains solely to how jsonToDocument converts a JSON string to an IS document and keeping data types such as number and boolean as strings instead of converting them to Java objects.
1 Like
Here is a Java service that does that and a number of other changes that some may find useful. Added the option to uppercase all field values.
/**
* No warranty. No support. Use at your own risk.
*
* Place this in whatever package works for you. We have our equivalent to WmPublic
* with a variety of general-purpose services. If you don't have such a package, would
* encourage that one be started.
*
* Comments tab:
Normalize a document by:
* Optionally convert all field names to uppercase
* Optionally convert all field values to uppercase
* Optionally convert non-string values to strings; e.g. Long, Integer, Boolean, etc. to string
- If fieldValuesToUpperCase is true, converts resulting string value to uppercase
* Optionally remove fields with a null value; includes empty arrays
* Optionally remove fields with an empty string value
* Optionally trimming all string fields; if the resulting value is empty,
the field will removed if removing fields with an empty string value is specified
* Optionally convert child documents with @xsi:nil="true" as an attribute to a null object; document
will be removed if removing fields with null value is specified; attribute prefix is assumed to be the
default "@" (can modify the service to add a parameter if something other than @ is ever needed)
* For string lists, only the trim and uppercase options apply; empty entries are not removed
The input document is unchanged if the cloneOption is "deep." It is cloned and the clone is normalized.
A deep clone is memory expensive so use "deep" only when absolutely necessary.
* Input on Input/Output tab:
document (Document)
cloneOption (String; picklist: none, shallow, deep)
fieldNamesToUpperCase (String: picklist: false, true)
fieldValuesToUpperCase (String: picklist: false, true)
objectsToStrings (String: picklist: false, true)
removeEmptyStringFields (String: picklist: false, true)
removeNullFields (String: picklist: false, true)
trimStringValues (String: picklist: false, true)
nilToNull (String: picklist: false, true)
* Output on Input/Output tab:
normalizedDocument (Document)
*/
public static final void normalizeDocument(IData pipeline) throws ServiceException {
IDataCursor idc = pipeline.getCursor();
IData document = IDataUtil.getIData(idc, "document");
String cloneOption = IDataUtil.getString(idc, "cloneOption");
boolean fieldNamesToUpperCase = IDataUtil.getBoolean(idc, "fieldNamesToUpperCase"); // defaults to false
boolean fieldValuesToUpperCase = IDataUtil.getBoolean(idc, "fieldValuesToUpperCase");
boolean objectsToStrings = IDataUtil.getBoolean(idc, "objectsToStrings");
boolean removeNullFields = IDataUtil.getBoolean(idc, "removeNullFields");
boolean removeEmptyStringFields = IDataUtil.getBoolean(idc, "removeEmptyStringFields");
boolean trimStringValues = IDataUtil.getBoolean(idc, "trimStringValues");
boolean nilToNull = IDataUtil.getBoolean(idc, "nilToNull");
int opt = 0;
if("shallow".equals(cloneOption))
opt = 1;
else if("deep".equals(cloneOption))
opt = 2;
try {
if (document != null) {
IData nDocument;
nDocument = (opt == 2 ? IDataUtil.deepClone(document)
: (opt == 1 ? IDataUtil.clone(document) : document));
normalizeDocument(nDocument,
fieldNamesToUpperCase, fieldValuesToUpperCase,
objectsToStrings, removeEmptyStringFields,
removeNullFields, trimStringValues, nilToNull);
IDataUtil.put(idc, "normalizedDocument", nDocument);
}
} catch (IOException e) {
throw new ServiceException(e);
} finally {
idc.destroy();
}
}
// --- <<IS-BEGIN-SHARED-SOURCE-AREA>> ---
private static boolean isNil(IData doc)
{
boolean isNil = false;
IDataCursor idc = doc.getCursor();
boolean moreData = idc.first();
while (moreData)
{
// Constraint: XML schema rules allow an element
// with xsi:nil="true" to have other attributes.
// Loop through all elements to see if all children
// are attributes. If not, then isNil cannot be true.
String key = idc.getKey();
if(key.startsWith("@"))
{
// Check if the child is "@xsi:nil" (allowing for any prefix, not just xsi)
if(idc.getKey().matches("@.+:nil") && ("true".equals(idc.getValue())))
isNil = true;
} else {
isNil = false; // Child element without the attribute prefix
break;
}
moreData = idc.next();
}
idc.destroy();
return isNil;
}
private static void normalizeDocument(IData doc, boolean fieldNamesToUpperCase, boolean fieldValuesToUpperCase, boolean objectsToStrings, boolean removeEmptyStringFields, boolean removeNullFields, boolean trimStringValues, boolean nilToNull)
throws ServiceException {
boolean moreData;
IDataCursor idc = doc.getCursor();
moreData = idc.first();
while (moreData) {
boolean deleteField = false;
if (removeNullFields && idc.getValue() == null) {
deleteField = true;
} else {
if (fieldNamesToUpperCase) {
idc.setKey((String) (idc.getKey()).toUpperCase());
}
Object theValue = idc.getValue();
if (theValue instanceof com.wm.data.IData[]) { // evaluates to false if theValue is null
IData[] currentDocList = (IData[]) theValue;
for (int i = 0; i < currentDocList.length; i++) {
normalizeDocument(currentDocList[i],
fieldNamesToUpperCase, fieldValuesToUpperCase,
objectsToStrings, removeEmptyStringFields,
removeNullFields, trimStringValues, nilToNull);
}
} else if (theValue instanceof com.wm.data.IData) {
if(nilToNull && isNil((IData) theValue))
{
if(removeNullFields)
deleteField = true;
else
idc.setValue(null);
} else {
normalizeDocument((IData) theValue,
fieldNamesToUpperCase, fieldValuesToUpperCase,
objectsToStrings, removeEmptyStringFields,
removeNullFields, trimStringValues, nilToNull);
}
} else if (theValue instanceof String[]) {
if(trimStringValues || fieldValuesToUpperCase) {
String[] list = (String[]) theValue;
for (int i = 0; i < list.length; i++) {
if(trimStringValues) list[i] = list[i].trim();
if(fieldValuesToUpperCase) list[i] = list[i].toUpperCase();
}
}
} else if (theValue instanceof java.lang.String) {
String s = (String) theValue;
if (trimStringValues || fieldValuesToUpperCase) {
if(trimStringValues) s = s.trim();
if(fieldValuesToUpperCase) s = s.toUpperCase();
idc.setValue(s);
}
if (removeEmptyStringFields && (s.length() == 0)) {
deleteField = true;
}
} else if ((theValue != null) && theValue.getClass().isArray()) {
if(removeNullFields && (java.lang.reflect.Array.getLength(theValue) == 0) ) {
deleteField = true;
}
} else if ((theValue != null) && objectsToStrings) {
String s = theValue.toString();
if(fieldValuesToUpperCase) s = s.toUpperCase();
idc.setValue(s);
}
}
// Both methods move the cursor
if (deleteField)
moreData = idc.delete();
else
moreData = idc.next();
}
idc.destroy();
}
// --- <<IS-END-SHARED-SOURCE-AREA>> ---
2 Likes
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.