Question about Decoder Microservice and DecoderResult

[dst (Daniel Steiner)]

Product/components used and version/fix level are you on:

Cumulocity 1015.0.309

Detailed explanation of the problem:

I have implemented a decoder microservice in Java that get called by an lwm2m custom action, queries an external API and then returns a DecoderResult object containing an Event with custom fragments. All seems to work fine, except that the custom fragments are missing in the created events. I used the following project as reference which suggests that custom fragments in events should be possible:

cumulocity-examples/sample-lwm2m-custom-decoder at develop ·...

Is this a known issue or am i doing something wrong here?

Example

Decoder service log output showing DecoderResult:

{"alarms":null,"dataFragments":null,"message":null,"success":true,"alarmTypesToUpdate":null,"events":[{"source":{"id":"7660465"},"type":"c8y_LocationUpdate","time":"2023-05-24T08:39:09.683+02:00","text":"Location update","c8y_Position":{"alt":null,"lng":11.56999804,"uncertainty":2342,"fulfilledWith":"WIFI","lat":44.12599975}}],"measurements":null}

Resulting cumulocity event:

{
	"creationTime": "2023-05-24T06:39:09.794Z",
	"source": {
		"name": "My LwM2M Device",
		"self": "https://TENANT_URL/inventory/managedObjects/7660465",
		"id": "7660465"
	},
	"type": "c8y_LocationUpdate",
	"lastUpdated": "2023-05-24T06:39:09.794Z",
	"self": "https://TENANT_URL/event/events/9365743",
	"time": "2023-05-24T08:39:09.683+02:00",
	"id": "9365743",
	"text": "Location update"
}

As can be seen the “c8y_Position” custom fragment is missing in the created event. If i take the JSON of the event produced by my microservice and create the event manually via a POST to the “/event/events” endpoint the custom fragments are preserved.

Is your question related to the free trial, or to a production (customer) instance?

Production instance

[Kai_Sieben]

Hi Daniel and warm welcome to the Tech Community Forums

As you already noted with a manual API call the fragments are preserved.
Hence in general this works of course, as any custom fragments can be added to an event.

Could you share details of your microservice and how it is doing the API call to update the event?
Are you using Java and the Microservice SDK?

Another smaller remark:
As the “c8y_Position” fragment is used by Cumulocity I would recommend to put any real custom attributes - like uncertainty and fulfilledWith - into a separate fragment, e.g. “custom_Position”.
This would separate your real custom fragments to those known to Cumulocity and can prevent any confusion later on.

"c8y_Position":{"lng":11.56999804,"lat":44.12599975},
"custom_Position": {"uncertainty":2342,"fulfilledWith":"WIFI"}

Regards Kai

[dst (Daniel Steiner)]

Hi Kai,

Thanks for the quick reply!

Yes i am using Java and the microservice SDK, i took the sample-lwm2m-custom-decoder as a reference. My current (working) solution is to not rely on the event creation by the platform from my DecoderResult “events” array but to POST the Event JSON (in the same format i wrote it to the DecoderResult before) directly to the /event/events endpoint, and just returning { “success”: true } as my DecoderResult.

I also tried to add a different custom fragment to my DecoderResult event (eg. “decoderCustomFragment”: true ), this also does not propagate to the resulting cumulocity event. Only the values of the properties “type”, “time” and “text” are present.

In the readme for the “sample-lwm2m-custom-decoder” under " Full decoder response sample" a custom fragment is shown as being part of a valid response, so i assumed this should work:

"events": [
{
    "source": {
        "id":"12345" },
    "type": "TestEvent 1",
    "text": "Data was decoded",
    "time": "2020-03-03T12:03:12.845Z",
    "myFragment": "my data"
},
...
]

Regards,
Daniel

[Stefan_Witschel]

Hi @dst ,

can you post a code snippet how you are adding the custom fragment to the eventRepresentation?
I think there is something wrong and we can correct that together when you post your code snippet.

[dst (Daniel Steiner)]

Hi @Stefan_Witschel

Sure here is a code excerpt for the DecoderResult creation:

			C8yPosition c8y_Position = new C8yPosition();
			c8y_Position.setLat( groundFixResponse.getLat() );
			c8y_Position.setLng( groundFixResponse.getLon() );
			c8y_Position.setUncertainty( groundFixResponse.getUncertainty() );
			c8y_Position.setFulfilledWith( groundFixResponse.getFulfilledWith() );

	        var source = new ManagedObjectRepresentation();
	        source.setId( sourceDeviceId );
	        
	        var event = new EventRepresentation();
	        event.setSource(source);
	        event.setType("c8y_LocationUpdate");
	        event.setDateTime(DateTime.now());
	        event.setText("Location update by single cell positioning");
	        event.setProperty("c8y_Position", c8y_Position);
	        event.setProperty("decoderCustomFragment", true);
	        
			DecoderResult decoderResult = new DecoderResult();
			decoderResult.addEvent(event, false);
			decoderResult.setSuccess(true);
			return decoderResult;

Thanks in advance!

[Kai_Sieben]

How is the DecoderResult handled afterwards?
Just creating the DecoderResult does not yet do any API call AFAIK.
You would need to handle the DecoderResult object in some way and take the event and send it to the platform.

[dst (Daniel Steiner)]

Since it’s a decoder microservice my application creates a REST endpoint “/decode” and is marked as decoder in my cumulocity.json:

"isDecoder": {
    "name":"LocationDecoderService"
  }

This endpoint gets called by the platform via lwm2m “CustomActions” when a specific value is transmitted by our devices. The microservice returns the DecoderResult as response body for the “/decode” endpoint. The whole functionality is implemented as described here:

cumulocity-examples/sample-lwm2m-custom-decoder at develop ·... - The LWM2M decoder interface

The expectation would then be as described here:

[Stefan_Witschel]

C8yPosition is a custom type I guess and cannot be deserialized to JSON on request.
When you are using standard types like “c8y.position” you can just do:

event.set(position)

for custom types you can use setProperty but have to make sure that the custom type extends svenson AbstractDynamicProperties.

Try to set a “String” as a customProperty for testing:

event.setProperty("decoderCustomFragment", "ThisIsAvalue");

[dst (Daniel Steiner)]

So i made some adjustments and further testing unfortunately with the same results. What i changed:

• Added extends AbstractDynamicProperties to my C8yPosition type and
• Changed set event.setProperty to a String:

	        event.setProperty("LocationService", "ThisIsAvalue");

When testing locally the response body of the “/decode” call is now:

{
    "events": [
        {
            "id": null,
            "type": "c8y_LocationUpdate",
            "time": "2023-05-24T12:51:50.395+00:00",
            "creationTime": null,
            "text": "Location update by single cell positioning",
            "externalSource": null,
            "source": {
                "id": "7660465"
            },
            "dateTime": "2023-05-24T12:51:50.395Z",
            "creationDateTime": null,
            "lastUpdatedDateTime": null,
            "LocationService": "ThisIsAvalue",
            "c8y_Position": {
                "attrs": {},
                "lat": 48.25999975,
                "lng": 16.30999804,
                "alt": null,
                "uncertainty": 2342,
                "fulfilledWith": "WIFI"
            }
        }
    ],
    "success": true
}

So extending AbstractDynamicProperties seems to just have added the “attrs”: {} object, otherwise the JSON looks the same as before.

I now add the event via http POST request (as before) AND also return the DecoderResponse as response body for handling in Cumulocity. As expected two events are generated, the event generated by the platform is unfortunatly still missing all custom fragments:

Event generated by Cumulocity:

        {
            "creationTime": "2023-05-24T13:01:42.977Z",
            "source": {
                "name": "My LwM2M Device",
                "self": "https://TENANT_URL/inventory/managedObjects/7660465",
                "id": "7660465"
            },
            "type": "c8y_LocationUpdate",
            "lastUpdated": "2023-05-24T13:01:42.977Z",
            "self": "https://TENANT_URL/event/events/9374975",
            "time": "2023-05-24T15:01:42.970+02:00",
            "id": "9374975",
            "text": "Location update by single cell positioning"
        }

Event generated by POST request to “/event/events”:

        {
            "creationTime": "2023-05-24T13:01:42.932Z",
            "source": {
                "name": "My LwM2M Device",
                "self": "https://TENANT_URL/inventory/managedObjects/7660465",
                "id": "7660465"
            },
            "type": "c8y_LocationUpdate",
            "lastUpdated": "2023-05-24T13:01:42.932Z",
            "self": "https://TENANT_URL/event/events/9378650",
            "time": "2023-05-24T15:01:42.928+02:00",
            "id": "9378650",
            "text": "Location update by single cell positioning",
            "LocationService": "ThisIsAvalue",
            "c8y_Position": {
                "fullfilledWith": "WIFI",
                "lng": 11.56999804,
                "uncertainty": 2342,
                "lat": 44.12599975
            }
        }

I kind of expected this since i checked to logs of the old version and deserialization seemed to work fine before. Would you happen to have a working example of some decoder code that adds a custom fragment?

Thanks, Daniel

[Stefan_Witschel]

Thanks for your testing!
ok, the attrs property looks wrong in the c8y_Position fragment, so serialization to JSON doesn’t seem to be the issue here.

This pretty sounds like an issue with the lwm2m-agent not recognizing any custom fragments for decoded events when mapping to Cumulocity IoT.

Are you able to create a support incident for this so we can track this issue?

[dst (Daniel Steiner)]

Ok thanks for the feedback, i will open a support ticket through our usual channels.

Regards,
Daniel

[dst (Daniel Steiner)]

After contacting Software AG R&D for feedback, i wanted to update this topic with an actual solution:

As it turns out i was adding my custom fragment to EventRepresentation object in the wrong way. What i was doing was to add the property by calling the setProperty(String name, Object value) method:

	C8yPosition c8y_Position = new C8yPosition( 44.12599975, 11.56999804, 2342,"WIFI");
	EventRepresentation event = new EventRepresentation();
	event.setProperty("c8y_Position",c8y_Position); // <- this is wrong
	DecoderResult decoderResult = new DecoderResult();
	decoderResult.addEvent(event, false);
	decoderResult.setSuccess(true);
	return decoderResult;

Which then creates a decoder result that looks like i posted above:

{
"events":[{
	"source":{"id":"7660465"},
	"type":"c8y_LocationUpdate",
	"time":"2023-05-24T08:39:09.683+02:00",
	"text":"Location update",
	"c8y_Position":{"alt":null,"lng":11.56999804,"uncertainty":2342,"fulfilledWith":"WIFI","lat":44.12599975}
}],
"success": true
}

As you can see the c8y_Position fragment is a direct property of the event, which is the syntax used when creating an event via the events API, and also the representation present in the decoder service documentation ( see Full decoder response sample in readme.md of the example).

Unfortunately this is NOT the syntax expected by in the decoder result of the lwm2m decoder. Here all custom fragments (so everything apart from text, type, time and source) need to be enclosed into attrs property like so:

{
"events":[{
	"source":{"id":"7660465"},
	"type":"c8y_LocationUpdate",
	"time":"2023-05-24T08:39:09.683+02:00",
	"text":"Location update",
	"attrs": {
		"c8y_Position":{"alt":null,"lng":11.56999804,"uncertainty":2342,"fulfilledWith":"WIFI","lat":44.12599975}
	}
}],
"success": true
}

This can be achieved in code by NOT using setProperty(String name, Object value) but the set(Object object, String propertyName) method of the EventRepresentation object like so:

	C8yPosition c8y_Position = new C8yPosition();
	EventRepresentation event = new EventRepresentation();
	event.set(c8y_Position,"c8y_Position"); // <- the right way to add custom fragments
	DecoderResult decoderResult = new DecoderResult();
	decoderResult.addEvent(event, false);
	decoderResult.setSuccess(true);
	return decoderResult;

Since this is not documented too well i was leaning on the example response in the sample-lwm2m-custom-decoder which unfortunately provides an invalid (or at least incomplete) representation of the way custom fragments need to be added, as well as on the way in which events are created via the events API which follows a different format to the one expected in the decoder result.

Hope this can help someone facing similar issues.

[Harrison_Pace]

Thanks for this, I am struggling to get the DecoderResult working and was also just making API calls directly to overcome this.

I found the Documentation lacking and was reliant on the same example description on Github. According to the documentation, it should be as simple as sending an array of measurements, events, alarms and dataFragments. But in practice it doesn’t seem to work.

I have attached an example of a response, I am returning from my Microservice which doesn’t work:
response.txt (5.4 KB)

Is it a requirement to strip the Source ID?

Can anyone chime in as to what needs changing and why?

[dst (Daniel Steiner)]

Hi Harrison,

I only worked with events (and java equivalent EventRepresentation) as a decoder result so far. Are you not seeing any measurements and alarms created or are just the custom fragments missing?

Are you using custom Objects for the Measurements or the default com.cumulocity.microservice.customdecoders.API.model.MeasurementDto and com.cumulocity.microservice.customdecoders.API.model.MeasurementValueDto? Can you share the code where you populate your DecoderResult?

Br,
Daniel

[Harrison_Pace]

Thanks Daniel,

I am not seeing any measurements or events been created, I am not using the SDK rather I am using a Flask with Python, I am manually defining the endpoint for /decode.

The code is segmented into classes. I will post the method which outputs the response of each:

DecoderResult:

def obj(self):
        return {
            "alarms": [alarm.obj() for alarm in self.alarms],
            "events": [event.obj() for event in self.events],
            "measurements": [measurement.obj() for measurement in self.measurements],
            "dataFragments" : [],
            "success": self.success
            }

Alarms:

def obj(self):
        return { "source": { "id": self.sourceDeviceId }, "type": str(self.fragmentName) + "." + str(self.seriesName), "text": self.text, 
                  "severity": self.severity, "status": self.status, "time": self.time}

Measurements:

def obj(self):
        return  { self.fragmentName : { self.seriesName : { "unit": self.unit, "value": self.value }},
            "time": self.time, "source": {"id": self.sourceDeviceId }, "type": self.fragmentName  } `

Event:

    def obj(self):
        return  {
                'source': {'id': self.sourceDeviceId},
                'type': self.event_type,
                'text': self.text,
                'time': self.time
            }

Not sure what I’m doing wrong here, but hopefully with the code and the response, it provides some insight?

Cheers

[sous (Sourajit Saha)]

Hi @Harrison_Pace
We talked earlier this morning. I checked the response body and found some issues there.

1. The “source” fragment needs to be there. Currently, I noticed an error because of the missing “source” fragment. I took a note of it and will be adding more proper error messages from the agent’s side. And also we need to update the document.
2. This is not an issue yet but you’ll run into this problem once you fix the ‘series’ issue. The additional fragment “mb_FlowError” will not be stored if the current response is used. This one works directly with the platform, but for the decoder results the JSON data for measurements has to be formed as specified here cumulocity-examples/sample-lwm2m-custom-decoder at develop · SoftwareAG/cumulocity-examples · GitHub

{
     "type":"c8y_example_lwm2m_decoder_binaryValues_byteIndex_1",
     "series":"binaryValueSeries",
     "time":"2019-02-07T11:05:55.272Z",
     "fragmentsToCopyFromSourceDevice":[
        "IMEI",
        "IMSI"
     ],
     "deviceFragmentPrefix":"device!Name",
     "includeDeviceName":false,
     "deviceNameFragment":null,
     "additionalProperties":{
        "foo":"bar"
     },
     "values":[
        {
           "seriesName":"byte 1",
           "unit":"unknown",
           "value":11
        }
     ]
  }

I hope this will solve your issue.