Data storage/ statistics

,
1

Product/components used and version/fix level are you on:

Cumulovity IoT version 1015.0.309

Detailed explanation of the problem:

While analyzing the data storage, I have noticed significant differences between the data stored in the customer tenant and the data sent from the customer’s device to that tenant. I’m looking for suggestions on how to query and identify these differences effectively. Any ideas?

Is your question related to the free trial, or to a production (customer) instance?

production

2

Hello Leandre,
please describe more in detail, which differences do you mean, e.g. are periodically data missing, a special type of data is missing or are measurement series cut?
Regards, Christian

1 Like
3

Hallo Christian, no data is missing as per our records. The tenant statistics indicate that storage in the C8Y database is consuming more data than is being transmitted from the customer’s device to C8Y. The device is transmitting around 10 MB pro day, but the tenant statics show 17 MB per day. I am keen to understand the factors contributing to this discrepancy and explore preventive measures.

4

How are you sending the data? It is not necessarily stored in the format you use for sending.

5

How can I find out, how and in which format the data is beeing stored?
Regards,
Leandre

6

Hi Leandre,

ok I understood you compare storage values in:
{{url}}/apps/administration/index.html#/tenantStatistics
[STORAGE (MB) and/or PEAK STORAGE (MB)]
with transmitted data measured by an external tool.

First idea: this storage shows the sum of all devices.
Do you have only one device receiving data in your tenant?

Second idea: Possibly you have an active Apama streaming analytics rule, which creates additional data for that device?
You can check the active rules in {{url}}/apps/streaminganalytics/index.html#/analyticsmodels/models

Which tenant and which device is affected?
If it is hosted by Software AG, then I can check it with my support user access.

Regards, Christian

1 Like
7

Hi Christian,

  • exactly, I am comparing tenant statistics with transmitted data measured by an external tool.
  • Currently, there is only one device in that tenant with active Apama streaming analytics rule. This might be the reason for the discrepancy. Thanks for your support.
  • It is not hosted by SAG, but I can temporaly add you as a new user.

Regardless of the above reasons. How can I find out, how and in which format the data is being stored?

8

In the database it is stored as JSON documents, always, no matter how the data is transmitted to Cumulocity. So when sending SmartREST via MQTT for example, resulting in a very small payload size during transmitting, the size in the database will be much bigger as it is still a JSON document in the end.

SmartREST:
200,c8y_Temperature,T,25

JSON in database (not complete):

{
	"self": "https://t14368213.eu-latest.cumulocity.com/measurement/measurements/106021068",
	"id": "106021068",
	"type": "c8y_Temperature",
	"source": {
		"self": "https://t14368213.eu-latest.cumulocity.com/inventory/managedObjects/101010649",
		"id": "101010649"
	},
	"time": "2023-03-23T00:00:06.197Z",
	"c8y_Temperature": {
		"T": {
			"value": 25
		}
	}
}

Also the platform adds additional information to the documents like “self”, “createdOn”, “lastUpdated” and other properties which are calculated and added, which you don’t even see in the REST interface.
When using Apama this is also true for additional calculated documents or properties.

This in sum could possibly explain the 10 to 16 MB discrepancy you’ve detected.

1 Like
9

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.