Service restarting while fetching 1500 values

,
1

Product/components used and version/fix level:

Cumulocity Production

Detailed explanation of the problem:

Hi, I have deployed one microservice which fetches around 1500 values three times for each M.O.
Now, while fetching the measurements from the DB, the service restarts.

In local it works perfectly fine but we are not able to identify why the service is restarting while fetching values from DB.

    try {
        subscriptionsService.runForEachTenant(() -> {

            MeasurementFilter filter = new MeasurementFilter().bySource(new GId(deviceID)).byType(type)
                    .byValueFragmentSeries(series).byDate(dateFrom, dateTo);
            Iterable<MeasurementRepresentation> measColl = measurementApi.getMeasurementsByFilter(filter).get(2000).allPages();

            List<MeasurementRepresentation> measList = StreamSupport.stream(measColl.spliterator(), false).toList();

            List<Double> list = fetchValues(measList, series); // This method separates the values from the sample structure.
            map.put("list", list);

        });

Sample Structure is like this:

               "measurement":  {
                "A": {
                    "value": 81.9
                },
                "B": {
                    "value": 26.55
                },
                "C": {
                    "value": 100
                },
                "D": {
                    "value": 102.37
                },
                "E": {
                    "value": 5625.96
                },
                "F": {
                    "value": 22129.44
                },
                "G": {
                    "value": 2.54
                },
                "H": {
                    "value": 99
                },
                "I": {
                    "value": 98
                },
                "J": {
                    "value": 247.22
                },
                "K": {
                    "value": 0
                },
                "L": {
                    "value": 98
                },
                "M": {
                    "value": 100
                },
                "N": {
                    "value": 393.87
                },
                "O": {
                    "value": 180.81
                },
                "P": {
                    "value": 804.66
                },
                "Q": {
                    "value": 78.03
                },
                "R": {
                    "value": 99
                },
                "S": {
                    "value": 84
                },
                "T": {
                    "value": 100
                },
                "U": {
                    "value": 5.11
                },
                "V": {
                    "value": 99
                },
                "W": {
                    "value": 87
                },
                "X": {
                    "value": 31.36
                },
                "Y": {
                    "value": 1188.39
                },
                "Z": {
                    "value": 817.82
                },
                "AA": {
                    "value": 90.64
                },
                "AB": {
                    "value": 521.41
                },
                "AC": {
                    "value": 42.34
                },
                "AD": {
                    "value": 196.16
                },
                "AE": {
                    "value": 46.24
                },
                "AF": {
                    "value": 724.43
                } //and so on
            }
        }

Let me know if you have any idea what the issue is.

2

What is your configured & assigned Memory for your microservice? Part of the cumulocity.json
It might be the pod or JVM is running Out of Memory and is restarted by Kubernetes.

A small improvement in your code (but unrelated): You could move the try - catch block within the loop (2 lines down). So if one tenant processing throws an exception the other tenants processing is not affected.

1 Like
3

Your code copies all measurements of all pages of your query into a list. Depending on your resources configuration and the size of your data this can easily consume more memory than available. Then your microservice process will be automatically killed and restarted.

I recommend using paging as intended and load smaller page sizes, process their content and only then load the next page. in your code this would mean iterating over pHMeasColl directly instead of copying its content first and using a page size < 100.

1 Like
4

The issue was the size only of the cpu in config. Thank you for the suggestion

5

Just for clarification: You increased the CPU cores in the manifest? Did you also increase the memory?

6

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.