I am facing a problem when running a microservice with multiple k8s instances. I need to check whether a managedObject with a name (for example test1) already exists (By using Java SDK and filtering managedObject by name), if the name “test1” does not exist so I will create a new mo with the name “test1”. The problem can be handled easily with Java Synchronization while running a single instance. But with multiple instances, there is a case when 2 instances call the API to check the existence at the same time and the result is both “not-exist” => 2 instances will create 2 managedObjects with the same name “test1”.
I have tried to use externalID because as I understand one externalID can only attach to one managedObject. I expect that if 2 instances try to set the same externalID (let’s say test2_externalID) to 2 managedObjects at the same time, one of them will throw an exception and I will use the other one. But the platform doesn’t throw any exceptions in this race condition, I can have one externalID attached to 2 managedObjects.
Could anyone suggest a solution to this problem?
Thanks.
Error messages / full error message screenshot / log file:
No error
Question related to a free trial, or to a production (customer) instance?
Usually load-balancing should prevent the same query to reach multiple instances of your microservice at the same time.
I will therefore make the asumption that those are 2 different queries but with a very short interval between them.
In a standard architecture you usually address this kind of problem with a distributed cache, or at least an event bus.
Sadly there is currently no such capability in Cumulocity although in the near future you will be able to access Pulsar directly from you microservice which will help implement this.
The only solution I could suggest is to try to lower the frequency at which messages reach the platform. This could be done by sending batch of events for instance instead of sending them one by one.
We need more details regarding your use-case to help you find the right solution.
that sounds weird. Can you share a screen maybe of your two managed objects with the exact same external IDs (type + ID equal)?
What happens if you search for the external ID? Will it return both managed objects?
Normally it returns this on the identity API if you create the same external ID multiple times:
409
{
"error": "identity/Duplicate",
"message": "ID [type=c8y_Serial, value=test123456] was already bound to different Global ID.",
"info": "https://www.cumulocity.com/guides/reference/rest-implementation//#a-name-error-reporting-a-error-reporting"
}
Our microservice is a decoder, it receives payload from the LWM2M agent and decodes it. The information decoded includes a name (ex: test1), and we need to check and create a managedObject with that name. It is possible that 2 payloads from the LWM2M send the same name, in that case, we only expect 1 mo with the name test1 will be created.
Because the microservice has multiple instances, 2 payloads with the same name test1 may be sent to 2 instances simultaneously, leading to my problem.
I am expecting something like that. But the platform only throws exception if you call that API one by one, not simultaneously (I think C8Y hasn’t handled synchronization for this case). You can see the pictures below, 2 mo have the same externalID:
Hi,
Ok, understood.
I see a way of dealing with that: instead of creating the mo right away from your decoder, rather create an operation, then have another ms subscribing (or maybe polling to be sure to avoid any race) to those operations and dealing with the mo creation.
Since the operations will be queued, it becomes impossible that they are processed simultaneously.
Therefore, you’ll just have to ignore the second operation once your mo is created by the first one.
Thanks for the details @elcarim1802!
I discussed it internally and it seems to be a known issue which is already fixed in future versions (1018.135.0+). Can you check which version you are running?
Creating a microservice MO and using operations could be a good workaround. Important is that you set the status of the operation to EXECUTING so the other microservice does not work on it in parallel.
So it seems your version does not contain the fix yes.
If you synchronously want to process them, yes you need one additional microservice handling the operations of a MO owned by the microservice. The decoder microservice should be re-implemented , putting the decoded message to an operation instead of directly trying create the managed object.
Alternatively you wait for the fix. With the next release it should be fixed.
