We´ve had some trouble with a flow service which cache seems to get currupt every now and then, or at least I think so.
The service is cached (4 hours) and uses the prefetch feature and retrieves a TPA with system specific settings, nothing else.
Just out of the blue the service starts taking approx. 5s to execute instead of <10ms. Sometime it happens when the prefetch is executed, sometimes not.
It does not help to clear the cache, and the only way to get rid of the behaviour is to turn off the cache feature, save the service, turn it on again and save again. It happens almost every day now and casues the thread count to raise very quickly (since the service is executet approx. 10 times / s) until it cant receive any more requests.
Could it be that the cache somehow gets corrupt?
We have 3 IS:es in a cluster and it allways happens to the same service and on every IS at the same time, which makes me think is a cache/cluster issue…
One idea is to turn off caching. The benefits are likely marginal and if it is misbehaving this way the downside now outweighs the upside.
Keep in mind that caching looks at and stores the entire pipeline, not just the inputs and outputs declared on the service. If the usage of the service has changed (called in different scenarios) then that might explain why the caching behavior has changed.
According to the IS there is only 1 cache entry (for this service) and the cache hit ratio is never less than 99%, so it really does seem to work the way it should. We have quite alot of configuration stored in different TPA´s for different solutions, and if we would turn the caching off for every service the database would probably go haywire
Looking at the memory usage I can´t see anything unusual when it happens, nor is it connected to the load which is pretty constant most of the day. The usage/behaviour has been the same the last 4 years, so I´m a bit puzzled. The only changes are environmental changes like patch levels etc.
I´ve opened up a service requests, but since I can´t reproduce the behaviour in a controlled manner it´s likely they will just close the ticket.
Any other ideas?
I havn´t dropped you idea about turning the cache feature of, just saving it for a last way out :p:
Hmm, seems like you’ve investigated/considered each of the things I would look at. At this point I’d suspect an obscure bug/issue with a recently applied patch. I know your pain in getting support to help reproduce/resolve the issue.
My only other idea is to use a test environment to determine if caching actually improves things. I know the TN services cache some elements already, though I don’t recall if it caches TPAs. If the difference in time of, say, 10,000 reads between cached and not cached is marginal, then perhaps you can avoid the headache of figuring out what’s pooched.
Yeah, probably some bug related to the current patch level (after all, it´s not like it´s never happened before ).
Anyway, I´ll set up a test and see how big the difference in time is - cause right now the only option is to turn the cache feature off.
I made a small test using soapUI, 10.000 transactions comparing times when the services is cached vs not cached. I also compared soap vs http get, not that its relevant for this issue…
I made the test for 1, 2 and 3 threads. Definition of assert: Time > 1000ms
The load in this test is significantly higher than in our production environment, so I´m not very worried about the nr of transactions with response time > 1000ms.
The transport protocol also adds some overhead, which does not exist when the service is invoked from another flow service.
I´ll go ahead and turn off the cache feature in production.
every now and then, or at least I think so.
).