Resuming a cron job after 502 bad gateway error

Product/components used and version/fix level:

Detailed explanation of the problem:

Hey guys!
I have the following code, every now then there is the bad gateway error on the tenant which causes the below code to stop working and the cron job never resumes even on the next day. Does anyone know how to get around this problem?

	@Scheduled(cron = "0 0 0 * * * ")
	public void triggerCalculation() {

		try {
			memoryUsageLogger.setFunctionRunning(true);
			LOG.info("Running trigger forecase calculation...");
			microserviceSubscriptionsService.runForEachTenant(() -> {
				try {
					long startTimeMillis = System.currentTimeMillis();

					LOG.info("Running forecast service for tenant: " + microserviceSubscriptionsService.getTenant());
					analyze(microserviceSubscriptionsService.getTenant());
					LOG.info("Trigger calculation process completed!");

					long endTimeMillis = System.currentTimeMillis();
					long elapsedTimeMillis = endTimeMillis - startTimeMillis;
					double elapsedTimeMinutesMillis = (double) elapsedTimeMillis / (60 * 1000);

					System.out.println(
							"Elapsed Time (System.currentTimeMillis()): " + elapsedTimeMinutesMillis + " minutes");

				} catch (Exception e) {
					LOG.error(e.getMessage());
				}
			});
			memoryUsageLogger.setFunctionRunning(false);
		} catch (Exception e) {
			LOG.error(e.getMessage());
		} catch (Throwable t) {
			LOG.error(t.getMessage());
		}

	}

Error messages / full error message screenshot / log file:

2024-02-20 10:04:48.616 ERROR 1 --- [subscriptions-0] .s.s.i.MicroserviceSubscriptionScheduler : Error while reacting on microservice subscription

com.cumulocity.sdk.client.SDKException: Error invoking GET http://cumulocity:8111/application/currentApplication/subscriptions
	at com.cumulocity.microservice.subscription.repository.impl.CurrentMicroserviceRepository.handleException(CurrentMicroserviceRepository.java:101) ~[microservice-subscription-1018.0.261.jar!/:na]
	at com.cumulocity.microservice.subscription.repository.impl.CurrentMicroserviceRepository.getSubscriptions(CurrentMicroserviceRepository.java:75) ~[microservice-subscription-1018.0.261.jar!/:na]
	at com.cumulocity.microservice.subscription.repository.impl.CurrentMicroserviceRepository.getSubscriptions(CurrentMicroserviceRepository.java:81) ~[microservice-subscription-1018.0.261.jar!/:na]
	at com.cumulocity.microservice.subscription.repository.MicroserviceSubscriptionsRepository.retrieveSubscriptions(MicroserviceSubscriptionsRepository.java:123) ~[microservice-subscription-1018.0.261.jar!/:na]
	at com.cumulocity.microservice.subscription.service.impl.MicroserviceSubscriptionsServiceImpl.retrieveSubscriptions(MicroserviceSubscriptionsServiceImpl.java:165) ~[microservice-subscription-1018.0.261.jar!/:na]
	at com.cumulocity.microservice.subscription.service.impl.MicroserviceSubscriptionsServiceImpl.subscribe(MicroserviceSubscriptionsServiceImpl.java:115) ~[microservice-subscription-1018.0.261.jar!/:na]
	at com.cumulocity.microservice.subscription.service.impl.MicroserviceSubscriptionScheduler.lambda$schedulePeriodicSubscription$0(MicroserviceSubscriptionScheduler.java:81) ~[microservice-subscription-1018.0.261.jar!/:na]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
Caused by: com.cumulocity.sdk.client.SDKException: Http status code: 502
Something went wrong. Failed to parse error message.
 tenant: t140168379 user: servicebootstrap_trigger-forecast-ms
	at com.cumulocity.sdk.client.ResponseParser.checkStatus(ResponseParser.java:81) ~[java-client-1018.0.261.jar!/:na]
	at com.cumulocity.sdk.client.ResponseParser.parse(ResponseParser.java:63) ~[java-client-1018.0.261.jar!/:na]
	at com.cumulocity.sdk.client.RestConnector.get(RestConnector.java:118) ~[java-client-1018.0.261.jar!/:na]
	at com.cumulocity.microservice.subscription.repository.impl.CurrentMicroserviceRepository.getSubscriptions(CurrentMicroserviceRepository.java:73) ~[microservice-subscription-1018.0.261.jar!/:na]
	... 11 common frames omitted```

Hi @Arsalan_Mahmood_Siddiqi ,

I suspect the following.

  • The default pool size of the Scheduler is 1. So it might be that on error case the scheduler thread is not released properly and blocking other threads to get started.

Actually you start a thread in a scheduler thread. You could increase logging to see if the threads are released properly. The one who is throwing the exception is the subscription thread for each tenant it tries to get a subscription, the scheduler threads seems to be fine and maybe do endless retries?!