Integration server v7.1.2 is not starting up properly. It gets stuck at enabling ports.
[ISP.0046.0012I] Enabling HTTP Listener on port 9999
[ISP.0046.0012I] Enabling HTTP Listener on port 5555
I also see below error.
[ISP.0090.0004E] pub.storage:put – Critical Exception: Cannot aquire the lock.
This is a repeated issue. Last time I started up server in a different port using ./server.sh -port and refreshed the default port and again started with the default port, then it started. One more time it took all weekend before finally coming up.
Need your suggestions on what could be the cause of this and how this can be fixed. I don’t think port alone is the issue here.
Is this have anything to do with DB issues like pool definition password expiry etc?
To add to above post, found below repeating in logs when started IS in debug mode. Because of this IS is taking long to start. Kindly suggest on how to fix this.
2016-05-02 11:24:58 GMT [WmDistributedLockManager.JdbcLockDbManager.selectOlderCountD] Found 0 locks older than 1462188118315
2016-05-02 11:24:58 GMT [WmDistributedLockManager.JdbcLockFactory.listOlderLocksD] Leave JdbcLockFactory.listOlderLocks(1462188118315), returning 0 locks
2016-05-02 11:25:03 GMT [WmDistributedLockManager.JdbcLockDbManager.selectOlderSqlD] selecting locks older than (SELECT EXCL_LOCK_ID, TARGET_ID, LOCK_HOLDER, LOCK_TYPE, LOCK_SHARE_COUNT, LOCK_TIMESTAMP FROM WM_EXCL_DIST_LOCK WHERE LOCK_TYPE=‘E’ AND LOCK_TIMESTAMP<?) with this sql: 1462188123324
2016-05-02 11:25:03 GMT [WmDistributedLockManager.JdbcLockDbManager.selectOlderCountD] Found 0 locks older than 1462188123324
2016-05-02 11:25:03 GMT [WmDistributedLockManager.JdbcLockDbManager.selectOlderSqlD] selecting locks older than (SELECT WM_EXCL_DIST_LOCK.EXCL_LOCK_ID, WM_EXCL_DIST_LOCK.TARGET_ID, WM_EXCL_DIST_LOCK.LOCK_HOLDER, WM_EXCL_DIST_LOCK.LOCK_TYPE, WM_EXCL_DIST_LOCK.LOCK_SHARE_COUNT, WM_EXCL_DIST_LOCK.LOCK_TIMESTAMP, WM_SHAR_DIST_LOCK.SHAR_LOCK_ID, WM_SHAR_DIST_LOCK.LOCK_HOLDER, WM_SHAR_DIST_LOCK.EXCL_LOCK_ID, WM_SHAR_DIST_LOCK.LOCK_TIMESTAMP FROM WM_EXCL_DIST_LOCK, WM_SHAR_DIST_LOCK WHERE WM_EXCL_DIST_LOCK.EXCL_LOCK_ID=WM_SHAR_DIST_LOCK.EXCL_LOCK_ID AND WM_EXCL_DIST_LOCK.LOCK_TYPE=‘S’ AND WM_SHAR_DIST_LOCK.LOCK_TIMESTAMP<?) with this sql: 1462188123324
From my understanding this SQL is harmless, you got this logs it is due to there is a sweeper thread running to clean up the expired locks, even there is no locks, the sweeper thread will still resumed.
May I know do you manage to bring your “IS” up? Do you have any other logs details.
Besides, for me I will suggest better change the “5555” port.
You were dealing a very old version 7.1.2 and make sure you clear all the repo and document store and audit files for a clean startup and if required copy/restore with the config back up files if this was some thing worked before.
The server came only after disabling the user packages. But couldn’t locate the startup service which was stalling the server startup in debug startup logs. Still no information about the lockmanager alert.
Based on your initial message, you should be able to find your problematic startup service by doing a “Find dependents” on the pub.storage:put service.
I’m guessing that either due to a bug in the application or due to an unexpected shutdown, your application left some locks in the database which are preventing a startup service from completing. Run the service pub.storage:listLocks to see if you can determine what happened. Assuming you find locks there that should no longer exist, you can use pub.storage:releaseLocks or pub.storage:unlock to get rid of them. The Built-in Services guide has more info on these services.
Last but not least, I strongly recommend taking a look at how and why you guys are using pub.storage. I’ve worked with several other customers in the past who ran into locking as well as performance issues when using these services and we were always able to find better solutions. I personally avoid these services unless absolutely necessary.