IS Responding very slowly after some actions

Hello There,

We have a problem with our DEV IS Server. We have around 15 packages. Usually, the IS responds very slowly. Below are the symptoms. We have the same amount of packages on TEST and PROD servers.

  1. Hangs when we save some code changes. It doesn’t matter, whether it is a big package or a small package. during this time, opening IS Admin is also an issue.
  2. Designer hangs, when i try to open a folder that has more elements (like doc, flow service etc…)
  3. If it hangs, it usually takes more than minutes some times, even 30 mins. I have to kill my designer and relaunch to do the code changes again. Some times, the code changes get saved.

Here are some of the details of the IS Server:

  1. Verison - 9.5.1.0

  2. We just have JDBC adapter and Broker.

  3. The services that were exposed were 98% REST services.

  4. Few of the memory assignment parameters:
    set JAVA_MIN_MEM=2048M
    set JAVA_MAX_MEM=4096M
    set JAVA_MAX_PERM_SIZE=1024M

      rem set JAVA_MAX_DIRECT_SIZE=-XX:MaxDirectMemorySize=128M
    
  5. Attached the thread dump when my designer hanged for more than 15 minutes.

  6. Windows Server 2008 R2 with 8GB RAM .

  7. CPU cores - 2
    8 ) Runs on SSD.

  8. Designer i use is 9.7 version.

Any help is really appreciated!

Thanks,
rAm
DEV IS Server Thread Dump.txt (84.4 KB)

Are you getting any out of memory memory issues?
Is the windows box dedicated to wM application/other applications as well?
Check whether there is sufficient heap memory has assigned to all components which are running in the same box?
Are you facing the issue with only designer/IS/Broker as well?
Have you tried to capture the statistics page when it hangs? This will help to identify the memory.

Hi Veera,

unfortunately the ThreadDump is incomplete.

I have tried to visualize it with Samurai ThreadDump tool, but it is not showing the thread table which allows to identify the blocking thread.

Which version of Designer 9.7 are using?
The RCP POC version or a real version?
If real version: Are there any Fixes applied to Designer?

Any Fixes applied to IS?

Regards,
Holger

Hello Krishna,

Really appreciate your reply. Please find below my responses for all your questions.

  1. No out of memory issues logged in the server or error logs.
  2. Windows Box is only dedicated to wM application
  3. I believe the heap memory is sufficient for the wM IS
  4. The issue is with IS and Designer. If i save some part of code, sometimes, even i add a new variable a document and save it, it takes a lot of time and at the same time, i can’t open IS admin as well. As soon as the IS admin recovers, i see the RAM used is around 1GB out of assigned 4 GB.
  5. I don’t see Memory Stats varying at all during the BAD state i.e. via IS Admin statistics. I clearly know the memory fluctuates, especially when we do a huge load test. But in this case, there is no variation at all.

Thanks,
rAm

@Holger - Thank you so much for coming forward to help me. I generated the Thread Dump from IS Admin. I copied every line to the note pad and attached in here. Do you think i am doing something wrong in here?

Attached a screenshot with the Designer version and updates.

This is a real version. Not a POC. I installed myself by downloading the image file and installed the designer on my machine.

This problem is not just with me. It is the same for each and every of my colleague. We have a mix of people using Mac OS and Windows 7.

Thanks,
rAm

Designer Version and Updates.png

Hi Veera,

can you provide a list of installed Fixes from UpdateManager?
For both (local Designer adn remote IS)?

Most likely this an IS issue instead of a Designer issue.
Can you check the network to the IS box?

Regards,
Holger

Veera – Involve your windows Admin to know what’s going on your windows machine where IS been installed. Check the load at different intervals. Analyze logs to find any errors corresponding to any of the IS components. There are many factors which may cause the issue you raised.

Thanks,

Yes please check with your win admins also the reason of slowness reaching IS unless you don’t see any abnormal things from the IS logs/thread performance etc…

Engage SAG support as well to troubleshoot more in officially.

HTH,
RMG

Hi Veera,
I had faced same problem before. Many people connected to a server, so server was slow down, when we openned or saved a flow service, it took long time. Sometime, someone couldn’t connect to server (other connected normally). I don’t know the root cause, but when we increased JVM Memory (x5), it returned normal. You can track it on Statistic Screen and try it.
If you can find out the root cause or another method, please let me know, because I also want to know it. :roll:

Hi,

please check if there any “Available Thread Threshhold” messages are logged to the IS server.log.

If so, you should consider increasing the amount of available threads in IS Admin under Settings -> Ressources.

Each active connection as well as any user being logged in requires a thread.

Regards,
Holger

Veera – what’s the current status of the issue? Did you able to figure any suspicious things around the IS or windows machine ?

Thanks,

Thank for your reply. We have updated the JVM memory of IS as well. I have listed all the Memory assignments to the IS in my first post. Could you please take a look at them. I believe that should be more than enough. That is same memory assignments done to our prod servers as well.

Thanks,
rAm

Hello Holger,

I don’t have update manager, i will see how i can updates applied to my designer. W.r.t. the IS, there were no updates applied to the IS version 9.5.1, attached screenshot of the about page in this post and also the resources page. I believe 300 max threads is more than enough for a DEV server and also i have never seen “Available Thread Threshhold” messages in the server.log.

Thanks,
rAm


I wasn’t able to involve windows admin yet. would like to invest some more time from my side.

Veera – I have gone through your 1st message on this thread, memory setting looking good, there are some other factors which have been causing this issue. Sure, spend some more time, keep analyze logs and involve Windows admins to know the facts.

Thanks,

Hi Veera,

please apply latest PRT Fix, Monitor Fix and IS Core Fix together with the required SCG-Fixes as there have been severel Performance- and Security-Issues fixed by these.

If you want to update Designer without UpdateManager check Empower and/or TechCommunity download section if there is an eclipse update site available.
If so download it and point Designer to search it for updates.

Regards,
Holger

Yes I echo the same please apply latest and greatest fixes available to resolve some of the issues (plan in your lower level environment first and test it out) apart from the Designer plug-in updates as well.

HTH,
RMG

Hi,

Try to increase memory in following file “ eclipse.ini “
I think you can see these two paramaters .

-Xms1536m
-Xmx3072m

Please increase these last two configuration values from default values to below one and lets see

Should be configuration:

-Xms2048m
-Xmx4096m

Thanks,
Vinay.

Thank you all for your replies.

I observed this one difference in the processor, rest all Hard Disk (SSD), CPU cores and RAM Assignments were same.

TEST - Dual 2.4GHZ Intel Pentium Pro/II/III (Intel® Xeon® CPU E7- 2870 @ 2.40 Ghz (2 processors))

DEV - Dual 2.5GHZ Intel Pentium III (Intel® Xeon® CPU E5540 @2.53Ghz (2 processors))

Does it make a big difference?

Hello Vinay,

I have applied the configurations as you said to my designer, earlier it was 2GB (Max). Which is more than enough for designer. But i increased to 4GB (max) and minimum to 2GB. But, i still didn’t see any improvement. Hoping this is on the windows machine.

Hi Veera,

this should not be the big difference.

Do you observe differences when trying to upload to the one or the other server?

Regards,
Holger