We experienced similar problem few months back.
To temporarily resolve the problem we scheduled a service that runs java System.gc(); I cannot say that this solves the problem. But in general, to my experience, it will reduce your “restart IS” frequency.
We also applied the latest patch of the HP-1.4.2 JVM and increase the Min heap to make it the closer with the max heap. (We use 1024M). This will in general increase the frequency of the Garbage Collection. You have to be cautious, because running gc to often may reduce your performance. This is where tuning will probably takes place. So, it is largely depends on your application and how heavy is your IS usage, in terms of mapping etc.
My best suggestion is to go through your services again. Most of the time you will probably find some unused pipeline that didn’t get cleared, db connection from wmDB don’t get closed properly, disabled flow-steps etc.
I feel that optimizing your services may reduce your memory problems or even solve it.
IS slow response sometimes also relates with the frequency of pub-sub of IS with the brokers, your message delivery mechanism and the size of the docs that get pass around. If you use workflow with large docsuments and comlicated hierarchical mapping, the IS of the workflow could potentially be running real slow, especially if you use WMTomcat to deploy your workflow.
So, my suggestion is look at your architecture of how do you use the IS and check all the services to make sure everything clean and in prod, remove all the disabled steps as well.