I have currently 7 instances on one physical box each using 2GB JVM memory and one of them with 3,5 GB JVM memory.
Remember that the instances also have a Tanuki Wrapper for controlling the instances, which requires additional RAM.
You will have to do some sort of load testing to figure out the right values for your system.
Performance is subjective, so one cannot provide an objective answer - that is, there is no “one-size-fits-all”, or even “best practices” to some extent. Here are some questions to get you started -
Do you have a performance issue you’re experiencing?
Why do you want to change the hardware now?
How did you decide that you need 4 instances/installations?
Why do these 4 installations/instances share the same physical machine?
Is this a development environment?
What are the transaction/request volumes that this machine is handling?
What are the average, peak and trough transaction throughputs across this machine?
What is the usage on the machine? CPU, RAM, Disk I/O, Latency?
Will vertical scaling (increasing the resources on the machine) suffice considering the cost/benefit ratio, or do you need to consider going horizontal (increasing the number of machines, i.e., instances)?
Having cautioned you above, the best performance benefits are unequivocally reaped from within the code - i.e., good interface design and development principles.
Share a description of your business case or the problem you’re facing, and the experts can guide you better.