For this you need to understand how the system architecture work. In a unix environment the OS has a set amount of memory eg. 4 GB. When you set the parameter for the IS JAVA_MIN_MEM=1024M, the JVM starts with an initial chunk of 1GB from the original 4GM set in the OS as physical memory.
Now as your IS starts processing the memory utilization of the server starts increasing depending on the kind of services running on your IS.
By the sound of it, it seems to be the case that the Garbage Collection might not have been working as efficiently as it should and there might be memory leaks in the code internally. I would recommend you have a manual GC set as a scheduled service in your IS. Just be careful that you dont set up GC as a scheduled service very frequently as it would make your system idle there by even reducing the performance of your server even further. Also check for variables not being dropped in your services. Its always a good idea to use clearPipeline service at the end of each of your flow services to clean up the pipeline to save memory.