I’m positive that you will have to configure heap space for Designer, since WmTestSuite (i.e., Unit Test Framework) is a Designer/Eclipse plugin - it doesn’t have it’s own JVM.
Read the “Working with Designer” guide for adjusting Designer memory settings.
In case the error happens at Server end, the default wrapper property for max memory (which is 512MB) needs to be checked. Memory resources need to be allocated based on ‘packages in scope’, lines of code in Services under tests & no. of tests that are being executed as part of Code Coverage.
Note: the memory would be freed up once test execution completes and reports are generated.
For Designer end error, you can specify max memory details in eclipse.ini.