Are you looking for programmatic way to search for tasks from IS? When you invoke SearchTasks service what search query do you pass in? You need to specify following configuration:
searchUserTasks → false
Terms →
Fields → processInstanceID
Operator → =
Value → your process instance ID
This error is not good. Not sure if this would help but you could increase the maximum JVM memory for My webMethods server (which embeds Task Engine) by modifying following file: Portal/server/default/bin/server.properties.(sh or bat)
Do you know how many tasks do you have in the system?
What exact database version do you have in use?
Could you search for Process Instance ID from Task List Management portlet?