Routing to multiple targets using ES 50 from IS 46

I have configured my adapters in IS 4.6 and using ES 5.0 for routing purpose only.

One of my IS service is publishing a doc to ES broker. The same doc is subscribed by two client groups (CG1 and CG2). Now i have configured 2 listeners in IS for CG1 and CG2 to subscribe the doc from ES to two different services in IS.

Now this setup works fine individually (i.e. one listener is up at a time).
But they do not work together (Only one of the listener/ service consumes the same doc twice and the other gets none).
Also, if i stop both of them, publish the doc and restart both the listeners, it works fine (each listener/ service consumes one doc each).

Please let me know if anybody has worked in similar scenario.