handleTransition isn’t a public service so there is no documentation for it. That said, in process models it is invoked under the control of a trigger in exactly the same way as any other service, so provided the Process Transition document arrives with the correct content, handleTransition will operate correctly.
In practice, this is similar to how the resubmission for a process step works if done through the Monitor UI or through the Monitor services.
For a public service to resubmit a failed step, you need to look at the Monitor services (see the Monitor Built-In Services Reference, particularly pub.monitor.process.instanceControl:resubmitInstanceStep).
This may give you the control you need, though if you are looking to do very rapid resubmission, note that these services work off what is in the audit database so the audit information must be processed first. Also, for resubmission to work, the step must be enabled for resubmission, the main impact of which is that the pipeline for the service is saved in the Audit database. Therefore your audit data will grow faster especially if you are not cleansing the pipeline (as recommended by best practice).
There are a few implementations out there that have used fairly specialised handling of process errors and other intervention mechanisms, but these have always been done with close co-operation of Software AG Professional Services. You are welcome to experiment with the behaviour of these services, but in most cases, bespoke invocations of handleTransition and the other non-public services wouldn’t be supported and may not always work the way you want them to.