May ensure this processing rules does not break your existing function . Also you have to handle logic for all the documents which will not be recognized by the system . if you say *** Can I trigger a processing rule (to invoke a service) that capture such EDI messages and implement custom alert mechanism ***
Scenario 1 : Sender : X and Receiver : Y and Document : Any.
In scenario when your system not able to recognize the message based on the Document type this rule will be invoked and do your custom logic. Think about how many processing rule you will be ended up in creating based on the sender and receiver .
Scenario 2 : Sender : X and Receiver : Y and Document : Any
This will be your default rule and should be bottom of your processing rule .
And there could be any other scenario …