Use a “TN gateway service” and provide hints to TN as to what the document type is. In other words, you tell TN what the doc type is rather than TN doing doc recognition.
Does all the Sender/Receiver/DocType showing as “unknown” when the payload extracted?
Either way you need to use routeFlatfile service for TN to recognize the flat file doc type and so if atleast the sender/receiver recognizes and it invokes the rule/processing service where you can extract content and convert to Stream to map it to routeFlatFile service thus it can get recognized in TN: