Sure. Integration Server supports WS-Security using Username, X509 Certificate or SAML tokens…
Well, that is, if the custom soap processor that you write from scratch supports those things. IS, out-of-the-box lacks any and all support for WS-Security today.
I’ve written custom soap processors to handle authentication and optionally authorization based on Username and SAML tokens now for four different projects so I can say with certainty that it is very, very possible. It just takes a bit of work… OK, a lot of work.
If you want to use XML digital signature or XML encryption in your tokens or messages that also is possible, but with even more work.
The Infravio acquisition provides some hope that WS-* , XML digsig and XML encryption support is not that far away, but the work to integrate Infravio X-Broker (to be relabeled SOA Management) with Integration Server has yet to be done.
Take a look at some of the custom soap processor threads in this forum for ideas of what is involved. A custom soap processor is typically just a Flow (or java) service that pre-processes soap requests received by Integration Server. You can make that Flow service do anything you need before the requested operation is invoked. Harder than it should be by far, but very feasible and highly performant when done right.