please remember that Broker 9.6 is the last version which will be available for Broker ever, as Broker will be replaced by Universal Messaging.
Broker 9.6 can be used with all recent wM versions from 9.6 onwards.
As long as the Machine can handle the load one Broker should be sufficient and can handle both types (default messaging and JMS messaging), if the Broker is given enough data storage size.
On my environments we are running two Broker Servers from one installation under one Broker Monitor:
One for handling general messaging and JMS messaging, but without Optimize traffic and one for the Optimize Traffic.
This approach has been choosen as Optimize had (and at least partially still has) difficulties when connecting to SSL-enabled and encrypted Broker Servers.
Especially the Designer has no properties for defining the keystore, truststore et al. when configuring Optimize connectivity.
Therefore the Optimize-Broker is not SSL-enabled and encrypted whereas the Broker the regular messaging and JMS messaging is SSL-enabled and encrypted as required by our security team.