Is your Broker on the mainframe (z/OS)? Is the Broker protected by EntireX Security? What version of EntireX Communicator is the Broker running under?
If EntireX Security is not being used, you can shut down a server from SMH. If you are running EXX721 or above with EntireX Security, it depends on the security rules established for the class/server/service whether you can shut down a service or not. Below EXX721, with EntireX Security, you would need access to the SAG.ETBCIS.CMD service in order to shut down any service. Above that level, you also need to logon to SMH with an account that has Register authority (eg for RACF, you need Control access to the class.server.service; for ACF2 you need Delete authority) - same level as required to register the server.
For Windows, if you configure the Java RPC Server as a Windows Service, you can start and stop the server from the Windows Services control panel. See “Operation and Monitoring of the Java RPC Server under Windows” for information on how to configure the service.