We have a problem with Mediator 7.2.1 when executing two sequences in parallel.
We first send a medium xml document (4 MB) to a first sequence. Normally the processing time take 20 minutes.
Then we send a very small document (60 KB) into an other sequence. But we get an out of memory from mediator and both processing stop.
We would like to know how mediator behave when receiving multiple concurrent request for different sequence? Does it start a new instance of mediator factory, does it serialize incoming messages?
The memory setting is normally enough. Tomca ?Xms384m ?Xmx1024m MediatorFactory ?Xms256m ?Xmx1024m and Mediator host default.
When receiving the second request none of the memory usage grows beyond pre-allocated memory (-Xms).
It is difficult to answer this kind of question without a lot more information. Perhaps this long and general answer will help.
Mediator 7.2.1 added support for multiple component factories. The router part of host manager, which runs in the web server, dispatches work to each factory in simple round-robin fashion. Each new request is dispatched to the next available factory in its routing table. At startup the host manager can start all of the factories defined to it or just one default factory. To add a new factory it must be started manually. The host manager will not dynamically start a component factory after the startup period.
There is a protection mechanism that allows the host manager to bypass a component factory when that factory exceeds a given level of memory allocation. This is the High Memory Mark defined on the Admin console. The router will not send new work to that factory until its memory use goes down to the Low Memory Mark. The purpose of this mechanism is to allow the factory to continue to process work that has already started. Also the Low Memory Mark allows the garbage collector to finish its job before new work is introduced. While this prevents new work from being added to the factory it does not prevent the work already in progress from consuming all of the memory in the JVM. When the work already in progress does consume all the memory, the JVM should expand its heap according to the -Xms/-Xmx specification. If that is exhausted it will fail.
There are some areas of concern in this scheme. It is new and we plan to make it more sophisticated in time.
- It is symetrical. Each message is treated like any other. The caching that occurs for sequences, models and stylesheets is duplicated in each component factory. There are cases where it would be efficient to dedicate a factory or group of factories to a type of sequence or some other parameter. For now you could accomplish this by creating another web server.
- If a new request is routed to a factory that is currently enabled and it is now determined that the High Memory Mark is exceeded (by work currently in progress), that new request is rejected with an error (I think the error code is =>500) and the factory is disabled for new work. You have to resubmit the request even if there is another factory available. In other words the first request to disable a factory is rejected, but other requests following it will be routed to another factory.
- If you set the High Memory Mark too low the JVM might stay at the -Xms minimum memory level but the factory will periodically get disabled and enabled. This occurs if the requests that do get processed exceed the High Memory Mark but do not consume enough memory to force the JVM to expand its heap.
- If you set the Low Memory Mark too low the factory might get disabled but not re-enabled. Garbage collection is the only mechanism for reducing the memory consumed in the factory. Mediator caches sequences and stylesheets in memory. If there are large sequences and stylesheets which keep the memory used above the Low Memory Mark the factory will not re-enable even if it is idle. You will have to shut down the factory and restart it manually.
The case below does not say what the High and Low Memory Marks are but I would suggest setting them higher, perhaps to 90% and 85% respectively, because there is a wide range between -Xms and -Xmx.
I wonder why it takes 20 minutes to process a 4MB document. Does the machine have real memory to back up the request for 1024M of virtual memory? Avoid using “//” in XPath expressions. This may cause the processor to examine every node of the document. The Xalan/Xerces processors will normally only expand the nodes that are necessary to process a request. Using “//” may require the entire document to be expanded in internal tables for a case where only a small fraction of the nodes are actually needed.