I have read your “Performance Report” and I have tried to get a solution for the customer’s problem.
The scenario is:
Computer A (clock speed 3 GHz, RAM 1 GB, Linux Red Hat Advanced Server 3.0)
XML Mediator is installed on this computer (Web application + Host Manager).
When JBoss is running, the free RAM is about 175 Mb
The CF (DefaultComponentFactory) has these settings:
The main Mediator sequence processes a xml customer request and stores information into Domino database.
Each xml request file is about 30 kb
Computer B (clock speed 3 GHz, RAM 2 GB, Linux Red Hat Advanced Server 3.0)
Tamino XML Server is installed on this computer.
Mediator logs its messages into Tamino XML Server
Computer C (Windows 2000 Server)
Lotus/Domino database is installed on this computer
Problem description
The customer wants to send many requests at the same time (About 1000 requests). That means invoking the Mediator’s sequence by code.
For example:
for i=1 to i=1000
call MasterSequenceIn(Document_i)
end for
If i<50 , the test is OK
If i>50 the customer gets Error 503
I have tried to evaluate the HighMemoryMark using the formula at page 21 of your document, in order to get the best architecture for the customer
In this formula I have to set
Now my problem is evaluating MemoryConsumption parameter.
I have installed jvmstat and I have monitored the DefaultComponentFactory.
I have used visualgc tool and obtained the output as shown in the attached image.
When I launch a generic Mediator sequence I see the Eden Space increasing suddenly.
Is the Eden Space the MemoryConsumption?
Or is it the sum of Eden Space, Old Gen and Perm Gen?
Can I evaluate MemoryConsumption by hand without JVMSTAT?
There are some issues with your initial configuration that will complicate the measurement of resource consumption. I will however attach a jvmstat program that works with java 1.4.x here since it is no longer available from sun. The one for java 1.5 does not work with earlier jvm architectures.
What complicates matters is that you have only one processor on your machine and have to share processing time with the system, jboss and the component factory. This increase dramatically the length of time each sequence will require to complete and act as a bottleneck. On top of that, the GC in your JVM is quite active with full-gc (from the graphic you sent it used over 16 seconds of processor time). This serves only to worsen the bottleneck situation. The current resources available are, as you have demonstrated to yourself, not nearly enough to ensure the parallel processing of 1000 messages.
If we were to take the numbers as they are and just multiply your basic configuration, we would calculate that a total of 20 component factories, each with 1GB or ram, would be required to fulfill the your requirements.
This sounds a little extreme and would probably be quite different in a multi-processor environment where the possibility to tune the gc in the jvm and process multiple sequences in parallel exists.
My initial suggestion is to add a server with 4-8 processors and 16-32GB of ram to host the required ComponentFactories. You should attempt to maintain the same number of ComponentFactories as you have processors - increasing the concurrentMsg.Max incrementally.
If you could, please use the jvmstat I am attaching here and let me know what the avg. memory consumption for one sequence is and the required time to complete jvmstat.zip (301 KB)
It is very difficult to resolve performance or resource issues like this without including the details of the Mediator application itself. Some specific questions come to mind.
You indicated that the customer is sending 1000 requests at the same time. How are they being sent - synchronous or asynchronous? E.g. does the client program wait for completion of one request before sending the next?
If the requests are synchronous, does the processing sequence do anything like to create additional processing threads that continue execution after the initial request completes?
Have you specified anything other than the default for the Component Factory’s max concurrent messages setting?
In general, I do not believe in throwing additional hardware at a performance problem until more information is known about the application.
Mike is right. This is why I asked you to provide the statistics from jvmstat. If you could provide us with some idea of the application and sequences, we could perhaps also provide some hints on how you might re-engineer to optimize performance, reducing the machine requirements.
in order to clarify the problem I will provide you the exporting file of the project.
The involved sequence is called MasterSequenceIn.xml
This sequence invokes other sequences and should be invoked in asynchronous way about 1000 times (in order to process about 1000 documents each after the other).
The customer is been testing the system using a PHP script in which there is a loop for, like:
for (i=1;i<=1000;i++){
call MasterSequenceIn.xml
}
Obviously, each call does not wait for completion of the previous one.
