Hi,
in Administrators_Guide i’ve found this note:
The Any server option does not specify an order in which
servers are used to execute tasks. In other words, no
load balancing is performed. Instead, an instance of
the scheduler runs on each server connected to the
database. Periodically, each instance checks the database
in which information about scheduled jobs is stored.
The first scheduler instance to find a task that is due
to start runs it, then marks the task as complete in the
database where information about scheduled jobs is
stored. The scheduler instances running on the other
servers connected to the database then know not to run
the task. This behavior will not change if you install a
third-party load balancer.
Now there still some questions of understanding:
Question 1: We see on our 4-node cluster that all scheduled tasks (Any Server) run on same node. Can we change this behaviour?
Question 2: What happens if we set a lower “Scheduler Thread Throttle” and the limit will be reached. Will this node allocate the task? In this case we expect this situation: Next Run (sec) = 0 Status = Active This would be a stupid solution regarding all other nodes are bored.
Question 3: Would the situation change if we use a synch clock service in the cluster?
Regards
Oli S.
Is all 4 IS nodes pointing to the same external database pool? You can check this from JDBC pools and check for IS Internal and Core Audit functions.
Thanks Mahesh,
yes, of course, we know how to setup an IS cluster 
Cheers!
Oli
Additional Info: we are using time sync services. Question 3 is obsolete.
It does require detailed analysis to say why one event occurred before another in a distributed clustered environment.
But as my prior experience, we do have some checklist for scheduled jobs running on ANY SEVER on your IS
1> Make sure there should not be server clock differences
2> Try configuring the same “watt.server.scheduler.maxWait” on all nodes, give it a try on QA or Test servers
3> Make sure all the nodes running IS has the same set of configurations (java, memory, network latency) without any difference.
If still, issues persist, better to involve support so that the issue can be looked at a more granular level.
Thanks Mahesh!
How do you understand this documentation text?
watt.server.scheduler.maxWait
Maximum time in seconds Integration Server waits between queries of the task queue.
The server periodically checks the queue for tasks that are scheduled to run. If there are
no tasks waiting to run, the server waits the maxWait time before checking the queue
again. If there are tasks waiting to run, the server checks again at the task’s schedule
execution time, or after the maxWait time, whichever is earlier. For example, if the
pending task is due to execute in 30 seconds and the maxWait time is 60, the server will
check the queue again in 30 seconds. The default is 60.
Note: You must ensure that the value you specify for this parameter must be less
than the minimum time interval set for a scheduled task.
It looks like this: One IS node checks the tasks in database and knows “there are 4 tasks waiting to run in 27 seconds, 7 tasks in 35 sec etc”. Other IS nodes have this information too (because they are polling the db). After 27 seconds each node want grab the 4 tasks. The first one wins and get all 4 scheduler task for execution. So the first node get them all? It processes them a long time (possibly due to slow SAP interfaces). After 35sec the same node gets the other 7 tasts etc.
It is not possible to synch the cluster node in ms. Even if, the first node seems to get all waiting tasks.
So your list is fine, but
1> small clock differences can’t be avoided (statement of our provider)
2> high watt.server.scheduler.maxWait could be an issue for short-time repeating tasks.
3> we have same config on all nodes and all server snuggle with their shared database
I want to understand the mechanism of scheduled task allocation to the different nodes. The SAG documentation is not very helpful.
It seems the stupid solution is (joke) 
set the server clocks every 30 seconds randomly different on all nodes. This way the first (“quickest”) node switches more often. :-/
I will write SAG Support just to explain this.
Cheers!
Oli
Response from SAG support for your information
[i]Hi Oliver,
I have been looking into this this further. (and also found some comments from the developer who maintains the Scheduler).
Your understanding of watt.server.scheduler.maxWait is correct. The scheduler thread checks periodically according to this parameter for any tasks that need to be started. All the tasks that need it are then started. The scheduler then calculates the interval until the next task will need to be started and sleeps for either that duration or maxWait, whichever is smaller.
If you want to do any logging, you can use IS Admin > Settings > Logging > Server logger > 137 User Task Scheduler = trace. Setting this will show what is happening (It you haven’t already tried this - it produces a lot of output, so you might only want to do it in test)
In a clustered environment, there is no synchronization. Each Integration Server has a scheduler thread, and whichever server gets to it first will run a task. There is no intention to make a task stick to one node, nor does it make any attempt to evenly distribute the load. Thus when there is a difference in clock between machines (even a small difference like yours) the first one to get the task wins.
I did notice on your database extract that a number of the tasks has the same next run value so they will get started at the same time by the same Integration Server.
I think it would be worth investigating if a time server can be used to keep the clock fully in sync. Even if it does not make them sync exactly to the ns, differences in workload/jvm/machine etc would make the distribution more random.
Trust this helps to explain. Let me know if you have further questions or if I can close the incident. [/i]