scheduler instance still active, recovered by other instance in the cluster
I'm noticing a strange behavior in clustering that is causing erratic behavior. Some of my jobs hang while waiting for outside web service calls, and they must hang for a while by necessity. If they hang for too long though, I notice the job starts to get picked up by other nodes in the cluster, even though this is a stateful job, and is not finished, so no other job of its type should fire. I'm also noticing this in the logs
This scheduler instance (SchedulerName) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior.
1) Is there some sort of Quartz thread which stands outside the cluster and looks for nodes that are hanged?
2) If so, does it then ask another node to deal with the same job?
3) How long must a node be hanging before Quartz asks another node to take the job?
4) Is that amount of time configurable?
5) Can I turn off this behavior entirely, since I expect hanging time in my job?
Things to know about my system:
MySQL database, not running schedulers in a web container, multiple Windows boxes, clocks ARE synchronized extremely regularly.