Quartz troubleshooting

Problem:
Quartz in clustered mode, nodes are sync via db.
There are many lines like these in logs:
[2015-07-04 00:00:01,620]  WARN [QuartzScheduler_scheduler-prov011435933239135_ClusterManager] (JobStoreSupport.java:3298) - This scheduler instance (prov011435933239135) is still active but was recovered by another instance in the cluster.  This may cause inconsistent behavior.

"The most probable cause of such a cluster-related error is having cluster nodes with system clock not synchronized."

Ref:
http://quartz-scheduler.org/documentation/faq

https://confluence.atlassian.com/display/JIRAKB/This+scheduler+instance+(%3Cnodename%3E)+is+still+active+but+was+recovered+by+another+instance+in+the+cluster

https://stackoverflow.com/questions/15701417/strange-behavior-of-quartz-in-cluster-configuration

web service client

Server is returning soap fault with http status 400

SOAP UI is displaying nicely

When using cxf Java client I get strange, misleading message:
org.apache.cxf.interceptor.Fault: Could not send Message.

On server side I see everything is as expected soap fault is returned with bad request status code 400.
After little debugging I found this in org.apache.cxf.transport.http.HTTPConduit class:

// "org.apache.cxf.transport.process_fault_on_http_400" property should be set in case a 

// soap fault because of a HTTP 400 should be returned back to the client (SOAP 1.2 spec)

So before making a call configure client:

Client client = ClientProxy.getClient(port);
client.getBus().setProperty("org.apache.cxf.transport.process_fault_on_http_400", true);

After that I get expected exception:
Caused by: org.apache.cxf.binding.soap.SoapFault: some validation message


One more example of calling web service with Apache CXF - now within the same project I'm doing integration test with Spock and SpringBootTest.
HelloWorldWs is interface annotated with @WebService

def setup() {
    JaxWsProxyFactoryBean proxyFactory = new JaxWsProxyFactoryBean()
    proxyFactory.setServiceClass(HelloWorldWs)
    this.pismaUrl = "http://localhost:" + this.port + "/services/pisma"

    proxyFactory.setAddress(pismaUrl)
    this.client = (HelloWorldWs) proxyFactory.create()

    Client client = ClientProxy.getClient(this.client);
    client.getBus().setProperty("org.apache.cxf.transport.process_fault_on_http_400", true);
}

web service client example

//web service interface to be called
@WebService(targetNamespace = "http://services.crm.ticketing.samara.com", name = "ticketingServicePortTypes")
@XmlSeeAlso({ObjectFactory.class})
public interface TicketingServicePortTypes {

// web service client that was generated with wsdltojava
// This class was generated by Apache CXF 3.0.3
TicketingService extends javax.xml.ws.Service

Java client that calls TicketingService, using basic authentication with u/p.
final TicketingService service = new TicketingService(); // should be singleton
final TicketingServicePortTypes port = service.getTicketingServicePort();
javax.xml.ws.BindingProvider bp = (BindingProvider) port;
log.trace("username: " + username + ", password: " + password);
bp.getRequestContext().put(BindingProvider.USERNAME_PROPERTY, username);
bp.getRequestContext().put(BindingProvider.PASSWORD_PROPERTY, password);
bp.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, "http://10.105.100.127:8445/");


Using spring cxf xml:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:task="http://www.springframework.org/schema/task" xmlns:jaxws="http://cxf.apache.org/jaxws"
xmlns:soap="http://cxf.apache.org/bindings/soap" xmlns:cxf="http://cxf.apache.org/core"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context
      http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/aop
    http://www.springframework.org/schema/aop/spring-aop.xsd
http://www.springframework.org/schema/task
http://www.springframework.org/schema/task/spring-task.xsd
http://cxf.apache.org/jaxws http://cxf.apache.org/schemas/jaxws.xsd
        http://cxf.apache.org/core http://cxf.apache.org/schemas/core.xsd
        http://cxf.apache.org/bindings/soap http://cxf.apache.org/schemas/configuration/soap.xsd
        ">




<!-- jaxws client to front end -->
<jaxws:client id="crmService"
serviceClass="com.samara.generated.crm.api.TicketingServicePortTypes"
address="${crm.url}" username="${crm.username}" password="${crm.password}">
<jaxws:binding>
<soap:soapBinding mtomEnabled="false" version="1.1" />
</jaxws:binding>
</jaxws:client>

</beans>


Generating random integers in a range

Random rand;
int randomNum = rand.nextInt((max - min) + 1) + min;

or

int randomNum = ThreadLocalRandom.current().nextInt(min, max + 1);


Ref:
stackoverflow - Generating random integers in a range with Java
Random - nextInt

Disjuction (OR operator)

Writing OR clause using criteria API.

In this case operation parameter can be passed as optional / varargs (arbitrary number of values ).

private void addOperationToCriteria(Criteria criteria, SomeOperation ... operation)
{
if (operation == null)
return;
if (operation.length == 1)
criteria.add(Restrictions.eq("operation", operation[0]));
else
{
Disjunction dis = Restrictions.disjunction(); // OR
for (SomeOperation o : operation)
{
dis.add(Restrictions.eq("operation", o));
}
criteria.add(dis);
}
}


Quartz with Spring and embedded database

I'm was testing quartz at localhost with H2 embedded database.
In this case I need quartz tables prepared.

In spring config define database and wha script to load:
<jdbc:embedded-database id="dataSource" type="H2">
<jdbc:script location="classpath:recycle/quartz-h2.sql" />
</jdbc:embedded-database>


Example of spring 3.2 integration with quart 1.8:

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns      = "http://www.springframework.org/schema/beans"
       xmlns:xsi  = "http://www.w3.org/2001/XMLSchema-instance"
       xmlns:task = "http://www.springframework.org/schema/task"

       xsi:schemaLocation = "http://www.springframework.org/schema/beans
                             http://www.springframework.org/schema/beans/spring-beans.xsd
                             http://www.springframework.org/schema/task
                             http://www.springframework.org/schema/task/spring-task.xsd">
                           
<!-- Setting pool to 10 threads -->
    <task:scheduler id="taskScheduler" pool-size="10"/>

    <!-- Scheduler -->
    <bean id="scheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
        <property name="configLocation" value="recycle/quartz.properties"/>
        <property name="dataSource" ref="dataSource"/>
        <property name="transactionManager" ref="transactionManager" />
        <property name="triggers">
        <list>
                <ref bean="txRecycleTrigger" />
            </list>
        </property>
        <!-- Fill the schedulerFactoryBeans scheduleContextMap with ALL spring
beans that shall be available for the Job implementations. The SpringBeanJobFactory will inject these beans into newly instantiated workers if they have a setter that corresponds to the beans key. This makes spring beans available even for (persisted and) deserialized Jobs. NOTE: This mechanism isn't supported by Spring 3.0 for the latest Quartz version 2.0.1 but for the previous version 1.8.5 -->
<property name="schedulerContextAsMap">
<map>
<entry key="recycleSimService" value-ref="RecycleSimService" />
<entry key="logEnabled" value="${job.logEnabled}" />
</map>
</property>
    </bean>
 
    <!-- Property ""MISFIRE_INSTRUCTION_FIRE_ONCE_NOW" means that The job is executed immediately after the scheduler discovers misfire situation.  -->
 
    <bean id="txRecycleTrigger" class="org.springframework.scheduling.quartz.CronTriggerFactoryBean">
        <property name="jobDetail"      ref  ="recycleSimJob" />
        <property name="cronExpression" value="${job.recycleScheduler.cronExpression}" />
        <property name="misfireInstructionName" value="MISFIRE_INSTRUCTION_FIRE_ONCE_NOW" />
    </bean>
    <bean id="recycleSimJob" class="org.springframework.scheduling.quartz.JobDetailFactoryBean">
      <property name="jobClass" value="com.samara.job.RecycleSimJob" />
    </bean>
 
</beans>



Job implementation

import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;
import org.quartz.StatefulJob;
import org.springframework.scheduling.quartz.QuartzJobBean;

public class RecycleSimJob extends QuartzJobBean implements StatefulJob
{
 private transient static final Logger log = LoggerFactory.getLogger(RecycleSimJob.class);

 private transient RecycleSimService recycleSimService;

 @Override
 protected void executeInternal(JobExecutionContext context) throws JobExecutionException
 {
  if (getRecycleSimService() == null)
  {
   log.warn("recycleSimService is not set!");
   return;
  }
  getRecycleSimService().recycleSims(new Date());
 }

 public RecycleSimService getRecycleSimService()
 {
  return recycleSimService;
 }

 public void setRecycleSimService(RecycleSimService recycleSimService)
 {
  this.recycleSimService = recycleSimService;
 }

}

Schedule job in real time

I'm using injected org.springframework.scheduling.quartz.SchedulerFactoryBean.

JobDetail jobDetail = new JobDetail("deactivation_" + orderId, DeactivateOrderJob.class);
Map<String, Long> params = new HashMap<String, Long>();
params.put("orderId", orderId);
jobDetail.setJobDataMap(new JobDataMap(params));
final SimpleTrigger simpleTrigger = new SimpleTrigger("deactivationTrigger_" + orderId, expiryTime);
schedulerFactory.getScheduler().scheduleJob(jobDetail, simpleTrigger);

DeactivateOrderJob will be waken and it will read orderId from map.

@Override
protected void executeInternal(JobExecutionContext context) throws JobExecutionException
{
     Map params = context.getJobDetail().getJobDataMap();
     Long orderId = (Long) params.get("orderId");
     .....
}