Clustered Quartz setup

This blog post is about configuring clustered Quartz with static configuration i.e. the job configuration remains the same for each software version.

Running clustered Quartz is easy, but it's not easy to setup the cluster. The problem is that how do you create and update Quartz jobs of a running system. In some systems, you might do the configuration by hand directly to database. In other systems, you want to avoid manual steps and let the system update itself automatically during version updates. Because I normally automate the installation of the projects I'm involved with, I want that the system does job maintenance automatically without manual database updating.

Quartz provides several examples for different kinds of use cases. One of the examples (example 13 in version 2.2.3) cover clustered Quartz. The example assumes that the processes are started so that the person or software that starts the process, knows when to clean existing jobs and when not. I don't find this approach very useful in real life, although it is of course a simple example.

When the Quartz using process starts, the process should somehow know when to update the jobs. This is somewhat difficult problem to solve. You shouldn't update jobs while they're running, because you might end up running the same job twice (even simultaneously), which is not nice behavior. You should also create the job configuration exactly once per system version (unless you're managing dynamically changing jobs).

One way to solve the job updating problem is that one of the nodes in the cluster knows that it's the primary (or lead or whatever) node and that it's also the first running node in the cluster. Only primary node is allowed to update the job configuration and other nodes are running against the configuration created by primary node. In my solution, primary node removes job data completely and recreates it based on the current configuration.

public void initialize() throws SchedulerException {
  if (quartzNodeDao.isPrimary()) {
    // only the "lead" process gets to update the jobs. Jobs are updated by removing all job data and recreating it    

    scheduler.clear();
    final int SECOND = 1 * 1000;
    this.demoTrigger = createTrigger(SECOND);
    JobDetail importJobDetail = JobBuilder
        .newJob(DemoJob.class)
        .withIdentity(DEMO_JOB, DEMO_GROUP)
        .storeDurably()
        .requestRecovery()
        .build();

    System.out.println("Chosen as master, created new scheduled job");
    scheduler.scheduleJob(importJobDetail, demoTrigger);
  } else {
    System.out.println("Not master, skipping job creation");
  }
}

The key thing in this solution is that each starting cluster node tries to insert the GIT software revision of the current release to the database and if the node is successful, it's the primary node.

public boolean isPrimary() {
  String sql = "INSERT INTO build_revision (revision) VALUES (?);";
  try(Connection connection = dataSource.getConnection()) {
    connection.setAutoCommit(false);
    try (PreparedStatement ps = connection.prepareStatement(sql)) {
      connection.setAutoCommit(false);
      ps.setString(1, revision);
      ps.executeUpdate();
      connection.commit();
      // insert was successful, so the caller is primary node      return true;
    }
  } catch (Exception e) {
    // Should be handled better in real life, but handling ignored in demo.
    // Most likely insert failed, because entry exists already, so caller
    // is not primary node.

    return false;
  }
}

After the Quartz job configuration has been done, it's irrelevant which one of the nodes is primary, because the job configuration won't change before next version. It's assumed that all the nodes are stopped in the installation so that only one version of the software is running at any given point of time. This solution works very nicely at least in AWS ECS environment.

Here's link to complete project: https://github.com/perttuta/quartz