是否有更好的方法来减少长时间运行的石英作业的执行时间？

Question

我有一个cron作业，该作业将在夜间某个时间触发，该作业会从数据库中获取大约100k的大量productId，并从一项耗时约700ms的服务中获取1个productId的API的所有产品的productInfo。

CronJob

public class GetProducts extends QuartzJobBean {

    @Autowired
    private ProductClient productClient;

    @Override
    protected void executeInternal(JobExecutionContext context) throws JobExecutionException {

        List<Long> ProductsIds = fetchAllProductIdsFromDb();

        Response<ProductClientResponse> response = null;

        for (Long productId : ProductsIds) {
            ProductClientRequestBody requestBody = new ProductClientRequestBody();
            requestBody.putIdInsideTheRequestBody(productId);
            response = productClient.getResult(requestBody).execute();

            if (response != null && response.isSuccessful()) {
                log.info("We have got successful response for {}", i);
            }
        }
    }
}

这里productClient是服务的翻新客户端。因此，这项工作在技术上将需要5个小时才能完成。

#============================================================================
# Configure Main Scheduler Properties
#============================================================================
org.quartz.scheduler.instanceName=QuartzScheduler
org.quartz.scheduler.instanceId=AUTO
#============================================================================
# Configure ThreadPool
#============================================================================
org.quartz.threadPool.class=org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount=80
org.quartz.threadPool.threadPriority=5
#============================================================================
# Configure JobStore
#============================================================================
org.quartz.jobStore.misfireThreshold=60000
org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.useProperties=true
org.quartz.jobStore.dataSource=myDS
org.quartz.jobStore.tablePrefix=QRTZ_
org.quartz.jobStore.isClustered=true
org.quartz.jobStore.clusterCheckinInterval=20000
#============================================================================
# Configure Datasources
#============================================================================
org.quartz.dataSource.myDS.driver=com.mysql.jdbc.Driver
org.quartz.dataSource.myDS.maxConnections=10
org.quartz.dataSource.myDS.validationQuery=select 0 from dual
org.quartz.scheduler.batchTriggerAcquisitionMaxCount =10
org.quartz.dataSource.myDS.URL=jdbc:mysql://127.0.0.1:3306/quartz
org.quartz.dataSource.myDS.user=root
org.quartz.dataSource.myDS.password=root

这是我的石英特性文件。我想知道是否有更好的方法来获取所有100k产品的ProductInfo。

一种方法

我为所有ProductId安排了10万个工作。而在集群环境中运行的石英将根据可用实例进行调度。

org.quartz.threadPool.threadCount = 80-此属性指出，在该服务的一个实例中，最多80个线程可以占用这些作业。 是吗？如果我有2个实例在运行，那么至少可以同时运行100-160个作业。我的方法正确吗？这样可以大大节省时间。

还有其他更好的方法吗？

Answer 1

注意：-这是为了说明可以进一步简化代码的方法（我尚未测试代码，可能会出现语法错误）

public class ProductsJob extends QuartzJobBean {

@Autowired
private ProductClient productClient;

@Override
protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
    List<Long> ProductsIds = fetchAllProductIdsFromDb();
    ProductsExecutor productsExecutor=new ProductsExecutor();
    productsExecutor.runTasks(ProductsIds);
}


}

public class ProductsExecutor {

private int threadPoolSize=10;
private ExecutorService executorService;

public ProductsExecutor()
{
    executorService =Executors.newFixedThreadPool(threadPoolSize);
}

public void runTasks(List<Integer> productIds)
{
    //Loop and execute the task and wait for them it necessary
    ProductsWorker worker=new ProductsWorker();
    worker.setProductId(productId);
     executorService.invokeAll(worker);
}


}

public class ProductsWorker implements Runnable {

private int productId;
@Override
public void run() {
    ProductClientRequestBody requestBody = new ProductClientRequestBody();
    requestBody.putIdInsideTheRequestBody(productId);
    response = productClient.getResult(requestBody).execute();

    if (response != null && response.isSuccessful()) {
        log.info("We have got successful response for {}");
    }

}

public void setProductId(int productId)
{
    this.productId=productId;
}

}