通过使用Spring Batch逐行插入数据来修复列表大小问题

Question

我有一个文件，该文件包含超过80k的数据行，它们通过解析并插入到表中。该文件是用管道分隔的，我只希望每行的前3个字段。当我测试spring boot程序时，我使它可以处理10行数据，但是当我尝试使用真实文件时，在ArrayIndexOutofBoundExcepetion: 1变量中得到了String sql。我相信有很多数据正在尝试存储在列表中。

我需要帮助来更改代码以一次读取一行，然后插入数据库中，直到覆盖文件中的所有行为止。我认为spring batch可以帮助我做到这一点，但是我并不熟悉它，也不会用我目前的知识和经验来编辑代码来逐行插入代码。

解析文件的方法

private List<ZygateEntity> parseZygateData() throws IOException {
        String filePath = "C:\\DEV\\Test_file.xlsx";

        List<String> lines = Files.readAllLines(Paths.get(filePath),Charset.forname("UTF-8"));

        // remove date and amount
        lines.remove(0);
        lines.remove(lines.size() - 1);

        return lines.stream()
                .map(s -> s.split("[|]")).map(val -> new ZygateEntity(val[0],val[1],val[2])).collect(Collectors.toList());
    }

插入表格的方法

public void insertZygateData(List<ZygateEntity> parseData) {
    String sql = "INSERT INTO Landing.midrange_xygate_load (account_name,command_name,system_name)"+
            "VALUES (:account_name,:command_name,:system_name)";

    for (ZygateEntity zygateInfo : parseData){
        SqlParameterSource source = new MapSqlParameterSource("account_name", zygateInfo.getAccountName())
                .addValue("command_name", zygateInfo.getCommandName())
                .addValue("system_name", zygateInfo.getSystemName());
        namedParameterJdbcTemplate.update(sql, source);
    }
}

堆栈跟踪

java.lang.ArrayIndexOutOfBoundsException: 1
    at com.test.dao.impl.ZygateDataLoadDAOImpl.lambda$parseZygateData$1(ZygateDataLoadDAOImpl.java:73)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at com.test.dao.impl.ZygateDataLoadDAOImpl.parseZygateData(ZygateDataLoadDAOImpl.java:74)
    at com.test.dao.impl.ZygateDataLoadDAOImpl$$FastClassBySpringCGLIB$$79d44833.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
    at com.test.dao.impl.ZygateDataLoadDAOImpl$$EnhancerBySpringCGLIB$$2892f5cf.parseZygateData(<generated>)
    at com.test.service.ZygateService.runZygateJob(ZygateService.java:39)
    at com.test.AlmxygatejobApplication.run(AlmxygatejobApplication.java:29)
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:784)
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:768)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
    at com.test.AlmxygatejobApplication.main(AlmxygatejobApplication.java:19)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)

Answer 1

看来您的输入文件有问题。检查您的输入文件，好像是一行（或多行），它有一个或更少的“ |” （您在|上分割字符串）。或检查文件末尾是否还有其他一些换行符。