如何在Vaadin中分段读取数据并导出为PDF

Question

我在 MySQL 中有 20 亿条记录。

我使用具有延迟加载功能的 Vaadin 框架 (https://vaadin.com/docs/v23/components/grid#lazy-loading) 和过滤器（带回调）：

private void showData(Filter filter) {
        positionGrid.setItems(query -> {
            var vaadinSortOrders = query.getSortOrders();
            var springSortOrders = new ArrayList<Sort.Order>();
            for (QuerySortOrder so : vaadinSortOrders) {
                String colKey = so.getSorted();
                if (so.getDirection() == SortDirection.ASCENDING) {
                    springSortOrders.add(Sort.Order.asc(colKey));
                } else if (so.getDirection() == SortDirection.DESCENDING) {
                    springSortOrders.add(Sort.Order.desc(colKey));
                }
            }
            return positionService.getAll(
                    filter,
                    PageRequest.of(query.getPage(),
                            query.getPageSize(),
                            Sort.by(springSortOrders))
            ).stream();
        });
    }

我需要导出PDF报告中的数据（我使用itextpdf库）。但我收到一个错误：

java.lang.OutOfMemoryError：Java 堆空间

我的代码：

private final PositionGrid positionGrid;

...

positionGrid.getLazyDataView().getItems().forEach(elem -> {
            table.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph(String.valueOf(num.incrementAndGet())).setTextAlignment(TextAlignment.CENTER)));
            table.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph(elem.getSsi().toString())).setTextAlignment(TextAlignment.CENTER));
            table.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph(PositionGrid.formatter.format(elem.getDatetime())).setTextAlignment(TextAlignment.CENTER)));
            table.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph(elem.getPosx().toString())).setTextAlignment(TextAlignment.CENTER));
            table.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph(elem.getPosy().toString())).setTextAlignment(TextAlignment.CENTER));
            table.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph((elem.getVelocity() != null) ? elem.getVelocity().toString() : "")).setTextAlignment(TextAlignment.CENTER));
            table.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph((elem.getRssi() != null) ? elem.getRssi().toString() : "")).setTextAlignment(TextAlignment.CENTER));
            table.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph((elem.getPathDelay() != null) ? elem.getPathDelay().toString() : "")).setTextAlignment(TextAlignment.CENTER));
            table.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph((elem.getBsId() != null) ? elem.getBsId().toString() : "")).setTextAlignment(TextAlignment.CENTER));
        });

请告诉我。如何读取部分数据（我需要将其写入 PDF 文件）？

这是我的实体：

@Entity
@Table(name = "Positions")
@Getter
@Setter
public class Position {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    @Column(name = "id", nullable = false)
    private Long id;

    @Column(name = "datetime", nullable = false)
    private Instant datetime;

    @Column(name = "posx", nullable = false)
    private Double posx;

    @Column(name = "posy", nullable = false)
    private Double posy;

    @Column(name = "velocity")
    private Double velocity;

    @Column(name = "rssi")
    private Short rssi;

    @Column(name = "pathDelay")
    private Short pathDelay;

    @Column(name = "bs_id")
    private Integer bsId;

    @Column(name = "ssi", nullable = false)
    private Long ssi;
}

Answer 1

感谢 cfrick 的帮助。

我的页面有很好的报告，并且我使用带有过滤器和 API 标准的延迟加载：

我的方法：

positionService.getAll(
                    filter,
                    PageRequest.of(query.getPage(),
                            query.getPageSize(),
                            Sort.by(springSortOrders))
            ).stream();

我的服务：

@Service
public class PositionCustomService extends CustomService<Position> {

    private final static Logger LOGGER = LogManager.getLogger(PositionCustomService.class);

    private final PositionRepository repository;

    @Autowired
    public PositionCustomService(PositionRepository repository) {
        this.repository = repository;
    }

    public Page<Position> getAll(PageRequest pageRequest) {
        return repository.findAll(pageRequest);
    }

    public List<Position> getAll(Filter filter) {
        return repository.findAll(bySearchWithFilter(filter, Position.class));
    }

    public Page<Position> getAll(Filter filter, PageRequest pageRequest) {
        return repository.findAll(bySearchWithFilter(filter, Position.class), pageRequest);
    }

    public Map<LocalDate, Long> getCountByDays(int days) {
        return repository.countByDays(days);
    }
}

我的定制服务：

public abstract class CustomService<T> {

    public static <T> Specification<T> bySearchWithFilter(Filter filter, Class<T> clazz) {

        return (Root<T> root, CriteriaQuery<?> query, CriteriaBuilder criterailBuilder) -> {

            List<Predicate> predicates = new ArrayList<>();

            // Add Predicates for tables to be joined
            List<JoinColumnProps> joinColumnProps = filter.getJoinColumnProps();

            if (joinColumnProps != null && !joinColumnProps.isEmpty()) {
                for (JoinColumnProps joinColumnProp : joinColumnProps) {
                    addJoinColumnProps(predicates, joinColumnProp, criterailBuilder, root);
                }
            }

            List<AttributeFilter> attributeFilters = filter.getAttributeFilters();

            if (attributeFilters != null && !attributeFilters.isEmpty()) {

                for (final AttributeFilter attributeFilter : attributeFilters) {
                    addPredicates(predicates, attributeFilter, criterailBuilder, root);
                }
            }

            if (predicates.isEmpty()) {
                return criterailBuilder.conjunction();
            }

            return criterailBuilder.and(predicates.toArray(new Predicate[0]));
        };
    }

    private static <T> void addJoinColumnProps(List<Predicate> predicates, JoinColumnProps joinColumnProp,
                                               CriteriaBuilder criterailBuilder, Root<T> root) {

        AttributeFilter attributeFilter = joinColumnProp.getSearchAttrFilter();
        Join<Object, Object> joinParent = root.join(joinColumnProp.getJoinColumnName());

        String property = attributeFilter.getField().getNameField();
        Path expression = joinParent.get(property);

        addPredicate(predicates, attributeFilter, criterailBuilder, expression);
    }

    private static <T> void addPredicates(List<Predicate> predicates, AttributeFilter attributeFilter,
                                          CriteriaBuilder criterailBuilder, Root<T> root) {
        String property = attributeFilter.getField().getNameField();
        Path expression = root.get(property);

        addPredicate(predicates, attributeFilter, criterailBuilder, expression);
    }

    private static void addPredicate(List<Predicate> predicates, AttributeFilter attributeFilter,
                                     CriteriaBuilder criterailBuilder, Path expression) {
        switch (attributeFilter.getOperator()) {
            case "=":
                predicates.add(criterailBuilder.equal(expression, attributeFilter.getValue()));
                break;
            case "LIKE":
                predicates.add(criterailBuilder.like(expression, "%" + attributeFilter.getValue() + "%"));
                break;
            case "IN":
                predicates.add(criterailBuilder.in(expression).value(attributeFilter.getValue()));
                break;
            case ">":
                predicates.add(criterailBuilder.greaterThan(expression, (Comparable) attributeFilter.getValue()));
                break;
            case "<":
                predicates.add(criterailBuilder.lessThan(expression, (Comparable) attributeFilter.getValue()));
                break;
            case ">=":
                predicates.add(criterailBuilder.greaterThanOrEqualTo(expression, (Comparable) attributeFilter.getValue()));
                break;
            case "<=":
                predicates.add(criterailBuilder.lessThanOrEqualTo(expression, (Comparable) attributeFilter.getValue()));
                break;
            case "!":
                predicates.add(criterailBuilder.notEqual(expression, attributeFilter.getValue()));
                break;
            case "IsNull":
                predicates.add(criterailBuilder.isNull(expression));
                break;
            case "NotNull":
                predicates.add(criterailBuilder.isNotNull(expression));
                break;
            default:
                throw new IllegalArgumentException(attributeFilter.getOperator() + " is not a valid predicate");
        }
    }
}

我的仓库：

    @Repository
    public interface PositionRepository extends JpaRepository<Position, Long>, JpaSpecificationExecutor<Position>, PositionRepositoryCustom {
    }


    @Repository
    public interface PositionRepositoryCustom {
    
        Map<LocalDate, Long> countByDays(int days);
    }

public class PositionRepositoryImpl implements PositionRepositoryCustom {

    @PersistenceContext
    private EntityManager entityManager;

    @Override
    public Map<LocalDate, Long> countByDays(int days) {

        CriteriaBuilder cb = entityManager.getCriteriaBuilder();
        CriteriaQuery cq = cb.createQuery();

        Root<Position> position = cq.from(Position.class);
        List<Predicate> predicates = new ArrayList<>();

        Instant endDate = Instant.now();
        Instant startDate = endDate.minus(days, ChronoUnit.DAYS)
                .truncatedTo(ChronoUnit.DAYS);

        Expression dateWithoutTimeExp = cb.function("date", Instant.class, position.get(PositionModelFilter.MESSAGE_DATE.getNameField()));

        cq.multiselect(dateWithoutTimeExp, cb.count(position.get(PositionModelFilter.ID.getNameField())));

        cq.groupBy(dateWithoutTimeExp);

        predicates.add(cb.between(position.get(PositionModelFilter.MESSAGE_DATE.getNameField()),
                startDate, endDate));
        cq.where(predicates.toArray(new Predicate[0]));

        Map<LocalDate, Long> resMap = new HashMap<>();
        entityManager.createQuery(cq).getResultList().forEach(elem -> {
            Object[] mas = (Object[])elem;
            resMap.put(((Date)mas[0]).toLocalDate(), (long)mas[1]);
        });

        return resMap;
    }
}

public interface JpaSpecificationExecutor<T> {
    Optional<T> findOne(@Nullable Specification<T> spec);

    List<T> findAll(@Nullable Specification<T> spec);

    Page<T> findAll(@Nullable Specification<T> spec, Pageable pageable);

    List<T> findAll(@Nullable Specification<T> spec, Sort sort);

    long count(@Nullable Specification<T> spec);

    boolean exists(Specification<T> spec);
}

我找到了如何处理大数据的唯一解决方案：我开始进行一些有限的查询：

int index = 0;
        int limit = 10_000;

        while (true) {
            log.debug("Загружается партия записей: " + index);
            List<Position> res = positionService.getAll(
                            positionFilterView.getFilter(),
                            PageRequest.of(index,
                                    limit,
                                    Sort.by(Sort.Order.asc("datetime")))
                    ).stream().collect(Collectors.toList());

            if (res.isEmpty()) break;

            Table dataTable = new Table(UnitValue.createPercentArray(columnWidths));
            res.forEach(elem -> {
                dataTable.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph(String.valueOf(num.incrementAndGet())).setTextAlignment(TextAlignment.CENTER)));
                dataTable.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph(elem.getSsi().toString())).setTextAlignment(TextAlignment.CENTER));
                dataTable.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph(PositionGrid.formatter.format(elem.getDatetime())).setTextAlignment(TextAlignment.CENTER)));
                dataTable.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph(elem.getPosx().toString())).setTextAlignment(TextAlignment.CENTER));
                dataTable.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph(elem.getPosy().toString())).setTextAlignment(TextAlignment.CENTER));
                dataTable.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph((elem.getVelocity() != null) ? elem.getVelocity().toString() : "")).setTextAlignment(TextAlignment.CENTER));
                dataTable.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph((elem.getRssi() != null) ? elem.getRssi().toString() : "")).setTextAlignment(TextAlignment.CENTER));
                dataTable.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph((elem.getPathDelay() != null) ? elem.getPathDelay().toString() : "")).setTextAlignment(TextAlignment.CENTER));
                dataTable.addCell(new Cell().setTextAlignment(TextAlignment.CENTER).add(new Paragraph((elem.getBsId() != null) ? elem.getBsId().toString() : "")).setTextAlignment(TextAlignment.CENTER));
            });

            pdfExporter.getDocument().add(dataTable);
            //pdfExporter.getDocument().flush();

            index++;
        }

唯一的问题是1分钟内将1万条记录写入文件。已经过去很长一段时间了（