在没有StackOverflowError的情况下序列化Java对象

问题描述 投票：8回答：3

我有一个非常大的Java对象，它在内存中表示带有顶点和边的图形。每个顶点都有一个与其连接的其他顶点的ArrayList（并且具有HashMap数据结构以及其他用途）。图形可以有几千个顶点，还有更多边。

当尝试使用Java的内置序列化（implements Serializable等）序列化图形时，我总是遇到StackOverflowError。将图形的其他属性设置为transient没有帮助，也没有将堆栈大小设置得更大（即-Xss1g或-Xss512m）。

我不认为我需要制作一个自定义writeObject方法，因为ArrayList和HashMap已经有自己的实现，这些都是在序列化时调用的。

我的问题是：有没有办法在没有获得StackOverflowError的情况下序列化已经在内存中的大型Java对象？

编辑：这是堆栈跟踪：

Exception in thread "main" java.lang.StackOverflowError
at java.lang.reflect.Method.invoke(Method.java:575)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:950)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1482)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
// Many more lines after this

以下是我的Graph课程的概述：

public class Graph implements Serializable {

/**
 * 
 */
private static final long serialVersionUID = -2632163054149021990L;
private ArrayList<Vertex> vertices;

private HashMap<Integer, Set<Vertex>> map;

public Graph(int rowMax, int colMax)
{
    map = new HashMap<Integer, Set<Vertex>>();

    this.vertices = new ArrayList<Vertex>();
}

public void connectVertices(Vertex u, Vertex v)
{
    u.addNeighbor(v);
    v.addNeighbor(u);
}

// other unrelated methods after this

这是我的Vertex课程：

public class Vertex implements Serializable {

/**
 * 
 */
private static final long serialVersionUID = 8520500010710631610L;
public int row;
public int col;
private ArrayList<Vertex> neighbors; // may change this to Set<Vertex>

public Vertex(int i, int j)
{
    this.row = i;
    this.col = j;
    this.neighbors = new ArrayList<Vertex>();
}

public boolean addNeighbor(Vertex v)
{
    this.neighbors.add(v);
    return true;
}

// unrelated methods after this

编辑2：此外，对于较小尺寸但有“邻居”的图形，没有这个问题。

java serialization stack-overflow
3个回答
4
投票

如果图形的深度太大而无法处理默认序列化，则序列化将抛出StackOverflowError。这是由于默认序列化在解析图形时递归地序列化每个节点。

平面结构将正常工作（例如，具有2000个子节点的父节点），但深层结构将失败（例如，具有2000个后代级别的节点）。

例如。堆栈溢出如下：

public class Node implements Serializable
{
    private ArrayList<Node> nodes = new ArrayList<Node>();

    public static void main(String[] args) throws Exception
    {
        Node node = new Node();
        int depth = 3000;

        // Add nodes chained down to specified depth
        Node last = node;
        for (int i = 0; i < depth; i++)
        {
            Node temp = new Node();
            last.nodes.add(temp);
            last = temp;
        }

        System.out.println("starting");

        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        ObjectOutputStream out = new ObjectOutputStream(bos);
        // Below line will cause a stack overflow.
        out.writeObject(node);

        System.out.println("done");
    }
}

您需要减少图形的深度以限制递归序列化调用的数量，或者编写自定义序列化来解决此问题。自定义序列化本质上需要是非递归的，不幸的是，初看起来似乎并非易事。

0
投票

我遇到了类似的问题。经过多次打猎，我发现了一个Kryo叉子，用于处理深层嵌套的物体。通过https://github.com/EsotericSoftware/kryo/issues/103，克隆和mvn clean install https://github.com/romix/kryo/tree/kryo-2.23-continuations。它目前是com.esotericsoftware.kryo:kryo:2.23-SNAPSHOT。

（附注：关于SO的问题有很多可以用这个回答。我应该发布答案的副本（如https://stackoverflow.com/a/43327778/513038），或者评论这个答案的链接，或者将问题标记为重复指向这个问题，或者是什么？）

0
投票

我知道这已经晚了几年，但是当我试图自己解决问题时，我通过谷歌遇到了这个问题。

我想要一个比编写自定义序列化代码要少得多的代码的解决方案。

问题是节点连接到节点连接到节点。可以从任何给定节点到达网络的很大一部分 - 对于无向图，它是每个节点。

当您尝试序列化节点时，每个可达节点将最终在堆栈上。

最简单的解决方案不是将节点直接连接到其邻居，而是添加一个间接层：

class Node{

    private final Map<NodeRef, Connection> forwardConnections = new HashMap<>();
    private final Map<NodeRef, Connection> reverseConnections = new HashMap<>();

    ...
}

class Connection{

    private final NodeRef source;
    private final NodeRef dest;
    private final ConnectionMetadata meta;

    public Connection(Node source, Node dest, ConnectionMetadata meta){
        this.source = new NodeRef(source);
        this.dest = new NodeRef(dest);
    }

    public Node getSource(){
        return source.resolve();
    }

    public Node getDest(){
        return dest.resolve();
    }

    public ConnectionMetadata getMeta(){
        return meta;
    }

}

public class NodeRef{

    private transient Node node;
    private final Network network;
    private final int[] uid;

    public NodeRef(Node node){
        this.node = node;
        this.network = node.getNetwork();
        this.uid = node.getUID();
    }

    //Won't be called during deserialisation
    public Node resolve(){
        if(node == null){
            node = network.resolve(uid);
        }
        return node;
    }

    //Will be called when connections maps are deserialised.
    public boolean equals(Object o){
        //you might also want to check that the networks are equal
        return (o instanceof NodeRef) && Arrays.equals(uid, ((NodeRef)o).uid);
    }


    //Will be called when connections maps are deserialised.
    public int hashCode(){
        return Arrays.hashCode(uid);
    }
}
最新问题
© www.soinside.com 2019 - 2024. All rights reserved.