使用NLP Stanford时如何处理与内存相关的异常？

Question

我正在尝试通过以下链接运行Class：Word2VecSentimentRNN：

https://github.com/deeplearning4j/dl4j-examples/blob/master/dl4j-examples/src/main/java/org/deeplearning4j/examples/recurrent/word2vecsentiment/Word2VecSentimentRNN.java

示例很大，因此此处给出示例链接。我还从以下链接下载了样本矢量文件：https://github.com/mmihaltz/word2vec-GoogleNews-vectors我收到以下错误：

Exception in thread "main" java.lang.OutOfMemoryError: Cannot allocate 3103474 + 3600000000 bytes (> Pointer.maxBytes)
    at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:484)
    at org.bytedeco.javacpp.Pointer.init(Pointer.java:118)
    at org.bytedeco.javacpp.FloatPointer.allocateArray(Native Method)
    at org.bytedeco.javacpp.FloatPointer.<init>(FloatPointer.java:68)
    at org.nd4j.linalg.api.buffer.BaseDataBuffer.<init>(BaseDataBuffer.java:457)
    at org.nd4j.linalg.api.buffer.FloatBuffer.<init>(FloatBuffer.java:57)
    at org.nd4j.linalg.api.buffer.factory.DefaultDataBufferFactory.createFloat(DefaultDataBufferFactory.java:238)
    at org.nd4j.linalg.factory.Nd4j.createBuffer(Nd4j.java:1201)
    at org.nd4j.linalg.factory.Nd4j.createBuffer(Nd4j.java:1176)
    at org.nd4j.linalg.api.ndarray.BaseNDArray.<init>(BaseNDArray.java:230)
    at org.nd4j.linalg.cpu.nativecpu.NDArray.<init>(NDArray.java:111)
    at org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory.create(CpuNDArrayFactory.java:247)
    at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:4261)
    at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:4227)
    at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:3501)
    at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.readBinaryModel(WordVectorSerializer.java:219)
    at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.loadGoogleModel(WordVectorSerializer.java:118)
    at com.nyu.sentimentanalysis.core.Word2VecSentimentRNN.run(Word2VecSentimentRNN.java:77)

我已尝试使用参数-Xmx2g and -Xms2g启动该应用程序。甚至不时更改值以检查它是否有用或起作用。请让我知道我该怎么办。在这里被锁起来。

Answer 1

我在运行标准Word2vec代码时遇到了这个问题，并且在使用OutOfMemory后系统死了。

以下设置对我有用，以使用Word2vec预训练模型为基于DL4J / ND4J的应用维持长期的生产负荷

java -Xmx2G -Dorg.bytedeco.javacpp.maxbytes = 6G -Dorg.bytedeco.javacpp.maxphysicalbytes = 6G