Cassandra - 由于java.lang.OutOfMemoryError：直接缓冲区内存，节点无法重启

问题描述 投票：1回答：1

我使用最新的C* java driver摄取20 GB的数据。我有一个5节点C *集群。正在摄取数据的客户端应用程序正在不属于C *集群（但是属于同一LAN的一部分）的本地节点上运行。我也在我摄取的桌子上使用cassandra-lucene-index索引。 [cqlsh 5.0.1 | Cassandra 3.0.8 | CQL规范3.4.0 |原生协议v4]

我的Java客户端应用程序以下列方式工作：

  1. 解析json文件
  2. 将构建一个PreparedStatement
  3. 我的文件中每个元素的一个BoundStatement使用executeAsync发送到C *集群[注意：有很多行被发送]

大约一半时间，我的两个节点死了，/var/log/cassandra/system.log显示

    ERROR [CompactionExecutor:1] JVMStabilityInspector.java:140 - JVM state determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_91]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.8.0_91]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_91]
        at org.apache.cassandra.utils.memory.BufferPool.allocate(BufferPool.java:108) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.utils.memory.BufferPool.access$1000(BufferPool.java:45) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.utils.memory.BufferPool$LocalPool.allocate(BufferPool.java:387) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.utils.memory.BufferPool$LocalPool.access$000(BufferPool.java:314) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:120) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:92) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.io.util.RandomAccessReader.allocateBuffer(RandomAccessReader.java:87) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.access$100(CompressedRandomAccessReader.java:38) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader$Builder.createBuffer(CompressedRandomAccessReader.java:275) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:74) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:59) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader$Builder.build(CompressedRandomAccessReader.java:283) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:145) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.io.util.SegmentedFile.createReader(SegmentedFile.java:133) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1711) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(AbstractSSTableIterator.java:93) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:46) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:36) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:62) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:580) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:492) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at com.stratio.cassandra.lucene.IndexService.read(IndexService.java:618) ~[cassandra-lucene-index-plugin-3.0.8.0.jar:na]
        at com.stratio.cassandra.lucene.IndexWriterWide.finish(IndexWriterWide.java:89) ~[cassandra-lucene-index-plugin-3.0.8.0.jar:na]
        at org.apache.cassandra.index.SecondaryIndexManager$IndexGCTransaction.commit(SecondaryIndexManager.java:958) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.compaction.CompactionIterator$1$1.onMergedRows(CompactionIterator.java:197) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:484) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:446) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:220) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:159) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:428) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:288) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:128) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:111) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:149) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263) ~[apache-cassandra-3.0.8.jar:3.0.8]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_91]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]

当我尝试重新启动两个节点时，会发生同样的异常。然后我决定调整MAX_HEAP_SIZE中的HEAP_NEWSIZE和cassandra-env.sh但节点只会耗尽所有可用内存然后再次死亡。我有一个82gb的ram节点和一个48gb的ram死了。我为4G, 8G, 24G尝试了MAX_HEAP_SIZE和800M, 2G的值HEAP_NEWSIZE，但他们无法重启。

system.log文件还显示堆的转储：

Heap dump file created

INFO  [CompactionExecutor:1] 2016-08-14 13:43:44,223 HeapUtils.java:136 -
 num     #instances         #bytes  class name
----------------------------------------------
   1:         20879     2883199896  [I
   2:       5228765      334640960  org.apache.cassandra.utils.btree.BTreeSearchIterator
   3:       1836788      196567272  [B
   4:       3625993      163640704  [Ljava.lang.Object;
   5:       1260125       60486000  java.nio.HeapByteBuffer
   6:       1686322       53962304  org.apache.cassandra.utils.MergeIterator$Candidate
   7:        412664       52984560  [C
   8:        965525       38621000  org.apache.cassandra.db.rows.BufferCell
   9:       1511370       36272880  java.util.ArrayList
  10:        768459       30738360  org.apache.cassandra.db.rows.BTreeRow$Builder
  11:        382981       30638480  org.apache.cassandra.io.compress.CompressedRandomAccessReader
  12:        848641       27156512  java.util.RandomAccessSubList
  13:        848467       27150944  java.util.AbstractList$ListItr
  14:        394468       25245952  java.nio.DirectByteBuffer
  15:        387495       24799680  java.nio.DirectByteBufferR
  16:        774559       24785888  org.apache.cassandra.utils.btree.BTree$Builder
  17:        382823       24500672  org.apache.cassandra.db.columniterator.SSTableIterator$ForwardIndexedReader
  18:        486500       23352000  org.apache.cassandra.db.columniterator.SSTableIterator
  19:        383067       21451752  org.apache.cassandra.db.ClusteringPrefix$Deserializer
  20:        667956       21374592  org.apache.cassandra.db.rows.BTreeRow
  21:        853186       20476464  java.util.Arrays$ArrayList
  22:        848462       20363088  java.util.SubList$1
  23:        486689       19467560  org.apache.cassandra.db.rows.SerializationHelper
  24:        486319       19452760  org.apache.cassandra.db.filter.ClusteringIndexNamesFilter$1
  25:        383067       18387216  org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer
  26:        404602       16184080  org.apache.cassandra.utils.MergeIterator$ManyToOne
  27:        487237       15591584  java.util.TreeMap$KeyIterator
  28:        382823       15312920  org.apache.cassandra.db.columniterator.AbstractSSTableIterator$IndexState
  29:        376176       15047040  sun.misc.Cleaner
  30:        469635       15028320  org.apache.cassandra.db.rows.EncodingStats
  31:        464757       14872224  com.google.common.collect.Iterators$11
  32:        404602       13837016  [Lorg.apache.cassandra.utils.MergeIterator$Candidate;
  33:        550947       13222728  java.lang.Long
  34:        404502       12598432  [Lorg.apache.cassandra.db.rows.Row;
  35:        768640       12298240  org.apache.cassandra.db.rows.BTreeRow$Builder$CellResolver
  36:        375337       12010784  java.nio.DirectByteBuffer$Deallocator
  37:        250019       10000760  org.apache.cassandra.db.rows.Row$Merger
  38:        250019       10000760  org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer
  39:        312161        9989152  com.google.common.collect.RegularImmutableList
  40:        154609        9894976  org.apache.cassandra.db.SinglePartitionReadCommand
  41:        383092        9194208  org.apache.cassandra.cache.KeyCacheKey
  42:        275673        8821536  java.util.ArrayList$Itr
  43:        155947        8733032  java.util.LinkedHashMap
  44:        344055        8257320  java.lang.Double
  45:        157837        7576176  java.util.TreeMap
  46:        314112        7538688  com.google.common.collect.Iterators$12
  47:        302346        7256304  com.google.common.collect.Collections2$TransformedCollection
  48:        125865        7048440  java.util.stream.ReferencePipeline$Head
  49:        125057        7003192  org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator
  50:        154483        6179320  com.stratio.cassandra.lucene.IndexWriterWide
  51:        383788        6140608  java.util.zip.CRC32
  52:        377473        6039568  java.lang.Integer
  53:        250019        6000456  org.apache.cassandra.db.rows.Row$Merger$CellReducer
  54:        236029        5844680  [Ljava.nio.ByteBuffer;
  55:        231113        5546712  java.lang.String
  56:        230392        5529408  org.apache.cassandra.db.Clustering
  57:        125057        5002280  org.apache.cassandra.db.rows.RangeTombstoneMarker$Merger
  58:        154949        4958368  org.apache.cassandra.db.filter.ColumnFilter
  59:        309592        4953472  com.google.common.collect.Iterables$3
  60:        154483        4943456  [Lorg.apache.cassandra.db.rows.ColumnData;
  61:        154483        4943456  [Lorg.apache.cassandra.db.rows.Row$Builder;
  62:        154483        4943456  org.apache.cassandra.db.rows.Rows$1
  63:        179483        4307592  com.google.common.collect.Iterators$5
  64:          8055        4213168  [J
  65:        125114        4003648  java.util.ArrayList$ArrayListSpliterator
  66:        125057        4001824  org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer
  67:        125060        3828904  [Lorg.apache.cassandra.db.DeletionTime;
  68:        125057        3828808  [Lorg.apache.cassandra.db.rows.RangeTombstoneMarker;
  69:        157055        3769320  com.google.common.collect.Iterables$2
  70:        155783        3738792  org.apache.cassandra.utils.Interval
  71:        154722        3713328  org.apache.cassandra.db.rows.ComplexColumnData
  72:        154677        3712248  org.apache.cassandra.db.filter.ClusteringIndexNamesFilter
  73:        154483        3707592  org.apache.cassandra.index.SecondaryIndexManager$IndexGCTransaction$1
  74:        150356        3608544  org.apache.cassandra.db.Columns
  75:         70425        3380400  [Lorg.apache.cassandra.db.ClusteringPrefix$Kind;
  76:        124989        2999736  org.apache.cassandra.db.rows.ComplexColumnData$Builder
  77:        124971        2999304  java.util.stream.MatchOps$$Lambda$105/1652649396
  78:        124971        2999304  java.util.stream.MatchOps$1MatchSink
  79:        124971        2999304  java.util.stream.MatchOps$MatchOp
  80:        173871        2781936  org.apache.cassandra.db.rows.CellPath$CollectionCellPath
  81:        155081        2481296  java.util.TreeMap$KeySet
  82:        154648        2474368  java.util.TreeSet
  83:        154482        2471712  com.stratio.cassandra.lucene.IndexWriterWide$$Lambda$146/22703726
  84:         16388        2228768  com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$PaddedAtomicReference
  85:         54841        1754912  java.util.HashMap$Node
  86:         36123        1444920  sun.nio.cs.UTF_8$Decoder
  87:         35748        1429920  org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
  88:         34554        1382160  java.util.TreeMap$Entry
  89:         11385        1126968  [Ljava.util.HashMap$Node;
  90:         18124        1014944  sun.nio.cs.UTF_8$Encoder
  91:         41388         993312  org.apache.cassandra.io.compress.CompressionMetadata$Chunk
  92:         37722         905328  java.lang.StringBuilder
  93:          7306         814736  java.lang.Class
  94:          8627         753872  [S
  95:         15574         747552  java.util.HashMap
  96:         14365         603008  [Ljava.lang.String;
  97:         23530         564720  java.util.EnumMap$EntryIterator$Entry
  98:         17382         556224  java.util.concurrent.ConcurrentHashMap$Node
  99:         22728         545472  org.apache.cassandra.db.ColumnFamilyStore$ViewFragment
 100:          6057         533016  java.lang.reflect.Method
 101:          8908         498848  org.apache.cassandra.utils.memory.BufferPool$Chunk
 102:         12317         450160  [Ljavax.management.ObjectName$Property;
 103:         18124         434976  javax.management.ObjectName$Property
 104:         10779         431160  java.util.LinkedHashMap$Entry
 105:         16300         391200  org.apache.cassandra.utils.btree.BTree$$Lambda$19/816798571
 106:          7827         375696  java.nio.HeapCharBuffer
 107:          9190         367600  java.lang.ref.Finalizer
 108:         16672         364336  [Ljava.lang.Class;
 109:          7538         361824  org.apache.cassandra.io.compress.CompressedRandomAccessReader$Builder
 110:         20623         329968  org.apache.cassandra.db.lifecycle.View$$Lambda$46/610085529
 111:          7491         299640  java.util.HashMap$KeyIterator
 112:          4551         291264  java.net.URL
 113:          9087         290784  sun.nio.fs.UnixPath
 114:           517         272976  [Lorg.apache.lucene.util.fst.FST$Arc;
 115:          7784         249088  java.io.File
 116:          3366         242352  java.lang.reflect.Field
 117:          3783         242112  java.util.regex.Matcher
 118:          4323         242088  java.util.zip.ZipFile$ZipFileInputStream
 119:          4322         242032  java.util.zip.ZipFile$ZipFileInflaterInputStream
 120:          9545         229080  java.util.concurrent.ConcurrentLinkedQueue$Node
 121:          9525         228600  com.google.common.collect.Iterators$8
 122:         14037         224592  java.lang.Object
 123:          3456         221184  io.netty.buffer.PoolSubpage
 124:          8942         214608  java.util.concurrent.CopyOnWriteArrayList$COWIterator
 125:          6702         214464  java.lang.StackTraceElement
 126:          2917         210024  org.apache.lucene.util.fst.FST$Arc
 127:          5119         204760  javax.management.MBeanPermission
 128:          6281         200992  java.io.FilePermission
 129:          4867         194680  javax.management.ObjectName
 130:          7807         187368  org.apache.cassandra.dht.Murmur3Partitioner$LongToken
 131:          7603         182472  org.apache.cassandra.io.util.MmappedRegions$Region
 132:          5624         179968  org.apache.cassandra.cql3.ColumnIdentifier
 133:          4330         173200  java.util.HashMap$ValueIterator
 134:          2901         162456  jdk.internal.org.objectweb.asm.Item
 135:          6604         158496  [Lorg.apache.cassandra.dht.Range;
 136:          6506         156152  [Ljava.security.ProtectionDomain;
 137:          1508         144768  java.util.jar.JarFile$JarFileEntry
 138:           323         140560  [Ljava.util.concurrent.ConcurrentHashMap$Node;
 139:          2143         137152  org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl
 140:          5708         136992  org.apache.cassandra.dht.Token$KeyBound
 141:          4196         134272  org.apache.cassandra.db.rows.BTreeRow$$Lambda$140/1296076467
 142:          5445         130680  com.google.common.collect.Iterators$PeekingImpl
 143:          3990         127680  com.codahale.metrics.LongAdder
 144:          2273         127288  org.apache.cassandra.db.RowUpdateBuilder
 145:          1460         116800  java.util.zip.ZipEntry
 146:          4843         116232  java.util.concurrent.ConcurrentSkipListMap$Node
 147:          4837         116088  com.sun.jmx.mbeanserver.StandardMBeanSupport
 148:          3616         115712  java.util.Collections$UnmodifiableMap
 149:          4514         108336  org.apache.lucene.util.BytesRef
 150:          1931         108136  javax.management.MBeanServerNotification
 151:          2654         106160  java.security.AccessControlContext
 152:          4320         103680  com.sun.jmx.mbeanserver.NamedObject
 153:          4311         103464  org.apache.cassandra.utils.btree.BTree$FiltrationTracker
 154:          4295         103080  org.apache.cassandra.utils.Pair
 155:          6259         100144  java.io.FilePermission$1
 156:          2044          98112  org.apache.lucene.index.FieldInfo
 157:          2003          96144  org.antlr.runtime.CommonToken
 158:           106          95840  [Ljdk.internal.org.objectweb.asm.Item;
 159:          2902          92864  org.apache.cassandra.db.CBuilder$ArrayBackedBuilder
 160:          1022          89936  org.apache.lucene.codecs.blocktree.FieldReader
 161:          2779          88928  java.util.EnumMap$EntryIterator
 162:          5557          88912  [Lorg.apache.cassandra.db.Clustering;
 163:          2197          87880  org.apache.cassandra.utils.concurrent.Ref$State
 164:           908          87168  org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer$NumericEntry
 165:          1785          85680  com.codahale.metrics.EWMA
 166:          2063          82520  java.util.WeakHashMap$Entry
 167:          2470          81824  [Ljava.lang.reflect.Method;
 168:          2017          80680  java.util.HashMap$EntryIterator
 169:          3226          77424  java.io.ExpiringCache$Entry
 170:           875          77416  [Ljava.lang.StackTraceElement;
 171:           455          77408  [Z
 172:          2412          77184  java.io.FileDescriptor
 173:          3216          77184  java.util.concurrent.ConcurrentLinkedDeque$Node
 174:          2300          73600  com.google.common.collect.Iterators$7
 175:          1022          73584  org.apache.lucene.util.fst.FST
 176:          2283          73056  org.apache.lucene.util.WeakIdentityMap$IdentityWeakReference
 177:          3036          72864  org.apache.cassandra.db.LivenessInfo
 178:          2857          71792  [Lorg.yaml.snakeyaml.tokens.Token$ID;
 179:          1774          70960  java.util.Formatter$FormatSpecifier
 180:          1102          70528  sun.nio.ch.FileChannelImpl
 181:          4393          70288  java.util.HashMap$KeySet
 182:          2189          70048  org.apache.cassandra.metrics.CassandraMetricsRegistry$MetricName
 183:          2886          69264  java.net.URLClassLoader$1
 184:          2871          68904  org.apache.cassandra.db.DeletionTime
 185:          2853          68472  org.apache.cassandra.dht.Range
 186:           164          68224  [Lorg.antlr.runtime.BitSet;
 187:          1694          67760  java.lang.ref.SoftReference
 188:           128          67584  [Lcom.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$PaddedAtomicReference;
 189:          4196          67136  org.apache.cassandra.db.Columns$$Lambda$139/703415756
 190:          4175          66800  com.google.common.collect.Iterators$3
 191:          3938          63008  java.util.HashSet
 192:          2599          62376  com.google.common.collect.ImmutableEntry
 193:          1526          61040  org.apache.cassandra.db.PreHashedDecoratedKey
 194:          1891          60512  java.net.InetAddress$InetAddressHolder
 195:          3780          60480  com.stratio.cassandra.lucene.IndexWriterWide$$Lambda$149/2109187587
 196:           924          59136  java.util.concurrent.ConcurrentHashMap
 197:          1457          58280  java.lang.ClassNotFoundException
 198:          2411          57864  java.lang.StringBuffer
 199:          1797          57504  java.util.TreeMap$EntryIterator
 200:          1412          56480  java.util.EnumMap
 201:          2349          56376  java.util.Date
 202:          1563          56072  [Ljava.util.Formatter$Flags;
 203:          1390          55600  java.util.ArrayList$SubList$1
 204:           868          55552  java.text.DecimalFormatSymbols
 205:          1384          55360  org.yaml.snakeyaml.error.Mark
 206:           686          54880  java.lang.reflect.Constructor
 207:          1361          54440  java.util.ArrayList$SubList
 208:          2267          54408  org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxGauge
 209:          2238          53712  sun.misc.ProxyGenerator$ConstantPool$IndirectEntry
 210:          2197          52728  org.apache.cassandra.utils.concurrent.Ref
 211:          2172          52128  java.util.Collections$1
 212:          1615          51680  java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
 213:          2103          50472  org.apache.cassandra.db.lifecycle.View$$Lambda$66/2068475002
 214:          3151          50416  java.util.HashMap$Values
 215:           134          50384  java.lang.Thread
 216:           272          50048  [Lorg.apache.cassandra.net.MessagingService$Verb;
 217:          2042          49008  javax.management.ObjectInstance
 218:          1225          49000  java.lang.invoke.MethodType
 219:            61          48776  [D
 220:          2005          48120  java.util.Formatter$FixedString
 221:          1490          47680  java.security.CodeSource
 222:           569          47184  [Ljava.util.WeakHashMap$Entry;
 223:           208          46592  jdk.internal.org.objectweb.asm.MethodWriter
 224:           827          46312  java.lang.invoke.MemberName
 225:          1447          46304  sun.misc.URLClassPath$JarLoader$2
 226:          1916          45984  java.util.concurrent.atomic.AtomicLong
 227:          1896          45504  java.lang.Class$MethodArray
 228:          1886          45264  java.net.Inet4Address
 229:          1845          44280  javax.management.MBeanTrustPermission
 230:          1830          43920  [Lorg.apache.lucene.util.packed.PackedInts$Format;
 231:          1339          42848  java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry
 232:           655          41920  java.util.stream.ReferencePipeline$3
 233:          1734          41616  com.google.common.collect.Iterables$6
 234:          2578          41248  com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$1
 235:          2560          40960  com.google.common.collect.Maps$6
 236:          1022          40880  org.apache.lucene.util.fst.BytesStore
 237:           842          40416  org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxTimer
 238:          1253          40096  java.lang.ref.ReferenceQueue
 239:           703          39368  java.util.concurrent.ConcurrentHashMap$EntryIterator
 240:           264          38016  com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$PaddedAtomicLong
 241:          1539          36936  org.apache.lucene.util.fst.ReverseBytesReader
 242:           655          36680  java.lang.Class$ReflectionData
 243:           648          36288  java.util.concurrent.ConcurrentHashMap$KeyIterator
 244:          2111          33776  java.util.concurrent.atomic.AtomicInteger
 245:           421          33680  org.apache.cassandra.db.rows.RowAndDeletionMergeIterator
 246:          2092          33472  org.apache.cassandra.utils.concurrent.Refs
 247:          1022          32704  org.apache.lucene.store.ByteArrayDataInput
 248:          1358          32592  org.apache.cassandra.gms.GossipDigest
 249:           789          31560  sun.nio.ch.FileChannelImpl$Unmapper
 250:           656          31488  java.util.StringTokenizer
...
3851:             1             16  sun.util.locale.provider.SPILocaleProviderAdapter
3852:             1             16  sun.util.resources.LocaleData
3853:             1             16  sun.util.resources.LocaleData$LocaleDataResourceBundleControl
Total      41014224     4650350712
  1. 我怎样才能让它们重新开始？我觉得他们试图重新构建一些排队的插入，因为我只是用我的客户端用writeAsync“垃圾”插入，而cassandra最终被插入的数量所淹没？
  2. 下次我提取数据时如何防止这种情况发生？
cassandra datastax-java-driver
1个回答
1
投票

这是因为为旧堆缓冲区占用了太多内存，因为旧版本的Java（1.8.102和1.9.104之前的版本）对该堆没有限制：qazxsw poi

尝试升级到更新的Java并添加：

https://support.datastax.com/hc/en-us/articles/360000863663-JVM-OOM-direct-buffer-errors-affected-by-unlimited-java-nio-cache

到-Djdk.nio.maxCachedBufferSize=1048576 或cassandra-env.sh。

最新问题
© www.soinside.com 2019 - 2024. All rights reserved.