使用局部线程避免创建映射以减少GC,但失败了

问题描述 投票:0回答:1

为了避免在线GC问题。

背景

原始地图将被复制到线程(包含在线程池中),并且在该线程内,可以更新复制的地图,并且在更新之后,某些复制的地图可能会反馈到原始地图]

考虑因素

与我本地的Mac书相比,在线条件有两个典型条件:

  • 性能更好的服务器(CPU,内存和IO)
  • 高吞吐量(百万级QPS)

当前解决方案

我现在正在使用ThreadLocal建立一个ReusableMap,以确保每个映射都已绑定到线程,并且当需要copy且已经创建了该线程的映射时,我们可以直接使用地图。

当然,我们需要先清除地图,然后从原始地图中复制内容

我以为这会减少GC,但是当我使用jmh运行一些测试并通过jvisualvm在Visual GC中监视结果时;我可悲的是发现它没有达到我的预期。仍然像以前一样有很多GC。


一些代码演示我刚才提到的内容:

public class ReusableHashMap {
    private static final String DEFAULT_MAP_KEY = "defaultMap";
    /**
     * weak or soft reference perhaps could be used: https://stackoverflow.com/a/299702/2361308
     * <p>
     * via the static ThreadLocal initialized, each thread will only see the value it set itself;
     */
    private static ThreadLocal<Map> theCache = new ThreadLocal<>();

    /**
     * the default usage when there is only one map passed from parent
     * thread to child thread.
     *
     * @param origMap the parent map
     * @param <K>     generic type for the key
     * @param <V>     generic type for the value
     * @return a map used within the child thread - the reusable map
     */
    public static <K, V> Map<K, V> getMap(Map<K, V> origMap) {
        return getMap(DEFAULT_MAP_KEY, origMap);
    }


    public static <K, V> Map<K, V> getMap() {
        return getMap(DEFAULT_MAP_KEY);
    }

    /**
     * clone the parent-thread map at the beginning of the child thread,
     * after which you can use the map as usual while it's thread-localized;
     * <p>
     * no extra map is created for the thread any more - preventing us from creating
     * map instance all the time.
     *
     * @param theMapKey the unique key to specify the map to be passed into the child thread;
     * @param origMap   the parent map
     * @param <K>       generic type for the key
     * @param <V>       generic type for the value
     * @return the cached map reused within the child thread
     */
    public static <K, V> Map<K, V> getMap(String theMapKey, Map<K, V> origMap) {
        Map<String, Map<K, V>> threadCache = theCache.get();
        if (Objects.isNull(threadCache)) {
            System.out.println("## creating thread cache");
            threadCache = new HashMap<>();
            theCache.set(threadCache);
        } else {
            System.out.println("**## reusing thread cache");
        }
        Map<K, V> cacheMap = threadCache.get(theMapKey);
        if (Objects.isNull(cacheMap)) {
            System.out.println("  ## creating thread map cache for " + theMapKey);
            cacheMap = new HashMap<>();
        } else {
            System.out.println("  **## reusing thread map cache for " + theMapKey);
            cacheMap.clear();
        }
        if (MapUtils.isNotEmpty(origMap)) {
            cacheMap.putAll(origMap);
        }
        threadCache.put(theMapKey, cacheMap);
        return cacheMap;
    }

    public static <K, V> Map<K, V> getMap(String theMapKey) {
        return getMap(theMapKey, null);
    }

    /**
     * @param size
     * @param lenLimit
     * @return
     */
    private static Map<String, String> generateAMap(int size, int lenLimit) {
        Map<String, String> res = new HashMap<>();
        String aKey = "key - ";
        String aValue = "value - ";
        for (int i = 0; i < size; ++i) {
            aKey = i + " - " + LocalDateTime.now().toString();
            aValue = i + " - " + LocalDateTime.now().toString() + aValue;
            res.put(aKey.substring(0, Math.min(aKey.length(), lenLimit)),
                    aValue.substring(0, Math.min(aValue.length(), lenLimit)));
        }
        return res;
    }


    public static void main(String[] args) throws Exception {
        org.openjdk.jmh.Main.main(args);
//        System.out.println(MyState.operationCount.get());
        sleep(30_000);
//        System.out.println(MyState.operationCount.get());
//        System.out.println(toJson(generateAMap(200, 200)));
    }

    @Benchmark
    @Fork(value = 1, warmups = 2, jvmArgs = {"-Xms100M", "-Xmx100M"})
    @Warmup(iterations = 3, time = 10)
    @BenchmarkMode(Mode.Throughput)
    @OutputTimeUnit(TimeUnit.MINUTES)
    @Measurement(iterations = 1, time = 2, timeUnit = TimeUnit.MINUTES)
    public void testMethod() throws Exception {
//        AtomicInteger THREAD_UUID = new AtomicInteger(1);
        ConcurrentHashMap<String, String> theResultMap = new ConcurrentHashMap<>();
        final Map<String, String> theParentMap0 = MyState.parentMapSmall_0;
        final Map<String, String> theParentMap1 = MyState.parentMapSmall_0;
//        final Map<String, String> theParentMap0 = MyState.parentMapMedium_0;
//        final Map<String, String> theParentMap1 = MyState.parentMapMedium_1;
        ThreadUtils.getTheSharedPool().submit(() -> {
            try {
//                MyState.operationCount.getAndIncrement();
//                Map<String, String> theChildMap0 = new HashMap<>(theParentMap0);
                Map<String, String> theChildMap0 = ReusableHashMap.getMap("test0", theParentMap0);
                String threadName = Thread.currentThread().getName();
//                    print(theChildMap0, "initial map");
                theChildMap0.put("test0", "child");
                theChildMap0.put("test" + threadName, "child");

//                Map<String, String> theChildMap1 = new HashMap<>(theParentMap1);
                Map<String, String> theChildMap1 = ReusableHashMap.getMap("test1", theParentMap1);
//                    print(theChildMap1, "initial map");
                theChildMap1.put("test1", "child");
                theChildMap1.put("test" + threadName, "child");

//                THREAD_UUID.incrementAndGet();
//                    theChildMap.put("test" + "Thread" + THREAD_UUID.getAndIncrement(), "child");
                for (int j = 0; j < 1_0; ++j) {
//                        print(theChildMap0, "theMapKey - test0");
//                        print(theChildMap1, "theMapKey - test1");
                    sleep(10);
                }
//                theResultMap.putAll(theChildMap0);
//                theResultMap.putAll(theChildMap1);
            } catch (Exception e) {
                System.err.println(e.getMessage());
            }
        }).get();
    }

    private void print(Object o) {
        print(o, "");
    }

    private void print(Object o, String content) {
        String s = String
                .format("%s: current thread: %s map: %s", content, Thread.currentThread().getName(), toJson(o));
        System.out.println(s);
    }

    @State(Scope.Benchmark)
    public static class MyState {
        static AtomicInteger operationCount = new AtomicInteger(1);
        // 20 & 200 -> 4k
        static Map<String, String> parentMapSmall_0 = generateAMap(20, 200);
        static Map<String, String> parentMapSmall_1 = generateAMap(20, 200);
        // 200 & 200 -> 22k
        static Map<String, String> parentMapMedium_0 = generateAMap(100, 200);
        static Map<String, String> parentMapMedium_1 = generateAMap(100, 200);

        // 200 & 200 -> 45k
//        static Map<String, String> parentMapMedium_0 = generateAMap(200, 200);
//        static Map<String, String> parentMapMedium_1 = generateAMap(200, 200);
    }
}
java multithreading garbage-collection jvm thread-local
1个回答
0
投票

HashMap包含内部的Node对象的数组,如果您调用hashMap.clear(),则将清除此数组,因此所有Node对象都不可用于垃圾回收。因此,缓存这样的地图根本无济于事。

如果您想限制GC的数量,您可以只使用ConcurrentHashMap吗?如果您需要在线程之间分配工作,则可以将键列表/键数组传递给它们应在其上操作的每个线程,然后返回更新值列表。没有您要解决的问题的准确描述,很难说出更多。

但是首先,您还应该考虑如果确实需要此操作,那么您真的要给Java施加很大压力,以解决GC的实际问题,只要您提供一些正常的内存量,这些问题就无法通过一些简单的调整来解决。到Java进程。

© www.soinside.com 2019 - 2024. All rights reserved.