HBase最佳实践-CMS GC调优

HBase发展到当下,对其进行的各种优化从未停止,而GC优化更是其中的重中之重。从0.94版本提出MemStoreLAB策略、Memstore Chuck Pool策略对写缓存Memstore进行优化开始,到0.96版本提出BucketCache以及堆外内存方案对读缓存BlockCache进行优化,再到后续2.0版本宣称会引入更多堆外内存,可见HBase会将堆外内存的使用作为优化GC的一个战略方向。然而无论引入多少堆外内存,都无法避免读写全路径使用JVM内存,就拿BucketCache中offheap模式来讲,即使HBase数据块是缓存在堆外内存的,但是在读取的时候还是会首先将堆外内存中的block加载到JVM内存中,再返回给用户。可见,无论使用多少堆外内存,对JVM内存的使用终究是绕不过去,既然绕不过去,就还是需要落脚于GC本身,对GC本身进行优化。本文就将会介绍HBase应用场景下CMS GC策略的调优技巧,后续还会针对另一业界开始使用的GC策略-G1GC策略在HBase应用场景下进行调优介绍。

CMS GC工作原理

如果看官已经对CMS GC工作原理比较熟悉,完全可以跳过本节内容,直接进入下节。如果看官还对CMS GC不是很了解,可以参考笔者之前的另一篇文章《HBase GC的前生今生-身世篇》,文中对JVM的内存结构以及CMS GC进行了相当详细的介绍。为了下文介绍方便,在此还是对其中的一些重要知识点进行提炼:

1. 整个JVM内存由Young区、Tenured区和Perm区三部分组成,其中Young区又分为一个Eden区和两个Survivor区

2. 整个对象生命周期简要说明(一定要烂熟于心,下文会一直用到):

(1)Young区:一个对象初始化之后,首先会进入Eden区,当Eden区满之后会触发一次Minor GC,Minor GC会检查Eden区所有对象是否依旧存活(是否有其他对象引用),如果存活,会将其从Eden区拷贝到Survivor区,并将这些存活对象的age加一,而死亡的对象会被作为垃圾回收。此时Eden区又空闲出来,等新对象填充,填充满之后再会触发Minor GC,如此往复。需要注意的是,每执行一次Minor GC,存活对象的age就会加一。

(2)Tenured区:一旦存活对象的age超多一定阈值就会晋升到Tenured区,因此可以理解为Tenured区一般存放长寿对象。很显然,随着时间流逝,Tenured区也会被填充满,此时就会触发CMS GC(old gc),这种GC相对比较复杂,由5个步骤组成,详见参考文章。

3. 无论是Minor GC还是CMS GC,都会’Stop-The-World’,即停止用户的一切线程,只留下gc线程回收垃圾对象。其中Minor GC的STW时间主要耗费在复制阶段,CMS GC的STW时间主要耗费在标示垃圾对象阶段。

GC调优目标

上节简单介绍了Java虚拟机的内存结构以及Java GC的基本知识,接下来会在此基础上介绍HBase集群中GC的几种参数调优技巧。在介绍具体的调优技巧之前,有必要先来看看GC调优的最终目标和基本原则:

1. 平均Minor GC时间尽可能短。因为整个Minor GC都处于STW,因此短时间Minor GC会使用户读写更加平稳,延迟可控。

2. CMS GC次数越少越好。时间越短越好。一方面是因为一次CMS GC一般都会引起至少秒级的应用暂停,对用户读写影响较大;另一方面频繁的CMS GC会产生大量的内存碎片,严重的时候会引起Full GC,导致RegionServer宕机。

下面对参数的调优技巧都谨遵以上原则,尤其对于HBase这类延迟敏感性项目而言,在尽量避免严重影响用户读写的情况下使得GC更加平稳、暂停时间更短!

CMS GC优化技巧

本节会针对HBase这一应用场景对JVM的各种GC参数进行分析,主要分三个阶段进行。第一阶段会介绍适用于所有场景下的GC参数配置,这些参数不需要太多解释读者就可以轻松理解;第二阶段和第三阶段分别就两组参数进行调优讲解,这两组参数一般会根据不同的应用场景进行设置才能使得GC效果最好,鉴于这两组参数的复杂性,我们会通过理论+实验的方式一一进行说明;

阶段一:默认推荐配置

在介绍具体的调优技巧之前,先来看看CMS GC涉及到的所有相关参数及其对应的意义,下面是最常见的参数:

-Xmx -Xms -Xmn -Xss -XX:MaxPermSize= M -XX:SurvivorRatio=S  -XX:+UseConcMarkSweepGC -XX:+UseParNewGC  -XX:+CMSParallelRemarkEnabled -XX:MaxTenuringThreshold=N -XX:+UseCMSCompactAtFullCollection  -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=C -XX:-DisableExplicitGC

通过上文对各个GC参数的说明,可以轻松得出第一阶段推荐的参数设置如下,这样的设置基本适用于所有的场景:

-XX:+UseConcMarkSweepGC -XX:+UseParNewGC  -XX:+CMSParallelRemarkEnabled  -XX:+UseCMSCompactAtFullCollection  -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=75% -XX:-DisableExplicitGC


调优预准备

上文通过解释各个GC参数意义给出了基本的推荐设置,同时也提到几个对性能影响重大的参数:Xmn、SurvivorRatio以及MaxTenuringThreshold,下面会通过理论推理+实验验证的方式对这几个参数在HBase系统的设置进行调优。在深入介绍调优技巧之前,需要额外针对三个相关部分预先做下讲解,这样可以更好地理解下文的实验数据分析。这三部分分别是:测试环境+测试基本条件,GC日志解释,HBase场景内存分析;

测试环境

首先就下文中实验测试的硬件拓扑、软件配置以及相关测试数据情况进行说明:


需要强调的是HBase全部配置为BucketCache模式,而不是LruBlockCache。使用了大量堆外内存作为读缓存,在很大程度上优化了GC,如下图:


上图是在两种缓存策略下GC表现情况,可见BucketCache模式比LruBlockCache模式GC表现好很多,强烈建议线上配置BucketCache模式。可能很多童鞋都测试过这两种模式下的GC、吞吐量、读写延迟等指标,看到测试结果都会很疑惑,BucketCache模式下的各项性能指标都比LruBlockCache差了好多,笔者也疑惑过,后来才明白:测试肯定是在基本全内存场景下进行的,这种情况下确实会是如此。读者可以想想为什么会如此,实在不明白可以参考之前一篇博文《BlockCache方案性能对比测试报告》。但是话又说回来,在大数据场景下又有多少业务会是全内存操作呢?

GC日志分析

介绍完实验基本条件后,再对GC日志进行简单的解释,方便下文对日志进行分析。需要注意只有在添加参数-XX:+PrintTenuringDistribution才能打印对应日志,强烈建议线上集群开启该参数,日志片段如下:

2016-07-26T10:37:16.933+0800: 227753.150: [GC2016-07-26T10:37:16.933+0800: 227753.150: [ParNew
Desired survivor size 268435456 bytes, new threshold 5 (max 15)
- age   1:   57523184 bytes,   57523184 total
- age   2:   80236520 bytes,  137759704 total
- age   3:   73226496 bytes,  210986200 total
- age   4:   50318392 bytes,  261304592 total
- age   5:   63166384 bytes,  324470976 total
- age   6:        240 bytes,  324471216 total
: 1268903K->305311K(1572864K), 0.0840620 secs] 26598675K->25635082K(66584576K), 0.0844700 secs] [Times: user=1.82 sys=0.08, real=0.08 secs]

上述日志片段分三部分进行解释:

第一部分:基本信息区,主要有两点需要重点关注,其一是Desired survivor size 268435456 bytes,该值通过SurvivorSize * TargetSurvivorRatio计算而来,默认TargetSurvivorRatio为50%,如果xmn为5g,SurvivorRatio为8的话,Desired Survivor Size 就等于256M;其二是new threshold 5 (max 15),括号中max 15表示对象晋级老生代的最大阈值为15,new threshold 5表示调整后的threshold为5,可见,threshold在整个过程中会不断调整,调整后的threshold表示不大于该值的所有age的对象大小总和刚好大于Desired Survivor Size,否则为最大threshold(15)。

第二部分:不同age对象分布区,第一列表示该Young区共分布有age在1~6的对象;第二列表示所在age含有的对象集所占内存大小,比如age为2的所有对象总大小为80236520 bytes;第三列表示小于对应age的所有对象占用内存的累加值,比如age2对应第二列137759704 total表示age为1和age为2的所有对象总大小;

第三部分:内存回收信息区,第一列表示Young区的内存回收情况,1268903K->305311K表示Young区回收前内存为1268903K,回收后变为305311K;第二列表示Jvm Heap的内存回收情况,26598675K->25635082K(66584576K) 表示当前Jvm总分配内存为66584576K,回收前对象占用内存为26598675K,回收后对象占用内存为25635082K;第三列表示回收时间,其中real表示本次gc所消耗的STW时间,即用户业务暂停时间。

HBase场景内存分析

通常来讲,每种应用都会有自己的内存对象特性,分类来说无非就两种:一种是短寿对象(指存活对象较短的对象,比如临时变量等)居多工程,比如大多数纯HTTP请求处理工程,短寿对象可能占到所有对象的70%左右;另一种是长寿对象(指存活对象较长的对象,比如TTL设置较长的缓存对象)居多工程,比如类似于HBase、Spark等这类大内存工程。具体以HBase为例,来看看具体的内存对象:

1. RPC请求对象,比如Request对象和Response对象,一般这些对象会随着短连接RPC的销毁而消亡,这些对象可以认为是短寿对象;

2. Memstore对象,HBase中Memstore中对象一般会持续存活较长时间,用户写入数据到Memstore中之后对象就一直存在,直至Memstore写满之后flush到HDFS。一般在写入QPS较高的情况下写满memstore也通常需要一个小时左右,可见Memstore对象肯定是长寿对象。另外,Memstore对象默认比较大,2M大小。

3. BlockCache对象,和Memstore对象一样,BlockCache对象一般也会在内存存活较长时间,属于长寿对象。这种对象默认64K大小。

因此可以看出,HBase系统属于长寿对象居多的工程,因此GC的时候只需要将RPC这类短寿对象在Young区淘汰掉就可以达到最好的GC效果。

阶段二:NewParSize调优

理论分析

NewParSize表示young区大小,而young区大小直接决定minor gc的频率。minor gc频率一方面决定单次minor gc的时间长短,gc越频繁,gc时间就越短;一方面决定对象晋升到老年代的量,gc越频繁,晋升到老年代的对象量就越大。解释起来就是:

1. 增大young区大小,minor gc频率降低,单次gc时间会较长(young区设置更大,一次gc就需要复制更多对象,耗时必然比较长),业务读写操作延迟抖动较大。反之,业务读写操作延迟抖动较小,比较平稳。

2. 减小young区大小,minor gc频率增快,但会加快晋升到老年代的对象总量(每gc一次,对象age就会加一,当age超过阈值就会晋升到老年代,因此gc频率越高,age就增加越快),潜在增加old gc风险。

因此设置NewParSize需要进行一定的平衡,不能设置太大,也不能设置太小。

实验结果

实验条件:分为独立对照试验,三台RegionServer分别设置Xmn为512m、2g、5g,Xmn越大,分配的Young区越大;SurvivorRatio和MaxTenuringThreshold取默认值;

实验结果曲线图:


结果分析

1. 图一是Xmn不同场景下总体的GC耗时曲线图,其中横坐标表示GC次数,纵坐标表示GC耗时(STW),单位ms。需要特别说明的是,这3条曲线是在相同时间段统计的,也就是说在这段时间内Xmn为512m的情况下GC次数最多,而相应的Xmn为5的情况下GC次数最少。

2. 图一整体上来看绿线尖峰很多而且很高,表示CMS GC较频繁,但绿线主体部分处于红线与蓝线之下,表示平均Minor GC耗时更短;蓝线GC次数最少,尖峰也比较突出,另外Minor GC相比红线和绿线耗时更长;红线的Minor GC耗时介于蓝线和绿线之间,尖峰比较平稳,表示CMS GC相对比较短暂;因此总体来看,红线代表的Xmn为2的场景下CMS GC更加合理,平均Minor GC相对不高,而相比之下,另外两种场景都有特别明显的缺陷,Xmn=2是一个最优的选择;图一只能直观上看出这么多,更加精确结果需要接着看图二和图三。

3. 图二主要统计Minor GC的主要指标:总GC次数以及平均单次Minor GC耗时。两者来看,更关注后者,因为后者决定了业务读写的延迟以及稳定度;由图中可以看出,Xmn512m的平均单次Minor GC耗时最少,其次是Xmn2g,最差是Xmn5g,达到了130ms左右,意味着在其Minor GC过程中所有业务读写延迟至少为130ms;这个也很好理解,Young区越小,Minor GC频率越高,单次Minor GC需要复制的对象数就越少,耗时越少;

4. 图三主要统计CMS GC(老年代GC)的主要指标:CMS GC次数以及平均单次老年代GC耗时(只算STW耗时);由图中可以看出,Xmn2g无论是GC次数还是GC耗时都更加优秀,相比之下Xmn512m就是最差的选择;解释起来也很简单,因为Young区设置太小,Minor GC频率高,对象age增加很快,很多对象就有可能因为age超过阈值(默认6)晋升到老年代,相对而言会更有可能引入大量短寿对象晋升老年代。而短寿对象相对而言会比较小,比如request、response等,大量小对象一旦进入老年代,就会导致CMS GC的时候需要标注更多对象,必然比较耗时;

实验结论

可见,测试结果基本和理论分析一致,Xmn设置过小会导致CMS GC性能较差,而设置过大会导致Minor GC性能较差,因此建议在JVM Heap为64g以上的情况下设置Xmn在1~3g之间,在32g之下设置为512m~1g;具体最好经过简单的线上调试;需要特别强调的是,笔者在很多场合都看到很多HBase线上集群会把Xmn设置的很大,比如有些集群Xmx为48g,Xmn为10g,查看日志发现GC性能极差:单次Minor GC基本都在300ms~500ms之间,CMS GC更是很多超过1s。在此强烈建议,将Xmn调大对GC(无论Minor GC还是CMS GC)没有任何好处,不要设置太大。

阶段三:增大Survivor区大小(减小SurvivorRatio) & 增大MaxTenuringThreshold

理论分析

上文讲过,一次Minor GC会将存活对象从Eden区(以及survivor from区)复制到Survivor区(to区),因此增大Survivor区可以容纳更多的存活对象。这样就会防止因为Survivor区太小导致很对存活对象还没有达到MaxTenuringThreshold阈值就直接进入老生代,潜在增大old gc的触发频率;但是Survivor区设置太大也会有一定的问题,Survivor设置较大会使得对象可以在Young区’待’的时间很长,但是对于一些长寿对象较多的场景下(比如HBase),大量长寿对象长时间待在Young区做很多’无谓’的复制,一定程度上增加Minor GC开销。

另外,增加MaxTenuringThreshold相当于提高了进入老年代的门槛,可以有效限制进入老年代的对象数。和Survivor设置相似,调整MaxTenuringThreshold也需要做一个取舍,设置太小会增加CMS GC的触发频率以及耗时,而设置太大则会在长寿对象较多场景下增加Minor GC开销。一般情况下,默认MaxTenuringThreshold=15已经相对比较大,不需要做任何调整。

实验结果

实验条件:分为独立对照试验,三台RegionServer分别设置SurvivorRatio为2、8、15,SurvivorRatio越大,Survivor区大小越小;MaxTenuringThreshold取默认值;其他:-Xmx64g,-Xmn2g;

实验结果曲线:

结果分析

1. 图一是SurvivorRatio在三种不同场景下对应的GC性能曲线图,大体可以看出蓝线Minor GC次数最多,绿线尖峰太多,即CMS GC性能最差;具体细节再来看图二和图三。

2. 图二主要统计Minor GC主要指标:平均单次Minor GC耗时三者基本相当,SurvivorRatio:2场景下稍微较高,这是因为SurvivorRatio=2对应的Survivor区较大,可以使得对象在Young区’待’的时间很长,在HBase这种长寿对象较多的情况下,可能会增加一些无谓的‘复制’开销(下文会通过日志分析详细解释)。另外,SurvivorRatio=2场景下Minor GC频率也比较高,可能的原因是因为在总Young大小确定的情况下,Survivor越大,Eden自然越小,Minor GC频率就会增大。可见,SurvivorRatio=2场景下Minor GC性能相对稍微较差。

3. 图三主要统计CMS GC主要指标:三者CMS GC次数基本相当,SurvivorRatio=2场景下单次CMS GC耗时最少,相比SurvivorRatio=8的场景耗时减少30%左右,性能最好;而相比之下SurvivorRatio=15场景下耗时最长,性能相当差;这是因为SurvivorRatio=2场景下存活对象可以长时间待在Young区,可以得到充分的淘汰,晋升到老生代的短寿小对象会比较少,因而CMS GC性能较好;相比SurvivorRatio=15会因为Survivor区设置太小,很多短寿小对象因为得不到充分的淘汰就会‘溢出’到老生代,导致CMS性能很差。

实验结论

可见,测试结果基本和理论分析也基本一致,对于Minor GC来说,SurvivorRatio设置对其影响不是很大。而对于CMS GC来说,将SurvivorRatio设置过大简直就是灾难,性能极其差。而和默认值SurvivorRatio=8相比,将SurvivorRatio调小有利于短寿小对象更充分地淘汰,因此建议将SurvivorRatio=2

CMS调优结论

1. 缓存模式采用BucketCache策略Offheap模式

2. 对于大内存(大于64G),采用如下配置:

-Xmx64g -Xms64g -Xmn2g -Xss256k -XX:MaxPermSize=256m -XX:SurvivorRatio=2  -XX:+UseConcMarkSweepGC -XX:+UseParNewGC 
-XX:+CMSParallelRemarkEnabled -XX:MaxTenuringThreshold=15 -XX:+UseCMSCompactAtFullCollection  -XX:+UseCMSInitiatingOccupancyOnly        
-XX:CMSInitiatingOccupancyFraction=75 -XX:-DisableExplicitGC

其中Xmn可以随着Java分配堆内存增大而适度增大,但是不能大于4g,取值范围在1~3g范围;SurvivorRatio一般建议选择为2;MaxTenuringThreshold设置为15;

3 对于小内存(小于64G),只需要将上述配置中Xmn改为512m-1g即可

总结

本文首先比较系统的介绍了CMS GC的相关知识,之后分三个阶段层层推进对HBase集群中相关重要参数的调优进行了详细说明,尤其后面两阶段通过理论推理以及实验验证的方式对两组核心参数进行了针对性调整,最终得出一个较为完整的CMS GC参数配置。读者可以参考该参数配置对集群进行调整,再通过日志查看调整效果~

范欣欣

网易杭州研究院技术专家。负责网易内部Hadoop&HBase等组件内核开发运维工作,擅长大数据领域架构设计,性能优化以及问题诊断。 著有《HBase原理与实践》一书。 微信公众号:大数据基建。 邮箱:libisthanks@gmail.com。

在 “HBase最佳实践-CMS GC调优” 上有 37 条评论

  1. 范老师 你这里说的hbase_heapsize 在我的配置里默认为1000MB,这个hbase_heapsize和我分配给读写缓存的regionserver_heapsize是啥关系?

      1. 是的 我的hadoop环境是有ambari搭的,hbase相关的内存分配只有hbase_heapsize(默认1000),另外就是regionserver_heapsize了,regionserver_heapsize的内存用来作为write buffer( hbase.regionserver.global.memstore)和read buffer(hfile.blockcahe.size) 请问我这种环境下如何进行内存的gc调优?

        1. hi ,
          1. gc调优一方面主要看readbuffer的策略,建议使用bucketblockcache替代当前的lrublockcache
          2. 另一方面需要关注jvm的gc参数,比如xmn参数以及suvivorratio参数两个,可以参考文章

          gc调优本身和hbase_heapsize以及regionserver_heapsize的大小没有太大关系,也和write_buffer以及read_buffer关系不大

  2. 博主您好,想请教一下,gc的日志,/var/log/hbase/gc.regionserver.log每次重启就会覆盖,能够配置成追加写吗。有时hbase报错,想去查gc的问题时,一旦重启就没了之前的gc日志信息了。

    1. 可以看看你的jvm配置 将GC日志设为3或者更多 -Xloggc:$HBASE_LOG_DIR/gc-regionserver-`date +%Y%m%d-%H-%M`.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=3

  3. 博主你好,我也是做了好多测试,发现不同读写比情况下LRU的性能(吞吐、延时)都要强于CBC,看到文章中也提到了这一点,但还是想不懂为什么CBC要差一些,所以很想请教博主?

    1. 因为CBC模式下bucketcache(offheap模式)使用堆外内存,堆外内存读取会比jvm内存读取复杂,流程更多,所以在全内存场景喜爱LRU完全好于CBC,在缓存基本不命中场景下两者吞吐量延迟基本相当

  4. 勘误:
    文章“阶段三:增大Survivor区大小(减小SurvivorRatio) & 增大MaxTenuringThreshold”一节的实验结论中,原文“而和默认值SurvivorRatio=8相比,将SurvivorRatio调大有利于短寿小对象更充分地淘汰,因此建议将SurvivorRatio=2”有误,应该是“将SurvivorRatio调小”。

  5. 范大神,能问下你们hbase集群zookeeper.session.timeout这个参数设置多大么,能否通过调大这个参数来降低因为GC导致的RS连接zk超时挂掉问题呢? 假如我调整到比如180秒会对hbase造成什么影响吗? 万分感谢!!

  6. 博主方便用 jmap -heap PID 显示出博主的配置,然后解释这些参数吗?这样可能更加方便大家看学习博主的配置。

    1. 没有用G1GC一方面是因为我们这边内存使用量没有那么大,另一方面G1GC要用好需要关注太多参数配置。不过以后是一个大的方向

      1. 我们的服务器都是190G左右的大内存机器, 如果G1GC使用得当, 是否可以用LruCache取代BucketCache?
        感觉BucketCache是CMS GC下的一种Workaround方案.

  7. 老师,您好。我们的HBase是小集群(10台),的CDH集群,内存都是用的默认值(RS的堆内存有4g),之前没有调优。我在用Flink批量写入HBase时,CDH经常飘红:原因是GC时间超阈值。我查了一下CMS日志,发现在并发清除阶段的时间久的离谱!我把比较夸张的一段贴在下面了,能否给个思路,感谢!
    2021-08-10T14:51:47.634+0800: [CMS-concurrent-sweep: 1.949/6.413 secs] [Times: user=14.85 sys=1.02, real=6.41 secs]
    2021-08-10T14:51:47.635+0800: [CMS-concurrent-reset-start]
    2021-08-10T14:51:47.650+0800: [CMS-concurrent-reset: 0.015/0.015 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
    2021-08-10T14:51:47.909+0800: [GC (Allocation Failure) 2021-08-10T14:51:47.909+0800: [ParNew: 73803K->7502K(76672K), 0.0433232 secs] 2985070K->2918830K(4185792K), 0.0434138 secs] [Times: user=0.04 sys=0.01, real=0.05 secs]
    2021-08-10T14:51:47.953+0800: [GC (CMS Initial Mark) [1 CMS-initial-mark: 2911327K(4109120K)] 2919091K(4185792K), 0.0095153 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
    2021-08-10T14:51:47.963+0800: [CMS-concurrent-mark-start]
    2021-08-10T14:51:48.146+0800: [GC (Allocation Failure) 2021-08-10T14:51:48.146+0800: [ParNew: 75662K->8512K(76672K), 0.0806900 secs] 2986990K->2923498K(4185792K), 0.0807806 secs] [Times: user=0.06 sys=0.02, real=0.08 secs]
    2021-08-10T14:51:48.296+0800: [CMS-concurrent-mark: 0.248/0.333 secs] [Times: user=1.49 sys=0.06, real=0.33 secs]
    2021-08-10T14:51:48.296+0800: [CMS-concurrent-preclean-start]
    2021-08-10T14:51:48.332+0800: [CMS-concurrent-preclean: 0.034/0.036 secs] [Times: user=0.21 sys=0.06, real=0.04 secs]
    2021-08-10T14:51:48.332+0800: [CMS-concurrent-abortable-preclean-start]
    2021-08-10T14:51:48.402+0800: [GC (Allocation Failure) 2021-08-10T14:51:48.402+0800: [ParNew: 76672K->8512K(76672K), 0.1289755 secs] 2991658K->2937785K(4185792K), 0.1290695 secs] [Times: user=0.11 sys=0.02, real=0.13 secs]
    2021-08-10T14:51:48.608+0800: [CMS-concurrent-abortable-preclean: 0.141/0.275 secs] [Times: user=1.61 sys=0.05, real=0.27 secs]
    2021-08-10T14:51:48.609+0800: [GC (CMS Final Remark) [YG occupancy: 51522 K (76672 K)]2021-08-10T14:51:48.609+0800: [Rescan (parallel) , 0.0422101 secs]2021-08-10T14:51:48.651+0800: [weak refs processing, 0.0000795 secs]2021-08-10T14:51:48.651+0800: [class unloading, 0.0492355 secs]2021-08-10T14:51:48.700+0800: [scrub symbol table, 0.0123413 secs]2021-08-10T14:51:48.713+0800: [scrub string table, 0.0009098 secs][1 CMS-remark: 2929273K(4109120K)] 2980795K(4185792K), 0.1049206 secs] [Times: user=0.10 sys=0.00, real=0.11 secs]
    2021-08-10T14:51:48.714+0800: [CMS-concurrent-sweep-start]
    2021-08-10T14:51:48.779+0800: [GC (Allocation Failure) 2021-08-10T14:51:48.779+0800: [ParNew: 76637K->8512K(76672K), 0.2079816 secs] 2996766K->2961111K(4185792K), 0.2080966 secs] [Times: user=0.21 sys=0.00, real=0.21 secs]
    2021-08-10T14:51:49.140+0800: [GC (Allocation Failure) 2021-08-10T14:51:49.140+0800: [ParNew: 76672K->8512K(76672K), 0.1998496 secs] 3003891K->2965198K(4185792K), 0.1999370 secs] [Times: user=0.20 sys=0.00, real=0.20 secs]
    2021-08-10T14:51:49.434+0800: [GC (Allocation Failure) 2021-08-10T14:51:49.434+0800: [ParNew: 76672K->8512K(76672K), 0.1821081 secs] 3021530K->2978797K(4185792K), 0.1822234 secs] [Times: user=0.18 sys=0.00, real=0.18 secs]
    2021-08-10T14:51:49.733+0800: [GC (Allocation Failure) 2021-08-10T14:51:49.733+0800: [ParNew: 76672K->8512K(76672K), 0.1504635 secs] 3024458K->2974818K(4185792K), 0.1505647 secs] [Times: user=0.15 sys=0.00, real=0.15 secs]
    2021-08-10T14:51:49.993+0800: [GC (Allocation Failure) 2021-08-10T14:51:49.993+0800: [ParNew: 76672K->8511K(76672K), 0.1936745 secs] 3016307K->2973449K(4185792K), 0.1937949 secs] [Times: user=0.19 sys=0.00, real=0.19 secs]
    2021-08-10T14:51:50.301+0800: [GC (Allocation Failure) 2021-08-10T14:51:50.302+0800: [ParNew: 76671K->8511K(76672K), 0.1576835 secs] 3013717K->2964476K(4185792K), 0.1578101 secs] [Times: user=0.16 sys=0.00, real=0.15 secs]
    2021-08-10T14:51:50.480+0800: [GC (Allocation Failure) 2021-08-10T14:51:50.480+0800: [ParNew: 76671K->8511K(76672K), 0.0860739 secs] 3030531K->2973237K(4185792K), 0.0861817 secs] [Times: user=0.07 sys=0.02, real=0.08 secs]
    2021-08-10T14:51:50.584+0800: [GC (Allocation Failure) 2021-08-10T14:51:50.584+0800: [ParNew: 76671K->8510K(76672K), 0.0512029 secs] 3040549K->2978473K(4185792K), 0.0513130 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:50.647+0800: [GC (Allocation Failure) 2021-08-10T14:51:50.647+0800: [ParNew: 76670K->7183K(76672K), 0.0521286 secs] 3045304K->2982489K(4185792K), 0.0522169 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:50.712+0800: [GC (Allocation Failure) 2021-08-10T14:51:50.712+0800: [ParNew: 75343K->6360K(76672K), 0.0444439 secs] 3049340K->2986350K(4185792K), 0.0445512 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:50.771+0800: [GC (Allocation Failure) 2021-08-10T14:51:50.771+0800: [ParNew: 74520K->8512K(76672K), 0.0515823 secs] 3052375K->2995976K(4185792K), 0.0516707 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:50.834+0800: [GC (Allocation Failure) 2021-08-10T14:51:50.834+0800: [ParNew: 76506K->6074K(76672K), 0.0452806 secs] 3063146K->3000358K(4185792K), 0.0453631 secs] [Times: user=0.03 sys=0.01, real=0.05 secs]
    2021-08-10T14:51:50.902+0800: [GC (Allocation Failure) 2021-08-10T14:51:50.902+0800: [ParNew: 74234K->8512K(76672K), 0.0504306 secs] 3063328K->3005131K(4185792K), 0.0505333 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:50.974+0800: [GC (Allocation Failure) 2021-08-10T14:51:50.974+0800: [ParNew: 76672K->8512K(76672K), 0.0627139 secs] 3070082K->3009574K(4185792K), 0.0628423 secs] [Times: user=0.05 sys=0.02, real=0.06 secs]
    2021-08-10T14:51:51.060+0800: [GC (GCLocker Initiated GC) 2021-08-10T14:51:51.060+0800: [ParNew: 76672K->8512K(76672K), 0.0589934 secs] 3074349K->3012650K(4185792K), 0.0590668 secs] [Times: user=0.06 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:51.133+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.133+0800: [ParNew: 76672K->7963K(76672K), 0.0530061 secs] 3079476K->3018288K(4185792K), 0.0530952 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:51.197+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.197+0800: [ParNew: 76123K->7852K(76672K), 0.0478507 secs] 3086001K->3025091K(4185792K), 0.0479499 secs] [Times: user=0.04 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:51.255+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.255+0800: [ParNew: 76012K->8512K(76672K), 0.0424995 secs] 3092647K->3031110K(4185792K), 0.0425745 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:51.310+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.310+0800: [ParNew: 76672K->8512K(76672K), 0.0484108 secs] 3097352K->3039524K(4185792K), 0.0484893 secs] [Times: user=0.04 sys=0.01, real=0.04 secs]
    2021-08-10T14:51:51.370+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.370+0800: [ParNew: 76672K->8397K(76672K), 0.0437014 secs] 3106640K->3046419K(4185792K), 0.0437746 secs] [Times: user=0.05 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:51.422+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.422+0800: [ParNew: 76557K->7104K(76672K), 0.0400171 secs] 3112953K->3049574K(4185792K), 0.0400884 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:51.473+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.473+0800: [ParNew: 75264K->8512K(76672K), 0.0368263 secs] 3116982K->3054561K(4185792K), 0.0369065 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:51.523+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.523+0800: [ParNew: 76672K->7656K(76672K), 0.0460787 secs] 3121546K->3065556K(4185792K), 0.0461540 secs] [Times: user=0.05 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:51.580+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.580+0800: [ParNew: 75816K->7498K(76672K), 0.0468142 secs] 3131558K->3069979K(4185792K), 0.0469190 secs] [Times: user=0.05 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:51.636+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.636+0800: [ParNew: 75658K->8512K(76672K), 0.0425241 secs] 3137002K->3074813K(4185792K), 0.0426331 secs] [Times: user=0.04 sys=0.01, real=0.04 secs]
    2021-08-10T14:51:51.697+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.697+0800: [ParNew: 76672K->8512K(76672K), 0.0428190 secs] 3142038K->3080283K(4185792K), 0.0429126 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:51.749+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.750+0800: [ParNew: 76672K->8512K(76672K), 0.0386611 secs] 3147631K->3085888K(4185792K), 0.0387424 secs] [Times: user=0.03 sys=0.01, real=0.04 secs]
    2021-08-10T14:51:51.799+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.799+0800: [ParNew: 76672K->8512K(76672K), 0.0412278 secs] 3151520K->3092529K(4185792K), 0.0413260 secs] [Times: user=0.04 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:51.853+0800: [GC (GCLocker Initiated GC) 2021-08-10T14:51:51.853+0800: [ParNew: 76672K->8512K(76672K), 0.0452774 secs] 3158908K->3099394K(4185792K), 0.0453738 secs] [Times: user=0.03 sys=0.01, real=0.04 secs]
    2021-08-10T14:51:51.909+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.909+0800: [ParNew: 76672K->8474K(76672K), 0.0494126 secs] 3164339K->3104267K(4185792K), 0.0494995 secs] [Times: user=0.04 sys=0.01, real=0.05 secs]
    2021-08-10T14:51:51.970+0800: [GC (Allocation Failure) 2021-08-10T14:51:51.970+0800: [ParNew: 76634K->8299K(76672K), 0.0509132 secs] 3169915K->3109519K(4185792K), 0.0510104 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:52.029+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.029+0800: [ParNew: 76459K->8208K(76672K), 0.0373005 secs] 3176254K->3115580K(4185792K), 0.0373899 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:52.074+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.074+0800: [ParNew: 76368K->7312K(76672K), 0.0367450 secs] 3181244K->3115986K(4185792K), 0.0368579 secs] [Times: user=0.03 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:52.120+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.120+0800: [ParNew: 75472K->8512K(76672K), 0.0401914 secs] 3183208K->3121999K(4185792K), 0.0403127 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:52.170+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.170+0800: [ParNew: 76672K->8512K(76672K), 0.0426470 secs] 3188910K->3130015K(4185792K), 0.0427650 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:52.230+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.230+0800: [ParNew: 76672K->8512K(76672K), 0.0503207 secs] 3192776K->3134704K(4185792K), 0.0504159 secs] [Times: user=0.03 sys=0.02, real=0.06 secs]
    2021-08-10T14:51:52.293+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.293+0800: [ParNew: 76672K->7979K(76672K), 0.0390631 secs] 3195745K->3135076K(4185792K), 0.0391420 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:52.342+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.342+0800: [ParNew: 76139K->7490K(76672K), 0.0349573 secs] 3197410K->3136267K(4185792K), 0.0350359 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
    2021-08-10T14:51:52.386+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.386+0800: [ParNew: 75650K->6531K(76672K), 0.0309209 secs] 3197520K->3133630K(4185792K), 0.0309972 secs] [Times: user=0.02 sys=0.01, real=0.03 secs]
    2021-08-10T14:51:52.425+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.425+0800: [ParNew: 74691K->8512K(76672K), 0.0336670 secs] 3197654K->3136810K(4185792K), 0.0337406 secs] [Times: user=0.03 sys=0.01, real=0.03 secs]
    2021-08-10T14:51:52.471+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.471+0800: [ParNew: 76672K->8512K(76672K), 0.0397881 secs] 3198152K->3138106K(4185792K), 0.0398787 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:52.521+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.521+0800: [ParNew: 76672K->8512K(76672K), 0.0328564 secs] 3202124K->3141177K(4185792K), 0.0329334 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
    2021-08-10T14:51:52.568+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.568+0800: [ParNew: 76672K->8512K(76672K), 0.0413346 secs] 3205766K->3145864K(4185792K), 0.0414457 secs] [Times: user=0.03 sys=0.01, real=0.05 secs]
    2021-08-10T14:51:52.624+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.624+0800: [ParNew: 76672K->7401K(76672K), 0.0427615 secs] 3212268K->3150330K(4185792K), 0.0428648 secs] [Times: user=0.05 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:52.686+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.686+0800: [ParNew: 75561K->6884K(76672K), 0.0400685 secs] 3215185K->3152788K(4185792K), 0.0401657 secs] [Times: user=0.03 sys=0.01, real=0.04 secs]
    2021-08-10T14:51:52.737+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.737+0800: [ParNew: 75044K->5857K(76672K), 0.0513699 secs] 3218967K->3154707K(4185792K), 0.0514839 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:52.797+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.797+0800: [ParNew: 74017K->5053K(76672K), 0.0406744 secs] 3221868K->3156472K(4185792K), 0.0407750 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:52.847+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.847+0800: [ParNew: 73213K->8512K(76672K), 0.0397704 secs] 3223979K->3161810K(4185792K), 0.0398779 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:52.899+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.899+0800: [ParNew: 76672K->7849K(76672K), 0.0438958 secs] 3229360K->3168201K(4185792K), 0.0440062 secs] [Times: user=0.03 sys=0.01, real=0.05 secs]
    2021-08-10T14:51:52.957+0800: [GC (Allocation Failure) 2021-08-10T14:51:52.957+0800: [ParNew: 76009K->7763K(76672K), 0.0364444 secs] 3233300K->3176812K(4185792K), 0.0365427 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:53.006+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.006+0800: [ParNew: 75923K->6555K(76672K), 0.0344456 secs] 3241790K->3177269K(4185792K), 0.0345565 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:53.057+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.057+0800: [ParNew: 74715K->8194K(76672K), 0.0315442 secs] 3240466K->3180486K(4185792K), 0.0316234 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
    2021-08-10T14:51:53.101+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.101+0800: [ParNew: 76354K->6693K(76672K), 0.0402031 secs] 3246292K->3183405K(4185792K), 0.0403126 secs] [Times: user=0.03 sys=0.02, real=0.04 secs]
    2021-08-10T14:51:53.151+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.151+0800: [ParNew: 74853K->8512K(76672K), 0.0368653 secs] 3248774K->3186488K(4185792K), 0.0369517 secs] [Times: user=0.04 sys=0.00, real=0.03 secs]
    2021-08-10T14:51:53.201+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.201+0800: [ParNew: 76672K->7846K(76672K), 0.0483054 secs] 3251478K->3189903K(4185792K), 0.0484085 secs] [Times: user=0.05 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:53.260+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.260+0800: [ParNew: 76006K->8512K(76672K), 0.0358117 secs] 3256451K->3193882K(4185792K), 0.0359015 secs] [Times: user=0.03 sys=0.01, real=0.03 secs]
    2021-08-10T14:51:53.320+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.320+0800: [ParNew: 76672K->7734K(76672K), 0.0464199 secs] 3253264K->3193290K(4185792K), 0.0465162 secs] [Times: user=0.03 sys=0.01, real=0.04 secs]
    2021-08-10T14:51:53.378+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.378+0800: [ParNew: 75894K->6746K(76672K), 0.0378897 secs] 3259489K->3194830K(4185792K), 0.0379893 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:53.455+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.455+0800: [ParNew: 74906K->8512K(76672K), 0.0650141 secs] 3258682K->3197584K(4185792K), 0.0651756 secs] [Times: user=0.05 sys=0.00, real=0.07 secs]
    2021-08-10T14:51:53.552+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.552+0800: [ParNew: 76672K->5695K(76672K), 0.0929924 secs] 3260641K->3196173K(4185792K), 0.0931994 secs] [Times: user=0.05 sys=0.01, real=0.09 secs]
    2021-08-10T14:51:53.684+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.684+0800: [ParNew: 73855K->5042K(76672K), 0.0615285 secs] 3259842K->3194613K(4185792K), 0.0617298 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
    2021-08-10T14:51:53.768+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.768+0800: [ParNew: 73202K->8512K(76672K), 0.0583413 secs] 3260122K->3196967K(4185792K), 0.0585096 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
    2021-08-10T14:51:53.856+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.856+0800: [ParNew: 76672K->5750K(76672K), 0.0667699 secs] 3255177K->3189423K(4185792K), 0.0669035 secs] [Times: user=0.05 sys=0.01, real=0.07 secs]
    2021-08-10T14:51:53.941+0800: [GC (Allocation Failure) 2021-08-10T14:51:53.941+0800: [ParNew: 73910K->8512K(76672K), 0.0709935 secs] 3255437K->3198192K(4185792K), 0.0711401 secs] [Times: user=0.06 sys=0.00, real=0.07 secs]
    2021-08-10T14:51:54.041+0800: [GC (Allocation Failure) 2021-08-10T14:51:54.041+0800: [ParNew: 76672K->6788K(76672K), 0.0488584 secs] 3264395K->3202314K(4185792K), 0.0492206 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:54.116+0800: [GC (Allocation Failure) 2021-08-10T14:51:54.116+0800: [ParNew: 74948K->8512K(76672K), 0.0605065 secs] 3269525K->3208524K(4185792K), 0.0606492 secs] [Times: user=0.04 sys=0.00, real=0.06 secs]
    2021-08-10T14:51:54.199+0800: [GC (Allocation Failure) 2021-08-10T14:51:54.199+0800: [ParNew: 76672K->8512K(76672K), 0.0437344 secs] 3274882K->3211719K(4185792K), 0.0439084 secs] [Times: user=0.04 sys=0.01, real=0.05 secs]
    2021-08-10T14:51:54.298+0800: [GC (Allocation Failure) 2021-08-10T14:51:54.298+0800: [ParNew: 76672K->8512K(76672K), 0.0620067 secs] 3276579K->3215324K(4185792K), 0.0621579 secs] [Times: user=0.05 sys=0.00, real=0.07 secs]
    2021-08-10T14:51:54.425+0800: [GC (Allocation Failure) 2021-08-10T14:51:54.425+0800: [ParNew: 76629K->8054K(76672K), 0.0450103 secs] 3276955K->3216113K(4185792K), 0.0451683 secs] [Times: user=0.03 sys=0.01, real=0.05 secs]
    2021-08-10T14:51:54.508+0800: [GC (Allocation Failure) 2021-08-10T14:51:54.508+0800: [ParNew: 76214K->8512K(76672K), 0.0432830 secs] 3281493K->3214555K(4185792K), 0.0434589 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:51:54.580+0800: [GC (Allocation Failure) 2021-08-10T14:51:54.580+0800: [ParNew: 76672K->5880K(76672K), 0.0419076 secs] 3276456K->3209229K(4185792K), 0.0420531 secs] [Times: user=0.05 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:54.709+0800: [GC (Allocation Failure) 2021-08-10T14:51:54.709+0800: [ParNew: 73999K->8512K(76672K), 0.0342041 secs] 3266325K->3200838K(4185792K), 0.0343394 secs] [Times: user=0.03 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:54.832+0800: [GC (Allocation Failure) 2021-08-10T14:51:54.832+0800: [ParNew: 76670K->5527K(76672K), 0.0423911 secs] 3257665K->3192776K(4185792K), 0.0425222 secs] [Times: user=0.03 sys=0.01, real=0.04 secs]
    2021-08-10T14:51:54.947+0800: [GC (Allocation Failure) 2021-08-10T14:51:54.947+0800: [ParNew: 73687K->4075K(76672K), 0.0339701 secs] 3248652K->3179040K(4185792K), 0.0341424 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:51:55.107+0800: [GC (Allocation Failure) 2021-08-10T14:51:55.107+0800: [ParNew: 72235K->2146K(76672K), 0.0572441 secs] 3225565K->3155476K(4185792K), 0.0574210 secs] [Times: user=0.04 sys=0.01, real=0.06 secs]
    2021-08-10T14:51:55.417+0800: [GC (Allocation Failure) 2021-08-10T14:51:55.417+0800: [ParNew: 70306K->8512K(76672K), 0.1606864 secs] 3211889K->3154957K(4185792K), 0.1608645 secs] [Times: user=0.14 sys=0.00, real=0.16 secs]
    2021-08-10T14:51:56.005+0800: [GC (Allocation Failure) 2021-08-10T14:51:56.006+0800: [ParNew: 76672K->8512K(76672K), 0.3085277 secs] 3180847K->3134649K(4185792K), 0.3089095 secs] [Times: user=0.31 sys=0.00, real=0.31 secs]
    2021-08-10T14:51:56.527+0800: [GC (Allocation Failure) 2021-08-10T14:51:56.527+0800: [ParNew: 76672K->8512K(76672K), 0.1977908 secs] 3164005K->3113200K(4185792K), 0.1979444 secs] [Times: user=0.16 sys=0.03, real=0.20 secs]
    2021-08-10T14:51:56.991+0800: [GC (Allocation Failure) 2021-08-10T14:51:56.991+0800: [ParNew: 76672K->8512K(76672K), 0.2410614 secs] 3129032K->3085993K(4185792K), 0.2411975 secs] [Times: user=0.24 sys=0.00, real=0.24 secs]
    2021-08-10T14:51:57.384+0800: [GC (Allocation Failure) 2021-08-10T14:51:57.384+0800: [ParNew: 76672K->8512K(76672K), 0.2582527 secs] 3102988K->3056474K(4185792K), 0.2584169 secs] [Times: user=0.26 sys=0.01, real=0.26 secs]
    2021-08-10T14:51:57.865+0800: [GC (Allocation Failure) 2021-08-10T14:51:57.865+0800: [ParNew: 76672K->8512K(76672K), 0.2842812 secs] 3103749K->3052060K(4185792K), 0.2844181 secs] [Times: user=0.19 sys=0.00, real=0.28 secs]
    2021-08-10T14:51:58.260+0800: [CMS-concurrent-sweep: 3.436/9.546 secs] [Times: user=31.79 sys=1.32, real=9.55 secs]
    2021-08-10T14:51:58.260+0800: [CMS-concurrent-reset-start]
    2021-08-10T14:51:58.285+0800: [CMS-concurrent-reset: 0.026/0.026 secs] [Times: user=0.10 sys=0.01, real=0.02 secs]
    2021-08-10T14:51:58.412+0800: [GC (Allocation Failure) 2021-08-10T14:51:58.412+0800: [ParNew: 76672K->8510K(76672K), 0.3329774 secs] 2824888K->2785491K(4185792K), 0.3331950 secs] [Times: user=0.32 sys=0.00, real=0.33 secs]
    2021-08-10T14:51:58.959+0800: [GC (Allocation Failure) 2021-08-10T14:51:58.959+0800: [ParNew: 76646K->8510K(76672K), 0.2450121 secs] 2853627K->2810727K(4185792K), 0.2451972 secs] [Times: user=0.24 sys=0.00, real=0.25 secs]
    2021-08-10T14:51:59.362+0800: [GC (Allocation Failure) 2021-08-10T14:51:59.362+0800: [ParNew: 76670K->8510K(76672K), 0.1749537 secs] 2878887K->2823111K(4185792K), 0.1751192 secs] [Times: user=0.18 sys=0.00, real=0.17 secs]
    2021-08-10T14:51:59.562+0800: [GC (Allocation Failure) 2021-08-10T14:51:59.562+0800: [ParNew: 76670K->8510K(76672K), 0.1022924 secs] 2891271K->2832777K(4185792K), 0.1024260 secs] [Times: user=0.08 sys=0.01, real=0.10 secs]
    2021-08-10T14:51:59.704+0800: [GC (Allocation Failure) 2021-08-10T14:51:59.704+0800: [ParNew: 76670K->8510K(76672K), 0.0780837 secs] 2900937K->2840444K(4185792K), 0.0782226 secs] [Times: user=0.07 sys=0.02, real=0.08 secs]
    2021-08-10T14:51:59.814+0800: [GC (Allocation Failure) 2021-08-10T14:51:59.814+0800: [ParNew: 76670K->8512K(76672K), 0.1094483 secs] 2908604K->2846490K(4185792K), 0.1096095 secs] [Times: user=0.11 sys=0.00, real=0.11 secs]
    2021-08-10T14:51:59.953+0800: [GC (Allocation Failure) 2021-08-10T14:51:59.953+0800: [ParNew: 76672K->8079K(76672K), 0.0929354 secs] 2914650K->2851766K(4185792K), 0.0931272 secs] [Times: user=0.09 sys=0.00, real=0.09 secs]
    2021-08-10T14:52:00.066+0800: [GC (Allocation Failure) 2021-08-10T14:52:00.066+0800: [ParNew: 76238K->8512K(76672K), 0.0810710 secs] 2919925K->2855938K(4185792K), 0.0812067 secs] [Times: user=0.07 sys=0.01, real=0.08 secs]
    2021-08-10T14:52:00.174+0800: [GC (Allocation Failure) 2021-08-10T14:52:00.174+0800: [ParNew: 76672K->8512K(76672K), 0.0700912 secs] 2924098K->2862790K(4185792K), 0.0702544 secs] [Times: user=0.07 sys=0.00, real=0.07 secs]
    2021-08-10T14:52:00.263+0800: [GC (Allocation Failure) 2021-08-10T14:52:00.263+0800: [ParNew: 76672K->8175K(76672K), 0.0549011 secs] 2930950K->2870154K(4185792K), 0.0549915 secs] [Times: user=0.05 sys=0.01, real=0.05 secs]
    2021-08-10T14:52:00.335+0800: [GC (Allocation Failure) 2021-08-10T14:52:00.335+0800: [ParNew: 76335K->7596K(76672K), 0.0487672 secs] 2938314K->2875396K(4185792K), 0.0488938 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:52:00.398+0800: [GC (Allocation Failure) 2021-08-10T14:52:00.398+0800: [ParNew: 75756K->6779K(76672K), 0.0431132 secs] 2943556K->2880817K(4185792K), 0.0432218 secs] [Times: user=0.04 sys=0.00, real=0.05 secs]
    2021-08-10T14:52:00.457+0800: [GC (Allocation Failure) 2021-08-10T14:52:00.457+0800: [ParNew: 74347K->6318K(76672K), 0.0494566 secs] 2948385K->2886503K(4185792K), 0.0495888 secs] [Times: user=0.05 sys=0.01, real=0.05 secs]
    2021-08-10T14:52:00.508+0800: [GC (CMS Initial Mark) [1 CMS-initial-mark: 2880184K(4109120K)] 2888212K(4185792K), 0.0055409 secs] [Times: user=0.00 sys=0.01, real=0.01 secs]
    2021-08-10T14:52:00.514+0800: [CMS-concurrent-mark-start]
    2021-08-10T14:52:00.553+0800: [GC (Allocation Failure) 2021-08-10T14:52:00.553+0800: [ParNew: 74478K->8512K(76672K), 0.0694369 secs] 2954663K->2899289K(4185792K), 0.0695788 secs] [Times: user=0.07 sys=0.00, real=0.07 secs]
    2021-08-10T14:52:00.645+0800: [GC (Allocation Failure) 2021-08-10T14:52:00.645+0800: [ParNew: 76672K->6658K(76672K), 0.0778403 secs] 2967449K->2903596K(4185792K), 0.0780144 secs] [Times: user=0.06 sys=0.00, real=0.08 secs]
    2021-08-10T14:52:00.744+0800: [GC (Allocation Failure) 2021-08-10T14:52:00.744+0800: [ParNew: 74818K->8090K(76672K), 0.0732794 secs] 2971756K->2910155K(4185792K), 0.0735057 secs] [Times: user=0.06 sys=0.02, real=0.07 secs]
    2021-08-10T14:52:00.845+0800: [GC (Allocation Failure) 2021-08-10T14:52:00.846+0800: [ParNew: 76250K->8512K(76672K), 0.0618596 secs] 2978315K->2914855K(4185792K), 0.0620692 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
    2021-08-10T14:52:00.932+0800: [GC (Allocation Failure) 2021-08-10T14:52:00.932+0800: [ParNew: 76672K->7485K(76672K), 0.0742042 secs] 2983015K->2922106K(4185792K), 0.0743640 secs] [Times: user=0.08 sys=0.00, real=0.07 secs]
    2021-08-10T14:52:01.032+0800: [GC (Allocation Failure) 2021-08-10T14:52:01.032+0800: [ParNew: 75645K->8512K(76672K), 0.0690928 secs] 2990266K->2928220K(4185792K), 0.0692745 secs] [Times: user=0.05 sys=0.01, real=0.07 secs]
    2021-08-10T14:52:01.146+0800: [GC (Allocation Failure) 2021-08-10T14:52:01.146+0800: [ParNew: 76672K->6054K(76672K), 0.0871127 secs] 2996380K->2933517K(4185792K), 0.0873339 secs] [Times: user=0.09 sys=0.00, real=0.09 secs]
    2021-08-10T14:52:01.269+0800: [GC (Allocation Failure) 2021-08-10T14:52:01.269+0800: [ParNew: 74214K->5615K(76672K), 0.0645569 secs] 3001677K->2938358K(4185792K), 0.0646747 secs] [Times: user=0.06 sys=0.01, real=0.07 secs]
    2021-08-10T14:52:01.377+0800: [GC (Allocation Failure) 2021-08-10T14:52:01.377+0800: [ParNew: 73775K->8512K(76672K), 0.1229126 secs] 3006518K->2947913K(4185792K), 0.1230907 secs] [Times: user=0.08 sys=0.00, real=0.13 secs]
    2021-08-10T14:52:01.536+0800: [GC (Allocation Failure) 2021-08-10T14:52:01.536+0800: [ParNew: 76672K->8512K(76672K), 0.0783862 secs] 3016073K->2957694K(4185792K), 0.0785993 secs] [Times: user=0.08 sys=0.00, real=0.08 secs]
    2021-08-10T14:52:01.649+0800: [GC (Allocation Failure) 2021-08-10T14:52:01.649+0800: [ParNew: 76672K->6880K(76672K), 0.0881369 secs] 3025854K->2968143K(4185792K), 0.0883075 secs] [Times: user=0.06 sys=0.01, real=0.09 secs]
    2021-08-10T14:52:01.780+0800: [GC (Allocation Failure) 2021-08-10T14:52:01.780+0800: [ParNew: 75040K->6525K(76672K), 0.0596453 secs] 3036303K->2973406K(4185792K), 0.0598347 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
    2021-08-10T14:52:01.871+0800: [GC (Allocation Failure) 2021-08-10T14:52:01.871+0800: [ParNew: 74685K->8512K(76672K), 0.0616404 secs] 3041566K->2980273K(4185792K), 0.0618093 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
    2021-08-10T14:52:01.978+0800: [GC (Allocation Failure) 2021-08-10T14:52:01.978+0800: [ParNew: 76672K->8512K(76672K), 0.0650502 secs] 3048433K->2990875K(4185792K), 0.0651814 secs] [Times: user=0.04 sys=0.02, real=0.07 secs]
    2021-08-10T14:52:02.078+0800: [GC (Allocation Failure) 2021-08-10T14:52:02.078+0800: [ParNew: 76672K->7966K(76672K), 0.0602499 secs] 3059035K->2997557K(4185792K), 0.0604530 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
    2021-08-10T14:52:02.176+0800: [GC (Allocation Failure) 2021-08-10T14:52:02.176+0800: [ParNew: 76126K->8512K(76672K), 0.0846645 secs] 3065717K->3006788K(4185792K), 0.0848287 secs] [Times: user=0.07 sys=0.00, real=0.09 secs]
    2021-08-10T14:52:02.293+0800: [GC (Allocation Failure) 2021-08-10T14:52:02.293+0800: [ParNew: 76672K->6723K(76672K), 0.0608728 secs] 3074948K->3012473K(4185792K), 0.0611540 secs] [Times: user=0.05 sys=0.01, real=0.06 secs]
    2021-08-10T14:52:02.385+0800: [GC (Allocation Failure) 2021-08-10T14:52:02.385+0800: [ParNew: 74883K->8512K(76672K), 0.0599966 secs] 3080633K->3019937K(4185792K), 0.0601551 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
    2021-08-10T14:52:02.463+0800: [GC (Allocation Failure) 2021-08-10T14:52:02.464+0800: [ParNew: 76672K->7683K(76672K), 0.0632457 secs] 3088097K->3026853K(4185792K), 0.0634091 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
    2021-08-10T14:52:02.550+0800: [GC (Allocation Failure) 2021-08-10T14:52:02.550+0800: [ParNew: 75843K->7519K(76672K), 0.0609457 secs] 3095013K->3034132K(4185792K), 0.0611051 secs] [Times: user=0.05 sys=0.01, real=0.06 secs]
    2021-08-10T14:52:02.633+0800: [GC (Allocation Failure) 2021-08-10T14:52:02.633+0800: [ParNew: 75679K->5616K(76672K), 0.0475727 secs] 3102292K->3037203K(4185792K), 0.0477518 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:52:02.702+0800: [GC (Allocation Failure) 2021-08-10T14:52:02.702+0800: [ParNew: 73776K->8512K(76672K), 0.0487756 secs] 3105363K->3045465K(4185792K), 0.0489264 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:52:02.776+0800: [GC (Allocation Failure) 2021-08-10T14:52:02.776+0800: [ParNew: 76672K->8512K(76672K), 0.0585709 secs] 3113625K->3050873K(4185792K), 0.0587311 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
    2021-08-10T14:52:02.871+0800: [GC (Allocation Failure) 2021-08-10T14:52:02.871+0800: [ParNew: 76672K->7355K(76672K), 0.0531329 secs] 3119033K->3057011K(4185792K), 0.0533236 secs] [Times: user=0.04 sys=0.01, real=0.05 secs]
    2021-08-10T14:52:02.964+0800: [GC (Allocation Failure) 2021-08-10T14:52:02.964+0800: [ParNew: 75515K->8512K(76672K), 0.0683422 secs] 3125171K->3063394K(4185792K), 0.0684656 secs] [Times: user=0.05 sys=0.00, real=0.07 secs]
    2021-08-10T14:52:03.058+0800: [GC (Allocation Failure) 2021-08-10T14:52:03.058+0800: [ParNew: 76672K->8426K(76672K), 0.0555080 secs] 3131554K->3069489K(4185792K), 0.0556489 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
    2021-08-10T14:52:03.165+0800: [GC (Allocation Failure) 2021-08-10T14:52:03.165+0800: [ParNew: 76586K->8512K(76672K), 0.0586728 secs] 3137649K->3078755K(4185792K), 0.0588389 secs] [Times: user=0.04 sys=0.02, real=0.06 secs]
    2021-08-10T14:52:03.240+0800: [GC (Allocation Failure) 2021-08-10T14:52:03.240+0800: [ParNew: 76672K->6694K(76672K), 0.0499738 secs] 3146915K->3085419K(4185792K), 0.0501495 secs] [Times: user=0.04 sys=0.01, real=0.05 secs]
    2021-08-10T14:52:03.353+0800: [GC (Allocation Failure) 2021-08-10T14:52:03.353+0800: [ParNew: 74323K->8512K(76672K), 0.0571585 secs] 3153048K->3095150K(4185792K), 0.0573519 secs] [Times: user=0.05 sys=0.00, real=0.06 secs]
    2021-08-10T14:52:03.432+0800: [GC (Allocation Failure) 2021-08-10T14:52:03.432+0800: [ParNew: 76672K->8512K(76672K), 0.0576388 secs] 3163310K->3102530K(4185792K), 0.0578180 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
    2021-08-10T14:52:03.513+0800: [GC (Allocation Failure) 2021-08-10T14:52:03.513+0800: [ParNew: 76672K->7076K(76672K), 0.0446805 secs] 3170690K->3108313K(4185792K), 0.0447927 secs] [Times: user=0.03 sys=0.01, real=0.04 secs]
    2021-08-10T14:52:03.580+0800: [GC (Allocation Failure) 2021-08-10T14:52:03.581+0800: [ParNew: 75236K->8512K(76672K), 0.0627490 secs] 3176473K->3118603K(4185792K), 0.0628633 secs] [Times: user=0.06 sys=0.00, real=0.06 secs]
    2021-08-10T14:52:03.674+0800: [GC (Allocation Failure) 2021-08-10T14:52:03.674+0800: [ParNew: 76672K->8512K(76672K), 0.0459355 secs] 3186763K->3126292K(4185792K), 0.0460489 secs] [Times: user=0.04 sys=0.00, real=0.05 secs]
    2021-08-10T14:52:03.740+0800: [GC (Allocation Failure) 2021-08-10T14:52:03.740+0800: [ParNew: 76672K->8041K(76672K), 0.0438355 secs] 3194452K->3132549K(4185792K), 0.0439911 secs] [Times: user=0.03 sys=0.01, real=0.04 secs]
    2021-08-10T14:52:03.810+0800: [GC (Allocation Failure) 2021-08-10T14:52:03.810+0800: [ParNew: 76201K->7044K(76672K), 0.0531183 secs] 3200709K->3137343K(4185792K), 0.0532748 secs] [Times: user=0.06 sys=0.00, real=0.05 secs]
    2021-08-10T14:52:03.884+0800: [GC (Allocation Failure) 2021-08-10T14:52:03.884+0800: [ParNew: 75204K->7012K(76672K), 0.0444330 secs] 3205503K->3143301K(4185792K), 0.0445440 secs] [Times: user=0.05 sys=0.00, real=0.04 secs]
    2021-08-10T14:52:03.964+0800: [GC (Allocation Failure) 2021-08-10T14:52:03.964+0800: [ParNew: 74337K->8512K(76672K), 0.0425666 secs] 3210627K->3149693K(4185792K), 0.0426866 secs] [Times: user=0.03 sys=0.01, real=0.04 secs]
    2021-08-10T14:52:04.037+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.037+0800: [ParNew: 76672K->8512K(76672K), 0.0449836 secs] 3217853K->3161059K(4185792K), 0.0450881 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:52:04.099+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.099+0800: [ParNew: 76672K->8149K(76672K), 0.0487994 secs] 3229219K->3167200K(4185792K), 0.0489302 secs] [Times: user=0.04 sys=0.01, real=0.05 secs]
    2021-08-10T14:52:04.161+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.161+0800: [ParNew: 76309K->7671K(76672K), 0.0497549 secs] 3235360K->3170125K(4185792K), 0.0499380 secs] [Times: user=0.03 sys=0.02, real=0.05 secs]
    2021-08-10T14:52:04.224+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.224+0800: [ParNew: 75831K->5734K(76672K), 0.0428152 secs] 3238285K->3175419K(4185792K), 0.0429549 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:52:04.279+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.279+0800: [ParNew: 73894K->8512K(76672K), 0.0334411 secs] 3243579K->3179043K(4185792K), 0.0335398 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:52:04.324+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.324+0800: [ParNew: 76672K->7447K(76672K), 0.0485755 secs] 3247203K->3189113K(4185792K), 0.0486971 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
    2021-08-10T14:52:04.398+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.399+0800: [ParNew: 75607K->7053K(76672K), 0.0469127 secs] 3257273K->3196761K(4185792K), 0.0470274 secs] [Times: user=0.04 sys=0.01, real=0.05 secs]
    2021-08-10T14:52:04.461+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.461+0800: [ParNew: 75213K->6678K(76672K), 0.0379589 secs] 3264921K->3201471K(4185792K), 0.0380645 secs] [Times: user=0.04 sys=0.00, real=0.03 secs]
    2021-08-10T14:52:04.522+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.522+0800: [ParNew: 74838K->5672K(76672K), 0.0339250 secs] 3269631K->3204124K(4185792K), 0.0340281 secs] [Times: user=0.04 sys=0.00, real=0.03 secs]
    2021-08-10T14:52:04.569+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.569+0800: [ParNew: 73832K->8512K(76672K), 0.0423512 secs] 3272284K->3213689K(4185792K), 0.0424905 secs] [Times: user=0.04 sys=0.00, real=0.05 secs]
    2021-08-10T14:52:04.632+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.632+0800: [ParNew: 76672K->6503K(76672K), 0.0386608 secs] 3281849K->3219342K(4185792K), 0.0387709 secs] [Times: user=0.03 sys=0.01, real=0.04 secs]
    2021-08-10T14:52:04.685+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.685+0800: [ParNew: 74663K->7971K(76672K), 0.0354556 secs] 3287502K->3225065K(4185792K), 0.0355460 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:52:04.804+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.804+0800: [ParNew: 76080K->5046K(76672K), 0.0337686 secs] 3293175K->3227973K(4185792K), 0.0338702 secs] [Times: user=0.03 sys=0.01, real=0.03 secs]
    2021-08-10T14:52:04.869+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.869+0800: [ParNew: 73206K->8512K(76672K), 0.0306703 secs] 3296133K->3231439K(4185792K), 0.0307643 secs] [Times: user=0.02 sys=0.01, real=0.04 secs]
    2021-08-10T14:52:04.929+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.929+0800: [ParNew: 76672K->3825K(76672K), 0.0289216 secs] 3299599K->3228937K(4185792K), 0.0290304 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
    2021-08-10T14:52:04.976+0800: [GC (Allocation Failure) 2021-08-10T14:52:04.977+0800: [ParNew: 71985K->5791K(76672K), 0.0300249 secs] 3297097K->3230903K(4185792K), 0.0301462 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
    2021-08-10T14:52:05.020+0800: [CMS-concurrent-mark: 1.135/4.506 secs] [Times: user=8.62 sys=0.50, real=4.51 secs]
    2021-08-10T14:52:05.020+0800: [CMS-concurrent-preclean-start]
    2021-08-10T14:52:05.108+0800: [CMS-concurrent-preclean: 0.085/0.087 secs] [Times: user=0.18 sys=0.01, real=0.08 secs]
    2021-08-10T14:52:05.108+0800: [CMS-concurrent-abortable-preclean-start]
    2021-08-10T14:52:05.117+0800: [GC (Allocation Failure) 2021-08-10T14:52:05.117+0800: [ParNew: 73951K->4097K(76672K), 0.0345694 secs] 3299063K->3229215K(4185792K), 0.0346693 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:52:05.165+0800: [CMS-concurrent-abortable-preclean: 0.020/0.057 secs] [Times: user=0.15 sys=0.00, real=0.06 secs]
    2021-08-10T14:52:05.166+0800: [GC (CMS Final Remark) [YG occupancy: 51668 K (76672 K)]2021-08-10T14:52:05.166+0800: [Rescan (parallel) , 0.0215304 secs]2021-08-10T14:52:05.188+0800: [weak refs processing, 0.0001314 secs]2021-08-10T14:52:05.188+0800: [class unloading, 0.0571061 secs]2021-08-10T14:52:05.245+0800: [scrub symbol table, 0.0128483 secs]2021-08-10T14:52:05.258+0800: [scrub string table, 0.0012081 secs][1 CMS-remark: 3225118K(4109120K)] 3276786K(4185792K), 0.0929698 secs] [Times: user=0.09 sys=0.00, real=0.09 secs]
    2021-08-10T14:52:05.259+0800: [CMS-concurrent-sweep-start]
    2021-08-10T14:52:05.264+0800: [GC (Allocation Failure) 2021-08-10T14:52:05.264+0800: [ParNew: 72257K->5375K(76672K), 0.0307294 secs] 3297361K->3230480K(4185792K), 0.0308196 secs] [Times: user=0.03 sys=0.01, real=0.03 secs]
    2021-08-10T14:52:05.311+0800: [GC (Allocation Failure) 2021-08-10T14:52:05.311+0800: [ParNew: 73535K->3765K(76672K), 0.0267926 secs] 3291303K->3221632K(4185792K), 0.0269032 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
    2021-08-10T14:52:05.359+0800: [GC (Allocation Failure) 2021-08-10T14:52:05.359+0800: [ParNew: 71925K->4820K(76672K), 0.0268482 secs] 3280749K->3213643K(4185792K), 0.0269288 secs] [Times: user=0.03 sys=0.01, real=0.03 secs]
    2021-08-10T14:52:05.640+0800: [GC (Allocation Failure) 2021-08-10T14:52:05.640+0800: [ParNew: 72931K->5346K(76672K), 0.0420384 secs] 3149713K->3082127K(4185792K), 0.0421933 secs] [Times: user=0.04 sys=0.00, real=0.04 secs]
    2021-08-10T14:52:05.958+0800: [GC (Allocation Failure) 2021-08-10T14:52:05.958+0800: [ParNew: 73506K->8512K(76672K), 0.1193349 secs] 2970402K->2917681K(4185792K), 0.1194334 secs] [Times: user=0.09 sys=0.03, real=0.12 secs]
    2021-08-10T14:52:06.231+0800: [GC (Allocation Failure) 2021-08-10T14:52:06.231+0800: [ParNew: 76672K->8512K(76672K), 0.1500085 secs] 2924015K->2871932K(4185792K), 0.1501161 secs] [Times: user=0.14 sys=0.01, real=0.15 secs]
    2021-08-10T14:52:06.523+0800: [GC (Allocation Failure) 2021-08-10T14:52:06.523+0800: [ParNew: 76672K->8512K(76672K), 0.1862389 secs] 2894255K->2850303K(4185792K), 0.1863612 secs] [Times: user=0.15 sys=0.03, real=0.19 secs]
    2021-08-10T14:52:06.841+0800: [GC (Allocation Failure) 2021-08-10T14:52:06.841+0800: [ParNew: 76672K->8512K(76672K), 0.1742799 secs] 2877944K->2833727K(4185792K), 0.1743679 secs] [Times: user=0.16 sys=0.02, real=0.17 secs]
    2021-08-10T14:52:07.145+0800: [GC (Allocation Failure) 2021-08-10T14:52:07.145+0800: [ParNew: 76672K->8512K(76672K), 0.1524512 secs] 2822532K->2773812K(4185792K), 0.1525384 secs] [Times: user=0.12 sys=0.03, real=0.15 secs]
    2021-08-10T14:52:07.378+0800: [CMS-concurrent-sweep: 1.202/2.119 secs] [Times: user=8.09 sys=0.38, real=2.12 secs]
    2021-08-10T14:52:07.378+0800: [CMS-concurrent-reset-start]

    1. 看gc日志其实还好 想这种并发清理耗时很长([CMS-concurrent-sweep: 1.202/2.119 secs] [Times: user=8.09 sys=0.38, real=2.12 secs])的问题不大,这个不会造成程序暂停。可能需要关注的是下面这些ygc,但是这些ygc不会造成RegionServer宕机。
      2021-08-10T14:51:58.412+0800: [GC (Allocation Failure) 2021-08-10T14:51:58.412+0800: [ParNew: 76672K->8510K(76672K), 0.3329774 secs] 2824888K->2785491K(4185792K), 0.3331950 secs] [Times: user=0.32 sys=0.00, real=0.33 secs]
      2021-08-10T14:51:58.959+0800: [GC (Allocation Failure) 2021-08-10T14:51:58.959+0800: [ParNew: 76646K->8510K(76672K), 0.2450121 secs] 2853627K->2810727K(4185792K), 0.2451972 secs] [Times: user=0.24 sys=0.00, real=0.25 secs]

发表评论

邮箱地址不会被公开。 必填项已用*标注