[arthas] UnsupportedOperationException: class redefinition failed: attempted to change the schema (add/remove fields)

问题描述

[arthas@1]$ trace cn.xx.dataservice.biz.dataservice.controller.v1.CommonSearchController datasetQuery
Affect(class count: 1 , method count: 1) cost in 262 ms, listenerId: 1
Enhance error! exception: java.lang.UnsupportedOperationException: class redefinition failed: attempted to change the schema (add/remove fields)
error happens when enhancing class: class redefinition failed: attempted to change the schema (add/remove fields), check arthas log: /root/logs/arthas/arthas.log

# tail -100 /root/logs/arthas/arthas.log
2022-11-24 13:57:21 [arthas-NettyHttpTelnetBootstrap-3-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xb7343ee3, L:/127.0.0.1:3658] ACTIVE
2022-11-24 13:57:21 [arthas-NettyWebsocketTtyBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x4b708046] REGISTERED
2022-11-24 13:57:21 [arthas-NettyWebsocketTtyBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x4b708046] BIND: /127.0.0.1:8563
2022-11-24 13:57:21 [arthas-NettyWebsocketTtyBootstrap-4-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0x4b708046, L:/127.0.0.1:8563] ACTIVE
2022-11-24 13:57:21 [arthas-NettyWebsocketTtyBootstrap-4-2] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xd9e71848] REGISTERED
2022-11-24 13:57:21 [arthas-NettyWebsocketTtyBootstrap-4-2] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xd9e71848] BIND: local:arthas-netty-localaddress
2022-11-24 13:57:21 [arthas-NettyWebsocketTtyBootstrap-4-2] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xd9e71848, L:local:arthas-netty-localaddress] ACTIVE
2022-11-24 13:57:21 [arthas-binding-thread] INFO  c.t.a.core.server.ArthasBootstrap -as-server listening on network=127.0.0.1;telnet=3658;http=8563;timeout=6000;
2022-11-24 13:57:21 [arthas-binding-thread] INFO  c.t.a.core.server.ArthasBootstrap -as-server started in 1285 ms
Arthas server already bind.
2022-11-24 13:57:21 [arthas-NettyHttpTelnetBootstrap-3-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xb7343ee3, L:/127.0.0.1:3658] READ: [id: 0x082ba92d, L:/127.0.0.1:3658 - R:/127.0.0.1:39284]
2022-11-24 13:57:21 [arthas-NettyHttpTelnetBootstrap-3-1] INFO  c.a.a.d.i.n.h.logging.LoggingHandler -[id: 0xb7343ee3, L:/127.0.0.1:3658] READ COMPLETE
2022-11-24 13:57:22 [arthas-NettyHttpTelnetBootstrap-3-2] INFO  c.t.a.core.shell.term.impl.Helper -Loaded arthas keymap file from com/taobao/arthas/core/shell/term/readline/inputrc
2022-11-24 13:57:35 [arthas-command-execute] INFO  c.t.arthas.core.advisor.Enhancer -enhance matched classes: [class cn.seres.bd.dataservice.biz.dataservice.controller.v1.CommonSearchController]
2022-11-24 13:57:35 [arthas-command-execute] ERROR c.t.arthas.core.advisor.Enhancer -Enhancer error, matchingClasses: [class cn.seres.bd.dataservice.biz.dataservice.controller.v1.CommonSearchController]
java.lang.UnsupportedOperationException: class redefinition failed: attempted to change the schema (add/remove fields)
        at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
        at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
        at com.taobao.arthas.core.advisor.Enhancer.enhance(Enhancer.java:396)
        at com.taobao.arthas.core.command.monitor200.EnhancerCommand.enhance(EnhancerCommand.java:168)
        at com.taobao.arthas.core.command.monitor200.EnhancerCommand.process(EnhancerCommand.java:115)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108)
        at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:385)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-11-24 14:00:07 [arthas-NettyHttpTelnetBootstrap-3-2] ERROR io.termd.core.io_error -Reported io error => closing
java.lang.NoClassDefFoundError: one/profiler/AsyncProfiler
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
        at java.lang.Class.getDeclaredMethod(Class.java:2128)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.shouldOverridesName(AnnotatedCommandImpl.java:33)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.name(AnnotatedCommandImpl.java:51)
        at com.taobao.arthas.core.shell.system.impl.InternalCommandManager.getCommand(InternalCommandManager.java:131)
        at com.taobao.arthas.core.shell.system.impl.InternalCommandManager.getCommand(InternalCommandManager.java:41)
        at com.taobao.arthas.core.shell.system.impl.JobControllerImpl.createProcess(JobControllerImpl.java:154)
        at com.taobao.arthas.core.shell.system.impl.JobControllerImpl.createJob(JobControllerImpl.java:88)
        at com.taobao.arthas.core.shell.system.impl.GlobalJobControllerImpl.createJob(GlobalJobControllerImpl.java:57)
        at com.taobao.arthas.core.shell.impl.ShellImpl.createJob(ShellImpl.java:98)
        at com.taobao.arthas.core.shell.handlers.shell.ShellLineHandler.createJob(ShellLineHandler.java:83)
        at com.taobao.arthas.core.shell.handlers.shell.ShellLineHandler.handle(ShellLineHandler.java:61)
        at com.taobao.arthas.core.shell.handlers.shell.ShellLineHandler.handle(ShellLineHandler.java:17)
        at com.taobao.arthas.core.shell.handlers.term.RequestHandler.accept(RequestHandler.java:22)
        at com.taobao.arthas.core.shell.handlers.term.RequestHandler.accept(RequestHandler.java:10)
        at io.termd.core.readline.Readline$Interaction.end(Readline.java:231)
        at io.termd.core.readline.Readline$Interaction.access$000(Readline.java:187)
        at io.termd.core.readline.Readline$2.apply(Readline.java:487)
        at io.termd.core.readline.Readline$Interaction.handle(Readline.java:272)
        at io.termd.core.readline.Readline$Interaction.access$200(Readline.java:187)
        at io.termd.core.readline.Readline.deliver(Readline.java:121)
        at io.termd.core.readline.Readline.access$500(Readline.java:38)
        at io.termd.core.readline.Readline$Interaction$2.accept(Readline.java:435)
        at io.termd.core.readline.Readline$Interaction$2.accept(Readline.java:429)
        at io.termd.core.tty.TtyEventDecoder.accept(TtyEventDecoder.java:93)
        at io.termd.core.tty.TtyEventDecoder.accept(TtyEventDecoder.java:26)
        at io.termd.core.tty.ReadBuffer.accept(ReadBuffer.java:44)
        at io.termd.core.tty.ReadBuffer.accept(ReadBuffer.java:28)
        at io.termd.core.io.BinaryDecoder.write(BinaryDecoder.java:117)
        at io.termd.core.io.BinaryDecoder.write(BinaryDecoder.java:65)
        at io.termd.core.telnet.TelnetTtyConnection.onData(TelnetTtyConnection.java:139)
        at io.termd.core.telnet.TelnetConnection.flushData(TelnetConnection.java:267)
        at io.termd.core.telnet.TelnetConnection.flushDataIfNecessary(TelnetConnection.java:257)
        at io.termd.core.telnet.TelnetConnection.receive(TelnetConnection.java:108)
        at io.termd.core.telnet.netty.TelnetChannelHandler.channelRead(TelnetChannelHandler.java:47)
        at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at com.alibaba.arthas.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at com.alibaba.arthas.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at com.alibaba.arthas.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at com.alibaba.arthas.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
        at com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
        at com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
        at com.alibaba.arthas.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at com.alibaba.arthas.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at com.alibaba.arthas.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at com.alibaba.arthas.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: one.profiler.AsyncProfiler
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at com.taobao.arthas.agent.ArthasClassloader.loadClass(ArthasClassloader.java:34)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 52 common frames omitted

[arthas@1]$ sm -d cn.seres.bd.dataservice.biz.dataservice.controller.v1.CommonSearchController
 declaring-class   cn.seres.bd.dataservice.biz.dataservice.controller.v1.CommonSearchController
 constructor-name  <init>
 modifier          public
 annotation
 parameters
 exceptions
 classLoaderHash   66629f63                                                                                                                                                                                                                                                      

 declaring-class   cn.xx.bd.dataservice.biz.dataservice.controller.v1.CommonSearchController
 constructor-name  <init>
 modifier          private
 annotation
 parameters        cn.xx.bd.dataservice.biz.dataservice.controller.v1.CommonSearchController$auxiliary$OrIwA4e9
 exceptions
 classLoaderHash   66629f63                                                                                                                                                                                                                                                      

 declaring-class  cn.xx.bd.dataservice.biz.dataservice.controller.v1.CommonSearchController
 method-name      datasetQuery$original$1pV75W4b
 modifier         private
 annotation
 parameters       java.lang.String
                  java.lang.String
                  cn.seres.bd.dataservice.common.dto.page.PageRequest
 return           cn.seres.bd.dataservice.common.dto.CommonResponse
 exceptions       cn.seres.bd.dataservice.common.exception.CommonException
 classLoaderHash  66629f63                                                                                                                                                                                                                                                       

 declaring-class  cn.xx.bd.dataservice.biz.dataservice.controller.v1.CommonSearchController
 method-name      datasetQuery
 modifier         public
 annotation       org.springframework.web.bind.annotation.PostMapping,io.swagger.annotations.ApiOperation
 parameters       java.lang.String
                  java.lang.String
                  cn.xx.bd.dataservice.common.dto.page.PageRequest
 return           cn.xx.bd.dataservice.common.dto.CommonResponse
 exceptions       cn.xx.bd.dataservice.common.exception.CommonException
 classLoaderHash  66629f63                                                                                                                                                                                                                                                       

 declaring-class  cn.xx.bd.dataservice.biz.dataservice.controller.v1.CommonSearchController
 method-name      datasetQuery$original$1pV75W4b$accessor$yu3zRiZn
 modifier         final
 annotation
 parameters       java.lang.String
                  java.lang.String
                  cn.xx.bd.dataservice.common.dto.page.PageRequest
 return           cn.xx.bd.dataservice.common.dto.CommonResponse
 exceptions       cn.xx.bd.dataservice.common.exception.CommonException
 classLoaderHash  66629f63                                                                                                                                                                                                                                                       

 declaring-class  cn.xx.bd.dataservice.biz.dataservice.controller.v1.CommonSearchController
 method-name      getSkyWalkingDynamicField
 modifier         public
 annotation
 parameters
 return           java.lang.Object
 exceptions
 classLoaderHash  66629f63                                                                                                                                                                                                                                                       

 declaring-class  cn.xx.bd.dataservice.biz.dataservice.controller.v1.CommonSearchController
 method-name      setSkyWalkingDynamicField
 modifier         public
 annotation
 parameters       java.lang.Object
 return           void
 exceptions
 classLoaderHash  66629f63                                                                                                                                                                                                                                                       

Affect(row-cnt:7) cost in 33 ms.

问题原因

出现这个异常的原因是很明确的，在更新字节码时，增加/删除了field。可能有下面的原因：

• JDK的bug，尽量用最新的JDK版本，特别是JDK8

比如： redefine热更新报错 #969 (comment)

• 其它的java agent修改了字节码，增加了 field。

比如jacoco，这个代码测试覆盖率工具，它会在类里增加一个static field来记录执行信息。

• 其它的java agent修改了字节码，但是在retransform时，生成同样的字节码

比如skywalking： 无法trace 和watch加载skywalking7 的jar中的类方法 #1141

• 在执行arthas redefine/retransform命令时，用户自己想替换的字节码有修改

可能是用户本地的JDK版本和线上的JDK版本不一致导致的

可能是用户自己修改了源码

解决方法

暂时关闭 skywalking

X参考文献

• class redefinition failed: attempted to change the schema (add/remove fields) 分析排查 - github/arthas 【推荐】
• watch命令报错class redefinition failed: attempted to change the schema (add/remove fields) - github/arthas
• k8s中使用arthas：Enhance error! exception: java.lang.UnsupportedOperationException: class redefinition f... - CSDN
• 线上排查神器arthas总结 - CSDN
• 解析Arthas协助排查线上skywalking不可用问题_java - AB教程网