Zipkin+Sleuth调用链监控集成和使用

背景与需求

• 跨微服务的API调用发生异常，要求快速定位出问题出在哪里。
• 跨微服务的API调用发生性能瓶颈，要求迅速定位出性能瓶颈。

集成

整体结构

整体机构为C/S模式，客户端(Sleuth)来监控采集调用链信息，汇报给服务端(Zipkin)，通过Zipkin提供的web页面来展示链路调用和异常信息，统计链路图等功能。如下图：

操作步骤

1. 引入依赖

<dependency>  <groupId>org.springframework.cloud</groupId>  <artifactId>spring-cloud-starter-zipkin</artifactId></dependency>

其中，zipkin中已经包含了 spring-cloud-starter-sleuth 的依赖，不需要再次引入。

2. 配置

日志输出级别

logging:  level:    com.alibaba.nacos: error    org.springframework.cloud.sleuth: debug

端口等配置

`spring:` `zipkin:` `base-url: http://localhost:9411/`

如果不配置端口等信息，默认会使用 localhost:9411 来访问服务，更多的Zipkin和Sleuth配置可以查看官方文档。

3. 启动Zipkin服务端

官网 https://zipkin.io/

有两种启动方式，第一种是使用Docker来启动：

docker run -d -p 9411:9411 openzipkin/zipkin

第二种是使用jar包的方式来启动：

curl -sSL https://zipkin.io/quickstart.sh | bash -sjava -jar zipkin.jar

4. 启动所有服务

如果不报错，日志输出中包含sleuth的信息，说明集成成功，然后访问Zipkin的服务来查看链路监控信息吧！

踩坑记录

报错情况一

2020-12-30 14:55:33.914 ERROR [log-manager,,,] 97245 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'queueMessageListener' defined in file [/Users/peiel/WorkSpace/youlu/myspringcloud/log-manager/target/classes/com/youlu/logmanager/queue/QueueMessageListener.class]: Initialization of bean failed; nested exception is java.lang.IllegalStateException: Need to invoke method 'listListener' found on proxy for target class 'QueueMessageListener' but cannot be delegated to target bean. Switch its visibility to package or protected.
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:603)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
  at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:895)
  at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
  at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
  at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
  at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
  at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
  at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
  at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
  at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
  at com.youlu.logmanager.LogManagerApplication.main(LogManagerApplication.java:23)
Caused by: java.lang.IllegalStateException: Need to invoke method 'listListener' found on proxy for target class 'QueueMessageListener' but cannot be delegated to target bean. Switch its visibility to package or protected.
  at org.springframework.aop.support.AopUtils.selectInvocableMethod(AopUtils.java:138)
  at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.createRunnable(ScheduledAnnotationBeanPostProcessor.java:514)
  at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.processScheduled(ScheduledAnnotationBeanPostProcessor.java:381)
  at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.lambda$null$1(ScheduledAnnotationBeanPostProcessor.java:362)
  at java.lang.Iterable.forEach(Iterable.java:75)
  at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.lambda$postProcessAfterInitialization$2(ScheduledAnnotationBeanPostProcessor.java:362)
  at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
  at org.springframework.scheduling.annotation.ScheduledAnnotationBeanPostProcessor.postProcessAfterInitialization(ScheduledAnnotationBeanPostProcessor.java:361)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:431)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)
  ... 16 common frames omitted

解决方式，根据日志查看，是日志服务中， @Scheduled 定时任务中方法报错，导致无法实例化该方法所属的类，仔细查看日志报错，其中说明了产生了访问可见性问题，修改该方法为public描述即可解决。

报错情况二

日志如下：

2021-03-15 15:01:11.023 ERROR [mdm,,,] 97454 --- [           main] o.s.boot.SpringApplication               : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'resourceRolesMapInitRunner': Invocation of init method failed; nested exception is org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 3 second(s)
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:895)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
    at com.youlu.mdm.MdmApplication.main(MdmApplication.java:17)
Caused by: org.springframework.dao.QueryTimeoutException: Redis command timed out; nested exception is io.lettuce.core.RedisCommandTimeoutException: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 3 second(s)
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:70)
    at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41)
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
    at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42)
    at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:273)
    at org.springframework.data.redis.connection.lettuce.LettuceKeyCommands.convertLettuceAccessException(LettuceKeyCommands.java:809)
    at org.springframework.data.redis.connection.lettuce.LettuceKeyCommands.del(LettuceKeyCommands.java:128)
    at org.springframework.data.redis.connection.DefaultedRedisConnection.del(DefaultedRedisConnection.java:82)
    at org.springframework.data.redis.core.RedisTemplate.lambda$delete$2(RedisTemplate.java:713)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:228)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:188)
    at org.springframework.data.redis.core.RedisTemplate.delete(RedisTemplate.java:713)
    at com.youlu.common.service.impl.RedisServiceImpl.del(RedisServiceImpl.java:36)
    at com.youlu.mdm.service.impl.MdmResourceServiceImpl.initResourceRolesMap(MdmResourceServiceImpl.java:105)
    at com.youlu.mdm.component.ResourceRolesMapInitRunner.initResourceRolesMap(ResourceRolesMapInitRunner.java:20)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)
    ... 19 common frames omitted
Caused by: io.lettuce.core.RedisCommandTimeoutException: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 3 second(s)
    at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:132)
    at io.lettuce.core.FutureSyncInvocationHandler.handleInvocation(FutureSyncInvocationHandler.java:69)
    at io.lettuce.core.internal.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:80)
    at com.sun.proxy.$Proxy161.del(Unknown Source)
    at org.springframework.data.redis.connection.lettuce.LettuceKeyCommands.del(LettuceKeyCommands.java:126)
    ... 34 common frames omitted
Caused by: io.lettuce.core.RedisCommandTimeoutException: Command timed out after 3 second(s)
    at io.lettuce.core.ExceptionFactory.createTimeoutException(ExceptionFactory.java:51)
    at io.lettuce.core.protocol.CommandExpiryWriter.lambda$potentiallyExpire$0(CommandExpiryWriter.java:167)
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
    at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

产生原因：仔细查看日志，发现其中 resourceRolesMapInitRunner 类执行报错，查看源代码，发现使用了 @PostConstruct 注解来执行方法，该注解的生命周期的触发是所属类Bean实例化后，此时我们 Sleuth 并没有实例化完成，导致执行该方法报错。解决方式：修改该类，让该方法在SpringBoot启动完成后执行（ApplicationRunner等），即可解决。

使用方式

打开我们线上的Zipkin（http://47.114.174.96:9411/zipkin），可以看到如下页面。

举个例子

比如说，我访问了接口，这个接口报错，返回500，这个时候，我们如何快速查询出这个接口具体的失败原因？第一步：筛选出最近的带有报错的链路。 如图，我们已经看到异常的原因和请求的path。

---

更多测试技术分享、学习资源以及一些其他福利可关注公众号：【Coding测试】获取：