java 分布式实践

spring boot cloud实践

开源的全链路跟踪很多，比如 Spring Cloud Sleuth + Zipkin，国内有美团的 CAT 等等。

其目的就是当一个请求经过多个服务时，可以通过一个固定值获取整条请求链路的行为日志，基于此可以再进行耗时分析等，衍生出一些性能诊断的功能。

不过对于我们而言，首要目的就是 Trouble Shooting，出了问题需要快速定位异常出现在什么服务，整个请求的链路是怎样的。

为了让解决方案轻量，我们在日志中打印 RequestId 以及 TraceId 来标记链路。

RequestId 在 Gateway 生成表示唯一一次请求，TraceId 相当于二级路径，一开始与 RequestId 一样，但进入线程池或者消息队列后，TraceId 会增加标记来标识唯一条路径。

举个例子，当一次请求向 MQ 发送一个消息，那么这个消息可能会被多个消费者消费，此时每个消费线程都会自己生成一个 TraceId 来标记消费链路。加入 TraceId 的目的就是为了避免只用 RequestId 过滤出太多日志。

实现上，通过 ThreadLocal 存放 APIRequestContext 串联单服务内的所有调用。

当跨服务调用时，将 APIRequestContext 信息转化为 HTTP Header，被调用方获取到 HTTP Header 后再次构建 APIRequestContext 放入 ThreadLocal，重复循环保证 RequestId 和 TraceId 不丢失即可。

如果进入 MQ，那么 APIRequestContext 信息转化为 Message Header 即可(基于 RabbitMQ 实现)。

当日志汇总到日志系统后，如果出现问题，只需要捕获发生异常的 RequestId 或是 TraceId 即可进行问题定位。

1. 功能介绍

2. 使用方法

启动udf-rabbitmq-producer：
启动udf-rabbitmq-comsumer：
查看日志： produce的日志

|2019-04-04 11:53:12.260| INFO|XNIO-1 task-10|udf.udf.rabbitmq.producer.controller.SendMessage:34||127.0.0.1|172.27.9.60|clientSysName|ito-rabbitmq-provider|1904041153122562832859336|BIZ|init rabbitmq2org.springframework.amqp.rabbit.core.RabbitTemplate@1b1f5012|

|2019-04-04 11:53:12.262| INFO|XNIO-1 task-10|udf.udf.rabbitmq.springboot.starter.postprocess.MDCMesagePostProcess:41||127.0.0.1|172.27.9.60|clientSysName|ito-rabbitmq-provider|1904041153122562832859336|BIZ|{"traceId":"1904041153122562832859336","clientSysName":"clientSysName","logType":"BIZ","request":"HttpServletRequestImpl [ POST /v1/sendMsg ]","clientNodeName":"127.0.0.1","serverNodeName":"172.27.9.60","transId":"transId","requestUrl":"http://127.0.0.1:10616/v1/sendMsg","serverSysName":"ito-rabbitmq-provider","startTime":"1554349992256"}|

|2019-04-04 11:53:12.264| INFO|XNIO-1 task-10|udf.udf.log.springboot.starter.filter.NormalAspect:52||127.0.0.1|172.27.9.60|clientSysName|ito-rabbitmq-provider|1904041153122562832859336|RESP|interfaceName=http://127.0.0.1:10616/v1/sendMsg^executeTime=8^desc=request:HttpServletRequestImpl [ POST /v1/sendMsg ],response:{"age":0,"name":"string"}|

comsumer的日志

|2019-04-04 11:53:12.295| INFO|dataDeepDealListenerContainer-6|udf.udf.rabbitmq.springboot.starter.postprocess.MDCReceivePostProcessors:52||127.0.0.1|172.27.9.60|ito-rabbitmq-provider|ito-rabbitmq-comsumer|1904041152498513697567629|BIZ|接收到的MDC{"traceId":"1904041153122562832859336","clientSysName":"clientSysName","logType":"BIZ","request":"HttpServletRequestImpl [ POST /v1/sendMsg ]","clientNodeName":"127.0.0.1","serverNodeName":"172.27.9.60","transId":"transId","requestUrl":"http://127.0.0.1:10616/v1/sendMsg","serverSysName":"ito-rabbitmq-provider","startTime":"1554349992256","__TypeId__":"udf.udf.rabbitmq.producer.model.SampleMessage"}|

|2019-04-04 11:53:12.296| INFO|dataDeepDealListenerContainer-6|udf.udf.rabbitmq.comsumer.mqlistener.SampleMessageListener:26||127.0.0.1|172.27.9.60|ito-rabbitmq-provider|ito-rabbitmq-comsumer|1904041153122562832859336|BIZ|成功处理MQ消息, 消息体：{"name":"string","age":0}|

实现要点：对rabbitmqtemplate进行增强,添加前置消息处理器：

@Bean

   RabbitTemplate reforeRabbitTemplate(ConnectionFactory connectionFactory,Jackson2JsonMessageConverter jackson2JsonMessageConverter)

   {

         RabbitTemplate rabbitTemplate =new RabbitTemplate(connectionFactory);

       rabbitTemplate.setBeforePublishPostProcessors(mdcMesagePostProcess);

       rabbitTemplate.setAfterReceivePostProcessors(mdcReceivePostProcessors);

       rabbitTemplate.setMessageConverter(jackson2JsonMessageConverter);

       logger.info("init rabbitmq"+rabbitTemplate);

       return rabbitTemplate;

   }

public class MDCMesagePostProcess implements MessagePostProcessor {

    private Logger logger= LoggerFactory.getLogger(MDCMesagePostProcess.class);

    @Override

    public Message postProcessMessage(Message message) throws AmqpException {

        Map<String, String> mdcContainer =MDC.getCopyOfContextMap();

        logger.info(JSON.toJSONString(mdcContainer));

        for (Map.Entry<String, String> m : mdcContainer.entrySet()) {

            message.getMessageProperties().setHeader(m.getKey(),m.getValue());

        }

        return message;

    }

}

@Service

public class MDCReceivePostProcessors implements MessagePostProcessor {

    @Value("${spring.application.name}")

    private String sysName;

    private Logger logger= LoggerFactory.getLogger(MDCReceivePostProcessors.class);

    @Override

    public Message postProcessMessage(Message message) throws AmqpException {

        Map<String, Object> mdc= message.getMessageProperties().getHeaders();

        Map<String, String> mdcString= new HashMap<String,String>();

        logger.info("接收到的MDC"+ JSON.toJSONString(mdc));

        for (Map.Entry<String, Object> m : mdc.entrySet()) {

            mdcString.put(m.getKey(),m.getValue().toString());

        }

        mdcString.put(LogKeyConstants.CLIENT_SYS_NAME, String.valueOf(mdc.get(LogKeyConstants.SERVER_SYS_NAME)));

        mdcString.put(LogKeyConstants.SERVER_SYS_NAME, sysName);

        MDC.setContextMap(mdcString);

        return message;

    }

}

消费者还是需要额外指定：

@Bean

    public SimpleMessageListenerContainer dataDeepDealListenerContainer(ConnectionFactory connectionFactory, Queue queue) {

        SimpleMessageListenerContainer container =new SimpleMessageListenerContainer();

        container.setConnectionFactory(connectionFactory);

        container.setQueues(queue);

        container.setMessageListener(sampleMessageListener);

        container.setAcknowledgeMode(AcknowledgeMode.MANUAL);

        container.setAfterReceivePostProcessors(mdcReceivePostProcessors);

        return container;

    }