阿里云k8s应用最新日志采集不到的问题

问题描述： 阿里云k8s应用日志之前一直都是可以正常的采集， 先出现一问题， 通过kibana 和阿里云的日志服务都没法展示最新的k8s应用的日志， 部分应用的最新日志有被采集到，但大部分应用日志没有做采集到。

通过 命令 curl '17*****0:9200/_cat/indices?v'   查看 

部分应用没有建立今天的日志索引

　　

删除了elasticsearch组件， 再重新安装以后， 问题依然存在

　　

通过
 kubectl delete DaemonSet log-pilot -n kube-system
   kubectl delete DaemonSet logtail-ds  -n kube-system 

删除了日志采集工具 log-pilot, logtail 后再尝试重新安装

　　

重新安装 log-pilot组件 

kubectl apply -f https://acs-logging.oss-cn-hangzhou.aliyuncs.com/log-pilot.yml
daemonset.extensions/log-pilot created

　　

kubectl get DaemonSet -n kube-system
NAME                       DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
cloud-controller-manager   3         3         3         3            3           node-role.kubernetes.io/master=   76d
flexvolume                 6         6         6         6            6           <none>                            76d
kube-flannel-ds            6         6         6         6            6           beta.kubernetes.io/arch=amd64     76d
kube-proxy-master          3         3         3         3            3           node-role.kubernetes.io/master=   76d
kube-proxy-worker          3         3         3         3            3           <none>                            76d
log-pilot                  6         6         6         6            6           <none>                            3s

　　

再次通过命令查看日志索引 ， 已经可以看到最新的应用日志索引

 curl '172.21.7.210:9200/_cat/indices?v'
health status index                                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   audit-c48ec13c7dbc94c6fba92748baaf296fc-2019.03.13 G7yQFpNVQfa-xh18CmVCvQ   5   1          0            0      1.1kb           486b
yellow open   d******-ui-access-2019.03.13                 oZ9qhpYzQQeRGVtAFWHEaQ   5   1          0            0       486b           486b
green  open   im*****ess-2019.03.13                Lq6meYJcQZGK0MdXgnN7bw   5   1          6            0      167kb         83.5kb
green  open   of******ess-2019.03.13                 PaeAsTGMTEKflSMwaq-V0A   5   1          4            0     83.2kb         41.6kb
yellow open   da*****rror-2019.03.13                   hoO90XVTTiOr8OExHZOoKQ   5   1          0            0      1.1kb           648b
yellow open   d******o-2019.03.13                       eWFqQEBFT92x1sQy4Vp2rg   5   1          0            0       486b           486b

　　

重新安装 logtail  组件 

helm del --purge alibaba-log-controller
release "alibaba-log-controller" deleted

[root@k8****03 ~]# wget http://logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/kubernetes/alicloud-log-k8s-install.sh -O alicloud-log-k8s-install.sh; chmod 744 ./alicloud-log-k8s-install.sh; sh ./alicloud-log-k8s-install.sh c48ec13c7d****296fc
--2019-03-13 14:30:38--  http://logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/kubernetes/alicloud-log-k8s-install.sh
Resolving logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com (logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com)... 47.110.177.92
Connecting to logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com (logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com)|47.110.177.92|:80... connected.
HTTP request sent, awaiting response... 200 OK

kubectl get DaemonSet -n kube-system
NAME                       DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE

log-pilot                  6         6         6         6            6           <none>                            6m
logtail-ds                 6         6         6         6            6           <none>                            10s