使用awk进行日志信息的分组统计

起因

这是今天我线上出了一个bug，需要查看日志并统计一个我需要的信息出现的频率，可以叫做分组统计。

---

日志文件部分内容

00:09:07.655 [showcase_backend][ShowcaseServiceImpl] WARN  - showcase add recommend from taobao failed |shopId=1482502318|itemId=548320850657||530|Remote service error|isv.item-recommend-service-error:SAVE_REQUEST_TOO_MANY_TIMES|
00:09:07.673 [showcase_backend][topsdk] ERROR - 2017-06-27 00:09:07.673^_^xxxxxxx^_^taobao.item.recommend.add^_^10.28.42.60^_^Linux^_^18^_^http://gwx.api.taobao.com/router/rest^_^app_key=xxxxx&partner_id=top-sdk-java-20170607&method=taobao.item.recommend.add&v=2.0&session=xxxxxxxx&format=json&sign_method=hmac&num_iid=5483473847602&timestamp=2017-06-27+00%3A09%3A07^_^{"error_response":{"code":530,"msg":"Remote service error","sub_code":"isv.item-recommend-service-error:SAVE_REQUEST_TOO_MANY_TIMES","sub_msg":"橱窗推荐商品失败","request_id":"qm4l3wu2cnhu"}}
00:09:07.673 [showcase_backend][ShowcaseServiceImpl] WARN  - showcase add recommend from taobao failed |shopId=14825032138|itemId=548347847602||530|Remote service error|isv.item-recommend-service-error:SAVE_REQUEST_TOO_MANY_TIMES|
00:09:07.695 [showcase_backend][ShowcaseServiceImpl] WARN  - showcase add recommend from taobao failed |shopId=148250232138|itemId=546199835284||530|Remote service error|isv.item-recommend-service-error:SAVE_REQUEST_TOO_MANY_TIMES|橱

---

目的

统计每个shopId出现的错误的次数

---

方案

使用awk的分组统计

---

awk代码

cat xxx.log | grep shopId | awk -F '|'  '{print $2}' | awk -F '=' '{s[$2] += 1} END {for (i in s) {print i, s[i]}}' 

---

部分统计效果如下

···

800599961375 273

800599958609 4

800599956923 674

800599960564 2

800599960811 2930

529692269 1153

800599959910 2174

800599961392 10

800599961165 1

800599959912 26

63456935 558

67376794 490

800599958627 119

800599960127 757

36903919 1263

800599960357 971

800599960604 1

800599961203 211

800599958858 400

800599960609 160

162307612 1

112880504 45

550108033 22

800599956751 1

800599958437 20

800599960164 206

800599959951 134

800599960393 151

60322340 326

72003720 239

800599959939 422

800599959491 1

800599959479 1

72743666 181

800599958012 7

800599957416 9

800599958702 18

63926103 76

800599958897 8241

···

---

分析awk代码逻辑

cat xxx.log | grep shopId | awk -F '|'  '{print $2}' | awk -F '=' '{s[$2] += 1} END {for (i in s) {print i, s[i]}}'

主要看这段

awk -F '|'  '{print $2}' | awk -F '=' '{s[$2] += 1} END {for (i in s) {print i, s[i]}}'

1.awk -F '|' '{print $2}' 分离出 shopId=1234343 这种格式的数据

2.awk -F '=' '{s[$2] += 1} 将shopId=12323443分离， 定义数组s，awk开始处理每行的数据 s[$2] += 1, 将$2的数据当成key存进arrays，value碰到一样的就加1

3.{for (i in s) {print i, s[i]}} 循环打印出数组的key和value