三剑客之awk数组实战

shell数组for循环

第一种：

#!/bin/sh

array=(1 2 3)

for i in ${array[*]}

 do

   echo $i

done

 

第二种：

#!/bin/sh

array=(1 2 3)

for ((i=0;i<${#array[*]};i++))

 do

   echo ${array[i]}

done

 

shell数组小结：

一.定义

静态数组：array=(1 2 3)

动态数组：array=($(ls))

二.打印

${array[*]}或${array[@]}  打印所有元素

${#array[*]}或${#array[@]}  打印数组长度

${array[i]}  打印单个元素  i是下标

 

sum++ 的意思是计算总的访问条目数：

[root@fuzhou89 2017-01-09]#  zcat 2017-01-09-12*|awk '{print $7}'|awk -F "/" '{print $3}' |awk '{a[$1]++;sum++}END{for(i in a)print i,100*a[i]/sum "%",a[i]" counts"}'|sort -rn -k3|head

mh.tiancity.com 51.5872% 165788 counts

w.tancdn.com 18.0503% 58009 counts

i.l.inmobicdn.net 13.6327% 43812 counts

book.img.ireader.com 6.99683% 22486 counts

kyfw.12306.cn 5.19986% 16711 counts

comic.img.ireader.com 0.923846% 2969 counts

i.ugc.corp3g.cn 0.536135% 1723 counts

img6.uuserv30.net 0.37962% 1220 counts

book.d.ireader.com 0.285337% 917 counts

bbs.res.flyme.cn 0.238663% 767 counts

 

统计url排名前十的url：

[root@chundianxin103 2017-05-17]# zcat 2017-05-17-11*|awk '{a[$7]++}END{for(i in a)print i" "a[i]}'|sort -rnk2|head

http://cdn.live.xunlei.com/cdn/data/1m/xl9.js 147257

http://pic5.qiyipic.com/common/20170310/c2e7c1252aa640a4a3db004f1e470f1b.png 15175

http://screenshot.dwstatic.com/fjp/lic_-8813209714620956024?imageview/0/w/180/h/136 5902

http://pic5.qiyipic.com/common/20170513/cc26f8a3972e403dbce28afedb337279.jpg 5636

http://d.ifengimg.com/w750_h350/p2.ifengimg.com/cmpp/2017/05/17/20c46d79446469eedf42614c09c33ed5_size499_w750_h350.jpeg 5541

http://s.autoimg.cn/www/common/images/icon-new.png? 5529

http://screenshot.dwstatic.com/fjp/lic_9064017321729548390?imageview/0/w/370/h/282 4298

http://pic5.qiyipic.com/common/lego/20170515/8565c6c54e1341f69fc0cfece57e6494.jpg 4280

http://pic5.qiyipic.com/image/20160914/da/47/v_110943644_m_601_m1_120_160.jpg 4279

http://pic5.qiyipic.com/image/20150312/65/80/da/a_50133980_m_601_m2_120_160.jpg 4161

统计访问ip排名前十的IP:

[root@chundianxin103 2017-05-17]# zcat 2017-05-17-11*|awk '{a[$1]++}END{for(i in a)print a[i],i}'|sort -rn|head

16212 123.172.168.110

7154 123.172.205.156

4326 222.169.185.251

4202 222.168.86.66

3939 36.48.166.159

3649 222.169.251.202

3305 222.168.86.67

3290 222.168.19.248

3056 222.168.7.38

3041 123.172.184.191

统计状态码：

[root@chundianxin103 2017-05-17]# zcat 2017-05-17-11*|awk '{a[$9]++}END{for(i in a)print a[i],i}'|sort -rn|head             

1849400 200

174733 304

6933 301

5965 404

4997 0

4162 302

3858 206

963 503

807

768 %Hs

统计404的url：

[root@chundianxin103 2017-05-17]# zcat 2017-05-17-11*|awk '$9==404{print $7}'|sort|uniq -c|sort -rn|head

    597 http://res.qxz.37wan.com/t1/assets/model/wing/10031/mid/ridemove-2.cvt?

    446 http://res.aojian.game2.com.cn/eflashdir121227/res/data/avatar/effect/eid10457.tse

    267 http://res.qxz.37wan.com/t1/assets/model/wing/10031/mid/rideattack4-3.cvt?

    249 http://res.qxz.37wan.com/t1/assets/model/wing/10031/mid/rideattack2-4.cvt?

    192 http://res.qxz.37wan.com/t1/assets/model/wing/10031/mid/rideattack2-2.cvt?

    174 http://dhsz.cdn.gop.yy.com/cdn/0/avatar/weapon/9210002/mounts/.7t?r=0

    164 http://static.xiyou.cntv.cn/2013/themes/common/css/style2013.css?

    162 http://res.longc.37wan.com/dwcq_00192/assets/images/dropRedPacket/limit.png

    155 http://img3.hdletv.com/https://img2-lemall.letvimg.com//wap/20170424/default/16080459777120959

    139 http://www.huya.com/kpl/undefined

统计11:00-11:04时间段5分钟内，状态码为404的url:

[root@chundianxin103 2017-05-17]# zcat 2017-05-17-11*|awk '$4~/2017:11:[0-4]/ && $9==404{print $7}'|sort|uniq -c|sort -rn|head

    597 http://res.qxz.37wan.com/t1/assets/model/wing/10031/mid/ridemove-2.cvt?

    398 http://res.aojian.game2.com.cn/eflashdir121227/res/data/avatar/effect/eid10457.tse

    267 http://res.qxz.37wan.com/t1/assets/model/wing/10031/mid/rideattack4-3.cvt?

    249 http://res.qxz.37wan.com/t1/assets/model/wing/10031/mid/rideattack2-4.cvt?

    192 http://res.qxz.37wan.com/t1/assets/model/wing/10031/mid/rideattack2-2.cvt?

    154 http://res.longc.37wan.com/dwcq_00192/assets/images/dropRedPacket/limit.png

    148 http://dhsz.cdn.gop.yy.com/cdn/0/avatar/weapon/9210002/mounts/.7t?r=0

    136 http://static.xiyou.cntv.cn/2013/themes/common/css/style2013.css?

    131 http://img3.hdletv.com/https://img2-lemall.letvimg.com//wap/20170424/default/16080459777120959

    124 http://www.huya.com/kpl/undefined

统计该时间段内的域名的总带宽：

[root@chundianxin103 2017-05-17]# zcat 2017-05-17-11*|awk '{print $7,$10}'|awk -F '[/ ]+' '/^http/{print $2,$NF}'|awk '{sum+=$2}END{print sum}'

72400490649

统计该时间段内的各个域名的带宽量：

[root@chundianxin103 2017-05-17]# zcat 2017-05-17-11*|awk '{print $7,$10}'|awk -F '[/ ]+' '/^http/{print $2,$NF}'|awk '{a[$1" "$2]++}END{for(i in a)print i}'|sort -rnk2|head

gcdl2.nearme.com.cn 531400529                

storedl3.nearme.com.cn 427488722

storedl3.nearme.com.cn 339729119

gcdl0.nearme.com.cn 299357279

upfiles.3304399.net 293824785

gcdl2.nearme.com.cn 287142326

apk.wsdl.vivo.com.cn 272977794

gcdl2.nearme.com.cn 241730669

gcdl2.nearme.com.cn 241730668

storedl5.nearme.com.cn 222720000

注释：这个单位是bps,比特位每秒，换算成Mbps ，/1024/1024

Mbps是Million bits per second的缩写，Mbps=Mbit/s即兆比特每秒，它的意思是时间内传输数据的平均比特数，其单位是比特每秒(bps),或千比特每秒(Kbps),或兆比特每秒(Mbps)。

换算单位关系为：1Kbps=1024bps,1Mbps=1024*1024bps（有时为了计算简单记作1Kbps=1000bps,1Mbps=1000000bps）。数据传输速率反映了终端设备之间的信息处理能力，它是一段时间的平均值，数值越大，说明速度越快。

 

 

############经典案例##################

我的日志格式为：
09:55:54: ERROR1 /tmp/error/log.3 50 times
09:56:09: ERROR1 /tmp/error/log.14 50 times
10:56:12: ERROR1 /tmp/error/log.14 100 times
10:56:23: ERROR2 /tmp/error/log.5 50 times
11:56:26: ERROR2 /tmp/error/log.1 50 times
11:56:27: ERROR2 /tmp/error/log.5 100 times
15:56:29: ERROR3 /tmp/error/log.1 100 times
15:56:32: ERROR3 /tmp/error/log.1 150 times
16:56:33: ERROR4 /tmp/error/log.6 50 times
16:56:36: ERROR4 /tmp/error/log.6 100 times
16:56:40: ERROR4 /tmp/error/log.12 50 times

我想统计某时段某种类型的告警次数，时段取第一列第一个：之前的字符，告警类型取第二列，值不定，后面的字符可以不管。我想输出结果为：
09时ERROR1告警为2条。
10时ERROR1告警为1条。
10时ERROR2告警为1条。
11时ERROR2告警为2条。
15时ERROR3告警为2条。
16时ERROR4告警为3条。

 

Answer

 

awk -F'[: ]+' '{a[$1"时"$4]++}END{for(i in a)print i"告警为"a[i]"条."}' ufile