awk词频统计

2018-01-03@中关村

有文本 a.log 如下，请做词频统计，统计出每个单词出现的频率并倒序排序。

The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

方法一

egrep -o "\b[[:alpha:]]+\b" a.log 

awk '{++count[$0]} END{for (word in count){ printf("%-20s%d\n",word,count[word]);}}'

sort -n -r -k2,2

- 首先通过egrep把文本内容拆成每行列出一个单词

　　- egrep -o 表示只打印匹配到的字符，由换行符分割

　　- \b 是正则表达式里的单词边界符

　　- [:alpha:] 是表示字母的字符类

- 其次通过awk统计每个单词出现的次数

root@standby [13:39:48]$ egrep -o "\b[[:alpha:]]+\b" a.log |awk '{++count[$0]} END{for (word in count){ printf("%-20s%d\n",word,count[word]);}}' |sort -n -r -k2,2 |head -20
is                  10
than                8
better              8
to                  5
the                 5
one                 3
of                  3
never               3
it                  3
idea                3
be                  3
Although            3
way                 2
should              2
s                   2
obvious             2
may                 2
implementation      2
If                  2
explain             2
root@standby [13:42:38]$

　　

方法二

awk '{for(i=1;i<=NF;i++) count[$i]++} END{ for(patten in count) printf("%-20s%d\n",patten,count[patten])}

注意：这种情况统计的就不是单词，而是按照字段统计的

root@standby [15:45:06]$ awk '{for(i=1;i<=NF;i++) count[$i]++} END{ for(patten in count) printf("%-20s%d\n",patten,count[patten])}' a.log |sort -n -r -k2,2 |head -20
is                  10
than                8
better              8
to                  5
the                 5
of                  3
be                  3
Although            3
way                 2
should              2
one                 2
never               2
may                 2
implementation      2
If                  2
idea.               2
explain,            2
do                  2
a                   2
Zen                 1
root@standby [15:45:14]$

参考：https://www.cnblogs.com/Peter2014/p/7596128.html

参考：http://bbs.chinaunix.net/thread-4102008-1-1.html