Python 进行查询日志查询条件分析

任务：crm日志的查询条件  每次是哪几个字段查，有几种组合 ，统计每种组合查询的量

日志样例：

 132.xxx.xx.x -  -  [-- ::] "GET /REST/HTableService?appId=crmyun&partition=2017&query=QUERY_TYPE1%3D%E6%8E%A5%E5%85%A5%E5%8F%B7%7Cand%7CQUERY_VALUE1%3D17727955834%7Cand%7CDATETIME%3E20170925000000000%7Cand%7CDATETIME%3C20170928000000000&version=1.0&tablename=TB_CRM_xxxx_xxxxx&method=getData&latnId=755&staffNo=GZTEST200&timestamp=1506585708188&signature=D73E9B59E08EA7B1C2D0DDA72AC957E4 HTTP/1.1"
 132.xxx.xx.x -  -  [-- ::] "GET /REST/HTableService?staffNo=xxTEST200&appId=crmyun&version=1.0&tablename=TB_CRM_xxxx_xxxxx&method=getData&timestamp=1505871359000&signature=6743AE272C10BCC2261E11AF4CA5EA19&charset=UTF-8&partition=2017&query=STAFF_ID=1212100141|and|DATETIME>20170917000000000|and|DATETIME<20170919000000000 HTTP/1.1"     

查询条件：query查询条件可以多个，用|and|分割。

步骤：

1、正则获取query查询条件组合

 query=QUERY_TYPE1%3D%E6%8E%A5%E5%%A5%E5%8F%B7%7Cand%7CQUERY_VALUE1%3D17727955834%7Cand%7CDATETIME%3E20170925000000000%7Cand%7CDATETIME%3C20170928000000000
 query=STAFF_ID=|and|DATETIME>|and|DATETIME<

2、截取query列表，得到查询条件组合；以"%7C|\|"分割，得到列表

 ['QUERY_TYPE1%3D%E6%8E%A5%E5%85%A5%E5%8F%B7', 'and', 'QUERY_VALUE1%3D17727955834', 'and', 'DATETIME%3E20170925000000000', 'and', 'DATETIME%3C20170928000000000']
 ['STAFF_ID=1212100141', 'and', 'DATETIME>20170917000000000', 'and', 'DATETIME<20170919000000000']

3、剔除 'and' 项(列表取[::2])得到新列表

 ['QUERY_TYPE1%3D%E6%8E%A5%E5%85%A5%E5%8F%B7', 'QUERY_VALUE1%3D17727955834', 'DATETIME%3E20170925000000000', 'DATETIME%3C20170928000000000']
 ['STAFF_ID=1212100141', 'DATETIME>20170917000000000', 'DATETIME<20170919000000000']

4、以'%3D|%3E|%3C|>|<|='分割，并将key放入set()中，得到去重后的结果

 ['QUERY_TYPE1', '%E6%8E%A5%E5%85%A5%E5%8F%B7']
 ['QUERY_VALUE1', '']
 ['DATETIME', '']
 ['DATETIME', '']

 ['STAFF_ID', '']
 ['DATETIME', '']
 ['DATETIME', '']

5、将列表key值放入set()中，得到结果；参考代码如下

 import sys
 import time
 import re

 def read_write():
     with open("C:\\Users\\admin\\Desktop\\c5.log", 'r') as f1:
         for line in f1.readlines():
             pattern = re.compile(r'query=.*?\s')
             results = re.search(pattern, line).group().split('&')
             for result in results:
                 if result.startswith("query"):
                     temp = result[6: ]
                     list = re.split("%7C|\|",temp)[::2]
 #                    print list
                     my_set = set()
                     for l in list:
                         arrya = re.split('%3D|%3E|%3C|>|<|=', l)
                         my_set.add(arrya[0])
                     print my_set
                     c = [i for i in my_set]
                     file = open("C:\\Users\\admin\\Desktop\\4.txt", 'a')
                     file.write(repr(c)+'\n')
                     file.close()

 if __name__ == '__main__':
     start = time.time()
     read_write()
     stop = time.time()
     print "running time is "+str(stop - start)