关于pv的那些事！！

遗留问题:whid=1969的日志记录是什么意思？

网站站点信息未分配的时候，会用1969去代替站点信息。

PV:页面浏览量(page view),用户每次打开或刷新一次网页即被计算一次.

关于pv的计算,我们首先考虑下最简单的情况:计算易迅网每天的的pv,这个pv不分站点和渠道.

我们打开跳板机,进入75机器的shell目录.

打开shell目录里的siteKeyData_AllSite.sh,开始研究这个文件.

我们注意到这个脚本文件里,有一个getPV函数,我们来看下这个函数.

pv计算规则:

①页面跳转,并且当前页面id合法;

②网站端的,不计无线;

③IFrame内嵌页卡不计入PV;

④外部投放的页卡不计入PV;

⑤符合以上4点的就算做一条pv,pv数加1.

我们根据以上5点,写一个测试脚本vic_AllSite_pv.sh.算一下20140909号的的一天的易迅网的pv量.

 #!/bin/bash

 ## file name: vic_AllSite_pv.sh

 ## example: ./vic_AllSite_pv.sh  

 #获取当前脚本所在路径
 currdir=$(cd "$(dirname "$")"; pwd)

 #进入脚本所在目录
 cd $currdir

 ## PHP path
 PHP_BIN="/usr/local/php/bin/php"

 ## 显示脚本的正确用法
 function showUsage()
 {
     echo "Usage: sh vic_AllSite_pv.sh  [log_file_time]" >&
     exit ;
 }

 ## 如果参数为0，则显示该脚本的用法
 if [ $# =  ]; then
     showUsage
 fi

 ## 变量初始化
 start_time=`date "+%Y-%m-%d %H:%M:%S.%N"`
 if [ $# =  ]; then
     log_file_time=$
 fi

 ## 处理结果的存储文件名
 result_pv_filename="../data_today/""vic_ALLSITE_PV""_""$log_file_time"

 ## site_key
 ## warehouseid    - 各分站总体数据
 ## warehouseid_1  - 各分站商品详情页数据

 ## get PV
 function getPV()
 {

     log_file_time=$
     log_filename="row_data_"$
     log_filename=$log_filename"*"

     awk -v fdate=$log_file_time -v ftype=$period_type -F"\t" '{
         ## 不是页面跳转或者当前页面id不合法的请求，过滤掉
         if($ !=  || int($) < ) next;

         ## 不计无线
         wh_id = int($);
         ## if ( (wh_id == ) || (wh_id == ) || (wh_id == ) || (wh_id == ) ) next;
         if ( (wh_id !=    ) &&
              (wh_id != ) &&
              (wh_id != ) &&
              (wh_id != ) &&
              (wh_id != ) &&
              (wh_id != ) &&
              (wh_id != ) ) next;

         ## IFrame内嵌页卡不计入PV
         if( (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == )  || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ) next;

         ## 外部投放的页卡不计入PV
         if( == $ && !match($, /buy\.(51buy|yixun)\.com/)) next;

         if((int($) %  == ) || match($, /item\.(51buy|yixun)\.com/)) {
             pvArr["0_1"] = pvArr["0_1"] + ;
         }

         if((int($) == ) || match($, /buy\.(51buy|yixun)\.com\/cart\.html/)) {
             pvArr["0_2"] = pvArr["0_2"] + ;
         }

         if((int($) == ) || (int($) == ) || match($, /buy\.(51buy|yixun)\.com\/order.html/)) {
             pvArr["0_3"] = pvArr["0_3"] + ;
         }

         ## key: 分站ID
         pvArr[""] = pvArr[""] + ;
         } END {
             for(key in pvArr)
             {
                print key"\t"pvArr[key] >  "../data_today/""ALLSITE_PV_"ftype"_"fdate
             }
         }' ../data_today/$log_filename
 }

 ## start execute
 getPV  $log_file_time

 ## store result to DB
 ##${PHP_BIN} -c . -f siteKeyDataStore_AllSite.php $result_pv_filename $result_uv_filename $result_btnuv_filename

 ## process end time
 end_time=`date "+%Y-%m-%d %H:%M:%S.%N"`

 echo -e "ALLSITE_PV_Data:""["",""$log_file_time""]:""$start_time""~""$end_time"

vic_AllSite_pv.sh

计算结果如下:

0 3430852 //全站一天的pv
0_1 1071985 //商品详情页pv
0_2 76135 //购物车pv
0_3 40871 //下单页pv

全站的pv比较好算,那么各分站的pv又该怎么去算呢.

其实有了前面的基础,分站的计算就比较简单了,我们更改一下脚本.

 #!/bin/bash

 ## file name: vic_Sites_pv.sh

 ## example: ./vic_Sites_pv.sh  

 #获取当前脚本所在路径
 currdir=$(cd "$(dirname "$")"; pwd)

 #进入脚本所在目录
 cd $currdir

 ## PHP path
 PHP_BIN="/usr/local/php/bin/php"

 ## 显示脚本的正确用法
 function showUsage()
 {
     echo "Usage: sh vic_Sites_pv.sh  [log_file_time]" >&
     exit ;
 }

 ## 如果参数为0，则显示该脚本的用法
 if [ $# =  ]; then
     showUsage
 fi

 ## 变量初始化
 start_time=`date "+%Y-%m-%d %H:%M:%S.%N"`
 if [ $# =  ]; then
     log_file_time=$
 fi

 ## 处理结果的存储文件名
 result_pv_filename="../data_today/""vic_SITES_PV""_""$log_file_time"

 ## site_key
 ## warehouseid    - 各分站总体数据
 ## warehouseid_1  - 各分站商品详情页数据

 ## get PV
 function getPV()
 {

     log_file_time=$
     log_filename="row_data_"$
     log_filename=$log_filename"*"

     awk -v fdate=$log_file_time -v ftype=$period_type -F"\t" '{
         ## 不是页面跳转或者当前页面id不合法的请求，过滤掉
         if($ !=  || int($) < ) next;

         ## 不计无线
         wh_id = int($);
         ## if ( (wh_id == ) || (wh_id == ) || (wh_id == ) || (wh_id == ) ) next;
         if ( (wh_id !=    ) &&
              (wh_id != ) &&
              (wh_id != ) &&
              (wh_id != ) &&
              (wh_id != ) &&
              (wh_id != ) &&
              (wh_id != ) ) next;

         ## IFrame内嵌页卡不计入PV
         if( (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == )  || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ||
             (int($) == ) || (int($) == ) ) next;

         ## 外部投放的页卡不计入PV
         if( == $ && !match($, /buy\.(51buy|yixun)\.com/)) next;

         if((int($) %  == ) || match($, /item\.(51buy|yixun)\.com/)) {
             pvArr[$"_1"] = pvArr[$"_1"] + ;
         }

         if((int($) == ) || match($, /buy\.(51buy|yixun)\.com\/cart\.html/)) {
             pvArr[$"_2"] = pvArr[$"_2"] + ;
         }

         if((int($) == ) || (int($) == ) || match($, /buy\.(51buy|yixun)\.com\/order.html/)) {
             pvArr[$"_3"] = pvArr[$"_3"] + ;
         }

         ## key: 分站ID
         pvArr[$] = pvArr[$] + ;
         } END {
             for(key in pvArr)
             {
                print key"\t"pvArr[key] >  "../data_today/""SITES_PV_"ftype"_"fdate
             }
         }' ../data_today/$log_filename
 }

 ## start execute
 getPV  $log_file_time

 ## store result to DB
 ##${PHP_BIN} -c . -f siteKeyDataStore_AllSite.php $result_pv_filename $result_uv_filename $result_btnuv_filename

 ## process end time
 end_time=`date "+%Y-%m-%d %H:%M:%S.%N"`

 echo -e "SITES_PV_Data:""["",""$log_file_time""]:""$start_time""~""$end_time"

计算结果如下：

2001_3 5789
1_3 16977
4001_1 76058
2001 550962
1969_1 22
4001_2 4795
3001 314763
1969_2 1
4001_3 2252
4001 239516
5001 134831
3001_1 93959
3001_2 6710
1001_1 291933
3001_3 3231
5001_1 40875
1001_2 19668
5001_2 2934
1001_3 11163
5001_3 1459
1969 19877
1 1283055
2001_1 181135
1_1 388003
2001_2 10970
1_2 31057
1001 887848

剩下的就是渠道pv数据的计算了.我们另起一章,详细解说.