js分析 汽_车_之_家 js生成css伪元素 hs_kw44_configUS::before

0.参考

https://developer.mozilla.org/zh-CN/docs/Web/CSS/Pseudo-elements

https://developer.mozilla.org/zh-CN/docs/Web/CSS/::before

https://developer.mozilla.org/en-US/docs/Web/API/Window/getComputedStyle

0.1 Use with pseudo-elements

getComputedStyle can pull style info from pseudo-elements (for example, ::after, ::before, ::marker, ::line-marker—see spec here).

<style>
 h3::after {
   content: ' rocks!';
 }
</style>

<h3>generated content</h3> 

<script>
  var h3 = document.querySelector('h3');
  var result = getComputedStyle(h3, ':after').content;

  console.log('the generated content is: ', result); // returns ' rocks!'
</script>

0.2 手动设置伪元素信息

  

0.3 理论上应该是可以直接通过 js 读取伪元素 content 的。。。然而这里读取不到 ::before 的 content

1.初步抓取

1.1不加载js VS 加载js

1.2 requests仅见一张table

In [1]: import requests
   ...: from scrapy import Selector
   ...:
   ...:
   ...: url = 'https://car.autohome.com.cn/config/series/3170.html'
   ...: s = requests.Session()
   ...: s.verify = False
   ...: s.headers = {
   ...:     'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36'
   ...: }
   ...:
   ...: r = s.get(url)
   ...: sel = Selector(text=r.text)
   ...:
   ...: sel.css('table')
   ...:

Out[1]: [<Selector xpath='descendant-or-self::table' data='<table class="t1 txtCen" width="100%">\n '>]

1.3 selenium可见所有table

In [3]: from scrapy import Selector
   ...: from selenium import webdriver
   ...:
   ...:
   ...: url = 'https://car.autohome.com.cn/config/series/3170.html'
   ...: dr = webdriver.Chrome()
   ...: dr.get(url)
   ...: sel = Selector(text=dr.page_source)
   ...: sel.css('table')
   ...:

DevTools listening on ws://127.0.0.1:12968/devtools/browser/2e9adb31-7510-421f-ac13-835350af144e
Out[3]:
[<Selector xpath='descendant-or-self::table' data='<table class="t1 txtCen" width="100%">\n '>,
 <Selector xpath='descendant-or-self::table' data='<table cellspacing="0" cellpadding="0" c'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_side" cellspacing="0" cel'>,
 <Selector xpath='descendant-or-self::table' data='<table cellspacing="0" cellpadding="0" c'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_0" cellspacing="0" cellpa'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_1" cellspacing="0" cellpa'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_2" cellspacing="0" cellpa'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_3" cellspacing="0" cellpa'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_4" cellspacing="0" cellpa'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_5" cellspacing="0" cellpa'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_100" cellspacing="0" cell'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_101" cellspacing="0" cell'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_102" cellspacing="0" cell'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_103" cellspacing="0" cell'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_104" cellspacing="0" cell'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_105" cellspacing="0" cell'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_106" cellspacing="0" cell'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_107" cellspacing="0" cell'>,
 <Selector xpath='descendant-or-self::table' data='<table id="tab_108" cellspacing="0" cell'>]

1.4 提取table信息

In [4]: for t in sel.css('table#tab_0'):
   ...:     print('@'*10)
   ...:     for row in t.xpath('.//tr'):
   ...:         print('-'*100)
   ...:         for col in row.xpath('.//th|.//td'):
   ...:             print(''.join(col.xpath('.//node()/text()').extract()), end='\t')
   ...:         print()
   ...:
@@@@@@@@@@
----------------------------------------------------------------------------------------------------
基本参数
----------------------------------------------------------------------------------------------------
厂      -       -       -       -       -       -       -       -       -       -       -       -
----------------------------------------------------------------------------------------------------
级别
----------------------------------------------------------------------------------------------------
能源类型        汽油    汽油    汽油    汽油    汽油    汽油    汽油    汽油    汽油    汽油    汽油    汽油
----------------------------------------------------------------------------------------------------
上市    2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10 2017.10
----------------------------------------------------------------------------------------------------
(kW)    110     110     110     110     140     140     110     110     110     110     140     140
----------------------------------------------------------------------------------------------------
(N·m)  250     250     250     250     320     320     250     250     250     250     320     320
----------------------------------------------------------------------------------------------------
发动机  1.4T 150马力 L4 1.4T 150马力 L4 1.4T 150马力 L4 1.4T 150马力 L4 2.0T 190马力 L4 2.0T 190马力 L4 1.4T 150马力 L4 1.4T 150马力 L4 1.4T 150马力 L4 1.4T 150马力 L4 2.0T 190马力 L4 2.0T 190马力 L4
----------------------------------------------------------------------------------------------------
变速箱  7挡双离合       7挡双离合       7挡双离合       7挡双离合       7挡双离合       7挡双离合       7挡双离合       7挡双离合       7挡双离合       7挡双离合       7挡双离合       7挡双离合
----------------------------------------------------------------------------------------------------
长*宽*高(mm)    4312*1785*1426  4321*1785*1426  4312*1785*1426  4321*1785*1426  4312*1785*1426  4321*1785*1426  4457*1796*1417  4462*1796*1417  4457*1796*1417  4462*1796*1417  4457*1796*1417  4462*1796*1417
----------------------------------------------------------------------------------------------------
车身结构        5门5座两厢车    5门5座两厢车    5门5座两厢车    5门5座两厢车    5门5座两厢车    5门5座两厢车    4门5座三厢车    4门5座三厢车    4门5座三厢车    4门5座三厢车    4门5座三厢车    4门5座三厢车
----------------------------------------------------------------------------------------------------
最高车速(km/h)  215     215     215     215     230     230     215     215     215     215     230     230
----------------------------------------------------------------------------------------------------
官方0-100km/h加速(s)    8.4     8.4     8.4     8.4     7.4     7.4     8.4     8.4     8.4     8.4     7.4     7.4
-----------------------------------------------------------------------------------------------------100km/h加速(s)        -       -       -       -       -       -       -       -       -       -       -       -
----------------------------------------------------------------------------------------------------
100-0km/h制动(m)        -       -       -       -       -       -       -       -       -       -       -       -
----------------------------------------------------------------------------------------------------
工信部(L/100km) 5.7     5.7     5.7     5.7     6.2     6.2     5.6     5.6     5.6     5.6     6.1     6.1
----------------------------------------------------------------------------------------------------
(L/100km)       -       -       -       -       -       -       -       -       -       -       -       -
----------------------------------------------------------------------------------------------------
整车    三10公里        三10公里        三10公里        三10公里        三10公里        三10公里        三10公里        三10公里        三10公里        三10公里        三10公里        三10公里

1.5 对比可见提取table信息不全

2.问题分析

2.1 开发者工具显示 '商' 为CSS伪元素的content属性

2.2 使用正常浏览器硬性刷新未见CSS文件存在有效对应信息

2.3 检查发现class name存在规律：hs_kw数字_configyk

.hs_kw22_configyk::before {
content: "商";
}

.hs_kw44_configyk::before {
content: "一汽";
}

2.4 selector提取存在多个class name后缀，而且无法提取到其中文字

In [19]: sel = Selector(text=dr.page_source)

In [20]: sel.css('span[class*="hs_kw"]').extract()
Out[20]:
['<span class="hs_kw34_configyk"></span>',
 '<span class="hs_kw66_configyk"></span>',
...
 '<span class="hs_kw42_optionLA"></span>',
]

2.5 使用正常浏览器全局搜索 hs_kw 和 hs_kw22_configyk 和 configyk ，仅发现关联文件 3170.html

2.6 Sources面板对 3170.html 进行美化排版，复制到 notepad++ 搜索关键字定位到 js 代码段

继续搜索

2.7 复制其中一个代码片段到 Sources 面板作为 snippet，移除头尾 <script> </script>，添加断点

2.8 运行代码片段，发现可疑变量

2.9 根据关键字定位到相关函数（运气好的话），添加log语句，重新运行输出中间变量

3 完整代码

http://www.cnblogs.com/my8100/p/9028359.html

4 运行结果

5.彩蛋（前端都这么直白？！）