curl 重定向问题
今天在curl
一个网站的时候遇到一个奇怪的问题,下面是输出:
lxg@lxg-X240:~$ curl -L http://www.yngs.gov.cn/ -v
* Hostname was NOT found in DNS cache
* Trying 116.52.12.163…
* Connected to www.yngs.gov.cn (116.52.12.163) port 80 (#0)
GET / HTTP/1.1
User-Agent: curl/7.38.0
Host: www.yngs.gov.cn
Accept: /< HTTP/1.1 302 Moved Temporarily
< Date: Wed, 04 Nov 2015 14:08:49 GMT
< Transfer-Encoding: chunked
< Location: http://www.yngs.gov.cn/newWeb/template/index.jsp
< Content-Type: text/html; charset=UTF-8
< Set-Cookie: JSESSIONID=SLyTW6RR3R7zPNkkvzvpj12Q1snzzvNFQjYPDbDhYbvgTXWhSnff!-995202664; path=/; HttpOnly
< X-Powered-By: *********
< Set-Cookie: SANGFOR_AD=20111157; path=/
<
* Ignoring the response-body
* Connection #0 to host www.yngs.gov.cn left intact
* Issue another request to this URL: ‘http://www.yngs.gov.cn/newWeb/template/index.jsp’
* Found bundle for host www.yngs.gov.cn: 0xb89840c0
* Re-using existing connection! (#0) with host www.yngs.gov.cn
* Connected to www.yngs.gov.cn (116.52.12.163) port 80 (#0)
GET /newWeb/template/index.jsp HTTP/1.1
User-Agent: curl/7.38.0
Host: www.yngs.gov.cn
Accept: /
……… //上面的输出一直重复
* Ignoring the response-body
* Connection #0 to host www.yngs.gov.cn left intact
* Maximum (50) redirects followed
curl: (47) Maximum (50) redirects followed
最后的错误显示超过了curl
设定的最大50次跳转。
从上面的输出来看访问http://www.yngs.gov.cn/的时候返回302跳转,跳转的url为http://www.yngs.gov.cn/newWeb/template/index.jsp,但是接着访问 http://www.yngs.gov.cn/newWeb/template/index.jsp的时候还是返回同样的302跳转,跳转后的地址是目标自身,这样肯定就会一直在 http://www.yngs.gov.cn/newWeb/template/index.jsp这个url上跳转,当超过curl
设定的默认最大跳转次数50以后就异常结束了。
既然curl
有问题那么试一下wget
命令看看吧,看这个命令是否也是会遇到同样的错误结果:
lxg@lxg-X240:~$ wget http://www.yngs.gov.cn/ –debug
—request begin—
GET / HTTP/1.1
User-Agent: Wget/1.16.1 (linux-gnu)
Accept: /
Accept-Encoding: identity
Host: www.yngs.gov.cn
Connection: Keep-Alive
—request end——response begin—
HTTP/1.1 302 Moved Temporarily
Date: Wed, 04 Nov 2015 14:18:51 GMT
Transfer-Encoding: chunked
Location: http://www.yngs.gov.cn/newWeb/template/index.jsp
Content-Type: text/html; charset=UTF-8
Set-Cookie: JSESSIONID=7JJTW6TLpKRF0vyNXtRpQrnZffkgDfB0vh6vDzQ9jhGNvRsmZxyv!-1122044597; path=/; HttpOnly
X-Powered-By: *********
Set-Cookie: SANGFOR_AD=20111151; path=/
—response end—302 Moved Temporarily
Stored cookie www.yngs.gov.cn -1 (ANY) / <session>
<insecure> [expiry none] JSESSIONID
7JJTW6TLpKRF0vyNXtRpQrnZffkgDfB0vh6vDzQ9jhGNvRsmZxyv!-1122044597Stored cookie www.yngs.gov.cn -1 (ANY) / <session> <insecure> [expiry none] SANGFOR_AD 20111151
Registered socket 3 for persistent reuse.
URI content encoding = “UTF-8”
位置:http://www.yngs.gov.cn/newWeb/template/index.jsp [跟随至新的 URL]URI content encoding = None
–2015-11-04 22:23:09– http://www.yngs.gov.cn/newWeb/template/index.jsp
再次使用存在的到 www.yngs.gov.cn:80 的连接。
Reusing fd 3.—request begin—
GET /newWeb/template/index.jsp HTTP/1.1
User-Agent: Wget/1.16.1 (linux-gnu)
Accept: /
Accept-Encoding: identity
Host: www.yngs.gov.cn
Connection: Keep-Alive
Cookie: JSESSIONID=7JJTW6TLpKRF0vyNXtRpQrnZffkgDfB0vh6vDzQ9jhGNvRsmZxyv!-1122044597; SANGFOR_AD=20111151
—request end——response begin—
HTTP/1.1 200 OK
Date: Wed, 04 Nov 2015 14:18:51 GMT
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
X-Powered-By: *********
—response end—
200 OK
上面是wget
的执行结果(去掉了一些多余的输出),我们看到wget
能正常的获取到http://www.yngs.gov.cn/的结果,并没有出现curl
遇到的死循环跳转的问题。那么也就是说这个网站本身是没有问题的,只是我们访问的时候可能缺少了一些参数。
接着我对比了一下curl
和wget
的request
、response
信息,我发现两者在第一次请求http://www.yngs.gov.cn/的时候request
、response
都是差不多的,不同的可能就是user-agent
。但是当再次请求302返回回来的redirect url http://www.yngs.gov.cn/newWeb/template/index.jsp的时候两者的request
中的参数就有一些不一样了,wget
的请求中是把第一次响应返回Cookie带上了,但是curl
却是Ignoring the response-body,忽略了第一次响应返回的数据,第二次请求的时候没有带上第一次请求返回的Cookie。
此时基本可以判断是因为curl
访问的时候默认忽略了response
返回的数据,redirect url的时候没有设置Cookie导致的,那么怎么来验证呢?
第一种方法就是禁掉wget
的Cookie看还能否正常获取内容:
lxg@lxg-X240:~$ wget http://www.yngs.gov.cn/ –debug –no-cookies
Setting –cookies (cookies) to 0
—request begin—
GET / HTTP/1.1
User-Agent: Wget/1.16.1 (linux-gnu)
Accept: /
Accept-Encoding: identity
Host: www.yngs.gov.cn
Connection: Keep-Alive
—request end——response begin—
HTTP/1.1 302 Moved Temporarily
Date: Wed, 04 Nov 2015 14:43:41 GMT
Transfer-Encoding: chunked
Location: http://www.yngs.gov.cn/newWeb/template/index.jsp
Content-Type: text/html; charset=UTF-8
Set-Cookie: JSESSIONID=SDLtW6ZdvQwPpqGR5mBf2N1TxChNlySvTN8lhDBTQpyP3KvDdr0R!-170174379; path=/; HttpOnly
X-Powered-By: *********
Set-Cookie: SANGFOR_AD=20111158; path=/—response end—
–2015-11-04 22:46:33– http://www.yngs.gov.cn/newWeb/template/index.jsp
再次使用存在的到 www.yngs.gov.cn:80 的连接。
Reusing fd 3.—request begin—
GET /newWeb/template/index.jsp HTTP/1.1
User-Agent: Wget/1.16.1 (linux-gnu)
Accept: /
Accept-Encoding: identity
Host: www.yngs.gov.cn
Connection: Keep-Alive
—request end——response begin—
HTTP/1.1 302 Moved Temporarily
Date: Wed, 04 Nov 2015 14:42:16 GMT
Transfer-Encoding: chunked
Location: http://www.yngs.gov.cn/newWeb/template/index.jsp
Content-Type: text/html; charset=UTF-8
Set-Cookie: JSESSIONID=YQJTW6ZLpWhm7pfr3LzL6lkQdQ1XbnBMCHQhjn7vZ2yptMJvsJvW!-1122044597; path=/; HttpOnly
X-Powered-By: *********
Set-Cookie: SANGFOR_AD=20111151; path=/
—response end—…………….
URI content encoding = None
已超过 20 次重定向。
我们看到wget
最后也是以尝试20次跳转失败结束。
第二中方法就是开启curl
的Cookie:
lxg@lxg-X240:~$ curl -L -b /tmp/curl.cookies http://www.yngs.gov.cn/
* Hostname was NOT found in DNS cache
* Trying 116.52.12.163…
* Connected to www.yngs.gov.cn (116.52.12.163) port 80 (#0)
GET / HTTP/1.1
User-Agent: curl/7.38.0
Host: www.yngs.gov.cn
Accept: /< HTTP/1.1 302 Moved Temporarily
< Date: Wed, 04 Nov 2015 14:55:53 GMT
< Transfer-Encoding: chunked
< Location: http://www.yngs.gov.cn/newWeb/template/index.jsp
< Content-Type: text/html; charset=UTF-8
* Added cookie
JSESSIONID=”lswQW6cZzRtvyGkkJm0hL8RscHT98bcC3YD4f4V1RCJvLLwb2ZMJ!-1122044597”
for domain www.yngs.gov.cn, path /, expire 0
< Set-Cookie: JSESSIONID=lswQW6cZzRtvyGkkJm0hL8RscHT98bcC3YD4f4V1RCJvLLwb2ZMJ!-1122044597; path=/; HttpOnly
< X-Powered-By: *********
* Added cookie SANGFOR_AD=”20111151” for domain www.yngs.gov.cn, path /, expire 0
< Set-Cookie: SANGFOR_AD=20111151; path=/
<
* Ignoring the response-body
* Connection #0 to host www.yngs.gov.cn left intact
* Issue another request to this URL: ‘http://www.yngs.gov.cn/newWeb/template/index.jsp’
* Found bundle for host www.yngs.gov.cn: 0xb8b74108
* Re-using existing connection! (#0) with host www.yngs.gov.cn
* Connected to www.yngs.gov.cn (116.52.12.163) port 80 (#0)
GET /newWeb/template/index.jsp HTTP/1.1
User-Agent: curl/7.38.0
Host: www.yngs.gov.cn
Accept: /
Cookie: JSESSIONID=lswQW6cZzRtvyGkkJm0hL8RscHT98bcC3YD4f4V1RCJvLLwb2ZMJ!-1122044597; SANGFOR_AD=20111151< HTTP/1.1 200 OK
< Date: Wed, 04 Nov 2015 14:55:53 GMT
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8
< X-Powered-By: *********
<
<!DOCTYPE html>
<html xmlns=”http://www.w3.org/1999/xhtml”>
<head>
………………..Connection #0 to host www.yngs.gov.cn left intact
</html>
curl
成功获取到了结果。
从上面的结果也就验证了之前的猜想。
curl -h
...
-L/--location Follow Location: hints (H)
--location-trusted Follow Location: and send authentication even
to other hostnames (H)
-m/--max-time <seconds> Maximum time allowed for the transfer
--max-redirs <num> Maximum number of redirects allowed (H)
--max-filesize <bytes> Maximum file size to download (H/F)
...
curl -L --max-redirs 100000 URL
curl 重定向问题的更多相关文章
- python --curl重定向到文件范例
import sys import os import subprocess import time start = time.time() old=sys.stdout f=open('test ...
- cURL.1 手册页
摘自http://blog.csdn.net/huangxy10/article/details/45717793 cURL.1 手册页 名称 cURL - transfer a URL 摘要 cUR ...
- nginx的安装及基本配置
在CentOS7(mini)上安装: [root@~ localhost]#lftp 172.16.0.1 lftp 172.16.0.1:/pub/Sources/7.x86_64/nginx> ...
- 使用curl获取Location:重定向后url
在php获取http头部信息上,php有个自带的函数get_headers(),我以前也是用这个的,听说效率在win上不咋地,再加上最近研究百度url无果,写了cURL获取重定向url的php代码来折 ...
- php 使用curl获取Location:重定向后url
在php获取http头部信息上,php有个自带的函数get_headers(),我以前也是用这个的,听说效率在win上不咋地,再加上最近研究百度url无果,写了cURL获取重定向url的php代码来折 ...
- centos shell编程5 LANMP一键安装脚本 lamp sed lnmp 变量和字符串比较不能用-eq cat > /usr/local/apache2/htdocs/index.php <<EOF重定向 shell的变量和函数命名不能有横杠 平台可以用arch命令,获取是i686还是x86_64 curl 下载 第三十九节课
centos shell编程5 LANMP一键安装脚本 lamp sed lnmp 变量和字符串比较不能用-eq cat > /usr/local/apache2/htdocs/ind ...
- curl抓取页面时遇到重定向的解决方法
用php的curl抓取网页遇到了问题,为阐述方便,将代码简化如下: <?php function curlGet($url) { $ch = curl_init(); curl_setopt($ ...
- 前端学HTTP之重定向和负载均衡
前面的话 HTTP并不是独自运行在网上的.很多协议都会在HTTP报文的传输过程中对其数据进行管理.HTTP只关心旅程的端点(发送者和接收者),但在包含有镜像服务器.Web代理和缓存的网络世界中,HTT ...
- PHP curl 函数
转载http://sunking.sinaapp.com/archives/111 最近使用curl的时候,发现了一个比较好用的函数,当然是初级者适用的一个函数,就是curl_getinfo(), 在 ...
随机推荐
- codeforces Round #440 A Search for Pretty Integers【hash/排序】
A. Search for Pretty Integers [题目链接]:http://codeforces.com/contest/872/problem/A time limit per test ...
- HDU 2547 无剑无我(数学)
#include<cstdio> #include<iostream> #include<cmath> int main() { double a,b,c,d,m; ...
- [xsy1140]求值
$\newcommand{ali}[1]{\begin{align*}#1\end{align*}}$题意:给定$n,b,c,d,e,a_{0\cdots n-1}$,令$x_k=bc^{4k}+dc ...
- 【最近公共祖先】【线段树】URAL - 2109 - Tourism on Mars
Few people know, but a long time ago a developed state existed on Mars. It consisted of n cities, nu ...
- 组合式MapReduce计算作业
1)迭代MapReduce计算任务,就是在一个循环内多次执行一个MapReduce. 2)顺序组合式MapReduce作业的执行 MapReduce1—>MapReduce2—>MapRe ...
- RabbitMq_05_Topics
Topics (using the .NET client) Prerequisites This tutorial assumes RabbitMQ isinstalled and running ...
- sqlserver 删除临时表
sqlserver 删除临时表 if object_id('tempdb..#tempTable') is not null Begin drop table #tempTable End
- [Git] 根据commiter过滤该用户的所有提交
git log --pretty=oneline --author="xxxx" -(n) 仅显示最近的 n 条提交 --since,--after 仅显示指定时间之后的提交 -- ...
- 设计工作-Axure
1,百度百科 http://baike.baidu.com/view/3332366.htm?fromtitle=axure&fromid=5056136&type=syn 2,官方网 ...
- RenderMonkey 练习 第一天 【opengl 纹理】
础实例: 我们首先实现一个带纹理模型的显示,大体了解RenderMonkey的操作方式. 1. 打开RenderMonkey, 右击WorkSpace的Effect WorkSpace结点,选择Add ...