httpClient download file(爬虫)
package com.opensource.httpclient.bfs;
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.http.Header;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
public class DownLoadFile
{
public String getFileNameByUrl(String url, String contentType)
{
url = url.substring(7);
if (contentType.indexOf("html") != -1)
{
url = url.replaceAll("[\\?/:*|<>\"]", "_") + ".html";
return url;
}
else
{
return url.replaceAll("[\\?/:*|<>\"]", "_") + "." + contentType.substring(contentType.lastIndexOf("/") + 1);
}
}
public void saveToLocal(byte[] data, String filePath)
{
try
{
DataOutputStream out = new DataOutputStream(new FileOutputStream(new File(filePath)));
for (int i = 0; i < data.length; i++)
out.write(data[i]);
out.flush();
out.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
public String downloadFile(String url)
throws ClientProtocolException, IOException
{
String filePath = null;
HttpClient httpClient = new DefaultHttpClient();
HttpGet get = new HttpGet(url);
HttpResponse rsp = httpClient.execute(get);
if (rsp.getStatusLine().getStatusCode() != HttpStatus.SC_OK)
{
System.err.println("Method failed: " + rsp.getStatusLine());
filePath = null;
}
Header[] header = rsp.getHeaders("Content-Type");
filePath = "D:\\" + getFileNameByUrl(url, header[0].getValue());
saveToLocal(rsp.toString().getBytes(), filePath);
return filePath;
}
public static void main(String[] args)
throws ClientProtocolException, IOException
{
DownLoadFile downLoadFile = new DownLoadFile();
String temp = downLoadFile.downloadFile("http://www.huawei.com/cn/");
System.out.println(temp);
}
}
httpClient download file(爬虫)的更多相关文章
- HttpClient的使用-爬虫学习1
HttpClient的使用-爬虫学习(一) Apache真是伟大,为我们提供了HttpClient.jar,这个HttpClient是客户端的http通信实现库,这个类库的作用是接受和发送http报文 ...
- Csharp:WebClient and WebRequest use http download file
//Csharp:WebClient and WebRequest use http download file //20140318 塗聚文收錄 string filePath = "20 ...
- 基于HttpClient实现网络爬虫~以百度新闻为例
转载请注明出处:http://blog.csdn.net/xiaojimanman/article/details/40891791 基于HttpClient4.5实现网络爬虫请訪问这里:http:/ ...
- [Powershell] FTP Download File
# Config $today = Get-Date -UFormat "%Y%m%d" $LogFilePath = "d:\ftpLog_$today.txt&quo ...
- Angular HttpClient upload file with FormData
从sof上找到一个example:https://stackoverflow.com/questions/46206643/asp-net-core-2-0-and-angular-4-3-file- ...
- FTP Download File By Some Order List
@Echo Off REM -- Define File Filter, i.e. files with extension .RBSet FindStrArgs=/E /C:".asp&q ...
- httpclient upload file
用httpclient upload上传文件时,代码如下: HttpPost httpPost = new HttpPost(uploadImg); httpPost.addHeader(" ...
- Download file using libcurl in C/C++
http://stackoverflow.com/questions/1636333/download-file-using-libcurl-in-c-c #include <stdio.h&g ...
- HttpClient的使用-爬虫学习(一)
Apache真是伟大,为我们提供了HttpClient.jar,这个HttpClient是客户端的http通信实现库,这个类库的作用是接受和发送http报文,引进这个类库,我们对于http的操作会变得 ...
随机推荐
- Android反编译APK
http://blog.csdn.net/vipzjyno1/article/details/21039349/ 关于被加壳工具添加我的apk如何脱壳 http://www.blogfshare.co ...
- KVO的实现原理探寻
@import url(http://i.cnblogs.com/Load.ashx?type=style&file=SyntaxHighlighter.css);@import url(/c ...
- python操作redis-set
#!/usr/bin/python #!coding: utf-8 import redis if __name__=="__main__": try: conn=redis.St ...
- QT:浮动的饼状统计图(自绘不规则窗口)
昨天在玩手机游戏时,见到一个游戏的饼状投票统计图很漂亮,所以我今天也花了一个下午来实现了一个类似的程序娱乐一下,呵呵. 思路:1:将窗体设为Qt::FramelessWindowHint和Qt::WA ...
- 设置edittext的hint位置
<EditText android:id="@+id/edt_content" android:layout_width="fill_parent" an ...
- 甲骨文推动Java进军“物联网”
该公司希望在嵌入式设备开发项目上Java可以取代C 随着周二宣布对嵌入式的Java版本进行升级,甲骨文希望扩展该平台到新一代连接设备,又名物联网.甲骨文还希望,Java可以在一些嵌入式开发项目 ...
- convert用法(数据库中原本储存的格式是Nvarchar,如何修改成datetime格式)
查询这张表得到的数据如图 select CONVERT(nvarchar,substring([purchase-date],1,4)) +'-'+CONVERT(nvarchar,substring ...
- 【ThinkPHP学习】ThinkPHP自己主动转义存储富文本编辑器内容导致读取出错
RT. ThinkPHP的conf文件里的Convention.php有一个配置选项 'DEFAULT_FILTER' => 'htmlspecialchars', // 默认參数过滤方法 用于 ...
- 关于OF和CF
很久很久前写的.越来越意识到作为一名科班出身的学生的重要性. 自己在使用IDA时,发现F5产生类似的这种代码. 其中有一句,v5 <= -141920797,我在想为什么是负数.如果把-1419 ...
- 【巧妙】【3-21个人赛】Problem C 01串
Problem C Time Limit : 3000/1000ms (Java/Other) Memory Limit : 65535/32768K (Java/Other) Total Sub ...