HttpClient抓取网页内容简单介绍

版本HttpClient3.1

1、GET方式

第一步、创建一个客户端，类似于你用浏览器打开一个网页

HttpClient httpClient = new HttpClient();

第二步、创建一个GET方法，用来获取到你需要抓取的网页URL

GetMethod getMethod = new GetMethod("http://www.baidu.com");

第三步、获得网址的响应状态码，200表示请求成功

int statusCode = httpClient.executeMethod(getMethod);

第四步、获取网页的源码

byte[] responseBody = getMethod.getResponseBody();

主要就这四步，当然还有其他很多东西，比如网页编码的问题

 public static String spiderHtml() throws Exception {
         //URL url = new URL("http://top.baidu.com/buzz?b=1");

         HttpClient client = new HttpClient();
         GetMethod method = new GetMethod("http://top.baidu.com/buzz?b=1");        

         int statusCode = client.executeMethod(method);
         if(statusCode != HttpStatus.SC_OK) {
             System.err.println("Method failed: "  + method.getStatusLine());
         }

         byte[] body = method.getResponseBody();
         String html = new String(body,"gbk");

2、Post方式

1 HttpClient httpClient = new HttpClient();  

        PostMethod postMethod = new PostMethod(UrlPath);
        postMethod.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,new DefaultHttpMethodRetryHandler());
        NameValuePair[] postData = new NameValuePair[2];
        postData[0] = new NameValuePair("username", "xkey");
        postData[1] = new NameValuePair("userpass", "********");
        postMethod.setRequestBody(postData);
        try {
            int statusCode = httpClient.executeMethod(postMethod);
            if (statusCode == HttpStatus.SC_OK) {
                byte[] responseBody = postMethod.getResponseBody();
                String html = new String(responseBody);
                System.out.println(html);
            }
        } catch (Exception e) {  
            System.err.println("页面无法访问");
        }finally{
         postMethod.releaseConnection();
     } 

相关链接：http://blog.csdn.net/acceptedxukai/article/details/7030700

http://www.cnblogs.com/modou/articles/1325569.html