CRAWING SPIDER

Goal -> Recursively list all links starting from a base URL.

1. Read page HTML.

2. Extract all links.

3. Repeat for each new link that is not already on the list.

#!/usr/bin/env python

import re

import requests

from urllib.parse import urljoin

target_url = "http://10.0.0.45/mutillidae/"

target_links = []

def extract_links_from(url):

    response = requests.get(url)

    return re.findall('(?:href=")(.*?")', response.content.decode())

def crawl(url):

    href_links = extract_links_from(url)

    for link in href_links:

        link = urljoin(url, link)

        if "#" in link:

            link = link.split("#")[0]

        if target_url in link and link not in target_links:

            target_links.append(link)

            print(link)

            crawl(link)

crawl(target_url)

The Python program runs perfectly.

http://10.0.0.45/mutillidae/favicon.ico"

http://10.0.0.45/mutillidae/styles/global-styles.css"

http://10.0.0.45/mutillidae/styles/ddsmoothmenu/ddsmoothmenu.css"

http://10.0.0.45/mutillidae/styles/ddsmoothmenu/ddsmoothmenu-v.css"

http://10.0.0.45/mutillidae/index.php?page=home.php"

http://10.0.0.45/mutillidae/index.php?page=login.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=login.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=login.php"

http://10.0.0.45/mutillidae/set-up-database.php"

http://10.0.0.45/mutillidae/index.php?page=show-log.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=show-log.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=show-log.php"

http://10.0.0.45/mutillidae/index.php?page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?page=credits.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=credits.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=credits.php"

http://10.0.0.45/mutillidae/"

http://10.0.0.45/mutillidae/index.php?page=user-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=user-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=user-info.php"

http://10.0.0.45/mutillidae/index.php?page=register.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=register.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=register.php"

http://10.0.0.45/mutillidae/index.php?page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php"

http://10.0.0.45/mutillidae/index.php?page=password-generator.php&username=anonymous"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=password-generator.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=password-generator.php"

http://10.0.0.45/mutillidae/index.php?page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?page=framing.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=framing.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=framing.php"

http://10.0.0.45/mutillidae/framer.html"

http://10.0.0.45/mutillidae/index.php?page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?page=installation.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=installation.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=installation.php"

http://10.0.0.45/mutillidae/documentation/mutillidae-installation-on-xampp-win7.pdf"

http://10.0.0.45/mutillidae/index.php?page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=home.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=home.php"

http://10.0.0.45/mutillidae/

http://10.0.0.45/mutillidae/?page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/?page=view-someones-blog.php"

http://10.0.0.45/mutillidae/?page=show-log.php"

http://10.0.0.45/mutillidae/?page=text-file-viewer.php"

http://10.0.0.45/mutillidae/?page=user-info.php"

http://10.0.0.45/mutillidae/?page=login.php"

http://10.0.0.45/mutillidae/?page=credits.php"

http://10.0.0.45/mutillidae/?page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?page=notes.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=notes.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=notes.php"

Python Ethical Hacking - WEB PENETRATION TESTING(4)的更多相关文章

  1. Python Ethical Hacking - WEB PENETRATION TESTING(1)

    WHAT IS A WEBSITE Computer with OS and some servers. Apache, MySQL ...etc. Cotains web application. ...

  2. Python Ethical Hacking - WEB PENETRATION TESTING(2)

     CRAWING DIRECTORIES Directories/folders inside the web root. Can contain files or other directories ...

  3. Python Ethical Hacking - WEB PENETRATION TESTING(5)

    Guessing Login Information on Login Pages Our target website: http://10.0.0.45/dvwa/login.php #!/usr ...

  4. Python Ethical Hacking - WEB PENETRATION TESTING(3)

    CRAWLING SUMMARY Our crawler so far can guess: Subdomains. Directories. Files. Advantages: ->Disc ...

  5. Ethical Hacking - Web Penetration Testing(13)

    OWASP ZAP(ZED ATTACK PROXY) Automatically find vulnerabilities in web applications. Free and easy to ...

  6. Ethical Hacking - Web Penetration Testing(8)

    SQL INJECTION WHAT IS SQL? Most websites use a database to store data. Most data stored in it(userna ...

  7. Ethical Hacking - Web Penetration Testing(10)

    SQL INJECTION SQLMAP Tool designed to exploit SQL injections. Works with many DB types, MySQL, MSSQL ...

  8. Ethical Hacking - Web Penetration Testing(6)

    REMOTE FILE INCLUSION Similar to local file inclusion. But allows an attacker to read ANY file from ...

  9. Ethical Hacking - Web Penetration Testing(4)

    CODE EXECUTION VULNS Allows an attacker to execute OS commands. Windows or Linux commands. Can be us ...

随机推荐

  1. Bestcoder Round8

    4989Summary 既然用C++了就偷懒直接用STL大法了 #include<iostream> #include<algorithm> #include<vecto ...

  2. c++运算符重及其调用

    本文参考自:https://blog.csdn.net/lisemi/article/details/93618161 运算符重载就是赋予运算符新功能,其本质是一个函数. 运算符重载时要遵循以下规则: ...

  3. skywalking中文文档

    https://github.com/apache/skywalking/blob/v5.0.0-alpha/docs/README_ZH.md 大家可以前往如下地址下载我们的发布包: l  Apac ...

  4. java scoket Blocking 阻塞IO socket通信三

    在NIO同步非阻塞的场景中和原来同步阻塞最大的却别就是引入了上面的Buffer对象,现在我们来学校上面的BUffer对象 我们来看看程序的代码: package bhz.nio.test; impor ...

  5. java-IO流(commons-io-2.6)使用教程

    工具库下载: https://pan.baidu.com/s/1tXXF4zjIfJ9ouObsU5RTpA  提取码:214v 1.打开IDEA 2.在模块下新建个lib文件夹将框架复制进去 3.点 ...

  6. 手摸手带你理解Vue的Computed原理

    前言 computed 在 Vue 中是很常用的属性配置,它能够随着依赖属性的变化而变化,为我们带来很大便利.那么本文就来带大家全面理解 computed 的内部原理以及工作流程. 在这之前,希望你能 ...

  7. Redis五种数据类型应用场景

    目录 1.1 回顾 2.1 应用场景 2.1.1 String 2.1.2 Hash 2.1.3 List 2.1.4 Zet 2.1.5 zset 3.1 小结 1.1 回顾 Redis的五种数据类 ...

  8. keras中loss与val_loss的关系

    loss是训练集的损失值,val_loss是测试集的损失值 以下是loss与val_loss的变化反映出训练走向的规律总结: train loss 不断下降,test loss不断下降,说明网络仍在学 ...

  9. P2194 HXY烧情侣【Tarjan】

    前言 当时和\(GYZ\)大佬一起做这个题,他表示这个题对他很不友好(手动滑稽) 题目描述 众所周知,\(HXY\) 已经加入了 \(FFF\) 团.现在她要开始喜(sang)闻(xin)乐(bing ...

  10. 简单的Linq查询语句

    下面我来我大家介绍几种简单的查询方式. 1.简单语法 这个LINQ语句的第一个关键字是from,from后面加的是范围变量,范围变量后加in,后加上事先实例化的模型,然后点出数据的来源. List是列 ...