CRAWING SPIDER

Goal -> Recursively list all links starting from a base URL.

1. Read page HTML.

2. Extract all links.

3. Repeat for each new link that is not already on the list.

#!/usr/bin/env python

import re

import requests

from urllib.parse import urljoin

target_url = "http://10.0.0.45/mutillidae/"

target_links = []

def extract_links_from(url):

    response = requests.get(url)

    return re.findall('(?:href=")(.*?")', response.content.decode())

def crawl(url):

    href_links = extract_links_from(url)

    for link in href_links:

        link = urljoin(url, link)

        if "#" in link:

            link = link.split("#")[0]

        if target_url in link and link not in target_links:

            target_links.append(link)

            print(link)

            crawl(link)

crawl(target_url)

The Python program runs perfectly.

http://10.0.0.45/mutillidae/favicon.ico"

http://10.0.0.45/mutillidae/styles/global-styles.css"

http://10.0.0.45/mutillidae/styles/ddsmoothmenu/ddsmoothmenu.css"

http://10.0.0.45/mutillidae/styles/ddsmoothmenu/ddsmoothmenu-v.css"

http://10.0.0.45/mutillidae/index.php?page=home.php"

http://10.0.0.45/mutillidae/index.php?page=login.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=login.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=login.php"

http://10.0.0.45/mutillidae/set-up-database.php"

http://10.0.0.45/mutillidae/index.php?page=show-log.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=show-log.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=show-log.php"

http://10.0.0.45/mutillidae/index.php?page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?page=credits.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=credits.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=credits.php"

http://10.0.0.45/mutillidae/"

http://10.0.0.45/mutillidae/index.php?page=user-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=user-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=user-info.php"

http://10.0.0.45/mutillidae/index.php?page=register.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=register.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=register.php"

http://10.0.0.45/mutillidae/index.php?page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php"

http://10.0.0.45/mutillidae/index.php?page=password-generator.php&username=anonymous"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=password-generator.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=password-generator.php"

http://10.0.0.45/mutillidae/index.php?page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?page=framing.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=framing.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=framing.php"

http://10.0.0.45/mutillidae/framer.html"

http://10.0.0.45/mutillidae/index.php?page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?page=installation.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=installation.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=installation.php"

http://10.0.0.45/mutillidae/documentation/mutillidae-installation-on-xampp-win7.pdf"

http://10.0.0.45/mutillidae/index.php?page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=home.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=home.php"

http://10.0.0.45/mutillidae/

http://10.0.0.45/mutillidae/?page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/?page=view-someones-blog.php"

http://10.0.0.45/mutillidae/?page=show-log.php"

http://10.0.0.45/mutillidae/?page=text-file-viewer.php"

http://10.0.0.45/mutillidae/?page=user-info.php"

http://10.0.0.45/mutillidae/?page=login.php"

http://10.0.0.45/mutillidae/?page=credits.php"

http://10.0.0.45/mutillidae/?page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?page=notes.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=notes.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=notes.php"

Python Ethical Hacking - WEB PENETRATION TESTING(4)的更多相关文章

  1. Python Ethical Hacking - WEB PENETRATION TESTING(1)

    WHAT IS A WEBSITE Computer with OS and some servers. Apache, MySQL ...etc. Cotains web application. ...

  2. Python Ethical Hacking - WEB PENETRATION TESTING(2)

     CRAWING DIRECTORIES Directories/folders inside the web root. Can contain files or other directories ...

  3. Python Ethical Hacking - WEB PENETRATION TESTING(5)

    Guessing Login Information on Login Pages Our target website: http://10.0.0.45/dvwa/login.php #!/usr ...

  4. Python Ethical Hacking - WEB PENETRATION TESTING(3)

    CRAWLING SUMMARY Our crawler so far can guess: Subdomains. Directories. Files. Advantages: ->Disc ...

  5. Ethical Hacking - Web Penetration Testing(13)

    OWASP ZAP(ZED ATTACK PROXY) Automatically find vulnerabilities in web applications. Free and easy to ...

  6. Ethical Hacking - Web Penetration Testing(8)

    SQL INJECTION WHAT IS SQL? Most websites use a database to store data. Most data stored in it(userna ...

  7. Ethical Hacking - Web Penetration Testing(10)

    SQL INJECTION SQLMAP Tool designed to exploit SQL injections. Works with many DB types, MySQL, MSSQL ...

  8. Ethical Hacking - Web Penetration Testing(6)

    REMOTE FILE INCLUSION Similar to local file inclusion. But allows an attacker to read ANY file from ...

  9. Ethical Hacking - Web Penetration Testing(4)

    CODE EXECUTION VULNS Allows an attacker to execute OS commands. Windows or Linux commands. Can be us ...

随机推荐

  1. rust 生命周期2

    之前定义的结构体,都是不含引用的. 如果想定义含引用的结构体,请定义生命周期注解 #[warn(unused_variables)] struct ImportantExcerpt<'a> ...

  2. 网站用https访问的问题

    网站挂到阿里云上, 可以http访问, 也可以https访问. 但是如果用https方式访问网站.发现接口报错. 因为接口只提供http方式. 在谷歌浏览器出现: Mixed Content: The ...

  3. 'ipconfig' 不是内部或外部命令，也不是可运行的程序 或批处理文件

    今天在学习的时候需要找本地ip地址,可是在命令行窗口却显示 百度之后发现原来是环境变量没配置的问题(其实之前是ok的,但应该是anconda安装的时候点了那个一键设置环境变量搞得本地的path里的数据 ...

  4. hdoj3791

    题目: Problem Description 判断两序列是否为同一二叉搜索树序列   Input 开始一个数n,(1<=n<=20) 表示有n个需要判断,n= 0 的时候输入结束.接下去 ...

  5. Jmeter Response Data 乱码问题解决方法

    1. 进入jmeter\apache-jmeter-4.0\bin,打开jmeter.properties 2. 搜索“sampleresult.default.encoding” 设置sampler ...

  6. Shiro简单加密服务

    编码/加密 在涉及到密码存储问题上,应该加密 / 生成密码摘要存储,而不是存储明文密码.比如之前的 600w csdn 账号泄露对用户可能造成很大损失,因此应加密 / 生成不可逆的摘要方式存储. Sh ...

  7. C#数据结构与算法系列（十二）：递归（Recursion）

    1.介绍 简单的说:递归就是方法自己调用自己,每次调用时传入不同的变量,递归有助于编程者解决复杂的问题,同时也让代码变得整洁 2.规则 执行一个方法时,就创建一个新的受保护的独立空间(栈空间) 方法的 ...

  8. 一文搞懂InnoDB索引存储结构

    参考资料:掘金小册:MySQL 是怎样运行的:从根儿上理解 MySQL B+树 我们知道,InnoDB是用B+树作为组织数据形式的数据结构.不论是存放用户记录的数据页,还是存放目录项记录的数据页,我们 ...

  9. java异常处理之try_catch_finally

    java异常处理之try_catch_finally 看下面的一个列子: public class TestException { int goabl=1; public TestException( ...

  10. asp.net Core依赖注入（自带的IOC容器）

    今天我们主要讲讲如何使用自带IOC容器,虽然自带的功能不是那么强大,但是胜在轻量级..而且..不用引用别的库. 在新的ASP.NET Core中,大量的采用了依赖注入的方式来编写代码. 比如,在我们的 ...