CRAWING SPIDER

Goal -> Recursively list all links starting from a base URL.

1. Read page HTML.

2. Extract all links.

3. Repeat for each new link that is not already on the list.

#!/usr/bin/env python

import re

import requests

from urllib.parse import urljoin

target_url = "http://10.0.0.45/mutillidae/"

target_links = []

def extract_links_from(url):

    response = requests.get(url)

    return re.findall('(?:href=")(.*?")', response.content.decode())

def crawl(url):

    href_links = extract_links_from(url)

    for link in href_links:

        link = urljoin(url, link)

        if "#" in link:

            link = link.split("#")[0]

        if target_url in link and link not in target_links:

            target_links.append(link)

            print(link)

            crawl(link)

crawl(target_url)

The Python program runs perfectly.

http://10.0.0.45/mutillidae/favicon.ico"

http://10.0.0.45/mutillidae/styles/global-styles.css"

http://10.0.0.45/mutillidae/styles/ddsmoothmenu/ddsmoothmenu.css"

http://10.0.0.45/mutillidae/styles/ddsmoothmenu/ddsmoothmenu-v.css"

http://10.0.0.45/mutillidae/index.php?page=home.php"

http://10.0.0.45/mutillidae/index.php?page=login.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=login.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=login.php"

http://10.0.0.45/mutillidae/set-up-database.php"

http://10.0.0.45/mutillidae/index.php?page=show-log.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=show-log.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=show-log.php"

http://10.0.0.45/mutillidae/index.php?page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?page=credits.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=credits.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=credits.php"

http://10.0.0.45/mutillidae/"

http://10.0.0.45/mutillidae/index.php?page=user-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=user-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=user-info.php"

http://10.0.0.45/mutillidae/index.php?page=register.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=register.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=register.php"

http://10.0.0.45/mutillidae/index.php?page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php"

http://10.0.0.45/mutillidae/index.php?page=password-generator.php&username=anonymous"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=password-generator.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=password-generator.php"

http://10.0.0.45/mutillidae/index.php?page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?page=framing.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=framing.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=framing.php"

http://10.0.0.45/mutillidae/framer.html"

http://10.0.0.45/mutillidae/index.php?page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?page=installation.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=installation.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=installation.php"

http://10.0.0.45/mutillidae/documentation/mutillidae-installation-on-xampp-win7.pdf"

http://10.0.0.45/mutillidae/index.php?page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=home.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=home.php"

http://10.0.0.45/mutillidae/

http://10.0.0.45/mutillidae/?page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/?page=view-someones-blog.php"

http://10.0.0.45/mutillidae/?page=show-log.php"

http://10.0.0.45/mutillidae/?page=text-file-viewer.php"

http://10.0.0.45/mutillidae/?page=user-info.php"

http://10.0.0.45/mutillidae/?page=login.php"

http://10.0.0.45/mutillidae/?page=credits.php"

http://10.0.0.45/mutillidae/?page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?page=notes.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=notes.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=notes.php"

Python Ethical Hacking - WEB PENETRATION TESTING(4)的更多相关文章

  1. Python Ethical Hacking - WEB PENETRATION TESTING(1)

    WHAT IS A WEBSITE Computer with OS and some servers. Apache, MySQL ...etc. Cotains web application. ...

  2. Python Ethical Hacking - WEB PENETRATION TESTING(2)

     CRAWING DIRECTORIES Directories/folders inside the web root. Can contain files or other directories ...

  3. Python Ethical Hacking - WEB PENETRATION TESTING(5)

    Guessing Login Information on Login Pages Our target website: http://10.0.0.45/dvwa/login.php #!/usr ...

  4. Python Ethical Hacking - WEB PENETRATION TESTING(3)

    CRAWLING SUMMARY Our crawler so far can guess: Subdomains. Directories. Files. Advantages: ->Disc ...

  5. Ethical Hacking - Web Penetration Testing(13)

    OWASP ZAP(ZED ATTACK PROXY) Automatically find vulnerabilities in web applications. Free and easy to ...

  6. Ethical Hacking - Web Penetration Testing(8)

    SQL INJECTION WHAT IS SQL? Most websites use a database to store data. Most data stored in it(userna ...

  7. Ethical Hacking - Web Penetration Testing(10)

    SQL INJECTION SQLMAP Tool designed to exploit SQL injections. Works with many DB types, MySQL, MSSQL ...

  8. Ethical Hacking - Web Penetration Testing(6)

    REMOTE FILE INCLUSION Similar to local file inclusion. But allows an attacker to read ANY file from ...

  9. Ethical Hacking - Web Penetration Testing(4)

    CODE EXECUTION VULNS Allows an attacker to execute OS commands. Windows or Linux commands. Can be us ...

随机推荐

  1. java中“”==“” equals hashcode的关系

    ava中的数据类型,可分为两类: 1.基本数据类型,也称原始数据类型.byte,short,char,int,long,float,double,boolean 他们之间的比较,应用双等号(==),比 ...

  2. C++的逗号运算符

    逗号运算符( , )是C++语言运算符中优先级最低的一种运算符,结合顺序是从左至右,用来顺序求值(最后一个逗号后面表达式的值作为整个表达式的值). 感觉这个东西还是挺冷门的,之前都不知道,平时也比较少 ...

  3. Docker（五）Docker镜像讲解

    Docker镜像讲解 镜像概念 镜像是一种轻量级.可执行的独立软件包,用来打包软件运行环境和基于运行环境开发的软件,它包含运行某个软件所需的所有内容,包括代码.运行时.库.环境变量和配置文件 Dock ...

  4. day19__生成器，迭代器

    一.列表解析 l1 = ['鸡蛋%s' % i for i in range(10) if i > 5] print(l1) # ['鸡蛋6', '鸡蛋7', '鸡蛋8', '鸡蛋9'] 这种方 ...

  5. mpvue实战-手势滑动导航栏

    写点东西记录一下美好时光,上周学习了一下通过mpuve开发微信小程序,看完文档,就准备撸起袖子加油干的时候,一开始就被支持手势滑动的导航栏给搞懵逼了.求助一波百度和谷歌未果后,只能自己动脑动手!为了给 ...

  6. P1640 [SCOI2010]连续攻击游戏【并查集】

    题目描述 lxhgww最近迷上了一款游戏,在游戏里,他拥有很多的装备,每种装备都有2个属性,这些属性的值用[1,10000]之间的数表示.当他使用某种装备时,他只能使用该装备的某一个属性.并且每种装备 ...

  7. hive中order by ,sort by ,distribute by, cluster by 的区别（**很详细**）

    hive 查询语法 select [all | distinct] select_ condition, select_ condition from table_name a [join table ...

  8. centos7在Evolution中配置163邮箱，被阻止收件解决方法

    config.mail.163.com/settings/imap/login.jsp?uid=xxxx@163.com

  9. C++版的网络数据包解析策略(升级版)

    初版:http://www.cnblogs.com/wjshan0808/p/6580638.html 说明:在实现了对应的接口后该策略可以适合绝大多数的网络数据包结构 首先,是三个接口 IProdu ...

  10. Python3笔记014 - 3.5 跳转语句

    第3章 流程控制语句 3.5 跳转语句 1.break 语句 while 条件表达式1: 语句块1 if 条件表达式2: break for 迭代变量 in 对象: 语句块1 if 条件表达式: br ...