CRAWING SPIDER

Goal -> Recursively list all links starting from a base URL.

1. Read page HTML.

2. Extract all links.

3. Repeat for each new link that is not already on the list.

#!/usr/bin/env python

import re

import requests

from urllib.parse import urljoin

target_url = "http://10.0.0.45/mutillidae/"

target_links = []

def extract_links_from(url):

    response = requests.get(url)

    return re.findall('(?:href=")(.*?")', response.content.decode())

def crawl(url):

    href_links = extract_links_from(url)

    for link in href_links:

        link = urljoin(url, link)

        if "#" in link:

            link = link.split("#")[0]

        if target_url in link and link not in target_links:

            target_links.append(link)

            print(link)

            crawl(link)

crawl(target_url)

The Python program runs perfectly.

http://10.0.0.45/mutillidae/favicon.ico"

http://10.0.0.45/mutillidae/styles/global-styles.css"

http://10.0.0.45/mutillidae/styles/ddsmoothmenu/ddsmoothmenu.css"

http://10.0.0.45/mutillidae/styles/ddsmoothmenu/ddsmoothmenu-v.css"

http://10.0.0.45/mutillidae/index.php?page=home.php"

http://10.0.0.45/mutillidae/index.php?page=login.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=login.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=login.php"

http://10.0.0.45/mutillidae/set-up-database.php"

http://10.0.0.45/mutillidae/index.php?page=show-log.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=show-log.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=show-log.php"

http://10.0.0.45/mutillidae/index.php?page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?page=credits.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=credits.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=credits.php"

http://10.0.0.45/mutillidae/"

http://10.0.0.45/mutillidae/index.php?page=user-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=user-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=user-info.php"

http://10.0.0.45/mutillidae/index.php?page=register.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=register.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=register.php"

http://10.0.0.45/mutillidae/index.php?page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php"

http://10.0.0.45/mutillidae/index.php?page=password-generator.php&username=anonymous"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=password-generator.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=password-generator.php"

http://10.0.0.45/mutillidae/index.php?page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?page=framing.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=framing.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=framing.php"

http://10.0.0.45/mutillidae/framer.html"

http://10.0.0.45/mutillidae/index.php?page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?page=installation.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=installation.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=installation.php"

http://10.0.0.45/mutillidae/documentation/mutillidae-installation-on-xampp-win7.pdf"

http://10.0.0.45/mutillidae/index.php?page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=home.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=home.php"

http://10.0.0.45/mutillidae/

http://10.0.0.45/mutillidae/?page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/?page=view-someones-blog.php"

http://10.0.0.45/mutillidae/?page=show-log.php"

http://10.0.0.45/mutillidae/?page=text-file-viewer.php"

http://10.0.0.45/mutillidae/?page=user-info.php"

http://10.0.0.45/mutillidae/?page=login.php"

http://10.0.0.45/mutillidae/?page=credits.php"

http://10.0.0.45/mutillidae/?page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?page=notes.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=notes.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=notes.php"

Python Ethical Hacking - WEB PENETRATION TESTING(4)的更多相关文章

  1. Python Ethical Hacking - WEB PENETRATION TESTING(1)

    WHAT IS A WEBSITE Computer with OS and some servers. Apache, MySQL ...etc. Cotains web application. ...

  2. Python Ethical Hacking - WEB PENETRATION TESTING(2)

     CRAWING DIRECTORIES Directories/folders inside the web root. Can contain files or other directories ...

  3. Python Ethical Hacking - WEB PENETRATION TESTING(5)

    Guessing Login Information on Login Pages Our target website: http://10.0.0.45/dvwa/login.php #!/usr ...

  4. Python Ethical Hacking - WEB PENETRATION TESTING(3)

    CRAWLING SUMMARY Our crawler so far can guess: Subdomains. Directories. Files. Advantages: ->Disc ...

  5. Ethical Hacking - Web Penetration Testing(13)

    OWASP ZAP(ZED ATTACK PROXY) Automatically find vulnerabilities in web applications. Free and easy to ...

  6. Ethical Hacking - Web Penetration Testing(8)

    SQL INJECTION WHAT IS SQL? Most websites use a database to store data. Most data stored in it(userna ...

  7. Ethical Hacking - Web Penetration Testing(10)

    SQL INJECTION SQLMAP Tool designed to exploit SQL injections. Works with many DB types, MySQL, MSSQL ...

  8. Ethical Hacking - Web Penetration Testing(6)

    REMOTE FILE INCLUSION Similar to local file inclusion. But allows an attacker to read ANY file from ...

  9. Ethical Hacking - Web Penetration Testing(4)

    CODE EXECUTION VULNS Allows an attacker to execute OS commands. Windows or Linux commands. Can be us ...

随机推荐

  1. Windows 程序设计(4) MFC-02 基本控件-上

    1. Button 按钮控件 1.1.按钮控件的基本使用 新建对话框工程,拖拽按钮控件,添加点击事件响应函数! a.双击模版进行添加: b.事件方式进行添加: button的常见事件类型 void C ...

  2. SpringMVC整合mybaitis

    目录 一.新建一个基于Maven的Web项目 二.创建数据库与表 三.添加依赖包 四.新建POJO实体层 五.新建MyBatis SQL映射层 六.完成Spring整合MyBatis配置 七.创建服务 ...

  3. cc23b_demo-函数对象c++ 调用操作符的重载与函数对象-//用模板定义一元谓词、代码示范

    //用模板定义一元谓词. #include <iostream> #include <vector> #include <algorithm> using name ...

  4. Kafka源码解析（二）---Log分析

    上一篇文章讲了LogSegment和Log的初始化,这篇来讲讲Log的主要操作有哪些. 一般来说Log 的常见操作分为 4 大部分. 高水位管理操作 日志段管理 关键位移值管理 读写操作 其中关键位移 ...

  5. 3、尚硅谷_SSM高级整合_创建Maven项目.avi

    Maven中dependencyManagement作用说明 在Maven多模块的时候,管理依赖关系是非常重要的,各种依赖包冲突,查询问题起来非常复杂,于是就用到了<dependencyMana ...

  6. 入门大数据---Kafka消费者详解

    一.消费者和消费者群组 在 Kafka 中,消费者通常是消费者群组的一部分,多个消费者群组共同读取同一个主题时,彼此之间互不影响.Kafka 之所以要引入消费者群组这个概念是因为 Kafka 消费者经 ...

  7. 逻辑式编程语言极简实现（使用C#） - 1. 逻辑式编程语言介绍

    相信很多朋友对于逻辑式编程语言,都有一种最熟悉的陌生人的感觉.一方面,平时在书籍.在资讯网站,偶尔能看到一些吹嘘逻辑式编程的话语.但另一方面,也没见过周围有人真正用到它(除了SQL). 遥记当时看&l ...

  8. IDEA开发工具使用 git 创建项目、拉取分支、合并分支

    转载自:https://blog.csdn.net/qq_39470733/article/details/80366435 工作中多人使用版本控制软件协作开发,常见的应用场景归纳如下: 假设小组中有 ...

  9. javaScript的三种储存方式

    (一).SessionStorage     会话储存 (二).localStorage           本地储存 (三).Cookier                   现实中为:饼干    ...

  10. 重学 Java 设计模式：实战备忘录模式「模拟互联网系统上线过程中，配置文件回滚场景」

    作者:小傅哥 博客:https://bugstack.cn - 原创系列专题文章 沉淀.分享.成长,让自己和他人都能有所收获! 一.前言 实现不了是研发的借口? 实现不了,有时候是功能复杂度较高难以实 ...