CRAWING SPIDER

Goal -> Recursively list all links starting from a base URL.

1. Read page HTML.

2. Extract all links.

3. Repeat for each new link that is not already on the list.

#!/usr/bin/env python

import re

import requests

from urllib.parse import urljoin

target_url = "http://10.0.0.45/mutillidae/"

target_links = []

def extract_links_from(url):

    response = requests.get(url)

    return re.findall('(?:href=")(.*?")', response.content.decode())

def crawl(url):

    href_links = extract_links_from(url)

    for link in href_links:

        link = urljoin(url, link)

        if "#" in link:

            link = link.split("#")[0]

        if target_url in link and link not in target_links:

            target_links.append(link)

            print(link)

            crawl(link)

crawl(target_url)

The Python program runs perfectly.

http://10.0.0.45/mutillidae/favicon.ico"

http://10.0.0.45/mutillidae/styles/global-styles.css"

http://10.0.0.45/mutillidae/styles/ddsmoothmenu/ddsmoothmenu.css"

http://10.0.0.45/mutillidae/styles/ddsmoothmenu/ddsmoothmenu-v.css"

http://10.0.0.45/mutillidae/index.php?page=home.php"

http://10.0.0.45/mutillidae/index.php?page=login.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=login.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=login.php"

http://10.0.0.45/mutillidae/set-up-database.php"

http://10.0.0.45/mutillidae/index.php?page=show-log.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=show-log.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=show-log.php"

http://10.0.0.45/mutillidae/index.php?page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=captured-data.php"

http://10.0.0.45/mutillidae/index.php?page=credits.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=credits.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=credits.php"

http://10.0.0.45/mutillidae/"

http://10.0.0.45/mutillidae/index.php?page=user-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=user-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=user-info.php"

http://10.0.0.45/mutillidae/index.php?page=register.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=register.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=register.php"

http://10.0.0.45/mutillidae/index.php?page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=view-someones-blog.php"

http://10.0.0.45/mutillidae/index.php?page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/index.php?page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=site-footer-xss-discussion.php"

http://10.0.0.45/mutillidae/index.php?page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=html5-storage.php"

http://10.0.0.45/mutillidae/index.php?page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=capture-data.php"

http://10.0.0.45/mutillidae/index.php?page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=dns-lookup.php"

http://10.0.0.45/mutillidae/index.php"

http://10.0.0.45/mutillidae/index.php?page=password-generator.php&username=anonymous"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=password-generator.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=password-generator.php"

http://10.0.0.45/mutillidae/index.php?page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=user-poll.php"

http://10.0.0.45/mutillidae/index.php?page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=set-background-color.php"

http://10.0.0.45/mutillidae/index.php?page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=pen-test-tool-lookup.php"

http://10.0.0.45/mutillidae/index.php?page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=text-file-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=browser-info.php"

http://10.0.0.45/mutillidae/index.php?page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=arbitrary-file-inclusion.php"

http://10.0.0.45/mutillidae/index.php?page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=secret-administrative-pages.php"

http://10.0.0.45/mutillidae/index.php?page=framing.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=framing.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=framing.php"

http://10.0.0.45/mutillidae/framer.html"

http://10.0.0.45/mutillidae/index.php?page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=change-log.htm"

http://10.0.0.45/mutillidae/index.php?page=installation.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=installation.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=installation.php"

http://10.0.0.45/mutillidae/documentation/mutillidae-installation-on-xampp-win7.pdf"

http://10.0.0.45/mutillidae/index.php?page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=documentation/vulnerabilities.php"

http://10.0.0.45/mutillidae/index.php?page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=documentation/how-to-access-Mutillidae-over-Virtual-Box-network.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=home.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=home.php"

http://10.0.0.45/mutillidae/

http://10.0.0.45/mutillidae/?page=add-to-your-blog.php"

http://10.0.0.45/mutillidae/?page=view-someones-blog.php"

http://10.0.0.45/mutillidae/?page=show-log.php"

http://10.0.0.45/mutillidae/?page=text-file-viewer.php"

http://10.0.0.45/mutillidae/?page=user-info.php"

http://10.0.0.45/mutillidae/?page=login.php"

http://10.0.0.45/mutillidae/?page=credits.php"

http://10.0.0.45/mutillidae/?page=source-viewer.php"

http://10.0.0.45/mutillidae/index.php?page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=usage-instructions.php"

http://10.0.0.45/mutillidae/index.php?page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=php-errors.php"

http://10.0.0.45/mutillidae/index.php?page=notes.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-hints&page=notes.php"

http://10.0.0.45/mutillidae/index.php?do=toggle-security&page=notes.php"

Python Ethical Hacking - WEB PENETRATION TESTING(4)的更多相关文章

  1. Python Ethical Hacking - WEB PENETRATION TESTING(1)

    WHAT IS A WEBSITE Computer with OS and some servers. Apache, MySQL ...etc. Cotains web application. ...

  2. Python Ethical Hacking - WEB PENETRATION TESTING(2)

     CRAWING DIRECTORIES Directories/folders inside the web root. Can contain files or other directories ...

  3. Python Ethical Hacking - WEB PENETRATION TESTING(5)

    Guessing Login Information on Login Pages Our target website: http://10.0.0.45/dvwa/login.php #!/usr ...

  4. Python Ethical Hacking - WEB PENETRATION TESTING(3)

    CRAWLING SUMMARY Our crawler so far can guess: Subdomains. Directories. Files. Advantages: ->Disc ...

  5. Ethical Hacking - Web Penetration Testing(13)

    OWASP ZAP(ZED ATTACK PROXY) Automatically find vulnerabilities in web applications. Free and easy to ...

  6. Ethical Hacking - Web Penetration Testing(8)

    SQL INJECTION WHAT IS SQL? Most websites use a database to store data. Most data stored in it(userna ...

  7. Ethical Hacking - Web Penetration Testing(10)

    SQL INJECTION SQLMAP Tool designed to exploit SQL injections. Works with many DB types, MySQL, MSSQL ...

  8. Ethical Hacking - Web Penetration Testing(6)

    REMOTE FILE INCLUSION Similar to local file inclusion. But allows an attacker to read ANY file from ...

  9. Ethical Hacking - Web Penetration Testing(4)

    CODE EXECUTION VULNS Allows an attacker to execute OS commands. Windows or Linux commands. Can be us ...

随机推荐

  1. 深入了解C#（TPL）之Parallel.ForEach异步

    前言 最近在做项目过程中使用到了如题并行方法,当时还是有点犹豫不决,因为平常使用不多, 于是借助周末时间稍微深入了下,发现我用错了,故此做一详细记录,希望对也不是很了解的童鞋在看到本文此文后不要再犯和 ...

  2. Linux下安装 Java

    一.在线下载 Java JDK 环境 (1)搜索 yum 库有什么 JDK 版本 [root@localhost ~]# yum search java | grep jdk ldapjdk-java ...

  3. Maven发展历史

    1.1 Maven是什么 Maven是一个项目管理和综合工具. Maven提供了开发人员构建一个完整的生命周期框架.开发者团队可以自动完成项目的基础工具建设, Maven使用标准的目录结构和默认构建生 ...

  4. java程序员软件测试技巧

    测试是开发的一个非常重要的方面,可以在很大程度上决定一个应用程序的命运.良好的测试可以在早期捕获导致应用程序崩溃的问题,但较差的测试往往总是导致故障和停机. 测试是开发的一个非常重要的方面,可以在很大 ...

  5. Win8.1安装配置64位Oracle Database 11g的详细图文步骤记录

    在开始Oracle数据安装之前建议:1.关闭本机的病毒防火墙.2.断开互联网. 这样可以避免解压缩丢失文件和安装失败. Step1 Oracle官网下载好Windows系统64位的安装包,下载速度慢的 ...

  6. No configuration file found and no output filename configured via Cli option.报错

    webpack手动配置webpack.config.js文件,打包时出现的报错,可以试试这种解决方案 报错如下: No configuration file found and no output f ...

  7. keras训练函数fit和fit_generator对比，图像生成器ImageDataGenerator数据增强

    1. [深度学习] Keras 如何使用fit和fit_generator https://blog.csdn.net/zwqjoy/article/details/88356094 ps:解决样本数 ...

  8. slow SQL

    一.介绍 慢查询日志可用于查找需要很长时间才能执行的查询,因此是优化的候选者.但是,检查长慢的查询日志可能是一项耗时的任务. 二.配置 # 查看: slow_query_log 慢SQL开关 slow ...

  9. 46道Linux面试题送给你(后续会不断更新)

    绝对路径用什么符号表示? 当前目录.上层目录用什么表示?主目录用什么表示? 切换目录用什么命令? 答案: # 绝对路径: 如/etc/init.d # 当前目录和上层目录: ./ ../ # 主目录: ...

  10. 【Xamarin.Forms 1】App的创建与运行

    引言 本篇文章将从介绍Xamarin.Forms创建开始. 开发环境 Visual Studio 2019 16.6.2 Xamarin.Forms 4.6.0.726 Android 5.0 (AP ...