Information Gathering Workflow

Nmap

sudo nmap --script http-auth --script-args http-auth.path=/login -p 80,443 <target-ip> 
sudo nmap --script http-devframework -p 80,443 <target-ip>
sudo nmap --script http-enum -p 80,443 <target-ip> 
sudo nmap --script http-headers -p 80,443 <target-ip> 
sudo nmap --script http-methods -p 80,443 <target-ip>

WHOIS Lookup

WHOIS provides domain registration details like owner, registrar, creation date, and contact info. This is foundational for identifying domain ownership and potential attack surfaces. Modern updates include using APIs or web interfaces to bypass command-line limitations (e.g., rate limits on CLI whois). Be aware of privacy protections like GDPR, which may redact personal data.

Commands/Tools

  • Command-Line WHOIS:

export TARGET="domain.tld"  
whois $TARGET
  • Output: Registration details, nameservers, expiry dates.

  • Caveat: Some registrars block bulk queries; use with sleep delays in scripts.

  • Modern Alternatives:

  • Setup Python Environment:

# Create the environment: Navigate to your project folder and run:
python3 -m venv my_project_env
# Activate the environment:
source my_project_env/bin/activate
# Install packages: Now you can use pip freely without affecting your system.
pip3 install <package-name>
# Deactivate: When you're done working, simply type:
deactivate

Use apt (For System-Wide Tools)

  • Use Python for structured output:

  • Install: pip3 install python-whois.

  • Why: Parses output into JSON-like structure for easier analysis.

    • Web-based: Browse WHOIS via sites like whois.icann.org or domaintools.com.

      • Tool Integration: Use browse_page tool with URL https://whois.icann.org/en/lookup?name=$TARGET and instructions like "Extract domain registration details, owner, and nameservers."

  • Other Tools:

    • web_search with query "whois $TARGET" for aggregated results from multiple sources.

    • For training: Compare outputs from different registrars to spot inconsistencies.

Enumeration

Note: when enumerating internal domains it may be helpful to create a host file entry:

DNS Enumeration

Explanation

DNS enumeration reveals IP addresses, mail servers, and other records tied to a domain, helping map network infrastructure. Types include A (IPv4), PTR (reverse lookup), TXT (SPF/DMARC), MX (mail), and ANY (all). Modernize by automating with scripts and using resilient nameservers to handle failures. This passive step avoids alerting targets.

Commands/Tools

  • Basic A Record:

  • Output: IP addresses associated with the domain.

  • PTR (Reverse Lookup):

  • Why: Identifies domain from IP, useful for verifying ownership.

  • ANY Records:

  • Caveat: Many servers block ANY queries for security; fall back to specific types.

  • TXT Records:

  • Useful for: Security configs like SPF to prevent email spoofing.

  • MX Records:

  • Output: Mail server priorities and hosts.

  • Modern Automation:

    • Python script for batch enumeration:

  • Install: pip install dnspython.

  • Why: Handles errors, supports custom resolvers.

  • Other Tools:

    • code_execution to run the above Python code dynamically.

    • web_search_with_snippets with query "DNS records for $TARGET" for quick public database snippets.

    • For advanced: Use x_keyword_search on X (Twitter) with query "$TARGET DNS leak" filter:links to find public discussions or leaks.

Passive Subdomain Enumeration

Explanation

Passive enumeration discovers subdomains without querying the target directly, using public datasets like certificate logs or search engines. This reduces detection risk. Modern updates include API integrations and semantic searches for broader coverage. Sources like VirusTotal or Censys aggregate data from scans.

Features/Tools

  • Install jq for JSON parsing.

  • Certificate Transparency:

  • Multi-Source Harvesting:

    • Create sources.txt with: baidu, bufferoverun, crtsh, hackertarget, otx, projectdiscovery, rapiddns, sublist3r, threatcrowd, trello, urlscan, virustotal, zoomeye.

  • Install: pip install theHarvester.- Why: Aggregates from multiple APIs; outputs JSON/XML.

  • Modern Alternatives:

    • Use browse_page with URL https://crt.sh/?q=%25.$TARGET and instructions "Extract and list unique subdomains from certificate logs."

    • Semantic search: x_semantic_search with query "subdomains of $TARGET" to find relevant X posts mentioning them.

  • Other Tools:

    • web_search with query "subdomains $TARGET site:github.com" for leaked repos.

    • Shodan.io API (via web_search query "shodan $TARGET subdomains") for IoT/exposed devices.

Passive Infrastructure Identification

Explanation

Identifies tech stack, historical changes, and archived content without active probes. Useful for spotting outdated software or forgotten endpoints. Modernize with URL archiving tools and API fetches.

Features/Tools

  • Install: go install github.com/tomnomnom/waybackurls@latest.

  • Output: URLs with timestamps for timeline analysis.

  • Modern Alternatives:

    • Python for processing:

  • Install: pip install waybackpy.

  • Other Tools:

    • browse_page with URL https://web.archive.org/web/*/$TARGET and instructions "Summarize changes in website structure over the last 5 years."

    • web_search_with_snippets query "historical infrastructure $TARGET" for quick archived insights.

Fingerprinting

Explanation

Fingerprinting detects web technologies, servers, and security like WAFs (Web Application Firewalls). This informs vulnerability scanning. Modernize with verbose outputs and integrations.

Commands/Tools

  • Retrieve HTTP Headers:

  • Output: Server type, cookies, security headers. Check for Verbose Server Information

Test for HTTP Methods

Download and Inspect Web Page Source

Check for SSL/TLS Certificate Details

Enumerate Subdomains or Endpoints

Probe for API Endpoints

Test for Directory Listing

Spoof User-Agent for Recon

Check for Robots.txt

  • Tech Identification:

  • Install: apt install whatweb.

  • Web Interfaces:

  • WAF Detection:

  • Install: pip install wafw00f.

  • Screenshots:

  • Install: Download from GitHub.

  • Software Scan:

  • Install: apt install nikto.

  • Modern Alternatives:

    • Use code_execution to run WhatWeb in a sandbox.

    • view_image for analyzing screenshots from Aquatone outputs.

  • Other Tools:

    • web_search query "fingerprint $TARGET site:builtwith.com" for tech stack via BuiltWith.

  • Python Automation:

  • Output: Raw certificate details, server response.

  • Why: Displays certificate chain and connection info.

  • Scan SSL/TLS Configuration:

  • Install: apt install sslscan.

  • Output: Supported ciphers, TLS versions, and certificate details.

  • Why: Identifies weak protocols or ciphers (e.g., deprecated TLS 1.0).

  • Test Specific TLS Versions:

  • Why: Confirms supported TLS versions; older versions (TLS 1.0/1.1) may indicate vulnerabilities.

    • Caveat: Some servers reject unsupported versions; expect connection errors.

  • Extract Certificate Content:

    • Manual Method:

      1. Open a browser and navigate to https://$TARGET.

      2. Click the lock icon in the URL bar.

      3. Export the certificate as a .pem file (most browsers support this under "Certificate Details").

      4. Analyze:

  • Output: Human-readable certificate details (e.g., issuer, SANs, expiry).

    • Automated Method, Python:

  • Install: pip install pyOpenSSL.

  • Why: Automates certificate extraction and parsing; extracts SANs for subdomain discovery.

Active Subdomain Enumeration

Explanation

Active methods query the target directly (e.g., brute-forcing), risking detection. Use sparingly in training. Modernize with rate-limiting and wordlist optimizations.

Brute-Force:

  • Install: apt install gobuster; Wordlists from SecLists GitHub.

  • Dnsenum:

  • Install: apt install dnsenum.

  • Other Tools:

    • web_search query "active subdomain enumeration tools 2025" for latest alternatives like Amass.

    • Integrate with code_execution for custom brute-force scripts

Virtual Hosts

Explanation

Discovers hidden hosts on the same IP. Useful for shared hosting environments.

Commands/Tools

  • Wordlist: Custom or from SecLists.

  • Other Tools:

    • browse_page on virtual host testing sites with instructions "List potential vhosts for $IP."

Crawling

Explanation

Crawling extracts links, forms, and data systematically. Scrapy is ideal for large-scale; modernize with handling JS-rendered sites.

Commands/Tools

  • Install Scrapy:

  • Custom Spider (ReconSpider Example):

    • Download: wget https://example.com/ReconSpider.zip; unzip ReconSpider.zip (update URL as needed).

  • Why: Gathers emails, links, metadata.

  • Modern Scrapy Spider: Create spider.py:

  • Other Tools:

    • browse_page with URL http://$TARGET and instructions "Crawl and summarize all internal links, forms, and endpoints."

    • web_search_with_snippets query "crawl $TARGET endpoints" for public crawls.

    • For videos/images: view_x_video or view_image if media URLs are found.

Check Comments in HTML Source

Purpose: Extract hidden comments in HTML for hints (e.g., API keys, endpoints).

  • Command:

  • Why: Filters HTML comments from page source.

  • Python Automation:

  • Install: pip install requests beautifulsoup4

  • Why: Parses comments reliably, handles dynamic content.

  • Alternative: browse_page on http://$TARGET with instructions "Extract all HTML comments from the source code."

Find Source Code

Purpose: Locate public source code on platforms like GitHub/GitLab for insights into the tech stack or vulnerabilities.

  • GitHub Dorks:

  • Search: Use GitHub’s search bar or web_search query "$TARGET language:Python site:github.com".

  • Python Automation:

  • Install: pip install requests

  • Why: Queries GitHub API for repositories; respects rate limits (add token for higher limits).

  • Alternatives:

    • web_search_with_snippets query "$TARGET site:gitlab.com | site:bitbucket.org" for other platforms.

    • x_keyword_search on X with query "$TARGET source code leak filter:links" for exposed repos.

HTTP Requests with Python

Purpose: Interact with web servers to extract data or test endpoints.

  • GET Request:

  • Why: Customizable for query parameters, authentication.

  • GET with Session:

  • Why: Persists cookies across requests for stateful interactions.

  • POST Request:

  • Why: Tests forms or API endpoints.

  • POST with Session:

  • Why: Maintains session for multi-step interactions.

  • Alternative: Use code_execution to run these scripts dynamically, saving results to JSON:

Automating Recon

FinalRecon

Installation:

Modules and options:

Recon Examples:

FinalRecon is a Python-based tool for web reconnaissance, offering modules for header analysis, SSL certificate checking, WHOIS lookup, crawling, DNS enumeration, subdomain enumeration, directory searching, port scanning, and more. Below are example commands to demonstrate its usage for training purposes.

1. Check HTTP Headers

Purpose: Retrieve header information such as server type, content-encoding, and scripting language from a target website. Command:

Description: This command fetches HTTP headers, which can reveal server software, caching mechanisms, or other configuration details useful for reconnaissance.

2. Analyse SSL Certificate

Purpose: Gather SSL/TLS certificate details, such as issuer, serial number, and expiration date, to assess the target's security. Command:

Description: This extracts SSL certificate information, helping identify potential misconfigurations or weak encryption.

3. Perform WHOIS Lookup

Purpose: Collect domain registration details, such as owner, registrar, and registration dates. Command:

Description: WHOIS data provides insights into the domain’s ownership and history, useful for understanding the target’s background.

4. Crawl Target Website

Purpose: Fetch web pages, links, JavaScript files, and other resources from the target site. Command:

Description: Crawling maps the website’s structure, identifying internal/external links, images, and scripts that may reveal sensitive endpoints.

5. DNS Enumeration

Purpose: Query DNS records (e.g., A, AAAA, MX, TXT) to gather network information. Command:

Description: This enumerates DNS records, which can reveal IP addresses, mail servers, or DMARC policies associated with the domain.

6. Subdomain Enumeration

Purpose: Discover subdomains of the target to identify additional attack surfaces. Command:

Description: Uses sources like Certificate Transparency logs to find subdomains. Requires API keys for some sources (configured in keys.json).

Purpose: Identify hidden directories or files on the target web server. Command:

Description: Searches for directories and files with specified extensions (e.g., .txt, .php, .html) using a custom wordlist for brute-forcing.

8. Fast Port Scan

Purpose: Scan for open ports on the target to identify running services. Command:

Description: Performs a fast scan of the top 1000 ports, useful for identifying services like HTTP, SSH, or FTP.

9. Full Reconnaissance

Purpose: Run all available modules for comprehensive reconnaissance. Command:

Description: Executes all scans (headers, SSL, WHOIS, crawl, DNS, subdomains, directories, ports) and saves results in a text file for analysis.

10. Custom Configuration with API Keys

Purpose: Enhance subdomain enumeration using external API keys (e.g., Shodan, VirusTotal). Command:

Description: Configures an API key in keys.json to leverage external data sources for more accurate subdomain enumeration.

11. Port Scanning

Identifying open ports is a fundamental step in network enumeration. Default Port Scan Scans for the most common TCP ports.

Custom Port Scan To check specific ports that you suspect might be open.

Top Ports Scan To scan the 'n' most common ports.

Notes

  • Installation: Clone from GitHub (git clone https://github.com/thewhiteh4t/FinalRecon.git), navigate to the directory, and install dependencies (pip3 install -r requirements.txt).

  • API Keys: Some modules (e.g., subdomain enumeration) use optional API keys. Edit finalrecon/conf/keys.json to add keys or set unused sources to null.

  • Output: Use the -o flag to export results in txt, xml, or csv formats (e.g., -o csv).

  • Ethical Use: Only scan targets you have permission to test. Unauthorized scanning is illegal.

  • Source: Examples are based on FinalRecon’s documentation and usage guide from its GitHub repository.

Last updated