SpiderFoot

During a penetration test, one of the most important aspects of engaging a target is information gathering. The more information you have coming into an attack, the more likely the attack is to succeed.

SpiderFoot is a reconnaissance tool that automatically queries over 100 public data sources (OSINT) to gather intelligence on IP addresses, domain names, e-mail addresses, names and more. You simply specify the target you want to investigate, pick which modules to enable and then SpiderFoot will collect data to build up an understanding of all the entities and how they relate to each other.

▒ What is OSINT?

OSINT (Open Source Intelligence) is data available in the public domain which might reveal interesting information about your target. This includes DNS, Whois, Web pages, passive DNS, spam blacklists, file meta data, threat intelligence lists as well as services like SHODAN, HaveIBeenPwned? and more.

▒ What can I do with SpiderFoot?

The data returned from a SpiderFoot scan will reveal a lot of information about your target, providing insight into possible data leaks, vulnerabilities or other sensitive information that can be leveraged during a penetration test, red team exercise or for threat intelligence. Try it out against your own network to see what you might have exposed!

✓ Installing:

$ pip install lxml netaddr M2Crypto cherrypy mako requests bs4 swing
$ git clone https://github.com/smicallef/spiderfoot.git
$ cd spiderfoot
~/spiderfoot$ pip install -r requirements.txt

✓ Running:

To run SpiderFoot, simply execute sf.py from the directory you extracted/pulled SpiderFoot into. Ensure you’re using Python 3; on some Linux distributions python is Python 2.7, so best to be explicit and use python3:

~/spiderfoot$ python3 sf.py
Attempting to verify database and update if necessary...
You must specify a target when running in scan mode. Try --help for guidance.

This is telling you that you’re missing command-line arguments, because SpiderFoot doesn’t know whether you want to run it in scan mode, or in Web UI mode.

To start SpiderFoot in Web UI mode, you need to tell it what IP and port to listen to. The below example binds SpiderFoot to localhost on port 5001:

~/spiderfoot$ python3 sf.py -l 127.0.0.1:5001

🚩 It's important to note that, by default, SpiderFoot doesn't use HTTPS or any form of authentication.

Avoid running it on a server/workstation that can be accessed from untrusted devices, as they will be able to control SpiderFoot remotely and initiate scans from your devices.