Web Content Discovery

How to discover files and directories.

Web Application Content Discovery

FeroxBuster

feroxbuster uses brute force combined with a wordlist to search for web content. These resources may store sensitive information about web applications and operational systems, such as source code, credentials, internal network addressing, etc.

• https://github.com/epi052/feroxbuster

The following command will execute a rate limited web brute force using a wordlist of your choosing. It will present as a web browser and filter out unwanted status codes, like HTTP 404. Adjust the --filter-lines value after the first run, when you can determine how many lines a false positive page contains.

./feroxbuster -k -u <URL> --rate-limit 100 --scan-limit 1 -a 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36' --filter-status 404,403,400,503,401,429 --extract-links -w <wordlist.txt> -n -t 100 --filter-lines 0

You may consider using content discovery wordlists from the following sources:

• https://github.com/danielmiessler/SecLists/tree/master/Discovery/Web-Content

API Content Discovery

KiteRunner

Kiterunner is a tool that is capable of performing traditional content discovery but also bruteforcing routes/endpoints in modern applications that use APIs.

• https://github.com/assetnote/kiterunner

The following command will use the default Assetnote wordlist and filter out any unwanted status codes, like HTTP 404. Adjust the --ignore-length value after the first run, when you can determine how many lines a false positive page contains.

./kr scan <URL> -A=apiroutes-240128 --fail-status-codes 400,401,404,403,501,502,426,411 --ignore-length 999999

You may consider using content discovery wordlists from the following sources:

• https://wordlists.assetnote.io/