How to Use the Shodan API at Scale

This is a quick post mostly for refreshing my memory in the future. I recently wanted to download the data Shodan had on a large corporate IP space with disparate ranges and several hundred thousand IP addresses for post processing.

As far as I can tell the Shodan help docs are scattered across too many pages and domains and subdomains. There are a few guides out there on the basics of Shodan CLI and API but I didn’t see anything that documented things at a slightly larger scale so here are a few quick notes on gathering this data. Shodan needs no introduction, and the basics are well covered so I’ll dive in.

API Plans and Credits and Scanning Credits and Download Credits and Oh My

API plans? Credits? What do you need to know to make sure you have the right one? In my case I just wanted to download the data Shodan had and for the moment didn’t need to make use of any scanning or real-time data feed features. The Freelancer plan would work in this case but fortunately for me, my employer is helping funding Shodan with a corporate license. This also gives me access to bulk lookups which speeds up time to results.

Pricing

If you’re not going past the first page and not using filters, no credits are being used and no account is required.

Credit Types

Obtaining data costs you “credits”. Depending how you access it you’re using different credit types. The following info comes from the Shodan Credits Explained page, which mostly just confuses me.

You can check your current usage at https://developer.shodan.io/dashboard

CLI Usage

This is straightforward and the docs are at https://cli.shodan.io/ . The CLI can also be used to search and parse data.

$ shodan search -h
shodan search [OPTIONS] <search query>
$ shodan search --fields ip_str,port,org,hostnames microsoft iis 6.0

$ shodan download -h
shodan download [OPTIONS] <filename> <search query>
$ shodan download --limit 100 file_name filter:query

$ shodan parse -h
shodan parse [OPTIONS] <filenames>
$ shodan parse --fields ip_str,port,org --separator , microsoft-data.json.gz

$ shodan convert -h
shodan convert [OPTIONS] <input file> <output format>

$shodan convert file_name.json.gz csv

Keep in mind, this is all in Python so the CLI tools can be trivially modified - find where your shodan.py files are installed and modify as you please. For example, shodan scan list only returns the last 10 results, but if you go look at the source, you’ll see you can quickly make changes to the REST calls (at ~/.local/lib/python3.6/site-packages/shodan/cli/scan.py on my system).

Downloading data with the API

There’s a pretty basic API example in the Shodan API Guide to get you started. In my case I had thousands of IPs and ranges I was looking at and the api.host() bulk lookup function was useful as it can take in array of 100 IPs per request. Note this feature requires a Corporate API plan.

import shodan
api = shodan.Shodan('YOUR CORPORATE API KEY')

hosts = api.host([
    '8.8.8.8',
    '8.8.4.4',
])

for info in hosts:
    print(info['ip_str'])

Once data has been downloaded, you can use the CLI to parse, but additional processing with Python can be useful. There are a few built in helper functions outlined on the Working with Shodan Data Files page. I ended up just writing my own, dumping the json in a format that still allowed the CLI to parse it.

The following scripts should help get you started with downloading data from Shodan, start scans, and parse out some CVEs from the downloaded data files.

Download Shodan Data from a list of CIDRs or IPs

Scan IPs from a file

Parse out CVEs and save to CSV

Of note: the banner specs if you’re parsing this data can be useful for types and optional fields - https://developer.shodan.io/api/banner-specification.

Also useful is the list of query filters (below), most of which can be used with shodan parse --fields <filter>.

Using the Scanning API

Just a quick blurb on this. Again, this requires a paid API plan. From the docs

Shodan crawls the entire Internet at least once a month, but if you want to request Shodan to scan a network immediately you can do so using the on-demand scanning capabilities of the API. A few common reasons to launch a scan are:

• Validate firewall rules
• Confirm issue was patched/ fixed
• Check custom ports

And a note about scan status. Due to the way banner grabbing and services enumeration is done, a scan status might say DONE but the results won’t actually be ready. DONE in this case means it’s been picked up and has started to process. It’s a known caveat and if you need results ASAP they can be picked up with Shodan network monitor or simply waiting some period of time before downloading the results. Check out the REST API docs on scan statuses.

Query Filters

From the API docs, a list of search query filters:

References

• https://developer.shodan.io/api
• https://shodan.readthedocs.io/en/latest/index.html
• https://help.shodan.io/mastery/working-with-shodan-data-files
• https://help.shodan.io/developer-fundamentals/looking-up-ip-info
• https://github.com/achillean/shodan-python
• https://danielmiessler.com/study/shodan/
• https://pen-testing.sans.org/blog/2015/12/08/effective-shodan-searches