Network Entity Reputation Database (NERD)

The NERD system gathers data about sources of cyber threats from a number of sources and builds a constantly-updated database of the known malicious network entities (currently only IP addresses). It provides detailed information about each of them, including when and where it was reported as malicious, auxiliary data such as hostname or geolocation, and a numeric summarization of the IP's reputation. Most of the data are publicly accessible via a web interface:

Access the NERD web interface HERE

Network entity profiles

NERD keeps a record for each IP address reported as a source of some network attack or other malicious activity. Such a record contains meta-data about the reports and other relevant information fetched from other sources (see below for the list of all used data sources). Most of the data is periodically updated. The web interface allows to get a detail of each individual IP address as well as search for IPs matching given criteria.

Basic information about related BGP prefixes, ASNs, registered IP blocks and organizations (from whois databases) is stored as well.

Moreover, the information known about each IP is summarized into a reputation score -- a number expressing the level of threat the IP address poses. Currently, quite a simple static formula is used to compute the reputation score, but there is an ongoing research about how to better estimate it.

Data sources

Data about detected malicious activities which are used to create IP profiles (so called "primary data") are taken from the following sources:

• Warden
  • Warden is the alert sharing system operated by CESNET, which collects reports from a number of systems, such as honeypots, netflow analyzers and other detectors of malicious traffic, deployed mostly (but not only) in Czech academic networks. Around a million alerts from over a dozen detection systems are shared every day.
  • Warden is a closed community, full alert data are not publicly available, but we provide basic meta-data, such as the number of alerts per category and IP, via NERD.
  • Until recently, this was the only primary data source of NERD and some parts of it still doesn't take the others into account, e.g. the "reputation score" is only computed from Warden events, filtering by attack category uses a taxonomy from Warden, etc. We are currently working on making NERD more general, i.e. less focused on Warden.
• Blacklists
  • NERD uses almost 50 public "blacklists" (i.e. plain text lists of malicious IP addresses) from around 20 providers. Each IP address appearing on any of these lists gets a record created in NERD.
• DShield
  • DShield is a collaborative firewall log correlation system run by SANS Institute. A large community of contributors share information about unexpected/unsolicited incoming connections into it, statistics about individual ports and IPs are available to anyone.
  • NERD uses a daily feed of all IPs reported to DShield and ingests into its database any IP address reported at least 10 times by at least 3 distinct targets.
• AlienVault Open Threat Exchange (OTX)
  • OTX is an open community portal where security researchers and professionals share millions of threat indicators daily.
  • NERD is subscribed to several users and pulses which we found reliable and useful (the AlienVault account and several honeypot feeds). It look for any IP addresses in these pulses and creates or updates corresponding IP profiles accordingly.
• MISP
  • NERD is connected to a large MISP instance operated by CIRCL (this one) and includes IP addresses from all events tagged with "tlp:white" (i.e. public information).

Each IP record (created based on data from one of the primary sources above) is further enriched by data from the following "secondary" sources:

• Hostname (as resolved by a reverse DNS query)
• Autonomous system number (ASN)
• whois information
• Geolocation (according to MaxMind's GeoLite2 database)
• Presence on blacklists that are not used as primary sources (either lists accessible via DNSBL only, or those whose entries are not necessarily malicious IPs, such as the list of TOR exit nodes)
• CESNET's passive DNS system
• Various tags describing the address, derived from other information

We are always looking for new data sources to include. If you have or know about some data that could be added, please contact us.

Access

The data are available via a simple web GUI and API. Both allow to show details about an entity (IP address, ASN, etc.) or search entities by various parameters.

Basic access to most of the data is allowed to anyone.

However, some of the information in the system may be sensitive and can not be accessible to public (e.g. raw alerts from Warden or data from MISP with TLP other than 'white'). Full access to the NERD system is therefore restricted to trusted partners only. Generally, it is granted to members of trusted CSIRT teams and well-known security researchers.

Source codes & more information

Source codes of all components of the NERD project are available on github. Documentation is available on project's wiki

Although we publish source codes, it's (currently) not expected that someone deploys his or her own instance (some parts would not work well without access to Warden data). Therefore, although there are some installation scripts that should make the installation easy, they are not well tested, documented and do not cover everything. If possible, we recommend to access the main instance operated by CESNET instead and share data or collaborate on development. Nevertheless, if you still need to deploy your own instance, do not hesitate to contact us for assistance.

For more information about how does it work, see our paper from CyberTIM workshop at ARES 2019.

Acknowledgments