Prioritizing alerts using automated open source intelligence

After a very busy 2011, I'm starting 2012 with a new year's resolution: "To write posts more often". And here is the first one...

Lately I've been working in how to enhance the data using open source intelligence information. And I'm amazed how much value you can get from it.

The idea is to use reputation information from public sources and correlate it with your internal events in order to prioritize alerts. For example, in IDS/IPS alerts, you can correlate the external IPs in the IDS signatures against a list of known Malware IPs and increase the priority of if you get a match. Of course you can extend this to domain names, urls, etc. and also to different log sources as firewalls, proxies and so on.

I've been looking into two open source utilities to retrieve reputation information in an automatic way:

• ArcOSI
  ArcOSI is a Python based utility available for Unix or Windows that scrapes several trusted open source intelligence sites for known malicious IP's and domains and streams them into ArcSight CEF format via Syslog for use in your SIEM content. http://code.google.com/p/arcosi/

• EnigmaIndicators
  Enigma is a bash script that parses known suspicious email address senders, email subjects, email sender and attached files, suspicious files, IP addresses, domains, web requested URLs, URL file names, top requested news feeds, suspicious user agent strings, and suspicious MD5 file hashes from open and custom close source intelligence feeds.  Enigma also calculates entropy (measures the randomness of possible outcome) against the relevant data it parses for advance heuristics detection within ArcSight. All parsed data is then brought into ArcSight via CEF syslog. http://enigmaindicators.codeplex.com/
  

The general idea in both utilities is the same:

1. Scrapes different open sources intelligence sites for known malware information as IPs, domains, urls, MD5 files hashes, email addresses, etc.
2. For each entry, create an CEF event with the source and type of intelligence and the threat information.
3. Send it via Syslog to a defined destination.

Both utilities are designed for a easy integration with ArcSight, using CEF, so no parser is needed.

Just properly defined active lists in ArcSight are needed. In general you'll need an IP-based list and one for each type of reputation information, 1 for malware IPs, 1 for domains, 1 for email address, etc... This will make easier the rules creation and also more efficient because the lists can end up pretty big. For example, in my last test with malware IPs, we are talking about 12K entries and for domains, 65K entries. I'd also recommend to set a TTL for the entries so they are automatically removed from the list.

You will also need to create rules to populate the lists when an event of this type is received. For it, just use the CEF fields to properly filter the events and use the "Add to active list" rule Action.

Once you have defined the rules and the lists are populated, the lists are ready to be used. For example, in the Common Condition Editor as an "InActiveList" condition for a rule definition or as filter in a Active Channel. 

As I said, both tools are designed to use CEF, so the integration with ArcSight is immediate. But it's really easy to customize the scripts, for example to write to a file, so the information can be used in RSA enVision (Import Watchlist) or in Q1 using reference sets.