MineMeld: threat intelligence automation – search received IoC events with Splunk [4]

This post is the fourth of a series on Threat Intelligence Automation topic.
Post 1: Architecture and Hardening of MineMeld
Post 2: Foundation: write a custom prototype and SOC integration
Post 3: Export internal IoC to the community
Post 5: Connect to a TAXII service

After having laid the foundations for building a community with the previous posts, it’s now time to make some advanced analysis of the received IoC.
In post 2 I integrated MineMeld output nodes into Splunk SOC near-real-time engine to automate SOC IoC access detection. This configuration strengthens the analysis and response capabilities of our SOC.

With this post I show you how to integrate MineMeld miners IoC events (update and withdraw of remote IoC) into Splunk engine so you can use Splunk search advanced features to have a deeper look into the IoC received from the miners.
This is also an important information for a SOC because if you have an IoC hit the first think to do is to understand where the IoC come from, if it was sent by more than one source etc

Each miner periodically polls external sources and emits:

• an UPDATE message, when an IoC is added (EMIT_UPDATE). In the following picture pack-lines[.]com domain has been added to italian CERT-PA Infosec domain feed (see post 2 for details);
• a WITHDRAW message, when an IoC is deleted (EMIT_WITHDRAW). In the following picture domain www[.]germamedic[.]it has been removed from italian CERT-PA Infosec domain feed (see post 2 for details).

MineMeld has a very simple search interface which allows to search for specific events (update or withdraw) and IoC details (URL, domain, md5, sha etc).
The goal is to integrate these events into Splunk engine and make some dashboard to search into MineMeld data and do advanced analysis; this configuration provides analysts with additional analysis capabilities.

There are no specif MineMeld prototype to connect to Splunk but I found a logstash connector and used it.
This prototype by default sends tcp data to local logstash istance (default to 127.0.0.1:5514).
Why not send the same logs to Splunk? Is just a matter of parsing on Splunk side 😉

STEP 1: clone logstash prototype

From prototypes page, clone the stdlib.localLogStash prototype to a new one minemeldlocal.LOG-TO-SPLUNK. While cloning change the 2 prototype parameters as follow:

• logstash_host: <YOUR SPLUNK IP ADDRESS>;
• logstash_port: 1534 (or any port where Splunk will listen for MineMeld data).

The new minemeldlocal.LOG-TO-SPLUNK prototype looks like this.

STEP 2: Install the Splunk app to parse MineMeld data

I wrote a simple Technology Addon (TA) to receive and parse MineMeld data on Splunk, you can find it on my githut repo.
Download TA-custom-minemeld_ioc file and install it (from web interface you need to convert to .tar.gz first) on your Splunk single-istance or on Splunk forwarders of your distributed deployment (see picture below).

MineMeld sends IoC updates/withdraw to Splunk as a JSON stream with multiple lines (1 event per line).
On Splunk there is the need to parse the stream on forwarder side and before sending data to the indexers.
Notes on Splunk config:

• data are stored in minemeld_ioc index, create it or adjust the index name as you want;
• data are indexed with sourcetype minemeld_ioc;
• this line tells Splunk to break multi-line JSON data in single events: BREAK_ONLY_BEFORE = ^\{

STEP 3: configure MineMeld to send logs to Splunk

Now that Splunk and MineMeld are ready, let’s proceed with the making of the new output node that sends JSON data to Splunk. This new node is based on the cloned prototype minemeldlocal.LOG-TO-SPLUNK.

The existing configuration is the one from post 2 with the 3 italian CERT-PA miners to be connected to the new output node LOG-TO-SPLUNK (see the below image).

From MineMeld CONFIG page, IMPORT the following config in APPEND mode and then COMMIT.

nodes:
 LOG-TO-SPLUNK:
  inputs:
   - CERT-PA_domains
   - CERT-PA_listip
   - CERT-PA_urls
  output: false
  prototype: minemeldlocal.LOG-TO-SPLUNK

You can verify that your output node is receiving update/whitdraw events checking LOG-TO-SPLUNK node logs on MineMeld.

STEP 4: create Splunk dashboards for analysis

Now it’s time to move to Splunk config.
First of all check if data are collected and stored in minemeld_ioc index with a simple query (index=minemeld_ioc). If not, start troubleshooting step 1-2-3 🙂

Then install on your Splunk search head – in case of distributed environment – or on your Splunk single-istance my MineMeld Analysis application.

The app has two views:

• Threat Intelligence Center: a summary of received events (update/whitdraw ). There are two clickable panels that drill-down to Threat Intelligence Search view:
  • Events trend: a graph panel that show aggregated events by message type (update/whitdraw) in the timeline (span time 5 minutes);
  • Last events: a table panel that show details of received events:
    • @indicator: the IoC received;
    • type: type of IoC (sha1, sha256, md5, domain, url);
    • message: update or whitdraw;
    • @origin: the miner that originate the message;
    • _time: index time for the event

• Threat Intelligence Search: a search interface for events. The drill-down by default redirects to the raw data search, but if you have Forsensic Investigator app installed just comment the javascript code of the view and you are redirected to VirusTotal Forensic Investigator search.

Follow a video of the app in action.

Enjoy!