MineMeld: threat intelligence automation – search received IoC events with Splunk [4]

This post is the fourth of a series on Threat Intelligence Automation topic.
Post 1: Architecture and Hardening of MineMeld
Post 2: Foundation: write a custom prototype and SOC integration
Post 3: Export internal IoC to the community
Post 5: Connect to a TAXII service

After having laid the foundations for building a community with the previous posts, it’s now time to make some advanced analysis of the received IoC.
In post 2 I integrated MineMeld output nodes into Splunk SOC near-real-time engine to automate SOC IoC access detection. This configuration strengthens the analysis and response capabilities of our SOC.

With this post I show you how to integrate MineMeld miners IoC events (update and withdraw of remote IoC) into Splunk engine so you can use Splunk search advanced features to have a deeper look into the IoC received from the miners.
This is also an important information for a SOC because if you have an IoC hit the first think to do is to understand where the IoC come from, if it was sent by more than one source etc

Searching for Cert-PA URL that containg PHP
Searching for Cert-PA URL that contain PHP

Each miner periodically polls external sources and emits:

  • an UPDATE message, when an IoC is added (EMIT_UPDATE). In the following picture pack-lines[.]com domain has been added to italian CERT-PA Infosec domain feed (see post 2 for details);
  • a WITHDRAW message, when an IoC is deleted (EMIT_WITHDRAW). In the following picture domain www[.]germamedic[.]it has been removed from italian CERT-PA Infosec domain feed (see post 2 for details).
CERT-PA_domain mIner events
CERT-PA_domain mIner events

MineMeld has a very simple search interface which allows to search for specific events (update or withdraw) and IoC details (URL, domain, md5, sha etc).
The goal is to integrate these events into Splunk engine and make some dashboard to search into MineMeld data and do advanced analysis; this configuration provides analysts with additional analysis capabilities.

There are no specif MineMeld prototype to connect to Splunk but I found a logstash connector and used it.
This prototype by default sends tcp data to local logstash istance (default to
Why not send the same logs to Splunk? Is just a matter of parsing on Splunk side 😉

STEP 1: clone logstash prototype

MineMeld local logstash prototype
MineMeld local logstash prototype

From prototypes page, clone the stdlib.localLogStash prototype to a new one minemeldlocal.LOG-TO-SPLUNK. While cloning change the 2 prototype parameters as follow:

  • logstash_host: <YOUR SPLUNK IP ADDRESS>;
  • logstash_port: 1534 (or any port where Splunk will listen for MineMeld data).

The new minemeldlocal.LOG-TO-SPLUNK prototype looks like this.

new prototype minemeldlocal.LOG-TO-SPLUNK
new prototype minemeldlocal.LOG-TO-SPLUNK

STEP 2: Install the Splunk app to parse MineMeld data

I wrote a simple Technology Addon (TA) to receive and parse MineMeld data on Splunk, you can find it on my githut repo.
Download TA-custom-minemeld_ioc file and install it (from web interface you need to convert to .tar.gz first) on your Splunk single-istance or on Splunk forwarders of your distributed deployment (see picture below).

Splunk distributed deployment architecture
Splunk distributed deployment architecture

MineMeld sends IoC updates/withdraw to Splunk as a JSON stream with multiple lines (1 event per line).
On Splunk there is the need to parse the stream on forwarder side and before sending data to the indexers.
Notes on Splunk config:

  • data are stored in minemeld_ioc index, create it or adjust the index name as you want;
  • data are indexed with sourcetype minemeld_ioc;
  • this line tells Splunk to break multi-line JSON data in single events: BREAK_ONLY_BEFORE = ^\{

STEP 3: configure MineMeld to send logs to Splunk

Now that Splunk and MineMeld are ready, let’s proceed with the making of the new output node that sends JSON data to Splunk. This new node is based on the cloned prototype minemeldlocal.LOG-TO-SPLUNK.

The existing configuration is the one from post 2 with the 3 italian CERT-PA miners to be connected to the new output node LOG-TO-SPLUNK (see the below image).

MineMeld output node for Splunk connection
MineMeld output node for Splunk connection

From MineMeld CONFIG page, IMPORT the following config in APPEND mode and then COMMIT.

   - CERT-PA_domains
   - CERT-PA_listip
   - CERT-PA_urls
  output: false
  prototype: minemeldlocal.LOG-TO-SPLUNK
LOG-TO-SPLUNK output node
LOG-TO-SPLUNK output node

You can verify that your output node is receiving update/whitdraw events checking LOG-TO-SPLUNK node logs on MineMeld.


STEP 4: create Splunk dashboards for analysis

Now it’s time to move to Splunk config.
First of all check if data are collected and stored in minemeld_ioc index with a simple query (index=minemeld_ioc). If not, start troubleshooting step 1-2-3 🙂

Splunk query: index=minemeld_ioc
Splunk query: index=minemeld_ioc

Then install on your Splunk search head – in case of distributed environment – or on your Splunk single-istance my MineMeld Analysis application.

The app has two views:

  • Threat Intelligence Center: a summary of received events (update/whitdraw ). There are two clickable panels that drill-down to Threat Intelligence Search view:
    • Events trend: a graph panel that show aggregated events by message type (update/whitdraw) in the timeline (span time 5 minutes);
    • Last events: a table panel that show details of received events:
      • @indicator: the IoC received;
      • type: type of IoC (sha1, sha256, md5, domain, url);
      • message: update or whitdraw;
      • @origin: the miner that originate the message;
      • _time: index time for the event
Threat Intelligence Center
Threat Intelligence Center
  • Threat Intelligence Search: a search interface for events. The drill-down by default redirects to the raw data search, but if you have Forsensic Investigator app installed just comment the javascript code of the view and you are redirected to VirusTotal Forensic Investigator search.
Threat Intelligence Search
Threat Intelligence Search

Follow a video of the app in action.


3 thoughts on “MineMeld: threat intelligence automation – search received IoC events with Splunk [4]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.