MineMeld: threat intelligence automation – foundation: write a custom prototype and SOC integration (Splunk) [2]

This post is the second of a series on Threat Intelligence Automation topic.
Post 1: Architecture and Hardening of MineMeld
Post 3: Export internal IoC to the community
Post 4: Search received IoC events with Splunk
Post 5: Connect to a TAXII service

On the first post of my threat intelligence automation jurney I wrote why I choosed MineMeld, the architecture implemented and the hardening steps. One of the goals is to connect MineMeld to heterogeneous external sources to get IoC (Indicators of Compromise) and integrate it into our i-SOC (Information Security Operation Center) near-real-time engine to get evidences of security events to be analyzed by i-SOC analysts.

In this post I show the foundation of the threat intelligence automation model: how I wrote a custom prototype to get the InfoSec feeds from italian CERT-PA (Public Administration – italian web site) and how I integrated these feeds into Splunk near-real-time engine.
I started with this integration because InfoSec has very good feeds (IP, URLs, domains) that are not just copy&paste from OSINT sources but are often updated and automatically analyzed to check that IoC are still “alive”.

This page and this page give you all the information needed to understand how MineMeld works, so RTFM before moving on 😉

InfoSec feeds are public available at following URLs as a list with some comment on top:

• IP addresses: https://infosec.cert-pa.it/analyze/listip.txt
• Domains: https://infosec.cert-pa.it/analyze/listdomains.txt
• URLs: https://infosec.cert-pa.it/analyze/listurls.txt

I wrote a local custom prototype to get InfoSec feeds; I sent also a PR to MineMeld prototypes github repo, and it was accepted so anyone can use upstream when will be available.

UPDATE 19 Sept 2017: Version 0.9.42 includes itcertpa prototype so if you upgrade or install last versione  from Ubuntu repos (version 0.9.11-1build1,  don’t know why…) you don’t need to follow the following section.

[IF MM_VERSION < 0.9.42]
Version < 0.9.42 don’t have itcertpa prototype installed so create the local file (beware to the name!!) /opt/minemeld/local/prototypes/itcertpa.yml  with following content (or download the file here).

url: https://infosec.cert-pa.it/
description: >
    Infosec feeds from italian CERT-PA, Computer Emergency Response Team - Public Administration
    Implemented prototypes:
    - itcertpa.IP: IP addresses (/32)
    - itcertpa.DOMAINS: domains
    - itcertpa.URLS: URLs

prototypes:
    IP:
        author: Giovanni Mellini
        development_status: STABLE
        node_type: miner
        indicator_types:
            - IPv4
        tags:
            - ConfidenceHigh
            - ShareLevelGreen
        description: Italian CERT-PA Infosec IP hosts indicators
        config:
            source_name: itcertpa.IP
            attributes:
                type: IPv4
                direction: inbound
                confidence: 80
                share_level: green
            ignore_regex: '^#.*'
            indicator:
                regex: '^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$'
            url: https://infosec.cert-pa.it/analyze/listip.txt
            age_out:
                default: null
                sudden_death: true
                interval: 600
        class: minemeld.ft.http.HttpFT

    DOMAINS:
        author: Giovanni Mellini
        development_status: STABLE
        node_type: miner
        indicator_types:
            - domain
        tags:
            - ConfidenceHigh
            - ShareLevelGreen
        description: Italian CERT-PA Infosec DOMAINS indicators
        config:
            source_name: itcertpa.DOMAINS
            attributes:
                type: domain
                direction: inbound
                confidence: 80
                share_level: green
            ignore_regex: '^#.*'
            indicator:
                regex: '^.*'
            url: https://infosec.cert-pa.it/analyze/listdomains.txt
            age_out:
                default: null
                sudden_death: true
                interval: 600
        class: minemeld.ft.http.HttpFT

    URLS:
        author: Giovanni Mellini
        development_status: STABLE
        node_type: miner
        indicator_types:
            - URL
        tags:
            - ConfidenceHigh
            - ShareLevelGreen
        description: Italian CERT-PA Infosec URLS indicators
        config:
            source_name: itcertpa.URLS
            attributes:
                type: URL
                direction: inbound
                confidence: 80
                share_level: green
            ignore_regex: '^#.*'
            indicator:
                regex: '^http.*'
            url: https://infosec.cert-pa.it/analyze/listurls.txt
            age_out:
                default: null
                sudden_death: true
                interval: 600
        class: minemeld.ft.http.HttpFT

After that, just in case, restart your MineMeld istance (# service minemeld restart) and logout/login from the web interface.
[FI]

From the prototypes page (from config page, the small button on bottom-right) search for itcertpa; the 3 new are available and ready to be used as miners (input sources).

Now it’s time to build the threat intelligence chain; the simplest way is to import the following configuration (copy&paste using import button in config page, please note that indentation is important). Later on the post I will describe in details the config.

nodes:
 aggregator-URL:
  inputs:
   - IT-CERT-PA_urls
  output: true
  prototype: stdlib.aggregatorURL 
 aggregator-IP:
  inputs:
   - IT-CERT-PA_listip
  output: true
  prototype: stdlib.aggregatorIPv4Inbound
 aggregator-DOMAIN:
  inputs:
   - IT-CERT-PA_domains
  output: true
  prototype: stdlib.aggregatorDomain
 4SPLUNK:
  inputs:
   - aggregator-DOMAIN
   - aggregator-IP
   - aggregator-URL
  output: false
  prototype: stdlib.feedHCWithValue
 IT-CERT-PA_domains:
  inputs: []
  output: true
  prototype: itcertpa.DOMAINS
 IT-CERT-PA_listip:
  inputs: []
  output: true
  prototype: itcertpa.IP
 IT-CERT-PA_urls:
  inputs: []
  output: true
  prototype: itcertpa.URLS

The result is the following (remember to commit the changes with the top-left button).

Few minutes later the miners start getting data in (nodes page).

Referring to the above picture:

1. Miners (green) get data from CERT-PA InfoSec feeds.
   All the miners set a share_level green and an high confidence (80) to the collected indicators and discard the comments with a regexp (ignore_regex: ‘^#.*’). They also set an age_out policy for the indicators; when they are removed from the feeds they disappear from the output of the miner.
   Follow details:
   • IT-CERT-PA_domains: this miner uses new itcertpa.DOMAINS prototype and allow all the rest (regex: ‘^.*’). Gets 144 indicators;
   • IT-CERT-PA_listip: this miner uses new itcertpa.IP prototype and allow only IP addresses (regex:’^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$’). Gets 52 indicators;
   • IT-CERT-PA_urls: this miner uses new itcertpa.URLS prototype and allow only http/s url (regex:’^http.*’). Gets 161 indicators;
2. Miners output is sent to the processors (red).
   In MineMeld each processor can handle just one type of indicator (IP, URL, domains, MD5 etc) so each miner is linked to the relevant processor (note that miners instead can handle multiple indicators). The processors, in a more complex scenario, aggregate IoC from different sources, deduplicate data received and age-out old entries; in this case the output nodes have the same indicators of the miners.
   Follow details:
   • aggregator-DOMAIN: gets IT-CERT-PA_domains miner in input (uses the standard stdlib.aggregatorURL prototype);
   • aggregator-IP:gets IT-CERT-PA_listip miner in input (uses the standard stdlib.aggregatorIPv4Inbound prototype);
   • aggregator-URL:gets IT-CERT-PA_domains miner in input (uses the standard stdlib.aggregatorDomain prototype);
3. Processors output is sent to the output node (yellow).
   All the processors (aggregator-DOMAIN, aggregator-IP, aggregator-URL) are linked to the 4SPLUNK output node. This is the core of integration with i-SOC near-real-time engine; the output node gets all the indicators (357 in this case) from the processors (remember in a more complex scenario IoC are aged-out and deduplicated). We just need to take them 🙂
   4SPLUNK output node uses stdlib.feedHCWithValue that is a standard prototype for High Confidence indicators (confidence > 75) that support enhanced output; tks again to Luigi Mori ( check this commit) this prototype allow me to consume the output directly in SPLUNK (more details later).

So at the end I have a list of indicators that can be accessed with a simple HTTP query to the output node 4SPLUNK.

The output is an aggregated list of domains, IPs (in a range format) and URLs. Useless for me in this format.

To use the data in the custom i-SOC application built on top of SPLUNK engine (see my italian talk at HackInBo for more details) I need to get the output in CSV format.
Easy. Just append some option 😉

$ curl -k https://192.168.X.X/feeds/4SPLUNK?v=csv&f=type&f=indicator&tr=1

• v=csv : export in CSV format;
• tr=1 : don’t use a range format for IP addresses;
• f=type&f=indicator : export both type and indicator fields (title in the first line of CSV file).

The output now looks like this, the threat intelligence feeds are ready to be consumed by i-SOC SPLUNK application 🙂

type,indicator
[...]
domain,05.microsoftcloudserver.com
[...]
IPv4,91.219.236.222
[...]
URL,http://185.22.64.46//01/index.php
[...]

On the SPLUNK search head server, where i-SOC application lives, data collected from navigation proxies “goes” into the accelarated Web data model (CIM compliant).
I scheduled following saved search (each 10 minutes) that check if there is any match with the indicators collected from MineMeld.

| tstats `summariesonly` max(_time) as _time,values(Web.status) as status from datamodel=Web by Web.action,Web.dest,Web.dst_ip,Web.url,Web.user,Web.src | `drop_dm_object_name("Web")`

| search 
[|inputlookup soc_ti.csv | search type="IPv4" | rename indicator AS dst_ip | fields dst_ip] OR 
[|inputlookup soc_ti.csv | search type="domain" | rename indicator AS dest | fields dest] OR 
[|inputlookup soc_ti.csv | search type="URL" | rename indicator AS url | fields url ]

| stats count by user,src,dest,url,dst_ip,action

The saved search:

• search into the accelerated SPLUNK Web data model (tstats) selecting relevant fields: action, dest, dst_ip, url, user, src;
• read the CSV file soc_ti.csv and search (OR) for hits into the 3 indicator types:
  • type=”IPV4″: rename the CSV field indicator into SPLUNK Web data model field dst_ip (destination IP address);
  • type=”domain”: rename the CSV field indicator into SPLUNK Web data model field dest (destination domain or IP address);
  • type=”URL”: rename the CSV field indicator into SPLUNK Web data model field url (requested URL).

The CSV file soc_ti.csv is written by a cron job (each 10 minutes); this job gets data from MineMeld and move it to SPLUNK i-SOC application dir /opt/splunk/splunk-sw/etc/apps/soc/lookups/.

#!/bin/bash
cd /tmp/
if [ ! -d /tmp/minemeld_appo/ ]; then
 mkdir minemeld_appo
fi
cd minemeld_appo
rm -f soc_ti.csv
/usr/bin/wget -O soc_ti.csv 'https://192.168.X.X/feeds/4SPLUNK?v=csv&f=type&f=indicator&tr=1' --no-check-certificate --timeout=2 --tries=5
if [ $? -eq 0 ];
then
 mv soc_ti.csv /opt/splunk/splunk-sw/etc/apps/soc/lookups/
else
 echo "ERROR"
 exit 1
fi
exit 0

Last but not least when an hit is found the SPLUNK i-SOC saved search runs a script (soc_incident_manager.py) that:

• create a Security Event on the i-SOC analysts dashboard;
• create a Ticket in i-SOC OTRS ticketing platform;
• make a drill-down link to help analysts to understand what happened asap.

Automation done 🙂