MineMeld: threat intelligence automation – export internal IoC to the community [3]

This post is the third of a series on Threat Intelligence Automation topic.
Post 1: Architecture and Hardening of MineMeld
Post 2: Foundation: write a custom prototype and SOC integration
Post 4: Search received IoC events with Splunk
Post 5: Connect to a TAXII service

After building the architecture and integrating the InfoSec feeds from italian CERT-PA into MineMeld and the near-real-time SOC engine, it’s time to put another brick to build an effective community: export internal IoC to the community in a standard format so authorized parties can get it and use them as they want.

STIX/TAXII Network
STIX/TAXII Network

The ultimate goal is to build a community that can share IoC using a standard language and a transport mechanism (STIX/TAXXI) getting data from heterogeneous sources (more integration examples in next posts) and injecting data into the community network.

So let’s start with the configuration steps.

Following picture shows an example of a target community network where:

  • there is a central hub that sends IoC to consumers and receives IoC from producers (spoke). This is typically done by a a CERT that – hopefully 🙂 – does analysis and contextualization before sending and after receiving IoC;
  • each peripheral entity (spoke) can also build peer-to-peer connections with other parties to directly share specific IoC.
Target community network example
Target community network example

In this post I show you how to become a producer exporting URL, IP and domains in STIX/TAXII format; the reference network architecture is described in post 1.

Reference architecture
Reference architecture

From MineMeld CONFIG page, IMPORT the following config (remember to COMMIT at the end):

  • press APPEND if you have a working config already in place, like the config made on post 2. Remember to do a BACKUP before, just in case;
  • press REPLACE if your current config can be erased.
nodes:
 TI-url-export:
  inputs: []
  output: true
  prototype: stdlib.listURLGeneric
 TI-domain-export:
  inputs: []
  output: true
  prototype: stdlib.listDomainGeneric
 TI-ipv4-export:
  inputs: []
  output: true
  prototype: stdlib.listIPv4Generic
 TAXII-output:
  inputs:
   - TI-url-export
   - TI-domain-export
   - TI-ipv4-export 
  output: false
  prototype: stdlib.taxiiDataFeed

At the end you have

TAXII export
TAXII export
  • TI-url-export
    The miner where we add URL (uses stdlib.listURLGeneric prototype);
  • TI-domain-export
    The miner where we add domains (uses stdlib.listDomainGeneric prototype);
  • TI-ipv4-export
    The miner where we add IP addresses (uses stdlib.listIPv4Generic prototype);
  • TAXII-output
    The output node that exports data received from the 3 miners in STIX/TAXII format (stdlib.taxiiDataFeed prototype). This is what I called the producer.

And this is the connection graph.

TAXII export connection graph
TAXII export connection graph

Now it’s time to fill the miners with some IoC.
This can be done from NODES page, clicking on each miner (TI-*-export), then on the INDICATORS icon (left) and then filling the fields (next image is for domain indicators).

Add indicators
Add indicators

After that the output node TAXII-output has the 3 indicators added.

Indicators added to TAXII output
Indicators added to TAXII output

The TAXII node is now ready to be queried from a TAXII client using the DISCOVERY SERVICE url: https://192.168.X.X/taxii-discovery-service 

TAXII output discovery service
TAXII output discovery service

To test the TAXII output we can use Cabby that is a simple python TAXII client implemetation. Here the install guide, just install virtualenv before (on Ubuntu: $ sudo apt-get isntall virtualenv).

I tested some command line tools as follow:

// note: "--verify no" option added because in test lab I have a self-signed certificate
// service discovery
(venv) gmellini@SandTigerShark:~$ taxii-discovery --host 192.168.X.X --path /taxii-discovery-service --https --verify no
2017-08-30 16:37:49,265 INFO: Sending Discovery_Request to https://192.168.X.X/taxii-discovery-service
2017-08-30 16:37:49,269 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
 InsecureRequestWarning)

2017-08-30 16:37:49,289 INFO: 3 services discovered
=== Service Instance ===
 Service Type: DISCOVERY
 Service Version: urn:taxii.mitre.org:services:1.1
 Protocol Binding: urn:taxii.mitre.org:protocol:http:1.0
 Service Address: https://192.168.X.X/taxii-discovery-service
 Message Binding: urn:taxii.mitre.org:message:xml:1.1
 Available: True
 Message: None

=== Service Instance ===
 Service Type: COLLECTION_MANAGEMENT
 Service Version: urn:taxii.mitre.org:services:1.1
 Protocol Binding: urn:taxii.mitre.org:protocol:http:1.0
 Service Address: https://192.168.X.X/taxii-collection-management-service
 Message Binding: urn:taxii.mitre.org:message:xml:1.1
 Available: True
 Message: None

=== Service Instance ===
 Service Type: POLL
 Service Version: urn:taxii.mitre.org:services:1.1
 Protocol Binding: urn:taxii.mitre.org:protocol:http:1.0
 Service Address: https://192.168.X.X/taxii-poll-service
 Message Binding: urn:taxii.mitre.org:message:xml:1.1
 Available: True
 Message: None

// get collection list
(venv) gmellini@SandTigerShark:~$ taxii-collections --path https://192.168.X.X/taxii-collection-management-service --verify no
2017-08-30 16:53:46,540 INFO: Sending Collection_Information_Request to https://192.168.X.X/taxii-collection-management-service
2017-08-30 16:53:46,544 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

=== Data Collection Information ===
  Collection Name: TAXII-output
  Collection Type: DATA_FEED
  Available: True
  Collection Description: TAXII-output Data Feed
  Supported Content:   urn:stix.mitre.org:xml:1.1.1
  === Polling Service Instance ===
    Poll Protocol: urn:taxii.mitre.org:protocol:http:1.0
    Poll Address: https://192.168.X.X/taxii-poll-service
    Message Binding: urn:taxii.mitre.org:message:xml:1.1
==================================

// poll discovered collections
(venv) gmellini@SandTigerShark:~$ taxii-poll --host 192.168.X.X --https --verify no --collection TAXII-output --discovery /taxii-discovery-service
2017-08-30 16:59:38,174 INFO: Polling using data binding: ALL
2017-08-30 16:59:38,175 INFO: Sending Discovery_Request to https://192.168.X.X/taxii-discovery-service
2017-08-30 16:59:38,179 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

2017-08-30 16:59:38,200 INFO: 3 services discovered
2017-08-30 16:59:38,200 INFO: Sending Poll_Request to https://192.168.X.X/taxii-poll-service
2017-08-30 16:59:38,203 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

<stix:STIX_Package xmlns:DomainNameObj="http://cybox.mitre.org/objects#DomainNameObject-1" xmlns:cybox="http://cybox.mitre.org/cybox-2" xmlns:cyboxCommon="http://cybox.mitre.org/common-2" xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2" xmlns:example="http://example.com" xmlns:indicator="http://stix.mitre.org/Indicator-2" xmlns:minemeld="https://go.paloaltonetworks.com/minemeld" xmlns:stix="http://stix.mitre.org/stix-1" xmlns:stixCommon="http://stix.mitre.org/common-1" xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:taxii="http://taxii.mitre.org/messages/taxii_xml_binding-1" xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1" xmlns:tdq="http://taxii.mitre.org/query/taxii_default_query-1" id="minemeld:indicator-d196ce3b-d27c-4176-a131-5111c7024411" version="1.1.1" timestamp="2017-08-30T13:53:38.193519+00:00">
 <stix:Indicators>
  <stix:Indicator id="minemeld:indicator-9dcf3ce3-305b-46c0-93db-88f9d937c69f" timestamp="2017-08-30T13:53:38.193821+00:00" xsi:type="indicator:IndicatorType">
   <indicator:Title>domain: wwwww.brokendomain.broken</indicator:Title>
   <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">URL Watchlist</indicator:Type>
   <indicator:Description>domain indicator from TI-domain-export</indicator:Description>
   <indicator:Observable id="minemeld:observable-8d6a2557-0831-4950-9092-7473bcd05010">
    <cybox:Title>FQDN: wwwww.brokendomain.broken</cybox:Title>
    <cybox:Object id="minemeld:DomainName-960ffcd7-e9d5-4b16-bf02-ecfa6379b5b4">
     <cybox:Properties xsi:type="DomainNameObj:DomainNameObjectType" type="FQDN">
      <DomainNameObj:Value>wwwww.brokendomain.broken</DomainNameObj:Value>
     </cybox:Properties>
    </cybox:Object>
   </indicator:Observable>
   <indicator:Confidence timestamp="2017-08-30T13:53:38.194039+00:00">
    <stixCommon:Value xsi:type="stixVocabs:HighMediumLowVocab-1.0">High</stixCommon:Value>
   </indicator:Confidence>
  </stix:Indicator>
 </stix:Indicators>
</stix:STIX_Package>
<stix:STIX_Package xmlns:AddressObj="http://cybox.mitre.org/objects#AddressObject-2" xmlns:cybox="http://cybox.mitre.org/cybox-2" xmlns:cyboxCommon="http://cybox.mitre.org/common-2" xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2" xmlns:example="http://example.com" xmlns:indicator="http://stix.mitre.org/Indicator-2" xmlns:minemeld="https://go.paloaltonetworks.com/minemeld" xmlns:stix="http://stix.mitre.org/stix-1" xmlns:stixCommon="http://stix.mitre.org/common-1" xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:taxii="http://taxii.mitre.org/messages/taxii_xml_binding-1" xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1" xmlns:tdq="http://taxii.mitre.org/query/taxii_default_query-1" id="minemeld:indicator-82cf103a-3bcf-4dc0-af60-c35c051b9ce8" version="1.1.1" timestamp="2017-08-30T13:54:28.029792+00:00">
 <stix:Indicators>
  <stix:Indicator id="minemeld:indicator-05a1fdff-35b0-4c12-8a23-2e7e74d96cd1" timestamp="2017-08-30T13:54:28.030074+00:00" xsi:type="indicator:IndicatorType">
   <indicator:Title>IPv4: 1.1.1.1/32</indicator:Title>
   <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">IP Watchlist</indicator:Type>
   <indicator:Description>IPv4 indicator from TI-ipv4-export</indicator:Description>
   <indicator:Observable id="minemeld:observable-f77e3d4d-62f6-4134-84b4-f2519338f672">
    <cybox:Title>IPv4: 1.1.1.1/32</cybox:Title>
    <cybox:Object id="minemeld:Address-dbda63c8-0b0e-4c83-abf8-8fd510c4fa6b">
     <cybox:Properties xsi:type="AddressObj:AddressObjectType" category="ipv4-addr">
      <AddressObj:Address_Value>1.1.1.1/32</AddressObj:Address_Value>
     </cybox:Properties>
    </cybox:Object>
   </indicator:Observable>
   <indicator:Confidence timestamp="2017-08-30T13:54:28.030253+00:00">
    <stixCommon:Value xsi:type="stixVocabs:HighMediumLowVocab-1.0">High</stixCommon:Value>
   </indicator:Confidence>
  </stix:Indicator>
 </stix:Indicators>
</stix:STIX_Package>
<stix:STIX_Package xmlns:URIObj="http://cybox.mitre.org/objects#URIObject-2" xmlns:cybox="http://cybox.mitre.org/cybox-2" xmlns:cyboxCommon="http://cybox.mitre.org/common-2" xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2" xmlns:example="http://example.com" xmlns:indicator="http://stix.mitre.org/Indicator-2" xmlns:minemeld="https://go.paloaltonetworks.com/minemeld" xmlns:stix="http://stix.mitre.org/stix-1" xmlns:stixCommon="http://stix.mitre.org/common-1" xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:taxii="http://taxii.mitre.org/messages/taxii_xml_binding-1" xmlns:taxii_11="http://taxii.mitre.org/messages/taxii_xml_binding-1.1" xmlns:tdq="http://taxii.mitre.org/query/taxii_default_query-1" id="minemeld:indicator-c58d05b6-55cc-4071-9446-9d7b38613bc6" version="1.1.1" timestamp="2017-08-30T13:56:16.463180+00:00">
 <stix:Indicators>
  <stix:Indicator id="minemeld:indicator-6ba0e59f-97e5-467e-a5de-c42353a53e87" timestamp="2017-08-30T13:56:16.463434+00:00" xsi:type="indicator:IndicatorType">
   <indicator:Title>URL: https://wwww.brokensite.com/compromised.php</indicator:Title>
   <indicator:Type xsi:type="stixVocabs:IndicatorTypeVocab-1.1">Domain Watchlist</indicator:Type>
   <indicator:Description>URL indicator from TI-url-export</indicator:Description>
   <indicator:Observable id="minemeld:observable-de9352e3-8b56-4ec4-ba79-508c14d702ab">
    <cybox:Title>URL: https://wwww.brokensite.com/compromised.php</cybox:Title>
    <cybox:Object id="minemeld:URI-82dd5db9-51fe-4c7b-86ff-e9c74b022aea">
    <cybox:Properties xsi:type="URIObj:URIObjectType" type="URL">
     <URIObj:Value>https://wwww.brokensite.com/compromised.php</URIObj:Value>
    </cybox:Properties>
    </cybox:Object>
   </indicator:Observable>
   <indicator:Confidence timestamp="2017-08-30T13:56:16.463591+00:00">
    <stixCommon:Value xsi:type="stixVocabs:HighMediumLowVocab-1.0">High</stixCommon:Value>
   </indicator:Confidence>
  </stix:Indicator>
 </stix:Indicators>
</stix:STIX_Package>

Cabby gets the 3 indicators (STIX) from TAXII-output collection. It works 🙂

The TAXII output is now available to anyone that knows the discovery URL.
Of course we can restrict the access only to authorized users in MineMeld.

To do this first create a new feed user from ADMIN – FEED USERS page (guest/guest) then an access tag for the user (public_taxii). This is the tag to be “attached” to the output feed to set authentication.

Feed user creation
Feed user creation

There is a warning; feed authentication must be explicitly enabled with following command (to see the new config just refresh the web interface with CTRL+F5).

$ sudo -u minemeld sh -c 'echo "FEEDS_AUTH_ENABLED: True" > /opt/minemeld/local/config/api/30-feeds-auth.yml'

Now from CONFIG page click on TAXII-output node and set public_taxii tag.

TAXII feed autentication
TAXII feed autentication

The non-authenticated Cabby service discovery command return an authorization error.

(venv) gmellini@SandTigerShark:~$ taxii-poll --host 192.168.X.X --https --verify no --collection TAXII-output --discovery /taxii-discovery-service
2017-08-30 17:47:36,842 INFO: Polling using data binding: ALL
2017-08-30 17:47:36,843 INFO: Sending Discovery_Request to https://192.168.X.X/taxii-discovery-service
2017-08-30 17:47:36,847 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

2017-08-30 17:47:36,866 ERROR: Can not autodiscover advertised services
2017-08-30 17:47:36,866 ERROR: HTTP Error: status code 401
(venv) gmellini@SandTigerShark:~$

Trying with guest credential works fine.

(venv) gmellini@SandTigerShark:~$ taxii-poll --host 192.168.X.X --https --verify no --collection TAXII-output --discovery /taxii-discovery-service --username guest --password guest
2017-08-30 17:49:59,900 INFO: Polling using data binding: ALL
2017-08-30 17:49:59,901 INFO: Sending Discovery_Request to https://192.168.X.X/taxii-discovery-service
2017-08-30 17:49:59,905 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

2017-08-30 17:49:59,925 INFO: 3 services discovered
2017-08-30 17:49:59,925 INFO: Sending Poll_Request to https://192.168.X.X/taxii-poll-service
2017-08-30 17:49:59,929 WARNING: /home/gmellini/venv/local/lib/python2.7/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

<stix:STIX_Package [...]

Last but not least when you enable feed authentication all your existing non-authenticated feeds stop working; you need to tell MineMeld that these feeds are anonymous feed adding anonymous tag of course; this is a default tag (you can see in the tag list).

Do this if you have an existing configuration or if you have in place the config from post 2; in this case 4SPLUNK output feed must look like this.

Anonymous feed settings
Anonymous feed settings

Enjoy!

4 thoughts on “MineMeld: threat intelligence automation – export internal IoC to the community [3]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s