MineMeld: threat intelligence automation – architecture and hardening [1]

This post is the first of a series on Threat Intelligence Automation topic
Post 2: Foundation: write a custom prototype and SOC integration
Post 3: Export internal IoC to the community
Post 4: Search received IoC events with Splunk
Post 5: Connect to a TAXII service

Last slide at my HackInBo talk (italian) was about how to automatically integrate threat intelligence feeds into our near-real-time Information Security Operation Center (i-SOC) SPLUNK engine to reduce the time spent by SOC security analysts on IoC (Indicators of Compromise) analysis.

At the time I was testing an open source project from PaloAlto: MineMeld. It was the right choice; after extensive tests MineMeld now help me to solve the challenges I had in the past while playing with IoC coming from various threat intelligence sources: collection automation, unduplication, aging and SOC integration.

MineMeld can also share our internal IoC to the italian infosec community we are now building from the ground. We are working hard on this and I’m really confident we will succeed (want to join? DM me on twitter or in the comments).

This first post details the installation steps and the architecture design I implemented to:

• collect feeds from external sources: this is done in MineMeld with the miner nodes;
• make available the feeds to trusted sources (internal and external): this is done in MineMeld with the ouput nodes. Output nodes export feeds (csv, text, TAXII etc) via HTTPS URLs;
• put data collected into our SOC engine: get automatically the feeds from i-SOC SPLUNK application in a usable format (csv in my case) to inject IoC in our real-time engine without human processing.

I suggest to read more here to understand how MineMeld works. Basically IoC are received (and tagged) from heterogeneous sources (miner nodes) and then processed to remove duplicates and old entries (processor nodes); at the end IoC are available(output nodes) in various formats (json, text, csv) to authorized clients.

The following picture shows the high-level design to comply to the above requirements (in green the miners, in yellow the output nodes).

I installed MineMeld following the Ubuntu 14.04 community guide; in few minutes I had the server up&running. The guide contains some hardening step but in my scenario it was not enough: I need to access MineMeld web interface only from my internal SOC network and – at the same time – export IoC to authorized users on Internet through a trusted web site (HTTPS).

By default MineMeld uses nginx as a reverse proxy to forward all the request coming to port tcp/443 (HTTPS) to the application server listening on local address 127.0.01:5000 (Gunicorn web server). Gunicorn reply both to admin interface and output feeds requests; this means that if you configure a static NAT to expose HTTPS service on Internet, everyone has access to admin interface. Not good at all.

So I modified nginx MineMeld configuration file /etc/nginx/sites-enabled/minemeld-web adding a new server section as follow. Please note in red my comments to better understand the configuration.

[ 15 Aug 2017: more enforcing on the config as suggested by iThreatHunt; disabled TLSv1 and 3DES cypher suite ]

server {
 # new listening interface on port tcp/10443
 # this port receive external Internet requests from the firewall
 # that does port forwarding from tcp/443 to tcp/10443
 listen 10443 ssl;

 server_name <PUBLIC FQDN>;

 # Your PUBLIC FQDN certifcate files here
 ssl_certificate /etc/nginx/ssl/public.crt;
 ssl_certificate_key /etc/nginx/ssl/private.rsa;
 ssl_trusted_certificate /etc/nginx/ssl/intermediate.pem;

 # HARDENING STEPS FOR HTTPS
 # https://weakdh.org/sysadmin.html
 # https://live.paloaltonetworks.com/t5/MineMeld-Discussions/HOWTO-MineMeld-Threat-Intelligence-Automation-architecture-and/m-p/171573
 ssl_protocols TLSv1.1 TLSv1.2;
 ssl_prefer_server_ciphers on;
 ssl_session_cache shared:SSL:10m;
 ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK';
 ssl_dhparam /etc/nginx/ssl/dhparams.pem;

 keepalive_timeout 5;

 # Custom errror pages for anyone trying to access not allowed web pages
 # error files stored in dir: /usr/share/nginx/html/
 error_page 404 /custom_404.html;
  location = /custom_404.html {
  root /usr/share/nginx/html;
  internal;
 }
 
 error_page 500 502 503 504 /custom_50x.html;
  location = /custom_50x.html {
  root /usr/share/nginx/html;
  internal;
 }

 # This section allow Internet access to text feeds (csv, txt, json)
 # Remote client can access it via this URL schema
 #  https://<FQDN>/feeds/<MINEMELD OUTPUT FEED NAME>
 location /feeds/ {
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_set_header Connection '';
  proxy_http_version 1.1;
  proxy_redirect off;
  proxy_buffering off;
  chunked_transfer_encoding off;
  proxy_cache off;
  proxy_read_timeout 120s;

  expires -1;

  # Forward /feeds/ requests to the application server
  proxy_pass http://app_server; 
 }
 
 # This section allow Internet access to STIX/TAXII feeds
 # Remote client can access it via TAXII service discovery service
 #  https://<FQDN>/taxii-discovery-service
 # We need to enable 3 URLs
 location /taxii-discovery-service {
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_set_header Connection '';
  proxy_http_version 1.1;
  proxy_redirect off;
  proxy_buffering off;
  chunked_transfer_encoding off;
  proxy_cache off;
  proxy_read_timeout 120s;
 
  expires -1;
  
  # Forward /taxii-discovery-services requests to the application server
  proxy_pass http://app_server; 
 }

 location /taxii-collection-management-service {
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_set_header Connection '';
  proxy_http_version 1.1;
  proxy_redirect off;
  proxy_buffering off;
  chunked_transfer_encoding off;
  proxy_cache off;
  proxy_read_timeout 120s;
 
  expires -1;
 
  # Forward/taxii-collection-management-services requests to the application server
  proxy_pass http://app_server; 
 }

 location /taxii-poll-service {
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_set_header Connection '';
  proxy_http_version 1.1;
  proxy_redirect off;
  proxy_buffering off;
  chunked_transfer_encoding off;
  proxy_cache off;
  proxy_read_timeout 120s;

  expires -1;

  # Forward /taxii-poll-services requests to the application server
  proxy_pass http://app_server; 
 }
 # End of STIX/TAXII feeds section
 
 # Error 404 if you are not doing the right request :)
 location / {
  return 404;
 }
}

So at the end the external firewall does NAT & Port Forwarding and all the requests directed to https[:]//<PUBLIC FQDN>/ are redirect to the DMZ server through the address https[:]//<DMZ IP>:10443

From Internet: only explicity allowed URLs are served (feeds), other requests receive a 404 error message.

From internal: admin interface and feeds are available on standard HTTPS connection and only from authorized clients.

In the next posts I will cover:

• setup of the miners: STIX/TAXXI, MISP, csv etc;
• feeds export in csv format and SPLUNK integration;
• feeds export in TAXII format.

Stay tuned.

Many thanks to Luigi Mori for its continued support.