Unix socket madness with Trisul and IDS alerts

One of the things Trisul can do is to merge rich traffic analytics data with traditional IDS alerts from systems like Snort and Suricata. Up until Trisul 5.5 you could connect Trisul into a single Unix Socket to which Snort or Barynard2 would write alerts. Once alerts were inside Trisul they would be merged with other types of NSM data in real time. You could then ask queries like “Show me list of flows that produced a HIGH priority alert”

Why Unix Sockets?

Unix sockets are old school, yet they are used in a suprisingly large number of systems even today for IPC (Interprocess communications). There are basically two ways you can read alerts from a logging system like Suricata.

  1. Traditional Log Tailing Observe an output log file and process any new alerts.
  2. Network messaging Just read messages off a network feed. This can be unix sockets, TCP/UDP sockets or higher abstractions like message queues.

The advantages of Unix Sockets we see are

  1. Secure – purely local, unlike TCP/UDP sockets
  2. Message based – Unix socket that use SOCK_DGRAM preserve message boundaries. You can simply read the messages as sent
  3. chmod – can be secured using traditional linux chmod style permissions. For network sockets you need to use ACLs or other mechanisms like CURVE or TLS
  4. No waldo – A most loved feature of IDS log tail is the so called waldo file. You need to know where to position yourself on the log file when you restart the system.

So there! We like Unix Sockets and try to use it even for shiny new output formats like Suricata’s EVE JSON output format.

Let us now see some code that shows you how you can use Trisul’s new LuaJIT API to interface with Unix Sockets.

Trisul 6.0 Platform

Trisul 6.0 is now positioned as a platform that leverages the power of LuaJIT to provide plug-in points into our processing pipeline. Now you can write an ‘inputfilter’ LUA script to listen to ANY type of feed that would result in an arbitrary alert.

New scripts on github

We just released four scripts on Github Trisul-Scripts

  1. suricata_eve.lua (github) – EVEExtensible Event Format is a modern output format that uses JSON. If you havent used Suricata’s EVE output format you really should check it out.
  2. suricata_eve_unixsocket.lua – Using a bit of LuaJIT FFI read EVE from a Unix Socket
  3. snort_unixsocket.lua – Traditional “-A unsock” output format from snort
  4. barnyard2_unixsocket.lua – Unified2 binary format from Barnyard2.

The input filter framework

Input filters are LUA scripts you write that can drive trisul. You can write custom PCAP readers, alert monitors, or custom flow-like records and use that as the input. In the illustration below you can see how multiple scripts can listen on different inputs. The Trisul platform will take care of the orchestration, threading, preventing starvation etc.

Using Unix Sockets. LuaJIT FFI to the rescue

Sometimes LuaJIT feels like cheating because you can drop down to standard C at anytime and call linux APIs and use C structures. If you are already familiar with LuaJIT FFI all of this will be familiar to you. Others can read the LUA code to understand how this C-LuaJIT interface works. Once you have setup the LuaJIT “cdefs”, read the JSON message from the socket, all you have to do is to map the EVE message to a Trisul alert format.

Lets take a look at how we map the EVE format.

From eve_unixsocket.lua

The real action happens in the step_alert function. You need to map the EVE types to a LUA table that Trisul understands as per the inputfilter API



  -- setup unixsocket using FFI

  -- read JSON EVE

  -- filter out non 'alert'  types

  -- map to input filter

  local ret =  {

      AlertGroupGUID='{9AFD8C08-07EB-47E0-BF05-28B4A7AE8DC9}',     -- Trisul alert group = External IDS 
      TimestampSecs = tv_sec,                                      -- Epoch based time stamps
      TimestampUsecs = tv_usec,
      SigIDKey = p.alert["signature_id"],                          -- SigIDKey is mandatory 
      SigIDLabel = p.alert["signature"],                           -- User Label for the above SigIDKey 
      SourceIP = p["src_ip"],                                      -- IP and Port pretty direct mappings
      SourcePort = p["src_port"],
      DestIP = p["dest_ip"],
      DestPort = p["dest_port"],
      Protocol = protocol_num(p["proto"]),                         -- convert TCP to 6 
      SigRev = p.alert["rev"],
      Priority = p.alert["severity"],
      ClassificationKey = p.alert["category"],
      AlertStatus=p.alert["action"],                                -- allowed/blocked like ALARM/CLEAR
      AlertDetails=p.alert["signature"]                             -- why waste a text field 'AlertDetails'?
  };


Most of the fields are straight up one-to-one mapping. I found the AlertStatus and EVE alert.action to be quite interesting. Apparently Suricata uses that field in IPS mode to indicate whether the event resulted in a BLOCK or ALLOW. By mapping it into Trisul AlertStatus you get a neat integration that result in something like below.

Notice the ‘allowed’ tag.

Explore the API

Over the next few days we will be writing about the other new LuaJIT APIs in Trisul. One such API is the File Extraction API.

Install Trisul now, no sign up needed

Installing Trisul Network Analytics 6.0 is painless. There are no sign ups. It is as simple as apt-get install or yum install Head over to the Downloads area to get started. The APIs are included as part of the base package.

Until next time.

Free Download Trisul 6.0 !

Trisul 5.5. updates fixes Netflow v9 and SFlow issues

New builds of Trisul Network Analytics 5.5 are now available.

Summary

They fix the following issues.

  • Netflow v9 when used with large flow-cache timeout not reporting correct duration of flows
  • When you Enable Netflow v9 Ingress and Egress on the same interface ip flow ingress and ip flow egress and you have the MergeMultipleSource option enabled in the Netflow Configuration File you may notice that only the egress traffic is counted and the ingress is ignored.
  • When Trisul processes SFlow from multiple IPs on the same devices, the traffic may be incorrectly counted.

If you are using Trisul in Netflow v9 (Ingress+Egress) mode or are using SFlow you are encouraged to update immediately. Free Download of Ubuntu 14.04 LTS and CentOS7

Big new Release 6.0 coming

A quick heads up. We have a big new release of Trisul 6.0 coming up in a couple of weeks. In 6.0 you can use Trisul in a distributed mode, write your own application logic using LUA, and even roll out a probe + cloud deployment mode.

Free Download Trisul 6.0
Trisul Network Analytics 6.0 is now available – Enjoy!

Threshold Band alerts HOWTO screencast

Trisul Network Analytics 5.5 introduced an extremely nifty feature called Threshold Band Alerting . We’ve talked about that feature in this blog post and in the documentation.

We find that some of our customers are not using the full power of the feature. The key thing to remember is that you can create bands on ANY traffic metric not just total bandwidth. One of our most useful bands in action at a customer site is one we’ve created on “Active Flows”. This has helped catch a number of suspect activity that were invisible if you just looked at total bandwidth.

Here is are two simple screencasts that shows you how to create bands of your own.

Screencast 1 : Create a new band

You can click on any key or inside the charts on the legend to go to the Key Dashboard.

The first screencast shows

  1. Clicking on a key to navigate to the key dashboard
  2. In this example – we are creating a band for INBOUND traffic
  3. Then clicking on Create Threshold Band and going through the form
  4. Checking the Compare Day box – which only compares Wednesdays with other Wednesdays to create a profile
  5. Using Number of Samples = 5 – which needs about 5 weeks of training data
Note If using the FREE license you may not be able to use the Compare Days feature because you only have 3 days of training data.

Click to play/pause

Screencast 2 : Viewing alerts and bands

Shows you how to view alerts and also to see the internals of how bands are calculated, how our algorithm excludes outliers at a data-series level as well as at an interval level.

The screencast shows

  1. Sample of how alerts are shown on the charts
  2. Clicking on a Alerts > Threshold Band to go to the alerts page
  3. The next page shows alerts that have fired, clicking on “Configure”
  4. Clicking on “View All Days” to see all band along with the training data

Click to play/pause

Get New Builds

We just uploaded new builds of Trisul 5.5. We encourage you to update.

We will also be releasing Trisul 6.0 a major new release in the next few weeks. Stay tuned and share your experiences with us.

Free Download Trisul 5.5
Download Trisul 5.5 today Spend your time on interesting problems not fighting tools.

Threshold Band alerts and other exciting features released in Trisul Network Analytics 5.5

We are happy to announce a new release of Trisul Network Analytics with exciting new features that make it even more effective in deep streaming analytics of network traffic.

Threshold Band alerting [ docs ]

We sat down with a few of our customers with the most sensitive networks and observed how various Traffic Metrics were responding to real life situations. This helped us develop a simple yet effective “Threshold Band” feature that we are releasing now. Essentially Trisul now lets you attach ‘threshold bands’ to any metric. Once you attach a band, Trisul automatically calculates an upper and lower bound for that metric for each time-of-day and day-of-week combination. The concept itself is old, but we have made it really easy to attach any metric to a band, which is automatically recomputed every night.


Actual metric shown against gray band of expected values

Currently we are using this to alert on statistical movement of a number of metrics such as

  • total bandwidth
  • concurrent flows : helped detect a DoS situation
  • netflow volumes
  • volumes of top 3 customers with revenue impact : if traffic below lower bound results in $ loss for them
  • number of flows being setup up / sec
  • number of flows terminated / sec

You can track any of the hundreds of thousands of metrics this way. In our next maintenance release, we are rolling out a
Read more about this feature in our Threshold Band Anomaly alert docs

Self monitoring

A new dashboard called Perf Stats now tracks metrics about Trisul’s own memory, CPU, disk usage. Quite surprisingly spikes in these third order metrics have led us to interesting findings.


Self monitoring CPU, Disk, and memory usage patterns of Trisul process

Active Keys Monitor

A new dashboard keeps track of how many unique keys (entities) are being monitored in each counter group. The use cases are to get a really good visibility of ‘unique things’ in your network from various viewpoints. Trisul can also track Unique X or Y using Cardinality counters. See the post on Hyperloglog counters


Trends of unique items being tracked in every counter group

New historical chart use interface

Now you can click on the ‘chart’ icon on any module to expand the chart to days or weeks.


Clicking on the highlighted icon brings up the explorer window as shown below


Zoom into any metrics without losing context

p.

Other changes

On the backend there are several exciting changes

  • LUA Scripting functionality has been added to counter group items
  • Automatic recovery of crashed databases in rare cases of power failure
  • New Netflow dashboards
  • New License page
  • Many UI fixes such as the ability to add any item to the menu in a single click
Free Download Trisul 5.5
Download Trisul 5.5 today Spend your time on interesting problems not fighting tools.