Trisul Network Analytics
Developer Zone

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Analyzing the WRCCDC PCAP dump using TrisulNSM : Part 1 Approach ======

The good folks at WRCCDC ((Western Regional Collegiate Cyber Defense Competition))  were kind enough to release [[https://archive.wrccdc.org/|packet captures]] (PCAPS) of the recently concluded event. The entire corpus is roughly 1TB.  Now the question is : **What are the tools that can help you unravel the information in the PCAPs?** 

This is Part-1 of a 3 Part series 

  * Part 1: Approach how to avoid getting overwhelmed by large PCAPS 
  * [[offline:wrccdc_pcaps_trisulnsm|Part 2: How to use the free TrisulNSM Docker Image to analyze the PCAP dump]]
  * [[offline:wrccdc_pcaps_results|Part 3: Screenshots & video of possible analysis paths (using TrisulNSM)]]


===== Where to start with giant PCAP dumps =====

The strategy to avoid getting overwhelmed by giant PCAPs is similar to capturing live traffic for the very first time from big networks. You start from knowledge about the organization and then build a baseline analysis. Then you can spread out into different analysis paths depending on what you are looking for. The tooling you have must support this process end to end. 

In this particular case, here is What we know for sure :
  - the PCAPs are published by WRCCDC
  - these are from a cyber defense competition 
  - they are big. Big enough to make packet level tools like Wireshark , NetworkMiner etc impractical as first-line tool 


We like to divvy up the work into two distinct tasksets

  - **Monitoring Tasks** : Look around at higher level trying to spot patterns, gain understanding, watch outliers. Goal here is to gain total visibility and then discover potential **Drilldown tasks**
  - **Drilldown Tasks** : You've already identified something that needs further investigation. You now need tooling to increase resolution on that path alone and complete the investigation. The ideal end point for this in NSM would be to drilldown to the packet level. 


===== Monitoring =====
 
We like to build a baseline understanding of the network from the PCAPs from the following four angles. In that order. 

==== 1: Traffic Analytics ====

First step is to get the "lay of the land". We created TrisulNSM to excel in this area. You can try answering these traffic questions. 

  * What is the duration of the PCAPs? How many packets in the PCAP dump? Bytes? How many Flows ? 
  * What does the overall bandwidth usage chart look like ? 
  * How much of that bandwidth went to external world, how much stayed inside? 
  * What is the size of the network we're looking at? How many unique "Internal IPs" are there?
  * Top Hosts, Applications
  * What is the app profile ? Is it mostly HTTPS, then HTTP, does it resemble a typical enterprise?
  * How many active flows ? How many total flows?
  * What hosts have most connections ?

==== 2: Flow Analytics ====

Once you get a solid understanding of the traffic profiles, next step is to ask basic questions about the flows (( Flows in TrisulNSM are defined as IP Flows that include TCP and UDP flows as well as IP level flows like GRE, IPSEC,etc)) . These yield the second level of understanding. 

  * What are the biggest flows we're seeing by volume of data transfer ? 
  * What are the longest running flows ?
  * Flows uploading data outside vs Flows downloading from outside.
  * Which hosts are responsible for the most number of flows over time?
  * Which apps have the most flows ? 



==== 3: Alerts ====

Next you can look at the same PCAPs from a security angle. You can use Snort or Suricata with all rules enabled and also use a Cyber Crime Intel feed to check your traffic. The TrisulNSM Docker image uses [[https://suricata-ids.org/|Suricata]] with all rules from the [[http://doc.emergingthreats.net/|Emerging Threats Open Ruleset]] to run a 2nd pass over the PCAP dump.  This gives a really good baseline security view from which you can tee off to more detailed exploration. 

  * How many alerts fired over time ? Time chart of alert activity.
  * How many Priority-1 alerts ? 
  * Any alerts fired from [[http://iplists.firehol.org/?ipset=firehol_level1|FireHOL Level-1]] (we like this list for its very low false positive rate). If anything fires here, you MUST look deeper. 

We generally enable all rules while looking at PCAPs because we have luxury of time and CPU. A large rule load can result in packet drops on live networks but doable in offline mode. 

==== 4: More advanced Traffic Analytics ====

Steps 1-3, will give you a rather solid foundation. By this time, you should have atleast a dozen potential starting points to dig deeper. For example : //"Hey, I am seeing a dozen 500MB+ downloads , need to check what it is".//  At this point, we like to dig a little deeper into advanced counter groups that are available in TrisulNSM out of the box. These give excellent medium resolution for you to investigate. Some of the useful counter groups are :

  * **SSL/TLS  counters** :  What kind of certificate activity are you seeing ? Who are the top CAs, TLS Organizations, Ciphers used?  Are there any self signed certificates going around ? 
  * **SNI** : Which are the top-K and bottom-K SNI (Server Name Indication) in TLS traffic you're seeing. 
  * **JA3 Hash** : The JA3 is a TLS client hello finger print. It is a very good soft indicator that can spot common client applications like Browsers. There are some good malware prints too. 
  * **Geo counters** : Traffic and Hits by country. You can open up the Bottom-K lists and explore deeper.
  * **HTTP  counters** :  HTTP Error codes by time. 

Trisul gives you 20 more counter groups, but the above five are good medium resolution starting points.



===== Drilling down  =====

The relevant factor in Drilldown is whether the analyst is looking for something specific. A security oriented analyst may only want to follow certain drilldown paths from certain starting points. A more general threat-hunting analyst might want to drilldown on all possible leads. These techniques are useful

  - **Top-K/Bottom-K** : You can use these to find out heavy hitters and rare items. 
  - **Graph Analytics** : Expand a particular item to reveal connected items.
  - **Resources** : View meta data like TLS Certificates, HTTP Headers, DNS records from lower resolution items such as an HTTP Error code or a rare TLS Cipher.
  - **Full Text Search** : For tools based on Log analysis such as Splunk or ELK you can query for logs and aggregations. TrisulNSM is based on streaming analytics but has limited FTS (Full Text Search) capability for HTTP Headers and TLS Certificates.
  - **Files** : Dump executables and potential troublesome downloads. TrisulNSM can extract files of any size and show them. 
  - **Flows** : The second last level of drill down. From a IP or App or other items you can pull up a list of flows. 
  - **Packets** : The final level of drill down.  After this you should have all the information to decide if any escalation is required for action outside the NSM toolset.  



==== Next ====

Enough of theory. [[offline:wrccdc_pcaps_trisulnsm|Part-2 of this series]] explains how you can get the TrisulNSM Docker image to run over the PCAP dump