docker:pcap_analysis
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
docker:pcap_analysis [2017/11/16 23:31] – [Single pass only] veera | docker:pcap_analysis [2018/03/02 17:27] (current) – [Instructions : How to run the Docker image over PCAPs] veera | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== How to analyze large PCAP dumps using the free TrisulNSM Docker ====== | + | ====== How to analyze large PCAP files using TrisulNSM Docker ====== |
- | Quite a few of us have large packet capture dumps, but we dont quite know what do to with them. Some of the things we do are | + | While live traffic capture is the predominant mode of Network Security Monitoring, it is also crucial to be able to load packet capture |
- | * Wireshark : Just keep the PCAP dumps around, in case you need to you can carve small pieces of them and load them into Wireshark | + | |
- | | + | |
- | | + | |
- | * Suricata / Snort : Run an IDS ruleset over the PCAPs. | + | |
- | * Security Onion : Play the PCAPs using TCP Replay and get all of the above. | + | - **Search vs Streaming** import PCAP is a bit harder for streaming pipelines like Trisul compared to Elastic Search backends. This is due to the possibility of the streaming window closing before all the events come in. |
- | * Moloch : Index the packets | + | |
- | * TrisulNSM : Traffic metrics tool at its core, but does flows, index packets, extracts metadata, and other NSM tasks | + | |
+ | ===== Tools ===== | ||
- | We just released a free [[https:// | ||
- | |||
- | ===== How it works : Two pass analysis ===== | ||
- | |||
- | When you specify '' | ||
- | |||
- | - **Pass 1 : Using Trisul**, we collect deep traffic metrics, reconstruct and analyze flows, extract metadata, index and store packets. | ||
- | - **Pass 2 : Using Suricata + Trisul**, we generate IDS alerts and convert those into metrics and graphs. Examples are vertices from a specific SIGID or Host. Top-k for hosts that are in attacker role, etc. This goes to Layer 1 | ||
- | |||
- | The final result is a merger of Layer 0 + Layer 1. You can pivot from alerts to flows to TLS certificates down to packets. | ||
- | |||
- | The following picture is what it looks like. | ||
- | {{ : | ||
+ | Here are some of the tools in the NSM ecosystem that generate various types of data that need to be orchestrated. | ||
+ | * **Argus or SiLK** : Try to index them from a flow perspective. Argus or SiLK can do this. | ||
+ | * **Bro** : turn the PCAPs into bro logs, which record | ||
+ | * **Suricata / Snort** : Run an IDS ruleset over the PCAPs. | ||
+ | * **Security Onion** : The NSM distro that packages everything you need. The latest version of Security Onion includes a script to automate this process.The backend storage and reporting is now Elastic Search. | ||
+ | * **Moloch** : Index raw packets for fast recall. Also stores into Elastic Search. | ||
+ | * **NTOP** : Traffic monitoring | ||
+ | * **Wireshark/ | ||
+ | * **TrisulNSM** : Traffic metrics at its core, but also does flows, packet indexiing, metadata extraction, and other NSM functions. Trisul uses Stream Processing instead of Search as its backend. The free license allows you to import any number of PCAPs as long each PCAP isnt longer than 3 days. | ||
+ | This article explains how the new [[https:// | ||
===== Instructions : How to run the Docker image over PCAPs ===== | ===== Instructions : How to run the Docker image over PCAPs ===== | ||
- | < | + | Put the PCAP dump into the shared docker |
- | We are assuming here that you have a Linux system with Docker installed. | + | |
- | </ | + | |
- | + | ||
- | First you need to create a //root volume//, say ''/ | + | |
- | Then put the PCAP dump into the trisul root volume so that the docker image can read it. | + | |
< | < | ||
- | mkdir /opt/trisul6_root | + | mkdir /opt/trisulroot |
- | cp / | + | cp myhugeCapture.pcap /opt/trisulroot |
</ | </ | ||
+ | Run the trisul6 docker image on the PCAP | ||
- | <note important> | + | <code bash> |
- | Ensure the file is uncompressed. Suricata cant work with gz/bzip2 at the moment. | + | |
- | </note> | + | |
+ | docker run --privileged=true \ | ||
+ | | ||
+ | --net=host | ||
+ | -v / | ||
+ | -d trisulnsm/ | ||
+ | --fine-resolution \ | ||
+ | | ||
- | Run the free trisul6 docker image on the PCAP like so | ||
- | |||
- | <code bash> | ||
- | docker run --name trisul1a --net=host \ | ||
- | -v / | ||
- | -d trisulnsm/ | ||
- | --pcap BSidesDE2017_PvJCTF.pcap | ||
</ | </ | ||
- | Now wait for the import to complete. | + | Now wait for the import to complete. |
- | To check on progress | + | To check on progress , tail the log and wait for confirmation |
< | < | ||
- | docker logs trisul1a | + | docker logs -f trisul1a |
</ | </ | ||
Line 78: | Line 68: | ||
<code bash> | <code bash> | ||
docker run --name trisul1a --net=host \ | docker run --name trisul1a --net=host \ | ||
- | -v /opt/trisul6_root:/ | + | -v /opt/trisulroot:/ |
-d trisulnsm/ | -d trisulnsm/ | ||
--pcap BSidesDE2017_PvJCTF.pcap \ | --pcap BSidesDE2017_PvJCTF.pcap \ | ||
Line 85: | Line 75: | ||
- | Note : For pcap import | + | ==== Analysis ==== |
+ | |||
+ | Once loaded | ||
+ | |||
+ | {{: | ||
+ | |||
+ | After you login here are some suggested steps | ||
+ | |||
+ | | ||
+ | | ||
+ | - Tools > Explore to query flows. | ||
+ | |||
+ | {{: | ||
+ | |||
+ | |||
+ | ===== Processing Compressed PCAP files ===== | ||
+ | |||
+ | Trisul can handle compressed PCAP files (gz, bz2) , a large number of PCAPs in a directory, or even a directory tree. Trisul will automatically process the files in order of the timestamp in the first packet in each file. However, Suricata isnt able to handle that. If you need the full IDS alerts + Traffic analytics then you need to process a single uncompressed file at at time. You can use mergecap to combine them outside of Trisul. | ||
+ | |||
===== Multiple imports ===== | ===== Multiple imports ===== | ||
Line 107: | Line 116: | ||
The docker image includes a [[https:// | The docker image includes a [[https:// | ||
+ | |||
+ | |||
+ | |||
+ | ===== How it works : Two pass analysis ===== | ||
+ | |||
+ | When you specify '' | ||
+ | |||
+ | - **Pass 1 : Using Trisul**, we collect deep traffic metrics, reconstruct and analyze flows, extract metadata, index and store packets. | ||
+ | - **Pass 2 : Using Suricata + Trisul**, we generate IDS alerts and convert those into metrics and graphs. Examples are vertices from a specific SIGID or Host. Top-k for hosts that are in attacker role, etc. This goes to Layer 1 | ||
+ | |||
+ | The final result is a merger of Layer 0 + Layer 1. You can pivot from alerts to flows to TLS certificates down to packets. | ||
+ | |||
+ | The following picture is what it looks like. | ||
+ | {{ : | ||
+ | |||
+ | |||
docker/pcap_analysis.1510855308.txt.gz · Last modified: 2017/11/16 23:31 by veera