Differences

This shows you the differences between two versions of the page.

--- pcaps:ixmgtool [2019/04/13 18:51] – [Merge multiple thin PCAP files into a single thick PCAP] veera
+++ pcaps:ixmgtool [2019/04/15 16:50] (current) – [Conclusion] veera
@@ Line 1: / Line 1: @@
-====== Merge multiple thin PCAP files into a single thick PCAP ======
+====== Merge multiple thin PCAP files into a single fat PCAP ======
 When you install Trisul Network Analytics , you get a free command line tool called ''trisul_ixmgtool''
@@ Line 7: / Line 7: @@
 ===== What is a FAT pcap file =====
-A FAT pcap file contains more unique flows and endpoints than a THIN pcap file.
+<note>A FAT pcap file contains more unique flows and endpoints than a THIN pcap file regardless of the actual bandwidth.
+</note>
 While testing NSM((Network Security Monitoring))  platforms we look for FAT pcap files because it stresses the memory and performance of algorithms.  Given a 10GB //thin// PCAP file with just 1 flow, and a 1GB //fat//  PCAP file with 100K flows - you should prefer the FAT file for testing.
@@ Line 17: / Line 18: @@
 ===== How is it different from mergecap  =====
-Mergecap  is a command line [[https://www.wireshark.org/docs/man-pages/mergecap.html|utility from the wireshark]] project.  It also combines multiple thin PCAP files into a single  fat PCAP file. But it preserves the timestamps, hence works to //fatten// the output PCAP if there is significant overlap in the time windows.
+Mergecap  is a command line [[https://www.wireshark.org/docs/man-pages/mergecap.html|utility from the wireshark]] project.  It also combines multiple thin PCAP files into a single  fat PCAP file. But it preserves the timestamps, hence works to //fatten// the output PCAP //only//  if there is significant overlap in the time windows.
-trisul_ixmgtool when run with the squish option , aligns the timestamps  of the files to the lowest timestamp and then processes the merge.  The following diagram illustrates the difference between mergecap and ixmgtool
+trisul_ixmgtool when run with the squish option , aligns the timestamps  of the files to the lowest timestamp and then processes the merge.  The following diagram illustrates the difference between mergecap and ixmgtool.
 {{:pcaps:ixmgtool.png |}}
+You can think of ixmgtool as combining the following three  operations
+  - Find the lowest timestamp from all the pcap files, and compute the deltas for each file
+  - Run ''editcap -t delta'' to transform the timestamps of each file
+  - Run ''mergecap'' on the transformed pcap files
-====== trisul_ixmgtool ======
+====== Using trisul_ixmgtool ======
 To get the free ixmgtool [[https://trisul.org/download|install Trisul Probe]] , you will find the trisul_ixmgtool in ''/usr/local/bin''
+**Usage**
+<code>
+unpl@unpl:~$ trisul_ixmgtool
+Usage : ixmgtool [-squish|-squish10]  -r home-dir f1 f2 f3 f4 f5 ..  -out outfile
+</code>
+**Options**
+  * ''-squish''  :  align the timestamps to the lowest found and merge
+  * ''-squish10'' : fatten by 10 TIMES by taking each TCP flow and making 10 extra duplicate flows by changing the source IP address 10 different IPs in the  10.0.0.x range
+If you run without the squish options, ixmgtool is the same as mergecap.
+===== Example run =====
+Say you have put 10 files in a directory  and you want to create a single FAT file.  If you are curious, we got these files from the good folks who run WRCCDC((The WRCCDC Cyber defense competition archives at https://archive.wrccdc.org/ ))
+<code>
+unpl@unpl:~/wr$ ls -lh
+total 2.5G
+-rw-rw-r-- 1 unpl unpl 119M Mar 15 20:14 wrccdc.regionals.2019-03-01.111129006380000.pcap
+-rw-rw-r-- 1 unpl unpl 112M Mar 15 20:14 wrccdc.regionals.2019-03-01.111133006390000.pcap
+-rw-rw-r-- 1 unpl unpl 124M Mar 15 20:14 wrccdc.regionals.2019-03-01.111138006400000.pcap
+-rw-rw-r-- 1 unpl unpl 125M Mar 15 20:14 wrccdc.regionals.2019-03-01.111143006410000.pcap
+-rw-rw-r-- 1 unpl unpl 106M Mar 15 20:14 wrccdc.regionals.2019-03-01.111147006420000.pcap
+-rw-rw-r-- 1 unpl unpl 110M Mar 15 20:14 wrccdc.regionals.2019-03-01.111151006430000.pcap
+-rw-rw-r-- 1 unpl unpl 107M Mar 15 20:14 wrccdc.regionals.2019-03-01.111155006440000.pcap
+-rw-rw-r-- 1 unpl unpl 105M Mar 15 20:14 wrccdc.regionals.2019-03-01.111159006450000.pcap
+-rw-rw-r-- 1 unpl unpl 112M Mar 15 20:14 wrccdc.regionals.2019-03-01.111203006460000.pcap
+-rw-rw-r-- 1 unpl unpl 119M Mar 15 20:14 wrccdc.regionals.2019-03-01.111206006470000.pcap
+-rw-rw-r-- 1 unpl unpl 113M Mar 15 20:14 wrccdc.regionals.2019-03-01.111210006480000.pcap
+-rw-rw-r-- 1 unpl unpl 118M Mar 15 20:14 wrccdc.regionals.2019-03-01.111215006490000.pcap
+</code>
+Running the following command
+<code>
+unpl@unpl:~/wr$ trisul_ixmgtool -squish -r . *.pcap -out fatone.pcap
+EOF on wrccdc.regionals.2019-03-01.111203006460000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111159006450000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111147006420000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111143006410000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111210006480000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111206006470000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111151006430000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111155006440000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111129006380000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111138006400000.pcap, bye !
+Done.
+</code>
+results in a fat pcap
+<code>
+unpl@unpl:~/wrccdc$ ls -lh fatone.pcap
+-rw------- 1 unpl unpl 1.2G Apr 13 13:29 fatone.pcap
+</code>
+To get a **really FAT pcap** you can use the ''-squish10'' option. This creates 10 dummy flows for each flow by manipulating the source IP to 10 different IPs in the 10.0.0.x range.
+<code>
+unpl@unpl:~/wrccdc$ trisul_ixmgtool -squish10 -r . *.pcap -out really_fatone.pcap
+5000000 Packets  15005458762 Bytes Time Fri Mar  1 19:11:31 2019-475695
+EOF on wrccdc.regionals.2019-03-01.111203006460000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111159006450000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111147006420000.pcap, bye !
+8000000 Packets  23767761206 Bytes Time Fri Mar  1 19:11:32 2019-940185
+EOF on wrccdc.regionals.2019-03-01.111143006410000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111210006480000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111206006470000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111151006430000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111155006440000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111129006380000.pcap, bye !
+EOF on wrccdc.regionals.2019-03-01.111138006400000.pcap, bye !
+unpl@unpl:~/wrccdc$ ls -lh really_fatone.pcap
+-rw------- 1 unpl unpl 13G Apr 13 13:35 really_fatone.pcap
+</code>
+====== Conclusion ======
+trisul_ixmgtool  can be used to create FAT pcaps. These can be very useful for stressing NSM solutons.  Using the squish options you can create a mega thick PCAP file for testing by throwing all your PCAP testing files in single directory from varying timestamps and creating a single thick one.
+Hope this is useful to the NSM community.
+To get the tool (it is free).  Install the Trisul Probe package for your platform from  the [[https://trisul.org/download|Trisul Download page]]