User Tools

Site Tools


pcaps:ixmgtool

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
pcaps:ixmgtool [2019/04/13 19:06] – [trisul_ixmgtool] veerapcaps:ixmgtool [2019/04/15 16:50] (current) – [Conclusion] veera
Line 1: Line 1:
-====== Merge multiple thin PCAP files into a single thick PCAP ======+====== Merge multiple thin PCAP files into a single fat PCAP ======
  
 When you install Trisul Network Analytics , you get a free command line tool called ''trisul_ixmgtool'' When you install Trisul Network Analytics , you get a free command line tool called ''trisul_ixmgtool''
Line 7: Line 7:
 ===== What is a FAT pcap file ===== ===== What is a FAT pcap file =====
    
-A FAT pcap file contains more unique flows and endpoints than a THIN pcap file.   +<note>A FAT pcap file contains more unique flows and endpoints than a THIN pcap file regardless of the actual bandwidth.    
 +</note>
  
 While testing NSM((Network Security Monitoring))  platforms we look for FAT pcap files because it stresses the memory and performance of algorithms.  Given a 10GB //thin// PCAP file with just 1 flow, and a 1GB //fat//  PCAP file with 100K flows - you should prefer the FAT file for testing. While testing NSM((Network Security Monitoring))  platforms we look for FAT pcap files because it stresses the memory and performance of algorithms.  Given a 10GB //thin// PCAP file with just 1 flow, and a 1GB //fat//  PCAP file with 100K flows - you should prefer the FAT file for testing.
Line 17: Line 18:
 ===== How is it different from mergecap  ===== ===== How is it different from mergecap  =====
  
-Mergecap  is a command line [[https://www.wireshark.org/docs/man-pages/mergecap.html|utility from the wireshark]] project.  It also combines multiple thin PCAP files into a single  fat PCAP file. But it preserves the timestamps, hence works to //fatten// the output PCAP if there is significant overlap in the time windows.+Mergecap  is a command line [[https://www.wireshark.org/docs/man-pages/mergecap.html|utility from the wireshark]] project.  It also combines multiple thin PCAP files into a single  fat PCAP file. But it preserves the timestamps, hence works to //fatten// the output PCAP //only//  if there is significant overlap in the time windows.
  
-trisul_ixmgtool when run with the squish option , aligns the timestamps  of the files to the lowest timestamp and then processes the merge.  The following diagram illustrates the difference between mergecap and ixmgtool+trisul_ixmgtool when run with the squish option , aligns the timestamps  of the files to the lowest timestamp and then processes the merge.  The following diagram illustrates the difference between mergecap and ixmgtool.  
  
 {{:pcaps:ixmgtool.png |}} {{:pcaps:ixmgtool.png |}}
  
 +You can think of ixmgtool as combining the following three  operations
 +  - Find the lowest timestamp from all the pcap files, and compute the deltas for each file
 +  - Run ''editcap -t delta'' to transform the timestamps of each file
 +  - Run ''mergecap'' on the transformed pcap files
  
-====== trisul_ixmgtool ======+====== Using trisul_ixmgtool ======
  
 To get the free ixmgtool [[https://trisul.org/download|install Trisul Probe]] , you will find the trisul_ixmgtool in ''/usr/local/bin'' To get the free ixmgtool [[https://trisul.org/download|install Trisul Probe]] , you will find the trisul_ixmgtool in ''/usr/local/bin''
Line 48: Line 53:
 ===== Example run ===== ===== Example run =====
  
-Say you have put 10 files in a directory  and you want to create a single FAT file. +Say you have put 10 files in a directory  and you want to create a single FAT file.  If you are curious, we got these files from the good folks who run WRCCDC((The WRCCDC Cyber defense competition archives at https://archive.wrccdc.org/ ))
  
 <code> <code>
Line 97: Line 102:
  
  
-To get a really fat pcap you can use the squish 10 option+To get a **really FAT pcap** you can use the ''-squish10'' option. This creates 10 dummy flows for each flow by manipulating the source IP to 10 different IPs in the 10.0.0.x range. 
  
  
 +
 +<code>
 +unpl@unpl:~/wrccdc$ trisul_ixmgtool -squish10 -r . *.pcap -out really_fatone.pcap
 +
 +5000000 Packets  15005458762 Bytes Time Fri Mar  1 19:11:31 2019-475695
 +EOF on wrccdc.regionals.2019-03-01.111203006460000.pcap, bye ! 
 +EOF on wrccdc.regionals.2019-03-01.111159006450000.pcap, bye ! 
 +EOF on wrccdc.regionals.2019-03-01.111147006420000.pcap, bye ! 
 +8000000 Packets  23767761206 Bytes Time Fri Mar  1 19:11:32 2019-940185
 +EOF on wrccdc.regionals.2019-03-01.111143006410000.pcap, bye ! 
 +EOF on wrccdc.regionals.2019-03-01.111210006480000.pcap, bye ! 
 +EOF on wrccdc.regionals.2019-03-01.111206006470000.pcap, bye ! 
 +EOF on wrccdc.regionals.2019-03-01.111151006430000.pcap, bye ! 
 +EOF on wrccdc.regionals.2019-03-01.111155006440000.pcap, bye ! 
 +EOF on wrccdc.regionals.2019-03-01.111129006380000.pcap, bye ! 
 +EOF on wrccdc.regionals.2019-03-01.111138006400000.pcap, bye ! 
 +
 +unpl@unpl:~/wrccdc$ ls -lh really_fatone.pcap 
 +-rw------- 1 unpl unpl 13G Apr 13 13:35 really_fatone.pcap
 +
 +</code>
 +
 +
 +====== Conclusion ======
 +
 +trisul_ixmgtool  can be used to create FAT pcaps. These can be very useful for stressing NSM solutons.  Using the squish options you can create a mega thick PCAP file for testing by throwing all your PCAP testing files in single directory from varying timestamps and creating a single thick one. 
 +
 +Hope this is useful to the NSM community. 
 +
 +
 +To get the tool (it is free).  Install the Trisul Probe package for your platform from  the [[https://trisul.org/download|Trisul Download page]]
  
  
pcaps/ixmgtool.1555162604.txt.gz · Last modified: 2019/04/13 19:06 by veera