trisul_flowcap : Prune PCAP files by capping flow payloads
Using this tool you can reduce size of Trisul PCAPs by capping how many bytes of each flow is stored. Almost all of the valuable intelligence is in the first 1MB or so of each flow, so this is an effective technique to dramatically reduce the size of PCAP files.
In Trisul, PCAP policies can be specified
- During live capture
- at capture time you can control PCAP policies see controlling packet storage
- Post capture
- trisul_flowcap can be used to prune these already collected files. Lets you create custom policies such as. Prune 10MB/flow for 1 month, then reduce to 100K/flow.
Some of the unique features of trisul_flowcap are Sampling, Rich ouput, Dry run
A hard cutoff destroys the statistical information convyed in a PCAP file. You cannot process the compressed capture file through a program to find how many bytes the flow actually transferred. Sampling helps here.
--capbytes-1000000 --samplerate=100 you instruct the program to Write the first 1MB of each flow, then for payloads after 1MB sample at 1:100 one every 100 packets, then adjust the WireLength to save the bytes that were skipped.
After processing is complete trisul_flowcap prints a rich performance summary.
Here is a sample output
DOCKER:unplprotectli:root oper$ trisul_flowcap -c /usr/local/etc/trisul-probe/domain0/probe0/context0/trisulProbeConfig.xml --capbytes=1000000 --samplerate=100 -i RCF_triscap.wseXmJ -o RCF_triscap.wseXmJ.flowcapped Progress : |||||||||||||||||||| 100% In Bytes : 998931121 (952.65 MB) Out Bytes : 313149410 (298.64 MB) Compression % : 68.6516 % Flows : 18072 Flows capped : 85 Flows capped % : 0.470341 % Method : sampled if over flow over 1000000 bytes Input : RCF_triscap.wseXmJ Output in : RCF_triscap.wseXmJ.flowcapped Elapsed time : 18 seconds DOCKER:unplprotectli:root oper$
Interpreting the output
The output says you have compressed the Trisul PCAP file by 68.6% , more than 1/3rd while only impacting 0.47% of the flows. 99.53% of the flows are left AS-IS.
It could be quite difficult to judge what is the optimum capture bytes. The
--dryrun option prints the statistice without actually pruning the input file.
man trisul_flowcap in the probe nodes to view the man page. The man page is reproduced below for reference.
trisul_flowcap – reduce size of packet capture files by using a flow cutoff
trisul_flowcap [−OPTIONS ] [−i Input PCAP] [−o Output PCAP]
trisul_flowcap is a tool to reduce the size of packet capture files. Trisul (RCF_) and PCAP file formats are supported. It can also be used to convert from Trisul to PCAP format and vice
trisul_flowcap takes advantage of the fact is that in network traffic an overwhelming percentage of traffic is carried by a small number of large flows called ’elephant flows’. This tool uses a flow cutoff method to only store the first X bytes of a flow.
A nice real world use case is a sliding resolution. Full content can be stored for 1 Week, then the first 10MB of each flow for upto 1 Month, then the first 50K for upto 3 months.
The Trisul config file (see FILES). This tool needs a trisul installation because
it requires the trisul packet dissection engine to locate the IP/TCP/UDP layers up from the link layers.
The input packet capture file. It can be a Trisul packet file usually named RCF_ or a PCAP
The input file format. Should
be pcap or trisul Default is :
The output file. This is optional when the —dryrun option is used. Note: The output file in Trisul format cannot be located on a tmpfs partition. You will get an error. This is due to the fact that the Trisul output uses a high throughput DIRECT write method that is not supported on a tmpfs partition.
The output format. Should be pcap or trisul Default is : trisul
Maximum number of raw bytes to
keep for each flow.
The maximum number of flows to keep track of before using a LRU algorithm to evict flows. This option can be used to control memory usage. Default is1000 flows.
Instead of completely truncating flows that cross the cutoff volume, you can specify a sample rate . With a sample rate you can save 1 every s packets for flows that cross the cutoff volume. The wirelen of the sampled packets written to the output are multiplied by the sample rate to preserve statistical and flow continuity in the output file. Default : 0 (no sampling is used)
Use this option to do a dry run to check the compression performance of this tool without writing an output file.
Dont show the progress bar. Use this when running this tool in a cron job or in the background
Print a help message
To compress a
Trisul format capture file RCF_triscap.P7T8Lw with a flow
cutoff of 1MB
-i RCF_triscap.P7T8Lw \
-o /home/bob/RCF_triscap.P7T8Lw.cut -n 1000000
To compress a
PCAP file Bigcapture.pcap into Bigcapture.pcap.cut with a
flow cutoff of 1MB
-i Bigcapture.pcap -I pcap \
-o Bigcapture.pcap.cut -O pcap -n 1000000
RCF_triscap.P7T8Lw with a flow cutoff of 1MB and a sampling
of 1:10 after that
-c ….trisulProbeConfig.xml \
-i RCF.triscap.P7T8Lw \
-o /home/bob/RCF_triscap.P7T8Lw.cut -n 1000000 -s 10
The Trisul probe configuration file. This is typically located in /usr/local/etc/trisul-probe under subdirectory domain0/probe0/context0. This file is requires because trisul_flowcap needs to use the probe protocol dissectors to locate the IP and IPv6 headers.
Trisul Network Analytics © 2020 https://trisul.org