tips:flowindextuning
Table of Contents
Tuning flow index database
The Trisul flow database is a high performance high throughput store specifically designed for Network Flow storage.
The tuning parameters are controlled by the following section in trisulHubConfig.xml Trisul Hub Configuration file
Flow database tuning parameters
<DBParameters> <FlowStream> <MicroSecondTimestamps>true</MicroSecondTimestamps> <kFLOWS_PER_BLOCK></kFLOWS_PER_BLOCK> <kBLOOM_AGG_SIZE></kBLOOM_AGG_SIZE> <kBUMPX_AGG_SIZE></kBUMPX_AGG_SIZE> <ZFLOWBLOCK_COMPRESSOR_CODE></ZFLOWBLOCK_COMPRESSOR_CODE> <!-- Bitmap indices - to disable set buckets to 0 --> <kHOST_BUMPX_BUCKETS></kHOST_BUMPX_BUCKETS> <kAPP_BUMPX_BUCKETS></kAPP_BUMPX_BUCKETS> <kPROTO_BUMPX_BUCKETS></kPROTO_BUMPX_BUCKETS> <kGEN1_BUMPX_BUCKETS></kGEN1_BUMPX_BUCKETS> <kGEN2_BUMPX_BUCKETS></kGEN2_BUMPX_BUCKETS> <kGEN3_BUMPX_BUCKETS></kGEN3_BUMPX_BUCKETS> <kAUX0_BUMPX_BUCKETS></kAUX0_BUMPX_BUCKETS> <kAUX1_BUMPX_BUCKETS></kAUX1_BUMPX_BUCKETS> <kAUX2_BUMPX_BUCKETS></kAUX2_BUMPX_BUCKETS> <kHOST24_BUMPX_BUCKETS></kHOST24_BUMPX_BUCKETS> <kHOST16_BUMPX_BUCKETS></kHOST16_BUMPX_BUCKETS> <!-- Bloom filter matrix.. set to bits and hashes to 0 to disable an index --> <kHOST_BLOOM_BITS> </kHOST_BLOOM_BITS> <kHOST_BLOOM_HASHES> </kHOST_BLOOM_HASHES> <kAPP_BLOOM_BITS> </kAPP_BLOOM_BITS> <kAPP_BLOOM_HASHES></kAPP_BLOOM_HASHES> <kPROTO_BLOOM_BITS></kPROTO_BLOOM_BITS> <kPROTO_BLOOM_HASHES></kPROTO_BLOOM_HASHES> <kGEN1_BLOOM_BITS></kGEN1_BLOOM_BITS> <kGEN1_BLOOM_HASHES> </kGEN1_BLOOM_HASHES> <kGEN2_BLOOM_BITS> </kGEN2_BLOOM_BITS> <kGEN2_BLOOM_HASHES> </kGEN2_BLOOM_HASHES> <kGEN3_BLOOM_BITS> </kGEN3_BLOOM_BITS> <kGEN3_BLOOM_HASHES> </kGEN3_BLOOM_HASHES> <kAUX0_BLOOM_BITS> </kAUX0_BLOOM_BITS> <kAUX0_BLOOM_HASHES> </kAUX0_BLOOM_HASHES> <kAUX1_BLOOM_BITS> </kAUX1_BLOOM_BITS> <kAUX1_BLOOM_HASHES> </kAUX1_BLOOM_HASHES> <kAUX2_BLOOM_BITS> </kAUX2_BLOOM_BITS> <kAUX2_BLOOM_HASHES> </kAUX2_BLOOM_HASHES> <kHOST16_BLOOM_BITS> </kHOST16_BLOOM_BITS> <kHOST16_BLOOM_HASHES> </kHOST16_BLOOM_HASHES> <kHOST24_BLOOM_BITS> </kHOST24_BLOOM_BITS> <kHOST24_BLOOM_HASHES> </kHOST24_BLOOM_HASHES> </FlowStream> </DBParameters>
Indices and Filters
The main tuning are the sizes of the Bitmap Indices and Bloom Filters. The details of the implementation are not important here. We talk about the tyeps and how to disable them.
If a Index or Filter for a particular type is disabled, the query performance with that type shall be slower.
Types and typical sizes at ISP scale
Bitmap indices
kHOST_BUMPX_BUCKETS | ip address v4 and v6 | 870M + 870M + 870M = 2.6G | 2.6G K1A + 2.6G K1Z + 2.6G K21= 7.8GG |
kAPP_BUMPX_BUCKETS | ports | 300MB | 4G |
kPROTO_BUMPX_BUCKETS | protocol | ||
kGEN1_BUMPX_BUCKETS | router IP | ||
kGEN2_BUMPX_BUCKETS | router Interface ingres | ||
kGEN3_BUMPX_BUCKETS | router interface egres | ||
kAUX0_BUMPX_BUCKETS | tag 0 | ||
kAUX1_BUMPX_BUCKETS | tag 1 | ||
kAUX2_BUMPX_BUCKETS | the main flow tag strings | ||
kHOST24_BUMPX_BUCKETS | net/24 | 860M | 2.6G |
kHOST16_BUMPX_BUCKETS | net/16 | 430M | 670M |
tips/flowindextuning.txt · Last modified: 2023/09/27 15:04 by veera