User Tools

Site Tools


tips:flowindextuning

Tuning flow index database

The Trisul flow database is a high performance high throughput store specifically designed for Network Flow storage.

The tuning parameters are controlled by the following section in trisulHubConfig.xml Trisul Hub Configuration file

Flow database tuning parameters

<DBParameters>
    <FlowStream>
        <MicroSecondTimestamps>true</MicroSecondTimestamps>
 
        <kFLOWS_PER_BLOCK></kFLOWS_PER_BLOCK>
        <kBLOOM_AGG_SIZE></kBLOOM_AGG_SIZE>
        <kBUMPX_AGG_SIZE></kBUMPX_AGG_SIZE>
        <ZFLOWBLOCK_COMPRESSOR_CODE></ZFLOWBLOCK_COMPRESSOR_CODE>
 
        <!-- Bitmap indices - to disable set buckets to 0 -->
        <kHOST_BUMPX_BUCKETS></kHOST_BUMPX_BUCKETS>
        <kAPP_BUMPX_BUCKETS></kAPP_BUMPX_BUCKETS>
        <kPROTO_BUMPX_BUCKETS></kPROTO_BUMPX_BUCKETS>
        <kGEN1_BUMPX_BUCKETS></kGEN1_BUMPX_BUCKETS>
        <kGEN2_BUMPX_BUCKETS></kGEN2_BUMPX_BUCKETS>
        <kGEN3_BUMPX_BUCKETS></kGEN3_BUMPX_BUCKETS>
        <kAUX0_BUMPX_BUCKETS></kAUX0_BUMPX_BUCKETS>
        <kAUX1_BUMPX_BUCKETS></kAUX1_BUMPX_BUCKETS>
        <kAUX2_BUMPX_BUCKETS></kAUX2_BUMPX_BUCKETS>
        <kHOST24_BUMPX_BUCKETS></kHOST24_BUMPX_BUCKETS>
        <kHOST16_BUMPX_BUCKETS></kHOST16_BUMPX_BUCKETS>
 
 
        <!-- Bloom filter matrix.. set to bits and hashes to 0 to disable an index -->
        <kHOST_BLOOM_BITS>  </kHOST_BLOOM_BITS>
        <kHOST_BLOOM_HASHES> </kHOST_BLOOM_HASHES>
        <kAPP_BLOOM_BITS> </kAPP_BLOOM_BITS>
        <kAPP_BLOOM_HASHES></kAPP_BLOOM_HASHES>
        <kPROTO_BLOOM_BITS></kPROTO_BLOOM_BITS>
        <kPROTO_BLOOM_HASHES></kPROTO_BLOOM_HASHES>
        <kGEN1_BLOOM_BITS></kGEN1_BLOOM_BITS>
        <kGEN1_BLOOM_HASHES> </kGEN1_BLOOM_HASHES>
        <kGEN2_BLOOM_BITS> </kGEN2_BLOOM_BITS>
        <kGEN2_BLOOM_HASHES> </kGEN2_BLOOM_HASHES>
        <kGEN3_BLOOM_BITS> </kGEN3_BLOOM_BITS>
        <kGEN3_BLOOM_HASHES> </kGEN3_BLOOM_HASHES>
        <kAUX0_BLOOM_BITS> </kAUX0_BLOOM_BITS>
        <kAUX0_BLOOM_HASHES> </kAUX0_BLOOM_HASHES>
        <kAUX1_BLOOM_BITS> </kAUX1_BLOOM_BITS>
        <kAUX1_BLOOM_HASHES> </kAUX1_BLOOM_HASHES>
        <kAUX2_BLOOM_BITS> </kAUX2_BLOOM_BITS>
        <kAUX2_BLOOM_HASHES> </kAUX2_BLOOM_HASHES>
        <kHOST16_BLOOM_BITS> </kHOST16_BLOOM_BITS>
        <kHOST16_BLOOM_HASHES> </kHOST16_BLOOM_HASHES>
        <kHOST24_BLOOM_BITS> </kHOST24_BLOOM_BITS>
        <kHOST24_BLOOM_HASHES> </kHOST24_BLOOM_HASHES>
 
    </FlowStream>
</DBParameters>

Indices and Filters

The main tuning are the sizes of the Bitmap Indices and Bloom Filters. The details of the implementation are not important here. We talk about the tyeps and how to disable them.

If a Index or Filter for a particular type is disabled, the query performance with that type shall be slower.

Types and typical sizes at ISP scale

Bitmap indices

kHOST_BUMPX_BUCKETS ip address v4 and v6 870M + 870M + 870M = 2.6G 2.6G K1A + 2.6G K1Z + 2.6G K21= 7.8GG
kAPP_BUMPX_BUCKETS ports 300MB 4G
kPROTO_BUMPX_BUCKETS protocol
kGEN1_BUMPX_BUCKETS router IP
kGEN2_BUMPX_BUCKETS router Interface ingres
kGEN3_BUMPX_BUCKETS router interface egres
kAUX0_BUMPX_BUCKETS tag 0
kAUX1_BUMPX_BUCKETS tag 1
kAUX2_BUMPX_BUCKETS the main flow tag strings
kHOST24_BUMPX_BUCKETS net/24 860M 2.6G
kHOST16_BUMPX_BUCKETS net/16 430M 670M
tips/flowindextuning.txt · Last modified: 2023/09/27 15:04 by veera