Netflow configuration file

How to edit

On the Trisul-Probe type sudo /usr/local/share/trisul-probe/cfgedit then select “Netflow” to edit. See Plugin Configuration

Location

In a file called PI-7CA09636-02D4-45E7-AA00-BE0D49B94E26.xml in /usr/local/etc/trisul-probe/domainX/probeX/contextX You can edit the config file directly by typing the full file name as shown below



# vi /usr/local/etc/trisul-probe/domain0/probe0/context0/PI-7CA09636-02D4-45E7-AA00-BE0D49B94E26.xml 

the above example shows how to edit the config for the default context

Parameters

The following nodes under the global Netflow policy.

Boolean parameters are case-insensitive. True, true, TRUE mean the same thing
Parameters Defaults Description
AppMode Application mode. Set this to ipdr to enable IPDR mode of Netflow processing. Leave it to blank for default processing
MeterHosts True Whether or not traffic stats for each host is metered.
MeterSubnets TRUE Whether or not traffic stats for each subnet is metered.
MeterApplications TRUE Whether or not traffic stats for each application is metered. In this context application refers to the UDP/TCP port or to special network level apps like ICMP, IPSEC, etc. In case of TCP/UDP ports, the default behavior is to only meter the lower numbered port. This works in most cases, but if you want to count both ports, set the MeterBothPorts parameter.
MeterNetflowSources TRUE If set, Trisul tracks traffic volume from each exporting router. The meters tracked are
Netflow export rate in bytes
Netflow export rate in records
Total traffic volume on router in bytes
MeterNetflowInterfaces TRUE If set, Trisul tracks traffic into and out of network interfaces.
MergeMultipleSources TRUE If set, duplicate flows from different routers will be merged.
MatchBiDirectionalFlows TRUE Netflow records are unidirectional. If set, Trisul will merge two unidirectional flows into one bidirectional flow.
If you set this option to false, Trisul will retain the uni-directional flow as-is.
UseRouterTimestamps FALSE If set, Trisul will use the timestamp from the Netflow records.
If not set, Trisul will use the timestamp at the server running Trisul. This is the default and recommended option unless you have routers which synchronize their timestamps automatically.
MergeHTTP FALSE For HTTP flows, ignore the Source Ephemeral Port and set it to 65535. NOTE When the MergeX option is used flows will not be terminate but timed out. Use the MergeX options to reduce the number of unique flows
MergeHTTPS FALSE For HTTPS flows, ignore the Source Ephemeral Port and set it to 65535.
MergeDNS FALSE For DNS flows, ignore the Source Ephemeral Port. Set to FALSE, every DNS transaction will be a separate flow
IgnoreESP FALSE Ignore IPSEC ESP flows. These are tunneled interfaces containing no flow information within them.
MeterHomeNetwork TRUE Classifies traffic relative to your Home Network.
INCOMING Destination IP is in your home network but Source IP is not
OUTGOING Source IP is in your home network but Destination IP is not
INTERNAL Both the Source IP and Destination IP are in your home network
TRANSIT Both the Source IP and Destination IP are not in your home network
MeterBothPorts FALSE When set to FALSE : The default behaviour of Trisul is to only meter the low numbered port. The assumption is that low numbered ports represent servers. This is usually accurate for traditional server applications that run below port 1024.
When set to TRUE : Meters both ports for applications above port 1024. Ports below 1024 (such as HTTP) are still counted in the normal way.
Set this to TRUE if you have P2P / VoIP Traffic you wish to track.
MeterTCPConnections TRUE Meter connection count for the flow TCP end point IPs. This allows you to get reports for hosts with maximum connections, etc
MeterAppConnections TRUE Meter connection count for source and destination ports for TCP based application.
FilterInterfacesInclude A comma separated list of interfaces.
When set, Trisul will accept and process flows that exit or enter these interfaces. This can be in routerip_ifindex format or Trisul key format
An example
If you only want to accept flows that use the WAN link (if Index = 108) on router (122.166.4.242) and GigE link (ifIndex = 120) also on router (122.166.4.242)
You should specify the interfaces as 122.166.2.242_108,122.166.242.100_120
FilterInterfacesExclude A comma separated list of interfaces.
When set, Trisul will only process flows that DO NOT use these interfaces.
The format is same as the parameter FilterInterfacesInclude
Note that you cannot use the FilterInterfacesInclude and Exclude parameters at the same time. This rule applies to all such parameters
FilterFlowASNInclude Only process netflow records containing one of the ASN in this comma separated list
FilterFlowASNExclude Only process netflow records which DO Not contain any ASN in this CSV list. USECASE Upstream ISP filtering out its downstream ISP ASNs for monitoring and logging purposes.
Feature used in ISP environments
FilterHostsInclude Only process netflow records containing IP address in this CSV list (IPv4 only)
FilterHostsExclude Only process netflow records not containing ANY IP in this CSV list (IPv4 only)
FilterAppsInclude Only process these ports. Use comma separated port numbers
CheckPacketsForCorruption FALSE Turns on heuristics to check each netflow record for corruption. When set to TRUE, Trisul validates the timestamps, packets, bytes, etc. We have seen unpatched routers transmit corrupted records, which can distort the metrics and charts.
LogTemplateMessages TRUE If enabled , Trisul will log template updates to the probe log files. Disable if you feel there are too many of these messages.
IgnoreV9EgressFromDevices Sometimes we find that customers erroneously enable both Netflow v9 INGRESS and EGRESS on all ports. This will result in double counting. The solution is to only configure INGRESS+EGRESS on a single port such as the uplink ports. Oftentimes, we are unable to get the Netflow change request done. This option allows you to adjust for this. This option contains a comma separated list of IP addresses of devices from which Trisul will ignore EGRESS netflow V9 records. The following example ignores Netflow v9 EGRESS from two devices
<IgnoreV9EgressFromDevices>172.16.13.31,172.16.13.36
Egress flow records are those with NF9/IPFIX direction field with value=1
IgnoreAllEgress FALSE Ignore all flow records generated by egress processing. Egress flow records are those Netflow v9 and IPFIX records with direction = 1 (egress)
IgnoreOutCounts Set this option to true to ignore out packets and out bytes sent incorrectly by some devices. Some firewalls send the same metrics in IN and OUT counts by misinterpreting the template as post out counts. Use this if you see the same values of IN/OUT on all flows
EnableShimTunnel FALSE If Enabled, Trisul will decapsulate the special SHIM tunnel used to forward Netflow packets from a remote network to the Trisul probe. Normal Netflow will also be processed correctly. For more information see on Github netflow-shim-tunnel
HomeASNumbers Enter a comma separated list of Home AS and Downstream AS numbers. Used in ISP deployments these are the AS numbers of the customer running Trisul. If Home AS numbers are present they are used instead of Home Network Prefixes to determine per-AS Upload (egress) and Download (ingress) traffic metrics per AS and also to split upstream vs downstream. For large and transit ISPs with changing downstream topologies see the parameter HomeASNumbersFile
AddEdges TRUE Adds ASN <—> PREFIX edges. If enables, it allows you to select a AS and see which PREFIXES are active in that AS. Here active means prefixes which had some traffic in the selected interval
MeterLinkAS FALSE Flow-Link-ASN is a special Crosskey Counter Group which tracks network interface > AS flows. Setting this option to TRUE, enables that counter group. You can then visualize these flows using the Sankey App
UsePostNATAddresses TRUE This is Neflow Template option supported on some devices such Palo Alto. Set this to TRUE to use Post-NAT IP address which represent the actual end user in our organization rather than the WAN address.
TemplateDumpIntervalSeconds 600 How frequently must the template database be dumped by the probes. These are shown on the Admin > Show Template DB page. Set this to 0 to prevent template databases being dumped
MeterZeroFlows FALSE Use case Large ISP compliance logging. When compliance only asks for connection logs without collecting byte and packet counts, enabel this option
AllowVariableLengthStrings FALSE IPFIX variable length strings in templates are used by some network vendors to export labels and tags using IPFIX template field type 65535
TagFlowsWithUserID FALSE Some vendors will export flows with a User-ID string which identifies the pre-NAT user of the flow, this appends a tag each flow. See Trisul Flow Taggers
MeterUserID FALSE If User-ID string is present, meter bytes, flows, connections for it
MeterAppID FALSE If App-ID string is present, meter bytes, flows, connections for it
AddL2AccountingBytes 0 Use this option if you want to closely match Netflow with SNMP metrics for accounting and billing purposes. Netflow being a Layer3 (IP) technology it does not account for link layer bytes and also any physical layer bytes such as Ethernet Frame Check sequence. A typical setting is to set this parameter to 18 to add 14 bytes of ethernet and 4 bytes of ethernet FCS to each packet. This option is typically only useful for customers who seek very close match with SNMP metrics for billing purposes
ReloadConfigCheckSeconds Set this to 60 to automatically check for changes to this configuration file every 60 seconds. If the modified timestamp of this configuration file changes, the configuration is automatically reloaded. The default behaviour is for Trisul Probe to require a restart for a Netflow configuration file change.
TagFlowsWithDirection FALSE Tag each flow with a direction relative to the Home Networks (see user guide Home networks section) based on the AS Number information.

Every flow is tagged with one of the following

[dir]internet
when one of source and dest AS is within home network and the other is external
[tag]internal
when both source and dest AS are within the home network
[tag]transit
when both source and dest AS are outside the home network

This allows you to search for flows by direction using the Explore Flows tool and the expression tag=[dir]transit
Feature used in ISP environments

HomeASNumbersFile Used with ISP licensed customers. Trisul automatically creates an online feed of currently observed downstream AS numbers based on BGP routing information from outside vantage points. Set this to the value downstream_origin_as.txt and configure the Geo plugin for automatic updates. For simpler applications see the parameter HomeASNumbers
Feature used in ISP environments
Use64BitHostByteOrder FALSE Some netflow implentations incorrectly send 64 bit counters in template fields as two 32 bit fields in host byte order. If you are seeing bizzarely large numbers for bandwidth, bytes, and packets – try setting this to TRUE
MeterTosAsDSCP TRUE Meter TOS byte as 6-bit DSCP + 2-bit ECN. This is then mapped to DSCP values in Trisul UI such as for example
   “08” → “CS1/AF11-13”
“0A” → “CS1/AF11-13”
SmartNATRemoval TRUE Set this to TRUE in firewall environments. When set to TRUE Trisul analyzers Netflow records and always selects the correct IP addresses to reflect the internal IPs rather than the public NAT IP
Home Prefixes File A Trisul routes database or a CSV file containing all prefixes to be treated as home or downstream prefixes. Typically this file is named 0.0.0.0_routes.sql3 setup during configuraiton for ISP deployment
Feature used in ISP environments
AutoloadPreviousTemplates false When set to true Trisul will reload the most recently known netflow template database and immediately start processing instead of waiting for templates. Use this option in networks where template intervals are long
PrintRecords Used in DDoS report mode to print raw netflow records automatically, Do not set this option manually

Sampling rates

Flow sampling is a technique used by very busy devices to reduce CPU load. If you confiygure a sampling rate of 128, the device will only process every 128 packets. In IPFIX technologies the sampling rate is typically sent in an options template. This section provides a manual way to configure sampling rates per router in the event you are not able to setup the options template. This is not a replacement for configuring sampling on the router or enabling the options template.

Add a separate line for each router as shown below


<SamplingRates>
    <Rate router="180.179.9.2"   rate="128"  />
</SamplingRates>

Default sampling rate

To create a default sampling rate for all routers which do not have an explicit sampling rate set, use the IP address 0.0.0.0. For example to configure a sampling rate of 1000 for all routers. Use


<SamplingRates>
    <Rate router="180.180.9.2"   rate="128"  />
    <Rate router="0.0.0.0"   rate="1000"  />
</SamplingRates>

Advanced

Static Templates

Typically you dont need to use this section. We have used this section in the past to address various firmware bugs in some devices which did not export Netflow templates correctly. Templates fields specified in this section will override those received on the network.