7.7. Complete IP Flow logger

7.7.1 DoT (India) Compliance

As per Dept of Telecommunications, India. All Internet Service Providers and holders of Mobile UMTS/CMTS/UL licensees are required to store the following information on a per flow basis.

Parameters to be collected

• Customer information obtained during customer acquisition.
• user id
• start time of flow
• end time of flow
• source ip address
• NAT source IPv4 or IPv6 address (if NAT is used)
• source port
• NAT source port (if NAT is used)
• Destination IPv4 of IPv6 address
• Destination Port

The compliance requirement calls the above information IPDR and NAT Syslog. Ideally they should be combined into one simple comprehensive per-flow log. This is what Trisul IPDR provides.

7.7.2 Obtaining data telemetry

The Trisul IPDR solution can accept a variety of inputs to create and maintain the IPDR log for compliance. The following diagram explains the architecture.

7.7.3 Configuring Trisul for IPDR

At ISP scale storing every single flow is a processor and disk intensive task. Hence this configuration cannot be mixed with the Netflow Traffic and Security Analytics feature except for very small ISPs. This section contains configuration tips to optimize this feature.

Use compressor lz4-ipv4-call-log-with-nat

Use a new flow compressor specifically designed for IPDR flow log. This high performance compressor can store a flow with NAT in as little as 14 bytes.

Open the trisulHubConfig.xml file and specify the following in Advanced DB Parameters.

• Use a new compressor lz4-ipv4-call-log-with-nat
• Disable microsecond timestamps

<DBParameters>
    <FlowStream>
        <MicroSecondTimestamps>false</MicroSecondTimestamps>
        <ZFLOWBLOCK_COMPRESSOR_CODE>lz4-ipv4-call-log-with-nat</ZFLOWBLOCK_COMPRESSOR_CODE>
...

7.7.4 The trisul-ipdr query service

The Trisul IPDR package comes with a powerful async query service called trisul-ipdr The features of this service are

1. Asynchronous – you can submit multiple long running queries for IP and then download the results when done
2. Results – the results will be in compliance format including the full IP details as well as the NAT (if applicable)
3. Dashboard – dashboard shows running queries, completed queries, download results
4. Cancel – allows cancel of long running queries, number of records are constantly updated
5. Audit log – all query submits including user name, submit time, submit params are stored in audit log
6. Statistics – Dashboard also shows important statistics about Total BW, Number of flows/min, DB growth etc.
7. Automatic FTP – for query dumps that are huge, the service automatically FTP’s results to a separate secure FTP server

To start this service

systemctl start trisul-ipdr 

7.7.5 Workflow

The system is designed to create a special login to the agent who will be performing the queries. This login has no other privileges other than do perform the query required for complaince. The powerful trisul_ipdr service described above ensures the data is provided as a download or pushed directly to a Secure FTP (SFTP) server. Sometimes we have noticed agent requests resulting in several GB of output which cannot be downloaded over a browser. See Configure IPDR Settings on how to setup the SFTP server.

The following diagram shows the workflow

• Agent login with special ID

  The agent is given a separate login and password with a dashboard that shows only one option. To retrieve IPDR logs.

• Agent enter query

  As per compliance requirements agent queries are based on IPv4 or IPv6 IP address. The agent enters the query here by

  • enter the IP address
    • source , destination, or NAT address supported
  • select a time window for the query

  The fields are marked by the arrows in the figure shown below

  
  

• Get results from IPDR Dashboard

  The IPDR service is asynchoronous, this allows agents to submit multiple long queries and wait for results without having to submit them one after another. The results are obtained in the following two methods

  Secure FTP or Download

  Download directly from Web Browser

  For queries that result in a reasonable file size (default 100MB) the agent can directly download the results by pressing the Download button. See “IPDR Settings > Download File Size”:/docs/ug/webadmin/ipdr-settings.html to adjust this limit.

  Get it from a secure FTP server

  Regardless of the limit, if a Secure FTP (SFTP) server is configured the results file is copied to that server automatically. The advantage of this option is very large file sizes can be sent automatically to the FTP server rather than the user having to download from the browser.

  File name

  Once completed the IPDR log will have a name like 765f9124f705735ad54f9c87223eecd4944a0aa7.txt This is to ensure the integrity of the file from tampering. The file name is SHA-1Hash.txt where the file contents are hashed to produce the file name.

  Cancellation

  The user can cancel long running tasks by pressing the Cancel button. The results that have been retrieved up until that point is immediately made available for download and/or FTP.

7.7.6 IPDR Control Dashboard

The dashboard is divided into two parts as shown below

Basic metrics

This shows key metrics of the flow rate, database size, and bandwidth

Control table

IPDR Queries that have been submitted and current status and download options.

Metrics panels

The following metric modules are shown for a statistical overview.

Control table

The control table has a row for every query that has been submitted. The following columns and options are shown

7.7.7 Tuning

We suggest the following configuration parameters for a minimal IPDR deployment.

References

Also see IPDR FTP Settings