The NetSage Pipeline

Description

The Netsage Flow Processing Pipeline is composed of several components for processing network flow data, including importing, deidentification, metadata tagging, flow stitching, etc. There are many ways the components can be combined, configured, and run. These documents will describe the standard "simple" set up and provide information for more complex configurations.

Data Collection

In Netsage, sensor(s) are network devices configured to collect flow data (tstat, sflow, or netflow) and send it to a "pipeline host" for processing.

Tstat flow data can be sent directly to the pipeline ingest RabbitMQ queue on the pipeline host using the Netsage tstat-transport tool. This can be installed as usual or via Docker.

Sflow and netflow data from configured routers should be sent to the pipeline host where it is collected and stored into nfcapd files using nfdump tools. The Netsage project has packaged the nfdump tools into a Docker container for ease of use.

Pipeline Components

The Netsage Flow Processing Pipeline is made of the following components (currently)

• Importer: Perl scripts on the pipeline host that read nfcapd flow files and send the flow data to a RabbitMQ queue. (Doc, in github)
• RabbitMQ: Used for message passing and queuing of tasks.
• Logstash pipeline: Performs a variety of operations on the flow data to transform it and add additional information. (Doc)
• Elasticsearch: Used for storing the final flow data.

Visualization

Grafana or Kibana can be used to visualize the data stored in elasticsearch. Netsage Grafana Dashboards are available in github.

Pipeline Installation

Originally, the pipeline was deployed by installing all of the components individually on one or more servers (the "BareMetal" or "Server" Install). More recently, we've also added a Docker deployment option. With simple pipelines having just one sflow and/or one netflow sensor (and any number of tstat sensors), the "Docker Simple" Install should suffice. The "Docker Advanced" guide will help when there are more sensors and/or other customizations required.