pktvisor summarizes data streams in real time and provides a clean, time-windowed HTTP interface and command line UI to the results.

pktvisor

This project is in active development.

pktvisor summarizes data streams in real time and provides a clean, time-windowed HTTP interface and command line UI to the results.

Summarized information includes, for example:

• Packet rates: 50th, 90th, 95th, 99th percentiles
• Packet counts by protocol and IP version
• Cardinality of set of source IPs and DNS qnames seen in window
• Top 10 heavy hitters: IPs, ASNs, Geo, DNS qnames, DNS slow xacts…

Although currently DNS and packet capture focused, it is designed to be used in broader contexts.

2019-2020© NSONE, Inc.

Overview

pktvisor consists of:

1. An agent which efficiently summarizes streams and exposes a REST API to collect the results
2. A terminal based, command line UI which can visualize the real-time summarized data
3. Tools for collecting and visualizing a globally distributed set of agents to a central location

The agent can also summarize pcap files.

[drafting]

API Documentation

The REST API documentation, including a description of the metrics that are available, is available in OpenAPI format. See the docs/ directory.

Getting Started

The easiest way to get started with pktvisor is to use the public docker image. The image contains both the command line UI and the collector daemon (agent).

# pull the container
docker pull ns1labs/pktvisor 
# start the agent/collector
docker run --rm --net=host -d ns1labs/pktvisor pktvisord any
# run the command line UI
docker run -it --rm --net=host ns1labs/pktvisor pktvisor

If many stats do not seem to be collecting, especially in/out stats, then you likely need to use -H to tell it which net prefix to use as the “host” - see the help section below.

See usage examples below for more complex scenarios, including Geo support.

There are currently no prebuilt operating system packages. If you would like to build your own executable, please see the Build section below.

Agent Usage

A collector daemon agent should be installed on each note to be monitored.

Current command line options are described with:

pktvisord --help

    Usage:
      pktvisord [-b BPF] [-l HOST] [-p PORT] [-H HOSTSPEC] [--periods P] [--summary] [--geo-city FILE] [--geo-asn FILE]
                [--max-deep-sample N]
                TARGET
      pktvisord (-h | --help)
      pktvisord --version

    pktvisord summarizes your data streams.

    TARGET is either a network interface, an IP address (4 or 6) or a pcap file (ending in .pcap or .cap)

    Options:
      -l HOST               Run metrics webserver on the given host or IP [default: localhost]
      -p PORT               Run metrics webserver on the given port [default: 10853]
      -b BPF                Filter packets using the given BPF string
      --geo-city FILE       GeoLite2 City database to use for IP to Geo mapping (if enabled)
      --geo-asn FILE        GeoLite2 ASN database to use for IP to ASN mapping (if enabled)
      --max-deep-sample N   Never deep sample more than N% of packets (an int between 0 and 100) [default: 100]
      --periods P           Hold this many 60 second time periods of history in memory [default: 5]
      --summary             Instead of a time window with P periods, summarize all packets into one bucket for entire time period.
                            Useful for executive summary of (and applicable only to) a pcap file. [default: false]
      -H HOSTSPEC           Specify subnets (comma separated) to consider HOST, in CIDR form. In live capture this /may/ be detected automatically
                            from capture device but /must/ be specified for pcaps. Example: "10.0.1.0/24,10.0.2.1/32,2001:db8::/64"
                            Specifying this for live capture will append to any automatic detection.
      -h --help             Show this screen
      --version             Show version

Command Line UI Usage

The command line UI connects to an agent to visualize real time stream summarization. It can connect to a local or remote agent.

Usage Examples

Starting the collector agent from Docker with GeoDB and Host options:

docker run --rm --net=host -d --mount type=bind,source=/opt/geo,target=/geo ns1labs/pktvisor pktvisord --geo-city /geo/GeoIP2-City.mmdb --geo-asn /geo/GeoIP2-ISP.mmdb -H 192.168.0.54/32,127.0.0.1/32 any

Running the console UI from Docker:

docker run -it --rm --net=host ns1labs/pktvisor pktvisor

Centralized Collection

pktvisor may be collected centrally to give a global view of the collected information. [drafting]

Host Concept

Ingress and egress (in/out) related metrics can only be calculated if the agent understands how to identify the host.

[drafting]

Build Dependencies

• CMake >= 3.8
• Linux or OSX
• C++ compiler supporting C++17
• PcapPlusPlus (NS1 fork: pending upstream merge) https://github.com/nsone/PcapPlusPlus

Optional

• MaxMind DB libmaxmindb

Building

Building is based on CMake.

Default build:

mkdir build; cd build
cmake ..
make

Building the docker image:

org="myorg"
image="mypktvisor"
tag="latest"
docker build -t ${org}/${image}:${tag} -f Dockerfile .

Contributions

Pull Requests and issues are welcome. See the NS1 Contribution Guidelines for more information.

License

This code is released under Apache License 2.0. You can find terms and conditions in the LICENSE file.