System-Introspected Network and Container Visibility: A Quick Start Guide

Recently, we have introduced the concept of network and container visibility through system introspection and also demonstrated its feasibility with an opensource library libebpfflow. In other words, by leveraging certain functionalities of the linux operating system, we are able to detect, count and measure the network activity that is taking place on a certain host. We have published a paper and also presented the work at the FOSDEM 2019 and therefore a detailed discussion falls outside the scope of this post. However, we would like to recall that information we are able to extract is very rich and is absolutely not limited to mere byte and packet counters. For example we can determine:

• All the TCP and UDP network communications, including their peers, ports, and status
• TCP counters and also retransmissions, out-of-orders, round-trip times, and other metrics which are useful as a proxy for the communication quality
• Users, processes and executables behind every communication  (eg., /bin/bash , executed with root privileges, is contacting a malware IP address)
• Container and orchestrator information, when available (e.g., /usr/sbin/dnsmasq is being run inside container dnsmasq which, in turn, belongs to Kubernetes pod kube-dns-6bfbdd666c-jjt75 within namespace kube-system)

By the way, do you know what is the really cool innovation behind all of this? Well, actually, is that we do not have to look at the packets to get this information out! This is why we also love to use the term packetless network visibility, which may seem an oxymore at first, but eventually it makes a lot of sense. Indeed, not looking at the packets is not only cool but it is also somehow necessary under certain circumstances. Think to multiple containers which are communicating together on the same host. Their packets would never leave the system and, thus, would never reach the network, making any mirror or TAP totally useless. In this case, having visibility into the inter-container communications would require an introspection-based approach such as the one we have proposed.

Getting Started

Ok so now that we have gone through a brief recap of our technology it is time to see it in action. To start you need two pieces:

• nprobe-agent, a small application which integrates libebpfflow and is responsible for performing system introspection
• ntopng, our visualization tool, which receives introspected data from nprobe-agent and visualizes it in an handy GUI

Configuration is straightforward. You can fire nprobe-agent with just a single option which basically tells it the address on which ntopng is listening for introspected data

# nprobe-agent -v --zmq tcp://127.0.0.1:1234c

In this example, we are going to use nprobe-agent and ntopng on the same host so we are safely using the loopback address 127.0.0.1 to make them communicate. Note, however, that this is not necessary as nprobe-agent and ntopng can run on two physically separate hosts and you can also run multiple nprobe-agent and let them export to the same instance of ntopng.

To collect data from nprobe-agent, ntopng can be started as follows

./ntopng -i tcp://*:1234c -m "192.168.2.0/24"

the -i option specifies on which port ntopng has to listen for incoming data (see the port is 1234, the same used for nprobe-agent) whereas the option -m specifies a local network of interest.

Once both applications are running, point your browser to the address of ntopng and you will start seeing network communications along with users, processes and container information. Cool, isn’t it?

Combining Network Packets with System Introspection

When you have packets, you can also combine them with data from system introspection. This is straightforward to do. You just have to indicate a packet interface in ntopng as a companion of the interface which is responsible for the collection of system introspection data from nprobe-agent.

For example, assuming an ntopng instance is monitoring the loopback interface lo in addition to receiving data from nprobe-agent as

 ./ntopng -i tcp://*:1234c -i lo -m "192.168.2.0/24"

We can declare the tcp://*:1234c as the companion of lo from the preferences as

From that point on, system-introspected data arriving at tcp://*:1234c will also be delivered to lo and automatically combined with real packets: