Even though these might look like cosmetic changes, we are committed to create better software to serve our community. In the past couple of weeks:

• We have migrated the server to a new site that is now IPv4 and IPv6. You can now access our software from IPv6 finally.
• We have moved nDPI, PF_RING and ntopng to github. We hope this will foster collaboration on our software. We are aware that there are areas where we can improve the github integration, but as we’re new to this technology, we invite those who are more expert than us to contribute.
• As many people complained about our web site (hard to find information, not designed for mobile devices) we have redesigned it and made it more modern. We hope you will enjoy it.
• We are aware that we need to integrate better testing tools in our code. While we are exploring testing tools, we have just started with travis so that at every commit our software is compiled and semi-tested.

If you have suggestions or be willing to help, please drop us a mail !

This is to announce the availability of nDPI 1.6, a maintenance release that consolidates this open source deep packet inspection library. This is going to be the last version of the 1.x branch, as we plan (see the enhancements we have planned) major changes for the 2.0 release.

Changelog:

• Moved to github, with continuous testing tools (Travis-CI) and automatic regression testing (via Travis) for improving library quality.
• New dissector for the QUIC protocol.
• Major enhancements for Skype, and Tor detection.
• Reduced memory usage per nDPI flow (up to 40% with respect to 1.5.1).
• Added protocol breed (safe, unsafe….) and thus further classify protocols onto smaller groups.
• Ported to the MIPS platform (Ubiquity Networks), and added fixes for the ARM platform.
• Updated Windows project for building nDPI under Windows.
• Improved dissectors: Facebook, DropBox, Viber, YouTube, RDP, Ayiya, RTCP/RTP,
• Added support for Vevo.

Please do not forget to star nDPI on github if you like this project.

After 9 months of development, we are pleased to announce the release of ntopng 2.0. This is a major release as we have reworked many application components and made the application robust and usable by mid/large companies and ISPs. We have created two versions of ntopng:

• Community edition: this is the standard ntopng that you can use free of charge and that implements a robust and easy to use web-based traffic monitoring application.
• Professional edition: an enhanced version of ntopng that includes modern reports and many new features listed below on this article. This edition is available at a little free that we can use to hire people to better serve all the ntop community. Note that we have created a cheap embedded/professional edition of ntopng usable on ARM-based (e.g. Raspberry PI and BeagleBoard) and MIPS-based (e.g. Ubiquity Networks EdgeRouter) boxes.

As of ntopng 2.0, the binary Windows version of the ntopng community (remember that the code base is the same for Unix and Windows) is free of charge: we want ntopng to be pervasive across platforms. All binary packages are work in professional edition for 10 minutes, and then switch back to the community edition: this allows you to see what is different in the professional edition.

The main new features of ntopng 2.0 include:

Professional Edition

• Dynamic dashboard that includes a realtime view of traffic.
• PDF-printable reports including top hosts/activities/protocols.
• Ability to operate in inline mode and thus implement a layer-7 firewall (even on low-end embedded boxes) and traffic shaper (yes we can drop traffic and assign to hosts a bandwidth to avoid them to monopolise the traffic).
• All graphs are rendered in a pretty way with zoomable (in and out) drill-down facility.
• Per-minute accurate reports (in JSON format) of top X activities so that users can use them to generate further traffic reports in addition to all those included in the pro version.
• Added SNMP support for visualising MIB-II host information through the ntopng web interface.

Community Edition

• Moved the code to GitHub for easier collaboration.
• Fixed several bugs present in the previous version.
• Added continuous testing tools (Travis-CI) and automatic regression testing (via Travis) for improving code quality.
• Added ability to aggregate traffic from various network interfaces on the same interface view while keeping interface traffic split. Example ntopng -i eth1, -i eth2 -i view:eth1,eth2
• Added support for the latest nDPI that includes support for various new protocols (e.g. QUIC) and new versions of existing ones (e.g. Skype). nDPI is also used to drop application traffic in the professional noting edition.
• Hardened the code to support mid/large organisations and high traffic volumes, as well for operating on hosts with little memory.
• Added network latency in flows (server vs client network latency).
• Added flow TCP traffic statistics (packets retransmitted, lost, and out of order).
• Enhanced HTML code to render better on devices of various sizes.
• Enhanced host alerts (including traffic quotas) and added interface alerts. You can now for instance generate traffic alerts when an interface has too much traffic or if a host has passed its daily traffic quota.
• Ability to sniff from netfilter interfaces
• Improvements on OS detection of remote hosts.
• Alerts are now generated when ntopng detects a flooder or a network scanner (as well when accessing malware sites [-c plugin])

• Integration of ntopng with nagios: you can now create nagios plugins to query ntopng and thus emit alerts based not traffic conditions.
• Ability to categorise malware (-c option) using the Google Safe Browsing API that replaces the block.si service present in ntopng 1.x.
• Packaging for Intel, ARM and MIPS platforms.
• Added ability to fine-tune RRD configurations.
• Added ability to generate a traffic report for all hosted HTTP servers (on local networks): ISPs can now create a hourly report of all the thousand of servers they are hosting.
• Ability to work behind an HTTP reverse proxy.
• Enhanced the ElasticSearch export facility to cope with latest additions such as host geolocation.
• Added support for NUMA core affinity.
• Enhanced host GeoIP location.
• Various fixes to the historical network interface.
• Added reports per AS, geo-location, network, HTTP servers.
• Added per-network RRDs.
• Fixed various bugs including a memory leak that was slowly exhausting memory.
• Added several fixes for enhancing security and preventing ntopng to be misused (from the security point of view).
• Added ability to disable HTTP authentication (partially or fully).
• ntopng can now be queries via HTTP tools such as curl or wget with authentication enabled.
• Added ability to dump specific traffic (e.g. of a selected host) or when specific traffic conditions arise (e.g. too much traffic) on a tap interface and attach applications such as Wireshark/tcpdump to it. Similarly added ability to dump traffic to disk in pcap format.
• Added HTTP virtual hosts support in HTML reports.
• Added ability to send data in Lua using UDP (for instance you can use it for exporting metrics to Graphite).
• Added experimental InfluxDB export (disabled by default as ‘as of today’ InfluxDB is not yet production ready).

Please do not forget to star ntopng on github if you like this project!

Links:

• GitHub source code
• Shop site for pro licenses. Do not forget that if you are a no-profit, or educational institution all products are free for you.
• CentOS/Ubuntu binary packages
• Windows x64 binary package
• ARM-based binary packages
• Ubiquity MIPS-based binary packages
• ntopng 2.0 User’s Guide

ntopng allows you to export monitoring data do external sources. For low-traffic sites, SQLite and the ntopng historical interface can be a good option. As your traffic increases you are forced to put your data on a database if you care about performance and long-term data persistency.
In future ntopng versions we will add support for additional databases, but for the time being we decided to start with the ELK (ElasticSearch + LogStash + Kibana) paradigm. In this case ElasticSearch (ES) is the database backend, and Kibana the GUI used to report data. As ntopng is able to natively export data in ElasticSearch, we do not need to use LogStash at all.

Supposed you have installed your ElasticSearch and Kibana instance on host XYZ (that can very well be the same host where ntopng is running) all you need to do to start data export is to start ntopng as follows:

ntopng -F “es;<ES Index Type>;<ES Index Name>;<ES URL>;<ES pwd>”

so something like this should work for most of you

ntopng -F “es;flows;ntopng-%Y.%m.%d;http://XYZ:9200/_bulk;”

In ES parlance an index is what a table is on a relational database. In order to avoid putting all data in a single index (ES can harvest old data with you by configuring the data retention), ntopng will create a daily index automatically for you by using the index name specified on the command line. By default (unless you configure it) ES does not use a password to protect data, so you can leave the password field blank. Make sure that you do not change the /_bulk/ URL as ES likes it that way (of course you can change the host name and port).

Once started, ntopng will push ES flows that are expired or periodically send (every 5 mins) partial flows for long lasting flows. By connecting to Kibana using a web browser you can immediately start seeing incoming flows appear in realtime.

You can click on each individual flow and display all the flow attributes

Now it is time you to create a custom dashboard and report you data on a graphical interface.

A good starting point is Qbana, a github project designed to display monitoring data produced by ntopng/nProbe using Kibana

We would be delighted if ntop users could contribute with Kibana dashboards that we could share inside the community. So please be generous and send us your contribution that we’ll then share with the ntop community.

Last week we have made a couple of presentations at SharkFest 2015, introducing to the Wireshark community what ntopng is about. The most interesting part has been the feedback of those who attended the talks who commented on “how to integrate ntopng with Wireshark”. My personal conclusion is that the best way of doing this, is to turn ntopng into a pre-processor for Wireshark; namely use ntopng to permanently monitor a network, detect anomalies, and then analyse them with Wireshark. In fact a packet sniffer cannot be used as a permanent monitoring tools, as it requires human supervision and expertise to use it and interpret results. On a large network for instance, it is unlikely that Wireshark can be deployed permanently in all the places in order to report packet anomalies, as this tool has not been designed for this task.

In ntopng 2.0 we have added the ability to both dump traffic on disk in pcap format, and create tap interfaces (remember to start ntopng with –enable-taps so that ntopng can create them) to which you can attach Wireshark.

This allows ntopng to stream Wireshark packets in realtime for specific hosts/flows or for specific events (e.g. a portscan).

The wireshark community has asked to make this mechanism even more flexible. Packet-oriented people like to see low-level details such as TCP window size that monitoring folks (such as most ntopng users) do not usually care about. What it has been asked is to make ntopng scriptable and decide via a lua script when a specific flow needs to be streamed to a tap/saved to a pcap file. For instance we have been asked to let users decide based on various conditions such as for instance if the TCP window scale exceeds a given value or if drops are more than a certain threshold. In essence turn ntopng into a more sophisticated Wireshark packet pre-processor.

The second feedback received is to add in ntopng the ability to report about specific low-level details such as the TCP window scaling value or the backlog of in-ACKed packets on a TCP connection.

We are planning to implement these things in the coming months. However if you have further ideas or requirements, please file an issue on GitHub so that we can track it and make ntopng even better than what it is today. Thank you!

This post will teach you how to create a cheap WiFi access point able to enforce layer-7 application protocols. In order to do this you can use a cheap RaspberryPi or BeagleBoard with a USB WiFi stick, or use a x86 PC.

The USB stick we use is the following

# lsusb
Bus 002 Device 003: ID 148f:5370 Ralink Technology, Corp. RT5370 Wireless Adapter


and once plugged onto a USB port it is immediately recognised by Linux (in this post we use Ubuntu Linux but other distro will work as well). As described in this article, you just need to edit /etc/network/interfaces and add something like

allow-hotplug wlan0
iface wlan0 inet static
address 192.168.100.1
netmask 255.255.255.0

as well create your /etc/hostapd/hostapd.conf configuration file similar to ours

# cat /etc/hostapd/hostapd.conf
interface=wlan0
driver=nl80211
ssid=ntop
hw_mode=g
channel=6
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_passphrase=ntop_password
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

Done this we’re ready to start ntopng. Before doing that we need to
# ethtool -K wlan0 gro off gso of
# ethtool -K eth0 gro off gso off tso off


that prevents the network adapter to return to user-space packets large than the MTU (this is a problem because ntopng acts as a bridge and this it cannot transmit packets when they are larger than the MTU). It is now time to start ntopng that will act as a bridge between the wired and the wireless interface:

ntopng -i bridge:eth0,wlan0


Now the interfaces are bridged and wireless clients can connect to the Internet without restrictions.

We use a MacBook connected to the WiFi as a client and we want to prevent our notebook to do any Skype traffic. In order to do that we need to click on the host name, that displays host details.

Then click on the “Modify Host Traffic Policy” that opens a new page where the protocol to drop can be specified for this host.

In the left panel, type Skype, then click on the right arrow to move the protocol under the black listed protocols list. In order to activate the configuration you need to click on “Set Protocol Policy and Shaper”.

As soon as Skype is started, you will see Skype flows being identified by ntopng with a strikethrough font to highlight that the traffic is blocked. You can verify this because the Skype client on the notebook is unable to login onto the Skype network and thus calls and messages are disabled, while all the other protocols work as usual. This is just the beginning as you can add traffic shaping and much more. At any time you can check how much traffic has been filtered by ntopng by looking the the interface statistics under the ntop’s Interfaces menu.

Now you know how to use ntopng to enforce network traffic policies. Using the flexible configuration GUI, you can set default policies on your network (for instance block FaceBook for everyone) and add exceptions (don’t block FaceBook for PC x,y,z). You can do all the configurations using the ntopng web GUI without using any command line tool. Just a couple of reminders:

• You can use ntopng as bridge with two (as just described) or three network interfaces. The only difference is that in two network interfaces mode, the WiFi client is unable to talk with the IP address of the wired interface (i.e. if you want to speak with the ntopng host, your wireless clients must contact the IP of the WiFi interface).
• Two network interface bridging is available only on ntopng packages build on August 3rd 2015 or later.
• ntopng bridging is available only in the pro version of ntopng, so make sure you have a license or that the demo mode is not yet expired.

Enjoy !

This is to announce the release of nDPI 1.7. In addition to many new/updated dissectors, the main change of this release is the ability to identify subprotocols. For instance a DNS request Facebook is not identified as DNS.Facebook (previously only Facebook). This is a great additions for apps that used nDPI to block protocols and that failed due to lack of subprotocol support. We have also revised the core library code so that plugin initialisation is now stored in the plugin itself, making the library core shorter and more readable.

This is the complete changelog of the 1.7 release:

• New dissectors
• QUIC
• WhatsApp Voice
• Stracraft
• Teredo
• Snapchat
• Simet
• OpenSignal
• 99Taxi
• GloboTV
• Various dissectors for tim.br
• provided mobile services
• Deezer
• Instagram
• Microsoft cloud services
• Twitch
• KakaoTalk Voice and Chat
• HotspotShield VPN
• Updated dissectors
• STUN
• NetFlow
• NTP
• Skype
• Oscar
• Facebook
• MPEG TS
• DNS
• SSL
• RTP
• Other changes
• Added support of protocol+subprotocol.
• Dissectors are now initialised inside the dissector code and not in the main code branch
• New API call for converting nDPI protocols IDs to names ndpi_protocol2name()
• Added tool for regression testing (tests/ directory)

At ntop we’ve been playing with NetFlow/IPFIX since more than 10 years and been part of its standardisation. While we acknowledge that concept of flow (a set of packets with common properties such as the same IP/port/protocol/VLAN) is still modern, the NetFlow format is now becoming legacy as we have already discussed some time ago. Modern data crunchers such as those belonging to the big data movement or emerging data storage systems (e.g. Solr or ELK) are  based on the concept that information has to be defined on an open format (usually JSON) and that it has to be stored on systems that are agnostic to the data nature. For years (but for many companies it is still like that today) network administrators considered NetFlow an exception, as it needed to be processed with special tools (often called NetFlow collectors) and that often have an extra price tag, simply because the protocol was “so special”. If you read some books just released, you will realise that even in late 2015 this message is still pushed out by NetFlow vendors. Unfortunately.

At ntop, we believe that this statement is not true, and that NetFlow is not special at all: it is just an old legacy format that makes difficult for people to use flow-centric information that instead is still very modern (with some limitations that we’ll discuss in a future post). That’s all. One of the problems of NetFlow/IPFIX is the protocol that lacks many modern features, and that pushes apps to deal with this binary format. People realised that and started to create conversion tools from NetFlow-to-something so that we can finally use flow information without the legacy of their format. For this reason we have started to add in nProbe new export formats that preserve the flow-information while exporting data in open formats. Currently nProbe can natively export flow-information in:

• NetFlow v5/v9/IPFIX for legacy apps and collectors.
• Log-based textual flow information for people who want to pipe/sed/grep on text.
• Flow-to-MySQL/MariaDB for filling up a DB with flow information, or with periodic batch database import by processing the textual logs using DB-import utilities.
• Flow-to-ElasticSearch export so that nProbe can create indexes and import data into ELK without the need of external log-processing tools such as Logstash that make the process more complicated and with extra latency.
• Flow-to-KAFKA exporting data into the Apache Kafka broker that is used also in networking.
• Flow-to-ZeroMQ export for simultaneously sending flow information in JSON format to non-legacy consumers such as ntopng.

Having a native flow export into the probe, not only removes the need to have conversion tools involved, but it speeds up the operations. It also avoids (at the source) to deal with issues such as various flow formats and dialects (for instance nProbe supports various extensions such as Cisco ASA or PaloAlto) that will transform your NetFlow experience into a nightmare.

In the original ntopng it was possible to navigate historical information using a so called “Historical Interface”. Such interface was a logical network interface able to read flow data from a SQLite archive and present them on the web interface. This approach had various limitations when it was used to navigate data on a long term window, as all flows had to be restored on memory before visualising them, process that can take a lot of memory and time when data cardinality increases.

In the ntopng 2.1 development version, we have completely rewritten the code for historical data exploration. Currently the supported database backends include MySQL and ElasticSearch, so using the -F flag, you can dump flow information on one of these two backends. For instance if you want to use MySQL you just need to do

ntopng -i eth0 -F "mysql;localhost;ntopng;flows;root;"

and ntopng will save flow information in the database. Doing that, ntopng will also display you an extended web interface for navigating though flow information. You can select flows according to various criteria, by selecting the graph icon inside ntopng (local) host and interface pages.

As you can see at the bottom of the graph you will find a small table containing the top flows of selected time period you selected. This table is interactive as all ntopng tables so you can click on column names to sort data, or click on the Info button to visualise the flow information. All activities on this table are not performed on in-memory data but rather on historical data present on the database. For instance if you click on the www.internetfestival.it link under the Info column, you will select all flows which contain such string (e.g. DNS queries or HTTP requests from/to such host) as depicted below.

You can click on the search criteria to change the type of information you want to retrieve, and you will see the results in the table below. The middle table named “Search Results” contains a summary of the retrieved information so that you can narrow your search if necessary. In case you are interested in downloading all flows matching your queries, you can click on the bottom download icon to download all of them. On the current implementation you can easily drill down to flow level and see exactly what flow has created the suspicious traffic peak that you have seen in ntopng traffic graphs.

In order to avoid filling up your MySQL database, under the ntopng preferences menu, you can specify how long you want to keep flows into MySQL so that ntopng can delete for you old flows and thus you can save space. Note that this setting applies only to MySQL as RRD-based graphs are not affected but this setting. This means that you can use MySQL to store short-term flows, while using RRD-based graphs for more long-term analysis.

This implementation is just the beginning of what you will see after ntopng 2.2 release. In fact we plan to add data analytics that go beyond simple data search/retrieval, periodic reports and much more. If you want to contribute, please send us patches and code enhancements and we’ll integrate them.

For years we have optimised PF_RING to support multi-10 Gbit/40 Gbit operations in zero-copy at line rate using ZC. Our users know that using PF_RING they can operate at line rate in RX+TX, balance packets across processes, drop/prioritise traffic etc etc. After a few years where commodity NICs (mostly Intel) combined with PF_RING  have reached basically the same performance of FPGA-based adapters, the rush towards 100 Gbit has revamped interested towards non-commodity NICs. Due to this, you can now find on the market FPGA-based network adapters from companies such as Accolade Technology, CSPI/Myricom, and Napatech. If you want to adopt them in your apps, you need to modify the application code, learn a new API, recode some components. If you want to move to another adapter, you have to start-over and do it again.

In order to avoid this, we have decided to give our PF_RING users the best experience by extending PF_RING to  natively support FPGA-based adapters. This means that you can take your existing PF_RING-based application or popular applications such as nProbe/n2disk, Wireshark, Suricata, Snort, Bro, Argus, and instead of waiting until somebody modifies their code, just run them on top of your favourite NIC. The only thing you have to care about is the device name, all the rest is the same. The names for network interfaces change according to the NIC manufacturer. Below you can read how to start nProbe (our PF_RING based NetFlow probe) when using some adapters we support:

• Accolade Technology: nprobe -i anic:1 (see README)
• CSPI/Myricom: nprobe -i myri:1 (see README)
• Napatech: nprobe -i nt:1 (see README)
• Intel: nprobe -i zc:eth1 (see README)
• Other adapters: nprobe -i eth1

Now you might wonder what are the differences between these adapters, and what are their costs. Well ntop is a software company and a vendor-neutral organisation whose goal is to serve our user community, not hardware vendors. We do not want to endorse product A or product B, but instead offer you the best experience. Our message is simple: if you develop your application on top of PF_RING or if you want to use popular open-source applications your investment will be preserved when moving from adapter A to adapter B. This is because PF_RING shields you from the underlying hardware as it will transparently take care of underlying hardware differences. For instance each network manufacturer who supports hardware timestamps has its own format. Instead of getting crazy supporting all of them in your application, PF_RING does it for you and gives you one single time format regardless of the network adapter.

Bottom line. Choose the adapter you like according to the features you need, and the budget you can allocate. PF_RING will support it. If you want to develop on cheap commodity adapters and then deploy your application at 100Gbit you can do that without changing a single line of code, but just changing the device name. We believe that this is a unique PF_RING feature: let end-users choose what NIC they like most for a project without being locked by a single vendor.

If you have the chance to come to the upcoming OISF 2015 Suricata conference, you can attend our speech about PF_RING. We would be glad to explain all this in detail. Hope to see you in Barcelona next month!

PS. if other network manufacturers would like their NIC be supported in PF_RING, of course they can contact us.

In the past few years we have tried to open PF_RING in an attempt to turn it into the “new pcap” API for packet processing. Recently we have added native support for speedy FPGA-based NICs and thus created a single API for efficient NIC-independent packet processing.

If you are interested in hearing more about this subject, you can have a look at the slides or watch the video of our presentation, held in Barcelona at the Suricata Conference 2015.

Enjoy!

This is to announce the release of PF_RING 6.2 that has several improvements with respect to the previous version. As previously accounted, we have extended support of non-Intel devices in PF_RING to provide you the best experience supporting many new devices (and a few more will come in the following months). We have specialised PF_RING for FPGA-based adapters, and added support of 100 Gbit adapter such as those manufactured by Accolade technology and Napatech. As you might have noticed, we have moved release versioning to odd/even numbers. An even minor version means that this is a stable release, whereas odd numbers are used for development versions. This means that while 6.2 is the current stable version, the development version will be 6.3.

Below you can find the complete changelog of this new release.

• PF_RING Library
  • Accolade Technology support
  • Myricom/CSPI NICs (ASIC/FPGA) support with APIv4
  • Napatech module compatibility with ntanl v.4.0.1
  • New API pfring_set_packet_slicing for packet slicing (hw support when available)
  • New pfring_open flag PF_RING_ZC_IPONLY_RSS to compute RSS on IP only (not 4-tuple)
  • Reworked pfring_recv_chunk API to handle generic segments
  • Packet-mode with Napatech using pfring_recv also when card is configured as PCAP
  • Improved pfring_print_parsed_pkt with STP support
  • Improved software filtering rules stats
  • Fix for capturing tx packets in stack mode
  • Removed libnuma dependency (using native numa support in PF_RING ZC)
  • Added pfring_config tool to print includes and libs used in the current PF_RING configuration
  • Fixed loops in stack mode (e.g. stack injected packets captured from kernel in 1-copy mode)
  • Fix for IPv6 in pfring_hash_pkt
  • License update to LGPL 2.1
• ZC Library
  • New API pfring_zc_set_rxfh_indir to reprogram RSS Redirection Table
  • New pfring_zc_open_device flag PF_RING_ZC_DEVICE_CAPTURE_TX to capture also TX direction on standard devices
  • Fixed e1000e caplen
  • Fixed i40e TX
  • Fixed device detach
  • Fixed DAQ-ZC IPC detach
  • Fix for re-running recv after a breakloop in pfring_zc_queue_breakloop
  • Fix for nanoseconds in case of software timestamps with ZC devices
• DNA Library
  • Fixed cluster queues initialisation in Libzero
• PF_RING-aware Libpcap
  • Fixed pcap_brekloop (tcpdump now handles sigterm correctly when there is no traffic)
• PF_RING kernel module
  • Compilation fixes for FC 22 with kernel 4.x
  • Fixed BPF support in Centos 7 (causing crashes)
  • Fixed concurrent /proc updates and interface name clashes
• PF_RING-aware/ZC Drivers
  • New e1000e driver v.3.2.7.1 with kernel 4.x support
  • New igb driver v.5.3.2.2 with kernel 4.x support
  • New ixgbe driver v.4.1.5 with kernel 4.x support
  • New i40e-zc driver v.1.3.39.1 with kernel 4.x support
  • Fixed rx stats in ifconfig/proc/ifstat/etc.
  • Fixed stats with X540
  • ixgbe: forcing drop_en on all queues when an rx queue is open
• DNA Drivers
  • New e1000e driver v.3.2.4.2
  • New igb driver v.5.3.2.2
  • New ixgbe driver v.4.1.2
  • igb hw timestamp change from 40 bit to 64 bit full timestamp
• Examples
  • zbounce: printing direction
  • zbalance_ipc: optimisation to reduce time-pulse thrad load on non-time-sensitive applications
  • pfsend: linux cooked packets support
  • pfsend: ability to specify egress rate in pps
• Snort DAQ
  • Fixed direction and L2 header in the inject function

After over 6 months of work, we’re pleased to announce the release of ntopng 2.2 (as already discussed even numbers identify stable releases whereas odd number development versions). The goal of this release has been to consolidate the existing work, fix issues reported by users, improve the reports we have introduced in 2.0 and pave the way for the next development iteration where we plan to add new features (we’ll present the roadmap in the next few weeks). The main new features of this release is the introduction of traffic profiles, new historical interface redesigned from scratch, various reports improvements, support of layer-2 traffic analysis,integration with NetFilter (Linux only).

Enjoy!

Changelog:

• Implementation of BPF-based traffic profiles (similar to NFsen named filters) , logical flow-based aggregations — e.g., Facebook traffic originating at host X. Real-time statistics as well as historical data are collected for each traffic profile
• Added a fine-grained network traffic breakdown that captures and stores ingress, egress, and inner traffic for each local network
• Redesign of historical interfaces. Historical interface data have been seamlessly integrated with real-time data
• Historical flow dump and runtime drill-down of historical data with support for MySQL and ElasticSearch
• Built-in support for protocols:
  • CAPWAP (Control And Provisioning of Wireless Access Points, <https://tools.ietf.org/html/rfc5415>)
  • BATMAN (<http://www.open-mesh.org/projects/open-mesh/wiki/BATMANConcept>)
  • TZSP (TaZmen Sniffer Protocol)
• Added SIP and RTP protocols information in flow details
• Additional MAC-based host calassification
• Added support for Linux TUN/TAP devices and integration with Linux Netfilter for capturing/dropping packets relying on the Linux firewall.
• Extended and improved supported OSes: EdgeOS, Centos 6/7, Ubuntu 12.04/14.04, Debian, Windows x64, Raspbian (Raspberry)
• Extended and improved supported architectures: x86, x86-64, MIPS, ARM.
• Documentation and User Guide significanly improved
• Added a great deal of READMEs, including ElasticSearch, bridging, traffic shaping and policing, NetBeans development
• Improved stability both under normal and high network loads
• Fixed many minor bugs

For all our Italian-speaking friends, yesterday I have been interviewed by an Italian magazine. This is the link to the interview.

Enjoy!

This tutorial shows how to properly configure nagios and ntopng (Professional) in order to send asynchronous ntopng-generated alerts to nagios.

Prerequisites

It is assumed that the following software is already installed and properly configured:

• nagios daemon
• nagios NSCA (Nagios Service Check Acceptor) daemon
• ntopng Professional

Please see the Resources section at the bottom of this page for useful links and guides on how to set-up nagios and NSCA daemons.

Tutorial Set-Up

This tutorial uses two hosts connected to the same LAN:

• An x86_64 ubuntu 14.04 host running nagios and nagios NSCA daemons (ip address 192.168.1.10). Daemon executables are installed under /usr/local/nagios/bin, whereas configuration files are stored in /usr/local/nagios/etc.
• A Mac Os X host running ntopng (ip address 192.168.1.6)

Nagios NSCA Configuration

NSCA daemon configuration is pretty straightforward. The only thing that is recommended is to enable the debug in the nsca.cfg file. Open the file /usr/local/nagios/etc/nsca.cfg, locate the line debug=0 and change it to 1:

[...]
# DEBUGGING OPTION
# This option determines whether or not debugging
# messages are logged to the syslog facility. 
# Values: 0 = debugging off, 1 = debugging on

debug=1
[...]

Then, save the file and restart the NSCA daemon. Once the debug is enabled, useful NSCA status information is sent to the syslog.

Nagios Configuration

In order to properly handle ntopng-generated alerts, nagios requires some extra configuration.

Nagios command

The first thing to do is to add a nagios command. The command is very simple since all the alert logic resides inside ntopng. Open up the nagios command.cfg file (/usr/local/nagios/etc/objects/commands.cfg) and add the following lines at the bottom:

define command{
 command_name check_dummy
 command_line $USER1$/check_dummy $ARG1$
}

Save and close.

Nagios Service

The second thing is to define a service template for asynchronous, passive services. This service will be extended by the ntopng host defined in the following section. Open the file templates.cfg (/usr/local/nagios/etc/objects/templates.cfg) and add the follwing service definition at the bottom

define service{
 use generic-service
 name passive_service
 active_checks_enabled 0
 passive_checks_enabled 1 # We want only passive checking
 flap_detection_enabled 0
 register 0 # This is a template, not a real service
 is_volatile 0
 check_period 24x7
 max_check_attempts 1
 normal_check_interval 5
 retry_check_interval 1
 check_freshness 0
 contact_groups admins
 check_command check_dummy!0
 notification_interval 120
 notification_period 24x7
 notification_options w,u,c,r
 stalking_options w,c,u
}

The service defined above has only passive checks and uses the previously created check_dummy command.

Nagios Host

Now it is possible to add a new nagios host definition for the host running ntopng. Create the file ntopnghost.cfg under /usr/local/nagios/etc/objects/ with the following contents:

#
# HOST DEFINITION
#
define host{
 name generic-ntopng ; The name of this host template
 use generic-host ; This template inherits other values from the generic-host template
 check_period 24x7 ; By default, Linux hosts are checked round the clock
 check_command check-host-alive ; Default command to check Linux hosts
 check_interval 5 ; Actively check the host every 5 minutes
 retry_interval 1 ; Schedule host check retries at 1 minute intervals
 max_check_attempts 10 ; Check each Linux host 10 times (max)
 contact_groups admins ; Notifications get sent to the admins by default
 register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
 }

define host{
 use generic-ntopng ; Name of host template to use
 ; This host definition will inherit all variables that are defined
 ; in (or inherited by) the linux-server host template definition.
 host_name ntopng-host
 alias Ntopng Host
 address 192.168.1.6
 }

#
# HOST GROUP DEFINITION
#
# Define an optional hostgroup for Linux machines
define hostgroup{
 hostgroup_name ntopng-servers ; The name of the hostgroup
 alias Ntopng Servers ; Long name of the group
 members ntopng-host ; Comma separated list of hosts that belong to this group
 }

#
# SERVICE DEFINITION
#
define service{
 use passive_service
 service_description NtopngAlert
 host_name ntopng-host
}

In the host definition part of the configuration file above, it is created a generic-ntopng host template that may be used for one or more hosts running ntopng. The second host definition, that inherits from generic-host, is the actual ntopng host. The host_name has been set to ntopng-host, and its ip address correspond to the Mac Os X machine running ntopng. It is important to jot down the host_name as it is necessary to set it inside ntopng configuration as well. Please make sure to change the ip address to the one of the host running your ntopng instance.

In the host group definition part of the configuration file, it is defined an host group for all the ntopng servers. Currently, the ntopng-servers group has only one host, namely, ntopng-host, but one may create a larger group with more than one ntopng monitoring host.

The third part, service definition, defines a service inheriting from the passive_service created in the section above. This service has a service_description equal to NtopngAlert and is associated to the host ntopng-host. The service_description string NtopngAlert should be kept in mind as it is required to set it into ntopng as well.

In order to tell nagios to read and use the ntopnghost.cfg file, it may be necessary to add the following line to the file (/usr/local/nagios/etc/nagios.cfg):

# Definitions for monitoring the ntopng host
cfg_file=/usr/local/nagios/etc/objects/ntopnghost.cfg

Nagios restart

Nagios configuration is done. To make sure the are no syntax errors in the edited files, we can run the following check

 /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

If everything is OK, nagios will output something like

Total Warnings: 0
Total Errors: 0
Things look okay - No serious problems were detected during the pre-flight check

If no errors and warnings are detected, then it is safe to restart the nagios daemon.

Ntopng Configuration

The host running ntopng must have the send_nsca utility. This utility is bundled and shipped with the sources of NSCA. Alternatively, it can be found in the repositories of the most common distributions.

The host used in this tutorial has the utility installed under /opt/local/sbin:

Simones-MacBook-Pro:nsca simone$ which send_nsca
/opt/local/sbin/send_nsca

We will drop this absolute path inside ntopng configuration page. It is also important to take note of the send_nsca configuration file absolute path. This is a very basic configuration file and doesn’t need to be edited but it is required by send_nsca to work properly. In the host used in the tutorial this file resides in /opt/local/etc/nsca/send_nsca.cfg

At this point no additional information is needed. Fire up the ntopng web interface and navigate to the Preferences page. There is a whole section dedicated to the nagios alerts.

The following configuration fields are available

Alerts To Nagios: Toggles nagios alerts on or off

Nagios NSCA Host: This field must be filled with the ip address of the host running the nagios NSCA daemon. In this tutorial the host is 192.168.1.10

Nagios NSCA Port: The port on which the NSCA daemon is listening. The daemon used in this tutorial listens on the default port 5667.

Nagios send_nsca executable: The absoulte path of the send_nsca executable, which is found in /opt/local/sbin/send_nsca in the ntopng host used in the tutorial

Nagios send_nsca configuration: The absolute path of the send_nsca configuration file, which is found in /opt/local/etc/nsca/send_nsca.cfg in the ntopng host used in this tutorial

Nagios host_name: The host_name exactly as specified in the nagios host definition for the ntopng host. The host name used in this tutorial is ntopng-host, as it has already been discussed in the nagios configuration section.

Nagios service_description: The service description exactly as specified in the nagios passive service definition for the ntopng host. The service_description used in this tutorial is NtopngAlert, as it has already been discussed in the nagios configuration section above.

At this point everything should be set up properly. Go to the ntopng web interface and arm an alert for a host of for a network. You will see it appearing both inside ntopng alerts page, as well as in the nagios web interface.

Since we have enabled NSCA debug, every time ntopng propagates an alert to nagios, the file /var/log/syslog of the nagios host should display a bunch of lines similar to the following:

Dec 13 12:01:00 simone-VirtualBox-1 nsca[17033]: Connection from 192.168.1.6 port 7410
Dec 13 12:01:00 simone-VirtualBox-1 nsca[17033]: Handling the connection...
Dec 13 12:01:00 simone-VirtualBox-1 nsca[17033]: SERVICE CHECK -> Host Name: 'ntopng-host', Service Description: 'NtopngAlert', Return
 Code: '0', Output: '[WARNING][THRESHOLD EXCEEDED] Threshold egress crossed by network 192.168.1.0/24 [134801 > 10]'
Dec 13 12:01:00 simone-VirtualBox-1 nagios: EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;ntopng-host;NtopngAlert;0;[WARNING][THRESHO
LD EXCEEDED] Threshold egress crossed by network 192.168.1.0/24 [134801 > 10]
Dec 13 12:01:00 simone-VirtualBox-1 nagios: PASSIVE SERVICE CHECK: ntopng-host;NtopngAlert;0;[WARNING][THRESHOLD EXCEEDED] Threshold e
gress crossed by network 192.168.1.0/24 [134801 > 10]
Dec 13 12:01:00 simone-VirtualBox-1 nsca[17033]: End of connection...

The syslog file is also very helpful when troubleshooting possible issues or missing alerts.

Resources

• Official guide to install nagios from sources on ubuntu.
• Tutorial to set-up the Nagios NSCA daemon.

2015 has been a year full of activities that allowed us to consolidate our tools and thus provide a better service to the community. In 2016 the plan is the following:

1. 100 Gbit
   As in 2015 we have added support for 40 Gbit in PF_RING, 2016 will be the year of 100 Gbit. We already support the Accolade and Napatech 100 Gbit NICs in PF_RING, but the plan is to make 100 Gbit commodity, and thus as soon as the new Intel Red Rock Canyon adapters will be available (we expect them Jan or Feb at latest) we will support them in PF_RING. This new adapter is very interesting as it supports various speeds 10/25/40/100 Gbit and it integrates an ethernet switch that we plan/hope to use to offload some tasks to the adapter instead of using the main CPU. In addition to Intel RRC we are adding support of additional 100Gbit adapters such as InveaTech 100Gbit adapters.
2. nProbe Cento
   As happened years ago when moving from 1 Gbit to 10 Gbit, the 100 Gbit challenge does not mean just more speed but it is a complete redesign of applications. Thanks to innovation in computing and to a mature PF_RING ZC framework, we want yo make 100 Gbit and multi-10Gbit monitoring commodity. For this reason next week at the Flocon conference, we will present a new version of nProbe named cento, that it is able to generate flows at 100 Gbit on a standard Intel-based server. This efficiency has allowed us to handle 10 Gbit of traffic (500k concurrent flows with ingress traffic of 14.88 M pps) on a single CPU core, that means for instance that you can do 40 Gbit netflow monitoring using a sub 1000$ Intel E3 server.
3. Flow monitoring and Security
   Last november at the Suricata conference, we have demonstrated for PF_RING can successfully accelerate applications such as Suricata, Snort and Bro. As often people want to have both flow evidence (on 100% of traffic) and run IDS on selected traffic (e.g. all but encrypted traffic), in cento we have built and engine that allows to do exactly this. This will promote IDS scalability (currently they can hardly handle 10 Gbit) at higher speeds while avoiding spending un-necessary time analysing not interesting traffic for an IDS (e.g. YouTube or Netflix traffic).
4. ntopng
   In the current development version of ntopng, we have implemented full Nagios support and supported nfsen-like filtering (soon we’ll add a post about it). This year we want to integrate ntopng with pfsense for classifying traffic that the firewall can then selectively drop, add traffic categorisation (e.g. divide traffic in categories such as social network, news, business…) so that we can drop/prioritise traffic not only based on application protocols but also based on information content. Another area of interest for most of our users, is the ability to classify traffic in categories (e.g. social network, sport, chat, …)  and decide what users can access what information; this is particularly interesting for schools and children so that inappropriate content is blocked. In essence we want to extend ntopng inline capabilities introduced with v2 to the next level to make this tool even more flexible.
5. Affordable Sensors Everywhere
   As our users know, one of the main ntop goals has been to make commodity what used to be very expensive. This year we want to combine ntopng/n2disk/nprobe (not all components will be necessary, the minimum is ntopng) to create a simple and user-friendly system able to serve needs of small networks as of a large enterprise. People should be able to permanently monitor their network activities by building themselves a network sensor based on the ntop software. In the current ntopng git development branch you can already see a preview of pcap-extraction capability integrated with flow-search stored by ntopng int MySQL.

For those attending the FloCon 2016 conference, we will organise a ntop BoF Wedn at 5.30PM where  we will cover this roadmap more in detail.

Stay tuned!

Topic: ntop Meetup – Affordable High-Speed Sensors Everywhere

After the Flocon 2016 meetup we held this week, next week we’re organising a meetup for our users of the bay area where we will discuss a new arguments including high-speed sensors and network analytics.

For those who have not attended the Flocon meetup (~25 people attended it, for a 30 min presentation followed by a 2h open discussion), the main comments have been:

• Network monitoring has to be integrated with security tools: people demand small machines able to visualise network traffic metrics as well report potential security violations.
• Inline monitoring we introduced in ntopng 2.x is interesting for many people, in particular small/medium business. We’ve been encouraged to move forward and introduce new features such as drop of selected domains/web-sites.
• The way to scale for CPU-bound applications (e.g. an IDS) happens reducing the ingress traffic the app has to process. The upcoming cento application is going exactly in this direction: user-configurable filtering of selected traffic (e.g. Netflix or encrypted traffic) so that applications such as n2disk or an IDS can operate on less traffic (saving on disk GB or SSL traffic isn’t that useful).
• We have been encouraged to progress with our 2016 roadmap as users are looking forward using low-cost sensors as the number of monitored devices is constantly increasing and thus monitoring costs must go down.

See you in San Francisco!

Last week we have previewed at FlowCon 2016 conference our new 100 Gbit probe called nProbe cento (cento is 100 in Italian). You can find our presentation slides here.

We believe that it is important to combine flow monitoring with security and packet to disk. This in an integrated manner, and not by using different un-correlated applications. Cento is the next generation probe able to generate flows at 100 Gbit line rate using an x86 PC and a 100 Gbit NIC on top of PF_RING ZC, while being able to selectively send packets to IDSs or packet-to-disk applications. In essence what we want to achieve with cento is: flows for everything, packets for something.

While we plan to release Cento sometime this spring, for those interested in testing it, you need to go to packages.ntop.org and install the development version of the nProbe package that includes a new binary named cento.

Stay tuned!

One of the great features of nfsen is the ability to specify filters for identifying specific traffic and thus aggregate and graph it. In ntopng we aggregate traffic per host and networks. However sometimes you want to aggregate using other criteria. Examples include:

• Traffic from host A to host B
• VPN traffic sent from host X to concentrator Y
• Facebook traffic sent from iPad 192.168.13.4
• ntopng web traffic

In order to implement these measurements, in ntopng we have introduced the concept of network profiles. Each profile is defined using the “Traffic Profiles” entry of the preferences menu of ntopng professional.

The preferences page contains a list of profiles defined with name and filter.

Contrary to nfsen that used its own syntax, here we decided to use BPF (used in tcpdump and wireshark) to define profiles. We have extended it so that you can include also the l7proto statement that is used to characterise nDPI protocols. Once you have saved the protocol you can see stats about it from the interface stats.

or from the flow page where flows are marked with the profile name.

Using this feature you can aggregate the traffic the way you want, as long as you can define the profile using a BPF filter, thing that is pretty likely to happen.

Enjoy!