Exactly 3 years elapsed from the introduction of nEdge (ntopng Edge), and despite the fact we haven’t posted much about it in our blog, this tool continued to grow, many features have been added over time, and we see that every time new users have the chance to try it, they are amazed about the capabilities it provides.

If it’s the first time you hear about nEdge, we suggest to read the introductory post which explains how nEdge enables Network administrators to enforce policies at Layer-7 on network users, the nEdge product page which is providing a summary of the features, and the User’s Guide if you want to get the details of the implementation, deployment, and policies configuration.

In this post we want to update you about some of the latest features we added to nEdge in the past months, as a result of a few inquiries we received for easying the deployment in Networks with multiple (physical or virtual) LANs. In fact, although it was already possible to configure multiple WAN interfaces in nEdge, to handle dynamic Multi-Path Routing, it was not possible to configure multiple LAN interfaces to connect multiple internal Networks (e.g. the staff and the guest network).

A releated feature that was also missing, is the ability to configure VLAN interfaces, which is also a key feature when handling multiple LANs (or WANs) as in most cases local network are virtualized and all traffic transits on a single link. Although it was possible to deploy nEdge as transparent bridge and enable the VLAN Trunk Bridging mode to handle traffic policing on a VLAN Trunk, it was not possible to configure and manage VLAN interfaces and treat them similar to physical interfaces when working in routing mode. This lets us save ports on the switch, and use a machine with only 2 ports for running nEdge.

Let’s see how all the above has been addressed in practice, and what are the steps to follow to configure and deploy nEdge in different network scenarios.

VLAN Configuration

As described in the Getting Started section of the User’s Guide, after installing and logging in the GUI for the first time, the system should be configured from the system interface, System -> System Setup page. Here it is possible to configure the Operating Mode (e.g. Routing) and select the Network Interfaces. At this point, after identifying the Network interfaces connected to our LAN or WAN Networks, it is possible to configure VLAN interfaces through the VLAN Configuration menu.

DHCP Service

nEdge provides the ability to configura a DHCP server for assigning IP addresses to the local Network. With the introduction of multiple LAN interfaces, we also added the ability to configure multiple DHCP servers to be able to assign IP addresses for all the different subnets. In order to do this, in the Interfaces Configuration section it is possible to assign the IP address to each LAN (or VLAN) interface, while in the DHCP Server section it is possible to select the IP address range and enable the service for each LAN (or VLAN) interface.

Multipath Routing

As described in the introduction, nEdge implements dynamic Multi-Path Routing, according to user-defined routing policies applied to the configured gateways. In most cases, a single WAN port is enough to route the traffic through one or more gateways, however this works in nEdge as long as all the gateways are on the same Network. If they are not on the same network, but you still want to use a single WAN port, it is now possible to use VLANs to configure one virtual Network interface for each gateway, and use them to configure routing policies.

Inter-LAN Filtering Rules

Having multiple local networks routed through nEdge introduces a new requirement, which is the ability to keep Network zones isolated, as they are usually used to segment the Network and control access to resources by means of Firewall policies. For this reason we introduced in nEdge the ability to define a Default Policy for Inter-LAN traffic, and define exceptions by means of filtering rules (e.g. accept traffic from Source IP A to Destination IP B or Local Network C). This can be configured from the System -> Inter-LAN Filtering Rules page.

MDNS forwarding

One last feature we added in this context is the support for multicast traffic forwarding. Multicast traffic, used for instance by the mDNS service (used to resolves hostnames within local Networks with no need of a local name server), is not forwarded between local networks. In nEdge we added the ability to broadcast this traffic, forwarding selected multicast packets according to the defined policies, between LAN interfaces. This allows zero-conf devices to keep working also across subnets and can be configured from the System -> Multicast Forwarding page.

Have fun with nEdge!

This is to announce that the registration for the ntop Conference 2023, 25 years since the first release of ntop, is now open.

Similar to past conferences, this event is divided into two days: the first day will be allocated for training on ntop products, the second day for the main conference and workshop.

You can read the conference and training agenda at the ntopConf 2023 page from which you can also reserve your seat.

Finally a few notes. In order to make this event effective we have decided that:

• The official conference language will be English. Non-English sessions will be limited and clearly marked. This is in order to make this event international and open to the whole ntop community.
• Conference will still be a free event (no admittance fee) but subject to registration.
• In the past, a large number of people registered to the event and didn’t show up, with an impact on organisation and logistics. A few days ahead of the event we will contact people, and those who will not confirm their participation will be removed from the attendance list.
• Record all conference sessions and make them available as video after the conference.
• We do not plan to have a live streaming as this is supposed to be a live event, where people should come, meet, and discuss.
• The training session will be limited in terms of number of seats. This is because we want to have a small group of motivated people that are somehow familiar with ntop tools or the topics (network visibility and cybersecurity). For this reason for training we will ask a small donation (from 1 Euro up) to make sure that if you register, you will also attend it. In other words, as it takes time to prepare the training, we want people to come motivated and willing to do exercises and learn.

Hope to see you in person in Pisa to celebrate the 25th anniversary of ntop !

ntopng has native blacklist support that enables generation of alerts when malware sites are contacted. You can enable/disable the list of active blacklist by accessing the blacklist page from the preferences menu of the left sidebar

and also configure the list properties such as refresh rate as well enable/disable them.

Now suppose you want to trigger an alert when contacting a specific IP address or a website (this regardless if using clear-text protocol such as HTTP or encrypted TLS-based communications). How can you do that? See it below:

1. Define a text file containing the website and/or IP addresses you want to be used to trigger an alert. The file format changes according to the filtering type: for IP based filtering this is the format to use, whereas for host/domain based filtering you can use this file as example. You can define one of them (or both) according to your needs.
2. Put the custom blacklist files you have defined on a web server that ntopng can access.
3. Now define a custom configuration file for each of your blacklist as described at this page. Doing this, you need to set the URL that you have used at the previous step in order ntopng to load the custom blacklist you have defined.
4. At this point ntopng is instructed to trigger an alert whenever traffic matches the custom IPs/hostnames that you have defined. You can find those alerts in the Flow page of the Alerts Explorer on the left menu sidebar.
5. Note that if you want to modify (e.g. add an IP or remove a hostname) your lists, you need to wait until ntopng reloads the list according to your preferences (e.g. daily), or you can force immediate blacklist reload by clicking on the reload button in Blacklist preferences.

If you have created custom blacklists you want to share with the ntop community, please do not forget to send us a pull request on github.

Enjoy !

ntopng creates timeseries for traffic by periodically (e.g. every minute) writing into RRD/Influx the traffic volume observed. Below you can see an example.

Traffic is used to keep track of the data volume exchanged. Over time timeseries are aggregated (roll-up) to save space, meaning for instance that 60 minute observations are used to compute a hourly observation. A timeseries rollup involves summarising the original time series data over larger time intervals. The purpose of doing a rollup is to reduce the volume of data and make it more manageable while preserving important information. Different rollup methods can be used, including:

1. Summarization: Summing up the values over each time interval.
2. Averaging: Taking the average of values within each interval.
3. Max/Min: Retaining the maximum or minimum value within each interval.
4. Sampling: Taking a sample data point within each interval.

The results is a new timeseries with a lower resolution as it has only one point per hour, instead of 60 individual observations. As in the hourly and minute timeseries the area under the curve (i.e. the volume of data exchanged) must be the same regardless of the timeseries resolution, the hourly point is computed as the average hourly value (among the 60 minute observations).

This means that if you display the above (last day) timeseries at a different resolution (e.g. week) as the average value is used, the timeseries points peaks will be lowered as each aggregated point will be computed as the average value of the period, thus smoothing the curve. The result is that the above timeseries when shown at 6 hour resolution reports 1.37 Gbps for TX, whereas the previous timeseries (daily) reported 0.95 Gbit.

This said, the traffic timeseries are good for computing data volume but not for handling throughout due to the use of average rollup. For this reason we have introduced (in dev branch, and soon stable when the new a new family of “Throughput” timeseries that are used to store throughput.

In this timeseries family, the rollup function is MAX instead of AVERAGE in order to make sure that the top throughput of the observation period is properly computed. This is the timeseries to use (instead of traffic) in case you care about the maximum observed period value that is usually the case with throughout.

Enjoy !

In the paper How the Great Firewall of China Detects and Blocks Fully Encrypted Traffic it is described a technique used in censorship to identify and block fully encrypted protocols. This technique, limited to TCP flows, uses a few techniques that are applied on the first TCP packet with payload, making it fast and convenient although with a small (< 1%)  percentage of false positives:

6 TCP 192.168.2.100:50860 <-> 185.88.236.110:5222 [proto: 305/Threema][IP: 305/Threema][Encrypted][Confidence: Match by IP][DPI packets: 13][cat: Chat/9][8 pkts/775 bytes <-> 5 pkts/472 bytes][Goodput ratio: 31/28][60.00 sec][bytes ratio: 0.243 (Upload)][IAT c2s/s2c min/avg/max/stddev: 1/29 9996/31 59845/33 22293/2][Pkt Len c2s/s2c min/avg/max/stddev: 66/66 97/94 257/146 62/33][Risk: ** Fully encrypted flow **][Risk Score: 50][Plen Bins: 0,50,25,0,0,25,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]

Enjoy !

While the concept of blacklist is very simple and many people are familiar with it, we know very little of how such blacklists have bene created. In particular we have asked ourselves a few simple questions:

• How effective are publicly available IP blacklists when used on my network?
• Can I trust them to block traffic or they contain false positives?
• Are they outdated or they are updated regularly?

In order to answer to the above questions, we have made a large scale study whose goal was to assess publicly available blacklists. The result of this study are presented in this paper that was presented at 6th International Workshop on Cyber Security (CSW-2023). Those who have no time to read the paper can glance through the presentation slides. The conclusions of the study are:

ntopng emits alerts in order to report relevant. They can be triggered by traffic thresholds, user scripts, behavioural checks, or due to Security issues, including those detected by IDS systems integrated with ntopng (the full list of built-in checks, and related alerts, that can be enabled in ntopng is available in the Alerts section of the documentation). Sometimes they are really critical and should be handled immediately to fix the problem, this is the case of Security events for instance (e.g. a compromised host that must be sanitized as soon as possible). In other cases they do not really need a manual intervention, as they report small issues with low severity, such as minor Network performance issues (e.g. a connection issue).

Alerts generated by ntopng can be notified to messaging systems or in many other ways using one of the Endpoints and Recipients available in the Notifications engine, which allows you to control which alerts should be delivered to each end recipient (e.g. based on the family, category, severity, hosts, etc.). However, in addition to notifications, alerts can also be inspected at any time through the Alerts Explorer, available in the ntopng user interface. The explorer allows you to navigate alerts by family, in a selected time range, and applying filters, based on many criteria, to be able to query the alerts database and retrieve what you need, in a few, simple steps.

In ntopng 5.6, and earlier versions, alerts for each family were organized into 3 tabs:

• Engaged
• Past
• Acknowledged

The Engaged tab was listing alerts related to ongoing issues, triggered and not yet released (e.g. a threshold exceeded, where the actual traffic is still above the threshold and not yet returned to normal levels). The Past tab was listing all historical alerts, related to past events which have been stored in the database. Then the user had the possibility to acknowledge alerts and move them to the Acknowledged tab, to leave in the Past tab only those that were not yet “processed” and “marked” by the user. This in theory was a good system to enable alerts management. As soon as there was an alert, the user was supposed to go to the Past tab, read them, and, after fixing the issue, mark them as acknowledged. Thus the Past tab was supposed to be almost empty. However, in practice, with the addition of several traffic checks to ntopng, on huge networks with many of them enabled, the system is usually reporting a large number of alerts, where most of them are minor Network issues that do not really require manual intervention for each of them individually, and this makes impossible to keep the Past tab empty.

For the above reasons, ntopng 5.7 introduces a new “inbox” design for the Alerts Explorer. The new interface consists of 3 new tabs: Engaged, Unread (“Require Attention“), All.

The Engaged tab (hourglass icon) remains unchanged, reporting alerts related to ongoing issues.

The All tab (drawer icon) reports all historical alerts, similar to an archive, including both unread and read/acknowledged alerts.

The Unread, or “Require Attention“, tab (eye icon) instead, lists only alerts that are supposed to require the user intervention (e.g. Security related) and that are very important. In fact, the system, for each individual check, decides whether an alert should be emitted as “unread” (for those that require attention) or already “read” (for those that are not really important to be analyzed by the user). For instance, security-related alerts are likely to appear in the Unread tab, while alerts related to minor Network issues will probably be archived in the All tab. However, minor alerts, contribute to the score of the host affected by those issues, and if the problem persists and occurs many times, you will definitely notice them as the score on the related host keeps growing (and an alert with higher severity is triggered as a consequence of this).

In short, the Unread tab is what the user should check, daily, to acknowledge alerts that require attention. In any case, all the alerts can still be inspected from the All tab.

We hope that this new design will simplify alert management and enable users to see what alerts are relevant without digging all of them.

Enjoy !

In a previous post, What’s In The (Alert) Inbox?, we’ve discussed how alerts are organised in the Alerts Explorer. The new “inbox” design allows us to cluster alerts into separate folders high-priority events, that require attention and needs to be addresses as soon as possible, from other minor events. This solves one issue: having all critical alerts under control, while still tracking and archiving all minor Network issues (that contribute to the hosts score, and may be still of interest when drilling down during our analysis).

In a system which analyses traffic of a large network, the amount of alerts (in particular those which are about Network issues) can be high, also depending on the number of Behavioural Checks we enable, and how we tune them. Checking the health of the Network, by looking at those alerts one by one, may definitely be a challenging and time-consuming task. For this reason some time ago we introduced in the Alerts Explorer the computation of the Top Alert Types, Clients, Servers, etc, to summarize what are the main issues affecting our networks, and who are the main actors.

This was really useful in understanding at a glance what’s going on. However this is somehow limited, as the information they provide are in many cases not enough for a real analysis and we still need to go through every single alert. For this reason we decided to add support for something more flexible: the ability to define custom queries.

With custom queries, the Alerts Explorer provides the ability to define additional views on top of the full list of alerts, which allows you to aggregate alerts based on arbitrary criteria for example. They are defined as “SQL Queries” using a simple JSON syntax, placed as .json files on the filesystem (this to promote easy extensibility from end-users), and automatically appear in a dropdown under the Queries section in the Alerts Explorer. It is possible to build a query which groups alerts based on Client, Server and Alert Type for instance, and list all alerts matching a specific 3-tuple from the table Actions.

ntopng already provides a few built-in custom queries for Flow and Host alerts. The corresponding definition is available on the filesystem as JSON files under /usr/share/ntopng/scripts/historical/alerts/{flow, host}/*.json

Adding a new custom query is as simple as placing one more JSON file under the same folder. Let’s see as an example the definition of a query grouping flow alerts on Client IP, Server IP and Alert Type:

{
    "name" : "Client / Server / Alert Type",
    "i18n_name" : "contacts_by_type",
    "select" : {
        "items" : [
            {
                "name" : "cli_ip"
            },
            {
                "name" : "srv_ip"
            },
            {
                "name" : "alert_id"
            },
            {
                "name" : "count",
                "func" : "COUNT",
                "param" : "*",
                "value_type" : "number"
            }
        ]
    },
    "groupby" : {
        "items" : [
            {
                "name" : "cli_ip"
            },
            {
                "name" : "srv_ip"
            },
            {
                "name" : "alert_id"
            }
        ]
    },
    "sortby" : {
        "items" : [
            {
                "name" : "count",
                "order" : "DESC"
            }
        ]
    }
}

The JSON format is self-explanatory: you can define the columns to show under the select tree, the columns on which the group by is applied under the groupby tree, and the default column on which sorting is applies under the sortby tree. Aggregation functions can also be defined as you can see for the count item, which is used in the example to display the number of alerts for each 3-tuple. For more complicated examples we recommend to take a look at the built-in query definitions available in the same folders.

For a full list of columns, please check the database schema under /usr/share/ntopng/httpdocs/misc/alert_store_schema.sql

Enjoy!

This is to announce a new PF_RING release 8.6 !

This stable release introduces a new Runtime component in PF_RING, which adds support for runtime filtering. This allows an external application to push filtering rules (through a Redis queue) while the socket is running, and offload them to the adapter when supported (e.g. on NVIDIA/Mellanox Connect-X adapters). This enables Zeek and Suricata “on-demand” at 100 Gbit as discussed in a previous post.

This release also adds support for Debian 12 and latest 6.x kernel shipped with Ubuntu 22 LTS. Many other improvements are available in this release, please check the full changelog below for the whole list! Enjoy!

Changelog

PF_RING Library

• New Runtime Manager for injecting and removing filtering rules to the socket via Redis
• Fix caplen/MTU on loopback capture

PF_RING Kernel Module

• Add support for probabilistic sampling with kernel capture

PF_RING Capture Modules and ZC Drivers

• Add initial support for NVIDIA/Mellanox BlueField
• Add Napatech ns timestamp in PCAP mode
• Add support for probabilistic sampling with userspace capture
• Optimize hw timestamping on ice adapters (Intel E810)
• Fix timestamp support when using the ZC burst API with ice adapters
• Fix drivers compilation on Kernel 6.x
• Fix drivers compilation on RH 8.8
• Fix memory leaks in PCAP module

FT Library

• Improve application protocol guess with nDPI

nPCAP

• Fix memory corruption in traffic extraction with big index files

PF_RING-aware Libpcap/Tcpdump

• Add PF_RING support to pcap_inject
• Fix pcap_read_pf_ring return code (number of packets)

Examples

• zbalance_ipc: add support for multiple balancer threads when using NVIDIA/Mellanox adapters
• pfsend: add -c option to balance on dest ip rather than src up
• pfcount: compute drop rate in packet mode only
• pfcount: report expired licenses
• Fix ftflow_dpdk compilation on DPDK 22 or later
• Fix memory leaks in pcount, alldevs, preflect, ftflow_pcap

Misc

• Add support for Debian 12
• Add libelf and libbpf dependencies to packages
• Add sbsigntool dependency which includes kmodsign required by dkms
• Add revision to pfring-dkms dependency in packages
• Fix check for init/systemd presence
• Cleanup support for legacy adapters

Sometimes, a critical issue shows up in your network and you’d like to be notified by ntopng on Telegram or by E-Mail. ntopng allows you to filter alerts for each recipient based on a few criteria including alert family, category, severity, or affected hosts. However in some case you want to be notified about a very specific alert, out of all alerts produced with the same family, category, severity.

For example, it’s important to be notified when an Interface has no traffic, or when a new device (MAC) connects or disconnects from your network, or if a SYN scan attack is in progress, however all these alerts have the same severity/category as other critical alerts that you could not be interested in receiving a message directly on your Telegram (for example).

For this reason, we decided to extend the recipients configuration to be able to deliver specific alerts, selected from the user to the endpoint we want. In this way, a user can send a notification (e.g.) on his Telegram, if some interface stops capturing traffic (No Traffic Activity alert).

Doing that today it’s simple!

From the Notification page, it’s possible to handle it, and when adding/editing a Recipient, a tab is available, from which it’s possible to select:

• ‘By Properties’ if the “old” way of filtering alerts is needed, so filter alerts by severity, by category, or by entity;
• ‘By Alerts’ if the “new” filtering is required and here it’s possible to select one or more alerts to deliver.

After selecting ‘By Alerts’ click on the dropdown and select the critical alerts to receive.

An other important point is that, as you wish to receive important alerts on your phone, you do not want to be spammed by ntopng!

So we added a new preference, still when adding/editing a Recipient, to silence the same exact alert if it triggers more than one time per hour, in this way only one alert of the same kind per hour is going to be delivered to the recipient.

We hope you enjoy this new feature!

A Bloom filter is. probabilistic data-structure used to test whether an element is present in a set. Blooms are affected by false positives, meaning that when a bloom returns true it does not mean that the searched element is part of the set but that it is “likely” to be part of the set. nDPI (and most tools ntop develops) uses Bloom filters in order to speed-up search operations by using a quick membership check that avoids slower checks. For instance if ntopng needs to know whether host A has talked with host B, instead of doing slow database queries or keeping memory hungry data-structures it uses blooms.

Bloom filters are implements as a fixed length bit array, where individual bits are set using a few hash functions. The false positive rate (p) is defined as (1 – e(-((k * n)/m)))^k  where (k) = number of hash functions ,(m) = number of bloom filter bits, (n) = number of filter items. A Bloom filter is usually built with a given capacity in order to match a given false-positive probability objective.

On one hand Bloom are efficient are they are both fast and compact in space. On the other hand, minimising the false positive rate requires to know in advance the number of filter items, that is usually unknown when the filter is created,  as we do not know in advance with how many different hosts a given host will talk to. The result is that either the filter is too short (in terms of number of bits) and this the false positive rate is much higher than expected, or too big and thus a lot of memory is wasted (as well performance due to memory faults).

In order to address this problem, nDPI implements two different bitmaps (full API details in ndpi_api.h):

• ndpi_bitmap that can store 32 bit hash values (e.g. you can use an IPv4 address as hash value). This is based on roaring bitmaps, a compressed bitmap.
• ndpi_bitmap64 that instead can store 64-bit hash values. This is based on fuse filters.

The advantage of these data-structures is that:

• Contrary to Blooms where you need to know in advance the value of (m) here we support the whole 32 or 64 bit range. So with the above bitmaps the false positive rate is constant according to (p).
• The space used is limited as these are compressed data-structures that can be manipulated (e.g. for membership checks) without decompressing them.
• Being them compact in memory and efficient during operations, they do offer the above advantages without a speed penalty with respect to Blooms.

Just to give you an idea, with ndpi_bitmap64 you can store 1 million 64 bit hash values using 11% of the space used to store such values as 64 bit integers. This means that not only using it you need ~1/10th of the memory used to keep the raw integers in memory, while being able to do membership queries such as you would do with hashes.

On a nutshell, the above nDPI bitmaps implement Bloom filters with a fixed/lowest false probability rate, limited memory usage and fast (~45 nsec on a Mac M1) membership query.

In a future blog post, we’ll explain how using them enabled nDPI to use < 200 kB for indexing in memory 30’000 Internet domain names, with respect to 24 MB previously used, this while keeping a fast(er) lookup time.

Enjoy !

PagerDuty is a popular incident-response platform that allows problem notifications to be delivered in a flexible way to the correct team member. We have integrated it in ntopng Enterprise and this post shows you howto configure it.

First of all you need to create a PagerDuty account and select a plan (there is a free one you can choose). Done that within PagerDuty you need to select “Event Orchestration” from the “Automation” menu and create a new event orchestration. Below you can see an example.

Once you saved it click on the Integrations menu and the integration key will be displayed. Now copy the integration key in the clipboard

and jump back to ntopng creating a new endpoint copying it the integration key

Then you need to create a ntopng recipient for this endpoint where you select what alerts to deliver to PagerDuty. Once you save the setting ntopng is ready to go and as soon as a new alert is generated, it will also be delivered to PagerDuty.

Enjoy ! 

ntop tools range from packet capture, traffic analysis and processing, and sometimes it is not easy to keep up on product updates as well master all the tools. This has been the driving force for organising ntop professional training: .

This is to announce that in May we have scheduled the next ntop Professional Training session. It will take place online (Microsoft Teams) on 7th, 9th, 14th, 16th, 21st, 23rd of November, 2023 at 3.00 PM CET (9.00 AM EDT). Training will be held in English language and each session lasts 90 minutes.

All registered attendees will receive, as part of the training, a license of ntopng Pro that you can use after the training for improving your skills.

Here, you can read more about all the training topics that will now cover SNMP support in ntopng more in detail. Shall you be interested to join this session, please book your seat. Shall you have questions, please feel free to mail training@ntop.org.

Enjoy !

Yesterday we have been invited to the NetEye Users Group Meeting to give a speech about monitoring and cybersecurity.

During the talk we covered out 25 years journey in this industry and the decisions we have made during that time:

• Network vendors provide (after 25 years) poor monitoring data: flaws, proprietary formats, sampling, device limitations didn’t change the landscape even though the NetFlow RFC 3954 is 20 years old, and IPFIX is basically just a cosmetic change.
• nDPI is 10 years old and it allowed us to provide contextual information even in encrypted communications.
• The challenge today is to anticipate, no longer to monitor. In order to do that we need to capitalize on pre-labelled data such as blacklists, as well monitor signals as detailed as possible in order to detect even tiny changes in our network traffic and actors.
• We use both threshold-based (for spotting too much/little) and behavioural based (for detecting changes in behaviour) for triggering alerts.
• Drill-down is mandatory: we need evidence of problems, not just problems, as they need to be fixed (the magic word is ‘remediation’). For this reason we have implemented in the ntop pipeline the complete alerts to flows to packets path.
•  Monitoring metrics must be HD (High Definition): packets/bytes were enough 20 years ago, today we need more detailed metrics that range from traffic quality to cybersecurity.

It took us 25 years to implement all this rooted on open source software we developed. Here you can find the presentation slides.

Enjoy !

The ntop conference and training 2023 was a success: more than 100 people attended it, some of them flying to Italy from other continents. This has been a special event as we have celebrated 25 years since the first release of the original ntop application, and 10 years of ntopng.

This was our first international event (previous conferences were in Italian) and we are happy of the outcome. For us a conference is a way to update our community about the progresses we have made, how the community uses our tools, and for collecting ideas for future releases. The user’s feedback is very valuable as it allows us to improve an focus on goals that matter to our community.

At the ntopConf 2023 page you can find presentation slides and videos of those people who donated their time to make this event happen.

Many thanks to our users and community for being with us for 25 years.

Threshold alerts and statistical alerts are two different methods for monitoring and detecting unusual or potentially problematic events in various systems, such as network monitoring where anomaly detection is essential. They differ in how they define and identify anomalies:

1. Threshold Alerts
   • Threshold alerts are based on fixed, predefined values or thresholds.
   • You set specific thresholds for one or more parameters or metrics within your system. When these parameters cross the predefined thresholds, an alert is triggered.
   • These thresholds are typically static and do not change automatically. You need to set and adjust them manually based on your system’s characteristics and expected behavior.
   • Threshold alerts are straightforward and easy to configure, making them suitable for situations where you have a clear understanding of what constitutes normal and abnormal behavior.
   • Common examples of threshold alerts include temperature sensors triggering an alert when a room becomes too hot or monitoring network traffic for bandwidth usage exceeding a specified limit.
2. Statistical Alerts
   • Statistical alerts, on the other hand, rely on statistical analysis and machine learning techniques to detect anomalies.
   • Instead of relying on fixed thresholds, statistical alerts use historical data to model what is considered “normal” behavior. They then identify deviations from this model as anomalies.
   • Statistical methods can be adaptive, meaning the system continuously learns and updates its understanding of normal behavior based on incoming data.
   • These alerts are more suitable for situations where the expected normal behavior may change over time or where it’s challenging to define specific thresholds.
   • Statistical alerting techniques include methods like Z-score analysis, Mahalanobis distance, or machine learning algorithms such as clustering, time series analysis, or deep learning.

The choice between threshold alerts and statistical alerts depends on the specific requirements of the monitoring system and the nature of the data being analysed. In some cases, a combination of both approaches may be used, with threshold alerts acting as a first line of defense for obvious issues and statistical alerts providing a more advanced and adaptable layer of anomaly detection for subtle or evolving problems.

In ntop we have implemented both methods in order to accommodate all user needs. You can trigger these alerts by setting a threshold on a single metric and let ntopng emit an alert when the threshold is crossed.

Behavioural checks contain several threshold based alerts: you set a threshold based on an arbitrary value you specify and that you think it’s reasonable for a given use case. For instance a usually contacts at most two DNS servers (primary and secondary) so setting a conservatory threshold to 7 will allow you to spot hosts that are misbehaving or DNS servers (in this case you need to set a Behavioural Exception to avoid triggering invalid alerts).

If you go under (ntopng left sidebar) menu Hosts -> Local Traffic Rules you can set more sophisticated alerts based on both an absolute threshold or with respect to the recent past.

In the above example you see a rule that for each local host, triggers and alert if the host traffic exceeds of 50% the traffic of the previous hour (you can set different frequencies such as 5 minutes or day). This rule can be defined for a specific host, or for all hosts. By selecting the rule type, you can specify similar rules to trigger other problems, such as a flow exporter (e.g. a NetFlow/sFlow device) that stops sending flows, or a network interface whose traffic is too much/little with respect to its recent past.

In case a threshold is crossed an alert is reported as shown in the above picture.

In addition to the above alerts, ntopng implements more sophisticated alerts based on statistical analysis that allow to detect changes in behaviour rather that using (semi-)static thresholds. 

In the above cases, ntopng continuously computes a range of lower/upper values that are considered normal for a given metric. These values are used to predict the range of the next metric that in case it is too low or too high, it triggers an alert.

Above you can see an example of behavioural timeseries analysis. Note that threshold are continuously computed with no human operator interaction necessary.

Final Remarks

In summary some of the statistical traffic analysis features of ntopng include:

1. Traffic Flow Analysis: ntopng can display detailed information about network flows, showing source and destination IP addresses, ports, protocol types, and the amount of data transferred. This information allows you to analyze traffic patterns statistically.
2. Historical Data: ntopng can store historical network traffic data and provide you with the ability to view and analyze this data over time. This feature allows you to identify trends, peak usage periods, and changes in network behavior.
3. Protocol Analysis: It provides detailed statistics on the usage of various network protocols, such as HTTP, DNS, FTP, and more. You can view which protocols are most heavily used and identify anomalies or unusual traffic patterns.
4. Application Visibility: ntopng can classify traffic by application, helping you understand which applications are consuming network resources. This is valuable for monitoring and optimizing network performance.
5. Anomaly Detection: While ntopng doesn’t provide advanced machine learning-based anomaly detection, it can help identify traffic patterns that deviate from the norm. Unusual spikes or deviations in network traffic can be indicative of potential issues.
6. Alerting: ntopng can generate alerts based on various criteria, including traffic thresholds. .
7. Traffic Reports: It can generate reports that include various traffic statistics, top talkers, top applications, and more, helping you analyze network traffic statistically.

Enjoy !

This is to announce the release of nDPI 4.8 that introduces various new protocols (in total 351 protocols and 53 risks), several internal changes to improve packet processing, extension of fuzzing to new components to improve coverage, new algorithms for handling lists with reduced memory and better performance. Protocol changes have been introduced not just for new protocols but also for keeping track of changes on exiting protocols such as QUIC and TLS. This said there are many changes under the hood that include contributions from many developers and that are listed in the changelog. For the new release we plan to add new dissectors and risks in addition to new protocol metadata, as well to keep removing legacy and now obsolete OpenDPI code. If you want to contribute to the next nDPI release please join us on GitHub.

Enjoy !

This is to announce the release of nProbe Cento 1.20, that is basically a maintenance release that fixes some issues, improved metadata export using nDPI, and adds new platform and distributions support.

Below you can find the whole changelog.

Enjoy !

Improvements

• Add ARM support
• Add support for dumping bad packets (–dump-bad-packets)
• Add support for the latest nDPI API
• Improve nDPI protocol guess

Fixes

• Fix bridge mode with standard drivers
• Fix max interface speed detection with comma-separated list
• Fix tx stats
• Fix banned search
• Fix permissions for the logrotate configuration file
• Disable userspace aggregation with kernel drivers (not required)
• More boundary checks

Misc/Changes

• Add support (package) for Debian 12 “Bookworm”
• Add support for quotes in conf file
• Change printed version to make it parseable
• Warn user on package update if maintenance is expired
• Update dependencies (add libelf1, libbpf0)

This is to announce the release of nProbe 10.4. In this version we have made several improvements (including support for new platforms and distributions) as well merged the agent code into the main code base (via -T) on both Linux and Windows. This feature allows you to export (for traffic originated or terminated on the host where nProbe runs) additional contextual information such as the user or process name that produced specific traffic flows. The agent mode is used in ntopng to implement the cloud mode support, that enables nProbe to be used as an agent on monitored hosts.

In addition to this, Kafka support, Amazon VPC handling, optimized Nokia IPFIX flow export  and various other minor changes are part of this release. You can read the whole changelog below.

Enjoy !

New Features

• Added cloud mode support for integrating with ntopng in cloud mode
• Reworked Kafka export support to make it robust and scalable
• Merged agent mode code in nProbe so that it is automatically enabled as the templace defined host
• RTP handling for Zoom/Ms Teams streams
• Added support for FreeBSD 14, Debian 12, ARM 64 bit

Command Line Options

• Added support for encryption to –zmq-publish-events
• Added support for multi-line json export (–zmq-format m)
• Added –cloud option to ease integration with ntopng in cloud mode
• Added –gtpv1-dont-export-flows-immediately and –gtpv2-dont-export-flows-immediately flags for disabling immediate flow export as soon as the first response packet is received
• Added –map-postnat
• Added –template-ids to specify the list of templates to accept

Improvements

• Implemented –map-ifnames for mapping interfaceNames to interfaceIds
• Implemented GTPV1_C2S_TEID/GTPV1_S2C_TEID and GTPV2_C2S_TEID/GTPV2_S2C_TEID
• Implemented ZMQ stress tester
• Implemented packet flow template
• Implemented supoort for proprietary Nokia IPFIX exports
• Improved VPC log scanning
• Improved nDPI protocol guess
• Improved packet decoding checkes
• Improved parameter check
• Improved tracing
• Added support for ndpi serializer in sflow counters export
• Added %FLOW_CONTENT_TYPE in order to export nDPI-dissected flow datatype (audio, video…)
• Added %FLOW_EXPORT_TIME that contains the epoch of a flow export
• Added -6 to sendPcap (Delivery of packets over IPv6)
• Added BPF filter in cloud mode
• Added BPF filter when started in –agent-mode
• Added Ent XL bundle support
• Added GTPV2_C2S_TEID and GTPV2_S2C_TEID
• Added Ms Teams support in RTP plugin
• Added Zoom RTP tracing
• Added check for avoiding crash when offload is on and packet length exceeds the snaplen size
• Added code for handling RTP stats when using dynamic payload (e.g. Teams)
• Added compressed AWS VPC log file support
• Added debian 12 support
• Added diameter handling of mesageId 300, 301, 303
• Added extra check
• Added support for %EXPORTER_IPV6_ADDRESS
• Added support of %INTERFACE_NAME %ACCOUNT_ID in VPC log collection
• Added test pcaps
• Added trace in case of being unable to delete a file
• Added tracing of log deletion
• Added FreeBSD 14 support

Fixes

• Fixed invalid print with large number of files
• Fixed for Win32 cross-compatibility
• Fixed for collecting bidirectional flows
• Fixed for various memory leaks
• Fixed netflow packets reforging during replay (it was adding 2 extra IP/UDP headers)
• Fixed DNS dissection
• Fixed ELK export with username/pwd Added support for ELK 8
• Fixed RTP detection with RTP flows starting with a STUN packet
• Fixed clickhouse export bug (missing username)
• Fixed flow direction due to a value overflow ntop/ntopng#6267
• Fixed invalid bytes/octets decoding during flow collection
• Fixes buffer overflow issues
• Fixes cache statistics
• Fixes for exporting process information in cloud mode
• Fixes #584
• Fixes sampling support Extended -p (aggregation) flag format

Misc

• Agent mode changes
• Changes due to nDPI API changes
• Changes for adding cloud support in Windows
• Code cleanup adding a new member to plugin dissector
• Do not enable the service on upgrade if disabled by the user
• Enable sflow counters export in TLV using numerial keys
• Enlarged Diameter port range 3868-3900
• Extended -p support with discard of OUT (NFv9)/Post (IPFIX) counters
• Fixed Kafka export format
• Minor fix in custom template handling
• Missing check to avoid using DPDK (when installed) if –with-dpdk is not used
• Now -3 allows to recursively scan a directory looking for AVC files
• Optimization to avoid handling binary data with HTTP plugin
• Print when running in Cloud mode. Disable demo mode when in Cloud mode.
• Removed –ucloud Added –instance-name that is exported in %NPROBE_INSTANCE_NAME
• Rework embedded check (at runtime now)
• Reworked kafka options handling
• Set edr_mode when the cloud license should unlock the endpoint mode only
• Workaround for non-standard DLT_NULL encapsulations
• aarch64 changes

This is to announce ntopng 6.0 a new major release that includes many new features and improvements:

• ntopng is no longer just a real-time traffic monitoring application: it can now track assets when offline and enable better investigations leveraging on improved historical traffic analysis.
• Implemented vulnerability reports that can scan hosts, ports, and look for CVEs. Even if other tools sport similar features, ntopng is unique in merging traffic analysis with vulnerability assessment. This means that you can position your CVEs with respect to real traffic (i.e. a severe vulnerability is not too problematic if nobody access the service) or discover open host ports never used in traffic (i.e. you better close them).
• The user interface has been greatly redesigned: we have implemented a flexible template-based reporting system and a new dashboard. In the next release ww’ll release an editor for customising it.
• OT/Scada support has been enhanced with new ModBus support and behavioural alerts.
• We have implemented basic features, currently in beta, for running ntopng/nProbe in cloud (you can read more here). This is just the beginning of a long journey for making ntop tools cloud-aware and in the following months we plan to finalise the development.

The list of features is very long and we will schedule a webinar within the next couple of weeks (we’ll send an announce on this blog) where we will walk through all the changes of this new release.

Below you can read a list of the main changes of this new release:

• New configurable Dashboard with new built-in templates.
• New configurable Traffic Report that can be delivered via email.
• New Vulnerability Scans & CVEs support.
• Introduced basic features for supporting ntop cloud [beta].
• Add support to Periodic Reports notified via Recipients (e.g. email).
• Add Inactive Hosts: track assets even when no longer online.
• Add PagerDuty and TheHive integration.
• Add Server Ports Analysis page.
• Enable multithreading in active measurements (more accurate).
• Migrate frontend chart timeseries library to Dygraph.
• Add support for MAC Address based RADIUS accounting.
• Improve OT, ICS, Scada support as well introduced support for Modbus protocol and alerts.
• Trigger External Host alerts directly from Lua (also for inactive hosts).
• Add support for LLDP id to MIB-II InterfaceId mapping.
• Add support for bidirectional rules.
• Add support for Enterprise XL bundle license.

You can read the Changelog for all details.

Enjoy !