This is to announce the release of nDPI 4.10. This release adds many improvements and new detected protocols. For this reason future releases will be scheduled more often on a 4 or 6 months (hard) basis in order to provide you constant updates on a predictable timeframe,

Beside adding many dissectors, this release paves the way towards First Packet Classification (FPC) that is an attempt (for selected protocols) to detect the application protocol DPI at the first packet of a connection. Of course this is a challenge, and it won’t be possible for all protocols, but we’re moving into that direction that could save a lot of memory and CPU cycles while nDPI-ing. This will be one of the main goals of the next release (that will probably be the first of the 5.0 series) along with other measurements we have in mind.

For the time being enjoy this release an for those interested in the complete changelog, you can find it in the rest of this post.

Enjoy !

Changelog

• Initial work towards First Packet Classification (FPC)

• Add OpenWire support (#2513)
• FPC: add DNS correlation (#2497)
• ipaddr2list.py, ndpi2timeline.py: reformatted (#2509)
• Add Nano (XNO) protocol support (#2508)
• Added ClickHouse protocol
• Add HLS support (#2502)
• Add infrastructure for explicit support of Fist Packet Classification (#2488)
• Add detection of Twitter bot (#2487)
• Added default port mappings to ndpiReader help -H (#2477)
• Add Ripe Atlas probe protocol. (#2473)
• Add ZUG consensus protocol dissector. (#2458)
• Added NDPI_PROBING_ATTEMPT risk
• DTLS: add support for DTLS 1.3 (#2445)
• Added dpi.compute_entropy configuration parameter
• Add Call of Duty Mobile support (#2438)
• Add Ethernet Global Data support (#2437)
• Viber: add detection of voip calls and avoid false positives (#2434)
• Add support for Mastodon, Bluesky and (FB-)Threads (#2418)
• Fixes JA4 computation adding a better GREASE detect funzion
• DTLS: add support for Alert message type (similar to TLS) (#2406)
• Add Adobe Connect support (#2407)
• Remove PPStream protocol and add iQIYI (#2403)
• Add BFCP protocol support (#2401)
• Add strlcpy implementation (#2395)
• Add KNXnet/IP protocol support (#2397)
• STUN: add support for ipv6 in some metadata (#2389)
• Implemented STUN peer_address, relayed_address, response_origin, other_address parsing Added code to ignore invalid STUN realm Extended JSON output with STUN information
• Add Label Distribution Protocol support (#2385)
• Add The Elder Scrolls Online support (#2376)
• Add Shellscript risk detection. (#2375)
• Add PE32/PE32+ risk detection (detect transmitted windows executables). (#2312)
• Added support for STUN Mapped IP address
• Added binary data transfer risk alert
• Add LoL: Wild Rift detection (#2356)
• STUN: add dissection of XOR-PEER-ADDRESS with ipv6 address
• Add FLUTE protocol dissector (#2351)
• Add PFCP protocol dissector (#2342)
• Add Path of Exile protocol dissector (#2337)
• Add NetEase Games detection support (#2335)
• Add Naraka Bladepoint detection support (#2334)
• Add BFD protocol dissector (#2332)
• Add DLEP protocol dissector (#2326)
• Add ANSI C12.22 protocol dissector (#2317)
• TLS: add configuration of JA* fingerprints (#2313)
• Add detection of Gaijin Entertainment games (#2311)
• Add new AppsFlyer domain (#2307)
• Add TencentGames protocol dissector (#2306)
• Add Gearman protocol dissector (#2297)
• Add Raft protocol dissector. (#2286)
• Add Radmin protocol dissector (#2283)
• Add STOMP protocol dissector (#2280)
• Add ElectronicArts detection support (#2274)
• Add Yojimbo (netcode) protocol dissector (#2277)
• Add a dedicated dissector for Zoom (#2265)
• Add Mumble detection support (#2269)
• Add KCP protocol dissector. (#2257)
• Add PIA (Private Internet Access) support (#2250)
• Add more adult content hostnames (#2247)
• Add Roughtime protocol dissector. (#2248)
• Add realtime protocol output to ndpiReader. (#2197)
• Add Google Chat support (#2244)
• ndpiReader: add breed stats on output used for CI (#2236)
• Add Ceph protocol dissector (#2242)
• Add HL7 protocol dissector (#2240)
• Add IEC62056 (DLMS/COSEM) protocol dissector (#2229)
• Add NoMachine NX protocol dissector (#2234)
• Add Apache Kafka protocol dissector (#2226)
• Add WebDAV detection support (#2224)
• Add JSON-RPC protocol dissector (#2217)
• Add OpenFlow protocol dissector (#2222)
• Add UFTP protocol dissector (#2215)
• Add HiSLIP protocol dissector (#2214)
• Add PROFINET/IO protocol dissector (#2213)
• Add Monero protocol classification. (#2196)
• Add Ether-S-Bus protocol dissector (#2200)
• Add IEEE C37.118 protocol dissector (#2193)
• Add ISO 9506-1 MMS protocol dissector (#2189)
• Add Beckhoff ADS protocol dissector (#2181)
• Add Schneider Electric’s UMAS detection support (#2180)
• Add Ether-S-I/O protocol dissector (#2174)
• Add Omron FINS protocol dissector (#2172)
• Rework S7Comm dissector; add S7Comm Plus support (#2165)
• Add OPC UA protocol dissector (#2169)
• Add RTPS protocol dissector (#2168)
• Add HART-IP protocol dissector (#2163)
• Add IEEE 1588-2008 (PTPv2) dissector (#2156)
• Added TeslaServices and improved TikTok host names. Fixes #2140. (#2144)
• Add ethereum protocol dissector. (#2111)
• Added generic Google Protobuf dissector. (#2109)
• Add CAN over Ethernet dissector.

• Enhanced PrimeVideo detection
• Enhanced ookla tracing
• Improved ICMP malformed packet risk description
• Improve detection of Cloudflare WARP traffic (#2491)
• tunnelbear: improve detection over wireguard (#2485)
• Improve detection of Twitter/X (#2482)
• Zoom: fix detection of screen sharing (#2476)
• Improved detection of Android connectiity checks
• Zoom: fix integer overflow (#2469)
• RTP/STUN: look for STUN packets after RTP/RTCP classification (#2465)
• Zoom: faster detection of P2P flows (#2467)
• Added NDPI_PROTOCOL_NTOP assert and removed percentage comparison (#2460)
• Add extra entropy checks and more precise(?) analysis. (#2383)
• STUN: improve extraction of Mapped-Address metadata (#2370)
• Added support for roaring bitmap v3 (#2355)
• Add more TencentGames signatures (#2354)
• Added DGA exception for Dropbox
• QUIC: add heuristic to detect unidirectional GQUIC flows (#2207)
• fuzzing: improve coverage (#2495)
• Improve detection of Cloudflare WARP traffic (#2491)
• fuzz: improve fuzzers using pl7m (#2486)
• wireshark: lua: minor improvements
• Improved logic for checking invalid DNS queries
• fuzz: improve fuzzing coverage (#2474)
• Improved Kafka dissector. (#2456)
• H323: improve detection and avoid false positives (#2432)
• Fix/improve fuzzing (#2426) (#2400)
• eDonkey: improve/update classification (#2410)
• Domain Classification Improvements (#2396)
• STUN: improve extraction of Mapped-Address metadata (#2370)
• Improve LoL: Wild Rift detection (#2359)
• Improve TencentGames detection (#2353)
• STUN: improve heurstic to detect old classic-stun
• ahocorasick: improve matching with subdomains (#2331)
• Improved alert on suspicious DNS traffic
• Telegram: improve identification
• Improved Telegram detection
• Improved modbus dissection to discard false positives
• Improved Polish gambling sites fetch script. (#2315)
• fuzz: improve fuzzing coverage (#2309)
• Improve normalization of flow->host_server_name (#2310)
• Improve ndpi_set_config error printing. (#2300)
• Improve MySQL detection (#2279)
• Improve handling of custom rules (#2276)
• Zoom: improve detection (#2270)
• Improved ndpi_get_host_domain
• Bittorrent: improve detection of UTPv1 (#2259)
• Improved uTorrent via utp (TCP-like streams over UDP). (#2255)
• fuzz: improve fuzzing coverage (#2239)
• fuzz: improve fuzzing coverage (#2220)
• Improved belgium gambling sites regex. (#2184)
• Improve CORBA detection (#2167)
• STUN: improve demultiplexing of DTLS packets (#2153)
• Improved TFTP. Fixes #2075. (#2149)
• fuzz: improve coverage and remove dead code (#2135)
• Improved Protobuf dissector. (#2119)
• Improved detection as non DGA for hostnames belnging to a CDN (#2068)
• Improved CryNetwork protocol dissector.

• Make the CI faster (#2475)
• Add a script to download/update the domain suffix list (#2321)
• Add identification of Huawei generic and cloud traffic (#2325)
• ndpiReader: improve the check on max number of pkts processed per flow (#2261)
• Added default port mappings to ndpiReader help -H (#2477)
• ndpiReader: restore ndpiReader -x $DOMAIN_NAME functionality (#2329)
• ndpiReader: improve the check on max number of pkts processed per flow (#2261)
• ndpiReader: fix memory leak
• Add realtime protocol output to ndpiReader. (#2197)
• ndpiReader: add breed stats on output used for CI (#2236)
• ndpiReader: avoid creating two detection modules when processing traffic/traces (#2209)
• ndpiReader: fix guessed_flow_protocols statistic (#2203)

This is to announce a new PF_RING release 8.8.0!

This release adds generic support for flow table offload, which is currently supported on Napatech adapters with Flow Manager enabled. This new technology has been successfully used to accelerate nProbe Cento when running with DPI enabled on multi 100 Gbit traffic (both passive and inline) and the work has been presented at IEEE HPSR (IEEE International Conference on High Performance Switching and Routing). This also adds support for zero-copy transmission on Napatech adapters, to reduce bandwidth utilisation and latency when forwarding traffic between interfaces.

In PF_RING 8.8.0 we also introduce support for the NVIDIA BlueField networking platform. It is now possible to run PF_RING on the ARM cores of the DPU and leverage on ZC acceleration for processing traffic in the adapter.

Many other improvements are available in this release, please check the full changelog below for the whole list.

Enjoy !

Changelog

Key Features

• Support for Napatech Flow Manager
•  Support for nVidia BlueField 2/3

PF_RING Library

• Add pfring_flow_update struct and pfring_recv_flow
• Add PF_RING_FLOW_OFFLOAD flag to enable Flow Table offload on supported adapters
• Add PF_RING_FLOW_OFFLOAD_AUTO_UNLEARN to control tcp auto unlearn
• Add PF_RING_KEEP_CRC flag (required by Napatech)
• Fix pfring_zc_send_pkt_multi return code

PF_RING Kernel Module

• Add flow_id to generic_flow_tuple_hw_rule
• Fix crash with more than 64 rss queues
• Various minor fixes

FT Library

• Add pfring_ft_zmq_get_stats API
• Add slice action

PF_RING Capture Modules and ZC Drivers

• Add support for Napatech Flow Manager (Flow Table offload)
• Add support for NVIDIA BlueField (ARM64)
• Add jumbo support for ice-zc
• Add support for Napatech zero-copy forwarding (pfring_send_last_rx_packet)
• Fix all Intel drivers compilation on latest kernels (including Ubuntu 24)
  • ixgbevf-zc driver update v.4.18.9
  • i40e-zc driver update v.2.24.6
  • iavf-zc driver update v.4.9.5
• Extend Napatech interface name to support adapter ID
• Fix drop counter on NVIDIA/Mellanox Connect-X (read phy discards in addition to buffer discards)
• Fix crash with more than 64 rss queues
• Fix iavf (i40e VF) ring initialization
• Fix driver dependency intel_auxiliary

PF_RING-aware Libpcap/Tcpdump

• New pfring-aware tcpdump 4.99.4

Examples

• New pfflow_offload sample app with support for Napatech Flow Manager
• pfsend
  • Fix ipv6 packet forging
  • Add eth header when replaying RAW capture files
  • Add -U / -K options to control flow distribution
  • Fix on the fly packets generation
  • Fix -A when -O is used
• pfsend_multichannel
  • Add -C to support multiple TX channels with Napatech
• pfbridge
  • Add support for Napatech zero-copy forwarding

Misc

• Improve pf_ringctl to auto generate hugepages.conf based on numa nodes
• Change MAX_CAPLEN to 65600 to correctly handle max jumbo packet size
• Remove deprecated code for Accolade and fm10k

This is to announce the release of nProbe 10.6 that includes many changes in a couple of selected areas:

• Mobile traffic analysis (GTPv1 and GTPv2) and GTP-C/GTP-U correlation has been rewritten to support complexity of modern mobile networks. 
• nProbe is now more friendly when talking ZMQ/Kafka (hence with ntopng) as it can report various statistics and export of specific information elements has been optimised to improve performance.

In addition nProbe supports the latest nDPI version that has been optimised in memory and that features almost 500 application protocols, that is a major step ahead with respect to the previous version. Furthermore, we have improved the analysis of multimedia and streaming protocols, as well collaboration tools such as Teams, Zoom and Meet: for these protocols RTP metrics such as MOS-like, jitter and packet loss have been improved.

Below you can find the complete nProbe 10.6 changelog.

Enjoy !

• Add –blacklists for configuring blacklists
• Add –ndpi-categories-dir
• Add –dns-dump-blacklisted for dumping only blacklisted client queries
• Add –gtpv1-track-imsi and –gtpv2-track-imsi
• Add –gtpv1-teid-cache-duration and –gtpv2-teid-cache-duration
• Add –gtp-use-host-in-tunnels
• Add –estimate-tcp-latency for making TCP network latency optionally estimated in case a flow start was not observed
• Add -gtp-use-host-in-tunnels for GTPv1
• Extend –map-ifnames option to accept a file in addition to the CLI

• New %JA4C_HASH IE
• New %GTPV1_GSN_ADDRESS_IPV4_A and %GTPV1_GSN_ADDRESS_IPV4_B containing the first and secondGSN IPv4 address
• New %FLOW_ENCRYPTED IE
• New %L7_DOMAIN_INFO IE similar to %L7_INFO but returns only the domain name of the host
• Add support for datalink 10 (raw packets)
• Add support for flow source for detecting how flows are generated (packets, collection of sflow/netflow, collection of sflow)
• Improve ZMQ events messages
• Improve HTTP dump to file (flush after write)
• Hash size is now automatically incresed when -M is used
• Various GTP-C/GTP-U improvements
  • Add support for DeleteBearerRequest in GTPv2
  • Enhanced %UPSTREAM_TUNNEL_ID %DOWNSTREAM_TUNNEL_ID with GTP gateways
• Improve RTP handling with Zoom and Teams
• Add Zoom p2p support
• Improve Zoom Media Encapsulation decoding
• Improve utility to export flows to Google Pub/Sub
  • Add native batching support
  • Add options to control import/export settings
• Add instance UUID for detecting the individual nprobe instances
  • Add NPROBE_UUID NPROBE_IP Information elements
• Add support fo HTTPS ports in the HTTP plugin
• Implement TCP flow swap in case SYN|ACK is observed before SYN
• Add @timestamp to the ELK plugin

• Fix Clickhouse database schema types
• Various RTP fixes
• Various GTP fixes
  • Restore GTP accounting with –imsi-apn-aggregation
  • Fix GTPv2 delete session handling
  • Add fixes for discarding negative GTP-C responses
  • Fix GTPv2 Bearer Context decoding that was previously unable to handle IPv4/IPv6 F-TEID
  • Fix crash with GTP flow aggregation
  • Fix GTP-C dump
• Fix DNS additional records decoding
• Fix –map-postnat cli option
• Fix plugin handling with null-ethernet (e.g. loopback) encapsulation
• Fix DHCP export
• Fix DHCP_CLIENT_NAME dissection
• Fix TOS handling
• Fix –in-iface-idx/–out-iface-idx values populated in case of packet capture from network device
• Fix endianess with –map-postnat
• Fix memory leaks
• Fix -p aggregation parsing
• Fix IMSI correlation
• Fix double Kafka flow export
• Fix IPS policing
• Fix nflite initialization

• Add support for the ntop Cloud
• ZMQ events are now emitted every 5 sec (used to be every second)
• Disabled JA3+ support in favour of JA4
• Diabled –collector-passthrough when ZMQ is in use as this can cause inconsistencies in ntopng side due to template format
• Add nprobe user to the ntop group
• Package for Ubuntu 24

This is to announce that Cento 2.0 is out! This new major release introduces many new great features.

First of all it adds support for offloading flows to Napatech SmartNICs featuring Flow Manager. This new feature has been presented at IEEE HPSR (IEEE International Conference on High Performance Switching and Routing) and demonstrated to provide a significant performance boost and dramatically reduce the PCIe and memory bandwidth utilisation when processing 100 Gbit (or more) links with full-speed traffic. This can be used both by standard cento to accelerate passive monitoring, and cento-bridge to accelerate packet forwarding and reduce latency (also in combination with the new zero-copy packet forwarding support).

This release also adds support for Avro serialization when exporting flows to Kafka, in addition to the JSON serialization. Support for custom templates has also been introduced to select the Information Elements to export when exporting to Kafka, both with Avro and JSON serialization formats.

From the long list of major improvements, it is worth to mention also the ability to configure, for each application protocol, a specific destination interface or queue when running cento in IDS mode, in combination with load-balancing. In addition to this, cento-ids is also now able to bufferize packets per flow, to store packets in memory until the layer-7 protocol has been detected, to make sure all packets for the flow are sent to the same/expected egress interface when a destination is configured per application protocol.

Please check the changelog below for the full list of new features and improvements.

Enjoy!

Changelog

New Features

• Flow table offload on Napatech adapters with Flow Manager support (–flow-offload option), on both passive and bridge mode
• Avro flow serialization when exporting to Kafka (–avro option)
• Template support (–template option) to select the Information Elements when exporting JSON or Avro to Kafka

Improvements

• Add supprot for zero-copy packet forwarding on Napatech adapters (–tx-offload option)
• Add support for interface notation with no rss queues on second interface (e.g. nt:stream[0-1],nt:1)
• Add support for bridging on a single interface (packet bouncing)
• Add support for shunting and slicing (via rules file) in bridge mode
• Add support for configuring egress interfaces or queues as actions in rules configuration when balancing traffic in ids mode
• Add support for buffering packets (per flow) in ids mode (–balanced-egress-buffer option)
• Add support for blacklists (–blacklists option)
• Add IMSI aggregation and caching using redis
• Add ability to load public TLD domain names
• Add support for GRE Transparent Ethernet Bridging
• Add UUID to exported metadata
• Improve Geolocation support with MaxMind DB
• Improve ZMQ export with high flow rate
• Optimize nDPI support

Fixes

• Fix PPPoE support

Misc/Changes

• Disable cento auto start on first install
• Add nDPI package dependency
• Add cento user to the ntop group
• Add libavro dependency
• Migrate from json-c to jansson due to Avro incompatibility
• Package for Ubuntu 24

We’re happy to announce ntopng 6.2, a 10 months long development cycle. We have changed a few things in the UI and under the hood.

• Many pages as the flow page have been rewritten from scratch for responsiveness and usability

• Mitre Att&ck has been integrated in alerts, flow risks and  dashboards.As you can see we now have implemented a remediation column that shows you some remediation actions to avoid specific issues to appear again in the future.
  
• ntopng 6.2 uses -60% of memory woth respect to 6.0 as already discussed in this blog post.
• Historical Flows can now be replayed to mimic a historical time machine
• ntopng is now ntop cloud aware, so you can control instances from the ntop cloud applications.

The list of new features it is very long and it deserves much more time than what we can offer with a blog post. For this reason we will publish specific posts in the next few days, and schedule a webinar in the coming weeks when most people will be back to work after the summer break. For those interested in knowing all changes in detail, the complete changelog is available at this page.

Enjoy and many thanks to all contributors that made this possible !

In mid August we have refreshed most ntop tools, that include new features, enhancements and fixes. This webinar will walk through the major release changes and show what you should expect from this new release. Finally we will present future work items and developments plans so you can comment and provide feedback.

The event is scheduled for Thu, September 5th at 15:00 CET (9 AM EST) and it will be streamed online. Attendance is free of charge, but you need to register at this page in order to reserve your seat.

ntop users who have enabled ClickHouse, know that they can search/aggregate/export historical flows and create customized reports. However, in the past months some of our users were uncomfortable of this approach as they preferred to seamlessly analyze historical as live data with the full power of ntopng.

In the latest ntopng version we have added a new “play” button shown in the picture below.

In order to use this new feature, you need to:

• Select the time span you are interested in (e.g. the last hour)
• Optionally you can set a filter (e.g. only traffic of host X)
• Click on the play button.

Once you have clicked on the button the following message is reported. Note that:

• in order to exhaust all the available memory, in case there are too many flows to extract, ntopng limits their number (currently to 2 millions).
• As data extraction can take a few seconds, please be patient (the speed depends on the ClickHouse extraction speed).

The extracted flows are used to populate a new interface named “Database”.

Once you select this interface, you can navigate all the pages as you do with live traffic. Note that in the menubar there is a blue icon (see the above picture) that shows if flow loading is still in progress or if the database extraction has been completed already. If you want you can make a new historical flows extraction, and in this case the previous data present in the Database interface will be overwritten.

Enjoy !

ntop tools range from packet capture, traffic analysis and processing, and sometimes it is not easy to keep up on product updates as well master all the tools. This has been the driving force for organising ntop professional training.

This is to announce that in October we have scheduled the next ntop Professional Training session. It will take place online (Microsoft Teams) on 15th, 17th, 22nd, 24th, 29th, 31st of October, 2024 at 3.00 PM CET (9.00 AM EDT). Training will be held in English language and each session lasts 90 minutes.

All registered attendees will receive, as part of the training, a license of ntopng Pro that you can use after the training for improving your skills as well the webinar link.

Here, you can read more about all the training topics. Shall you be interested to join this session, please book your seat. attendees will have access to all session recordings. Shall you have questions, please feel free to mail training@ntop.org.

Starting with nDPI 4.10, we have introduced a new feature called First Packet Classification (FPC). Goal of this technique is to address one problem of DPI that detects a protocol only when traffic has been dissected. This means that for TLS you need a few packets (usually between 5 and 10) for protocol dissection, as nDPI has to wait until TLS handshake packets are exchanged. This can be a problem in particular when DPI is used with inline traffic (e.g. on a IPS) as the decision about the application protocol cannot be taken at the first packet, hence some non-legitimate traffic (i.e. the first few flow packets until DPI is completed) can flow instead of being blocked.

The idea of FPC is to detect a protocol at the first packet so that we can reduce the workload on nDPI and address the above issue. As of today FPC is an ongoing development and we plan to extend it in the coming months. Currently it supports three modes:

• NDPI_FPC_CONFIDENCE_DPI
  A protocol is detected at the first packet using DPI. This can happen with protocols such as QUIC and SNMP where the first packet is enough fo deciding about a protocol. This typically happens only for UDP traffic as yo can imagine.

• NDPI_FPC_CONFIDENCE_IP
  nDPI includes IP address lists about known services (e.g. the list of Facebook IP addresses published by Meta), so once an IP of a flow matches one of these addresses it is detected reliably.

• NDPI_FPC_CONFIDENCE_DNS
  nDPI implements an LRU cache feeded by the DNS nDPI dissector. In essence when nDPI detects a DNS request for a host name for which the application protocol is known, this information is stored in the cache. Example, suppose nDPI detected a DNS request for host e12.whatsapp.net that has been resolved to IP address 15.197.206.217. As nDPI matches this host as WhatsApp as it has internally the list of WhatsApp domains, it stores in the cache  15.197.206.217 = WhatsApp. If in the near future nDPI sees traffic for host 15.197.206.217, it labels it as WhatsApp. This is a great hint for nDPI, in particular for TCP-based protocols such as TLS that as discussed earlier would require many more packets before the DPI detection would complete.

As said, this is just the beginning of FPC. In the near future we plan to extend it and add more aggressive caching. However thanks to this feature the nDPI performance has been already enhanced and detection time minimized. For all FPC details start here.

Stay tuned !

Next year the ntop community will meet in Zürich, Switzerland  for a two days event (training and conference) on May 7 and 8th.

As already happened in the past, we want to meet our users and discuss with them what we have done and what are the future directions to take. This event will not happen without our community hence we are looking for speakers willing to present  interesting use cases, solutions, challenges, report experiences or anything that is relevant for our community.

We have selected Zürich as location in the hope to have many users living in German-speaking countries attending this event. On the other hand not all of you will be able to travel, hence we accept both on-site and remote presentations. The official language of the event will be English.

Shall you be interested in submitting a talk, please visit the conference page and submit your contribution.

Enjoy !

Today we report how ntopng has been used by Alabus AG to improve the corporate security (German version down this page).

Enjoy !

PS. ntop users are very welcome to contact us reporting how they use ntop tools.

ntop is used as a basis for analyzing the entire network traffic and it generates a very large number of daily alerts, which are caused by known and unknown anomalies and then it historizes all network flow data for possible later forensics.
As an SME, we do not have the necessary resources to process all these alerts manually on a 24/7 basis.

Alerts over 15h

However, the flexibility of ntop enabled us to create our own add-in with our software platform. All flow and host alerts from level warning onwards are read out immediately across all interfaces via the ntop API, pre-filtered and aggregated according to our own rules.

The information is then enriched with the help of AbuseIP DB and public blocklists. If a detected IP address is found both in AbuseIPDB and in another blocklist, it is automatically added to our own blocklist and as a result we do have a near-time, not supervised active response system.

Automatically consolidated block list over 15h

These simple additions aggregate the ntop alert data by 98% and lead to a few residual alerts that can be mastered by SMEs.

The decision to switch to the open-source solution ntop and to add a little of our own effort has given us much greater network transparency and traceability and, finally reduced our network monitoring costs by factors.

alabus is a Swiss technology company that offers standard solutions for the insurance market (software and operations) in Switzerland. As a technology supplier and operator of standard solutions in a regulated market, certifications such as ISO 27001 are a basis for ensuring risk management in a traceable and transparent manner.

ntop wird zur Analyse des gesamten Netzwerk-Traffics als Basis verwendet, generiert daraus täglich eine sehr grosse Anzahl von Alerts, welche durch bekannte Anomalien verursacht werden und historisiert anschliessend sämtliche Netzwerk-Flow-Daten für eine allfällige spätere Forensik.

Für die manuelle 7/24 Abarbeitung all dieser Alerts stehen uns als KMU jedoch die notwendigen Ressourcen nicht zur Verfügung. Die schiere Menge an Alert-Meldungen ist in allen gängigen regelbasierten Überwachungssystemen häufig ein Human-Ressourcenissue.

Die Flexibilität von ntop ermöglichte es uns jedoch, ein eigenes Add-On mit unserer Software-Plattform zu erstellen. Dabei werden alle Flow- und Host-Alerts ab Level Warning via ntop API über sämtliche Interfaces zeitnah ausgelesen und nach eigenen Regeln vorgefiltert und aggregiert.
Anschliessend findet eine Anreicherung der Informationen mit Hilfe von AbuseIPDB sowie öffentlichen Blocklists statt. Wird eine entdeckte IP-Adresse sowohl in AbuseIPDB als auch in einer weiteren Blockliste gefunden, so wird diese vollautomatisch auf die eigene Blocklist gesetzt und es entsteht ein Near Time Active Response System.
Diese einfachen Ergänzungen aggregieren die ntop Alert-Daten um 98% und führen zu einer Anzahl Rest-Alerts, die auch als KMU zu meistern sind.

Die Entscheidung, auf die OpenSource Lösung ntop zu wechseln und ein wenig Eigenleistung beizufügen, hat uns eine wesentlich höhere Netzwerk-Transparenz und Nachvollziehbarkeit gebracht und last but not least konnten unsere Netzwerk-Überwachungs-Kosten stark reduziert werden.

alabus ist ein Schweizer Technologieunternehmen, welches Standardlösungen für den Versicherungsmarkt (Software und Betrieb) in der Schweiz anbietet.

Als Technologielieferant und Betreiber von Standard Lösungen in einem regulierten Markt, sind entsprechende Zertifizierungen wie ISO 27001 eine Basis, um Risiko-Management nachvollziehbar und transparent sicherzustellen.

Die damit verbundenen Aufgaben wurden in der Vergangenheit im Bereich Netzwerk und Security mit kommerziellen Produkten adressiert, erfüllten aber nie die eigenen Qualitätsansprüche, welche sich im Wesentlichen auf Anpassbarkeit und Flexibilität beziehen.

Auf der Suche nach einer Weiterentwicklung in der Unterstützung der Netzwerk Überwachung und Sicherheit stiessen wir auf die Open-Source Lösung ntop, welche durch Luca Deri, einem italienischen Wissenschafter an der Universität Pisa, 1998 initiiert wurde.

Der Einsatz von ntop, sowie die Möglichkeit, unsere individuellen Wünsche einfach und effizient selber erweitern zu können, führte dazu, dass sämtliche im Einsatz befindlichen kommerziellen Produkte ersetzt werden konnten, und wir nun in der Lage sind, unseren Kunden nachvollziehbar, transparent und automatisiert die Sicherheit im Netz zu garantieren.

In flow (sFlow/NetFlow/IPFIX) collection, nProbe acts as a “flow processor” for ntopng . nProbe is responsible for sending ntopng flows after they have been processed that includes

• Collection mode: flow normalization that is the process of converting flows on a format that ntopng can understand. This happens if flow exporter devices (e.g. a router) use custom information elements. In addition nProbe takes care of difference in flow format between sFlow and NetFlow/IPFIX that despite of the common word “Flow” are very different in format.
• Probe mode: convert captured packets into flows that are then exported to ntopng.

When you configure flow collection you have two options that are described below. These solutions are pretty similar, and you need to choose which one fits your needs based on your firewall rules (who is the connector initiator?) and traffic policy (do you want to merge nProbe traffic on the ntopng side?).

Probe Mode

In collector mode ntopng connects to the various nProbes (i.e. ntopng is the connector initiator). In this case you need to define in ntopng one interface (-i) per remote nProbe you intend to connect to. If you need to aggregate multiple ntopng interfaces into one you can add “-i view:all” to merge them up onto a view interface.

Collector Mode

In probe mode multiple nProbes connects to same ntopng interface (i.e. nProbe is the connector initiator). In this case you need to define in ntopng one interface (-i) to which all remote nProbes will connect to. As this is in collector more, note that you need to add a small ‘c’ at the end of the interface definition in ntopng. In this setup all probe traffic is automatically merged into a single ntopng interface.

Enjoy !

In order to assist our community with 24/7 support, we have built an AI/LLM-based bot that has been trained on the ntop documentation (all products including ntopng, nProbe, nDPI…) and blog posts on this website. Currently this service is available in beta version and it is accessible using Discord on our ntop server (read here how to access it).

You can use it asking questions in plain English/German/Italian/French/Dutch/Spanish…. so we hope that the language barrier will finally be solved.

Please send us your comments and in case there is anybody willing to help us writing documentation or support material, please contact us.

Enjoy !

ntopng is not typically classified as an Intrusion Detection System (IDS) in the traditional sense, but it does have some features that overlap with IDS functionalities. Let me explain the differences and how ntopng might serve a similar role:

What is ntopng?

ntopng is an open-source network traffic monitoring tool that provides visibility into network traffic and performance. It is primarily used for:

• Network Monitoring: Tracking traffic flows, bandwidth usage, and the behaviour of network devices.
• Traffic Analysis: Deep Packet Inspection (DPI) based on nDPI to analyse protocols, applications, and performance.
• Real-time Visualisation: Giving insights into network traffic in real-time, including hosts, flows, and application usage.

While ntopng offers valuable data for monitoring and performance management, it lacks many of the core features of a dedicated IDS, such as signature-based attack detection or active prevention mechanisms.

Why ntopng is not a traditional IDS:

1. No Signature-Based Detection: Traditional IDS tools like Snort or Suricata work by comparing network traffic against a database of attack signatures. They detect known attacks like malware, port scanning, or exploit attempts by matching traffic patterns. ntopng does not have such a signature database.
2. Limited Focus on Security Alerts: While ntopng can provide visibility into anomalies or unusual traffic patterns, it is not primarily focused on alerting administrators to potential security threats like an IDS would.
3. No Active Prevention: Intrusion Prevention Systems (IPS), which are often paired with IDS, not only detect but also block or mitigate detected threats in real-time. ntopng lacks this capability. Note that ntopng Edge features active traffic blocking based on DPI and behavioural traffic analysis.

How ntopng can be Used Like an IDS:

While ntopng isn’t a full-fledged IDS, it can complement an IDS or offer some level of security monitoring by providing:

1. Anomaly Detection: ntopng has basic functionality for detecting anomalies, such as unusual traffic patterns, bandwidth spikes, or unknown protocols. These could indicate malicious activity like Distributed Denial of Service (DDoS) attacks or unauthorised access attempts.
2. Flow Monitoring: By monitoring traffic flows, ntopng can help identify suspicious behaviour, such as frequent connections to external IPs, unexpected services, or unusual bandwidth consumption. This can aid in identifying threats or compromised devices.
3. Integration with IDS Tools: ntopng can be used alongside traditional IDS tools like Snort or Suricata to enhance network visibility. It can capture and visualise network traffic, making it easier to investigate the incidents flagged by an IDS.
4. Deep Packet Inspection (DPI): ntopng uses DPI to analyse the content of traffic, providing insights into the types of applications or protocols being used. This can help detect the misuse of certain applications or suspicious traffic on non-standard ports.

Combining ntopng with IDS/IPS

• To get the best of both worlds, you can use ntopng for detailed traffic analysis and monitoring, while pairing it with an IDS/IPS (like Snort, Suricata, or Zeek) for active intrusion detection and response.
• ntopng can also export flows to tools like ELK (Elasticsearch, Logstash, Kibana) for correlation with IDS alerts.

Conclusion

While ntopng is not a dedicated IDS, it can be used to monitor network traffic for anomalies, provide valuable network visibility, and aid in threat detection. However, for serious security monitoring, it is best used alongside traditional IDS/IPS solutions.

Would you like help setting up ntopng with another IDS for combined monitoring?

Enjoy !

We continually strive to make the software configuration and management more flexible and easier for the users. To this end, we are excited to announce the launch of a new way of licensing the software feature: the centralised License Manager (LM). This tool simplifies software license management by dynamically allocating licenses to various application instances running within your network.

The LM is another option you can use in addition to “traditional” systemId-based licenses that we use today.

What is the centralised License Manager?

Managing software licenses across multiple instances within a network can be a complex task. As your infrastructure scales, maintaining the software and licenses updated becomes a challenge, especially in dynamic environments with containerised application instances which are continuously created and destroyed. The License Manager aims to finally solve this problem.

The License Manager is a software application that allows you to deploy all your ntop licenses in one place, a central location to which ntop applications will connect to. Differently from “classic” ntop licenses that are bound to a host permanent SystemId, License Manager licenses are tied to the host where the License Manager runs, regardless of where the applications (e.g. nProbe or ntopng) are running. Please note that the License Manager runs in your private network, and it does not require access to the Cloud. It implements an on-premise license verification and enforcement.

In fact, with this tool, licenses are no longer tied (and manually installed) to specific devices. Instead, they are dynamically assigned to software instances on demand. If an instance no longer requires a license, it is automatically revoked and reassigned to another. The License Manager consolidates all your licenses into one location, and also offers a complete overview over their distribution in a web-based interface. By allowing licenses to “float” between instances, you can ensure that your license pool is being used effectively. This prevents over-licensing and under-utilization, optimizing your investment in ntop solutions. When an instance shuts down or no longer needs the license, the License Manager will automatically reallocate that license to an active instance. 

How It Works

The License Manager operates as a hub within your network, managing the flow of licenses between instances. Here’s a high-level overview of how it works:

• License Pooling: licenses for the License Manager are added to a centralized license pool, and it is possible to monitor and manage this pool from the web interface.
• License Allocation: application instances running on your network request licenses from the pool when they are launched.
• Automatic Reallocation: as soon as a license is no longer needed by a particular instance, it is returned to the pool, ready for reallocation to another instance.
• Auditing: the tool maintains logs and reports, helping you stay compliant with license terms and providing clear insight into license usage across the network. This dramatically simplifies audit processes.
• Authentication: a user Token can be used to authenticate the application to control which instances are allowed to allocate licenses (multiple tokens are supported to handle multiple groups).

The License Manager is designed to adapt to your evolving network needs, delivering flexibility, efficiency, and complete control over your ntop software licenses. Whether you are scaling up for a large project, running multiple instances across geographically distributed networks, or simply looking to streamline your license management, this tool provides the solution you need.

As with traditional licenses, the LM is deployed on premise and it does not require Internet access in order to operate.

We are confident that this new feature will make managing your ntop licenses easier and more efficient. For more information about the License Manager please visit the documentation, which contains instructions on how to get started and set it up.

Currently the License Manager is used for large customers, service providers or users with dynamic container environments. Both “traditional” and LM licences prices have the same cost but you cannot use a traditional license with the LM as described above in the documentation.

Stay tuned for more updates!

Depending on the network topology and configuration, your monitoring tools can receive the same traffic multiple times. This problem is called data duplication. Duplication can happen at packet or flow level:

• Packet duplication
  The same packet is received multiple (usually twice) times, either one after the other, or within a short mount of time. Note that this has nothing to do with TCP data retransmission that is a totally different scenario.
• Flow duplication
  Two or more flow-devices observe the same traffic, and emit the same flow at the same time.

In both cases, the goal is to keep the first packet/flow and discard the duplicates. The main difference between these two use cases, is that with packets the deduplication time window measured in msec for flows in seconds. 

Packet Deduplication

In this case we can offer a few options

• If you have a few Gbits of traffic, nProbe features packet deduplication via the –enable-ipv4-deduplication command line flag that is used to discard consecutive IPv4 packet copies. You can use nProbe to feed ntopng and thus deduplicate in ntopng.
• if you have more traffic or if your duplicated packets are not one after the other, you need to use nDedup, a tool part of the n2disk package that basically buffers packets in memory and discard duplicates.
  You need to deploy nDedup as a bridge tool that discards duplicated packets and forward “clean” traffic to an interface to which you attach you favorite monitoring tools. Note that the faster is the network, the more memory you need to keep packets in memory, so avoid large windows (50 ms or more) that they might require a lot of memory at high speed rates.

Flow Deduplication

As with packet deduplication, nProbe can be used to deduplicate flows when used to collect flows. In this case nProbe keeps flows in cache for some time and if for a collected flow there is already another flow with the same key in memory, such flow is discarded. As explained in this post, you can use –flow-deduplication <interval (sec)> can be used to specify the sliding window duration, during which the duplication check is applied. Considered that flow timeout in exporters can be different, you can use windows of 30 sec, compared to msec you can use for packets. Also in this case, thanks to nProbe, your ntopng installation can benefit from flow deduplication when used in flow collection mode.

Enjoy !

Many users requested us a simple way to visualise hosts activity overtime. In essence have the ability to answer questions like:

• What hosts were active during the week-end
• When a host is using most of the network.
• What hosts were active when a certain event happened.

This is what hosts activity monitor does.

In the dev branch, ntopng has been enhanced with a new menu entry under the hosts page, that shows in a heatmap the activity of local hosts. From the menubar it is possible to specify an arbitrary periodo or predefines ones such as last hour/day/month. The darker is the bar, the more traffic a given hosts has sent/received during such period of time. So easy.

Enjoy !

Last week during SharkFest Europe 2024 we have presented what are network fingerprints and how they work.

During the talk we (Luca and Ivan) have described how we have extended nDPI with support of network fingerprints, and how this work has been also integrated in Wireshark. We believe that fingerprints are an interesting technology that can help in better understanding the nature of traffic flows, detect inconsistencies on crafted traffic (e.g. a Windows box that pretends to impersonate an iOS device), and of course in cybersecurity. In the coming months we plan to do a native Wireshark integration and also to use them in ntop tools such as ntopng.

For those how didn’t have the chance to attend the presentation, these are the presentation slides.

Enjoy !

Last week we have attended Suricon 2024, the annual conference about Suricata and presented our work on how nDPI has been integrated with Suricata. At ntop we like to contribute to other open source projects we use and like, such as Suricata and Wireshark. One of the main limitations of Suricata is its inability to monitor many protocols (currently the engine supports ~20 protocols compared to 450+ protocols supported by nDPI) and the lack of behaviour analysis that would very well complement Suricata signature-based analysis. These have been the reasons why we have decided to write some code to integrated Suricata with nDPI. Our code contribution is close to be merged into Suricata and we hope this will happen in the next couple fo weeks.

For those who have been unable to attend Suricon, these are our presentation slides to see what we have presented in Madrid.

Enjoy !

Sometimes it happens that your router is congested, and you ask yourself “How is it possible?” or “Who is responsible for congesting the network?” or “Which router/port is congested?”.

You could simply answer the last question by using the SNMP/Flow Exporters Usage: HowTo Monitor SNMP Interfaces Utilisation and Congestion Rate; but what about the other two?

Let’s start by looking at SNMP. As explained in the previous post, if SNMP is enabled on the routers/switches, using ntopng it is possible to figure out if an interface is congested. On the other hand, by using NetFlow/IPFix/sFlow we could understand who is responsible for producing/receiving the traffic. and eventually what activity that host/s is/are currently doing. So by combining these two pieces of information, we can achieve our monitoring goal. 

First, let’s start ntopng with ClickHouse enabled (-F option).
Then enable SNMP on all the Routers/Switches we want to monitor and poll those devices by adding them in ntopng.

Then export the flows from that device towards an nProbe, connected to ntopng (See Using ntopng with nProbe).

In order to have detailed SNMP measurements, we can increase standard 5 min polling to 1 min polling via Settings -> Preferences -> Timeseries, scroll down up to the Flow Exporter Timeseries and switch the “Flow Exporters and nProbes Timeseries Resolution” to 1m.

Now is to disable usage calculation all the noise i.e. thos devices we are not interested in.

Suppose that we are interested just in the 192.168.2.134 device (our router/switch); jump to SNMP and remove from the Usage page, every device other then the above switch. In order to achieve that, for all device we want to skip, go to the Configuration page (in green) and set the “Exclude From Usage” preference (see below).

Now everything is set up and you can analyse your network: go to SNMP Devices -> Usage page; where you should be able to see each monitored device/interface.

From this page you can click on the congested Interface and jump to the details of the interface. Then zoom in the period of time of the congestion.

On this page you can see that the congestion happened in the Uplink (out) direction. Here you can click on the Uplink (Out) Usage link (blue rectangle in the above picture) for correlating this information with Historical flows and figure out what was the root cause.

If you have read until here, you have learnt how to to monitor your Routers/Switches, check if a traffic congestion happened and understand what has beenthe root cause.

Enjoy!