Recently we have compared the use of nDPI in a realtime application (ntopng) and a near-realtime (nProbe). We have captured a short pcap with some mixed traffic and analysed it with both applications. The expectation was to find comparable results between the two applications, but this happened only partially. This blog posts explains the main differences between the two tools and why there are some discrepancies in results.
In our tests, we have configured both nProbe and ntopng to analyze the same pcap and write results on two different ClickHouse tables, so we can easily compare results. The first difference is the total traffic volume: nProbe is a NetFlow-probe that accounts traffic from the IP layer, not considering the ethernet (and VLAN) layers, whereas ntopng accounts all traffic. This means that if you want to compare data you need to deduct ethernet overhead from ntopng data.
| Application | ClickHouse SQL Query |
|---|---|
| nProbe | select IPV4_SRC_ADDR,IPV4_DST_ADDR,IPV6_SRC_ADDR,IPV6_DST_ADDR,L4_SRC_PORT,L4_DST_PORT,(IN_PKTS+OUT_PKTS) as PKTS, (IN_BYTES+OUT_BYTES) AS BYTES from flows ORDER BY BYTES; |
| ntopng | select TOTAL_BYTES,IPV4_SRC_ADDR,IPV6_SRC_ADDR,IPV4_DST_ADDR,IPV6_DST_ADDR,IP_SRC_PORT,IP_DST_PORT,PACKETS,TOTAL_BYTES-(PACKETS*14) as T FROM flows ORDER BY T ASC; |
Another difference between the two application is in how nDPI works. In case of a detected protocol using DPI (e.g. DNS or TLS) there are not differences. Instead for protocols such as Ookla (speedtest) or BitTorrent where the detection is based on a cache filled up with signalling data, timing is important. A ntopng is pure realtime application, nDPI detection based on cache happens when enough packets have been processed for a given flow, hence in case of no match a guess or a generic protocol (i,e. HTTP for Ookla) is used.
In the case of nProbe (and nProbe Cento), nDPI works the same way but in case of no match the guess is made during flow export towards the collector. This means that in the case of nProbe, the chance to find an entry in the nDPI cache is higher with respect to ntopng this is because guess can be made several seconds (or even one minute) after nDPI detection completed. The results is that for non DPI-detected flows, the nProbe detection can be more accurate than ntopng simply because nProbe does not have the same time constraints that ntopng has.
Hope this post has clarified how the two applications work, so that you can now understand why you might see tiny differences (that are not bugs but ‘by design’) between the two applications.
Enjoy !