FOSDEM is the leading open source conference in Europe and it will take place this week-end in Brussels, Belgium. As we have developed open source software since 20 years, we believe we have right to belong to this community. This year we’ll have a stand on Sunday (Building K – Level 2) and also a talk in the Monitoring and Observability room on Sunday.

The ntop core team will attend the conference, and it would be great for us to meet our community, and distribute you some gadgets. We need your feedback.

Hope to see you tomorrow and Sunday at FOSDEM!

This February we’ll introduce ntopng v4 and we’re starting to write some blog posts to preview the new features. Let’s start with the user interface. Since v1 the UI has always been the same. People however asked us some more flexible layout where it is possible for instance to switch across network interfaces in a breeze. Furthermore the pervasive use of dark themes was also a driving force towards changes. While the UI in 4.2 will integrate new changes we already planned (for instance to switch from realtime to historical view, while currently it is mixed) let’s start with a new sidebar design that should leave more space on wide screens.

With the new UI, already part of the dev branch/packages, you can have both dark and light theme that you can choose/switch from the preferences panel.

So you can decide which one fits best for your (eye) needs. Please let us know on the telegram channel your feedback as we’re still making little changes.

Enjoy!

JA3 is a popular method to fingerprint TLS connections used by many monitoring tools and IDSs. JA3 focuses on encryption options specified during TLS connection setup to fingerprint the encryption library used by the application.

Image courtesy of Cisco

So in essence the same JA3 fingerprint will match multiple applications, making JA3 unreliable (when used as single feature) to fingerprint traffic. There are several JA3 fingerprint databases available on the Internet you can use to identify (remember with some grade of uncertainty, thus with false positives) client applications or malware applications.

Now suppose you want to ask yourself a different question. What is the nature of the client behind a TLS request. Is this a web browser or a command line application? Is this perhaps Google Chrome or Apple Safari? Identify a browser vs a different type of application can help to prevent (or score differently) connections, not to say that this (usually via JavaScript redirect) is a common practice for WAF (Web Application Firewalls) and Ant-DDoS protection tools to keep away non-browser applications.

Another question to ask is: what is the nature of this communication? Is this a web application (hence HTML/Javascript or similar) or perhaps a TLS VPN where there is no interest in HTTP at all? The latter in particular is very important to spot VPNs in the wild (remember that popular VPN such as OpenVPN are already detected by nDPI, so we want go beyond this type of detection).

In TLS, and in particular in the Client Hello packet, there is more than encryption information that is then ignored by JA3. In order to speedup protocol negotiation, years ago a new TLS extension called ALPN (Application-Layer Protocol Negotiation) has been defined. In essence the client application tells the contacted server what are the protocols the client supports for data exchange, hence after the TLS connection has been setup.

In addition to ALPN there is another extension (not used by JA3) that can help with application fingerprinting and that is the list of encryption protocol versions supported by the client. Contrary to ALPN, this field won’t help understanding the nature of the client application but it can definitively help with fingerprinting.

Let’s look at some examples.

Apple Safari	Google Chrome	Mozilla Firefox

As you can see, combining ALPN with TLS protocol version, you can fingerprint these three applications. However if you look closer to ALPN you will notice that all three applications support various HTTP versions, ranging from HTTP 1.1 to 2.0 to SPDY.

Let’s now look at some command line tools to see how they behave.

OpenVPN	curl	git

Both Curl and git commands speak HTTP and thus they interact with a web server, whereas OpenVPN does not have such extension in the ClientHello that means there is not HTTP server on the server side and thus that this is not web browser-like communication but it has a different nature as you would expect from a VPN.

You might now wonder how to do this analysis without using Wireshark, knowing where in the traffic to look at and how to interpret the information. Well, nDPI is the correct answer. Below you can see how nDPI (in particular the output below comes from ndpiReader) fingerprints the traffic

TCP 192.168.2.17:49360 <-> 31.13.86.52:443 [proto: 91.211/TLS.Instagram][cat: SocialNetwork/6][153 pkts/11644 bytes <-> 206 pkts/284089 bytes][Goodput ratio: 13.2/95.2][2.91 sec][ALPN: http/1.1][TLS Supported Versions: TLSv1.3,TLSv1.3 (Fizz)][bytes ratio: -0.921 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 23.0/0.1 2756/16 247.4/1.2][Pkt Len c2s/s2c min/avg/max/stddev: 66/66 76.1/1379.1 592/1454 67.7/296.4][TLSv1.2][Client: scontent-mxp1-1.cdninstagram.com][JA3C: 44dab16d680ef93487bc16ad23b3ffb1]

Now that you can have all this analysis ready to use, you can contribute to nDPI development by coding a simple script that can

• Identify HTTP vs non-HTTP based TLS communications
• Fingerprint web browsers using the method shown above.

Enjoy!

This year the annual ntop conference will take place in Milano, Italy on June 9-10, at Università Bocconi, one of the most prestigious university in Italy. As usual the first day will be used to train people on ntop tools and the second day is dedicated to the conference and community meeting.

As this is an open event, we believe the ntop community should attend it and also make it. For this reason we have created a call for presentation where we ask all of you to propose topics or give a talk at the event. As you can read this time we would like to have also talks in English in addition to those in Italian as in the past conferences.

All details can be found here.

Please contribute, we want to hear your voice!See you in Milano this June.

This is to announce the new stable release of nDPI 3.2. The main trend of nDPI is to move from “simple” application protocol detection towards behavioral traffic interpretation. This has been implemented with the integration of modules for detecting attacks (e.g. SQL injections and XSS in HTTP) and behavioral indications on packet length/time/entropy as well indicators used for creating simple indicators typical of IDS systems. In essence nDPI is moving from protocol reporting to comprehensive traffic interpretation.

nDPI now includes functions for efficiently serialising data in both JSON and binary format, that are used for instance by ntopng/nprobe to communicate more efficiently than before using less bandwidth and being faster.

In addition thanks to the work of Philippe Antoine, nDPI is now continuously analysed by ClusterFuzz for searching memory leaks, invalid memory accesses all based on libfuzz. In essence nDPI is a better product now.

Enjoy!

This is the complete changelog:

• New Features
  • New API calls
    • Protocol detection: ndpi_is_protocol_detected
    • Categories: ndpi_load_categories_file / ndpi_load_category
    • JSON/TLV serialization: ndpi_serialize_string_boolean / ndpi_serialize_uint32_boolean
    • Patricia tree: ndpi_load_ipv4_ptree
    • Module initialization: ndpi_init_detection_module / ndpi_finalize_initalization
    • Base64 encoding: ndpi_base64_encode
    • JSON export: ndpi_flow2json
    • Print protocol: ndpi_get_l4_proto_name / ndpi_get_l4_proto_info
  • Libfuzz integration
  • Implemented Community ID hash (API call ndpi_flowv6_flow_hash and ndpi_flowv4_flow_hash)
  • Detection of RCE in HTTP GET requests via PCRE
  • Integration of the libinjection library to detect SQL injections and XSS type attacks in HTTP requests
• New Supported Protocols and Services
  • TLS: new decoder
  • Added ALPN support
  • Added export of supported version in TLS header
  • Added Telnet dissector with metadata extraction
  • Added Zabbix dissector
  • Added POP3/IMAP metadata extraction
  • Added FTP user/password extraction
  • Added NetBIOS metadata extraction
  • Added Kerberos metadata extraction
  • Implemented SQL Injection and XSS attack detection
  • Host-based detection improvements and changes
  • Added Microsoft range
  • Added twitch.tv website
  • Added brasilbandalarga.com.br and .eaqbr.com.br as EAQ
  • Added 20.180.0.0/14, 20.184.0.0/13 range as Skype
  • Added 52.84.0.0/14 range as Amazon
  • Added ^pastebin.com
  • Changed 13.64.0.0/11 range from Skype to Microsoft
  • Refreshed Whatsapp server list, added *whatsapp-*.fbcdn.net IPs
  • Added public DNSoverHTTPS servers
• Improvements
  • Reworked and improved the TLS dissector
  • Reworked Kerberos dissector
  • Improved DNS response decoding
  • Support for DNS continuous flow dissection
  • Improved Python bindings
  • Improved Ethereum support
  • Improved categories detection with streaming and HTTP
  • Support for IP-based detection to compute the application protocol
  • Renamed protocol 104 to IEC60870 (more meaningful)
  • Added failed authentication support with FTP
  • Renamed DNSoverHTTPS to handle bot DoH and DoT
  • Implemented stacked DPI decoding
  • Improvements for CapWAP and Bloomberg
  • Improved SMB dissection
  • Improved SSH dissection
  • Added capwap support
  • Modified API signatures for ndpi_ssl_version2str / ndpi_detection_giveup
  • Removed ndpi_pref_http_dont_dissect_response / ndpi_pref_dns_dont_dissect_response (replaced by ndpi_extra_dissection_possible)
•  Fixes
  • Fixed memory invalid access in SMTP and leaks in TLS
  • Fixed a few memory leaks
  • Fixed invalid memory access in a few protocol dissectors (HTTP, memcached, Citrix, STUN, DNS, Amazon Video, TLS, Viber)
  • Fixed IPv6 address format across the various platforms/distributions
  • Fixed infinite loop in ndpi_workflow_process_packet
  • Fixed SHA1 certificate detection
  • Fixed custom protocol detection
  • Fixed SMTP dissection (including email)
  • Fixed Telnet dissection and invalid password report
  • Fixed invalid category matching in HTTP
  • Fixed Skype and STUN false positives
  • Fixed SQL Injection detection
  • Fixed invalid SMBv1 detection
  • Fixed SSH dissection
  • Fixed ndpi_ssl_version2str
  • Fixed ndpi_extra_dissection_possible
  • Fixed out of bounds read in ndpi_match_custom_category
• Misc
  • ndpiReader
    • CSV output enhancements
    • Added tunnelling decapsulation
    • Improved HTTP reporting
    • Added scan and HTTP attacks (XSS, SQL Injection) detection

This is to announce a new PF_RING major release 7.6.

Besides bug fixes and drivers updates to improve compatibility with latest kernels (including those shipped with Debian 10 and CentOS 8) this release includes many enhancements to the PF_RING FT library, which delivers unprecedented flexibility and all the features a flow-based packet processing application requires. Latest additions include:.

• Flow slicing: the library delivers periodic flow updates, no need to wait for flow termination.
• Tunnels decoding: packets are decapsulated and information about the tunnel are exposed by the library.
• More flow metadata: more L7 flow metadata are exposed by the library for common protocols (e.g. HTTP, DNS, TLS, ICMP), additional metadata can be extracted by accessing the nDPI handle directly.
• Flow termination reason: exported flow now include the termination reason (e.g. max lifetime, idle time, tcp termination, etc.).
• More control on the engine: more advanced settings are now available to control the behaviour of the flow table and deep packet inspection engine.

Writing a fast traffic classifier with metadata extraction and extended capabilities is nothing more than a school homework now.

This is the complete changelog:

• PF_RING Library
  • New pfring_open flag PF_RING_TX_BPF to evaluate the BPF filter also for TX
  • New pfring_open flag PF_RING_FLOW_OFFLOAD_TUNNEL to dissect tunneled traffic in flow-offload mode
  • New pfring_open flag PF_RING_DISCARD_INJECTED_PKTS to discard stack-injected packets
• ZC Library
  • New API call pfring_zc_close_device to close a ZC interface
  • New ‘flags’ parameter to pfring_zc_create_cluster
  • Fixed memory allocation in case of more than 4GB of buffer size
• FT Library
  • New API call pfring_ft_set_filter_all_protocols to reset all filtering rules
  • New API call pfring_ft_set_license to set a license at runtime
  • New API call pfring_ft_flow_get_ndpi_handle to access the flow nDPI handle
  • New pfring_ft_l7_protocol_id, pfring_ft_get_ndpi_handle to access the nDPI handle
  • New pfring_ft_flow_value status field to get flow termination reason
  • New PFRING_FT_TABLE_FLAGS_DPI_EXTRA flag to enable extra metadata extraction
  • New PFRING_FT_DECODE_TUNNELS flag to decode tunnels, new tunnel_type item in the flow value
  • New flow slicing support (pfring_ft_flow_set_flow_slicing API)
  • Added CAPWAP support
  • Added flow metadata for HTTP/DNS/TLS
  • Added global ‘default’ section to the rules configuration file
  • Added dpi_min_num_tcp_packets / dpi_min_num_udp_packets to the configuration file
  • Added flow_idle_timeout / flow_lifetime_timeout to the configuration file
  • Added src/dst mac to the exported flow key
  • Added ICMP type/code to flow metadata
  • Added flags to flow metadata
  • Added custom flow actions to be defined by the user
  • Added pfring_ft_load_configuration_ext API
  • Improved protocol detection for some protocols like Skype
  • Improved metadata extraction for some protocols like Telnet
  • Improved pfring_ft_license to return the duration also in demo mode
  • Changed l7_detected callback: this is called before the flow_packet callback now
  • Changed pfring_ft_create_table and pfring_ft_flow_value to allocate user metadata as part of the flow structure
  • Fixed filtering/shunting of custom protocols
  • Fixed protocol detection in case of guess
  • Fixed pfring_ft_set_l7_detected_callback user parameter handling
• PF_RING-aware Libpcap
  • Fixed device name check during socket initialization to handle long interface names
  • Fixed loop break
• PF_RING Kernel Module
  • Added new clustering mode cluster_per_flow_ip_with_dup_tuple
  • Allow any to capture from any namespace (on the host only)
  • Remapping ifindex to an internal device index to handle ifindex growing indefinitely
  • Fixed kernel crash parsing malformed packets (12 nested QinQ VLAN headers with GRE)
  • Fixed possible race condition
  • Fixed QinQ VLAN and VLAN offload support
  • Fixed concurrent access to the ring in case of loopback device and bridge
  • Compilation fixes for kernel 5.x
  • Reworked max ring size check to handle cases like jumbo frames
  • Improved promisc management
• PF_RING Capture Modules
  • New AF_XDP capture module
  • Napatech library update, fixed findalldev
  • Accolade library update, fixed caplen vs orig len, new env var ACCOLADE_FLOW_IDLE_TIMEOUT
  • Myricom library update, license fix with port aggregation
  • DAG library update
• ZC Drivers
  • New ixgbe-zc driver v.5.5.3
  • Support for Intel X550
  • Compilation fixes for kernel 5.x
  • Handling if up/down when the interface is in use by ZC
• nBPF
  • Added support to match custom fields through a callback (nbpf_set_custom_callback)
• Examples
  • zcount improvements:
    • Added -T option to capture TX
  • zbalance_ipc improvements:
    • Fixed -m 4/5/6 with multiple applications and more than 32 queues
    • New -E option to enable debug mode
    • New -C <FT config file> and -O <nDPI proto file> options
  • ftflow_dpdk improvements:
    • More stats: drops, hw stats, per-queue throughput
    • New options to control the link status, flow control, autoneg, port speed, checksum offload
    • New -P option to set the TX rate
    • New TX test mode and -T option to set the packet len
    • New -F option to enable forwarding
    • New -m <len> option to set the mtu
    • Capture-only mode
    • Forward optimizations
  • ftflow_pcap improvements:
    • Support for processing a PCAP file
    • New -p <proto.txt> option
    • New -F <file> option to configure filtering/shunting rules
  • pfsend improvements:
    • New -8 <num> option to send the same packets <num> times before moving to the next
    • New -B <bpf> option to set a BPF filter
    • New -t option to forge N different source port
    • New -A option to generate increasing number of flows
  • pfcount improvements:
    • New -R option to disable RSS reprogramming
  • pfbridge now discards injected packets
• Misc
  • New pf_ringcfg script to automatically configure pf_ring and drivers
  • New pre/post scripts executed by systemd before/after loading pf_ring and drivers
  • Improved hugepages configuration with multiple nodes
  • npcap library enhancelents, storage utility functions fix for NFS

Enjoy!

Last year you have read how we have integrated Suricata support into ntopng. While an IDS is a good source of data, it is just a sensor, how has no knowledge of the big network picture including the network overview, past host/flow history and device type. In essence an IDS is a nice to have but it’s not enough. What it is necessary is the ability to analyse traffic, learn what is wrong, compare current behaviour with the past, and draw some conclusions (i.e. read them as emit alerts) that could be used by network analysis to understand what’s wrong in the network. This is in essence what we have implemented in the upcoming ntopng.

All is driven by user scripts that can be accessed from the setting menu.

Scripts are executed periodically (e.g. every minute) or at events (i.e. when a flow protocol is detected). Here you can see the entities on which scripts are applied.

Below you can see flow scripts

and host scripts

You can modify the script thresholds by clicking on the edit button where you can specify parameters that drive the alert generation.

Scripts are used to generate alerts that are then consolidated per entity (hosts, autonomous systems, networks…) with a numerical value called score that is computed every minute based on the alerts generated by the scripts.

Under the flow/hosts page you can sort items according to the score to immediately see the consolidated view of where the problems are. Just sort according to the score to let ntopng drive you where problems are.

If you click on a host you can see the current number of issues reported, and the active alerts number.

Clicking on the chart next to the score field, you can see how the host score changed overtime.

Of course you can use the alert and flow explorer to drill-down as see the exact nature of the network issues.

All this is part will be part of the upcoming ntopng v4. Enjoy with the dev build now, and soon with v4.

NetFlow/IPFIX specifications have not considered privacy and confidentiality important. Exported flows are sent over unencrypted channels that prevent them to be exchanged on public networks unless techniques such as VPNs are used. Today encryption is no longer an option, and thus we have added encryption support in all our tools when flows are exchanged over ZMQ channels (e.g. when nProbe sends flows to ntopng). In order to use encryption a private/public keypair needs to be generated on the collector side (i.e. ntopng) and configured on all the probe applications sending flows to ntopng (e.g. nprobe or cento). In order to do this, the following steps needs to be performed:

1. Start ntopng adding the —zmq-encryption option. This option will generate, if not existing, the encryption keypair for all configured ZMQ interfaces.
2. Connect to the ntopng web gui and select the ZMQ interface information page and copy the value you have to put in –zmq-encryption-key <…> parameter
   
3. Start nprobe sending flows to the collector ZMQ interface according. In the above example something like nprobe --zmq tcp://127.0.0.1:1234 --zmq-encryption-key 'k8{zm*.r&99Iavul#bt9(H:[8RYco?Ko58l&6(((' -i ethX will do the job

As ZMQ keypairs are generated per interface, you need to make sure that the pair <nprobe, ntopng> will use the same encryption key values for the same ZMQ interface.

Note:

• If you have multiple ZMQ interfaces, each interface will have a different keypair that you need to configure on the corresponding probe.
• Data encryption is currently available in nightly builds and will be included in the next stable release.

You can now safely exchange flows over public networks overcoming the limitations of NetFlow/IPFIX.

Enjoy!

This is to introduce nProbe 9.0 stable release whose the two main features are traffic behaviour analysis and high speed flow collection.

Traffic Behaviour Analysis

When in 2002 nProbe development started, the idea was to create a drop-in replacement for physical probes present in routers. Later the advent of IPFIX pushed the monitoring community towards standardisation of flow exports, and promoted interoperability across probes and collectors. Then the market started to ask solutions for visibility (and not just traffic accounting), and we developed nDPI for going beyond port and protocols and tell exactly the application protocol being used. In the past months, in addition to further improving DPI, we have decided to got into a new land of traffic behaviour analysis. The recent changes in nDPI are just a part of the work we’re carrying on and that will be completed in the next few months. In essence the idea is to use tools such as nProbe to fingerprint traffic not just in terms of protocol, but also in terms of behaviour, i.e. how a certain host behaves when using a specific protocol when talking with a remote site. The reasons are manyfold, including the ability to interpret encrypted traffic that is no longer inspectable and thus that leaves network analysts blind. This is the direction we’ve taken and that will be consolidated in the next nProbe release. In essence nProbe has to be able to answer questions like: my surveillance camera is working normally or some changes in behaviour happened?

Behaviour analysis in nProbe is based on time and payload packet length bins, that are basically containers of values belonging to a specific range.

• %SEQ_PLEN
  it classifies packets payload (past 3WH for TCP) for the first 256 flow packets into 6 bins: <= 64, 65-128, 129-256, 257-512, 513-1024, >= 1025. Example if your payload bytes are 100,400,256,1064,1400 the SEQ_PLEN will be 0,1,1,1,0,2.
• %SEQ_TDIFF
  it classifies packet IAT (inter-arrival time) (past 3WH for TCP) for the first 256 flow packets into 6 bins: <= 1 ms, 1-5 ms, 5-10 ms, 10-50 ms, 50-100 ms, 100+ ms.
• %ENTROPY_CLIENT_BYTES and %ENTROPY_SERVER_BYTES
  Byte entropy on packet payload for both directions. A high value means that the bytes are more spread (high variance) with respect to low values where data is more predictable. Many protocols have typical values: DNS 4.2, TLS 7.7, NetFlow 4, SkypeCall 6.

These values are the first pillar for behaviour analysis. For instance you can easily detect a scan or hearthbleed time/length bins, or see if a flow misbehaves when their distribution changes with respect to typical values. If you apply this to host level (ntopng already does it) you can complete this analysis in a simple yet effective way.

High Speed Flow Collection

Traditionally nProbe has focused mostly on the probe side. Over the years, with NetFlow/IPFIX being more pervasive in network devices, we have decided to enhance the flow collection capabilities of nProbe. This is in particular since people started to use ntopng to visualise data (10 years ago most people used nProbe to export flows towards commercial collectors and not to self-visualise them with ntop tools). In order to increase the typical nProbe collection speed, we have first implemented –disable-cache for collected flows to disable the nProbe caching and thus emit flows quicker. In this release we introduce a new option named –collector-passthrough that is basically a 1:1 flow conversion from collected flows to ntopng (so the idea of this flag is to use it with ntopng for increasing the collection speed. To further improve the speed we are now using the new nDPI data serialiser that can write data in both JSON (legacy) and binary format that should be used when performance matters. Note that with –collector-passthrough you do not need to specify –disable-cache as in passthrough mode the cache is totally bypassed and thus it has not effect.

The following table reports test outcome on a Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz, binding nProbe to a single core (yes. if you have 4 cores you can multiply the numbers below by a factor close to 4).

Bottom line. With nProbe 9.0 you have a collection improvement with respect to the old version of 6.6x. Not bad! Note that for binary mode you need ntopng version 3.9 or better.

Flow Collection Across Firewalls

Firewalls and security devices are compulsory devices in every modern network in order to enforce security policies. However if you have a router/switch/probe outside of your firewall (e.g. on a DMZ or public cloud) and you want to collect NetFlow/sFlow/IPFIX using nprobe sitting behind a firewall, this can be a problem. Another problem that sometime people has to face with, is the need to collect flows over the Internet: this is because flows are sent in clear text and transit on a public network is never a good idea. For this reason we have developed a free, nProbe companion tool, named flowRelay that solves this problem. This is how flowRelay works.

$ flowRelay -h
Welcome to flowRelay v1.0: sFlow/NetFlow/IPFIX flow relay
Copyright 2019-20 ntop.org
flowRelay [-v] [-h] -z <ZMQ enpoint>] -c <port>
   -z <ZMQ enpoint> | Where to connect to or accept connections from.
                    | Examples:
                    |   -z tcp://*:5556c        [collector mode]
   -c <port>        | Flow collection port
   -k <ZMQ key>     | ZMQ encryption public key
   -v               | Verbose
   -h               | Help

In essence nProbe connects to flowRelay (the other way round won’t work due to firewall/NAT) that will then forward flows to nProbe via the encrypted ZMQ channel (you can read more here). flowRelay will not dissect the flow packets but it will check if the received traffic is really NetFlow/sFlow/IPFIX, this to avoid sending nProbe data that is not appropriate. Of course for maximum security you can set an ACL that allows only selected probes/routers/switches to send data to the flowRelay.

Enjoy!

Changelog

New Features

• Added flowRelay tool to collect flows through a firewall.
• Data serialisation based on nDPI: data can now be serialised in either JSON or binary format.
• Implemented %SEQ_PLEN, %SEQ_TDIFF, %ENTROPY_CLIENT_BYTES and %ENTROPY_SERVER_BYTES.
• Implemented –dump-bad-packets in collector mode for dumping invalid collected packets
• Improved GTPv2 plugin with dissection of many new fields.
• Implemented GTP v1/v2 per IMSI and APN aggregation via –imsi-apn-aggregation.
• Added HPERM and TZSP encapsulation support.
• Added %SSL_CIPHER %SSL_UNSAFE_CIPHER %SSL_VERSION.
• Implemented JA3 support via %JA3C_HASH and %JA3S_HASH.
• Implemented %SRC_HOST_NAME and %DST_HOST_NAME for host symbolic name export (if known).
• Implemented IP-in-IP support.
• Various Diameter plugin improvements.
• Implemented –collector-passthrough.
• Added support for IXIA packet trailer.
• Extended statistics reported via ZMQ so ntopng can better monitor the nProbe status.
• Implemented ZMQ encryption.
• Various Kafka export improvements (export plugin).
• Added support for the latest ElasticSearch format (export plugin).
• Implemented flow collection drop counters on UDP socket (Linux only).

Fixes

• Fixed NetFlow Lite processing.
• Implemented various checks for discarding corrupted packets that caused nProbe to crash.
• Fixed flow upscale calculation (e.g. with sFlow traffic).
• Application latency is now computed properly for some specific TCP flows with retransmissions.
• Fixed DNS dissection over TCP.
• Improvements in Out-of-Order and retransmission accounting.
• Fixes for %EXPORTER_IPV4_ADDRESS with sFlow.
• Support for the new GeoIP database format.
• Extended the @NTOPNG@ template macro with new fields supported by ntopng.
• Fixed bug in RTP plugin that was leading in some cases nprobe to crash.
• Various minor fixes in the nProbe engine.

This is to introduce the new nScrub 1.4 stable.

Besides a few bug fixes (mainly to the API) this release introduces many improvements, including:

• Full IPv6 support both in routing and bridge mode.
• Improved TCP protection, it is now possible to use SYN Proxy in asymmetric mode.
• Hardware bypass with watchdog support as failover mechanism in case of system failures or to handle maintenance.
• New plugins SDK to easily extend the core engine with custom protection algorithms.
• Native systemd support for multiple instances to handle multiple network segments.
• Support for Ubuntu 18, Debian 10, CentOS 8.

We are quite sure you will like and take advantage of every little improvement we made in this release. Enjoy!

Changelog

• Engine

  • Support for IPv6 neigh table
  • Support for IPv6 routing table
  • Pure SYN Cookie with encryption is used when in SYN Proxy mode is selected in asymmetric mode
  • More event types in event notification, sending notificaitons when an event terminates, added the ‘status’ field to the events
  • Full bypass support in routing mode
  • Uniq event identifier
  • Added options to use different ports for transmission (this is needed by Napatech as streams are RX only), Napatech is supported in transparent bridge only
  • TCP flags sanity check happens only if there is any 3whs check enabled
  • Hardware bypass watchdog support, integration with Silicom hw bypass adapters
  • Automatically setting hw bypass (if any) on application shutdown
  • Introduced HSP service type (Generic Hosting Service Provider)

• Plugins

  • New SDK (sample plugin available to get started)
  • Traffic blocking with pre/post hooks
  • Ability to inject packets
  • Inspection of both WAN and LAN traffic
  • Callback called on SIGUSR1 for reloading the configuration
  • Plugins stats (pre/post discard/forward counters) in the target stats

• API

  • API to discard http requests which are not matching a list of hostnames
  • API to purge all targets
  • API to print the full list of global VLAN mappings
  • API to set a limit to the IP whitelist to engage session whitelisting automatically
  • API to control the TCP check engage
  • API to drop TCP SYN packets with payload
  • API to set max DNS subdomain length (block watertorture)
  • API to drop TCP SYN with no options
  • API to purge all attackers (and delete all lists)
  • API and cli command to set the WAN/LAN interface IP address (required with DPDK TAP interfaces in routing mode)
  • Support for subnets without mask
  • Target stats now supports regex to select targets
  • Add bytes stats per protocol per target
  • wl_threshold: automatically turn off when IP whitelist size comes back below the threshold
  • Add gateway mac address to the arp table returned via REST
  • Added ability to overwrite the default virtual scrubber 0.0.0.0/0 – 0::0/0
  • Dynamic purge fix
  • Attacker search optimizations
  • Attackers list pagination, added list size when listing lists
  • String patterns removal using the ‘-‘ special char
  • Add discard reason to target stats
  • Fixed lists counters
  • Fix all lists purging
  • Stats fixes

• Tools

  • New nscrub-bl to manage blacklists
  • nscrub-cli
    • History now keeps also wrong commands
    • Add new command to load list from file in nscrub-cli
    • Load lists in batch mode
    • Fix integer parameters
  • nscrub-export reworked and improved options
    • Full configuration backup/restore
    • Add -i option to select the nscrub instance

• Packages

  • New Ubuntu 18 package
  • New Debian 10 package
  • New CentOS 8 package

• Misc

  • Systemd support for multiple nscrub instances
  • The nscrub service is now ‘PartOf’ the pf_ring service
  • Running nscrub as ‘nscrub’ by default, and falling back to nobody if it does not exist

This is to introduce you the latest n2n 2.6 stable release. This is mostly a maintenance release to address the issues of 2.4 that has been the first release since a long time of silence. The main features are

• AES encryption that features an overall speed bump (12x speed) and security with respect to twofish used in the previous n2n version.
• Extensive Windows and OpenWRT support.
• Full peer-to-peer topology support.
• Stable and more resilient connection.

Below you can find the complete changelog..

Enjoy!

Changelog

– Add ability to specify a whitelist of allowed communities on the supernode
– Implement local peers discovery via multicast
– Windows compilation fixes and instructions
– MacOS compilation fixes and instructions
– Add support for multiple edge systemd services
– Implement AES encryption for increased security and throughput
– Add benchmark tool for the encryption throughput
– Improve the connection stability and the chances to establish a P2P connection
– Remove keyschedule support to simplify the encryption code
– Integrate the changes made in the meyerd fork of n2n
– Implement packet stats for P2P vs supernode communication
– Replace peers linked list with hash table for faster lookup in big networks
– Automatically drop privileges to user n2n
– n2n version improvements
– Add support for ARM64 build
– Instructions and makefile file to build n2n on OpenWRT
– More options to control MTU, P2P connections, TOS and log verbosity
– Implement a wireshark dissector for the n2n protocol
– Remove calls to system() in tuntap_linux and use netlink instead
– Implement n2n-decode utility to decode and dump traffic to PCAP

After over one year of work, we’re proud to announce you that ntopng 4.0 is finally out. In this time we have redesigned ntopng for speed and openness, by breaking apart the existing monolithic C++ engine into a Lua-scriptable micro-engine. This is to enable people to contribute to the project without them being scared of coding in C++.

The major breakthroughs we have brought with this release are:

• A plugin engine that allows anyone with some basic Lua coding skills to tap straight into every single flow, host, or other network element and perform custom actions such as the triggering of alerts
• A fresh graphical user interface, featuring new icons, a refreshed look and feel, and light/dark themes
• The active monitoring of hosts Round Trip Times (RTTs) for HTTP, HTTPS, ICMP and ICMPv6
• Many cybersecurity-related features, including SSH and TLS fingerprinting, TLS analysis and alerting, and Suricata events integration

The plugin engine is probably most important breakthrough of v4: all the entities (e.g. flows, hosts, networks, SNMP…) are now driven by scripts that are executed based upon events. A typical event is a new flow dissected, a SNMP polling session, or a host just appeared on the network. Scripts can perform actions (e.g. monitor RTT of selected hosts issuing periodic ping) or trigger alerts based on thresholds or severity reported by the score. More in detail:

• Flows: flow plugins execute when a new flow is detected, when it’s been active for a certain time, or when it’s gone idle. Want to create an alert as soon as a flow is found to be going to a suspicious country? Need to detect Dropbox flows that are transferring too much data or outside business hours? Those are just a couple of things that can be done with flow plugins!
• Hosts: host plugins execute periodically on all active hosts. This allows anyone to create custom actions of virtually any kind. For example, host plugins can be written to detect hosts that are doing too much traffic, or to spot those that are probing the network with SYN scans, just to name a few.
• Other network elements: other plugins can be executed for instance when an interface is dropping too many packets, when the load of the system on top of which is running ntopng exceeds a given threshold, or when an interface of an SNMP device goes down, among many others.

Everything happens through the GUI: enable/disable/view source code / set alarm thresholds. You can read more in the development section of the user’s guide for developing your own plugin.

As you can have noticed from the screenshots above, v4 has a new, responsive look and feel. The navigation bar has been moved from the top to the left of the page to optimally use the space available on wide screens which are common today. In addition, the status bar, which was previously placed at the bottom of the page, has  been moved to the top, to make the essential ntopng status information always visible and available to the user. The user interface is now skinnable through themes, including a dark theme designed to reduce eyestrain to those that spend a lot of time in front of the screen.

As previously said, we have decided to enrich ntopng with new cybersecurity-oriented features mostly based on the work we have done on nDPI. In particular, 4.0 features SSH fingerprinting with hassh, TLS fingerprinting with JA3. Fingerprints are good indicators for good/bad traffic, especially today where the traffic is encrypted and thus it is increasingly difficult to understand it. Check out “Effective TLS Fingerprinting Beyond JA3” and “How Encryption Changed Network Traffic (Monitoring). Finally.” for extra information. TLS is not only fingerprinted, it is also thoroughly analyzed to check (and alert) when invalid, expired or mismatched certificates are detected or when certain cipher suites are weak. Everything about the TLS analysis is explained in the article “TLS/SSL Analysis: When Encryption and Safety Are Not Alike“. In terms of security, v4.0 can also easily be connected with Suricata for unprecedented visibility into traffic and security events!

Another new feature of v4 is the ability to actively monitor hosts latencies (aka RTT, round trip time): we’re complementing traditional passive traffic monitoring with active probing for keeping track of selected relevant assets. Currently, RTT probes are available for HTTP, HTTPS, ICMP and ICMPv6 and allow the user to quantify hosts RTT, and receive alerts when anomalous RTT values are detected or when monitored hosts become unreachable.

A comprehensive list of changes introduced in this major release is available in the changelog. As usual, ntopng installation instructions are available at packages.ntop.org. Try and experiment it, suggestions and contributions are very welcome! And why not joining us on our official telegram channel @ntop_community?

This release has been a major effort, and we expect people to contribute with plugins and scripts. We aim at creating a single comprehensive tool for network security and visibility. Hope to see your pull requests soon on GitHub.

Enjoy!

Meetup Logistics

For a long time out MacOS users asked a native ntopng package.

Even though we use MacOS and Linux to develop our tools, we didn’t consider important to revamp the MacOS installer (ntopng 1.x has a native MacOS installer) as there are solutions such as brew.sh that allow you to install packages. However such tools are not for all Mac users who not necessarily like to use the command line. Thus we have created a native package that installs in one click all the required components (e.g. Redis) needed by ntopng.

You can now go to packages.ntop.org and click on your platform for downloading the package you are looking for.

If you need to know more about ntopng, you can glance through the user’s guide that contains detailed installation instructions.

The latest stable ntopng 4.0 features a Round Trip Time (RTT) monitor which is capable of pinging hosts on a minute-by-minute basis to check:

• IP reachability with ICMP and ICMPv6 pings
• Web servers functionality with HTTP and HTTPS pings

Checks account for the RTT, that is, the time it takes to reach a certain host and receive a response from it. ntopng shows all the RTT-monitored hosts under the System Interface

For each monitored host ntopng offers the following information

• The full URL of the monitored host. The URL is the concatenation of the host name or IP address with a prefix which indicates the type of probe, namely one icmp, icmpv6, http or https
• A link to a RTT chart with all the minute-by-minute historical measures
• A threshold, expressed in milliseconds, used to trigger alerts
• The time of the last measurement performed
• The latest IP address resolved for the host name indicated in the URL
• The value, in milliseconds, of the latest measured RTT
• Buttons to delete or edit the configured host

Hosts can be added to the RTT monitor using the plus sign in the top-right corner of the table

Hosts can be added either by IP address or with their symbolic host name. In case a symbolic name is specified, ntopng will resolve it before performing the ping. A measurment needs to be specified as well in order to indicate the type of ping desired. Finally, an RTT threshold is requested for the alerts generation.

RTT Alerts

Alerts are triggered when the measured RTT is above the configured threshold, or when the host is unreachable. Configuring a threshold is contextual to the addition of an host among those monitored. Alerts are shown under the triangle entry of the RTT menu bar.

Alerts can either be engaged, for currently ongoing issues, or past, for issues occurred in the past.

RTT Timeseries

RTT timeseries are written automatically by default for any of the RTT-monitored hosts.

Conclusions

ntopng 4.0 is becoming an increasingly active monitoring tool. It adds to the active network discovery and SNMP polling of its predecessor ntopng 3.8 also ICMP, ICMPv6, HTTP and HTTPS ping capabilities. Keeping an eye on hosts on a minute-by-minute basis is now a straightforward thing to do using the RTT Monitor.

Soon we will make this monitoring mechanism more extensible so that people can add their own monitoring tool to make it more flexible than just latency monitoring. Enjoy!

The latest ntopng 4.0 has a renewed look. The main changes we have introduced are:

• An always-on-top status bar. Key information on the health and status of the network is essential for the analyst and it must be always visible and easily accessible. This is why we have introduced an always-on-top fixed status bar with key information such as network throughput, active hosts, flows, and ongoing alerts. This information was previously placed at the bottom of every page so it was difficult to access it and a lot of scrolling was necessary.
• An optimized layout for wide screens. 16:9 wide screens are common so it is important to optimize the layout of the pages for this kind of screens. Hence, we have introduced a vertical main menu bar on the left. The main ntopng menu bar was formerly horizontal and this was taking precious space that can now be used for actual page data.
• Dark and light themes. Being able to choose the colors of the graphical user interface it is not only a matter of personal preferences, it is also very helpful for those who spend a lot of time in front of a screen. Having darker colors helps relieving eyestrain of people who look at screens for hours and hours every day. Therefore, to help those people, we have added a dark theme. Another light theme is also available for those who prefer a bright interface. Those two new themes come in addition to the default one.

Want to see the results? Here are some screenshots

Well, now its time for you to try ntopng 4.0 out! Check this page out to find a package for your distro: https://packages.ntop.org/.

Ongoing health emergency demands business to enable employee work from home: call it smart working or (better) remote working. This process puts pressure on the company Internet connection as many (if not all) the activities need to be done remotely. Working from home is a good practice in order to avoid extra personal stress and reduce virus spread, this only if company’s Internet lines have enough capacity to handle all the remote workers.

How can you measure this extra traffic (with respect to days before the emergency) and optimise your infrastructure? Now this is possible deploying ntopng on your company network. Simply enabling the syslog export from your VPN Concentrator and having the right ntopng version (this feature is implemented in the current development version of ntopng) you can do that: you could be enabled to verify in real time what’s your remote workers are doing, in a ntopng fashion.

For example you can do assurance of remote workers traffic, performance, behaviours and all the other things ntopng allow for traffic analysis. Starting from version 4.0 you can not only visualize the actual traffic ntopng is analyzing, but you can correlate also the trusting level of the traffic and measuring it with extra level of confidence: black listing, encryption level, behavior score, etc etc. As described in the users guide, you can define a syslog interface and specify the log producer configuration.

For instance in the above example the system collects flows from three different devices each with a different vendor configuration. Thanks to this ntopng is the main place to control and visualize your network health status, not only with mirrored interfaced, but with help of external feeds such as nprobe and syslog:

• nProbe can collect local or remote device traffic in order to be visualized and tracked via ntopng panel.
• syslog enables ntopng to correlate remote VPN username with a network address.

Let’s now consider a simple scenario: the firewall that segment the network send flows to ntopng and nprobe that accept syslog from VPN Concentrator.

Components

• Firewall. It export sFlow/nFlow/IPFIX to nProbe that send them in ZMQ format to ntopng.
• VPN Concentrator. It send syslog to ntopng.
• ntopng aggregate flows and correlate VPN IP Addresses with remote Username
• nprobe collect firewall flows and send them to ntopng via ZMQ.

Our Monitoring Goal

• Do assurance of firewalled traffic, this starting from remote users.
• Have historical and statistical data about VPN Users and firewall flows.
• Optimises the bandwidth based on monitored metrics.

Don’t forget that if you have multiple branches or location you could reach the same goals deploying remote nProbe instances to collect local traffic, and send it to central ntopng in encrypted way without NAT boundaries.

In other words, there are no technical limitations to the solution but only limits that you don’t want to overcome. Do you need more info? Feel free to contact us.

In v4 we have introduced active monitoring in ntopng and since then we are improving in the 4.1.x development versions. In order to enable it you have to selectd the “System” interface from the top menubar and select “Active Monitoring” from the left “System” menu.

On that page you will see a table containing all your measurements. The code has been written in a modular way, so that available measurements can be extended as necessary simply adding a new lua script. As you can see from the measurements source, you simply have to create a new lua file for performing the measurement and throw it in the measurement folder for being executed.

Currently the following measurement families are supported:

• ICMP ping: send a scheduled ping (e.g. every minute) and measure the RTT (Round Trip Time).
• Continuous ICMP ping: same as above ping but continuous (send a ping every 3 seconds) to better evaluate continuous network reachability and thus service availability.
• Speedtest: evaluates the available bandwidth (up and down) as well the latency using the speedtest.net service.
• HTTP/HTTPS: periodic test HTTP(S) availability by contacting a remote web server.

The goal of the above measurements is to perform periodic activities and trigger alerts when something goes wrong.

For every measurement there is a threshold that is checked and in case the measurement does not satisfy the threshold, an alarm is triggered.

Such alarm stays open until the problem is solved that happens when the measurement is under the threshold. Note that for certain measurements such as continuous ping the threshold is an upper threshold: as the goal of this test is to continuously monitor service availability (measured as the number of successful ping replies with respect to issued ping requests), in this case this is an upper threshold (i.e. trigger an alert when availability goes below a certain threshold).

This is the current state of the art. What we’re trying to do is implement flexible alert generation. This means that we would like to alert selected recipients for a given alarm (today we alert all recipients the same way, instead of alerting recipient A for measurement X, and recipients B,C for measurement Y), and execute for instance a recovery action when something goes wrong.

Furthermore we’re working at additional measurements and we would like our community to contribute: please raise your hand.

Enjoy!

Trickbot is a malware distributed via malspam, spam emails containing links for downloading malicious files that infect computers.  A pcap file of a trickbot infection named 2019-09-25-Trickbot-gtag-ono19-infection-traffic.pcap can be downloaded at this URL. You can analyse the file using nDPI as follows

ndpiReader -i 2019-09-25-Trickbot-gtag-ono19-infection-traffic.pcap -v 2 -J > /tmp/trickbot.txt

Let’s now open the trickbot.txt file and search for “Risk”. This is a tag added to nDPI analysed flows when there is a non-zero security risk associated with the analysed traffic. Most problems include:

• Obsolete TLS protocol with self-signed certificate

  TCP 10.9.25.101:49457 <-> 187.58.56.26:449 [byte_dist_mean: 119.838346][byte_dist_std: 53.974475][entropy: 4.786248][total_entropy: 2546.284046][score: 0.9964][proto: 91/TLS][cat: Web/5][10 pkts/1532 bytes <-> 11 pkts/2197 bytes][Goodput ratio: 64/73][26.63 sec][bytes ratio: -0.178 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 0/0 3328/819 20084/4504 6493/1528][Pkt Len c2s/s2c min/avg/max/stddev: 54/54 153/200 459/1376 133/376][Risk: ** Self-signed Certificate **** Obsolete TLS version (< 1.1) **][TLSv1][JA3S: 623de93db17d313345d7ea481e7443cf][Issuer: C=AU, ST=Some-State, O=Internet Widgits Pty Ltd][Subject: C=AU, ST=Some-State, O=Internet Widgits Pty Ltd][Certificate SHA-1: DD:EB:4A:36:6A:2B:50:DA:5F:B5:DB:07:55:9A:92:B0:A3:52:5C:AD][Validity: 2019-07-23 10:32:39 - 2020-07-22 10:32:39][Cipher: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]

• Binary executable download (this is how trickbot infects the PCs when people click on malspam links)

  TCP 10.9.25.101:49197 <-> 185.98.87.185:80 [byte_dist_mean: 144.678397][byte_dist_std: 72.649105][entropy: 0.000000][total_entropy: 0.000000][score: 0.9990][proto: 7/HTTP][cat: Web/5][163 pkts/9113 bytes <-> 371 pkts/520336 bytes][Goodput ratio: 3/96][69.52 sec][Host: 185.98.87.185][bytes ratio: -0.966 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 623/25 60010/4824 5733/276][Pkt Len c2s/s2c min/avg/max/stddev: 54/54 56/1403 204/1514 16/164][URL: 185.98.87.185/tablone.png[StatusCode: 200][ContentType: image/png][UserAgent: WinHTTP loader/1.0][Risk: ** Binary application transfer **][PLAIN TEXT (GET /tablone.png HTTP/1.1)]

• Known protocols (TLS) on non-standard ports

  TCP 10.9.25.101:49464 <-> 170.238.117.187:8082 [byte_dist_mean: 71.693632][byte_dist_std: 36.108533][entropy: 4.780162][total_entropy: 20344.368866][score: 0.0085][proto: 7/HTTP][cat: Web/5][9 pkts/4738 bytes <-> 8 pkts/556 bytes][Goodput ratio: 89/22][59.19 sec][Host: 170.238.117.187][bytes ratio: 0.790 (Upload)][IAT c2s/s2c min/avg/max/stddev: 0/842 8335/9414 46230/46230 15705/18411][Pkt Len c2s/s2c min/avg/max/stddev: 54/54 526/70 1514/174 623/40][URL: 170.238.117.187:8082/ono19/BACHMANN-BTO-PC_W617601.AC3B679F4A22738281E6D7B0C5946E42/90[StatusCode: 0][ContentType: multipart/form-data][UserAgent: test][Risk: ** Known protocol on non standard port **][PLAIN TEXT (POST /ono)]

The above evidences are already enough to say that host 10.9.25.101 is doing something malicious that is worth to be analysed. However ntopng can help you do it more easily by means of the graphical interface and also because it enriches nDPI analysis with:

• Use of IP blacklists to identify compromised hosts.
• Malware anomaly score used to easily spot sources of trouble.
• Alert analysis accumulated per host, AS, network…

The same pcap can be ntopng analysed as follows

ntopng -i Downloads/Trickbot/2019-09-25-Trickbot-gtag-ono19-infection-traffic.pcap

Going in the Hosts => Host Explorer page it is possible to select “Alerted Flows” to analyse hosts according to the number of alerted flows they have active. As you can see you can immediately realise that two hosts, BACHMAN-BTO-PC and 187.58.56.26 and the two outliers, and two suspicious hosts.

Clicking on BACHMAN-BTO-PC you can see that this host as a score of over 12’000 that is clearly an indication of security issues.

A very high score of over 2500 is also reported for the other host 187.58.56.26.

Exploring the host flows we see that out local infected host BACHMAN-BTO-PC is uploading data via HTTP to 170.238.117.187

This looks even more suspicious. Let’s analyse this host that is very bad as ntopng marks it as blacklisted, being his IP listed on a blacklist ntopng has accessed (see this page for learning more about ntopng blacklists).

In order to complete our investigations, let’s check the reputation of this host 170.238.117.187 by clicking on the VirusTotal icon that bring us to this page

Our analysis is over. We have identified the local infected host, and where it is uploading our files.

Are you interested in knowing more about nDPI and traffic analysis? You can join us this week, May 14th 14:30 CET at RIPE 80. The presentation will be delivered from remote: just register for the RIPE meeting and enjoy it from wherever you are.

Enjoy analysing malware network traffic with nDPI and ntopng.