[Bro] af_packet comparison to PF_RING ZC/DNA for Bro (in light of recent Suricata tuning paper)

[Jan Grashöfer]

Hi Gary,

> After reading over the paper Michal and others worked on concerning 
> tuning Suricata for best performance with AF_Packet I'm wondering how 
> af_packet performance compares to pf_ring DNA/ZC (with the commercially 
> licensed drivers, not just vanilla) especially when it comes to Bro.

unfortunately I cannot provide any numbers. My main motivation for using
AF_Packet with Bro was the ease of use. Especially the PF_RING ZC
drivers caused issues in my environment, which I struggled to debug.
Given the extra cost of building this and that myself I chose AF_Packet.

> Is af_packet generally sufficient for Bro when it comes to monitoring 
> 100G+ networks using a cluster of commodity servers with Intel X520 NICs?

Good question. Someone should test this :)

> Is the distro shipped driver for something like an up-to-date Ubuntu 
> 16.04 (4.4 kernel) server sufficient or do you really need to compile 
> the driver from source to enable some extended features, or to get a 
> properly patched driver etc? I could see some benefits to just using the 
> distro packaged driver and not having to compile the driver from scratch 
> or rely on dkms when patching sensors. I've had this go very wrong a few 
> times.

For me (CentOS 7) the packaged driver worked well with AF_Packet. But if
you want to tune things for maximal performance, I would recommend using
the latest drivers. E.g., from time to time looking at the code might
help in this case to understand what's going on.

> Are there any gotchas where running one or the other might be the better 
> way to go? Examples (want to use some bro feature such as capstats, or 
> want to see VLAN tags in Bro logs, something else is broken or not 
> performing as expected)

I haven't used capstats but if I remember correctly, it is kind of
deprecated as it relies on libpcap. One should be able to obtain the
same information from other sources.

VLAN tags are indeed an issue using AF_Packet. For consistency reasons,
the kernel extracts VLAN tags even if there is no hardware VLAN
offloading (in contrast to Bro, Suricata can handle this due to its
monolithic structure). Actually that's something on my list.

Finally, one has to be careful regarding the kernel used. There is a bug
concerning AF-Packet's symmetric hashing that has been fixed in recent
kernels
(https://bro-tracker.atlassian.net/browse/BIT-1575?focusedCommentId=29627#comment-29627).

> Does af_packet or the Bro plugin for it have a way to deal with multiple 
> NICS (one per numa node), sort of like how pf_ring has dnacluster and 
> zbalance_ipc?

In theory configuring a set of workers per NUMA node using separate NICs
shouldn't be an issue. The only thing is that you won't get load
balancing across the NICs. I am not sure how well this works with
PF_RING, though.

> Feel free to share any other relevant considerations.

In addition to the VLAN stuff I have a couple of other things on my
list, which might allow some tuning. Unfortunately this list hasn't seen
much progress lately as I don't have access to a test setup. So there
might be room for improvement.

Jan