[Bro] AF_PACKET on Ubuntu 16.04

[Gary Faulkner]

I tried to follow the guide fairly closely and adapt for Bro with the
exception of BIOS level tuning (which I plan to investigate later).
rxhash is set to off. I was cpu pinning bro before, so I am continuing
to do so. Settings are below as well as a rough script I am tweaking to
load them.

For troubleshooting purposes I decided not to simplify the script with a
loop as I was running into some issues with command order (especially
with set_irq_affinity placement) as well as a couple unsupported options:

Features for eth4:

rx-checksumming: off
tx-checksumming: off
        tx-checksum-ipv4: off
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: on [fixed]
hw-tc-offload: off [fixed]


#!/bin/bash

#Unload any existing module and load with new parameters
rmmod ixgbe
modprobe ixgbe MQ=0,0,0,0 RSS=1,1,1,1 VMDQ=0,0,0,0
InterruptThrottleRate=12500,12500,12500,12500 FCoE=0,0,0,0 LRO=0,0,0,0
vxlan_rx=0,0,0,0
sleep 1

#Disable irqbalance to stop bouncing interrupts between cores
killall irqbalance
sleep 1

#Enable interfaces in promisc mode
ip link set eth4 promisc on arp off up
ip link set eth6 promisc on arp off up
sleep 1

#Disable IPv6 on interfaces
echo 1 > /proc/sys/net/ipv6/conf/eth4/disable_ipv6
echo 1 > /proc/sys/net/ipv6/conf/eth6/disable_ipv6

#Enable Jumbo Frames (MTU of 9216 used on routers)
ip link set dev eth4 mtu 9216
ip link set dev eth6 mtu 9216

#Enforce a single RX queue
ethtool -L eth4 combined 1
ethtool -L eth6 combined 1

#Manage interrupts
ethtool -C eth4 adaptive-rx on rx-usecs 100
ethtool -C eth6 adaptive-rx on rx-usecs 100

#Lower the NIC ring descriptor size
ethtool -G eth4 rx 512
ethtool -G eth6 rx 512

#Disable pause frames
#ethtool -A eth4 autoneg off
#ethtool -A eth6 autoneg off
ethtool -A eth4 rx off tx off
ethtool -A eth6 rx off tx off

#Disable offloading features
ethtool -K eth4 rx off
ethtool -K eth4 tx off
ethtool -K eth4 tso off
ethtool -K eth4 ufo off
ethtool -K eth4 gso off
ethtool -K eth4 gro off
ethtool -K eth4 lro off
ethtool -K eth4 tx-nocache-copy off
ethtool -K eth4 rxhash off
ethtool -K eth4 ntuple off
ethtool -K eth4 sg off
ethtool -K eth4 txvlan off
ethtool -K eth4 rxvlan off
ethtool -K eth6 rx off
ethtool -K eth6 tx off
ethtool -K eth6 tso off
ethtool -K eth6 ufo off
ethtool -K eth6 gso off
ethtool -K eth6 gro off
ethtool -K eth6 lro off
ethtool -K eth6 tx-nocache-copy off
ethtool -K eth6 rxhash off
ethtool -K eth6 ntuple off
ethtool -K eth6 sg off
ethtool -K eth6 txvlan off
ethtool -K eth6 rxvlan off

#Set irq affinity
/bin/bash ./set_irq_affinity 2 eth4
/bin/bash ./set_irq_affinity 3 eth6

On 2/22/2017 5:51 PM, Michał Purzyński wrote:
> Have you disabled hardware hash with ethtool? By default kernel will use the card hash which is asymmetric.
>
> You can verify it with ethtool -k <card>
>
> Look for rxhash - should be disabled.
>
> https://github.com/pevma/SEPTun
>
> Should show you how to prepare your system, you can ignore the core isolation and affinity for bro.
>
>> On Feb 22, 2017, at 3:29 PM, Gary Faulkner <gfaulkner.nsm at gmail.com> wrote:
>>
>> I'm trying AF_PACKET with Bro, but seem to be running a kernel and 
>> driver combo that doesn't appear to properly support symmetric hashing. 
>> I'm on Ubuntu 16.04 with kernel 4.4.0-59-generic. From what I can tell 
>> the patches should have been added around kernel 4.4.0-39 or so, but 
>> Justin's verification tool and Bro both seem to agree that it is broken 
>> on my system. I've tried with the OS supplied IXGBE driver (4.2.1-k) as 
>> well as compiling from scratch using a recent IXGBE directly from Intel 
>> (5.0.4). Is there a known working kernel and driver combo for Ubuntu 
>> 16.04, or are the necessary patches still not pushed into 16.04?
>>
>> Thanks,
>>
>> Gary
>>
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro