[Bro] Tap configuration

Fri Jan 20 07:29:55 PST 2017

Sorry for the delayed response Jon,

We’ve been tracking the "Suricata Extreme Performance Tuning" efforts where they’re hitting 20Gbps on a single box. They have a pretty good write-up of their research into the flow of a packet through the hardware/software layers: https://github.com/pevma/SEPTun <https://github.com/pevma/SEPTun>

The RSS=1 setting is to avoid packet re-ording impacts. Are you using PF_RING DNA with symmetric RSS?

One new issue we’re working on since switching from pf_ring to af_packet is that the CPU that worker-1 is pinned to runs close to 100% while the 9 other workers are all much lower than that. We’re been able to determine that its due to the Intel ‘set_irq_affinity’ script and its pinning every IRQ call for the NIC (and single RSS to the same CPU, which is the first core in that NUMA node.

We’re always looking for tuning feedback/best practices so I appreciate your questions.

-Dave

> On Jan 15, 2017, at 11:38 PM, Zeolla at GMail.com <zeolla at gmail.com> wrote:
> 
> So I'm not sure I follow exactly why you'd want to specifically emphasize keeping packets in the L3 cache.  Is there a specific hardware configuration where this makes more sense?
> 
> As of right now, I do pretty much the same thing you posted earlier except I map the # of RX queues to the # of physical CPU cores and maximize the NIC ring descriptor size.
> 
> Jon
> 
> On Sat, Jan 14, 2017 at 2:35 PM Dave Crawford <bro at pingtrip.com <mailto:bro at pingtrip.com>> wrote:
> This is what I use in my sensor /etc/network/interfaces config along with a custom “post-up” script. I use Debian for my Bro clusters, so your application will differ. I’m also using af_packet (v4.8.0 kernel) so some of the performance settings may need to be adjusted for your setup. My tuning is aimed at keeping the packets in L3 cache on the CPU vid the NIC hardware, hence the reduced rings. 
> 
> auto eth6
> iface eth6 inet manual
>   up ip link set $IFACE promisc on arp off mtu 1500 up
>   down ip link set $IFACE promisc off down
>   post-up /opt/tools/post-up_settings.sh $IFACE
> 
> 
> And the /opt/tools/post-up_settings.sh script:
> 
> #!/bin/bash
> 
> IFACE=$1
> 
> if [[ -n "$IFACE" ]]; then
>     
>     # Lower the NIC ring descriptor size
>     /sbin/ethtool -G $IFACE rx 512
> 
>     # Disable offloading functions
>     for i in rx tx sg tso ufo gso gro lro rxhash ntuple txvlan rxvlan; do ethtool -K $IFACE $i off; done
>     
>     # Enforce a single RX queue
>     /sbin/ethtool -L $IFACE combined 1
> 
>     # Disable pause frames
>     /sbin/ethtool -A $IFACE rx off tx off
>     
>     # Limit the maximum number of interrupts per second
>     /sbin/ethtool -C $IFACE adaptive-rx on rx-usecs 100
> 
>     # Disable IPv6
>     /bin/echo 1 > /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6
> 
>     # Pin IRQ to local CPU
>     /opt/tools/set_irq_affinity local $IFACE
> fi
> 
> -Dave
> 
>> On Jan 13, 2017, at 3:28 PM, Daniel Manzo <daniel.manzo at bayer.com <mailto:daniel.manzo at bayer.com>> wrote:
>> 
>> Thank you for the help. I tried the settings, but I have noticed any difference in packets. The main test that I am doing is that I would open two putty sessions to the server, and have one running capstats on eth12 while my other session was downloading a 1GB file to /dev/null. Last week, I was able to see the packets increase greatly via capstats, but now they stay steady at 7 or 8 packets per second.
>> 
>> Best regards,
>> Dan Manzo
>> 
>> -----Original Message-----
>> From: Seth Hall [mailto:seth at icir.org <mailto:seth at icir.org>] 
>> Sent: Friday, January 13, 2017 9:29 AM
>> To: Daniel Manzo
>> Cc: Neslog; Hosom, Stephen M; Bro-IDS
>> Subject: Re: [Bro] Tap configuration
>> 
>> I would recommend leaving checksum validation on in Bro, but disable checksum offloading on the NIC.
>> 
>> I typically point people to this blog post by Doug Burks (of the SecurityOnion project)...
>> 	http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html <http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html>
>> 
>> There is one further thing I would recommend though that we discovered well after this blog post was written.  If you are using an Intel NIC with the ixgbe driver, your nic has a feature called "flow director" that you will want to disable because it will negatively impact your analysis by reordering packets.  It can be disabled like this on linux:
>> 	ethtool -L eth12 combined 1
>> 
>> This will cause your NIC to have only a single hardware queue which will disable the flow director feature and prevent your NIC from reordering packets.  Do that along with the suggestions in the blog post above and things should be better.
>> 
>>  .Seth
>> 
>> 
>>> On Jan 13, 2017, at 8:58 AM, Daniel Manzo <daniel.manzo at bayer.com <mailto:daniel.manzo at bayer.com>> wrote:
>>> 
>>> I have tried disabling checksum offloading, but still no luck. Here is the ifcfg file for my eth interface:
>>> 
>>> DEVICE=eth12
>>> ONBOOT=yes
>>> BOOTPROTO=static
>>> PROMISC=yes
>>> USERCTL=no
>>> 
>>> Freundliche Grüße / Best regards,
>>> 
>>> Dan Manzo
>>> Asst Analyst I
>>> ________________________
>>> 
>>> Bayer: Science For A Better Life
>>> 
>>> Bayer U.S. LLC
>>> Country Platform US
>>> Scientific Computing Competence Ctr
>>> Bayer Road
>>> 15205 Pittsburgh (PA), United States
>>> Tel:                        +1 412 7772171 <tel:(412)%20777-2171>
>>> Mobile:                +1 412 5258332 <tel:(412)%20525-8332>
>>> E-mail:  daniel.manzo at bayer.com <mailto:daniel.manzo at bayer.com>
>>> 
>>> From: Neslog [mailto:neslog at gmail.com <mailto:neslog at gmail.com>] 
>>> Sent: Thursday, January 12, 2017 4:59 PM
>>> To: Hosom, Stephen M
>>> Cc: Bro-IDS; Daniel Manzo
>>> Subject: Re: [Bro] Tap configuration
>>> 
>>> I've had success disabling checksum. 
>>> ignore_checksums
>>> 
>>> 
>>> On Jan 12, 2017 2:24 PM, "Hosom, Stephen M" <hosom at battelle.org <mailto:hosom at battelle.org>> wrote:
>>> Have you looked into checksum offloading? If enabled, it can result in Bro not producing many of the logs you would expect.
>>> 
>>> From: bro-bounces at bro.org <mailto:bro-bounces at bro.org> [mailto:bro-bounces at bro.org <mailto:bro-bounces at bro.org>] On Behalf Of Daniel Manzo
>>> Sent: Thursday, January 12, 2017 11:05 AM
>>> To: bro at bro.org <mailto:bro at bro.org>
>>> Subject: [Bro] Tap configuration
>>> 
>>> Hi all,
>>> 
>>> I have Bro 2.4 configured on a RHEL 6.8 server and was wondering how to properly configure the network interfaces so that Bro can see as much of the network traffic as possible. My tap is connected in line with the network, and I believe that I was previously seeing the correct traffic, but now Bro has reporting much less information. I want to make sure that I have the interfaces configured correctly before moving on to troubleshooting other areas. Currently, I have two eth interfaces set up in PROMISC mode. Thank you for the help
>>> 
>>> Best regards,
>>> Dan Manzo
>>> 
>>> 
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
>> 
>> 
>> 
>> --
>> Seth Hall
>> International Computer Science Institute
>> (Bro) because everyone has a network
>> http://www.bro.org/ <http://www.bro.org/>
>> 
>> 
>> _______________________________________________
>> Bro mailing list
>> bro at bro-ids.org <mailto:bro at bro-ids.org>
>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org <mailto:bro at bro-ids.org>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro <http://mailman.icsi.berkeley.edu/mailman/listinfo/bro>
> -- 
> Jon
> 
> Sent from my mobile device
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170120/153c0f26/attachment-0001.html 