[Bro] Bro Digest, Vol 121, Issue 10
fatema bannatwala
fatema.bannatwala at gmail.com
Thu May 12 07:23:02 PDT 2016
Hi Li,
I don't know how much relevant this solution is to the case presented, but
might worth a try.
We run a bro cluster of 4 worker nodes and a manager. We recently started
seeing a lot of capture loss (>60%)
and tried doing some tuning of the interfaces:
Turned off Tx and Rx check-summing on the NIC, hence reduced the lag
between packets captured by interface and packets processed by BRO.
Also, check-sum calculation is default in BRO, hence turning it off on
interface won't create any security issues.
$ sudo ethtool -K em1 rx off
Actual changes:
rx-checksumming: off
large-receive-offload: off [requested on]
$ sudo ethtool -K em1 tx off
Actual changes:
tx-checksumming: off
tx-checksum-ipv4: off
tx-checksum-ipv6: off
tx-checksum-sctp: off
tcp-segmentation-offload: off
tx-tcp-segmentation: off [requested on]
tx-tcp6-segmentation: off [requested on]
$ sudo ethtool -K em1 sg off
Actual changes:
scatter-gather: off
tx-scatter-gather: off
generic-segmentation-offload: off [requested on]
$ sudo ethtool -K em1 tso off
$ sudo ethtool -K em1 gso off
$ sudo ethtool -K em1 gro off
This reduced the capture loss % to below 1%, and the cluster is not seeing
any capture loss till date.
Thanks,
Fatema.
On Tue, May 10, 2016 at 3:00 PM, <bro-request at bro.org> wrote:
> Send Bro mailing list submissions to
> bro at bro.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> or, via email, send a message with subject or body 'help' to
> bro-request at bro.org
>
> You can reach the person managing the list at
> bro-owner at bro.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bro digest..."
>
>
> Today's Topics:
>
> 1. bro cluster packet loss with pf_ring_zc (Bowen Li)
> 2. Re: bro cluster packet loss with pf_ring_zc (Gary Faulkner)
> 3. Re: bro ids icmp and attack signatures (Seth Hall)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 10 May 2016 11:50:56 +0800
> From: Bowen Li <newfire.bw at gmail.com>
> Subject: [Bro] bro cluster packet loss with pf_ring_zc
> To: bro at bro.org
> Message-ID:
> <
> CAJ17UJdzkj8N0uD2Sn-vrAmtcw8OQATKj2Wm5HMqWDmdxf1H1w at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Dear list,
>
> I?m using Bro 2.4.1 stable and PF_RING_ZC to analysis network traffic. The
> peak flow of the traffic almost close to 1G/bps(the full load of NIC), and
> the number of data packet in traffic may reach 200,000 pps. PF_RING_ZC
> zbalance_ipc shows pf_ring has no packet loss and broctl netstats
> shows that bro cluster have lost most of the packets, but the link number
> is equal to the receive packet number.
>
> When handle the packets, almost all of the cpus are in full load status, so
> I suspect that the processor of the server limits the packet processing
> speed, so the bro-cluster have to drop packets.
>
> So my question is now the performance of the server under 200,000 pps cases
> a packet loss in bro is normal or not.
>
> Here is the CPU info:
> Architecture: x86_64
> CPU op-mode(s): 32-bit, 64-bit
> Byte Order: Little Endian
> CPU(s): 32
> On-line CPU(s) list: 0-31
> Thread(s) per core: 2
> Core(s) per socket: 8
> Socket(s): 2
> NUMA node(s): 2
> Vendor ID: GenuineIntel
> CPU family: 6
> Model: 45
> Model name: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
> Stepping: 6
> CPU MHz: 1317.937
> BogoMIPS: 4419.58
> Virtualization: VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache: 256K
> L3 cache: 20480K
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30
> NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
>
> Here is the memory info:
> total used free shared buff/cache
> available
> Mem: 65759080 13018468 31412324 132364 21328288
> 52079776
> Swap: 29241340 0 29241340
>
> Is anyone able to help me?
>
>
>
> Thanks in advance,
>
> Bowen Li
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160510/5b5ed9b7/attachment-0001.html
>
> ------------------------------
>
> Message: 2
> Date: Tue, 10 May 2016 10:27:13 -0500
> From: Gary Faulkner <gfaulkner.nsm at gmail.com>
> Subject: Re: [Bro] bro cluster packet loss with pf_ring_zc
> To: Bowen Li <newfire.bw at gmail.com>, bro at bro.org
> Message-ID: <d4f0a530-29c8-b24d-f1b0-27e04af211d4 at gmail.com>
> Content-Type: text/plain; charset="windows-1252"
>
> I guess the first thing is to ask how much packet loss? You'll never
> fully eliminate it, but a good cluster set up can keep your average loss
> under 1%. I could ask a whole lot of questions about your cluster set
> up, but it is probably easier if you can share a redacted version of
> your node.conf (redact public IPs, DNS names, any sensitive info etc).
> You could also try running the capture loss script and doing some
> analysis on which and how many workers are dropping packets over time.
> Keep in mind if you are doing any kind of partial flow shunting this
> could skew the results. You could also look at stats.log if you have it
> enabled. If one or two workers are really dropping packets during an
> interval of time, but the rest look OK this could be traffic related
> (some large flows). If it is across the board you may need to look more
> closely at your cluster set-up for sub-optimal configuration or
> over-subscription.
>
> ~Gary
>
>
> On 5/9/16 10:50 PM, Bowen Li wrote:
> > Dear list,
> >
> > I?m using Bro 2.4.1 stable and PF_RING_ZC to analysis network traffic.
> The
> > peak flow of the traffic almost close to 1G/bps(the full load of NIC),
> and
> > the number of data packet in traffic may reach 200,000 pps. PF_RING_ZC
> > zbalance_ipc shows pf_ring has no packet loss and broctl netstats
> > shows that bro cluster have lost most of the packets, but the link number
> > is equal to the receive packet number.
> >
> > When handle the packets, almost all of the cpus are in full load status,
> so
> > I suspect that the processor of the server limits the packet processing
> > speed, so the bro-cluster have to drop packets.
> >
> > So my question is now the performance of the server under 200,000 pps
> cases
> > a packet loss in bro is normal or not.
> >
> > Here is the CPU info:
> > Architecture: x86_64
> > CPU op-mode(s): 32-bit, 64-bit
> > Byte Order: Little Endian
> > CPU(s): 32
> > On-line CPU(s) list: 0-31
> > Thread(s) per core: 2
> > Core(s) per socket: 8
> > Socket(s): 2
> > NUMA node(s): 2
> > Vendor ID: GenuineIntel
> > CPU family: 6
> > Model: 45
> > Model name: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
> > Stepping: 6
> > CPU MHz: 1317.937
> > BogoMIPS: 4419.58
> > Virtualization: VT-x
> > L1d cache: 32K
> > L1i cache: 32K
> > L2 cache: 256K
> > L3 cache: 20480K
> > NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30
> > NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
> >
> > Here is the memory info:
> > total used free shared buff/cache
> > available
> > Mem: 65759080 13018468 31412324 132364 21328288
> > 52079776
> > Swap: 29241340 0 29241340
> >
> > Is anyone able to help me?
> >
> >
> >
> > Thanks in advance,
> >
> > Bowen Li
> >
> >
> >
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160510/bc98bba6/attachment-0001.html
>
> ------------------------------
>
> Message: 3
> Date: Tue, 10 May 2016 12:09:04 -0400
> From: Seth Hall <seth at icir.org>
> Subject: Re: [Bro] bro ids icmp and attack signatures
> To: "Mostafa Abdallah. Ammar" <mostafaammar at aast.edu>
> Cc: "bro at bro.org" <bro at bro.org>
> Message-ID: <839261C2-AE66-47FF-9AD7-D7FD0CB3E2E3 at icir.org>
> Content-Type: text/plain; charset=iso-8859-1
>
> I would look into what icmp messages you are seeing over ICMP that is
> causing this. This is probably just due to some aspect of how router
> solicitation or neighbor solicitation happens. I would also create a pcap
> containing a test case where you know this to trigger correctly so that you
> can have a repeatable test.
>
> .Seth
>
>
> > On May 9, 2016, at 9:20 AM, Mostafa Abdallah. Ammar <
> mostafaammar at aast.edu> wrote:
> >
> > Dear All,
> >
> > I tried the following script icmptest.bro (attached) while running
> remote syslog, all the messages on syslog are regarding ipv6 and not ipv4
> is there an explanation for that .
> >
> > 05-09-2016 14:56:23 Local7.Info 10.0.1.153 May 9 14:55:45
> ubuntu-HVM-domU bro_notice: 1462798535.800222 - - - - - - -
> - - DetectICMPSHell:: ICMP connection threshold exceeded :
> fe80::1d26:ba55:fc1c:4a8 - - - - - bro Notice::ACTION_LOG
> 3600.000000 F - - - - -
> > Best Regards,
> >
> > Eng. Mostafa Abdallah Ammar,Msc.
> > Information Security and Auditing Supervisor
> > CCIE security #23971
> > Arab Academy For Science And Technology & maritime Transport
> > Computer Networks & Data Center (CNDC)
> > Mobile: 002 01001983674
> > From: Mostafa Abdallah. Ammar
> > Sent: Thursday, May 05, 2016 4:42 PM
> > To: bro at bro.org
> > Subject: bro ids icmp and attack signatures
> >
> > Dear All,
> >
> > I am new to bro ids , I installed successfully bro ids , and added a tap
> to network to it , and for example if I accessed a website on a machine I
> can see in http.log the website I accessed and if the wqebsite is ssl i can
> see in ssl.lot and x509.log the certificate info
> >
> > my question is :
> >
> > I want when I ping i see a notification for this ping (I tried and could
> not find)
> >
> > can I use signatures like snort with bro that generate logs when
> receiving an attack and generate log with signature ID
> >
> > Please provide reply with some details as I am new to bro.
> >
> >
> > Best Regards,
> >
> > Eng. Mostafa Abdallah Ammar,Msc.
> > Information Security and Auditing Supervisor
> > CCIE security #23971
> > Arab Academy For Science And Technology & maritime Transport
> > Computer Networks & Data Center (CNDC)
> > Mobile: 002 01001983674
> > <icmptest.bro>_______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> --
> Seth Hall
> International Computer Science Institute
> (Bro) because everyone has a network
> http://www.bro.org/
>
>
>
>
> ------------------------------
>
> _______________________________________________
> Bro mailing list
> Bro at bro.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
>
> End of Bro Digest, Vol 121, Issue 10
> ************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20160512/f6f18f68/attachment-0001.html
More information about the Bro
mailing list