[Bro] Myricom and Bro... show of hands for successful deployments on 10G links (with > 5Gpbs)

[Harry Hoffman]

Hi Vlad,

Absolutely. Sorry if that was vague or cryptic.

What I meant was that using the myricom test utilities I can capture everything on the wire. These utilities don’t write to disk so they only show that there’s not a issue with nic to memory transfers.

Once I fire up bro one worker consistently pegs a core at 100% and I drop greater then 1/2 of packets. The rate of drop isn’t as severe with tools like tcpdump but I assume that the difference in processing that bro does with packets.

All of these is running on a Dell R710 with 2 Xeon CPUs at 2.8GHz with 6 cores each (HT disabled)  and 96GB of RAM and two SSD drives for data each 700GB in size. We moved to the Dell specifically to test whether or not using SSD drives gave a performance boost in writing to disk.

We’re using the myricom tools (/opt/snf/bin/myri_counters) to determine dropped packets, via SNF drop ring full,  due to the application (tcpdump, bro, etc) being too slow to grab packets from the ring buffer.

As an initial, memory only test, we’ve run /opt/snf/bin/tests/snf_simple_recv and /opt/snf/bin/tests/snf_multi_recv. Both run without any drops and output shows an avg of 7Gbps on the wire. Running either test for extended periods of time does not cause the values in “SNF drop ring full” to increment.

/usr/local/bro/etc/node.cfg looks like (as you can see we’re attempting to tweak performance via the various SNF env variables. There’s no difference noticed using pin_cpus:

[manager]
type=manager
host=localhost
#
[proxy-1]
type=proxy
host=localhost
#
[worker-1]
type=worker
host=localhost
interface=p1p1
lb_method=myricom
lb_procs=10
#pin_cpus=2,3,4,5,6,7,8,9,10,11
env_vars="SNF_DEBUG_MASK=0x03,SNF_DATARING_SIZE=1024MB,SNF_DESCRING_SIZE=256MB”



We do have a rather large networks.cfg with 1439 entries in it.

Any suggestions would be greatly appreciated!  We had a “beefy-er Cisco 1u UCS box with 2 x 8 cores and the same memory and that didn’t hold up either. There were no free pci slots to test SSD drives which is why we’re testing with the Dell.

Cheers,
Harry


On Aug 22, 2014, at 2:44 PM, Vlad Grigorescu <vlad at grigorescu.org> wrote:

> Hi Harry,
> 
> Can you expand on "allowing both capture and writing to disk?" Carnegie Mellon runs a Bro cluster with Myricom NICS, which works well. However, the manager is on a box that doesn't have any workers on it (and thus doesn't receive any traffic), so I haven't had any I/O contention from network traffic and log writing. Is that what you're referring to?
> 
> We're seeing about 16 Gbps and dropping < 1% (around 0.1% most of the time, I believe). That's split up over 4 rather beefy boxes, though.
> 
>   --Vlad
> 
> 
> On Thu, Aug 21, 2014 at 10:26 AM, Harry Hoffman <hhoffman at ip-solutions.net> wrote:
> Hi All,
> 
> So, I’m writing to hopefully get a show of hands from those of your out there who’ve employed Myricom cards to capture packets on your 10G links.
> 
> I’ll start by saying that while the myricom cards we have in place do a fine job of capturing I’ve been unable to find the secret sauce that allows both capture and writing to disk in a way that doesn’t drop significant amounts of packets using either bro, tcpdump, snort, suricata.
> 
> For those of you out there using myricom cards in conjunction with your favorite tools (bro of course ;-)  ) can you let me know what data rate your Myricom cards are seeing and what (assuming some) percentage of packets you are dropping?
> 
> If you aren’t dropping anything I’d love to know more about your setup! :-)
> 
> Cheers,
> Harry
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20140822/e9ceafb7/attachment.html