[Bro] To proxy or not to proxy...

Harry Hoffman hhoffman at ip-solutions.net
Thu Apr 2 14:42:27 PDT 2015


So, I disable hyperthreading by default and yes, I leave a the CPUs for workers and CPU for proxy.

It's a fairly remarkable difference.

Cheers,
HarryOn Apr 2, 2015 5:22 PM, Adam Pumphrey <apumphrey at ivsec.com> wrote:
>
> Great discussion and pointers.  I’m working on a similar performance tuning and stabilization effort. 
>
> I took a closer look to verify and I can confirm Aashish’s statement about the numbering of cores on Linux.  We’re running CENTOS 6.2.  This box has 2 hyperthreaded hex-core procs.   All physical cores are assigned sequential ID’s in socket/core order, then hyperthreaded cores are assigned sequential ID’s in socket/core order.  
>
> Here’s what we end up with:
>
> __socket0 (P/H)__
> 0/12
> 1/13
> 2/14
> 3/15
> 4/16
> 5/17
>
> __socket1 (P/H)__
> 6/18
> 7/19
> 8/20
> 9/21
> 10/22
> 11/23
>
> Adam
>
> > On Apr 2, 2015, at 1:17 PM, Aashish Sharma <asharma at lbl.gov> wrote:
> > 
> > Same here: I have a proxy for every 10 workers on each of the physical box (which runs workers) in the cluster.
> > 
> > Ah! regarding CPU pinning: 
> > 
> >> fairly well. You may want to reduce your worker count a bit to leave 
> >> enough CPUs for the proxies. Out of curiosity are you pinning your 
> >> workers to dedicated CPU cores? If you are not it could be that your 
> >> workers are bouncing between cores due to hyper-threading which can 
> >> cause them to stomp all over each other. I found pinning workers to 
> >> cores helped tremendously when it came to worker health.
> > 
> > I agree completely! 
> > 
> > Also, Make sure that you have enough cores to run workers on. 
> > 
> > With respect to CPU pinning, on *FreeBSD*, CPUs are numbered as :
> > P = physical core 
> > H = Hyperthread core
> > 
> > 0/1 = P/H 
> > 2/3 = P/H 
> > 4/5 = P/H 
> > ...
> > ...
> > 11/12=P/H
> > 
> > You certainly don't want to pin_cpu on FreeBSD as 0,1,2,3 but instead pin_cpu=0,2,4,6,8.... (or 1,3,5,7...)
> > 
> > However, I beleive Linux does it different. While I have not yet looked at a Linux's box, I believe its scheme for hex-core processor is 
> > 
> > 0/6=P/H
> > 1/7=P/H
> > 2/8=P/H
> > ..
> > ..
> > 5/12=P/H
> > 
> > so you might want to pin_cpu on linux as: pin_cpu=01,2,3,4,5 or (6,7,8,9,10,11,12)
> > 
> > Make sure you leave a few cores alone for proxy and other tasks when pinning. 
> > 
> > Oh, btw, we have found no noticible difference in performance at all, when you pin a bro process on only physical core vs only hyperthreded cores. But make sure you don't pin bro processes on both P/H at the same time.
> > 
> > Now, it would be great if someone can confirm the linux side of the story. or shed more light on cpu_pinning. 
> > 
> > 
> > 
> > 
> > Aashish 
> > 
> > 
> > On Thu, Apr 02, 2015 at 11:24:31AM -0500, Gary Faulkner wrote:
> >> I'm currently running a separate box that has the manager and proxies on 
> >> it, but I did just as you describe at one point and it seemed to work 
> >> fairly well. You may want to reduce your worker count a bit to leave 
> >> enough CPUs for the proxies. Out of curiosity are you pinning your 
> >> workers to dedicated CPU cores? If you are not it could be that your 
> >> workers are bouncing between cores due to hyper-threading which can 
> >> cause them to stomp all over each other. I found pinning workers to 
> >> cores helped tremendously when it came to worker health.
> >> 
> >> ~Gary
> >> 
> >> On 4/1/2015 7:52 PM, Harry Hoffman wrote:
> >>> Hi folks,
> >>> 
> >>> So in my continuing pursuit of perfecting my Bro setup I found that adding a proxy on every box that also runs workers keeps bro much happier then a single manager/proxy box with one or more worker(s) boxes.
> >>> 
> >>> Prior to adding the additional proxies bro workers would die due to resource constraints.
> >>> 
> >>> Are other folks doing this?
> >>> 
> >>> Cheers,
> >>> Harry
> >>> 
> >>> 
> >>> 
> >>> _______________________________________________
> >>> Bro mailing list
> >>> bro at bro-ids.org
> >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >> 
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> > 
> > -- 
> > Aashish Sharma (asharma at lbl.gov) 
> > Cyber Security, 
> > Lawrence Berkeley National Laboratory  
> > http://go.lbl.gov/pgp-aashish 
> > Office: (510)-495-2680  Cell: (510)-612-7971
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro



More information about the Bro mailing list