[Bro] To proxy or not to proxy...
Harry Hoffman
hhoffman at ip-solutions.net
Thu Apr 2 14:42:27 PDT 2015
So, I disable hyperthreading by default and yes, I leave a the CPUs for workers and CPU for proxy.
It's a fairly remarkable difference.
Cheers,
HarryOn Apr 2, 2015 5:22 PM, Adam Pumphrey <apumphrey at ivsec.com> wrote:
>
> Great discussion and pointers. I’m working on a similar performance tuning and stabilization effort.
>
> I took a closer look to verify and I can confirm Aashish’s statement about the numbering of cores on Linux. We’re running CENTOS 6.2. This box has 2 hyperthreaded hex-core procs. All physical cores are assigned sequential ID’s in socket/core order, then hyperthreaded cores are assigned sequential ID’s in socket/core order.
>
> Here’s what we end up with:
>
> __socket0 (P/H)__
> 0/12
> 1/13
> 2/14
> 3/15
> 4/16
> 5/17
>
> __socket1 (P/H)__
> 6/18
> 7/19
> 8/20
> 9/21
> 10/22
> 11/23
>
> Adam
>
> > On Apr 2, 2015, at 1:17 PM, Aashish Sharma <asharma at lbl.gov> wrote:
> >
> > Same here: I have a proxy for every 10 workers on each of the physical box (which runs workers) in the cluster.
> >
> > Ah! regarding CPU pinning:
> >
> >> fairly well. You may want to reduce your worker count a bit to leave
> >> enough CPUs for the proxies. Out of curiosity are you pinning your
> >> workers to dedicated CPU cores? If you are not it could be that your
> >> workers are bouncing between cores due to hyper-threading which can
> >> cause them to stomp all over each other. I found pinning workers to
> >> cores helped tremendously when it came to worker health.
> >
> > I agree completely!
> >
> > Also, Make sure that you have enough cores to run workers on.
> >
> > With respect to CPU pinning, on *FreeBSD*, CPUs are numbered as :
> > P = physical core
> > H = Hyperthread core
> >
> > 0/1 = P/H
> > 2/3 = P/H
> > 4/5 = P/H
> > ...
> > ...
> > 11/12=P/H
> >
> > You certainly don't want to pin_cpu on FreeBSD as 0,1,2,3 but instead pin_cpu=0,2,4,6,8.... (or 1,3,5,7...)
> >
> > However, I beleive Linux does it different. While I have not yet looked at a Linux's box, I believe its scheme for hex-core processor is
> >
> > 0/6=P/H
> > 1/7=P/H
> > 2/8=P/H
> > ..
> > ..
> > 5/12=P/H
> >
> > so you might want to pin_cpu on linux as: pin_cpu=01,2,3,4,5 or (6,7,8,9,10,11,12)
> >
> > Make sure you leave a few cores alone for proxy and other tasks when pinning.
> >
> > Oh, btw, we have found no noticible difference in performance at all, when you pin a bro process on only physical core vs only hyperthreded cores. But make sure you don't pin bro processes on both P/H at the same time.
> >
> > Now, it would be great if someone can confirm the linux side of the story. or shed more light on cpu_pinning.
> >
> >
> >
> >
> > Aashish
> >
> >
> > On Thu, Apr 02, 2015 at 11:24:31AM -0500, Gary Faulkner wrote:
> >> I'm currently running a separate box that has the manager and proxies on
> >> it, but I did just as you describe at one point and it seemed to work
> >> fairly well. You may want to reduce your worker count a bit to leave
> >> enough CPUs for the proxies. Out of curiosity are you pinning your
> >> workers to dedicated CPU cores? If you are not it could be that your
> >> workers are bouncing between cores due to hyper-threading which can
> >> cause them to stomp all over each other. I found pinning workers to
> >> cores helped tremendously when it came to worker health.
> >>
> >> ~Gary
> >>
> >> On 4/1/2015 7:52 PM, Harry Hoffman wrote:
> >>> Hi folks,
> >>>
> >>> So in my continuing pursuit of perfecting my Bro setup I found that adding a proxy on every box that also runs workers keeps bro much happier then a single manager/proxy box with one or more worker(s) boxes.
> >>>
> >>> Prior to adding the additional proxies bro workers would die due to resource constraints.
> >>>
> >>> Are other folks doing this?
> >>>
> >>> Cheers,
> >>> Harry
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Bro mailing list
> >>> bro at bro-ids.org
> >>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >>
> >> _______________________________________________
> >> Bro mailing list
> >> bro at bro-ids.org
> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >
> > --
> > Aashish Sharma (asharma at lbl.gov)
> > Cyber Security,
> > Lawrence Berkeley National Laboratory
> > http://go.lbl.gov/pgp-aashish
> > Office: (510)-495-2680 Cell: (510)-612-7971
> > _______________________________________________
> > Bro mailing list
> > bro at bro-ids.org
> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>
> _______________________________________________
> Bro mailing list
> bro at bro-ids.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
More information about the Bro
mailing list