[Bro] bro and pf_ring zc configuration success stories

[radek]

That's megabits, as reported by capstats total.

cpu is:
https://gist.github.com/radoslawc/376ddb061354aec40e376214f6d830cc
nic is:
05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
Network Connection (rev 01)

-n=20,1 creates two applications, one with all traffic divided to 20 rings
(so effectively you've got 20 interfaces to attach 20 processes/threads
whatever you've got) to read packets, and another application to attach one
process to consume the same traffic (for example zcount).

Netstats is reporting 700 megabits per second (which I've assumed is amount
of traffic bro handles and drops rest, or am I wrong?), traffic that this
sensor receives is from 2 to 3.5 Gbit/s from Ixia's Breaking Point traffic
generator.

I've moved zbalance_ipc to core #1 and started bro with 4 workers:

[worker-0]
type=worker
host=localhost
interface=pf_ring::zc:27 at 0
pin_cpus=2

bound to cpu # 2,3,4,5

maybe 30 seconds into test:

[BroControl] > capstats

Interface             kpps       mbps       (10s average)
----------------------------------------
[BroControl] > netstats
   worker-0: 1506615620.937720 recvd=3019521 dropped=23608217 link=3019521
   worker-1: 1506615621.137749 recvd=2785062 dropped=18098988 link=2785062
   worker-2: 1506615621.337785 recvd=3285288 dropped=18621993 link=3285288
   worker-3: 1506615621.537699 recvd=2746902 dropped=17934732 link=2746902


all 4 cores at 100%

with zc -> dummy on 4 interfaces:

[worker-0]
type=worker
host=localhost
interface=pf_ring::dummy0
pin_cpus=2

traffic per dummy device is now:

=========================
Absolute Stats: [550'822 pkts total][0 pkts dropped][0.0% dropped]
[550'822 pkts rcvd][172'616'004 bytes rcvd][550'799.41 pkt/sec][1'380.87
Mbit/sec]
=========================
Actual Stats: [274'414 pkts rcvd][1'000.04 ms][274'402.74 pps][0.69 Gbps]
=========================

and drop rate:

[BroControl] > netstats
   worker-0: 1506616144.443402 recvd=435806 dropped=2060714 link=435806
   worker-1: 1506616148.260893 recvd=769123 dropped=1168980 link=769123
   worker-2: 1506616148.374244 recvd=774666 dropped=1157254 link=774666
   worker-3: 1506616148.639837 recvd=768808 dropped=1180893 link=768808


Best regards
Rado


On 28 September 2017 at 17:49, Azoff, Justin S <jazoff at illinois.edu> wrote:

> > On Sep 28, 2017, at 11:12 AM, radek <radoslawc at gmail.com> wrote:
> >
> > Hi!
> > Yes this was my initial setup (with dummy interfaces), I've used worker
> definition like you've suggested (pf_ring::dummy{0..19}) - before I was
> using interface=dummy{0..19}
> > It works, with the same traffic replayed, netstats:
> >
> > https://gist.github.com/radoslawc/4ca4d2f8bb0e7a2e5763d53eb31b59de
> >
> > so almost no drops,
> >
> > capstats returns nothing with interface=pf_ring::dummy{0..19}, with
> interface=dummy{0..19} it worked, but that's not the issue.
> >
> > Here's htop btw:
> > https://imgur.com/a/99ETo
> >
>
>
> Initially you said
>
> > This is with traffic around 700 - 800 Mbit/s
>
> Did you mean 700 megabits/sec or megabytes/sec ?
>
> At 700 Mbits/sec I'd expect the load on 20 workers to be almost nothing.
> What model CPU is in this box?
>
> > My question is with using dummy interfaces, doesn't it defeat purpose of
> zerocopy? It has to pass packets trough kernel to dummy interface.
>
> It's what they recommend, so it's probably fine...
>
> Another issue I see with your configuration is that you are passing -g=2
> to zbalance_ipc, which tells it to bind to core 2.  You should specifically
> bind zbalance_ipc and bro to different cores.
>
> I'm also not sure what the -n=20,1 does and if that should just be -n=20.
>
>
>
> > Also I've used worker definition for 20 of them:
> >
> > [worker-0]
> > type=worker
> > host=localhost
> > interface=pf_ring::zc:27 at 0
> > pin_cpus=1
> >
> > and result was identical as with using:
> >
> > [worker-0]
> > type=worker
> > host=localhost
> > interface=zc:27
> > lb_method=pf_ring
> > lb_procs=20
> >
>
> Can you just run 4 workers and see how it works?  You don't need 20 cores
> to handle 700mbit.  I just checked one of our worker boxes that is
> currently getting around 4000mbit with 14 workers and the cpus are at about
> 70%
>
>
>
> —
> Justin Azoff
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170928/decdc530/attachment-0001.html