[Zeek] "bro-cluster-in-a-box-setup" to "zeek-cluster-in-a-box-setup"?

Tue Feb 11 18:47:26 PST 2020

Give me a week, I already started working on it. I'll be in touch with
Amber. to post it to the official Zeek blog (and only there).

On Fri, Feb 7, 2020 at 11:29 AM Scott Wang <scwang+bro at sfu.ca> wrote:

> +1 on the article (or is it +4 now?)
>
>
> On Feb 07, 2020, at 05:23, Edgmand, Craig <craig.edgmand at okstate.edu>
> wrote:
>
> +1 on the article.
>
>
>
> *From:* zeek-bounces at zeek.org <zeek-bounces at zeek.org> *On Behalf Of *Michal
> Purzynski
> *Sent:* Thursday, February 6, 2020 4:54 PM
> *To:* Justin Hayek <jdhayek at protonmail.com>
> *Cc:* Paul Sibley <Paul.Sibley at canarie.ca>; zeek <zeek at zeek.org>
> *Subject:* Re: [Zeek] "bro-cluster-in-a-box-setup" to
> "zeek-cluster-in-a-box-setup"?
>
>
>
> **External Email - Please verify sender email address before responding.**
>
> Sure, you can run af_packet on any device, including
> device-made-of-devices, any virtual and physical interface and a
> combination thereof. The whole af_packet mechanism (they call it "taps"
> internally) works on a higher level.
>
>
>
> Now let's address the elephant in the room, shall we.
>
>
>
> IPv4 is correctly hashed on relatively modern kernels (I believe RHEL 7.4
> has a fix for that) - so you can use the cluster_flow mode.
>
> IPv6 seems to have problems, sometimes - I can see it correctly hashed
> most of the time (but not always).
>
>
>
> What we do on production, is we let card hash packets by src + dst IP
> address (and never ports, because fragments don't have port numbers), with
> the symmetric key, offloading disabled, correct number of queues set and
> cluster_qm.
>
>
>
> If the community is interested I can have an article out in a week - just
> need to know if there's someone who wants that?
>
>
>
> On Thu, Feb 6, 2020 at 2:30 PM Justin Hayek <jdhayek at protonmail.com>
> wrote:
>
> You can absolutely do this. We are using af_packet and bonded interfaces
> throughout the majority of our deployments (approximately 1800 sensors).
>
>
>
> We decided on af_packet as it was included in recent (at the time 2yrs
> ago) kernels. I can't speak to non-Debian based distro's, but we haven't
> seen any issues related to the use of af_packet.
>
>
>
> -Justin
>
>
>
>
>
>
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> On Thursday, February 6, 2020 7:04 AM, Joe Blow <blackhole.em at gmail.com>
> wrote:
>
>
>
> Would love to hear this confirmed with no performance issues.
>
>
>
> Cheers,
>
>
>
> JB
>
>
>
> Sent via BlackBerry Hub+ Inbox for Android
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.blackberry.hub&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828081603&sdata=M%2F1JueQnIyKgRX7jnBU9gH02YALsQaTDDn0Z%2FAbvIk8%3D&reserved=0>
>
> *From:* justin at corelight.com
>
> *Sent:* February 5, 2020 5:26 PM
>
> *To:* michalpurzynski1 at gmail.com
>
> *Cc:* Paul.Sibley at canarie.ca; zeek at zeek.org
>
> *Subject:* Re: [Zeek] "bro-cluster-in-a-box-setup" to
> "zeek-cluster-in-a-box-setup"?
>
> OOOH!  You can bond two interfaces together and run af_packet on the bond0
> interface? that works?!?
>
>
>
> On Wed, Feb 5, 2020 at 5:13 PM Michał Purzyński <
> michalpurzynski1 at gmail.com> wrote:
>
> There's a law that if you say pf_ring and af_packet 3 times, Michal shows
> up.
>
>
>
> I don't see many (any?) reasons for using pf_ring, TBH, if you have a
> modern kernel or a decent network card (Mellanox, Intel, etc). And I still
> owe the community the article to show how to use the af_packet correctly :/
>
>
>
> The case where one has inputs from multiple taps, to multiple network
> ports will be handled the same way by af_packet, if interfaces are bonded
> or bridged and by pf_ring. None of them buffers data and processes them at
> L4 and deals with out of order, etc.
>
>
>
> On Wed, Feb 5, 2020 at 2:04 PM Scott Wang <scwang+bro at sfu.ca> wrote:
>
> At the Canarie workshop, Steve Smoot from Corelight suggested using
> pf_ring still. Any thoughts/comments on switching to af_packet? Advantages
> vs Disadvantages?
>
>
>
> Regards,
>
> Scott
>
>
>
> On Feb 05, 2020, at 12:48, Justin Azoff <justin at corelight.com> wrote:
>
>
>
> Hi!
>
>
>
> It shouldn't be that hard to update to 3.x..
>
>
>
> - bro-pkg should be swapped out with the renamed zkg
>
> - the python2 references can likely be changed to 3
>
> - caf no longer needs to be installed separately
>
> - geoip and databases needs to be swapped out with maxminddb versions,
> might need a license
>
> - probably worth it to switch to af_packet from pf_ring.. pf_ring was only
> used initially to easily support capturing directly from both halves of a
> tap, which might not be a requirement anymore.
>
>
>
> My schedule is a bit crazy for the next week, but once I have some time to
> work on it I should be able to get things updated pretty quickly.. There's
> really not much to it.
>
>
>
>
>
>
>
> On Wed, Feb 5, 2020 at 12:38 PM Paul Sibley <Paul.Sibley at canarie.ca>
> wrote:
>
> Hello Zeek Community,
>
>
>
> I am working on a project where Zeek has been deployed in two phases.
> During the first phase, some participants used “
> https://github.com/ncsa/bro-cluster-in-a-box-setup
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fncsa%2Fbro-cluster-in-a-box-setup&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828091568&sdata=fQfU7TwSzNFS71cpv9lJX%2BgJsBMdjMxBlBKsGumxjM0%3D&reserved=0>”
> script to assist in, and automate a lot of the installation process.
>
>
>
> Since then we have entered the phase in our project where more
> participants have been added, CentOS 8 is preferred, and we are using Zeek
> 3.0.1.
>
>
>
> I wonder if any consideration, or work has been done, in updating the
> bro-cluster-in-a-box script to work with the updated OS and Zeek version.
> Any information would be appreciated.
>
>
>
> Thanks in advance,
>
>
>
> Paul Sibley
>
> _______________________________________________
>
> Zeek mailing list
>
> zeek at zeek.org
>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.icsi.berkeley.edu%2Fmailman%2Flistinfo%2Fzeek&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828091568&sdata=GJLJCcUpjnPuIhkuOGUVKOj8pbkiOJjyKbAbgq0H2Nc%3D&reserved=0>
>
>
>
>
>
> --
>
> Justin
>
> _______________________________________________
>
> Zeek mailing list
>
> zeek at zeek.org
>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.icsi.berkeley.edu%2Fmailman%2Flistinfo%2Fzeek&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828091568&sdata=GJLJCcUpjnPuIhkuOGUVKOj8pbkiOJjyKbAbgq0H2Nc%3D&reserved=0>
>
>
>
> _______________________________________________
>
> Zeek mailing list
>
> zeek at zeek.org
>
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.icsi.berkeley.edu%2Fmailman%2Flistinfo%2Fzeek&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7C2f7d95d07a784b1410e208d7ab57c87b%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637166265828101517&sdata=ZSD4CJbQpeb9pgmH6T8WUB8WpyLcEMrNtMmbVqggq1g%3D&reserved=0>
>
>
>
>
>
> --
>
> Justin
>
>
>
> _______________________________________________
> Zeek mailing list
> zeek at zeek.org
> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/zeek
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/zeek/attachments/20200211/8f7892c8/attachment-0001.html 