From oldpopsong at qq.com  Sat Jul  6 22:08:12 2019
From: oldpopsong at qq.com (=?ISO-8859-1?B?U29uZw==?=)
Date: Sun, 7 Jul 2019 13:08:12 +0800
Subject: [Zeek-Dev] Dealing with directional out-of-order packet?
In-Reply-To: <CAMzgZ0KP2U6m7iGbSiq=J8T7_k=DE4_K4b_phwUjmXK5JgLcFw@mail.gmail.com>
References: <tencent_24D7E74D078B6CA75EB78303@qq.com>
	<CAMzgZ0KP2U6m7iGbSiq=J8T7_k=DE4_K4b_phwUjmXK5JgLcFw@mail.gmail.com>
Message-ID: <tencent_2D8BEC5929FFF5D1758851FC@qq.com>

Thank you for your advice.


I agree with you that it is a capture setup problem so I gave up.


Just for the record, if anybody want to try to fix this kind of out-of-order,
avoid to access the derived analyzer class in .pac file, just use the base
class TCP_ApplicationAnalyzer, like below:


%extern{
#include "analyzer/protocol/tcp/TCP.h"
%}

refine connection HBASE_Conn += {
    %member{
        // gap recording structure
    %}

    function set_gap(seq: uint64 ,len: int, orig: bool): void
        %{
        if (orig)
            ...;
        else
            ...;
        %}

    function proc_hbase_preamble(msg: HBASE_Preamble): bool
        %{
        auto t = ((analyzer::tcp::TCP_ApplicationAnalyzer *)
            (bro_analyzer()))->TCP();



If you try to type cast bro_analyzer() back to the original
analyzer::HBase:HBASE_Analyzer *, then you may encounter
troublesome header file circular reference/forward declaration
problem.


Then just call your set_gap() in HBASE_Analyzer::Undelivered()
method like this:


void HBASE_Analyzer::Undelivered(uint64 seq, int len, bool orig)
    {
    tcp::TCP_ApplicationAnalyzer::Undelivered(seq, len, orig);
    had_gap = true;
    interp->NewGap(orig, len);
    interp->set_gap(seq, len, orig);
    }



And don't forget to remove the return statement in
HBASE_Analyzer::DeliverStream() method when has_gap is true.



And remember that the gapped data will be processed normally
when they finally arrived.



Obviously the real implementation code will be a mess.


song

------------------ Original ------------------
From:  "Jon Siwek";<jsiwek at corelight.com>;
Date:  Jun 26, 2019
To:  "Song"<oldpopsong at qq.com>; 
Cc:  "zeek-dev at zeek.org"<zeek-dev at zeek.org>; 
Subject:  Re: [Zeek-Dev] Dealing with directional out-of-order packet?



On Mon, Jun 24, 2019 at 6:34 PM Song <oldpopsong at qq.com> wrote:

> So my question is: is this other-side gap information available in
> Binpac?

Is it already available in the form you'd prefer?  Possibly not.  You
can maybe look at whether TCP_Analyzer::IsPartial() and
TCP_Analyzer::HadGap() could be used to detect the situation.

Can the exact information you want be made available?  If this is in
the context of your own custom analyzer, you can store any arbitrary
state in the analyzer that you want and access it from BinPAC code.

Ideally, this sounds like something to fix in the capture setup, not
to try to generally workaround in BinPAC/Zeek.

- Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190707/56d3d1be/attachment.html 

From dopheide at es.net  Fri Jul 12 15:45:40 2019
From: dopheide at es.net (Michael Dopheide)
Date: Fri, 12 Jul 2019 17:45:40 -0500
Subject: [Zeek-Dev] Zeek and the myricom package plugin
Message-ID: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>

Background:  We like to run 'master', but with the name change it broke too
many things and we had to stick to 2.6.2 for the time being.  Since then,
I've started trying to convert our ansible scripts and prepare to make the
jump.  Today I discovered the bro-myricom package would fail to build.

Has anyone attempted this yet?  I can get it to build with a couple minor
changes:

 src/Myricom.cc
<https://github.com/dopheide-esnet/zeek-myricom/commit/6e348389c4baa8a84b5657fefb4fc7b3dcdcf3bf#diff-218a2fe5a883051dbe028818da882eee>

@@ -1,4 +1,4 @@
#include "bro-config.h"
#include "zeek-config.h"

 tests/Scripts/get-bro-env
<https://github.com/dopheide-esnet/zeek-myricom/commit/3792461a35ec82c16efb6330cbf47f356317880c#diff-a4ddd0a8408a88cfa2a4ce426baff951>

<https://github.com/dopheide-esnet/zeek-myricom/commit/3792461a35ec82c16efb6330cbf47f356317880c#diff-a4ddd0a8408a88cfa2a4ce426baff951>
@@
-8,7 +8,7 @@ bro=`cat ${base}/../../build/CMakeCache.txt | grep BRO_DIST |
cut -d = -f 2`
if [ "$1" = "brobase" ]; then
echo ${bro}
elif [ "$1" = "bropath" ]; then
${bro}/build/bro-path-dev
${bro}/build/zeek-path-dev

(These are part of https://github.com/dopheide-esnet/zeek-myricom)

Unfortunately, we end up with another problem:
zeek -N
internal error in /home/zeek/zeek-myricom/build/scripts/./init.bro, line
37: bad reference count [0]

Line 37 is just SNF_RSS_IP:
        const snf_rss_mode: set[RssField] = {
                SNF_RSS_IP,
                SNF_RSS_SRC_PORT,
                SNF_RSS_DST_PORT
        } &redef;

Clearly I'm not gonna get lucky with an easy fix.  Doing a simple
search/replace for bro/zeek across the whole tree doesn't seem viable as
things like bro-bif.h haven't changed names.  Has anyone started digging
into how plugin packages will need to be updated?

Thanks,
-Dop
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190712/b50e5386/attachment-0001.html 

From jsiwek at corelight.com  Fri Jul 12 17:52:41 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 12 Jul 2019 17:52:41 -0700
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
Message-ID: <CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>

On Fri, Jul 12, 2019 at 3:46 PM Michael Dopheide <dopheide at es.net> wrote:

> Background:  We like to run 'master', but with the name change it broke
> too many things and we had to stick to 2.6.2 for the time being.  Since
> then, I've started trying to convert our ansible scripts and prepare to
> make the jump.  Today I discovered the bro-myricom package would fail to
> build.
>
> Has anyone attempted this yet?  I can get it to build with a couple minor
> changes:
>
>  src/Myricom.cc
> <https://github.com/dopheide-esnet/zeek-myricom/commit/6e348389c4baa8a84b5657fefb4fc7b3dcdcf3bf#diff-218a2fe5a883051dbe028818da882eee>
>
> @@ -1,4 +1,4 @@
> #include "bro-config.h"
> #include "zeek-config.h"
>
>
Can you give more info on how to reproduce this one?  The master branch
should currently be installing "zeek-config.h" along with a symlink to it
named "bro-config.h", with the ideal being that people wouldn't have to
make this change.

IIRC, since we changed our default install prefix from /usr/local/bro to
/usr/local/zeek, there's also a bit different logic if we find someone is
going to install over an old Bro installation that was still at the old
prefix, so might be good to know if you're installing fresh or upgrading
from the old version.



>  tests/Scripts/get-bro-env
> <https://github.com/dopheide-esnet/zeek-myricom/commit/3792461a35ec82c16efb6330cbf47f356317880c#diff-a4ddd0a8408a88cfa2a4ce426baff951>
>
>
> <https://github.com/dopheide-esnet/zeek-myricom/commit/3792461a35ec82c16efb6330cbf47f356317880c#diff-a4ddd0a8408a88cfa2a4ce426baff951> @@
> -8,7 +8,7 @@ bro=`cat ${base}/../../build/CMakeCache.txt | grep BRO_DIST |
> cut -d = -f 2`
> if [ "$1" = "brobase" ]; then
> echo ${bro}
> elif [ "$1" = "bropath" ]; then
> ${bro}/build/bro-path-dev
> ${bro}/build/zeek-path-dev
>
>
This one just looks like an oversight on our end, we'll need to keep
creating (or symlinking) that "build/bro-path-dev" file.


> Unfortunately, we end up with another problem:
> zeek -N
> internal error in /home/zeek/zeek-myricom/build/scripts/./init.bro, line
> 37: bad reference count [0]
>
> Line 37 is just SNF_RSS_IP:
>         const snf_rss_mode: set[RssField] = {
>                 SNF_RSS_IP,
>                 SNF_RSS_SRC_PORT,
>                 SNF_RSS_DST_PORT
>         } &redef;
>

This doesn't look related to the Bro-Zeek renaming, but possibly some enum
optimizations we did that are being tickled by what this plugin is doing.
Particularly there's an enum being declared/defined twice:

https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/src/myricom.bif#L9-L15

and

https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/scripts/init.bro#L26-L32

Possibly old versions of Bro were fine with that happening, but not
anymore.  Generally seems odd that we don't explicitly give an error
message to indicate the same enum being defined in two separate places.

I'll look more into what the proper fix is next week, but if you're just
looking to try to get something working in the meantime, a workaround may
be to comment out or remove the entire RssField enum definition inside the
init.bro script.


> Clearly I'm not gonna get lucky with an easy fix.  Doing a simple
> search/replace for bro/zeek across the whole tree doesn't seem viable as
> things like bro-bif.h haven't changed names.  Has anyone started digging
> into how plugin packages will need to be updated?
>

Generally the idea is to be mostly backward compatible so people aren't
forced to make any changes to external plugins, but looks like there's a
couple small things we overlooked or had not tested that we'll want to fix
before the 3.0 release, so thanks for the early feedback.

- Jon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190712/c5b82c8c/attachment-0001.html 

From michalpurzynski1 at gmail.com  Fri Jul 12 18:22:13 2019
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Fri, 12 Jul 2019 18:22:13 -0700
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
Message-ID: <D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>

We just migrated to master with the name change and the afpacket plugin, so I know the Zeek code is OK.


> On Jul 12, 2019, at 5:52 PM, Jon Siwek <jsiwek at corelight.com> wrote:
> 
> 
> 
>> On Fri, Jul 12, 2019 at 3:46 PM Michael Dopheide <dopheide at es.net> wrote:
>> Background:  We like to run 'master', but with the name change it broke too many things and we had to stick to 2.6.2 for the time being.  Since then, I've started trying to convert our ansible scripts and prepare to make the jump.  Today I discovered the bro-myricom package would fail to build.
>> 
>> Has anyone attempted this yet?  I can get it to build with a couple minor changes:
>> 
>>  src/Myricom.cc 
>> @@ -1,4 +1,4 @@
>> 	#include "bro-config.h"
>> 	#include "zeek-config.h"
>> 
> 
> Can you give more info on how to reproduce this one?  The master branch should currently be installing "zeek-config.h" along with a symlink to it named "bro-config.h", with the ideal being that people wouldn't have to make this change.
> 
> IIRC, since we changed our default install prefix from /usr/local/bro to /usr/local/zeek, there's also a bit different logic if we find someone is going to install over an old Bro installation that was still at the old prefix, so might be good to know if you're installing fresh or upgrading from the old version.
> 
>  
>>  tests/Scripts/get-bro-env 
>> @@ -8,7 +8,7 @@ bro=`cat ${base}/../../build/CMakeCache.txt | grep BRO_DIST | cut -d = -f 2`
>> 	if [ "$1" = "brobase" ]; then
>> 	    echo ${bro}
>> 	elif [ "$1" = "bropath" ]; then
>> 	    ${bro}/build/bro-path-dev
>> 	    ${bro}/build/zeek-path-dev
>> 
> 
> This one just looks like an oversight on our end, we'll need to keep creating (or symlinking) that "build/bro-path-dev" file.
> 
>> 
>> Unfortunately, we end up with another problem:
>> zeek -N
>> internal error in /home/zeek/zeek-myricom/build/scripts/./init.bro, line 37: bad reference count [0]
>> 
>> Line 37 is just SNF_RSS_IP:
>>         const snf_rss_mode: set[RssField] = {
>>                 SNF_RSS_IP,
>>                 SNF_RSS_SRC_PORT,
>>                 SNF_RSS_DST_PORT
>>         } &redef;
> 
> This doesn't look related to the Bro-Zeek renaming, but possibly some enum optimizations we did that are being tickled by what this plugin is doing.  Particularly there's an enum being declared/defined twice:
> 
> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/src/myricom.bif#L9-L15
> 
> and
> 
> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/scripts/init.bro#L26-L32
> 
> Possibly old versions of Bro were fine with that happening, but not anymore.  Generally seems odd that we don't explicitly give an error message to indicate the same enum being defined in two separate places.
> 
> I'll look more into what the proper fix is next week, but if you're just looking to try to get something working in the meantime, a workaround may be to comment out or remove the entire RssField enum definition inside the init.bro script.
>  
>> Clearly I'm not gonna get lucky with an easy fix.  Doing a simple search/replace for bro/zeek across the whole tree doesn't seem viable as things like bro-bif.h haven't changed names.  Has anyone started digging into how plugin packages will need to be updated?
> 
> Generally the idea is to be mostly backward compatible so people aren't forced to make any changes to external plugins, but looks like there's a couple small things we overlooked or had not tested that we'll want to fix before the 3.0 release, so thanks for the early feedback.
> 
> - Jon
> _______________________________________________
> zeek-dev mailing list
> zeek-dev at zeek.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190712/2f835ef6/attachment-0001.html 

From dopheide at es.net  Fri Jul 12 19:07:54 2019
From: dopheide at es.net (Michael Dopheide)
Date: Fri, 12 Jul 2019 21:07:54 -0500
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
Message-ID: <CAFfiZAWGzERY8wNjoewg=eQ9=wft5ioC7UTVCn704iRo-HRbUw@mail.gmail.com>

>
>
> Can you give more info on how to reproduce this one?  The master branch
> should currently be installing "zeek-config.h" along with a symlink to it
> named "bro-config.h", with the ideal being that people wouldn't have to
> make this change.
>
> IIRC, since we changed our default install prefix from /usr/local/bro to
> /usr/local/zeek, there's also a bit different logic if we find someone is
> going to install over an old Bro installation that was still at the old
> prefix, so might be good to know if you're installing fresh or upgrading
> from the old version.
>
>
This is a fresh install.  bro-config.h is a symlink in the installed
directory, but I think the issue is that the plugin is building against
bro-dist, the source tree and [bro-dist]/build/bro-config.h doesn't exist.
I can verify this by making that symlink exist.


> This one just looks like an oversight on our end, we'll need to keep
> creating (or symlinking) that "build/bro-path-dev" file.
>

Right on.


> This doesn't look related to the Bro-Zeek renaming, but possibly some enum
> optimizations we did that are being tickled by what this plugin is doing.
> Particularly there's an enum being declared/defined twice:
>
>
> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/src/myricom.bif#L9-L15
>
> and
>
>
> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/scripts/init.bro#L26-L32
>
> Possibly old versions of Bro were fine with that happening, but not
> anymore.  Generally seems odd that we don't explicitly give an error
> message to indicate the same enum being defined in two separate places.
>
> I'll look more into what the proper fix is next week, but if you're just
> looking to try to get something working in the meantime, a workaround may
> be to comment out or remove the entire RssField enum definition inside the
> init.bro script.
>

Might give that shot depending how the weekend goes.  Not in a rush, so we
can catch up next week.

Generally the idea is to be mostly backward compatible so people aren't
> forced to make any changes to external plugins, but looks like there's a
> couple small things we overlooked or had not tested that we'll want to fix
> before the 3.0 release, so thanks for the early feedback.
>

Wait, are we the only ones that try to run 'master' in production?  :)

Side conversation, what are people's thoughts on how to handle package
transition to zeek?  Using bro-myricom as an example, I made a zeek-myricom
just to help me keep in clear in my head.  Repos could just be renamed, but
I can see situations where that breaks auto-update procedures without some
hand holding.  I haven't looked too closely to see if zkg supports both
bro-pkg and zkg in the 'depends' field for meta data.

-Dop
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190712/7aec2cc3/attachment.html 

From jan.grashoefer at gmail.com  Mon Jul 15 06:56:52 2019
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Mon, 15 Jul 2019 15:56:52 +0200
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
Message-ID: <8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>

So https://github.com/J-Gras/bro-af_packet-plugin/issues/11 isn't an 
issue anymore due to backwards compatible symlinks?

On 13/07/2019 03:22, Micha? Purzy?ski wrote:
> We just migrated to master with the name change and the afpacket plugin, so I know the Zeek code is OK.
> 
> 
>> On Jul 12, 2019, at 5:52 PM, Jon Siwek <jsiwek at corelight.com> wrote:
>>
>>
>>
>>> On Fri, Jul 12, 2019 at 3:46 PM Michael Dopheide <dopheide at es.net> wrote:
>>> Background:  We like to run 'master', but with the name change it broke too many things and we had to stick to 2.6.2 for the time being.  Since then, I've started trying to convert our ansible scripts and prepare to make the jump.  Today I discovered the bro-myricom package would fail to build.
>>>
>>> Has anyone attempted this yet?  I can get it to build with a couple minor changes:
>>>
>>>   src/Myricom.cc
>>> @@ -1,4 +1,4 @@
>>> 	#include "bro-config.h"
>>> 	#include "zeek-config.h"
>>>
>>
>> Can you give more info on how to reproduce this one?  The master branch should currently be installing "zeek-config.h" along with a symlink to it named "bro-config.h", with the ideal being that people wouldn't have to make this change.
>>
>> IIRC, since we changed our default install prefix from /usr/local/bro to /usr/local/zeek, there's also a bit different logic if we find someone is going to install over an old Bro installation that was still at the old prefix, so might be good to know if you're installing fresh or upgrading from the old version.
>>
>>   
>>>   tests/Scripts/get-bro-env
>>> @@ -8,7 +8,7 @@ bro=`cat ${base}/../../build/CMakeCache.txt | grep BRO_DIST | cut -d = -f 2`
>>> 	if [ "$1" = "brobase" ]; then
>>> 	    echo ${bro}
>>> 	elif [ "$1" = "bropath" ]; then
>>> 	    ${bro}/build/bro-path-dev
>>> 	    ${bro}/build/zeek-path-dev
>>>
>>
>> This one just looks like an oversight on our end, we'll need to keep creating (or symlinking) that "build/bro-path-dev" file.
>>
>>>
>>> Unfortunately, we end up with another problem:
>>> zeek -N
>>> internal error in /home/zeek/zeek-myricom/build/scripts/./init.bro, line 37: bad reference count [0]
>>>
>>> Line 37 is just SNF_RSS_IP:
>>>          const snf_rss_mode: set[RssField] = {
>>>                  SNF_RSS_IP,
>>>                  SNF_RSS_SRC_PORT,
>>>                  SNF_RSS_DST_PORT
>>>          } &redef;
>>
>> This doesn't look related to the Bro-Zeek renaming, but possibly some enum optimizations we did that are being tickled by what this plugin is doing.  Particularly there's an enum being declared/defined twice:
>>
>> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/src/myricom.bif#L9-L15
>>
>> and
>>
>> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/scripts/init.bro#L26-L32
>>
>> Possibly old versions of Bro were fine with that happening, but not anymore.  Generally seems odd that we don't explicitly give an error message to indicate the same enum being defined in two separate places.
>>
>> I'll look more into what the proper fix is next week, but if you're just looking to try to get something working in the meantime, a workaround may be to comment out or remove the entire RssField enum definition inside the init.bro script.
>>   
>>> Clearly I'm not gonna get lucky with an easy fix.  Doing a simple search/replace for bro/zeek across the whole tree doesn't seem viable as things like bro-bif.h haven't changed names.  Has anyone started digging into how plugin packages will need to be updated?
>>
>> Generally the idea is to be mostly backward compatible so people aren't forced to make any changes to external plugins, but looks like there's a couple small things we overlooked or had not tested that we'll want to fix before the 3.0 release, so thanks for the early feedback.
>>
>> - Jon
>> _______________________________________________
>> zeek-dev mailing list
>> zeek-dev at zeek.org
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> 
> 
> _______________________________________________
> zeek-dev mailing list
> zeek-dev at zeek.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> 

From justin at corelight.com  Mon Jul 15 07:44:51 2019
From: justin at corelight.com (Justin Azoff)
Date: Mon, 15 Jul 2019 10:44:51 -0400
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
Message-ID: <CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>

Speaking of that, you made this commit:

https://github.com/J-Gras/bro-af_packet-plugin/commit/5a5d8bb74f162841111b26880137f51683e60ac1

which has the new changes(from the skeleton?) that allows it to be built
without the full source tree, but the myricom package is still using the
old cmake bits.

On Mon, Jul 15, 2019 at 9:57 AM Jan Grash?fer <jan.grashoefer at gmail.com>
wrote:

> So https://github.com/J-Gras/bro-af_packet-plugin/issues/11 isn't an
> issue anymore due to backwards compatible symlinks?
>
> On 13/07/2019 03:22, Micha? Purzy?ski wrote:
> > We just migrated to master with the name change and the afpacket plugin,
> so I know the Zeek code is OK.
> >
> >
> >> On Jul 12, 2019, at 5:52 PM, Jon Siwek <jsiwek at corelight.com> wrote:
> >>
> >>
> >>
> >>> On Fri, Jul 12, 2019 at 3:46 PM Michael Dopheide <dopheide at es.net>
> wrote:
> >>> Background:  We like to run 'master', but with the name change it
> broke too many things and we had to stick to 2.6.2 for the time being.
> Since then, I've started trying to convert our ansible scripts and prepare
> to make the jump.  Today I discovered the bro-myricom package would fail to
> build.
> >>>
> >>> Has anyone attempted this yet?  I can get it to build with a couple
> minor changes:
> >>>
> >>>   src/Myricom.cc
> >>> @@ -1,4 +1,4 @@
> >>>     #include "bro-config.h"
> >>>     #include "zeek-config.h"
> >>>
> >>
> >> Can you give more info on how to reproduce this one?  The master branch
> should currently be installing "zeek-config.h" along with a symlink to it
> named "bro-config.h", with the ideal being that people wouldn't have to
> make this change.
> >>
> >> IIRC, since we changed our default install prefix from /usr/local/bro
> to /usr/local/zeek, there's also a bit different logic if we find someone
> is going to install over an old Bro installation that was still at the old
> prefix, so might be good to know if you're installing fresh or upgrading
> from the old version.
> >>
> >>
> >>>   tests/Scripts/get-bro-env
> >>> @@ -8,7 +8,7 @@ bro=`cat ${base}/../../build/CMakeCache.txt | grep
> BRO_DIST | cut -d = -f 2`
> >>>     if [ "$1" = "brobase" ]; then
> >>>         echo ${bro}
> >>>     elif [ "$1" = "bropath" ]; then
> >>>         ${bro}/build/bro-path-dev
> >>>         ${bro}/build/zeek-path-dev
> >>>
> >>
> >> This one just looks like an oversight on our end, we'll need to keep
> creating (or symlinking) that "build/bro-path-dev" file.
> >>
> >>>
> >>> Unfortunately, we end up with another problem:
> >>> zeek -N
> >>> internal error in /home/zeek/zeek-myricom/build/scripts/./init.bro,
> line 37: bad reference count [0]
> >>>
> >>> Line 37 is just SNF_RSS_IP:
> >>>          const snf_rss_mode: set[RssField] = {
> >>>                  SNF_RSS_IP,
> >>>                  SNF_RSS_SRC_PORT,
> >>>                  SNF_RSS_DST_PORT
> >>>          } &redef;
> >>
> >> This doesn't look related to the Bro-Zeek renaming, but possibly some
> enum optimizations we did that are being tickled by what this plugin is
> doing.  Particularly there's an enum being declared/defined twice:
> >>
> >>
> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/src/myricom.bif#L9-L15
> >>
> >> and
> >>
> >>
> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/scripts/init.bro#L26-L32
> >>
> >> Possibly old versions of Bro were fine with that happening, but not
> anymore.  Generally seems odd that we don't explicitly give an error
> message to indicate the same enum being defined in two separate places.
> >>
> >> I'll look more into what the proper fix is next week, but if you're
> just looking to try to get something working in the meantime, a workaround
> may be to comment out or remove the entire RssField enum definition inside
> the init.bro script.
> >>
> >>> Clearly I'm not gonna get lucky with an easy fix.  Doing a simple
> search/replace for bro/zeek across the whole tree doesn't seem viable as
> things like bro-bif.h haven't changed names.  Has anyone started digging
> into how plugin packages will need to be updated?
> >>
> >> Generally the idea is to be mostly backward compatible so people aren't
> forced to make any changes to external plugins, but looks like there's a
> couple small things we overlooked or had not tested that we'll want to fix
> before the 3.0 release, so thanks for the early feedback.
> >>
> >> - Jon
> >> _______________________________________________
> >> zeek-dev mailing list
> >> zeek-dev at zeek.org
> >> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> >
> >
> > _______________________________________________
> > zeek-dev mailing list
> > zeek-dev at zeek.org
> > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> >
> _______________________________________________
> zeek-dev mailing list
> zeek-dev at zeek.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>


-- 
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190715/3210626c/attachment.html 

From dopheide at es.net  Mon Jul 15 09:32:39 2019
From: dopheide at es.net (Michael Dopheide)
Date: Mon, 15 Jul 2019 11:32:39 -0500
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
Message-ID: <CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>

Updating myricom to build against the install tree looks like it's going to
work and will be much cleaner.

-Dop


On Mon, Jul 15, 2019 at 9:46 AM Justin Azoff <justin at corelight.com> wrote:

> Speaking of that, you made this commit:
>
>
> https://github.com/J-Gras/bro-af_packet-plugin/commit/5a5d8bb74f162841111b26880137f51683e60ac1
>
> which has the new changes(from the skeleton?) that allows it to be built
> without the full source tree, but the myricom package is still using the
> old cmake bits.
>
> On Mon, Jul 15, 2019 at 9:57 AM Jan Grash?fer <jan.grashoefer at gmail.com>
> wrote:
>
>> So https://github.com/J-Gras/bro-af_packet-plugin/issues/11 isn't an
>> issue anymore due to backwards compatible symlinks?
>>
>> On 13/07/2019 03:22, Micha? Purzy?ski wrote:
>> > We just migrated to master with the name change and the afpacket
>> plugin, so I know the Zeek code is OK.
>> >
>> >
>> >> On Jul 12, 2019, at 5:52 PM, Jon Siwek <jsiwek at corelight.com> wrote:
>> >>
>> >>
>> >>
>> >>> On Fri, Jul 12, 2019 at 3:46 PM Michael Dopheide <dopheide at es.net>
>> wrote:
>> >>> Background:  We like to run 'master', but with the name change it
>> broke too many things and we had to stick to 2.6.2 for the time being.
>> Since then, I've started trying to convert our ansible scripts and prepare
>> to make the jump.  Today I discovered the bro-myricom package would fail to
>> build.
>> >>>
>> >>> Has anyone attempted this yet?  I can get it to build with a couple
>> minor changes:
>> >>>
>> >>>   src/Myricom.cc
>> >>> @@ -1,4 +1,4 @@
>> >>>     #include "bro-config.h"
>> >>>     #include "zeek-config.h"
>> >>>
>> >>
>> >> Can you give more info on how to reproduce this one?  The master
>> branch should currently be installing "zeek-config.h" along with a symlink
>> to it named "bro-config.h", with the ideal being that people wouldn't have
>> to make this change.
>> >>
>> >> IIRC, since we changed our default install prefix from /usr/local/bro
>> to /usr/local/zeek, there's also a bit different logic if we find someone
>> is going to install over an old Bro installation that was still at the old
>> prefix, so might be good to know if you're installing fresh or upgrading
>> from the old version.
>> >>
>> >>
>> >>>   tests/Scripts/get-bro-env
>> >>> @@ -8,7 +8,7 @@ bro=`cat ${base}/../../build/CMakeCache.txt | grep
>> BRO_DIST | cut -d = -f 2`
>> >>>     if [ "$1" = "brobase" ]; then
>> >>>         echo ${bro}
>> >>>     elif [ "$1" = "bropath" ]; then
>> >>>         ${bro}/build/bro-path-dev
>> >>>         ${bro}/build/zeek-path-dev
>> >>>
>> >>
>> >> This one just looks like an oversight on our end, we'll need to keep
>> creating (or symlinking) that "build/bro-path-dev" file.
>> >>
>> >>>
>> >>> Unfortunately, we end up with another problem:
>> >>> zeek -N
>> >>> internal error in /home/zeek/zeek-myricom/build/scripts/./init.bro,
>> line 37: bad reference count [0]
>> >>>
>> >>> Line 37 is just SNF_RSS_IP:
>> >>>          const snf_rss_mode: set[RssField] = {
>> >>>                  SNF_RSS_IP,
>> >>>                  SNF_RSS_SRC_PORT,
>> >>>                  SNF_RSS_DST_PORT
>> >>>          } &redef;
>> >>
>> >> This doesn't look related to the Bro-Zeek renaming, but possibly some
>> enum optimizations we did that are being tickled by what this plugin is
>> doing.  Particularly there's an enum being declared/defined twice:
>> >>
>> >>
>> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/src/myricom.bif#L9-L15
>> >>
>> >> and
>> >>
>> >>
>> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/scripts/init.bro#L26-L32
>> >>
>> >> Possibly old versions of Bro were fine with that happening, but not
>> anymore.  Generally seems odd that we don't explicitly give an error
>> message to indicate the same enum being defined in two separate places.
>> >>
>> >> I'll look more into what the proper fix is next week, but if you're
>> just looking to try to get something working in the meantime, a workaround
>> may be to comment out or remove the entire RssField enum definition inside
>> the init.bro script.
>> >>
>> >>> Clearly I'm not gonna get lucky with an easy fix.  Doing a simple
>> search/replace for bro/zeek across the whole tree doesn't seem viable as
>> things like bro-bif.h haven't changed names.  Has anyone started digging
>> into how plugin packages will need to be updated?
>> >>
>> >> Generally the idea is to be mostly backward compatible so people
>> aren't forced to make any changes to external plugins, but looks like
>> there's a couple small things we overlooked or had not tested that we'll
>> want to fix before the 3.0 release, so thanks for the early feedback.
>> >>
>> >> - Jon
>> >> _______________________________________________
>> >> zeek-dev mailing list
>> >> zeek-dev at zeek.org
>> >> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>> >
>> >
>> > _______________________________________________
>> > zeek-dev mailing list
>> > zeek-dev at zeek.org
>> > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>> >
>> _______________________________________________
>> zeek-dev mailing list
>> zeek-dev at zeek.org
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>
>
>
> --
> Justin
> _______________________________________________
> zeek-dev mailing list
> zeek-dev at zeek.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190715/b374f65d/attachment-0001.html 

From jsiwek at corelight.com  Mon Jul 15 10:05:54 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Mon, 15 Jul 2019 10:05:54 -0700
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAFfiZAWGzERY8wNjoewg=eQ9=wft5ioC7UTVCn704iRo-HRbUw@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<CAFfiZAWGzERY8wNjoewg=eQ9=wft5ioC7UTVCn704iRo-HRbUw@mail.gmail.com>
Message-ID: <CAMzgZ0+Zir1aSxtnuv-_gMCjVXD_wzQm0tsAEz3k5J+HoKBAyw@mail.gmail.com>

On Fri, Jul 12, 2019 at 7:08 PM Michael Dopheide <dopheide at es.net> wrote:

> Side conversation, what are people's thoughts on how to handle package transition to zeek?  Using bro-myricom as an example, I made a zeek-myricom just to help me keep in clear in my head.  Repos could just be renamed, but I can see situations where that breaks auto-update procedures without some hand holding.  I haven't looked too closely to see if zkg supports both bro-pkg and zkg in the 'depends' field for meta data.

* It's up to the package maintainer whether they want to rename -- we
can't force them.

* For packages maintained on GitHub (effectively all of them), when a
person renames a repository, GitHub automatically allows accessing the
repo via the old name/URL as a redirect to the new URL.  So people who
had installed with bro-pkg/zkg using the old URL should still be fine
going forward.

* The one thing that may break is if users had done an explicit `@load
bro-myricom` somewhere, so I've provided an `aliases` metadata field
for package maintainers to add in order to still support loading via
old names:

    https://docs.zeek.org/projects/package-manager/en/stable/package.html#aliases-field

* zkg does recognize either "zkg" or "bro-pkg" in the `depends` field.
Also "zeek" or "bro" are accepted.  However, the new "zeek" and "zkg"
names aren't recognized until zkg v2.0, so if the goal is to still
support users of bro-pkg, you'd still need to use "bro" and "bro-pkg".
Full docs:

    https://docs.zeek.org/projects/package-manager/en/stable/package.html#depends-field

- Jon


From seth at corelight.com  Tue Jul 16 07:29:24 2019
From: seth at corelight.com (Seth Hall)
Date: Tue, 16 Jul 2019 10:29:24 -0400
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
	<CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
Message-ID: <33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>

I'll take a look at it.

   .Seth

On 15 Jul 2019, at 12:32, Michael Dopheide wrote:

> Updating myricom to build against the install tree looks like it's 
> going to
> work and will be much cleaner.
>
> -Dop
>
>
> On Mon, Jul 15, 2019 at 9:46 AM Justin Azoff <justin at corelight.com> 
> wrote:
>
>> Speaking of that, you made this commit:
>>
>>
>> https://github.com/J-Gras/bro-af_packet-plugin/commit/5a5d8bb74f162841111b26880137f51683e60ac1
>>
>> which has the new changes(from the skeleton?) that allows it to be 
>> built
>> without the full source tree, but the myricom package is still using 
>> the
>> old cmake bits.
>>
>> On Mon, Jul 15, 2019 at 9:57 AM Jan Grash?fer 
>> <jan.grashoefer at gmail.com>
>> wrote:
>>
>>> So https://github.com/J-Gras/bro-af_packet-plugin/issues/11 isn't an
>>> issue anymore due to backwards compatible symlinks?
>>>
>>> On 13/07/2019 03:22, Micha? Purzy?ski wrote:
>>>> We just migrated to master with the name change and the afpacket
>>> plugin, so I know the Zeek code is OK.
>>>>
>>>>
>>>>> On Jul 12, 2019, at 5:52 PM, Jon Siwek <jsiwek at corelight.com> 
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>> On Fri, Jul 12, 2019 at 3:46 PM Michael Dopheide 
>>>>>> <dopheide at es.net>
>>> wrote:
>>>>>> Background:  We like to run 'master', but with the name change it
>>> broke too many things and we had to stick to 2.6.2 for the time 
>>> being.
>>> Since then, I've started trying to convert our ansible scripts and 
>>> prepare
>>> to make the jump.  Today I discovered the bro-myricom package would 
>>> fail to
>>> build.
>>>>>>
>>>>>> Has anyone attempted this yet?  I can get it to build with a 
>>>>>> couple
>>> minor changes:
>>>>>>
>>>>>>   src/Myricom.cc
>>>>>> @@ -1,4 +1,4 @@
>>>>>>     #include "bro-config.h"
>>>>>>     #include "zeek-config.h"
>>>>>>
>>>>>
>>>>> Can you give more info on how to reproduce this one?  The master
>>> branch should currently be installing "zeek-config.h" along with a 
>>> symlink
>>> to it named "bro-config.h", with the ideal being that people 
>>> wouldn't have
>>> to make this change.
>>>>>
>>>>> IIRC, since we changed our default install prefix from 
>>>>> /usr/local/bro
>>> to /usr/local/zeek, there's also a bit different logic if we find 
>>> someone
>>> is going to install over an old Bro installation that was still at 
>>> the old
>>> prefix, so might be good to know if you're installing fresh or 
>>> upgrading
>>> from the old version.
>>>>>
>>>>>
>>>>>>   tests/Scripts/get-bro-env
>>>>>> @@ -8,7 +8,7 @@ bro=`cat ${base}/../../build/CMakeCache.txt | 
>>>>>> grep
>>> BRO_DIST | cut -d = -f 2`
>>>>>>     if [ "$1" = "brobase" ]; then
>>>>>>         echo ${bro}
>>>>>>     elif [ "$1" = "bropath" ]; then
>>>>>>         ${bro}/build/bro-path-dev
>>>>>>         ${bro}/build/zeek-path-dev
>>>>>>
>>>>>
>>>>> This one just looks like an oversight on our end, we'll need to 
>>>>> keep
>>> creating (or symlinking) that "build/bro-path-dev" file.
>>>>>
>>>>>>
>>>>>> Unfortunately, we end up with another problem:
>>>>>> zeek -N
>>>>>> internal error in 
>>>>>> /home/zeek/zeek-myricom/build/scripts/./init.bro,
>>> line 37: bad reference count [0]
>>>>>>
>>>>>> Line 37 is just SNF_RSS_IP:
>>>>>>          const snf_rss_mode: set[RssField] = {
>>>>>>                  SNF_RSS_IP,
>>>>>>                  SNF_RSS_SRC_PORT,
>>>>>>                  SNF_RSS_DST_PORT
>>>>>>          } &redef;
>>>>>
>>>>> This doesn't look related to the Bro-Zeek renaming, but possibly 
>>>>> some
>>> enum optimizations we did that are being tickled by what this plugin 
>>> is
>>> doing.  Particularly there's an enum being declared/defined twice:
>>>>>
>>>>>
>>> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/src/myricom.bif#L9-L15
>>>>>
>>>>> and
>>>>>
>>>>>
>>> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/scripts/init.bro#L26-L32
>>>>>
>>>>> Possibly old versions of Bro were fine with that happening, but 
>>>>> not
>>> anymore.  Generally seems odd that we don't explicitly give an error
>>> message to indicate the same enum being defined in two separate 
>>> places.
>>>>>
>>>>> I'll look more into what the proper fix is next week, but if 
>>>>> you're
>>> just looking to try to get something working in the meantime, a 
>>> workaround
>>> may be to comment out or remove the entire RssField enum definition 
>>> inside
>>> the init.bro script.
>>>>>
>>>>>> Clearly I'm not gonna get lucky with an easy fix.  Doing a simple
>>> search/replace for bro/zeek across the whole tree doesn't seem 
>>> viable as
>>> things like bro-bif.h haven't changed names.  Has anyone started 
>>> digging
>>> into how plugin packages will need to be updated?
>>>>>
>>>>> Generally the idea is to be mostly backward compatible so people
>>> aren't forced to make any changes to external plugins, but looks 
>>> like
>>> there's a couple small things we overlooked or had not tested that 
>>> we'll
>>> want to fix before the 3.0 release, so thanks for the early 
>>> feedback.
>>>>>
>>>>> - Jon
>>>>> _______________________________________________
>>>>> zeek-dev mailing list
>>>>> zeek-dev at zeek.org
>>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>>>
>>>>
>>>> _______________________________________________
>>>> zeek-dev mailing list
>>>> zeek-dev at zeek.org
>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>>>
>>> _______________________________________________
>>> zeek-dev mailing list
>>> zeek-dev at zeek.org
>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>>
>>
>>
>> --
>> Justin
>> _______________________________________________
>> zeek-dev mailing list
>> zeek-dev at zeek.org
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>
> _______________________________________________
> zeek-dev mailing list
> zeek-dev at zeek.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev

--
Seth Hall * Corelight, Inc * www.corelight.com


From dopheide at es.net  Tue Jul 16 07:37:14 2019
From: dopheide at es.net (Michael Dopheide)
Date: Tue, 16 Jul 2019 09:37:14 -0500
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
	<CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
	<33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>
Message-ID: <CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>

Seth,

github.com/dopheide-esnet/zeek-myricom contains Jan?s changes as well as
removes the enum duplicate if you want to steal those.

Dop


On Tue, Jul 16, 2019 at 9:29 AM Seth Hall <seth at corelight.com> wrote:

> I'll take a look at it.
>
>    .Seth
>
> On 15 Jul 2019, at 12:32, Michael Dopheide wrote:
>
> > Updating myricom to build against the install tree looks like it's
> > going to
> > work and will be much cleaner.
> >
> > -Dop
> >
> >
> > On Mon, Jul 15, 2019 at 9:46 AM Justin Azoff <justin at corelight.com>
> > wrote:
> >
> >> Speaking of that, you made this commit:
> >>
> >>
> >>
> https://github.com/J-Gras/bro-af_packet-plugin/commit/5a5d8bb74f162841111b26880137f51683e60ac1
> >>
> >> which has the new changes(from the skeleton?) that allows it to be
> >> built
> >> without the full source tree, but the myricom package is still using
> >> the
> >> old cmake bits.
> >>
> >> On Mon, Jul 15, 2019 at 9:57 AM Jan Grash?fer
> >> <jan.grashoefer at gmail.com>
> >> wrote:
> >>
> >>> So https://github.com/J-Gras/bro-af_packet-plugin/issues/11 isn't an
> >>> issue anymore due to backwards compatible symlinks?
> >>>
> >>> On 13/07/2019 03:22, Micha? Purzy?ski wrote:
> >>>> We just migrated to master with the name change and the afpacket
> >>> plugin, so I know the Zeek code is OK.
> >>>>
> >>>>
> >>>>> On Jul 12, 2019, at 5:52 PM, Jon Siwek <jsiwek at corelight.com>
> >>>>> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Fri, Jul 12, 2019 at 3:46 PM Michael Dopheide
> >>>>>> <dopheide at es.net>
> >>> wrote:
> >>>>>> Background:  We like to run 'master', but with the name change it
> >>> broke too many things and we had to stick to 2.6.2 for the time
> >>> being.
> >>> Since then, I've started trying to convert our ansible scripts and
> >>> prepare
> >>> to make the jump.  Today I discovered the bro-myricom package would
> >>> fail to
> >>> build.
> >>>>>>
> >>>>>> Has anyone attempted this yet?  I can get it to build with a
> >>>>>> couple
> >>> minor changes:
> >>>>>>
> >>>>>>   src/Myricom.cc
> >>>>>> @@ -1,4 +1,4 @@
> >>>>>>     #include "bro-config.h"
> >>>>>>     #include "zeek-config.h"
> >>>>>>
> >>>>>
> >>>>> Can you give more info on how to reproduce this one?  The master
> >>> branch should currently be installing "zeek-config.h" along with a
> >>> symlink
> >>> to it named "bro-config.h", with the ideal being that people
> >>> wouldn't have
> >>> to make this change.
> >>>>>
> >>>>> IIRC, since we changed our default install prefix from
> >>>>> /usr/local/bro
> >>> to /usr/local/zeek, there's also a bit different logic if we find
> >>> someone
> >>> is going to install over an old Bro installation that was still at
> >>> the old
> >>> prefix, so might be good to know if you're installing fresh or
> >>> upgrading
> >>> from the old version.
> >>>>>
> >>>>>
> >>>>>>   tests/Scripts/get-bro-env
> >>>>>> @@ -8,7 +8,7 @@ bro=`cat ${base}/../../build/CMakeCache.txt |
> >>>>>> grep
> >>> BRO_DIST | cut -d = -f 2`
> >>>>>>     if [ "$1" = "brobase" ]; then
> >>>>>>         echo ${bro}
> >>>>>>     elif [ "$1" = "bropath" ]; then
> >>>>>>         ${bro}/build/bro-path-dev
> >>>>>>         ${bro}/build/zeek-path-dev
> >>>>>>
> >>>>>
> >>>>> This one just looks like an oversight on our end, we'll need to
> >>>>> keep
> >>> creating (or symlinking) that "build/bro-path-dev" file.
> >>>>>
> >>>>>>
> >>>>>> Unfortunately, we end up with another problem:
> >>>>>> zeek -N
> >>>>>> internal error in
> >>>>>> /home/zeek/zeek-myricom/build/scripts/./init.bro,
> >>> line 37: bad reference count [0]
> >>>>>>
> >>>>>> Line 37 is just SNF_RSS_IP:
> >>>>>>          const snf_rss_mode: set[RssField] = {
> >>>>>>                  SNF_RSS_IP,
> >>>>>>                  SNF_RSS_SRC_PORT,
> >>>>>>                  SNF_RSS_DST_PORT
> >>>>>>          } &redef;
> >>>>>
> >>>>> This doesn't look related to the Bro-Zeek renaming, but possibly
> >>>>> some
> >>> enum optimizations we did that are being tickled by what this plugin
> >>> is
> >>> doing.  Particularly there's an enum being declared/defined twice:
> >>>>>
> >>>>>
> >>>
> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/src/myricom.bif#L9-L15
> >>>>>
> >>>>> and
> >>>>>
> >>>>>
> >>>
> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/scripts/init.bro#L26-L32
> >>>>>
> >>>>> Possibly old versions of Bro were fine with that happening, but
> >>>>> not
> >>> anymore.  Generally seems odd that we don't explicitly give an error
> >>> message to indicate the same enum being defined in two separate
> >>> places.
> >>>>>
> >>>>> I'll look more into what the proper fix is next week, but if
> >>>>> you're
> >>> just looking to try to get something working in the meantime, a
> >>> workaround
> >>> may be to comment out or remove the entire RssField enum definition
> >>> inside
> >>> the init.bro script.
> >>>>>
> >>>>>> Clearly I'm not gonna get lucky with an easy fix.  Doing a simple
> >>> search/replace for bro/zeek across the whole tree doesn't seem
> >>> viable as
> >>> things like bro-bif.h haven't changed names.  Has anyone started
> >>> digging
> >>> into how plugin packages will need to be updated?
> >>>>>
> >>>>> Generally the idea is to be mostly backward compatible so people
> >>> aren't forced to make any changes to external plugins, but looks
> >>> like
> >>> there's a couple small things we overlooked or had not tested that
> >>> we'll
> >>> want to fix before the 3.0 release, so thanks for the early
> >>> feedback.
> >>>>>
> >>>>> - Jon
> >>>>> _______________________________________________
> >>>>> zeek-dev mailing list
> >>>>> zeek-dev at zeek.org
> >>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> zeek-dev mailing list
> >>>> zeek-dev at zeek.org
> >>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> >>>>
> >>> _______________________________________________
> >>> zeek-dev mailing list
> >>> zeek-dev at zeek.org
> >>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> >>>
> >>
> >>
> >> --
> >> Justin
> >> _______________________________________________
> >> zeek-dev mailing list
> >> zeek-dev at zeek.org
> >> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> >>
> > _______________________________________________
> > zeek-dev mailing list
> > zeek-dev at zeek.org
> > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190716/d137f04d/attachment-0001.html 

From seth at corelight.com  Tue Jul 16 07:48:33 2019
From: seth at corelight.com (Seth Hall)
Date: Tue, 16 Jul 2019 10:48:33 -0400
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
	<CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
	<33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>
	<CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>
Message-ID: <C19A4FC5-99F6-494B-BA41-60ABD8A9FCEF@corelight.com>

Ah, thanks!  I'll pull from there.

  .Seth

On 16 Jul 2019, at 10:37, Michael Dopheide wrote:

> Seth,
>
> github.com/dopheide-esnet/zeek-myricom contains Jan?s changes as 
> well as
> removes the enum duplicate if you want to steal those.
>
> Dop
>
>
> On Tue, Jul 16, 2019 at 9:29 AM Seth Hall <seth at corelight.com> wrote:
>
>> I'll take a look at it.
>>
>>    .Seth
>>
>> On 15 Jul 2019, at 12:32, Michael Dopheide wrote:
>>
>>> Updating myricom to build against the install tree looks like it's
>>> going to
>>> work and will be much cleaner.
>>>
>>> -Dop
>>>
>>>
>>> On Mon, Jul 15, 2019 at 9:46 AM Justin Azoff <justin at corelight.com>
>>> wrote:
>>>
>>>> Speaking of that, you made this commit:
>>>>
>>>>
>>>>
>> https://github.com/J-Gras/bro-af_packet-plugin/commit/5a5d8bb74f162841111b26880137f51683e60ac1
>>>>
>>>> which has the new changes(from the skeleton?) that allows it to be
>>>> built
>>>> without the full source tree, but the myricom package is still 
>>>> using
>>>> the
>>>> old cmake bits.
>>>>
>>>> On Mon, Jul 15, 2019 at 9:57 AM Jan Grash?fer
>>>> <jan.grashoefer at gmail.com>
>>>> wrote:
>>>>
>>>>> So https://github.com/J-Gras/bro-af_packet-plugin/issues/11 isn't 
>>>>> an
>>>>> issue anymore due to backwards compatible symlinks?
>>>>>
>>>>> On 13/07/2019 03:22, Micha? Purzy?ski wrote:
>>>>>> We just migrated to master with the name change and the afpacket
>>>>> plugin, so I know the Zeek code is OK.
>>>>>>
>>>>>>
>>>>>>> On Jul 12, 2019, at 5:52 PM, Jon Siwek <jsiwek at corelight.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Fri, Jul 12, 2019 at 3:46 PM Michael Dopheide
>>>>>>>> <dopheide at es.net>
>>>>> wrote:
>>>>>>>> Background:  We like to run 'master', but with the name change 
>>>>>>>> it
>>>>> broke too many things and we had to stick to 2.6.2 for the time
>>>>> being.
>>>>> Since then, I've started trying to convert our ansible scripts and
>>>>> prepare
>>>>> to make the jump.  Today I discovered the bro-myricom package 
>>>>> would
>>>>> fail to
>>>>> build.
>>>>>>>>
>>>>>>>> Has anyone attempted this yet?  I can get it to build with a
>>>>>>>> couple
>>>>> minor changes:
>>>>>>>>
>>>>>>>>   src/Myricom.cc
>>>>>>>> @@ -1,4 +1,4 @@
>>>>>>>>     #include "bro-config.h"
>>>>>>>>     #include "zeek-config.h"
>>>>>>>>
>>>>>>>
>>>>>>> Can you give more info on how to reproduce this one?  The master
>>>>> branch should currently be installing "zeek-config.h" along with a
>>>>> symlink
>>>>> to it named "bro-config.h", with the ideal being that people
>>>>> wouldn't have
>>>>> to make this change.
>>>>>>>
>>>>>>> IIRC, since we changed our default install prefix from
>>>>>>> /usr/local/bro
>>>>> to /usr/local/zeek, there's also a bit different logic if we find
>>>>> someone
>>>>> is going to install over an old Bro installation that was still at
>>>>> the old
>>>>> prefix, so might be good to know if you're installing fresh or
>>>>> upgrading
>>>>> from the old version.
>>>>>>>
>>>>>>>
>>>>>>>>   tests/Scripts/get-bro-env
>>>>>>>> @@ -8,7 +8,7 @@ bro=`cat ${base}/../../build/CMakeCache.txt |
>>>>>>>> grep
>>>>> BRO_DIST | cut -d = -f 2`
>>>>>>>>     if [ "$1" = "brobase" ]; then
>>>>>>>>         echo ${bro}
>>>>>>>>     elif [ "$1" = "bropath" ]; then
>>>>>>>>         ${bro}/build/bro-path-dev
>>>>>>>>         ${bro}/build/zeek-path-dev
>>>>>>>>
>>>>>>>
>>>>>>> This one just looks like an oversight on our end, we'll need to
>>>>>>> keep
>>>>> creating (or symlinking) that "build/bro-path-dev" file.
>>>>>>>
>>>>>>>>
>>>>>>>> Unfortunately, we end up with another problem:
>>>>>>>> zeek -N
>>>>>>>> internal error in
>>>>>>>> /home/zeek/zeek-myricom/build/scripts/./init.bro,
>>>>> line 37: bad reference count [0]
>>>>>>>>
>>>>>>>> Line 37 is just SNF_RSS_IP:
>>>>>>>>          const snf_rss_mode: set[RssField] = {
>>>>>>>>                  SNF_RSS_IP,
>>>>>>>>                  SNF_RSS_SRC_PORT,
>>>>>>>>                  SNF_RSS_DST_PORT
>>>>>>>>          } &redef;
>>>>>>>
>>>>>>> This doesn't look related to the Bro-Zeek renaming, but possibly
>>>>>>> some
>>>>> enum optimizations we did that are being tickled by what this 
>>>>> plugin
>>>>> is
>>>>> doing.  Particularly there's an enum being declared/defined twice:
>>>>>>>
>>>>>>>
>>>>>
>> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/src/myricom.bif#L9-L15
>>>>>>>
>>>>>>> and
>>>>>>>
>>>>>>>
>>>>>
>> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/scripts/init.bro#L26-L32
>>>>>>>
>>>>>>> Possibly old versions of Bro were fine with that happening, but
>>>>>>> not
>>>>> anymore.  Generally seems odd that we don't explicitly give an 
>>>>> error
>>>>> message to indicate the same enum being defined in two separate
>>>>> places.
>>>>>>>
>>>>>>> I'll look more into what the proper fix is next week, but if
>>>>>>> you're
>>>>> just looking to try to get something working in the meantime, a
>>>>> workaround
>>>>> may be to comment out or remove the entire RssField enum 
>>>>> definition
>>>>> inside
>>>>> the init.bro script.
>>>>>>>
>>>>>>>> Clearly I'm not gonna get lucky with an easy fix.  Doing a 
>>>>>>>> simple
>>>>> search/replace for bro/zeek across the whole tree doesn't seem
>>>>> viable as
>>>>> things like bro-bif.h haven't changed names.  Has anyone started
>>>>> digging
>>>>> into how plugin packages will need to be updated?
>>>>>>>
>>>>>>> Generally the idea is to be mostly backward compatible so people
>>>>> aren't forced to make any changes to external plugins, but looks
>>>>> like
>>>>> there's a couple small things we overlooked or had not tested that
>>>>> we'll
>>>>> want to fix before the 3.0 release, so thanks for the early
>>>>> feedback.
>>>>>>>
>>>>>>> - Jon
>>>>>>> _______________________________________________
>>>>>>> zeek-dev mailing list
>>>>>>> zeek-dev at zeek.org
>>>>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> zeek-dev mailing list
>>>>>> zeek-dev at zeek.org
>>>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>>>>>
>>>>> _______________________________________________
>>>>> zeek-dev mailing list
>>>>> zeek-dev at zeek.org
>>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>>>>
>>>>
>>>>
>>>> --
>>>> Justin
>>>> _______________________________________________
>>>> zeek-dev mailing list
>>>> zeek-dev at zeek.org
>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>>>
>>> _______________________________________________
>>> zeek-dev mailing list
>>> zeek-dev at zeek.org
>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>
>> --
>> Seth Hall * Corelight, Inc * www.corelight.com
>>

--
Seth Hall * Corelight, Inc * www.corelight.com


From seth at corelight.com  Tue Jul 16 09:33:28 2019
From: seth at corelight.com (Seth Hall)
Date: Tue, 16 Jul 2019 12:33:28 -0400
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
	<CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
	<33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>
	<CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>
Message-ID: <5C47D64E-59E3-49CC-9539-74D08E3650F1@corelight.com>

Does this branch look crazy to you at all?

https://github.com/sethhall/bro-myricom/tree/zeek-updates

My plan is to rename the whole project to zeek-myricom and set the alias 
as suggested by Jon once I have some external validation that this 
branch works for more people than just me. :)

   .Seth

On 16 Jul 2019, at 10:37, Michael Dopheide wrote:

> Seth,
>
> github.com/dopheide-esnet/zeek-myricom contains Jan?s changes as 
> well as
> removes the enum duplicate if you want to steal those.
>
> Dop
>
>
> On Tue, Jul 16, 2019 at 9:29 AM Seth Hall <seth at corelight.com> wrote:
>
>> I'll take a look at it.
>>
>>    .Seth
>>
>> On 15 Jul 2019, at 12:32, Michael Dopheide wrote:
>>
>>> Updating myricom to build against the install tree looks like it's
>>> going to
>>> work and will be much cleaner.
>>>
>>> -Dop
>>>
>>>
>>> On Mon, Jul 15, 2019 at 9:46 AM Justin Azoff <justin at corelight.com>
>>> wrote:
>>>
>>>> Speaking of that, you made this commit:
>>>>
>>>>
>>>>
>> https://github.com/J-Gras/bro-af_packet-plugin/commit/5a5d8bb74f162841111b26880137f51683e60ac1
>>>>
>>>> which has the new changes(from the skeleton?) that allows it to be
>>>> built
>>>> without the full source tree, but the myricom package is still 
>>>> using
>>>> the
>>>> old cmake bits.
>>>>
>>>> On Mon, Jul 15, 2019 at 9:57 AM Jan Grash?fer
>>>> <jan.grashoefer at gmail.com>
>>>> wrote:
>>>>
>>>>> So https://github.com/J-Gras/bro-af_packet-plugin/issues/11 isn't 
>>>>> an
>>>>> issue anymore due to backwards compatible symlinks?
>>>>>
>>>>> On 13/07/2019 03:22, Micha? Purzy?ski wrote:
>>>>>> We just migrated to master with the name change and the afpacket
>>>>> plugin, so I know the Zeek code is OK.
>>>>>>
>>>>>>
>>>>>>> On Jul 12, 2019, at 5:52 PM, Jon Siwek <jsiwek at corelight.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Fri, Jul 12, 2019 at 3:46 PM Michael Dopheide
>>>>>>>> <dopheide at es.net>
>>>>> wrote:
>>>>>>>> Background:  We like to run 'master', but with the name change 
>>>>>>>> it
>>>>> broke too many things and we had to stick to 2.6.2 for the time
>>>>> being.
>>>>> Since then, I've started trying to convert our ansible scripts and
>>>>> prepare
>>>>> to make the jump.  Today I discovered the bro-myricom package 
>>>>> would
>>>>> fail to
>>>>> build.
>>>>>>>>
>>>>>>>> Has anyone attempted this yet?  I can get it to build with a
>>>>>>>> couple
>>>>> minor changes:
>>>>>>>>
>>>>>>>>   src/Myricom.cc
>>>>>>>> @@ -1,4 +1,4 @@
>>>>>>>>     #include "bro-config.h"
>>>>>>>>     #include "zeek-config.h"
>>>>>>>>
>>>>>>>
>>>>>>> Can you give more info on how to reproduce this one?  The master
>>>>> branch should currently be installing "zeek-config.h" along with a
>>>>> symlink
>>>>> to it named "bro-config.h", with the ideal being that people
>>>>> wouldn't have
>>>>> to make this change.
>>>>>>>
>>>>>>> IIRC, since we changed our default install prefix from
>>>>>>> /usr/local/bro
>>>>> to /usr/local/zeek, there's also a bit different logic if we find
>>>>> someone
>>>>> is going to install over an old Bro installation that was still at
>>>>> the old
>>>>> prefix, so might be good to know if you're installing fresh or
>>>>> upgrading
>>>>> from the old version.
>>>>>>>
>>>>>>>
>>>>>>>>   tests/Scripts/get-bro-env
>>>>>>>> @@ -8,7 +8,7 @@ bro=`cat ${base}/../../build/CMakeCache.txt |
>>>>>>>> grep
>>>>> BRO_DIST | cut -d = -f 2`
>>>>>>>>     if [ "$1" = "brobase" ]; then
>>>>>>>>         echo ${bro}
>>>>>>>>     elif [ "$1" = "bropath" ]; then
>>>>>>>>         ${bro}/build/bro-path-dev
>>>>>>>>         ${bro}/build/zeek-path-dev
>>>>>>>>
>>>>>>>
>>>>>>> This one just looks like an oversight on our end, we'll need to
>>>>>>> keep
>>>>> creating (or symlinking) that "build/bro-path-dev" file.
>>>>>>>
>>>>>>>>
>>>>>>>> Unfortunately, we end up with another problem:
>>>>>>>> zeek -N
>>>>>>>> internal error in
>>>>>>>> /home/zeek/zeek-myricom/build/scripts/./init.bro,
>>>>> line 37: bad reference count [0]
>>>>>>>>
>>>>>>>> Line 37 is just SNF_RSS_IP:
>>>>>>>>          const snf_rss_mode: set[RssField] = {
>>>>>>>>                  SNF_RSS_IP,
>>>>>>>>                  SNF_RSS_SRC_PORT,
>>>>>>>>                  SNF_RSS_DST_PORT
>>>>>>>>          } &redef;
>>>>>>>
>>>>>>> This doesn't look related to the Bro-Zeek renaming, but possibly
>>>>>>> some
>>>>> enum optimizations we did that are being tickled by what this 
>>>>> plugin
>>>>> is
>>>>> doing.  Particularly there's an enum being declared/defined twice:
>>>>>>>
>>>>>>>
>>>>>
>> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/src/myricom.bif#L9-L15
>>>>>>>
>>>>>>> and
>>>>>>>
>>>>>>>
>>>>>
>> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/scripts/init.bro#L26-L32
>>>>>>>
>>>>>>> Possibly old versions of Bro were fine with that happening, but
>>>>>>> not
>>>>> anymore.  Generally seems odd that we don't explicitly give an 
>>>>> error
>>>>> message to indicate the same enum being defined in two separate
>>>>> places.
>>>>>>>
>>>>>>> I'll look more into what the proper fix is next week, but if
>>>>>>> you're
>>>>> just looking to try to get something working in the meantime, a
>>>>> workaround
>>>>> may be to comment out or remove the entire RssField enum 
>>>>> definition
>>>>> inside
>>>>> the init.bro script.
>>>>>>>
>>>>>>>> Clearly I'm not gonna get lucky with an easy fix.  Doing a 
>>>>>>>> simple
>>>>> search/replace for bro/zeek across the whole tree doesn't seem
>>>>> viable as
>>>>> things like bro-bif.h haven't changed names.  Has anyone started
>>>>> digging
>>>>> into how plugin packages will need to be updated?
>>>>>>>
>>>>>>> Generally the idea is to be mostly backward compatible so people
>>>>> aren't forced to make any changes to external plugins, but looks
>>>>> like
>>>>> there's a couple small things we overlooked or had not tested that
>>>>> we'll
>>>>> want to fix before the 3.0 release, so thanks for the early
>>>>> feedback.
>>>>>>>
>>>>>>> - Jon
>>>>>>> _______________________________________________
>>>>>>> zeek-dev mailing list
>>>>>>> zeek-dev at zeek.org
>>>>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> zeek-dev mailing list
>>>>>> zeek-dev at zeek.org
>>>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>>>>>
>>>>> _______________________________________________
>>>>> zeek-dev mailing list
>>>>> zeek-dev at zeek.org
>>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>>>>
>>>>
>>>>
>>>> --
>>>> Justin
>>>> _______________________________________________
>>>> zeek-dev mailing list
>>>> zeek-dev at zeek.org
>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>>>
>>> _______________________________________________
>>> zeek-dev mailing list
>>> zeek-dev at zeek.org
>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>>
>> --
>> Seth Hall * Corelight, Inc * www.corelight.com
>>

--
Seth Hall * Corelight, Inc * www.corelight.com


From dopheide at es.net  Tue Jul 16 10:07:29 2019
From: dopheide at es.net (Michael Dopheide)
Date: Tue, 16 Jul 2019 12:07:29 -0500
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <5C47D64E-59E3-49CC-9539-74D08E3650F1@corelight.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
	<CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
	<33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>
	<CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>
	<5C47D64E-59E3-49CC-9539-74D08E3650F1@corelight.com>
Message-ID: <CAFfiZAVzee1i52JE1t3qxrBCD0Hd_ozz5y6Sr1KZsHvLoGrzGA@mail.gmail.com>

It builds and works for me against master.  Should we change the
bro-pkg/zkg requirement to > 2.0?  It should be backwards compatible only
as far as when all the required includes, etc were included in the install
tree.  Not sure how far back that change was made.

-Dop

On Tue, Jul 16, 2019 at 11:33 AM Seth Hall <seth at corelight.com> wrote:

> Does this branch look crazy to you at all?
>
> https://github.com/sethhall/bro-myricom/tree/zeek-updates
>
> My plan is to rename the whole project to zeek-myricom and set the alias
> as suggested by Jon once I have some external validation that this
> branch works for more people than just me. :)
>
>    .Seth
>
> On 16 Jul 2019, at 10:37, Michael Dopheide wrote:
>
> > Seth,
> >
> > github.com/dopheide-esnet/zeek-myricom contains Jan?s changes as
> > well as
> > removes the enum duplicate if you want to steal those.
> >
> > Dop
> >
> >
> > On Tue, Jul 16, 2019 at 9:29 AM Seth Hall <seth at corelight.com> wrote:
> >
> >> I'll take a look at it.
> >>
> >>    .Seth
> >>
> >> On 15 Jul 2019, at 12:32, Michael Dopheide wrote:
> >>
> >>> Updating myricom to build against the install tree looks like it's
> >>> going to
> >>> work and will be much cleaner.
> >>>
> >>> -Dop
> >>>
> >>>
> >>> On Mon, Jul 15, 2019 at 9:46 AM Justin Azoff <justin at corelight.com>
> >>> wrote:
> >>>
> >>>> Speaking of that, you made this commit:
> >>>>
> >>>>
> >>>>
> >>
> https://github.com/J-Gras/bro-af_packet-plugin/commit/5a5d8bb74f162841111b26880137f51683e60ac1
> >>>>
> >>>> which has the new changes(from the skeleton?) that allows it to be
> >>>> built
> >>>> without the full source tree, but the myricom package is still
> >>>> using
> >>>> the
> >>>> old cmake bits.
> >>>>
> >>>> On Mon, Jul 15, 2019 at 9:57 AM Jan Grash?fer
> >>>> <jan.grashoefer at gmail.com>
> >>>> wrote:
> >>>>
> >>>>> So https://github.com/J-Gras/bro-af_packet-plugin/issues/11 isn't
> >>>>> an
> >>>>> issue anymore due to backwards compatible symlinks?
> >>>>>
> >>>>> On 13/07/2019 03:22, Micha? Purzy?ski wrote:
> >>>>>> We just migrated to master with the name change and the afpacket
> >>>>> plugin, so I know the Zeek code is OK.
> >>>>>>
> >>>>>>
> >>>>>>> On Jul 12, 2019, at 5:52 PM, Jon Siwek <jsiwek at corelight.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Fri, Jul 12, 2019 at 3:46 PM Michael Dopheide
> >>>>>>>> <dopheide at es.net>
> >>>>> wrote:
> >>>>>>>> Background:  We like to run 'master', but with the name change
> >>>>>>>> it
> >>>>> broke too many things and we had to stick to 2.6.2 for the time
> >>>>> being.
> >>>>> Since then, I've started trying to convert our ansible scripts and
> >>>>> prepare
> >>>>> to make the jump.  Today I discovered the bro-myricom package
> >>>>> would
> >>>>> fail to
> >>>>> build.
> >>>>>>>>
> >>>>>>>> Has anyone attempted this yet?  I can get it to build with a
> >>>>>>>> couple
> >>>>> minor changes:
> >>>>>>>>
> >>>>>>>>   src/Myricom.cc
> >>>>>>>> @@ -1,4 +1,4 @@
> >>>>>>>>     #include "bro-config.h"
> >>>>>>>>     #include "zeek-config.h"
> >>>>>>>>
> >>>>>>>
> >>>>>>> Can you give more info on how to reproduce this one?  The master
> >>>>> branch should currently be installing "zeek-config.h" along with a
> >>>>> symlink
> >>>>> to it named "bro-config.h", with the ideal being that people
> >>>>> wouldn't have
> >>>>> to make this change.
> >>>>>>>
> >>>>>>> IIRC, since we changed our default install prefix from
> >>>>>>> /usr/local/bro
> >>>>> to /usr/local/zeek, there's also a bit different logic if we find
> >>>>> someone
> >>>>> is going to install over an old Bro installation that was still at
> >>>>> the old
> >>>>> prefix, so might be good to know if you're installing fresh or
> >>>>> upgrading
> >>>>> from the old version.
> >>>>>>>
> >>>>>>>
> >>>>>>>>   tests/Scripts/get-bro-env
> >>>>>>>> @@ -8,7 +8,7 @@ bro=`cat ${base}/../../build/CMakeCache.txt |
> >>>>>>>> grep
> >>>>> BRO_DIST | cut -d = -f 2`
> >>>>>>>>     if [ "$1" = "brobase" ]; then
> >>>>>>>>         echo ${bro}
> >>>>>>>>     elif [ "$1" = "bropath" ]; then
> >>>>>>>>         ${bro}/build/bro-path-dev
> >>>>>>>>         ${bro}/build/zeek-path-dev
> >>>>>>>>
> >>>>>>>
> >>>>>>> This one just looks like an oversight on our end, we'll need to
> >>>>>>> keep
> >>>>> creating (or symlinking) that "build/bro-path-dev" file.
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Unfortunately, we end up with another problem:
> >>>>>>>> zeek -N
> >>>>>>>> internal error in
> >>>>>>>> /home/zeek/zeek-myricom/build/scripts/./init.bro,
> >>>>> line 37: bad reference count [0]
> >>>>>>>>
> >>>>>>>> Line 37 is just SNF_RSS_IP:
> >>>>>>>>          const snf_rss_mode: set[RssField] = {
> >>>>>>>>                  SNF_RSS_IP,
> >>>>>>>>                  SNF_RSS_SRC_PORT,
> >>>>>>>>                  SNF_RSS_DST_PORT
> >>>>>>>>          } &redef;
> >>>>>>>
> >>>>>>> This doesn't look related to the Bro-Zeek renaming, but possibly
> >>>>>>> some
> >>>>> enum optimizations we did that are being tickled by what this
> >>>>> plugin
> >>>>> is
> >>>>> doing.  Particularly there's an enum being declared/defined twice:
> >>>>>>>
> >>>>>>>
> >>>>>
> >>
> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/src/myricom.bif#L9-L15
> >>>>>>>
> >>>>>>> and
> >>>>>>>
> >>>>>>>
> >>>>>
> >>
> https://github.com/sethhall/bro-myricom/blob/89815d89e0ba6957149521cf99e608c0dc909f6d/scripts/init.bro#L26-L32
> >>>>>>>
> >>>>>>> Possibly old versions of Bro were fine with that happening, but
> >>>>>>> not
> >>>>> anymore.  Generally seems odd that we don't explicitly give an
> >>>>> error
> >>>>> message to indicate the same enum being defined in two separate
> >>>>> places.
> >>>>>>>
> >>>>>>> I'll look more into what the proper fix is next week, but if
> >>>>>>> you're
> >>>>> just looking to try to get something working in the meantime, a
> >>>>> workaround
> >>>>> may be to comment out or remove the entire RssField enum
> >>>>> definition
> >>>>> inside
> >>>>> the init.bro script.
> >>>>>>>
> >>>>>>>> Clearly I'm not gonna get lucky with an easy fix.  Doing a
> >>>>>>>> simple
> >>>>> search/replace for bro/zeek across the whole tree doesn't seem
> >>>>> viable as
> >>>>> things like bro-bif.h haven't changed names.  Has anyone started
> >>>>> digging
> >>>>> into how plugin packages will need to be updated?
> >>>>>>>
> >>>>>>> Generally the idea is to be mostly backward compatible so people
> >>>>> aren't forced to make any changes to external plugins, but looks
> >>>>> like
> >>>>> there's a couple small things we overlooked or had not tested that
> >>>>> we'll
> >>>>> want to fix before the 3.0 release, so thanks for the early
> >>>>> feedback.
> >>>>>>>
> >>>>>>> - Jon
> >>>>>>> _______________________________________________
> >>>>>>> zeek-dev mailing list
> >>>>>>> zeek-dev at zeek.org
> >>>>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> zeek-dev mailing list
> >>>>>> zeek-dev at zeek.org
> >>>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> >>>>>>
> >>>>> _______________________________________________
> >>>>> zeek-dev mailing list
> >>>>> zeek-dev at zeek.org
> >>>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Justin
> >>>> _______________________________________________
> >>>> zeek-dev mailing list
> >>>> zeek-dev at zeek.org
> >>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> >>>>
> >>> _______________________________________________
> >>> zeek-dev mailing list
> >>> zeek-dev at zeek.org
> >>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> >>
> >> --
> >> Seth Hall * Corelight, Inc * www.corelight.com
> >>
>
> --
> Seth Hall * Corelight, Inc * www.corelight.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190716/561c856b/attachment-0001.html 

From jsiwek at corelight.com  Tue Jul 16 10:36:16 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 16 Jul 2019 10:36:16 -0700
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAFfiZAVzee1i52JE1t3qxrBCD0Hd_ozz5y6Sr1KZsHvLoGrzGA@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
	<CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
	<33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>
	<CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>
	<5C47D64E-59E3-49CC-9539-74D08E3650F1@corelight.com>
	<CAFfiZAVzee1i52JE1t3qxrBCD0Hd_ozz5y6Sr1KZsHvLoGrzGA@mail.gmail.com>
Message-ID: <CAMzgZ0KQMLBTEKNKjyD8p3bD68xeJ5pMtc9cbWK2giNBt4w0Ag@mail.gmail.com>

On Tue, Jul 16, 2019 at 10:14 AM Michael Dopheide <dopheide at es.net> wrote:
>
> It builds and works for me against master.  Should we change the bro-pkg/zkg requirement to > 2.0?

At a glance, I only see a "bro-pkg >=1.5.0" requirement due to use of
`aliases` metadata field.

> It should be backwards compatible only as far as when all the required includes, etc were included in the install tree.  Not sure how far back that change was made.

Bro 2.6 is when headers started to get installed such that packages
could be independent from the source-tree, so adding a "bro >=2.6.0"
requirement makes sense to me.

- Jon

From seth at corelight.com  Tue Jul 16 10:55:07 2019
From: seth at corelight.com (Seth Hall)
Date: Tue, 16 Jul 2019 13:55:07 -0400
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAMzgZ0KQMLBTEKNKjyD8p3bD68xeJ5pMtc9cbWK2giNBt4w0Ag@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
	<CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
	<33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>
	<CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>
	<5C47D64E-59E3-49CC-9539-74D08E3650F1@corelight.com>
	<CAFfiZAVzee1i52JE1t3qxrBCD0Hd_ozz5y6Sr1KZsHvLoGrzGA@mail.gmail.com>
	<CAMzgZ0KQMLBTEKNKjyD8p3bD68xeJ5pMtc9cbWK2giNBt4w0Ag@mail.gmail.com>
Message-ID: <4E6585BF-468C-4D4C-82C2-E7839BECAB87@corelight.com>

Ok, I updated the dependencies listed bro-pkg.cfg to get the correct 
bro-pkg and bro.

Are there suggestions about how I should merge this branch into master 
and not break people installing this on current releases?

   .Seth

On 16 Jul 2019, at 13:36, Jon Siwek wrote:

> On Tue, Jul 16, 2019 at 10:14 AM Michael Dopheide <dopheide at es.net> 
> wrote:
>>
>> It builds and works for me against master.  Should we change the 
>> bro-pkg/zkg requirement to > 2.0?
>
> At a glance, I only see a "bro-pkg >=1.5.0" requirement due to use of
> `aliases` metadata field.
>
>> It should be backwards compatible only as far as when all the 
>> required includes, etc were included in the install tree.  Not sure 
>> how far back that change was made.
>
> Bro 2.6 is when headers started to get installed such that packages
> could be independent from the source-tree, so adding a "bro >=2.6.0"
> requirement makes sense to me.
>
> - Jon

--
Seth Hall * Corelight, Inc * www.corelight.com

From jsiwek at corelight.com  Tue Jul 16 13:12:41 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Tue, 16 Jul 2019 13:12:41 -0700
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <4E6585BF-468C-4D4C-82C2-E7839BECAB87@corelight.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
	<CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
	<33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>
	<CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>
	<5C47D64E-59E3-49CC-9539-74D08E3650F1@corelight.com>
	<CAFfiZAVzee1i52JE1t3qxrBCD0Hd_ozz5y6Sr1KZsHvLoGrzGA@mail.gmail.com>
	<CAMzgZ0KQMLBTEKNKjyD8p3bD68xeJ5pMtc9cbWK2giNBt4w0Ag@mail.gmail.com>
	<4E6585BF-468C-4D4C-82C2-E7839BECAB87@corelight.com>
Message-ID: <CAMzgZ0+CKePTdxw+J8DFpYhnnsp7cnVNox2fKfzgHuH3D4YYuw@mail.gmail.com>

On Tue, Jul 16, 2019 at 10:55 AM Seth Hall <seth at corelight.com> wrote:

> Are there suggestions about how I should merge this branch into master
> and not break people installing this on current releases?

Best would be to actually test against 2.6 and master to see if anything breaks.

At a minimum, actually I see these probably need to change to be compatible:

* get-bro-env: if "zeek-path-dev" doesn't exist fallback to "bro-path-dev"

* broctl/myricom.py: may need a try block to see if the ZeekControl
stuff exists, else still use BroControl

- Jon

From michalpurzynski1 at gmail.com  Tue Jul 16 13:35:31 2019
From: michalpurzynski1 at gmail.com (=?UTF-8?B?TWljaGHFgiBQdXJ6ecWEc2tp?=)
Date: Tue, 16 Jul 2019 13:35:31 -0700
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAFfiZAWGzERY8wNjoewg=eQ9=wft5ioC7UTVCn704iRo-HRbUw@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<CAFfiZAWGzERY8wNjoewg=eQ9=wft5ioC7UTVCn704iRo-HRbUw@mail.gmail.com>
Message-ID: <CAJ6bFK2pprRZS8ksTUhAodtBSaQw3o9Rk8oOkSTbqQfi--_Erg@mail.gmail.com>

On Fri, Jul 12, 2019 at 7:15 PM Michael Dopheide <dopheide at es.net> wrote:

>
> Generally the idea is to be mostly backward compatible so people aren't
> forced to make any changes to external plugins, but looks like there's a
> couple small things we overlooked or had not tested that we'll want to fix
> before the 3.0 release, so thanks for the early feedback.
>
> Wait, are we the only ones that try to run 'master' in production?  :)
>

Nope. I can count maybe weeks when Mozilla was running a stable release in
production - and we've been rocking this boat since like 7 years now?

And now I feel old.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190716/b5ebb285/attachment.html 

From dopheide at es.net  Fri Jul 19 13:52:06 2019
From: dopheide at es.net (Michael Dopheide)
Date: Fri, 19 Jul 2019 15:52:06 -0500
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAMzgZ0+CKePTdxw+J8DFpYhnnsp7cnVNox2fKfzgHuH3D4YYuw@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
	<CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
	<33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>
	<CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>
	<5C47D64E-59E3-49CC-9539-74D08E3650F1@corelight.com>
	<CAFfiZAVzee1i52JE1t3qxrBCD0Hd_ozz5y6Sr1KZsHvLoGrzGA@mail.gmail.com>
	<CAMzgZ0KQMLBTEKNKjyD8p3bD68xeJ5pMtc9cbWK2giNBt4w0Ag@mail.gmail.com>
	<4E6585BF-468C-4D4C-82C2-E7839BECAB87@corelight.com>
	<CAMzgZ0+CKePTdxw+J8DFpYhnnsp7cnVNox2fKfzgHuH3D4YYuw@mail.gmail.com>
Message-ID: <CAFfiZAW6uqivJ7DuvzCbiy1bjuokoQt1SwZpiuF=Q6Qqkeo1=w@mail.gmail.com>

More testing revealed another gotcha against master...

CMake is expecting to be able to copy myricom.bif.bro to it's destination,
but only myricom.bif.zeek exists.

make
...
[ 11%] [BIFCL] Processing src/myricom.bif
Error copying file "myricom.bif.bro" to
"/home/zeek/.zkg/testing/zeek-myricom/clones/zeek-myricom/build/lib/bif/myricom.bif.bro".

Not sure if that's just a problem on my end or more fundamental to the name
change with plugins.

-Dop

On Tue, Jul 16, 2019 at 3:12 PM Jon Siwek <jsiwek at corelight.com> wrote:

> On Tue, Jul 16, 2019 at 10:55 AM Seth Hall <seth at corelight.com> wrote:
>
> > Are there suggestions about how I should merge this branch into master
> > and not break people installing this on current releases?
>
> Best would be to actually test against 2.6 and master to see if anything
> breaks.
>
> At a minimum, actually I see these probably need to change to be
> compatible:
>
> * get-bro-env: if "zeek-path-dev" doesn't exist fallback to "bro-path-dev"
>
> * broctl/myricom.py: may need a try block to see if the ZeekControl
> stuff exists, else still use BroControl
>
> - Jon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190719/e7872963/attachment.html 

From jsiwek at corelight.com  Fri Jul 19 19:59:23 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Fri, 19 Jul 2019 19:59:23 -0700
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAFfiZAW6uqivJ7DuvzCbiy1bjuokoQt1SwZpiuF=Q6Qqkeo1=w@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
	<CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
	<33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>
	<CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>
	<5C47D64E-59E3-49CC-9539-74D08E3650F1@corelight.com>
	<CAFfiZAVzee1i52JE1t3qxrBCD0Hd_ozz5y6Sr1KZsHvLoGrzGA@mail.gmail.com>
	<CAMzgZ0KQMLBTEKNKjyD8p3bD68xeJ5pMtc9cbWK2giNBt4w0Ag@mail.gmail.com>
	<4E6585BF-468C-4D4C-82C2-E7839BECAB87@corelight.com>
	<CAMzgZ0+CKePTdxw+J8DFpYhnnsp7cnVNox2fKfzgHuH3D4YYuw@mail.gmail.com>
	<CAFfiZAW6uqivJ7DuvzCbiy1bjuokoQt1SwZpiuF=Q6Qqkeo1=w@mail.gmail.com>
Message-ID: <CAMzgZ0LcAk0nzfnCfxzRY_Ye+D1LU1OdPGmyp2_sQDCqYxyEfQ@mail.gmail.com>

On Fri, Jul 19, 2019 at 1:52 PM Michael Dopheide <dopheide at es.net> wrote:

> CMake is expecting to be able to copy myricom.bif.bro to it's destination, but only myricom.bif.zeek exists.
>
> make
> ...
> [ 11%] [BIFCL] Processing src/myricom.bif
> Error copying file "myricom.bif.bro" to "/home/zeek/.zkg/testing/zeek-myricom/clones/zeek-myricom/build/lib/bif/myricom.bif.bro".

* The thing which produces the .bif.bro versus .bif.zeek file is
`bifcl`.  Seems like you've got the new version since it made a
.bif.zeek

* The thing which generates a Makefile that expects to copy .bif.bro
versus .bif.zeek is cmake/BifCl.cmake.  Seems you may have an old
version of this since it created a Makefile that still expects a
.bif.bro

I didn't test building the myricom plugin, but a simple plugin
containing a .bif doesn't seem to generally have this problem when
using a fully updated zeek/master: it produced and installed a
.bif.zeek file.

Maybe verify you've got a fully updated and consistent Zeek build
and/or give the full command and build log you're running.  The
configuration output from the build log could have a hint that it's
picking up an unexpected Bro/Zeek build or install path.

If installing via zkg, an outdated "zeek_dist" in ~/.zkg/config is a
problem if it's still using a version of the myricom package that uses
it, else if there's an old version of "bro-config" (now "zeek-config")
in PATH, that could be a problem.

- Jon

From michalpurzynski1 at gmail.com  Fri Jul 19 21:07:48 2019
From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=)
Date: Fri, 19 Jul 2019 21:07:48 -0700
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <CAMzgZ0LcAk0nzfnCfxzRY_Ye+D1LU1OdPGmyp2_sQDCqYxyEfQ@mail.gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
	<CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
	<33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>
	<CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>
	<5C47D64E-59E3-49CC-9539-74D08E3650F1@corelight.com>
	<CAFfiZAVzee1i52JE1t3qxrBCD0Hd_ozz5y6Sr1KZsHvLoGrzGA@mail.gmail.com>
	<CAMzgZ0KQMLBTEKNKjyD8p3bD68xeJ5pMtc9cbWK2giNBt4w0Ag@mail.gmail.com>
	<4E6585BF-468C-4D4C-82C2-E7839BECAB87@corelight.com>
	<CAMzgZ0+CKePTdxw+J8DFpYhnnsp7cnVNox2fKfzgHuH3D4YYuw@mail.gmail.com>
	<CAFfiZAW6uqivJ7DuvzCbiy1bjuokoQt1SwZpiuF=Q6Qqkeo1=w@mail.gmail.com>
	<CAMzgZ0LcAk0nzfnCfxzRY_Ye+D1LU1OdPGmyp2_sQDCqYxyEfQ@mail.gmail.com>
Message-ID: <F963DA64-E3F1-4113-8B4F-8EAB69A2FCCE@gmail.com>

I tested it today and was able to build the myricom plugin without any problems.

It was a clean system, with the fresh copy of Zeek master and a clone of the plugin from Seth repo.

That was without using zkg mind you.


> On Jul 19, 2019, at 7:59 PM, Jon Siwek <jsiwek at corelight.com> wrote:
> 
>> On Fri, Jul 19, 2019 at 1:52 PM Michael Dopheide <dopheide at es.net> wrote:
>> 
>> CMake is expecting to be able to copy myricom.bif.bro to it's destination, but only myricom.bif.zeek exists.
>> 
>> make
>> ...
>> [ 11%] [BIFCL] Processing src/myricom.bif
>> Error copying file "myricom.bif.bro" to "/home/zeek/.zkg/testing/zeek-myricom/clones/zeek-myricom/build/lib/bif/myricom.bif.bro".
> 
> * The thing which produces the .bif.bro versus .bif.zeek file is
> `bifcl`.  Seems like you've got the new version since it made a
> .bif.zeek
> 
> * The thing which generates a Makefile that expects to copy .bif.bro
> versus .bif.zeek is cmake/BifCl.cmake.  Seems you may have an old
> version of this since it created a Makefile that still expects a
> .bif.bro
> 
> I didn't test building the myricom plugin, but a simple plugin
> containing a .bif doesn't seem to generally have this problem when
> using a fully updated zeek/master: it produced and installed a
> .bif.zeek file.
> 
> Maybe verify you've got a fully updated and consistent Zeek build
> and/or give the full command and build log you're running.  The
> configuration output from the build log could have a hint that it's
> picking up an unexpected Bro/Zeek build or install path.
> 
> If installing via zkg, an outdated "zeek_dist" in ~/.zkg/config is a
> problem if it's still using a version of the myricom package that uses
> it, else if there's an old version of "bro-config" (now "zeek-config")
> in PATH, that could be a problem.
> 
> - Jon
> _______________________________________________
> zeek-dev mailing list
> zeek-dev at zeek.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev


From dopheide at es.net  Sat Jul 20 15:58:14 2019
From: dopheide at es.net (Michael Dopheide)
Date: Sat, 20 Jul 2019 17:58:14 -0500
Subject: [Zeek-Dev] Zeek and the myricom package plugin
In-Reply-To: <F963DA64-E3F1-4113-8B4F-8EAB69A2FCCE@gmail.com>
References: <CAFfiZAUNb-Vddfd=TZQDE_pMUfVchS=1XwmyzuqeBDUfGO0qSw@mail.gmail.com>
	<CAMzgZ0KaHmzr+AKGqC4abp3426FrKtD3ui1k+pxYNkJEzjVaqg@mail.gmail.com>
	<D3895C64-E6BC-440A-9820-A497A78633EA@gmail.com>
	<8123862a-b290-ce4b-e774-11679aaf9d3f@gmail.com>
	<CAPfnCujOMTf6vBNz33bJNi=v-3OoXO660p+vr-8W4eF3yjW0jw@mail.gmail.com>
	<CAFfiZAXjc=xfiKPGCZGveL5SYX1bS8Ca99V+Z2Q9eKxQX9jwkQ@mail.gmail.com>
	<33330BBF-E765-410F-AF18-FA6413AB3362@corelight.com>
	<CAFfiZAWWKg6ohp3tan6heL50ptf3qTEjn0WPygrBziNW5z=U6w@mail.gmail.com>
	<5C47D64E-59E3-49CC-9539-74D08E3650F1@corelight.com>
	<CAFfiZAVzee1i52JE1t3qxrBCD0Hd_ozz5y6Sr1KZsHvLoGrzGA@mail.gmail.com>
	<CAMzgZ0KQMLBTEKNKjyD8p3bD68xeJ5pMtc9cbWK2giNBt4w0Ag@mail.gmail.com>
	<4E6585BF-468C-4D4C-82C2-E7839BECAB87@corelight.com>
	<CAMzgZ0+CKePTdxw+J8DFpYhnnsp7cnVNox2fKfzgHuH3D4YYuw@mail.gmail.com>
	<CAFfiZAW6uqivJ7DuvzCbiy1bjuokoQt1SwZpiuF=Q6Qqkeo1=w@mail.gmail.com>
	<CAMzgZ0LcAk0nzfnCfxzRY_Ye+D1LU1OdPGmyp2_sQDCqYxyEfQ@mail.gmail.com>
	<F963DA64-E3F1-4113-8B4F-8EAB69A2FCCE@gmail.com>
Message-ID: <CAFfiZAVEWHRLXTbEfQ8FYN93kadB0XOQnMrA4L9S0bX6Cvq4BA@mail.gmail.com>

Yeah, I think my problem was just overlap, but I feel we should keep a list
of these kinds of things to look out for in the transition.  For instance,
remembering to remove cron jobs that run as the 'bro' user when that user
no longer exists.  We've also got systems where /usr/local/bro/logs is a
separate filesystem.

-Dop



On Fri, Jul 19, 2019 at 11:07 PM Micha? Purzy?ski <
michalpurzynski1 at gmail.com> wrote:

> I tested it today and was able to build the myricom plugin without any
> problems.
>
> It was a clean system, with the fresh copy of Zeek master and a clone of
> the plugin from Seth repo.
>
> That was without using zkg mind you.
>
>
> > On Jul 19, 2019, at 7:59 PM, Jon Siwek <jsiwek at corelight.com> wrote:
> >
> >> On Fri, Jul 19, 2019 at 1:52 PM Michael Dopheide <dopheide at es.net>
> wrote:
> >>
> >> CMake is expecting to be able to copy myricom.bif.bro to it's
> destination, but only myricom.bif.zeek exists.
> >>
> >> make
> >> ...
> >> [ 11%] [BIFCL] Processing src/myricom.bif
> >> Error copying file "myricom.bif.bro" to
> "/home/zeek/.zkg/testing/zeek-myricom/clones/zeek-myricom/build/lib/bif/myricom.bif.bro".
> >
> > * The thing which produces the .bif.bro versus .bif.zeek file is
> > `bifcl`.  Seems like you've got the new version since it made a
> > .bif.zeek
> >
> > * The thing which generates a Makefile that expects to copy .bif.bro
> > versus .bif.zeek is cmake/BifCl.cmake.  Seems you may have an old
> > version of this since it created a Makefile that still expects a
> > .bif.bro
> >
> > I didn't test building the myricom plugin, but a simple plugin
> > containing a .bif doesn't seem to generally have this problem when
> > using a fully updated zeek/master: it produced and installed a
> > .bif.zeek file.
> >
> > Maybe verify you've got a fully updated and consistent Zeek build
> > and/or give the full command and build log you're running.  The
> > configuration output from the build log could have a hint that it's
> > picking up an unexpected Bro/Zeek build or install path.
> >
> > If installing via zkg, an outdated "zeek_dist" in ~/.zkg/config is a
> > problem if it's still using a version of the myricom package that uses
> > it, else if there's an old version of "bro-config" (now "zeek-config")
> > in PATH, that could be a problem.
> >
> > - Jon
> > _______________________________________________
> > zeek-dev mailing list
> > zeek-dev at zeek.org
> > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190720/54353267/attachment.html 

From robin at corelight.com  Tue Jul 23 08:09:47 2019
From: robin at corelight.com (Robin Sommer)
Date: Tue, 23 Jul 2019 08:09:47 -0700
Subject: [Zeek-Dev] Zeek 3.0.0 schedule
Message-ID: <20190723150947.GB75025@corelight.com>

Some of us met last week to discuss the timeline for Zeek 3.0.0. This
release will undertake the Big Zeekification(TM), and we?ll need to be
careful to avoid breaking existing setups wherever we can. With that
in mind, this is the current plan:

    - Code freeze on August 5. We?ll do a few days of internal testing
      on git master and then release a public beta version for broader
      testing.

    - We?re planning for four weeks of testing the beta version and
      will aim to address any issues coming up from the renaming (and
      otherwise) during that time. We?ll make further beta versions
      available during that period as needed.

    - Assuming things go well, we?ll release 3.0.0 in early September.

Zeek 3.0.0 will be our first long-term stable release. According to
the new release schedule, we?ll support it with critical fixes for a
year (see https://blog.zeek.org/2019/04/new-zeek-release-schedule.html).
The next feature release will be 3.1.0, four months after 3.0.0.

The remaining work before the 3.0.0 code freeze is tracked through
this GitHub Project: https://github.com/zeek/zeek/projects/3.

For people comfortable running git master, now would be a good time to
give the current state a try. Please let us know if you run into any
issues, in particular if something in your existing setup breaks due
to the renaming. If you believe it?s something that we can/should fix,
please file a ticket on GitHub. If you have any specific advice for
people on how to adjust aspects of their setups to Zeek 3, please send
a note to this mailing list; we?re working on a HOWTO for user-facing
implications of the renaming and will include any input we receive.

Thanks for helping us test!

Robin


-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com

From jsiwek at corelight.com  Wed Jul 24 16:29:27 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Wed, 24 Jul 2019 16:29:27 -0700
Subject: [Zeek-Dev] Zeek 3.0.0+ "master" versioning process change
Message-ID: <CAMzgZ0JF_kR6TQ73k85XuLz_tc5+qR2zGoDPDzRZ0XUha43-6w@mail.gmail.com>

I'm suggesting a small change to our usual development and versioning
process of the "master" branch: replace post-release versioning with
pre-release versioning.  E.g. once we enter the 3.0.0 beta period,
"master" will start using versions like 3.1.0-alpha.X where the old
process would have used 3.0.0-X (where X is number of commits since
the git tag).

Full description: https://github.com/zeek/zeek/issues/488

Please raise any potential issues soon if you find any.

- Jon

From johanna at icir.org  Wed Jul 24 17:54:42 2019
From: johanna at icir.org (Johanna Amann)
Date: Wed, 24 Jul 2019 17:54:42 -0700
Subject: [Zeek-Dev] Zeek 3.0.0+ "master" versioning process change
In-Reply-To: <CAMzgZ0JF_kR6TQ73k85XuLz_tc5+qR2zGoDPDzRZ0XUha43-6w@mail.gmail.com>
References: <CAMzgZ0JF_kR6TQ73k85XuLz_tc5+qR2zGoDPDzRZ0XUha43-6w@mail.gmail.com>
Message-ID: <F0C77C41-D072-4BD7-8F7A-5F4EDB51F31E@icir.org>

Hi,

I generally like this - the only thing that I am not sure about is the 
alpha label.

I get that it works great with alphabetic ordering - but for me alpha 
tends to signify some kind of test release. Could we perhaps make it 
-dev? :). If we then use -rc instead of -beta we have alphabetical 
ordering again - and to me this tends to represent what we are doing 
better in any case?

Johanna

On 24 Jul 2019, at 16:29, Jon Siwek wrote:

> I'm suggesting a small change to our usual development and versioning
> process of the "master" branch: replace post-release versioning with
> pre-release versioning.  E.g. once we enter the 3.0.0 beta period,
> "master" will start using versions like 3.1.0-alpha.X where the old
> process would have used 3.0.0-X (where X is number of commits since
> the git tag).
>
> Full description: https://github.com/zeek/zeek/issues/488
>
> Please raise any potential issues soon if you find any.
>
> - Jon
> _______________________________________________
> zeek-dev mailing list
> zeek-dev at zeek.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev

From johanna at icir.org  Wed Jul 24 18:02:19 2019
From: johanna at icir.org (Johanna Amann)
Date: Wed, 24 Jul 2019 18:02:19 -0700
Subject: [Zeek-Dev] Zeek 3.0.0+ "master" versioning process change
In-Reply-To: <F0C77C41-D072-4BD7-8F7A-5F4EDB51F31E@icir.org>
References: <CAMzgZ0JF_kR6TQ73k85XuLz_tc5+qR2zGoDPDzRZ0XUha43-6w@mail.gmail.com>
	<F0C77C41-D072-4BD7-8F7A-5F4EDB51F31E@icir.org>
Message-ID: <DCB66096-9F9D-4BC4-8528-6F7920671E9B@icir.org>

Actually, thinking about it some more - could we just not have the 
-alpha (or -dev) label, and go back to how it was before - with a 
changed meaning?

so - just 3.1.0-[commit-number] for the development builds. This would 
still work in a sem-ver style since?

3.1.0-12 < 3.1.0-beta.1 < 3.1.0

if I understand the semver standard correctly. (Numeric identifiers 
always have a lower precedence than non-numeric identifiers; it also 
works on a ascii-compare).

Johanna

On 24 Jul 2019, at 17:54, Johanna Amann wrote:

> Hi,
>
> I generally like this - the only thing that I am not sure about is the
> alpha label.
>
> I get that it works great with alphabetic ordering - but for me alpha
> tends to signify some kind of test release. Could we perhaps make it
> -dev? :). If we then use -rc instead of -beta we have alphabetical
> ordering again - and to me this tends to represent what we are doing
> better in any case?
>
> Johanna
>
> On 24 Jul 2019, at 16:29, Jon Siwek wrote:
>
>> I'm suggesting a small change to our usual development and versioning
>> process of the "master" branch: replace post-release versioning with
>> pre-release versioning.  E.g. once we enter the 3.0.0 beta period,
>> "master" will start using versions like 3.1.0-alpha.X where the old
>> process would have used 3.0.0-X (where X is number of commits since
>> the git tag).
>>
>> Full description: https://github.com/zeek/zeek/issues/488
>>
>> Please raise any potential issues soon if you find any.
>>
>> - Jon
>> _______________________________________________
>> zeek-dev mailing list
>> zeek-dev at zeek.org
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> _______________________________________________
> zeek-dev mailing list
> zeek-dev at zeek.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev

From jsiwek at corelight.com  Wed Jul 24 20:36:20 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Wed, 24 Jul 2019 20:36:20 -0700
Subject: [Zeek-Dev] Zeek 3.0.0+ "master" versioning process change
In-Reply-To: <DCB66096-9F9D-4BC4-8528-6F7920671E9B@icir.org>
References: <CAMzgZ0JF_kR6TQ73k85XuLz_tc5+qR2zGoDPDzRZ0XUha43-6w@mail.gmail.com>
	<F0C77C41-D072-4BD7-8F7A-5F4EDB51F31E@icir.org>
	<DCB66096-9F9D-4BC4-8528-6F7920671E9B@icir.org>
Message-ID: <CAMzgZ0KOKdhnUV50=zWR_wrQYYMCqRUdO4Qub=k0vgtBL=q4Ow@mail.gmail.com>

On Wed, Jul 24, 2019 at 6:02 PM Johanna Amann <johanna at icir.org> wrote:
>
> Actually, thinking about it some more - could we just not have the
> -alpha (or -dev) label, and go back to how it was before - with a
> changed meaning?
>
> so - just 3.1.0-[commit-number] for the development builds.

Our versioning script uses the last-reachable tag in "master".  At the
time we start the 3.1.0 development cycle, we don't have that 3.1.0
tag, and also that tag won't ever be made along the "master" branch,
it will be made sometime later within the "release/3.1" branch.

> > I generally like this - the only thing that I am not sure about is the
> > alpha label.
> >
> > I get that it works great with alphabetic ordering - but for me alpha
> > tends to signify some kind of test release.

What's meant by "test release" here ?

Could essentially consider any given commit in "master" to be a "test
release" -- and if we decide to be more formal/vocal about providing
builds of "master" (e.g. the OBS nightlies), then "alpha" may describe
exactly what you think it signifies ?

- Jon

From johanna at icir.org  Thu Jul 25 07:40:37 2019
From: johanna at icir.org (Johanna Amann)
Date: Thu, 25 Jul 2019 07:40:37 -0700
Subject: [Zeek-Dev] Zeek 3.0.0+ "master" versioning process change
In-Reply-To: <CAMzgZ0KOKdhnUV50=zWR_wrQYYMCqRUdO4Qub=k0vgtBL=q4Ow@mail.gmail.com>
References: <CAMzgZ0JF_kR6TQ73k85XuLz_tc5+qR2zGoDPDzRZ0XUha43-6w@mail.gmail.com>
	<F0C77C41-D072-4BD7-8F7A-5F4EDB51F31E@icir.org>
	<DCB66096-9F9D-4BC4-8528-6F7920671E9B@icir.org>
	<CAMzgZ0KOKdhnUV50=zWR_wrQYYMCqRUdO4Qub=k0vgtBL=q4Ow@mail.gmail.com>
Message-ID: <220CFB25-1462-49A7-839A-F5A67DFA12F7@icir.org>



On 24 Jul 2019, at 20:36, Jon Siwek wrote:

> On Wed, Jul 24, 2019 at 6:02 PM Johanna Amann <johanna at icir.org> 
> wrote:
>>
>> Actually, thinking about it some more - could we just not have the
>> -alpha (or -dev) label, and go back to how it was before - with a
>> changed meaning?
>>
>> so - just 3.1.0-[commit-number] for the development builds.
>
> Our versioning script uses the last-reachable tag in "master".  At the
> time we start the 3.1.0 development cycle, we don't have that 3.1.0
> tag, and also that tag won't ever be made along the "master" branch,
> it will be made sometime later within the "release/3.1" branch.

I might be slow here - but doesn?t the same problem apply to the 
proposed naming scheme?

So - you proposed master using 3.1.0-alpha.X. I was asking why we 
can?t just do 3.1.0-X instead, given that in semver numbering 
everything still stays consistent. I agree that this will need changes 
to our versioning scripts :)

>>> I generally like this - the only thing that I am not sure about is 
>>> the
>>> alpha label.
>>>
>>> I get that it works great with alphabetic ordering - but for me 
>>> alpha
>>> tends to signify some kind of test release.
>
> What's meant by "test release" here ?
>
> Could essentially consider any given commit in "master" to be a "test
> release" -- and if we decide to be more formal/vocal about providing
> builds of "master" (e.g. the OBS nightlies), then "alpha" may describe
> exactly what you think it signifies ?

True. I still like the sound of -dev and -rc better; and just not having 
a -alpha/-dev label even more - but I admit that that is a purely 
personal preference to some degree.

Another advantage of them is that they keep the version numbers somewhat 
shorter :)

Johanna

From robin at corelight.com  Thu Jul 25 08:12:17 2019
From: robin at corelight.com (Robin Sommer)
Date: Thu, 25 Jul 2019 08:12:17 -0700
Subject: [Zeek-Dev] Zeek 3.0.0+ "master" versioning process change
In-Reply-To: <CAMzgZ0KOKdhnUV50=zWR_wrQYYMCqRUdO4Qub=k0vgtBL=q4Ow@mail.gmail.com>
References: <CAMzgZ0JF_kR6TQ73k85XuLz_tc5+qR2zGoDPDzRZ0XUha43-6w@mail.gmail.com>
	<F0C77C41-D072-4BD7-8F7A-5F4EDB51F31E@icir.org>
	<DCB66096-9F9D-4BC4-8528-6F7920671E9B@icir.org>
	<CAMzgZ0KOKdhnUV50=zWR_wrQYYMCqRUdO4Qub=k0vgtBL=q4Ow@mail.gmail.com>
Message-ID: <20190725151217.GG89002@corelight.com>

Using "3.1.0-X" would also feel semantically a bit confusing I think
as we'd be changing the meaning of a scheme we're already using.

I like the idea of using "dev.X" and "rcX". I was originally feeling
similar about "alpha" but the sorting is a nice property to have.
Swtiching from "beta" to "rc" would address that.

In the end, either scheme works for me.

Robin

On Wed, Jul 24, 2019 at 20:36 -0700, Jonathan Siwek wrote:

> On Wed, Jul 24, 2019 at 6:02 PM Johanna Amann <johanna at icir.org> wrote:
> >
> > Actually, thinking about it some more - could we just not have the
> > -alpha (or -dev) label, and go back to how it was before - with a
> > changed meaning?
> >
> > so - just 3.1.0-[commit-number] for the development builds.
> 
> Our versioning script uses the last-reachable tag in "master".  At the
> time we start the 3.1.0 development cycle, we don't have that 3.1.0
> tag, and also that tag won't ever be made along the "master" branch,
> it will be made sometime later within the "release/3.1" branch.
> 
> > > I generally like this - the only thing that I am not sure about is the
> > > alpha label.
> > >
> > > I get that it works great with alphabetic ordering - but for me alpha
> > > tends to signify some kind of test release.
> 
> What's meant by "test release" here ?
> 
> Could essentially consider any given commit in "master" to be a "test
> release" -- and if we decide to be more formal/vocal about providing
> builds of "master" (e.g. the OBS nightlies), then "alpha" may describe
> exactly what you think it signifies ?
> 
> - Jon
> _______________________________________________
> zeek-dev mailing list
> zeek-dev at zeek.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev




-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com

From tim at corelight.com  Thu Jul 25 09:31:33 2019
From: tim at corelight.com (Tim Wojtulewicz)
Date: Thu, 25 Jul 2019 09:31:33 -0700
Subject: [Zeek-Dev] Zeek 3.0.0+ "master" versioning process change
In-Reply-To: <20190725151217.GG89002@corelight.com>
References: <CAMzgZ0JF_kR6TQ73k85XuLz_tc5+qR2zGoDPDzRZ0XUha43-6w@mail.gmail.com>
	<F0C77C41-D072-4BD7-8F7A-5F4EDB51F31E@icir.org>
	<DCB66096-9F9D-4BC4-8528-6F7920671E9B@icir.org>
	<CAMzgZ0KOKdhnUV50=zWR_wrQYYMCqRUdO4Qub=k0vgtBL=q4Ow@mail.gmail.com>
	<20190725151217.GG89002@corelight.com>
Message-ID: <E67914AE-6A18-41FC-8968-4BA8CE7C0916@corelight.com>

I like -dev and -rc more as well. -alpha on the master branch seems weird to me. I feel like alpha implies we actually did an official branched release, and not just commits from the main line where we?re doing development.

Tim

> On Jul 25, 2019, at 8:12 AM, Robin Sommer <robin at corelight.com> wrote:
> 
> Using "3.1.0-X" would also feel semantically a bit confusing I think
> as we'd be changing the meaning of a scheme we're already using.
> 
> I like the idea of using "dev.X" and "rcX". I was originally feeling
> similar about "alpha" but the sorting is a nice property to have.
> Swtiching from "beta" to "rc" would address that.
> 
> In the end, either scheme works for me.
> 
> Robin
> 
> On Wed, Jul 24, 2019 at 20:36 -0700, Jonathan Siwek wrote:
> 
>> On Wed, Jul 24, 2019 at 6:02 PM Johanna Amann <johanna at icir.org> wrote:
>>> 
>>> Actually, thinking about it some more - could we just not have the
>>> -alpha (or -dev) label, and go back to how it was before - with a
>>> changed meaning?
>>> 
>>> so - just 3.1.0-[commit-number] for the development builds.
>> 
>> Our versioning script uses the last-reachable tag in "master".  At the
>> time we start the 3.1.0 development cycle, we don't have that 3.1.0
>> tag, and also that tag won't ever be made along the "master" branch,
>> it will be made sometime later within the "release/3.1" branch.
>> 
>>>> I generally like this - the only thing that I am not sure about is the
>>>> alpha label.
>>>> 
>>>> I get that it works great with alphabetic ordering - but for me alpha
>>>> tends to signify some kind of test release.
>> 
>> What's meant by "test release" here ?
>> 
>> Could essentially consider any given commit in "master" to be a "test
>> release" -- and if we decide to be more formal/vocal about providing
>> builds of "master" (e.g. the OBS nightlies), then "alpha" may describe
>> exactly what you think it signifies ?
>> 
>> - Jon
>> _______________________________________________
>> zeek-dev mailing list
>> zeek-dev at zeek.org
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev
> 
> 
> 
> 
> -- 
> Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com
> _______________________________________________
> zeek-dev mailing list
> zeek-dev at zeek.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev



From jsiwek at corelight.com  Thu Jul 25 09:35:43 2019
From: jsiwek at corelight.com (Jon Siwek)
Date: Thu, 25 Jul 2019 09:35:43 -0700
Subject: [Zeek-Dev] Zeek 3.0.0+ "master" versioning process change
In-Reply-To: <220CFB25-1462-49A7-839A-F5A67DFA12F7@icir.org>
References: <CAMzgZ0JF_kR6TQ73k85XuLz_tc5+qR2zGoDPDzRZ0XUha43-6w@mail.gmail.com>
	<F0C77C41-D072-4BD7-8F7A-5F4EDB51F31E@icir.org>
	<DCB66096-9F9D-4BC4-8528-6F7920671E9B@icir.org>
	<CAMzgZ0KOKdhnUV50=zWR_wrQYYMCqRUdO4Qub=k0vgtBL=q4Ow@mail.gmail.com>
	<220CFB25-1462-49A7-839A-F5A67DFA12F7@icir.org>
Message-ID: <CAMzgZ0+o2+bxjUM0HzuZesQzdm-5i+4w+0guaT-iszMU0E8imA@mail.gmail.com>

On Thu, Jul 25, 2019 at 7:40 AM Johanna Amann <johanna at icir.org> wrote:

> > Our versioning script uses the last-reachable tag in "master".  At the
> > time we start the 3.1.0 development cycle, we don't have that 3.1.0
> > tag, and also that tag won't ever be made along the "master" branch,
> > it will be made sometime later within the "release/3.1" branch.
>
> I might be slow here - but doesn?t the same problem apply to the
> proposed naming scheme?

No, provided our versioning script keeps relying on last-reachable-tag
in master, we are free to create a 3.1.0-alpha tag in master, but we
aren't free to create a 3.1.0 tag in master.  That would mean we're
tagging a final 3.1.0 release way too early.  We could move the 3.1.0
tag later, but in the meantime I don't want to have to communicate to
people looking at the tags that it's not really an official release
yet (e.g. GitHub will automatically start listing it as a release).

> So - you proposed master using 3.1.0-alpha.X. I was asking why we
> can?t just do 3.1.0-X instead, given that in semver numbering
> everything still stays consistent. I agree that this will need changes
> to our versioning scripts :)

We *can* use 3.1.0-X to get a similar ordering property, but there's
reasons not to *want* to do that:

* We'd need a completely different versioning script/process, one that
doesn't rely on git tags.
* It's changing the meaning of X.Y.Z-[commit #] to mean "pre-release"
rather than "post-release".
* That potentially creates a bigger difference/inconsistency between
what sub-projects are doing for versioning.

> True. I still like the sound of -dev and -rc better; and just not having
> a -alpha/-dev label even more - but I admit that that is a purely
> personal preference to some degree.

My main reason for preferring alpha/beta is "it's less different than
before", otherwise don't have much argument against dev/rc.

- Jon


From robin at corelight.com  Thu Jul 25 15:37:05 2019
From: robin at corelight.com (Robin Sommer)
Date: Thu, 25 Jul 2019 15:37:05 -0700
Subject: [Zeek-Dev] Zeek 3.0.0+ "master" versioning process change
In-Reply-To: <CAMzgZ0+o2+bxjUM0HzuZesQzdm-5i+4w+0guaT-iszMU0E8imA@mail.gmail.com>
References: <CAMzgZ0JF_kR6TQ73k85XuLz_tc5+qR2zGoDPDzRZ0XUha43-6w@mail.gmail.com>
	<F0C77C41-D072-4BD7-8F7A-5F4EDB51F31E@icir.org>
	<DCB66096-9F9D-4BC4-8528-6F7920671E9B@icir.org>
	<CAMzgZ0KOKdhnUV50=zWR_wrQYYMCqRUdO4Qub=k0vgtBL=q4Ow@mail.gmail.com>
	<220CFB25-1462-49A7-839A-F5A67DFA12F7@icir.org>
	<CAMzgZ0+o2+bxjUM0HzuZesQzdm-5i+4w+0guaT-iszMU0E8imA@mail.gmail.com>
Message-ID: <20190725223705.GC92271@corelight.com>



On Thu, Jul 25, 2019 at 09:35 -0700, Jonathan Siwek wrote:

> My main reason for preferring alpha/beta is "it's less different than
> before", otherwise don't have much argument against dev/rc.

Let's just do dev/rc then, seems that's what more people prefer. And
then we'll go ahead with your scheme for 3.0.0, that should work well.

Robin

-- 
Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com

From jsbarber60 at gmail.com  Mon Jul 29 09:12:47 2019
From: jsbarber60 at gmail.com (Jeff Barber)
Date: Mon, 29 Jul 2019 10:12:47 -0600
Subject: [Zeek-Dev] PR coordination with external testing suite
Message-ID: <CABLqKG+MXGq1_3g51-nX3efeUOL+wVBiF79y-Enpi5b5cRjCXg@mail.gmail.com>

Hi Zeek devs,

I found an issue with duplicate packets not getting properly marked as TCP
retransmissions. I generated a PR for it on Friday (
https://github.com/zeek/zeek/pull/494), including the necessary changes to
make the built-in test suite pass.

However, the automatic Travis CI process that gets executed for a PR is
also downloading tests from an external repo:
https://github.com/zeek/zeek-testing and those tests are failing -- because
the fix changes the connection history values in all the connections (many
traces actually have duplicate packets in them that were previously not
noted in the history string).

The real question here is:
I don't mind updating the zeek-testing repo for the necessary changes, but
how would I coordinate that change? "this zeek PR needs to go with that
zeek-testing PR"

Jeff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190729/2f3e8db4/attachment.html 

From tim at corelight.com  Mon Jul 29 09:21:51 2019
From: tim at corelight.com (Tim Wojtulewicz)
Date: Mon, 29 Jul 2019 09:21:51 -0700
Subject: [Zeek-Dev] PR coordination with external testing suite
In-Reply-To: <CABLqKG+MXGq1_3g51-nX3efeUOL+wVBiF79y-Enpi5b5cRjCXg@mail.gmail.com>
References: <CABLqKG+MXGq1_3g51-nX3efeUOL+wVBiF79y-Enpi5b5cRjCXg@mail.gmail.com>
Message-ID: <22987894-FAC9-4916-94CD-008E25C17CA2@corelight.com>

Just make a note in both PRs that they need to be merged at the same time and link the two PRs to each other. Someone will get them both.

Tim

> On Jul 29, 2019, at 9:12 AM, Jeff Barber <jsbarber60 at gmail.com> wrote:
> 
> Hi Zeek devs,
> 
> I found an issue with duplicate packets not getting properly marked as TCP retransmissions. I generated a PR for it on Friday (https://github.com/zeek/zeek/pull/494 <https://github.com/zeek/zeek/pull/494>), including the necessary changes to make the built-in test suite pass. 
> 
> However, the automatic Travis CI process that gets executed for a PR is also downloading tests from an external repo: https://github.com/zeek/zeek-testing <https://github.com/zeek/zeek-testing> and those tests are failing -- because the fix changes the connection history values in all the connections (many traces actually have duplicate packets in them that were previously not noted in the history string).
> 
> The real question here is:
> I don't mind updating the zeek-testing repo for the necessary changes, but how would I coordinate that change? "this zeek PR needs to go with that zeek-testing PR"
> 
> Jeff
> 
> _______________________________________________
> zeek-dev mailing list
> zeek-dev at zeek.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190729/2fcec930/attachment.html