From asharma at lbl.gov  Mon Oct  3 12:11:50 2016
From: asharma at lbl.gov (Aashish Sharma)
Date: Mon, 3 Oct 2016 12:11:50 -0700
Subject: [Bro-Dev] [archive log failure]
Message-ID: <20161003191144.GB56932@mac-4.local>

I see notifications as following: 

----- Forwarded message from Xxxxxxx  -----

Date: Mon, 3 Oct 2016 11:54:39 -0700 (PDT)
From: 
To: 
Subject: [bro-cluster] archive log failure

Unable to archive one or more logs in directory:
/usr/local/bro/spool/tmp/post-terminate-worker-2016-10-03-09-40-35-36665
Check the post-terminate.out file in that directory for any error messages.
-- 
[Automatically generated.]

But then there is no spool/tmp/post-terminate-* 

However, in my /usr/local/bro/logs directory I see these folders emerge:

drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 20rk-5-9
drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 20rk-5-5
drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 20rk-5-8
drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 20rk-5-7
drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 20rk-5-10
drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 20rk-5-4
drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 20rk-5-3
drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 20rk-5-2
drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 20rk-5-1
drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 20rk-5-6
drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 20ox-5-
drwxr-xr-x    2 bro  bro   6.5k Oct  3 11:56 2016-10-03
drwxr-xr-x    2 bro  bro   512B Oct  3 12:01 20na--

Now, I do use the followign setting in broctl.cfg:

# change log naming
MakeArchiveName = /usr/local/bro-cpp/common/scripts/makelocal-archivename-2.1

However above been there forever and I don't recall these archive failure messages or these directories show up until I moved to : bro version 2.5-beta-debug

Aashish 

From dnthayer at illinois.edu  Mon Oct  3 13:02:14 2016
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Mon, 3 Oct 2016 15:02:14 -0500
Subject: [Bro-Dev] [archive log failure]
In-Reply-To: <20161003191144.GB56932@mac-4.local>
References: <20161003191144.GB56932@mac-4.local>
Message-ID: <fdab3e24-f34a-c6d6-d888-6eef98df3757@illinois.edu>

Those archive log failure emails are a new feature in version 2.5.
The only purpose of the emails is to make it easier to notice when
such an error occurs (i.e., these emails do not indicate a new type
of error condition).
Previously, if such a failure occurred, the only way you would know
is if you noticed missing logs in one of the subdirectories of
the <PREFIX>/logs/ directory, or if you noticed the presence of
a new spool/tmp/post-terminate-* directory.

As for the strange directory names, one possible reason could be your
make-archive-name script is producing bad output.



On 10/3/16 2:11 PM, Aashish Sharma wrote:
> I see notifications as following:
>
> ----- Forwarded message from Xxxxxxx  -----
>
> Date: Mon, 3 Oct 2016 11:54:39 -0700 (PDT)
> From:
> To:
> Subject: [bro-cluster] archive log failure
>
> Unable to archive one or more logs in directory:
> /usr/local/bro/spool/tmp/post-terminate-worker-2016-10-03-09-40-35-36665
> Check the post-terminate.out file in that directory for any error messages.
>

From asharma at lbl.gov  Mon Oct  3 13:18:37 2016
From: asharma at lbl.gov (Aashish Sharma)
Date: Mon, 3 Oct 2016 13:18:37 -0700
Subject: [Bro-Dev] [archive log failure]
In-Reply-To: <fdab3e24-f34a-c6d6-d888-6eef98df3757@illinois.edu>
References: <20161003191144.GB56932@mac-4.local>
	<fdab3e24-f34a-c6d6-d888-6eef98df3757@illinois.edu>
Message-ID: <20161003201830.GD56932@mac-4.local>

HI Daniel, 

> As for the strange directory names, one possible reason could be your
> make-archive-name script is producing bad output.

make-archive-name script does correctly archive the logs to 2016-10-03 folder. 

This is the contect of the script:

$ cat make-archive-name 

name=$1
flavor=$2
opened=$3
closed=$4
host=`hostname -s`

day=`echo $opened  | awk -F - '{printf "%s-%s-%s", $1, $2, $3}'`
from=`echo $opened | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'`
to=`echo $closed | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'`

if [ "$closed" != "" ]; then
   echo $day/$name.$host.$day-$from-$to
else
   echo $day/$name.$host.$day-$from-current
fi

=== 

Hereis output of  20rk-5-8 directory for example: 

~/logs/20rk-5-8]$ ls -altrh
total 40
-rw-r--r--    1 bro  bro    20B Sep 28 17:39 drop-debug.log.cluster.20rk-5-8-::-17:39:24.gz
-rw-r--r--    1 bro  bro    20B Oct  3 09:40 drop-debug.log.cluster.20rk-5-8-::-09:40:35.gz
drwxr-xr-x  196 bro  bro   6.0k Oct  3 11:54 ..
drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 .
-rw-r--r--    1 bro  bro    20B Oct  3 11:54 drop-debug.log.cluster.20rk-5-8-::-11:54:38.gz


Since make-archive-name does archive logs as expected not sure how to address 20rk-5-8 issue. secondly, why would these directories be in ~/logs instead of ../spool/tmp ? 


Aashish 

On Mon, Oct 03, 2016 at 03:02:14PM -0500, Daniel Thayer wrote:
> Those archive log failure emails are a new feature in version 2.5.
> The only purpose of the emails is to make it easier to notice when
> such an error occurs (i.e., these emails do not indicate a new type
> of error condition).
> Previously, if such a failure occurred, the only way you would know
> is if you noticed missing logs in one of the subdirectories of
> the <PREFIX>/logs/ directory, or if you noticed the presence of
> a new spool/tmp/post-terminate-* directory.
> 
> As for the strange directory names, one possible reason could be your
> make-archive-name script is producing bad output.
> 
> 
> 
> On 10/3/16 2:11 PM, Aashish Sharma wrote:
> >I see notifications as following:
> >
> >----- Forwarded message from Xxxxxxx  -----
> >
> >Date: Mon, 3 Oct 2016 11:54:39 -0700 (PDT)
> >From:
> >To:
> >Subject: [bro-cluster] archive log failure
> >
> >Unable to archive one or more logs in directory:
> >/usr/local/bro/spool/tmp/post-terminate-worker-2016-10-03-09-40-35-36665
> >Check the post-terminate.out file in that directory for any error messages.
> >

From dnthayer at illinois.edu  Mon Oct  3 13:45:54 2016
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Mon, 3 Oct 2016 15:45:54 -0500
Subject: [Bro-Dev] [archive log failure]
In-Reply-To: <20161003201830.GD56932@mac-4.local>
References: <20161003191144.GB56932@mac-4.local>
	<fdab3e24-f34a-c6d6-d888-6eef98df3757@illinois.edu>
	<20161003201830.GD56932@mac-4.local>
Message-ID: <78f4b18a-93d9-7bb5-015b-3c129a78b5ec@illinois.edu>

Your make-archive-name script works for me.

The next thing to check is your copy of
the <PREFIX>/share/broctl/scripts/archive-log and
the <PREFIX>/share/broctl/scripts/post-terminate script.
Check if you made any changes to those scripts (a bug in
those scripts could potentially run make-archive-name with
invalid parameters).


On 10/3/16 3:18 PM, Aashish Sharma wrote:
> HI Daniel,
>
>> As for the strange directory names, one possible reason could be your
>> make-archive-name script is producing bad output.
>
> make-archive-name script does correctly archive the logs to 2016-10-03 folder.
>
> This is the contect of the script:
>
> $ cat make-archive-name
>
> name=$1
> flavor=$2
> opened=$3
> closed=$4
> host=`hostname -s`
>
> day=`echo $opened  | awk -F - '{printf "%s-%s-%s", $1, $2, $3}'`
> from=`echo $opened | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'`
> to=`echo $closed | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'`
>
> if [ "$closed" != "" ]; then
>    echo $day/$name.$host.$day-$from-$to
> else
>    echo $day/$name.$host.$day-$from-current
> fi
>
> ===
>
> Hereis output of  20rk-5-8 directory for example:
>
> ~/logs/20rk-5-8]$ ls -altrh
> total 40
> -rw-r--r--    1 bro  bro    20B Sep 28 17:39 drop-debug.log.cluster.20rk-5-8-::-17:39:24.gz
> -rw-r--r--    1 bro  bro    20B Oct  3 09:40 drop-debug.log.cluster.20rk-5-8-::-09:40:35.gz
> drwxr-xr-x  196 bro  bro   6.0k Oct  3 11:54 ..
> drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 .
> -rw-r--r--    1 bro  bro    20B Oct  3 11:54 drop-debug.log.cluster.20rk-5-8-::-11:54:38.gz
>
>
> Since make-archive-name does archive logs as expected not sure how to address 20rk-5-8 issue. secondly, why would these directories be in ~/logs instead of ../spool/tmp ?
>
>
> Aashish
>
> On Mon, Oct 03, 2016 at 03:02:14PM -0500, Daniel Thayer wrote:
>> Those archive log failure emails are a new feature in version 2.5.
>> The only purpose of the emails is to make it easier to notice when
>> such an error occurs (i.e., these emails do not indicate a new type
>> of error condition).
>> Previously, if such a failure occurred, the only way you would know
>> is if you noticed missing logs in one of the subdirectories of
>> the <PREFIX>/logs/ directory, or if you noticed the presence of
>> a new spool/tmp/post-terminate-* directory.
>>
>> As for the strange directory names, one possible reason could be your
>> make-archive-name script is producing bad output.
>>
>>
>>
>> On 10/3/16 2:11 PM, Aashish Sharma wrote:
>>> I see notifications as following:
>>>
>>> ----- Forwarded message from Xxxxxxx  -----
>>>
>>> Date: Mon, 3 Oct 2016 11:54:39 -0700 (PDT)
>>> From:
>>> To:
>>> Subject: [bro-cluster] archive log failure
>>>
>>> Unable to archive one or more logs in directory:
>>> /usr/local/bro/spool/tmp/post-terminate-worker-2016-10-03-09-40-35-36665
>>> Check the post-terminate.out file in that directory for any error messages.
>>>

From asharma at lbl.gov  Mon Oct  3 14:04:30 2016
From: asharma at lbl.gov (Aashish Sharma)
Date: Mon, 3 Oct 2016 14:04:30 -0700
Subject: [Bro-Dev] [archive log failure]
In-Reply-To: <78f4b18a-93d9-7bb5-015b-3c129a78b5ec@illinois.edu>
References: <20161003191144.GB56932@mac-4.local>
	<fdab3e24-f34a-c6d6-d888-6eef98df3757@illinois.edu>
	<20161003201830.GD56932@mac-4.local>
	<78f4b18a-93d9-7bb5-015b-3c129a78b5ec@illinois.edu>
Message-ID: <20161003210426.GE56932@mac-4.local>

> the <PREFIX>/share/broctl/scripts/archive-log and
> the <PREFIX>/share/broctl/scripts/post-terminate script.

No modifications on either scripts as far as I can tell:

MD5 (share/broctl/scripts/archive-log) = 0d61804be56f8a61c18c6612bad486c8
MD5 (share/broctl/scripts/post-terminate) = ad4b56dfcfe8c1796a0a755e37256dda

Aashish 

On Mon, Oct 03, 2016 at 03:45:54PM -0500, Daniel Thayer wrote:
> Your make-archive-name script works for me.
> 
> The next thing to check is your copy of
> the <PREFIX>/share/broctl/scripts/archive-log and
> the <PREFIX>/share/broctl/scripts/post-terminate script.
> Check if you made any changes to those scripts (a bug in
> those scripts could potentially run make-archive-name with
> invalid parameters).
> 
> 
> On 10/3/16 3:18 PM, Aashish Sharma wrote:
> >HI Daniel,
> >
> >>As for the strange directory names, one possible reason could be your
> >>make-archive-name script is producing bad output.
> >
> >make-archive-name script does correctly archive the logs to 2016-10-03 folder.
> >
> >This is the contect of the script:
> >
> >$ cat make-archive-name
> >
> >name=$1
> >flavor=$2
> >opened=$3
> >closed=$4
> >host=`hostname -s`
> >
> >day=`echo $opened  | awk -F - '{printf "%s-%s-%s", $1, $2, $3}'`
> >from=`echo $opened | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'`
> >to=`echo $closed | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'`
> >
> >if [ "$closed" != "" ]; then
> >   echo $day/$name.$host.$day-$from-$to
> >else
> >   echo $day/$name.$host.$day-$from-current
> >fi
> >
> >===
> >
> >Hereis output of  20rk-5-8 directory for example:
> >
> >~/logs/20rk-5-8]$ ls -altrh
> >total 40
> >-rw-r--r--    1 bro  bro    20B Sep 28 17:39 drop-debug.log.cluster.20rk-5-8-::-17:39:24.gz
> >-rw-r--r--    1 bro  bro    20B Oct  3 09:40 drop-debug.log.cluster.20rk-5-8-::-09:40:35.gz
> >drwxr-xr-x  196 bro  bro   6.0k Oct  3 11:54 ..
> >drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 .
> >-rw-r--r--    1 bro  bro    20B Oct  3 11:54 drop-debug.log.cluster.20rk-5-8-::-11:54:38.gz
> >
> >
> >Since make-archive-name does archive logs as expected not sure how to address 20rk-5-8 issue. secondly, why would these directories be in ~/logs instead of ../spool/tmp ?
> >
> >
> >Aashish
> >
> >On Mon, Oct 03, 2016 at 03:02:14PM -0500, Daniel Thayer wrote:
> >>Those archive log failure emails are a new feature in version 2.5.
> >>The only purpose of the emails is to make it easier to notice when
> >>such an error occurs (i.e., these emails do not indicate a new type
> >>of error condition).
> >>Previously, if such a failure occurred, the only way you would know
> >>is if you noticed missing logs in one of the subdirectories of
> >>the <PREFIX>/logs/ directory, or if you noticed the presence of
> >>a new spool/tmp/post-terminate-* directory.
> >>
> >>As for the strange directory names, one possible reason could be your
> >>make-archive-name script is producing bad output.
> >>
> >>
> >>
> >>On 10/3/16 2:11 PM, Aashish Sharma wrote:
> >>>I see notifications as following:
> >>>
> >>>----- Forwarded message from Xxxxxxx  -----
> >>>
> >>>Date: Mon, 3 Oct 2016 11:54:39 -0700 (PDT)
> >>>From:
> >>>To:
> >>>Subject: [bro-cluster] archive log failure
> >>>
> >>>Unable to archive one or more logs in directory:
> >>>/usr/local/bro/spool/tmp/post-terminate-worker-2016-10-03-09-40-35-36665
> >>>Check the post-terminate.out file in that directory for any error messages.
> >>>

From dnthayer at illinois.edu  Mon Oct  3 23:06:37 2016
From: dnthayer at illinois.edu (Daniel Thayer)
Date: Tue, 4 Oct 2016 01:06:37 -0500
Subject: [Bro-Dev] [archive log failure]
In-Reply-To: <20161003210426.GE56932@mac-4.local>
References: <20161003191144.GB56932@mac-4.local>
	<fdab3e24-f34a-c6d6-d888-6eef98df3757@illinois.edu>
	<20161003201830.GD56932@mac-4.local>
	<78f4b18a-93d9-7bb5-015b-3c129a78b5ec@illinois.edu>
	<20161003210426.GE56932@mac-4.local>
Message-ID: <6a388bdc-e124-be57-a428-7af9684d8ce9@illinois.edu>

OK, so the next thing to check is if the archive-log and
make-archive-name scripts are receiving command-line parameters
that are in the expected format.  Maybe add a line like this
to those scripts:

   echo $@ >> /tmp/something.out

I'd suggest also checking if there's anything strange in
the <PREFIX>/spool/manager/stderr.log file.


On 10/3/16 4:04 PM, Aashish Sharma wrote:
>> the <PREFIX>/share/broctl/scripts/archive-log and
>> the <PREFIX>/share/broctl/scripts/post-terminate script.
>
> No modifications on either scripts as far as I can tell:
>
> MD5 (share/broctl/scripts/archive-log) = 0d61804be56f8a61c18c6612bad486c8
> MD5 (share/broctl/scripts/post-terminate) = ad4b56dfcfe8c1796a0a755e37256dda
>
> Aashish
>
> On Mon, Oct 03, 2016 at 03:45:54PM -0500, Daniel Thayer wrote:
>> Your make-archive-name script works for me.
>>
>> The next thing to check is your copy of
>> the <PREFIX>/share/broctl/scripts/archive-log and
>> the <PREFIX>/share/broctl/scripts/post-terminate script.
>> Check if you made any changes to those scripts (a bug in
>> those scripts could potentially run make-archive-name with
>> invalid parameters).
>>
>>
>> On 10/3/16 3:18 PM, Aashish Sharma wrote:
>>> HI Daniel,
>>>
>>>> As for the strange directory names, one possible reason could be your
>>>> make-archive-name script is producing bad output.
>>>
>>> make-archive-name script does correctly archive the logs to 2016-10-03 folder.
>>>
>>> This is the contect of the script:
>>>
>>> $ cat make-archive-name
>>>
>>> name=$1
>>> flavor=$2
>>> opened=$3
>>> closed=$4
>>> host=`hostname -s`
>>>
>>> day=`echo $opened  | awk -F - '{printf "%s-%s-%s", $1, $2, $3}'`
>>> from=`echo $opened | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'`
>>> to=`echo $closed | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'`
>>>
>>> if [ "$closed" != "" ]; then
>>>   echo $day/$name.$host.$day-$from-$to
>>> else
>>>   echo $day/$name.$host.$day-$from-current
>>> fi
>>>
>>> ===
>>>
>>> Hereis output of  20rk-5-8 directory for example:
>>>
>>> ~/logs/20rk-5-8]$ ls -altrh
>>> total 40
>>> -rw-r--r--    1 bro  bro    20B Sep 28 17:39 drop-debug.log.cluster.20rk-5-8-::-17:39:24.gz
>>> -rw-r--r--    1 bro  bro    20B Oct  3 09:40 drop-debug.log.cluster.20rk-5-8-::-09:40:35.gz
>>> drwxr-xr-x  196 bro  bro   6.0k Oct  3 11:54 ..
>>> drwxr-xr-x    2 bro  bro   512B Oct  3 11:54 .
>>> -rw-r--r--    1 bro  bro    20B Oct  3 11:54 drop-debug.log.cluster.20rk-5-8-::-11:54:38.gz
>>>
>>>
>>> Since make-archive-name does archive logs as expected not sure how to address 20rk-5-8 issue. secondly, why would these directories be in ~/logs instead of ../spool/tmp ?
>>>
>>>
>>> Aashish
>>>
>>> On Mon, Oct 03, 2016 at 03:02:14PM -0500, Daniel Thayer wrote:
>>>> Those archive log failure emails are a new feature in version 2.5.
>>>> The only purpose of the emails is to make it easier to notice when
>>>> such an error occurs (i.e., these emails do not indicate a new type
>>>> of error condition).
>>>> Previously, if such a failure occurred, the only way you would know
>>>> is if you noticed missing logs in one of the subdirectories of
>>>> the <PREFIX>/logs/ directory, or if you noticed the presence of
>>>> a new spool/tmp/post-terminate-* directory.
>>>>
>>>> As for the strange directory names, one possible reason could be your
>>>> make-archive-name script is producing bad output.
>>>>
>>>>
>>>>
>>>> On 10/3/16 2:11 PM, Aashish Sharma wrote:
>>>>> I see notifications as following:
>>>>>
>>>>> ----- Forwarded message from Xxxxxxx  -----
>>>>>
>>>>> Date: Mon, 3 Oct 2016 11:54:39 -0700 (PDT)
>>>>> From:
>>>>> To:
>>>>> Subject: [bro-cluster] archive log failure
>>>>>
>>>>> Unable to archive one or more logs in directory:
>>>>> /usr/local/bro/spool/tmp/post-terminate-worker-2016-10-03-09-40-35-36665
>>>>> Check the post-terminate.out file in that directory for any error messages.
>>>>>

From jan.grashoefer at gmail.com  Tue Oct  4 14:09:54 2016
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Tue, 4 Oct 2016 23:09:54 +0200
Subject: [Bro-Dev] Updating NEWS for 2.5
In-Reply-To: <A6808A24-C798-4364-B902-37940E917799@icir.org>
References: <20160808190459.GK52034@icir.org>
	<e50d086e-d306-6fe6-25b4-5902576d40d0@gmail.com>
	<20160809152120.GQ52034@icir.org>
	<60728D5A-75E2-43B9-A76B-FAFB49CA7EC2@gmail.com>
	<70d9858d-4f6e-6a70-6663-732275e145e5@gmail.com>
	<CAJ6bFK3KqrnccjKBu9SgoUW7JUT10ybU_D9WouAzD-_UjJrKSA@mail.gmail.com>
	<C54A7347-053D-4736-B528-91BDF485577C@icir.org>
	<73947624-41EA-44DF-836F-5ED43CEF7A70@gmail.com>
	<5fd75102-9d4f-fbb8-e255-d0e520317b37@gmail.com>
	<A6808A24-C798-4364-B902-37940E917799@icir.org>
Message-ID: <48dff4b0-b50f-bfd1-3ee4-c43bcea3adae@gmail.com>

Meanwhile I have added Michal's suggestions and he reviewed again:
https://gist.github.com/J-Gras/3ff4d5308a69e91fb61c65c12ecb818c
Feel free to publish :)

Jan

From slagell at illinois.edu  Tue Oct  4 14:13:03 2016
From: slagell at illinois.edu (Slagell, Adam J)
Date: Tue, 4 Oct 2016 21:13:03 +0000
Subject: [Bro-Dev] Updating NEWS for 2.5
In-Reply-To: <48dff4b0-b50f-bfd1-3ee4-c43bcea3adae@gmail.com>
References: <20160808190459.GK52034@icir.org>
	<e50d086e-d306-6fe6-25b4-5902576d40d0@gmail.com>
	<20160809152120.GQ52034@icir.org>
	<60728D5A-75E2-43B9-A76B-FAFB49CA7EC2@gmail.com>
	<70d9858d-4f6e-6a70-6663-732275e145e5@gmail.com>
	<CAJ6bFK3KqrnccjKBu9SgoUW7JUT10ybU_D9WouAzD-_UjJrKSA@mail.gmail.com>
	<C54A7347-053D-4736-B528-91BDF485577C@icir.org>
	<73947624-41EA-44DF-836F-5ED43CEF7A70@gmail.com>
	<5fd75102-9d4f-fbb8-e255-d0e520317b37@gmail.com>
	<A6808A24-C798-4364-B902-37940E917799@icir.org>
	<48dff4b0-b50f-bfd1-3ee4-c43bcea3adae@gmail.com>
Message-ID: <CFC6F247-D7E0-4886-B31D-87A104EBE4E5@illinois.edu>

Thanks. Let?s space out our blogs a little bit (We posted one yesterday on bro-pkg) and save this one for after the 2.5 release.

> On Oct 4, 2016, at 4:09 PM, Jan Grash?fer <jan.grashoefer at gmail.com> wrote:
> 
> Meanwhile I have added Michal's suggestions and he reviewed again:
> https://gist.github.com/J-Gras/3ff4d5308a69e91fb61c65c12ecb818c
> Feel free to publish :)
> 
> Jan
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

------

Adam J. Slagell
Chief Information Security Officer
Director, Cybersecurity Division
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info

"Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure."








-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161004/8bf57d13/attachment.bin 

From robin at icir.org  Fri Oct  7 07:36:50 2016
From: robin at icir.org (Robin Sommer)
Date: Fri, 7 Oct 2016 07:36:50 -0700
Subject: [Bro-Dev] Outstanding 2.5 tickets recap
Message-ID: <20161007143650.GU66878@icir.org>

We're very close. :-) I wanted to summarize the remaining tickets with
some notes/questions to the people who I believe have the token there
right now:

- SMB file not used?, https://bro-tracker.atlassian.net/browse/BIT-1721
  Seth/Vlad: I noticed yesterday that an SMB .bif doesn't seem to be
  used?

- Log file list is out of date, https://bro-tracker.atlassian.net/browse/BIT-1720
  Could somebody take this one on: what are the new log files in 2.5,
  we need to add them to the overview in the docs.

- ssh_auth_failed raised multiple times in single connection, https://bro-tracker.atlassian.net/browse/BIT-1641
  Vlad, how far are you implementing these changes?

- missing uid field in SMB1 script, https://bro-tracker.atlassian.net/browse/BIT-1688?filter=10001
  Seth?

- missing certain logs if logger node is enabled, https://bro-tracker.atlassian.net/browse/BIT-1700
  Seth, does adding that 3rd option for now looks like a viable
  compromise?

- Very long "pipe_name" in dce_rpc.log, https://bro-tracker.atlassian.net/browse/BIT-1702
  Seth/Vlad, sounds like the current suspicion is that this is trouble with fragmentation?

- Python 3.5 compatibility in broccoli-python, https://bro-tracker.atlassian.net/browse/BIT-1711?filter=10001
  Seth, this is just waiting on more feedback on the problem I think

- Py3 compatibility, https://bro-tracker.atlassian.net/browse/BIT-1719
  Daniel, any idea yet what's going on here? Could it be btest-related
  (given that core tests seem to be failing, which shouldn't have a
  Python dependency otherwise?)


Robin


-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From johanna at icir.org  Fri Oct  7 14:06:53 2016
From: johanna at icir.org (Johanna Amann)
Date: Fri, 7 Oct 2016 14:06:53 -0700
Subject: [Bro-Dev] Potential of including TLSv1.3 support in Bro 2.5
Message-ID: <20161007210653.cpseh2iva3svkydn@wifi203.sys.ICSI.Berkeley.EDU>

I just finished a branch that adds support for TLSv1.3 to Bro (branch
topic/johanna/tls13, important commit:
https://github.com/bro/bro/commit/fdef28ce7c3455d43267ab07dbb8ad96c9ea3890).

What do people think of the idea of adding that patch to the upcoming Bro
2.5 release?

I know that we are quite late in the current release process and that we
should not really make any feature changes after releasing the beta.  It
would, however, be neat to be able to support TLSv1.3 starting the moment
that people actually start to use it; without that support, we will only
have empty lines in ssl.log for these connections. Furthermore, the
changes that are needed to support TLSv1.3 have nearly no interaction with
the code that is used to parse earlier versions of TLS. Even if there are
problems with the code (or if the on-the-wire format still changes), the
only thing that should happen is that binpac throws errors. Which is
exactly what already happens now when throwing TLSv1.3 sessions at the
current master versions of Bro.

Thanks,
 Johanna

From jan.grashoefer at gmail.com  Fri Oct  7 14:54:33 2016
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Fri, 7 Oct 2016 23:54:33 +0200
Subject: [Bro-Dev] Potential of including TLSv1.3 support in Bro 2.5
In-Reply-To: <20161007210653.cpseh2iva3svkydn@wifi203.sys.ICSI.Berkeley.EDU>
References: <20161007210653.cpseh2iva3svkydn@wifi203.sys.ICSI.Berkeley.EDU>
Message-ID: <80a63c29-e063-09fe-8c21-2316b63a9e7c@gmail.com>

> What do people think of the idea of adding that patch to the upcoming Bro
> 2.5 release?

I like the idea! +1

Jan

From asharma at lbl.gov  Fri Oct  7 17:03:43 2016
From: asharma at lbl.gov (Aashish Sharma)
Date: Fri, 7 Oct 2016 17:03:43 -0700
Subject: [Bro-Dev] Potential of including TLSv1.3 support in Bro 2.5
In-Reply-To: <20161007210653.cpseh2iva3svkydn@wifi203.sys.ICSI.Berkeley.EDU>
References: <20161007210653.cpseh2iva3svkydn@wifi203.sys.ICSI.Berkeley.EDU>
Message-ID: <20161008000340.GI75590@mac-4.local>

I think the current feature freeze is a self-imposed limit out of coding discipline - but it ok to make exceptions.  Esp since 2.6 would be long way away. 

Risky as it is, It seems like inclusion of this code isn't going to cause any significant problems. FWIW, I can run this branch on my end for until release happens. 

Aashish 

On Fri, Oct 07, 2016 at 02:06:53PM -0700, Johanna Amann wrote:
> I just finished a branch that adds support for TLSv1.3 to Bro (branch
> topic/johanna/tls13, important commit:
> https://github.com/bro/bro/commit/fdef28ce7c3455d43267ab07dbb8ad96c9ea3890).
> 
> What do people think of the idea of adding that patch to the upcoming Bro
> 2.5 release?
> 
> I know that we are quite late in the current release process and that we
> should not really make any feature changes after releasing the beta.  It
> would, however, be neat to be able to support TLSv1.3 starting the moment
> that people actually start to use it; without that support, we will only
> have empty lines in ssl.log for these connections. Furthermore, the
> changes that are needed to support TLSv1.3 have nearly no interaction with
> the code that is used to parse earlier versions of TLS. Even if there are
> problems with the code (or if the on-the-wire format still changes), the
> only thing that should happen is that binpac throws errors. Which is
> exactly what already happens now when throwing TLSv1.3 sessions at the
> current master versions of Bro.
> 
> Thanks,
>  Johanna
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

From johanna at icir.org  Fri Oct  7 17:18:06 2016
From: johanna at icir.org (Johanna Amann)
Date: Fri, 07 Oct 2016 17:18:06 -0700
Subject: [Bro-Dev] Potential of including TLSv1.3 support in Bro 2.5
In-Reply-To: <20161008000340.GI75590@mac-4.local>
References: <20161007210653.cpseh2iva3svkydn@wifi203.sys.ICSI.Berkeley.EDU>
	<20161008000340.GI75590@mac-4.local>
Message-ID: <D7A295AB-B45D-4AFE-B1AD-C6D4E5AC2DCB@icir.org>

I would be happy if you test this branch - however, you are actually 
unlikely to trigger the new code paths. TLS 1.3 is still in the 
development stage, so much that I doubt that you will even encounter a 
single connection that uses it. At the moment, you have to enable it by 
hand in the development edition of browsers, and more or less compile 
your own server that is able to speak it.

(That being said, I am quite confident the on-the-wire format won't 
change significantly enough anymore that the new analyzer won't be able 
to parse it.)

Johanna

On 7 Oct 2016, at 17:03, Aashish Sharma wrote:

> I think the current feature freeze is a self-imposed limit out of 
> coding discipline - but it ok to make exceptions.  Esp since 2.6 would 
> be long way away.
>
> Risky as it is, It seems like inclusion of this code isn't going to 
> cause any significant problems. FWIW, I can run this branch on my end 
> for until release happens.
>
> Aashish
>
> On Fri, Oct 07, 2016 at 02:06:53PM -0700, Johanna Amann wrote:
>> I just finished a branch that adds support for TLSv1.3 to Bro (branch
>> topic/johanna/tls13, important commit:
>> https://github.com/bro/bro/commit/fdef28ce7c3455d43267ab07dbb8ad96c9ea3890).
>>
>> What do people think of the idea of adding that patch to the upcoming 
>> Bro
>> 2.5 release?
>>
>> I know that we are quite late in the current release process and that 
>> we
>> should not really make any feature changes after releasing the beta.  
>> It
>> would, however, be neat to be able to support TLSv1.3 starting the 
>> moment
>> that people actually start to use it; without that support, we will 
>> only
>> have empty lines in ssl.log for these connections. Furthermore, the
>> changes that are needed to support TLSv1.3 have nearly no interaction 
>> with
>> the code that is used to parse earlier versions of TLS. Even if there 
>> are
>> problems with the code (or if the on-the-wire format still changes), 
>> the
>> only thing that should happen is that binpac throws errors. Which is
>> exactly what already happens now when throwing TLSv1.3 sessions at 
>> the
>> current master versions of Bro.
>>
>> Thanks,
>>  Johanna
>> _______________________________________________
>> bro-dev mailing list
>> bro-dev at bro.org
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>

From seth at icir.org  Fri Oct  7 18:43:26 2016
From: seth at icir.org (Seth Hall)
Date: Fri, 7 Oct 2016 21:43:26 -0400
Subject: [Bro-Dev] Outstanding 2.5 tickets recap
In-Reply-To: <20161007143650.GU66878@icir.org>
References: <20161007143650.GU66878@icir.org>
Message-ID: <1F57A41D-63A3-40BA-8EC8-42E1B598A729@icir.org>


> On Oct 7, 2016, at 10:36 AM, Robin Sommer <robin at icir.org> wrote:
> 
> - SMB file not used?, https://bro-tracker.atlassian.net/browse/BIT-1721
>  Seth/Vlad: I noticed yesterday that an SMB .bif doesn't seem to be
>  used?

Vlad said he tested compiling that in and no tests were affected.  I'll test it with my private test suite as well.

> - missing uid field in SMB1 script, https://bro-tracker.atlassian.net/browse/BIT-1688?filter=10001
>  Seth?

Sorry, I think know the fix, I just need to get to it.

> - missing certain logs if logger node is enabled, https://bro-tracker.atlassian.net/browse/BIT-1700
>  Seth, does adding that 3rd option for now looks like a viable
>  compromise?

Yep, that's fine, but I left a note on the ticket with some more thoughts.

> - Very long "pipe_name" in dce_rpc.log, https://bro-tracker.atlassian.net/browse/BIT-1702
>  Seth/Vlad, sounds like the current suspicion is that this is trouble with fragmentation?

Yeah, it's that plus an actual parser error.  I have a sample and I'm working on it now.

> - Python 3.5 compatibility in broccoli-python, https://bro-tracker.atlassian.net/browse/BIT-1711?filter=10001
>  Seth, this is just waiting on more feedback on the problem I think

Left a note.

I'll be working on stuff this weekend... again. :)

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From vlad at grigorescu.org  Sat Oct  8 17:38:39 2016
From: vlad at grigorescu.org (Vlad Grigorescu)
Date: Sat, 8 Oct 2016 19:38:39 -0500
Subject: [Bro-Dev] Potential of including TLSv1.3 support in Bro 2.5
In-Reply-To: <D7A295AB-B45D-4AFE-B1AD-C6D4E5AC2DCB@icir.org>
References: <20161007210653.cpseh2iva3svkydn@wifi203.sys.ICSI.Berkeley.EDU>
	<20161008000340.GI75590@mac-4.local>
	<D7A295AB-B45D-4AFE-B1AD-C6D4E5AC2DCB@icir.org>
Message-ID: <CANUJ_F=-UY7e-Mhvpnar0WE0p52HsV-sN_inwdaHTkY3uDYedw@mail.gmail.com>

Well, I should point out that Cloudflare enabled it a couple of weeks ago:
https://blog.cloudflare.com/introducing-tls-1-3/

I was able to connect with my usual browser and grab a PCAP (after setting
the option in about:config). It seems to run just fine against the branch
(attached, in case it's of any use).

Is there any way to detect TLS 1.3 with git master? I wouldn't expect to
see any, but I've been surprised once or twice before. I ran the PCAP
against master, and while I did get an ssl.log, I didn't see anything in
there that would indicate it's TLS1.3.

  --Vlad

On Fri, Oct 7, 2016 at 7:18 PM, Johanna Amann <johanna at icir.org> wrote:

> I would be happy if you test this branch - however, you are actually
> unlikely to trigger the new code paths. TLS 1.3 is still in the
> development stage, so much that I doubt that you will even encounter a
> single connection that uses it. At the moment, you have to enable it by
> hand in the development edition of browsers, and more or less compile
> your own server that is able to speak it.
>
> (That being said, I am quite confident the on-the-wire format won't
> change significantly enough anymore that the new analyzer won't be able
> to parse it.)
>
> Johanna
>
> On 7 Oct 2016, at 17:03, Aashish Sharma wrote:
>
> > I think the current feature freeze is a self-imposed limit out of
> > coding discipline - but it ok to make exceptions.  Esp since 2.6 would
> > be long way away.
> >
> > Risky as it is, It seems like inclusion of this code isn't going to
> > cause any significant problems. FWIW, I can run this branch on my end
> > for until release happens.
> >
> > Aashish
> >
> > On Fri, Oct 07, 2016 at 02:06:53PM -0700, Johanna Amann wrote:
> >> I just finished a branch that adds support for TLSv1.3 to Bro (branch
> >> topic/johanna/tls13, important commit:
> >> https://github.com/bro/bro/commit/fdef28ce7c3455d43267ab07dbb8ad
> 96c9ea3890).
> >>
> >> What do people think of the idea of adding that patch to the upcoming
> >> Bro
> >> 2.5 release?
> >>
> >> I know that we are quite late in the current release process and that
> >> we
> >> should not really make any feature changes after releasing the beta.
> >> It
> >> would, however, be neat to be able to support TLSv1.3 starting the
> >> moment
> >> that people actually start to use it; without that support, we will
> >> only
> >> have empty lines in ssl.log for these connections. Furthermore, the
> >> changes that are needed to support TLSv1.3 have nearly no interaction
> >> with
> >> the code that is used to parse earlier versions of TLS. Even if there
> >> are
> >> problems with the code (or if the on-the-wire format still changes),
> >> the
> >> only thing that should happen is that binpac throws errors. Which is
> >> exactly what already happens now when throwing TLSv1.3 sessions at
> >> the
> >> current master versions of Bro.
> >>
> >> Thanks,
> >>  Johanna
> >> _______________________________________________
> >> bro-dev mailing list
> >> bro-dev at bro.org
> >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> >
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161008/b9d08f22/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tls13draft16-firefoxdevedition51.0a2-cloudflare.pcap
Type: application/octet-stream
Size: 119578 bytes
Desc: not available
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161008/b9d08f22/attachment-0001.obj 

From robin at icir.org  Mon Oct 10 08:59:47 2016
From: robin at icir.org (Robin Sommer)
Date: Mon, 10 Oct 2016 08:59:47 -0700
Subject: [Bro-Dev] Potential of including TLSv1.3 support in Bro 2.5
In-Reply-To: <20161008000340.GI75590@mac-4.local>
References: <20161007210653.cpseh2iva3svkydn@wifi203.sys.ICSI.Berkeley.EDU>
	<20161008000340.GI75590@mac-4.local>
Message-ID: <20161010155947.GD39716@icir.org>



On Fri, Oct 07, 2016 at 17:03 -0700, you wrote:

> Esp since 2.6 would be long way away.

Yeah, indeed. I just looked over the changes, they look fine and
rather unobtrusive to me (not that I would know TLS 1.3 :). I'd be
fine with merging as well, given that we'll need another round of
testing anyways with the SMB updates, so the TLS code will see some
more exercising (for regressions at least) then as well. Unless I hear
objections, I'll go ahead and merge.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From johanna at icir.org  Mon Oct 10 20:26:34 2016
From: johanna at icir.org (Johanna Amann)
Date: Mon, 10 Oct 2016 20:26:34 -0700
Subject: [Bro-Dev] Potential of including TLSv1.3 support in Bro 2.5
In-Reply-To: <CANUJ_F=-UY7e-Mhvpnar0WE0p52HsV-sN_inwdaHTkY3uDYedw@mail.gmail.com>
References: <20161007210653.cpseh2iva3svkydn@wifi203.sys.ICSI.Berkeley.EDU>
	<20161008000340.GI75590@mac-4.local>
	<D7A295AB-B45D-4AFE-B1AD-C6D4E5AC2DCB@icir.org>
	<CANUJ_F=-UY7e-Mhvpnar0WE0p52HsV-sN_inwdaHTkY3uDYedw@mail.gmail.com>
Message-ID: <CDC6265A-F75A-4BEB-91E9-BA1CCF54B213@icir.org>



On 8 Oct 2016, at 17:38, Vlad Grigorescu wrote:

> Well, I should point out that Cloudflare enabled it a couple of weeks 
> ago:
> https://blog.cloudflare.com/introducing-tls-1-3/

You actually got that to run? I did not manage to get any client to 
successfully negotiate TLS 1.3 with them and set up my own server in the 
end. But perhaps they updated in the last few days...

> I was able to connect with my usual browser and grab a PCAP (after 
> setting
> the option in about:config). It seems to run just fine against the 
> branch
> (attached, in case it's of any use).
>
> Is there any way to detect TLS 1.3 with git master? I wouldn't expect 
> to
> see any, but I've been surprised once or twice before. I ran the PCAP
> against master, and while I did get an ssl.log, I didn't see anything 
> in
> there that would indicate it's TLS1.3.

Well, it will show up as a binpac error while parsing a specific TLS 
message. Not the best way to do it ;)

Johanna

From johanna at icir.org  Thu Oct 13 14:52:21 2016
From: johanna at icir.org (Johanna Amann)
Date: Thu, 13 Oct 2016 14:52:21 -0700
Subject: [Bro-Dev] Potential of including TLSv1.3 support in Bro 2.5
In-Reply-To: <20161007210653.cpseh2iva3svkydn@wifi203.sys.ICSI.Berkeley.EDU>
References: <20161007210653.cpseh2iva3svkydn@wifi203.sys.ICSI.Berkeley.EDU>
Message-ID: <20161013215221.va5natunm3eyi3mc@wifi155.sys.ICSI.Berkeley.EDU>

As a follow-up: since all responses were positives, I filed a
merge-request for this and it should (hopefully) make it into 2.5.

Merge-request for those who want to follow it:
https://bro-tracker.atlassian.net/browse/BIT-1727

Johanna

On Fri, Oct 07, 2016 at 02:06:53PM -0700, Johanna Amann wrote:
> I just finished a branch that adds support for TLSv1.3 to Bro (branch
> topic/johanna/tls13, important commit:
> https://github.com/bro/bro/commit/fdef28ce7c3455d43267ab07dbb8ad96c9ea3890).
> 
> What do people think of the idea of adding that patch to the upcoming Bro
> 2.5 release?
> 
> I know that we are quite late in the current release process and that we
> should not really make any feature changes after releasing the beta.  It
> would, however, be neat to be able to support TLSv1.3 starting the moment
> that people actually start to use it; without that support, we will only
> have empty lines in ssl.log for these connections. Furthermore, the
> changes that are needed to support TLSv1.3 have nearly no interaction with
> the code that is used to parse earlier versions of TLS. Even if there are
> problems with the code (or if the on-the-wire format still changes), the
> only thing that should happen is that binpac throws errors. Which is
> exactly what already happens now when throwing TLSv1.3 sessions at the
> current master versions of Bro.
> 
> Thanks,
>  Johanna
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 

From hlin33 at illinois.edu  Mon Oct 17 12:24:09 2016
From: hlin33 at illinois.edu (Hui Lin (Hugo) )
Date: Mon, 17 Oct 2016 14:24:09 -0500
Subject: [Bro-Dev] OpenFlow Analyzer
Message-ID: <CAKq214ks818odONErNYm7+EQR5+--ZpevWAzfme4-kR9Wh8xqg@mail.gmail.com>

Hi

My understanding is that Bro has some northbound API to openflow switches
or controllers. I am wondering whether any development branch has analyzer
of openflow controllers. Just try to see whether I can use Bro to analyze
some controller-to-switch traffics.

Best,

Hui


-- 
Hui Lin
Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
DEPEND (http://depend.csl.illinois.edu/)
ECE, Uni. of Illinois at Urbana-Champaign
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161017/331090fb/attachment.html 

From slagell at illinois.edu  Mon Oct 17 12:37:22 2016
From: slagell at illinois.edu (Slagell, Adam J)
Date: Mon, 17 Oct 2016 19:37:22 +0000
Subject: [Bro-Dev] OpenFlow Analyzer
In-Reply-To: <CAKq214ks818odONErNYm7+EQR5+--ZpevWAzfme4-kR9Wh8xqg@mail.gmail.com>
References: <CAKq214ks818odONErNYm7+EQR5+--ZpevWAzfme4-kR9Wh8xqg@mail.gmail.com>
Message-ID: <265593D5-1457-44B3-A05D-CFA0F28FA9F8@illinois.edu>

Have you looked at the netcontrol framework in Bro, developed by Johanna?

> On Oct 17, 2016, at 2:24 PM, Hui Lin (Hugo) <hlin33 at illinois.edu> wrote:
> 
> Hi 
> 
> My understanding is that Bro has some northbound API to openflow switches or controllers. I am wondering whether any development branch has analyzer of openflow controllers. Just try to see whether I can use Bro to analyze some controller-to-switch traffics. 
> 
> Best,
> 
> Hui
> 
> 
> -- 
> Hui Lin
> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/ <http://hlin33.web.engr.illinois.edu/>) 
> DEPEND (http://depend.csl.illinois.edu/ <http://depend.csl.illinois.edu/>)
> ECE, Uni. of Illinois at Urbana-Champaign 
> 
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

------

Adam J. Slagell
Chief Information Security Officer
Director, Cybersecurity Division
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info

"Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 








-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161017/e6664a79/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3579 bytes
Desc: not available
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161017/e6664a79/attachment.bin 

From hlin33 at illinois.edu  Mon Oct 17 12:45:30 2016
From: hlin33 at illinois.edu (Hui Lin (Hugo) )
Date: Mon, 17 Oct 2016 14:45:30 -0500
Subject: [Bro-Dev] OpenFlow Analyzer
In-Reply-To: <4d5f0c63ca30482ba5b510909e8fea6e@CITESHT4.ad.uillinois.edu>
References: <CAKq214ks818odONErNYm7+EQR5+--ZpevWAzfme4-kR9Wh8xqg@mail.gmail.com>
	<4d5f0c63ca30482ba5b510909e8fea6e@CITESHT4.ad.uillinois.edu>
Message-ID: <CAKq214nPBW1vqK1UnF2Dn5Mgp53mUNCKMKE9aUWVXV=7UktF0w@mail.gmail.com>

Actually, netcontrol is what I thought of as northbound APIs. I actually
just wonder whether Bro has analyzer for openflow protocol or not (some
refer them as southbound traffics). It not, I probably need to use
wireshark to output the pcap and analyze them manually.

On Mon, Oct 17, 2016 at 2:37 PM, Slagell, Adam J <slagell at illinois.edu>
wrote:

> Have you looked at the netcontrol framework in Bro, developed by Johanna?
>
> On Oct 17, 2016, at 2:24 PM, Hui Lin (Hugo) <hlin33 at illinois.edu> wrote:
>
> Hi
>
> My understanding is that Bro has some northbound API to openflow switches
> or controllers. I am wondering whether any development branch has analyzer
> of openflow controllers. Just try to see whether I can use Bro to analyze
> some controller-to-switch traffics.
>
> Best,
>
> Hui
>
>
> --
> Hui Lin
> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
> DEPEND (http://depend.csl.illinois.edu/)
> ECE, Uni. of Illinois at Urbana-Champaign
>
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
>
> ------
>
> Adam J. Slagell
> Chief Information Security Officer
> Director, Cybersecurity Division
> National Center for Supercomputing Applications
> University of Illinois at Urbana-Champaign
> www.slagell.info
>
> "Under the Illinois Freedom of Information Act (FOIA), any written
> communication to or from University employees regarding University business
> is a public record and may be subject to public disclosure."
>
>
>
>
>
>
>
>
>


-- 
Hui Lin
Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/)
DEPEND (http://depend.csl.illinois.edu/)
ECE, Uni. of Illinois at Urbana-Champaign
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161017/3a688a14/attachment.html 

From slagell at illinois.edu  Mon Oct 17 12:47:54 2016
From: slagell at illinois.edu (Slagell, Adam J)
Date: Mon, 17 Oct 2016 19:47:54 +0000
Subject: [Bro-Dev] OpenFlow Analyzer
In-Reply-To: <CAKq214nPBW1vqK1UnF2Dn5Mgp53mUNCKMKE9aUWVXV=7UktF0w@mail.gmail.com>
References: <CAKq214ks818odONErNYm7+EQR5+--ZpevWAzfme4-kR9Wh8xqg@mail.gmail.com>
	<4d5f0c63ca30482ba5b510909e8fea6e@CITESHT4.ad.uillinois.edu>
	<CAKq214nPBW1vqK1UnF2Dn5Mgp53mUNCKMKE9aUWVXV=7UktF0w@mail.gmail.com>
Message-ID: <AF8F70E6-7A11-48C0-B217-5A04701947C0@illinois.edu>

I get you now. I am not aware of an open flow protocol analyzer in Bro.

> On Oct 17, 2016, at 2:45 PM, Hui Lin (Hugo) <hlin33 at illinois.edu> wrote:
> 
> Actually, netcontrol is what I thought of as northbound APIs. I actually just wonder whether Bro has analyzer for openflow protocol or not (some refer them as southbound traffics). It not, I probably need to use wireshark to output the pcap and analyze them manually. 
> 
> On Mon, Oct 17, 2016 at 2:37 PM, Slagell, Adam J <slagell at illinois.edu <mailto:slagell at illinois.edu>> wrote:
> Have you looked at the netcontrol framework in Bro, developed by Johanna?
> 
>> On Oct 17, 2016, at 2:24 PM, Hui Lin (Hugo) <hlin33 at illinois.edu <mailto:hlin33 at illinois.edu>> wrote:
>> 
>> Hi 
>> 
>> My understanding is that Bro has some northbound API to openflow switches or controllers. I am wondering whether any development branch has analyzer of openflow controllers. Just try to see whether I can use Bro to analyze some controller-to-switch traffics. 
>> 
>> Best,
>> 
>> Hui
>> 
>> 
>> -- 
>> Hui Lin
>> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/ <http://hlin33.web.engr.illinois.edu/>) 
>> DEPEND (http://depend.csl.illinois.edu/ <http://depend.csl.illinois.edu/>)
>> ECE, Uni. of Illinois at Urbana-Champaign 
>> 
>> _______________________________________________
>> bro-dev mailing list
>> bro-dev at bro.org <mailto:bro-dev at bro.org>
>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev <https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.icsi.berkeley.edu_mailman_listinfo_bro-2Ddev&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=gMEsgy9kNQo7aTfyIJsOSuw4Z57hfQyz6uV2H4S9PvE&m=6uypPBIs5nv0peW6XK_o8f3Tu4OHNbnBILH4E2KzmI0&s=S1oVBNplwXmgrFjCjOVpo2f7jqaRkA83lof6A3C0K3c&e=>
> 
> ------
> 
> Adam J. Slagell
> Chief Information Security Officer
> Director, Cybersecurity Division
> National Center for Supercomputing Applications
> University of Illinois at Urbana-Champaign
> www.slagell.info <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slagell.info&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=gMEsgy9kNQo7aTfyIJsOSuw4Z57hfQyz6uV2H4S9PvE&m=6uypPBIs5nv0peW6XK_o8f3Tu4OHNbnBILH4E2KzmI0&s=si3U_XSgz2LTo9UrJ_nkxfh3RIFE1cVuMrTHK3PX6Sk&e=>
> 
> "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> Hui Lin
> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/ <http://hlin33.web.engr.illinois.edu/>) 
> DEPEND (http://depend.csl.illinois.edu/ <http://depend.csl.illinois.edu/>)
> ECE, Uni. of Illinois at Urbana-Champaign 
> 

------

Adam J. Slagell
Chief Information Security Officer
Director, Cybersecurity Division
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info

"Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." 








-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161017/0b118ff0/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3579 bytes
Desc: not available
Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20161017/0b118ff0/attachment-0001.bin 

From johanna at icir.org  Tue Oct 18 12:59:20 2016
From: johanna at icir.org (Johanna Amann)
Date: Tue, 18 Oct 2016 15:59:20 -0400
Subject: [Bro-Dev] OpenFlow Analyzer
In-Reply-To: <AF8F70E6-7A11-48C0-B217-5A04701947C0@illinois.edu>
References: <CAKq214ks818odONErNYm7+EQR5+--ZpevWAzfme4-kR9Wh8xqg@mail.gmail.com>
	<4d5f0c63ca30482ba5b510909e8fea6e@CITESHT4.ad.uillinois.edu>
	<CAKq214nPBW1vqK1UnF2Dn5Mgp53mUNCKMKE9aUWVXV=7UktF0w@mail.gmail.com>
	<AF8F70E6-7A11-48C0-B217-5A04701947C0@illinois.edu>
Message-ID: <519BAF4B-2143-4DA2-A19C-7560FB14DA63@icir.org>

Just to add to this - there is no analyzer and so far this is also not 
planned. In addition to that - OpenFlow encourages use of TLS, so you 
(hopefully) should not actually see a lot of unencrypted OF traffic on 
the wire.

Johanna

On 17 Oct 2016, at 15:47, Slagell, Adam J wrote:

> I get you now. I am not aware of an open flow protocol analyzer in 
> Bro.
>
>> On Oct 17, 2016, at 2:45 PM, Hui Lin (Hugo) <hlin33 at illinois.edu> 
>> wrote:
>>
>> Actually, netcontrol is what I thought of as northbound APIs. I 
>> actually just wonder whether Bro has analyzer for openflow protocol 
>> or not (some refer them as southbound traffics). It not, I probably 
>> need to use wireshark to output the pcap and analyze them manually.
>>
>> On Mon, Oct 17, 2016 at 2:37 PM, Slagell, Adam J 
>> <slagell at illinois.edu <mailto:slagell at illinois.edu>> wrote:
>> Have you looked at the netcontrol framework in Bro, developed by 
>> Johanna?
>>
>>> On Oct 17, 2016, at 2:24 PM, Hui Lin (Hugo) <hlin33 at illinois.edu 
>>> <mailto:hlin33 at illinois.edu>> wrote:
>>>
>>> Hi
>>>
>>> My understanding is that Bro has some northbound API to openflow 
>>> switches or controllers. I am wondering whether any development 
>>> branch has analyzer of openflow controllers. Just try to see whether 
>>> I can use Bro to analyze some controller-to-switch traffics.
>>>
>>> Best,
>>>
>>> Hui
>>>
>>>
>>> -- 
>>> Hui Lin
>>> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/ 
>>> <http://hlin33.web.engr.illinois.edu/>)
>>> DEPEND (http://depend.csl.illinois.edu/ 
>>> <http://depend.csl.illinois.edu/>)
>>> ECE, Uni. of Illinois at Urbana-Champaign
>>>
>>> _______________________________________________
>>> bro-dev mailing list
>>> bro-dev at bro.org <mailto:bro-dev at bro.org>
>>> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev 
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__mailman.icsi.berkeley.edu_mailman_listinfo_bro-2Ddev&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=gMEsgy9kNQo7aTfyIJsOSuw4Z57hfQyz6uV2H4S9PvE&m=6uypPBIs5nv0peW6XK_o8f3Tu4OHNbnBILH4E2KzmI0&s=S1oVBNplwXmgrFjCjOVpo2f7jqaRkA83lof6A3C0K3c&e=>
>>
>> ------
>>
>> Adam J. Slagell
>> Chief Information Security Officer
>> Director, Cybersecurity Division
>> National Center for Supercomputing Applications
>> University of Illinois at Urbana-Champaign
>> www.slagell.info 
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slagell.info&d=DQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=gMEsgy9kNQo7aTfyIJsOSuw4Z57hfQyz6uV2H4S9PvE&m=6uypPBIs5nv0peW6XK_o8f3Tu4OHNbnBILH4E2KzmI0&s=si3U_XSgz2LTo9UrJ_nkxfh3RIFE1cVuMrTHK3PX6Sk&e=>
>>
>> "Under the Illinois Freedom of Information Act (FOIA), any written 
>> communication to or from University employees regarding University 
>> business is a public record and may be subject to public disclosure."
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> -- 
>> Hui Lin
>> Ph.D. Candidate (http://hlin33.web.engr.illinois.edu/ 
>> <http://hlin33.web.engr.illinois.edu/>)
>> DEPEND (http://depend.csl.illinois.edu/ 
>> <http://depend.csl.illinois.edu/>)
>> ECE, Uni. of Illinois at Urbana-Champaign
>>
>
> ------
>
> Adam J. Slagell
> Chief Information Security Officer
> Director, Cybersecurity Division
> National Center for Supercomputing Applications
> University of Illinois at Urbana-Champaign
> www.slagell.info
>
> "Under the Illinois Freedom of Information Act (FOIA), any written 
> communication to or from University employees regarding University 
> business is a public record and may be subject to public disclosure."
>
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev


From jsiwek at illinois.edu  Mon Oct 24 07:22:13 2016
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Mon, 24 Oct 2016 14:22:13 +0000
Subject: [Bro-Dev] bro-pkg support for centralized package structures?
Message-ID: <7565EBB7-CA12-4044-84C6-D70B2F66DD89@illinois.edu>

An argument was made in a GitHub issue [1] that the decentralized model for packages where each package is a distinct git repo may be inconvenient for some use-cases and that allowing multiple packages to reside in single git repo would be better.

I think supporting that would significantly change the design and goals of the package manager, so I wonder what ideas/thoughts others have about the current model vs. the proposed one?

- Jon

[1] https://github.com/bro/package-manager/issues/2

From seth at icir.org  Mon Oct 24 07:29:36 2016
From: seth at icir.org (Seth Hall)
Date: Mon, 24 Oct 2016 10:29:36 -0400
Subject: [Bro-Dev] bro-pkg support for centralized package structures?
In-Reply-To: <7565EBB7-CA12-4044-84C6-D70B2F66DD89@illinois.edu>
References: <7565EBB7-CA12-4044-84C6-D70B2F66DD89@illinois.edu>
Message-ID: <B1EF7531-574E-40E2-9B1A-11B5AAAC6042@icir.org>


> On Oct 24, 2016, at 10:22 AM, Siwek, Jon <jsiwek at illinois.edu> wrote:
> 
> I think supporting that would significantly change the design and goals of the package manager, so I wonder what ideas/thoughts others have about the current model vs. the proposed one?

I don't think we should change the design of the package manager based on that use case.  I think that the user is currently experiencing a moment in time where virtually no packages exist yet and assuming that no packages will exist in the future based on that fact.  All of the packages that the user placed into their big script repository are scripts that hopefully will eventually be turned into packages and more easily installed.

The user could still package all of their scripts into a single repository and have a script mechanism for choosing which scripts to actually load on different systems (i.e. @if directives choosing to load certain scripts on different systems, etc).

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/



From asharma at lbl.gov  Mon Oct 24 14:34:45 2016
From: asharma at lbl.gov (Aashish Sharma)
Date: Mon, 24 Oct 2016 14:34:45 -0700
Subject: [Bro-Dev] Adding event/notice at the end of log rotation
Message-ID: <20161024213443.GH55649@mac-711.local>

Would it be possible (also suggestion on what might be the best way) to add an event/execute a script once log-rotation/compression is complete. 

Use case: We archive the logs to a mass storage while leaving a local copy for N days. Right now, its a guessing game on when to run the nightly archive (ie once log compression and local rotation is complete). 

thanks, 
Aashish 

From robin at icir.org  Thu Oct 27 07:28:42 2016
From: robin at icir.org (Robin Sommer)
Date: Thu, 27 Oct 2016 07:28:42 -0700
Subject: [Bro-Dev] bro-pkg support for centralized package structures?
In-Reply-To: <B1EF7531-574E-40E2-9B1A-11B5AAAC6042@icir.org>
References: <7565EBB7-CA12-4044-84C6-D70B2F66DD89@illinois.edu>
	<B1EF7531-574E-40E2-9B1A-11B5AAAC6042@icir.org>
Message-ID: <20161027142842.GD88929@icir.org>



On Mon, Oct 24, 2016 at 10:29 -0400, you wrote:

> I don't think we should change the design of the package manager based
> on that use case.

I concur. While I can see the point that was brought up, I don't think
it will be much of an issue for most users. I think we even discussed
this alternative structure early on during the design phase. The
assumption of one package per repository simplifies the package
manager significantly, which to me justifies the trade-off here. I
could see reconsidering this as we gain more experience with use cases
and generally how people end up using the package manager, but for now
I don't see a need change anything. I doubt that the current model is
what would prevent people from creating packages.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From robin at icir.org  Thu Oct 27 16:28:19 2016
From: robin at icir.org (Robin Sommer)
Date: Thu, 27 Oct 2016 16:28:19 -0700
Subject: [Bro-Dev] Package manager meta data
Message-ID: <20161027232819.GD20933@icir.org>

Playing with Bro packages and bundles, there's one thing where I ended
up a bit confused and that's the meta data. We have two places right
now that store meta data about a package: there's the central set
stored with the package source (bro-pkg.index), and then there's the
set coming with the package itself (bro-pkg.meta). However, I'm a bit
lost what goes where, and which information should remain accessible
once things are bundled up.

Seems bro-pkg.meta generally has "description" and "tags" at least.
However, that information seems lost once I bundle a package up and
don't have access to the index repository anymore. There doesn't seem
to be an expectation that bro-pkg.meta would have similar information,
so there's nothing there to fill the gap.

Then, bro-pkg.meta has a "version" field (the docs show that as the
one field to put in there). I believe that version isn't really used
anywhere other than showing it as part of the meta data output in
"info". In Python it also maps to PackageInfo.metadata_version. But
then there's also PackageStatus.current_version, which is from git
(although I'm not sure how I would actually set that; is it just any
tag?).

Once packages go through bundle/unbundle I cannot run "info" when I'm
offline, and hence I don't get some of the information anymore (same
from Python with PackageInfo I believe)

Ideally, what I'd like to have is a single interface (speaking in
Python API terms) for a package that gives me all the meta-data
consistently, pulling it out from where's it's stored and maintaining
it when bundling. For things like "description", "tags", it could pull
them out of bro-pkg.index by default, and maybe override them from
bro-pkg.meta if they are defined there (so that one can set them even
if there's no package source to begin with). And bundles would then
carry the information through, probably inside their manifest. For
version information, bro-pkg could start with the git branch/tag but
override it with values from bro-pkg.index and bro-pkg.meta if defined
there. That way people could choose how to do their versioning. The
API would just offer a single version() doing the right thing, and
"bro-pkg info" would show that.

Does this make sense? I'm not quite sure about the reasons for the
current structure, I'm mostly thinking from the perspective of
information about the package I'd like to have access to easily, and
where it's coming from.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From jsiwek at illinois.edu  Fri Oct 28 11:50:12 2016
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Fri, 28 Oct 2016 18:50:12 +0000
Subject: [Bro-Dev] Package manager meta data
In-Reply-To: <20161027232819.GD20933@icir.org>
References: <20161027232819.GD20933@icir.org>
Message-ID: <9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu>


> On Oct 27, 2016, at 6:28 PM, Robin Sommer <robin at icir.org> wrote:
> 
> Then, bro-pkg.meta has a "version" field (the docs show that as the
> one field to put in there). I believe that version isn't really used
> anywhere other than showing it as part of the meta data output in
> "info?.

Right, the ?version? field in bro-pkg.meta doesn?t serve an actual function at the moment.  I was thinking about removing it (more on that in a response below).

> then there's also PackageStatus.current_version, which is from git
> (although I'm not sure how I would actually set that; is it just any
> tag?).

If you mean what control does the author of a package have over how that field gets populated: they can use git tags or branches.

With respect to the way users follow package updates, I expected the two common cases to be either tracking a particular git branch or by only updating to release versions.  A ?release version? here is any git tag that looks like a SemVer (e.g. 1.0.0).  So a user might do things like:

bro-pkg install ?version 1.0.0 foo

or

bro-pkg install ?version master foo

> Ideally, what I'd like to have is a single interface (speaking in
> Python API terms) for a package that gives me all the meta-data
> consistently, pulling it out from where's it's stored and maintaining
> it when bundling. For things like "description", "tags", it could pull
> them out of bro-pkg.index by default, and maybe override them from
> bro-pkg.meta if they are defined there (so that one can set them even
> if there's no package source to begin with).

That could be misleading.  The purpose of putting those tags in bro-pkg.index is because they are related to package ?discoverability? and it?s easier to search that data if it?s located in a single repo, the package source, rather than having to download the entire universe of packages up front.

If bro-pkg.meta is allowed to ?override? those fields just for the purpose of displaying metadata in the ?info? command, it?s misleading because that is not the actual data that?s searchable via the ?search? command.

If it?s important to always have some sort of ?description? field to display from the ?info? command, which would help situations where a package was never located via a package source (e.g. installed directly by git URL), then I?d suggest maybe either displaying contents of the README or just having a differently-named field.  Since the user has already found the package, maybe it?s more useful for them to have a ?usage? or ?quickstart? field that gives them some high-level view of the API or options to tweak.

Technically, people can put any field they want in bro-pkg.meta and ?info? will display it, so it?s just a matter of documenting suggested practices.

> And bundles would then
> carry the information through, probably inside their manifest.

Yeah, it would be fine to bind whatever metadata is available in the current bro-pkg.index to a package as it gets bundled or installed and to show that in the ?info? command.

> For
> version information, bro-pkg could start with the git branch/tag but
> override it with values from bro-pkg.index and bro-pkg.meta if defined
> there. That way people could choose how to do their versioning. The
> API would just offer a single version() doing the right thing, and
> "bro-pkg info" would show that.

How would those other versioning schemes actually work, though?

A user needs to be able to install a particular package version, so bro-pkg needs to be able to associate a version with a git commit.

If versioning is done in bro-pkg.meta within a package, I?m not sure how to locate a particular version.  Wouldn?t I have to brute force search through all commits?  Even then, it?s possible the author screwed up and has multiple commits with the same version in bro-pkg.meta, which one does bro-pkg choose?

If versioning is done in bro-pkg.index, the author could create mappings of ?version -> commit hash?.  But if we?re talking about the official bro ?packages? source, they have to submit a pull request each time they want to change that information and someone on the Bro team needs to approve it.

Sticking with standard git mechanisms seems like the best thing to do here.  Tracking updates to branches ?just works? and if people create tags that conform to semantic versioning then tracking updates to stable releases is also straight forward.  This method also works for installing packages directly via explicit git URL (not via a package source).  There?s also no way to accidentally create versioning ambiguities because git is the sole authority on versioning.

> Does this make sense?

I'll need to see your responses to above points, but the point about ?info? showing consistent metadata and working when offline makes sense.  Set of potential changes:

1) Documentation updates: get rid of references to ?version? in bro-pkg.meta, make it clear to use SemVer-formatted git tags for release versioning.

2) When a package is installed or bundled/unbundled, include a copy of its data from the currently-available bro-pkg.index.  This data is bound to that particular installation of the package.

3) The ?info? command should first check if a package is installed and display all metadata bound to that installation before trying to download direct from the origin repo or looking it up in package source.  This makes ?info? work offline when the package is installed.

Let me know how that looks.

- Jon


From jan.grashoefer at gmail.com  Fri Oct 28 15:52:58 2016
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Sat, 29 Oct 2016 00:52:58 +0200
Subject: [Bro-Dev] Package manager meta data
In-Reply-To: <9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu>
References: <20161027232819.GD20933@icir.org>
	<9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu>
Message-ID: <f31c758a-ff81-45d3-cd80-1b1daa3c2e25@gmail.com>

> That could be misleading.  The purpose of putting those tags in bro-pkg.index is because they are related to package ?discoverability? and it?s easier to search that data if it?s located in a single repo, the package source, rather than having to download the entire universe of packages up front.
> 
> If bro-pkg.meta is allowed to ?override? those fields just for the purpose of displaying metadata in the ?info? command, it?s misleading because that is not the actual data that?s searchable via the ?search? command.

I totally agree that overriding information can be misleading but it
might be worth compared to the other options.

> If it?s important to always have some sort of ?description? field to display from the ?info? command, which would help situations where a package was never located via a package source (e.g. installed directly by git URL), then I?d suggest maybe either displaying contents of the README or just having a differently-named field.  Since the user has already found the package, maybe it?s more useful for them to have a ?usage? or ?quickstart? field that gives them some high-level view of the API or options to tweak.

For me, introducing another field in bro-pkt.meta just increases
confusion. Displaying the README seems to be the easiest solution but I
think the behavior is still a bit confusing. Correct me if I am wrong
but bro-pkg.meta contains stuff like script_dir and dependencies (so
rather technically), whereas bro-pkg.index contains the descriptive
information like info text and tags (which is metadata, too, one could
even argue it's "more meta" than script_dir etc.).

>From the user/developer perspective and excluding the technical
realization, I think the most desirable solution would be to have a
single file to put the meta data in, so that a package is completely
self-describing. This would also allow to provide different descriptions
for different versions.

Regarding the technical solution, I'll try to sum up: Using a
distributed structure implies that important information is distributed,
too. I think the first question is, where to aggregate the information?
One could either maintain a cache in every client or integrate it into
the list of packages aka the public repository (current implementation).
The second question would be, whether and how to synchronize the
information? If the info is part of the repository this can be either
done manually (more or less the overriding solution of the current
implementation, assuming that the developers keep meta data in sync) or
automatically (e.g., by a script that fetches meta data of packages once
a day). If the cache is part of the client, this could be done based on
an expiration threshold or intentionally by the user (similar to dnf).
Finally one could drop the requirement of synced package and repository
meta data, risking to confuse the users. In that case the information
contained in the package should be used whenever possible (e.g., the
info command for a not installed package could obtain the most recent
information from the package's git repo).

The current implementation avoids synchronization by splitting the
information at the cost of non-self-describing packages. In general I
would vote for completely self-describing packages (might also make
bundling easier). The different aggregation and synchronization options
have their own pros and cons.

Another question: Now that repositories only contain bro-pkg.index files
with links instead of submodules, how are deleted/unavailable packages
detected/removed?

Best regards,
Jan

From jsiwek at illinois.edu  Sat Oct 29 10:01:31 2016
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Sat, 29 Oct 2016 17:01:31 +0000
Subject: [Bro-Dev] Package manager meta data
In-Reply-To: <f31c758a-ff81-45d3-cd80-1b1daa3c2e25@gmail.com>
References: <20161027232819.GD20933@icir.org>
	<9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu>
	<f31c758a-ff81-45d3-cd80-1b1daa3c2e25@gmail.com>
Message-ID: <B429D7D3-D2DC-40D8-B851-902FDF31A7D2@illinois.edu>


> On Oct 28, 2016, at 5:52 PM, Jan Grash?fer <jan.grashoefer at gmail.com> wrote:
> 
> Correct me if I am wrong
> but bro-pkg.meta contains stuff like script_dir and dependencies (so
> rather technically), whereas bro-pkg.index contains the descriptive
> information like info text and tags (which is metadata, too, one could
> even argue it's "more meta" than script_dir etc.).

That?s right.  The way I was thinking about how it?s split up is: if the metadata is related to how users will search for and discover new packages, then put it bro-pkg.index.  Else it?s likely related to how the package will interoperate with bro, bro-pkg, other packages, etc., and that goes in bro-pkg.meta.

> I think the most desirable solution would be to have a
> single file to put the meta data in, so that a package is completely
> self-describing. This would also allow to provide different descriptions
> for different versions.

Yes, I also think each package maintaining just it?s own, single metadata file is better.  It also means that if the package author ever registered their package with multiple sources, they don?t have to maintain the same bro-pkg.index in multiple places.

I don?t remember if we just settled on the current implementation because it was quick/easy or there were objections to other more complicated technical solutions.

> Regarding the technical solution, I'll try to sum up: Using a
> distributed structure implies that important information is distributed,
> too. I think the first question is, where to aggregate the information?
> One could either maintain a cache in every client or integrate it into
> the list of packages aka the public repository

Aggregating it into the package source is a better solution than having every client do it.  The later isn?t going to scale well:  the client will take longer and longer over time as more and more packages get registered to a source.  Also takes longer as a function of total number of release versions a package has because we are collecting metadata for each version.  Rather not ask users to just get used to developing more patience over time.

> The second question would be, whether and how to synchronize the
> information? If the info is part of the repository this can be either
> done manually (more or less the overriding solution of the current
> implementation, assuming that the developers keep meta data in sync) or
> automatically (e.g., by a script that fetches meta data of packages once
> a day).

I?d opt for a daily cron job to aggregate metadata into package sources.

> If the cache is part of the client, this could be done based on
> an expiration threshold or intentionally by the user (similar to dnf).
> Finally one could drop the requirement of synced package and repository
> meta data, risking to confuse the users. In that case the information
> contained in the package should be used whenever possible (e.g., the
> info command for a not installed package could obtain the most recent
> information from the package's git repo).

It?s not a problem for the metadata to be out of sync for a day since only the ?search? command is going to be using the aggregated data.  Other commands would have direct access to accurate metadata since they?ve already cloned the package locally.

It would also be trivial to give users access to the aggregation tool if they have a problem with potentially using day-old metadata in their searches and are prepared to wait however long the aggregation process takes.

E.g. we add this command/flag: `bro-pkg refresh ?aggregate-metadata`

Then the only difference between the daily aggregation process and a user is that the daily process does a `git commit && git push` in the locally cloned package source that bro-pkg is using internally.

> Another question: Now that repositories only contain bro-pkg.index files
> with links instead of submodules, how are deleted/unavailable packages
> detected/removed?

At the moment, they?d have to be removed manually whenever someone notices or reports it.

If we switch to automated metadata aggregation, removal of nonexistent packages could naturally be a part of that.

- Jon


From robin at icir.org  Sat Oct 29 11:28:02 2016
From: robin at icir.org (Robin Sommer)
Date: Sat, 29 Oct 2016 11:28:02 -0700
Subject: [Bro-Dev] Package manager meta data
In-Reply-To: <B429D7D3-D2DC-40D8-B851-902FDF31A7D2@illinois.edu>
References: <20161027232819.GD20933@icir.org>
	<9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu>
	<f31c758a-ff81-45d3-cd80-1b1daa3c2e25@gmail.com>
	<B429D7D3-D2DC-40D8-B851-902FDF31A7D2@illinois.edu>
Message-ID: <20161029182802.GC21981@icir.org>

I'll just jump in at the end of the thread: Switching to fully
self-describing packages sounds great to me, that should solve all the
issues I noticed. I also don't quite recall the reasoning for arriving
at the current scheme, but it was probably a combination of iterating
over the design a few times, along with a desire to keep it simple.
But having something like a cronjob, or git hook, trigger a rebuild of
a central cache seems easy enough, and would be a major usability
improvement.

I believe the main thing to consider is making it really easy for
package sources (in particular external ones not maintained by the
project) to run the meta data aggregation. Maybe that additional "git
commit && push" could even be integrated into an additional
server-side bro-pkg command. One could then drive that from either
cron or git hooks (if a source operator can do hooks that will avoid
any delays at all).

Robin

On Sat, Oct 29, 2016 at 17:01 +0000, you wrote:

> 
> > On Oct 28, 2016, at 5:52 PM, Jan Grash?fer <jan.grashoefer at gmail.com> wrote:
> > 
> > Correct me if I am wrong
> > but bro-pkg.meta contains stuff like script_dir and dependencies (so
> > rather technically), whereas bro-pkg.index contains the descriptive
> > information like info text and tags (which is metadata, too, one could
> > even argue it's "more meta" than script_dir etc.).
> 
> That?s right.  The way I was thinking about how it?s split up is: if the metadata is related to how users will search for and discover new packages, then put it bro-pkg.index.  Else it?s likely related to how the package will interoperate with bro, bro-pkg, other packages, etc., and that goes in bro-pkg.meta.
> 
> > I think the most desirable solution would be to have a
> > single file to put the meta data in, so that a package is completely
> > self-describing. This would also allow to provide different descriptions
> > for different versions.
> 
> Yes, I also think each package maintaining just it?s own, single metadata file is better.  It also means that if the package author ever registered their package with multiple sources, they don?t have to maintain the same bro-pkg.index in multiple places.
> 
> I don?t remember if we just settled on the current implementation because it was quick/easy or there were objections to other more complicated technical solutions.
> 
> > Regarding the technical solution, I'll try to sum up: Using a
> > distributed structure implies that important information is distributed,
> > too. I think the first question is, where to aggregate the information?
> > One could either maintain a cache in every client or integrate it into
> > the list of packages aka the public repository
> 
> Aggregating it into the package source is a better solution than having every client do it.  The later isn?t going to scale well:  the client will take longer and longer over time as more and more packages get registered to a source.  Also takes longer as a function of total number of release versions a package has because we are collecting metadata for each version.  Rather not ask users to just get used to developing more patience over time.
> 
> > The second question would be, whether and how to synchronize the
> > information? If the info is part of the repository this can be either
> > done manually (more or less the overriding solution of the current
> > implementation, assuming that the developers keep meta data in sync) or
> > automatically (e.g., by a script that fetches meta data of packages once
> > a day).
> 
> I?d opt for a daily cron job to aggregate metadata into package sources.
> 
> > If the cache is part of the client, this could be done based on
> > an expiration threshold or intentionally by the user (similar to dnf).
> > Finally one could drop the requirement of synced package and repository
> > meta data, risking to confuse the users. In that case the information
> > contained in the package should be used whenever possible (e.g., the
> > info command for a not installed package could obtain the most recent
> > information from the package's git repo).
> 
> It?s not a problem for the metadata to be out of sync for a day since only the ?search? command is going to be using the aggregated data.  Other commands would have direct access to accurate metadata since they?ve already cloned the package locally.
> 
> It would also be trivial to give users access to the aggregation tool if they have a problem with potentially using day-old metadata in their searches and are prepared to wait however long the aggregation process takes.
> 
> E.g. we add this command/flag: `bro-pkg refresh ?aggregate-metadata`
> 
> Then the only difference between the daily aggregation process and a user is that the daily process does a `git commit && git push` in the locally cloned package source that bro-pkg is using internally.
> 
> > Another question: Now that repositories only contain bro-pkg.index files
> > with links instead of submodules, how are deleted/unavailable packages
> > detected/removed?
> 
> At the moment, they?d have to be removed manually whenever someone notices or reports it.
> 
> If we switch to automated metadata aggregation, removal of nonexistent packages could naturally be a part of that.
> 
> - Jon
> 
> _______________________________________________
> bro-dev mailing list
> bro-dev at bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
> 



-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From robin at icir.org  Sat Oct 29 12:11:17 2016
From: robin at icir.org (Robin Sommer)
Date: Sat, 29 Oct 2016 12:11:17 -0700
Subject: [Bro-Dev] Package manager meta data
In-Reply-To: <20161029182802.GC21981@icir.org>
References: <20161027232819.GD20933@icir.org>
	<9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu>
	<f31c758a-ff81-45d3-cd80-1b1daa3c2e25@gmail.com>
	<B429D7D3-D2DC-40D8-B851-902FDF31A7D2@illinois.edu>
	<20161029182802.GC21981@icir.org>
Message-ID: <20161029191117.GD21981@icir.org>



On Sat, Oct 29, 2016 at 11:28 -0700, I wrote:

> (if a source operator can do hooks that will avoid any delays at all).

Ah, that's not true of course, it only applies to initial setup of a
package.

Robin

-- 
Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin

From jan.grashoefer at gmail.com  Sun Oct 30 16:20:03 2016
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Mon, 31 Oct 2016 00:20:03 +0100
Subject: [Bro-Dev] Package manager meta data
In-Reply-To: <B429D7D3-D2DC-40D8-B851-902FDF31A7D2@illinois.edu>
References: <20161027232819.GD20933@icir.org>
	<9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu>
	<f31c758a-ff81-45d3-cd80-1b1daa3c2e25@gmail.com>
	<B429D7D3-D2DC-40D8-B851-902FDF31A7D2@illinois.edu>
Message-ID: <12c859cf-b998-e726-c852-1caa32333e16@gmail.com>

> Aggregating it into the package source is a better solution than having every client do it.  The later isn?t going to scale well:  the client will take longer and longer over time as more and more packages get registered to a source.  Also takes longer as a function of total number of release versions a package has because we are collecting metadata for each version.  Rather not ask users to just get used to developing more patience over time.

I am not sure how strong the effect might be but this is definitely a
good point!

> I?d opt for a daily cron job to aggregate metadata into package sources.

For me this seems to be the easiest solution providing a maximum of
usability, too.

> It?s not a problem for the metadata to be out of sync for a day since only the ?search? command is going to be using the aggregated data.  Other commands would have direct access to accurate metadata since they?ve already cloned the package locally.

What about the "info" command? If a package is not installed it would be
nice if the command would obtain the latest meta data from the package's
repository as well.

> It would also be trivial to give users access to the aggregation tool if they have a problem with potentially using day-old metadata in their searches and are prepared to wait however long the aggregation process takes.
> 
> E.g. we add this command/flag: `bro-pkg refresh ?aggregate-metadata`
> 
> Then the only difference between the daily aggregation process and a user is that the daily process does a `git commit && git push` in the locally cloned package source that bro-pkg is using internally.

That sounds great to me! This would be exactly the behavior I would
expect based on other package managers I have used so far.

>> Another question: Now that repositories only contain bro-pkg.index files
>> with links instead of submodules, how are deleted/unavailable packages
>> detected/removed?
> 
> At the moment, they?d have to be removed manually whenever someone notices or reports it.
> 
> If we switch to automated metadata aggregation, removal of nonexistent packages could naturally be a part of that.

Thanks for the clarification. Automatic removal might be a bit
aggressive but a warning would be very helpful I guess.

Best regards,
Jan

From jsiwek at illinois.edu  Mon Oct 31 09:31:14 2016
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Mon, 31 Oct 2016 16:31:14 +0000
Subject: [Bro-Dev] Package manager meta data
In-Reply-To: <12c859cf-b998-e726-c852-1caa32333e16@gmail.com>
References: <20161027232819.GD20933@icir.org>
	<9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu>
	<f31c758a-ff81-45d3-cd80-1b1daa3c2e25@gmail.com>
	<B429D7D3-D2DC-40D8-B851-902FDF31A7D2@illinois.edu>
	<12c859cf-b998-e726-c852-1caa32333e16@gmail.com>
Message-ID: <536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu>


> On Oct 30, 2016, at 6:20 PM, Jan Grash?fer <jan.grashoefer at gmail.com> wrote:
> 
> What about the "info" command? If a package is not installed it would be
> nice if the command would obtain the latest meta data from the package's
> repository as well.

Yep, I imagined ?info? would still grab the latest metadata as well.

> Thanks for the clarification. Automatic removal might be a bit
> aggressive but a warning would be very helpful I guess.

>From the user?s perspective, if they already installed a package that later becomes unavailable, they?d see a warning when they do ?refresh?, but otherwise they could continue using the installed package.

But within the package source, what reason is there to keep old packages listed if their git URL no longer points to a valid package?  What would a user do with that information?  Theoretically, if the package was just temporarily unavailable, the next time the aggregation process runs, it would get listed again and users can seamlessly start receiving updates for it again.

- Jon


From jan.grashoefer at gmail.com  Mon Oct 31 10:03:51 2016
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Mon, 31 Oct 2016 18:03:51 +0100
Subject: [Bro-Dev] Package manager meta data
In-Reply-To: <536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu>
References: <20161027232819.GD20933@icir.org>
	<9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu>
	<f31c758a-ff81-45d3-cd80-1b1daa3c2e25@gmail.com>
	<B429D7D3-D2DC-40D8-B851-902FDF31A7D2@illinois.edu>
	<12c859cf-b998-e726-c852-1caa32333e16@gmail.com>
	<536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu>
Message-ID: <1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com>

>> Thanks for the clarification. Automatic removal might be a bit
>> aggressive but a warning would be very helpful I guess.
> 
> From the user?s perspective, if they already installed a package that later becomes unavailable, they?d see a warning when they do ?refresh?, but otherwise they could continue using the installed package.
> 
> But within the package source, what reason is there to keep old packages listed if their git URL no longer points to a valid package?  What would a user do with that information?  Theoretically, if the package was just temporarily unavailable, the next time the aggregation process runs, it would get listed again and users can seamlessly start receiving updates for it again.

I wasn't precise, sorry. I thought of an temporary unavailable package
repository and had in mind that it would be deleted from the upstream
package source repository by a cron job. I guess you are talking about a
local copy of that source repo. In that case deleting unavailable
packages wouldn't harm. I thought about supporting the operator of the
upstream source repository in cleaning the repo. However, this will
require manual interaction anyway and hopefully isn't a use case that
will be encountered soon :)

Best regards,
Jan

From jsiwek at illinois.edu  Mon Oct 31 10:27:49 2016
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Mon, 31 Oct 2016 17:27:49 +0000
Subject: [Bro-Dev] Package manager meta data
In-Reply-To: <1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com>
References: <20161027232819.GD20933@icir.org>
	<9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu>
	<f31c758a-ff81-45d3-cd80-1b1daa3c2e25@gmail.com>
	<B429D7D3-D2DC-40D8-B851-902FDF31A7D2@illinois.edu>
	<12c859cf-b998-e726-c852-1caa32333e16@gmail.com>
	<536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu>
	<1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com>
Message-ID: <AE5C690C-5C18-4343-B175-D9A14010CFA6@illinois.edu>


> On Oct 31, 2016, at 12:03 PM, Jan Grash?fer <jan.grashoefer at gmail.com> wrote:
> 
> I thought about supporting the operator of the
> upstream source repository in cleaning the repo. However, this will
> require manual interaction anyway and hopefully isn't a use case that
> will be encountered soon :)

I was thinking that even the upstream source repo could clean out invalid packages automatically since their metadata/listing is no longer useful to anyone.  Did you have more thoughts/concerns about why that might require manual interaction?

- Jon


From jan.grashoefer at gmail.com  Mon Oct 31 10:41:49 2016
From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=)
Date: Mon, 31 Oct 2016 18:41:49 +0100
Subject: [Bro-Dev] Package manager meta data
In-Reply-To: <AE5C690C-5C18-4343-B175-D9A14010CFA6@illinois.edu>
References: <20161027232819.GD20933@icir.org>
	<9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu>
	<f31c758a-ff81-45d3-cd80-1b1daa3c2e25@gmail.com>
	<B429D7D3-D2DC-40D8-B851-902FDF31A7D2@illinois.edu>
	<12c859cf-b998-e726-c852-1caa32333e16@gmail.com>
	<536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu>
	<1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com>
	<AE5C690C-5C18-4343-B175-D9A14010CFA6@illinois.edu>
Message-ID: <b3bc08f2-d8c4-4a76-99a5-b8cc1c1c17f2@gmail.com>

> I was thinking that even the upstream source repo could clean out invalid packages automatically since their metadata/listing is no longer useful to anyone.  Did you have more thoughts/concerns about why that might require manual interaction?

I just don't get this:

> Theoretically, if the package was just temporarily unavailable, the next time the aggregation process runs, it would get listed again

How, if it is completely removed?

Jan

From jsiwek at illinois.edu  Mon Oct 31 11:52:41 2016
From: jsiwek at illinois.edu (Siwek, Jon)
Date: Mon, 31 Oct 2016 18:52:41 +0000
Subject: [Bro-Dev] Package manager meta data
In-Reply-To: <b3bc08f2-d8c4-4a76-99a5-b8cc1c1c17f2@gmail.com>
References: <20161027232819.GD20933@icir.org>
	<9A7795D3-3C47-475B-AEB1-2CEDF9BB7AE4@illinois.edu>
	<f31c758a-ff81-45d3-cd80-1b1daa3c2e25@gmail.com>
	<B429D7D3-D2DC-40D8-B851-902FDF31A7D2@illinois.edu>
	<12c859cf-b998-e726-c852-1caa32333e16@gmail.com>
	<536BD67A-7A29-4B3F-9ABB-38DA079C53F3@illinois.edu>
	<1760cb97-de75-1cf7-b42e-899e4688418f@gmail.com>
	<AE5C690C-5C18-4343-B175-D9A14010CFA6@illinois.edu>
	<b3bc08f2-d8c4-4a76-99a5-b8cc1c1c17f2@gmail.com>
Message-ID: <1303E27E-FC8D-4FA5-AD68-DA64FC401F84@illinois.edu>


> On Oct 31, 2016, at 12:41 PM, Jan Grash?fer <jan.grashoefer at gmail.com> wrote:
> 
>> Theoretically, if the package was just temporarily unavailable, the next time the aggregation process runs, it would get listed again
> 
> How, if it is completely removed?

Oh, duh, I see what you mean.  I guess the answer is related to something we haven?t yet spec?d out: how should the structure of a package source?s index files change to adapt to the new scheme of aggregating metadata?

A package source could look like:

https://github.com/bro/packages
	0xxon/
		packages.index
		bro-sumstats-counttable.meta
	sethhall/
		packages.index
		credit-card-exposure.meta
		ssn-exposure.meta
		domain-tld.meta

Contents of sethhall/packages.index:

	https://github.com/sethhall/credit-card-exposure
	https://github.com/sethhall/ssn-exposure
	https://github.com/sethhall/domain-tld

Contents of sethhall/ssn-exposure.meta:

	# Automatically generated, do not edit.
	[master]
	url = https://github.com/sethhall/ssn-exposure
	tags = file analysis, social security number, ssn, dlp, data loss
	description = Detect and log US Social Security numbers.
	script_dir = scripts

	[1.0.0]
	?

	[2.0.0]
	?

The packages.index files are manually modified by users during the act of package registration.  The *.meta files are automatically created by the metadata aggregation process as it crawls the URLs listed in packages.index.

If a package is in packages.index, we say that its state is ?registered?.  Then, once it has a *.meta file, we say that its state is ?listed?.  If a package is ?listed?, then bro-pkg users can see it show up from ?search? and ?list? commands.  If the metadata aggregation process finds an invalid/unreachable package, it removes it?s *.meta file, but keeps it ?registered" in packages.index, so the next crawl will still attempt to list the package in case it was just temporarily unavailable.

Thoughts?  Is it useful to collect metadata for each version or just the latest?  ?Latest" here would mean the latest release version tag or, if none exist, the latest master branch commit.

If per-version metadata collection isn?t needed, the structure outlined above still works, but the existing structure would alsol: just stick latest metadata directly into bro-pkg.index (mixing autogenerated data w/ user-entered data).

- Jon