From asharma at lbl.gov Mon Apr 1 10:15:31 2019 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 1 Apr 2019 10:15:31 -0700 Subject: [Zeek-Dev] Bro 2.5.4 In-Reply-To: References: Message-ID: <20190401171530.GB67110@MacPro-2331.local> What Seth suggest is a better way to do this. I also do one more thing - when logs rotate overnight - we add the hostname to the log file names too. So our logs from different bro instances are named like: conn.log.cluster-1-mgr.2019-03-29-00:00:00-00:00:00.gz conn.log.wireless-bro.2019-03-29-00:00:00-00:00:00.gz etc. How we do this: Basically in broctl.cfg I've added: # change log naming MakeArchiveName = /usr/local/bro/common/scripts/makelocal-archivename-2.1 and here is the makelocal-archivename-2.1 $ cat makelocal-archivename-2.1 #! /usr/bin/env bash # # $Id: makelocal-archivename-2.1 572 2014-10-11 19:18:11Z aashish $ # # Returns a path for archived log files. This script is called # once for each log file being archived. Usage is: # # make-archive-name # # basename: The base file name of the log file being archived (e.g., conn.log). # timestamp-when-opened: The timestamp when the log file being archived was created. # timestamp-when-closed: The timestamp when the log file being archived was finished. # # Times are given in the form "year-month-day-hour-minute-second", # e.g., "2010-03-30-13-12-04" # # The script must return the path under which the file should be # archived. If it's a relative path, it will be interpreted as # relative to BroControl's standard log directory. # # Note that even though the logs will later be compressed, this # script should return the filename without any .gz extension; that # extension will be appended later. echo $@ >> /var/tmp/make-archive-name-2.1.out name=$1 flavor=$2 opened=$3 closed=$4 host=`hostname -s` day=`echo $opened | awk -F - '{printf "%s-%s-%s", $1, $2, $3}'` from=`echo $opened | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'` to=`echo $closed | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'` if [ "$closed" != "" ]; then echo $day/$name.$host.$day-$from-$to else echo $day/$name.$host.$day-$from-current fi On Thu, Jan 24, 2019 at 01:10:50PM -0500, Seth Hall wrote: > > > On 3 Jan 2019, at 11:01, Rajput, Jawad (CONTR) wrote: > > > Is there a way to add Bro server hostname field into all the Bro log > > types? We have 5 Bro servers capturing traffic on different network > > nodes, we are trying to add each server/sensor hostname into all the log > > types so analyst can identify where the logs are coming from. > > Yes! > > We added a log extension mecahnism a while ago. Here's a snippet you could > start from... > > ```bro > option my_server_name = ""; > > type MyLogExtension: record { > server_name: string &log; > }; > > function add_my_log_extension(path: string): MyLogExtension > { > return MyLogExtension($server_name = my_server_name); > } > > > redef Log::default_ext_func = add_my_log_extension; > ``` > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com > _______________________________________________ > zeek-dev mailing list > zeek-dev at zeek.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev From vern at corelight.com Mon Apr 8 13:02:17 2019 From: vern at corelight.com (Vern Paxson) Date: Mon, 08 Apr 2019 13:02:17 -0700 Subject: [Zeek-Dev] connection $history - 'g' for gap Message-ID: <20190408200217.1C1DE74113F@rock.ICSI.Berkeley.EDU> I'm finding it would be handy to be able to glance at a connection log line and know that the analysis for the connection experienced a content gap. For example, this can immediately explain why DPD failed to identify a known server. Proposal: add 'g'/'G' connection history values, scaled in the same exponential way as for 'c', 't' and 'w'. Any thoughts/objections before I go ahead and implement this? Vern From jmellander at lbl.gov Mon Apr 8 17:05:50 2019 From: jmellander at lbl.gov (Jim Mellander) Date: Mon, 8 Apr 2019 17:05:50 -0700 Subject: [Zeek-Dev] connection $history - 'g' for gap In-Reply-To: <20190408200217.1C1DE74113F@rock.ICSI.Berkeley.EDU> References: <20190408200217.1C1DE74113F@rock.ICSI.Berkeley.EDU> Message-ID: It might be valuable to have some (optional) way of accessing the byte counts consisting the content gap(s). If the content gap is somewhere in a long tail, but DPD still fails, then the explanation could be something other than a content gap. On the other hand, maybe you're just thinking about content gaps at the head of a connection before it has been fully analyzed. On Mon, Apr 8, 2019 at 1:09 PM Vern Paxson wrote: > I'm finding it would be handy to be able to glance at a connection log line > and know that the analysis for the connection experienced a content gap. > For example, this can immediately explain why DPD failed to identify a > known server. > > Proposal: add 'g'/'G' connection history values, scaled in the same > exponential way as for 'c', 't' and 'w'. > > Any thoughts/objections before I go ahead and implement this? > > Vern > _______________________________________________ > zeek-dev mailing list > zeek-dev at zeek.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190408/da2e8930/attachment.html From seth at corelight.com Tue Apr 9 04:36:21 2019 From: seth at corelight.com (Seth Hall) Date: Tue, 9 Apr 2019 13:36:21 +0200 Subject: [Zeek-Dev] connection $history - 'g' for gap In-Reply-To: <20190408200217.1C1DE74113F@rock.ICSI.Berkeley.EDU> References: <20190408200217.1C1DE74113F@rock.ICSI.Berkeley.EDU> Message-ID: <88ECA166-1A7C-4376-B4F3-B25DC6B0D3F2@corelight.com> I think it would be useful. I can't believe we hadn't already thought of adding that! .Seth -- Seth Hall * Corelight, Inc * www.corelight.com > On Apr 8, 2019, at 10:02 PM, Vern Paxson wrote: > > I'm finding it would be handy to be able to glance at a connection log line > and know that the analysis for the connection experienced a content gap. > For example, this can immediately explain why DPD failed to identify a > known server. > > Proposal: add 'g'/'G' connection history values, scaled in the same > exponential way as for 'c', 't' and 'w'. > > Any thoughts/objections before I go ahead and implement this? > > Vern > _______________________________________________ > zeek-dev mailing list > zeek-dev at zeek.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev From michalpurzynski1 at gmail.com Tue Apr 9 05:57:58 2019 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Tue, 9 Apr 2019 14:57:58 +0200 Subject: [Zeek-Dev] connection $history - 'g' for gap In-Reply-To: <88ECA166-1A7C-4376-B4F3-B25DC6B0D3F2@corelight.com> References: <20190408200217.1C1DE74113F@rock.ICSI.Berkeley.EDU> <88ECA166-1A7C-4376-B4F3-B25DC6B0D3F2@corelight.com> Message-ID: It?s one of these brilliant ideas that I cannot believe we haven?t got yet ;) Let?s implement it. > On Apr 9, 2019, at 1:36 PM, Seth Hall wrote: > > I think it would be useful. I can't believe we hadn't already thought of adding that! > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com > > >> On Apr 8, 2019, at 10:02 PM, Vern Paxson wrote: >> >> I'm finding it would be handy to be able to glance at a connection log line >> and know that the analysis for the connection experienced a content gap. >> For example, this can immediately explain why DPD failed to identify a >> known server. >> >> Proposal: add 'g'/'G' connection history values, scaled in the same >> exponential way as for 'c', 't' and 'w'. >> >> Any thoughts/objections before I go ahead and implement this? >> >> Vern >> _______________________________________________ >> zeek-dev mailing list >> zeek-dev at zeek.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev > > _______________________________________________ > zeek-dev mailing list > zeek-dev at zeek.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev From justin at corelight.com Tue Apr 9 09:35:42 2019 From: justin at corelight.com (Justin Azoff) Date: Tue, 9 Apr 2019 12:35:42 -0400 Subject: [Zeek-Dev] connection $history - 'g' for gap In-Reply-To: References: <20190408200217.1C1DE74113F@rock.ICSI.Berkeley.EDU> Message-ID: On Mon, Apr 8, 2019 at 8:13 PM Jim Mellander wrote: > It might be valuable to have some (optional) way of accessing the byte > counts consisting the content gap(s). If the content gap is somewhere in a > long tail, but DPD still fails, then the explanation could be something > other than a content gap. > > On the other hand, maybe you're just thinking about content gaps at the > head of a connection before it has been fully analyzed. > This is the missed_bytes field: missed_bytes: count &log &default = 0 &optional Indicates the number of bytes missed in content gaps, which is representative of packet loss. A value other than zero will normally cause protocol analysis to fail but some analysis may have been completed prior to the packet loss. -- Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190409/df2c5fca/attachment.html From jmellander at lbl.gov Tue Apr 9 09:53:57 2019 From: jmellander at lbl.gov (Jim Mellander) Date: Tue, 9 Apr 2019 09:53:57 -0700 Subject: [Zeek-Dev] connection $history - 'g' for gap In-Reply-To: References: <20190408200217.1C1DE74113F@rock.ICSI.Berkeley.EDU> Message-ID: Thanks. I was thinking of something a bit different - the total amount of the content gap is useful, but in some cases it might be useful to know where the content gaps occurred, whether in the head of the connection, which likely is impactful for protocol analysis, or in a long tail, where it probably doesn't affect analysis. Perhaps some tunable setting indicating that "I only care about content gaps in the first 10K (or whatever) of the connection" could address that... On Tue, Apr 9, 2019 at 9:36 AM Justin Azoff wrote: > > > On Mon, Apr 8, 2019 at 8:13 PM Jim Mellander wrote: > >> It might be valuable to have some (optional) way of accessing the byte >> counts consisting the content gap(s). If the content gap is somewhere in a >> long tail, but DPD still fails, then the explanation could be something >> other than a content gap. >> >> On the other hand, maybe you're just thinking about content gaps at the >> head of a connection before it has been fully analyzed. >> > > This is the missed_bytes field: > > missed_bytes: count &log &default = 0 &optional > Indicates the number of bytes missed in content gaps, which is > representative of packet loss. A value other than zero will normally cause > protocol analysis to fail but some analysis may have been completed prior > to the packet loss. > > -- > Justin > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190409/458c2d37/attachment-0001.html From anthony.kasza at gmail.com Tue Apr 9 21:11:00 2019 From: anthony.kasza at gmail.com (anthony kasza) Date: Tue, 9 Apr 2019 22:11:00 -0600 Subject: [Zeek-Dev] connection $history - 'g' for gap In-Reply-To: References: <20190408200217.1C1DE74113F@rock.ICSI.Berkeley.EDU> Message-ID: I like the idea of logging gap ranges for a connection. Could a vector be used to store gap start and gap stop offsets? -AK On Tue, Apr 9, 2019, 11:01 Jim Mellander wrote: > Thanks. I was thinking of something a bit different - the total amount of > the content gap is useful, but in some cases it might be useful to know > where the content gaps occurred, whether in the head of the connection, > which likely is impactful for protocol analysis, or in a long tail, where > it probably doesn't affect analysis. > > Perhaps some tunable setting indicating that "I only care about content > gaps in the first 10K (or whatever) of the connection" could address that... > > On Tue, Apr 9, 2019 at 9:36 AM Justin Azoff wrote: > >> >> >> On Mon, Apr 8, 2019 at 8:13 PM Jim Mellander wrote: >> >>> It might be valuable to have some (optional) way of accessing the byte >>> counts consisting the content gap(s). If the content gap is somewhere in a >>> long tail, but DPD still fails, then the explanation could be something >>> other than a content gap. >>> >>> On the other hand, maybe you're just thinking about content gaps at the >>> head of a connection before it has been fully analyzed. >>> >> >> This is the missed_bytes field: >> >> missed_bytes: count &log &default = 0 &optional >> Indicates the number of bytes missed in content gaps, which is >> representative of packet loss. A value other than zero will normally cause >> protocol analysis to fail but some analysis may have been completed prior >> to the packet loss. >> >> -- >> Justin >> > _______________________________________________ > zeek-dev mailing list > zeek-dev at zeek.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190409/d58e3850/attachment.html From seth at corelight.com Tue Apr 9 21:55:58 2019 From: seth at corelight.com (Seth Hall) Date: Wed, 10 Apr 2019 06:55:58 +0200 Subject: [Zeek-Dev] connection $history - 'g' for gap In-Reply-To: References: <20190408200217.1C1DE74113F@rock.ICSI.Berkeley.EDU> Message-ID: <6D4ADBE5-47A4-42AE-B337-ADA6478FFD46@corelight.com> I don't think we'd add that to the default log, but you should be able to write a script that does something like that. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com > On Apr 10, 2019, at 6:11 AM, anthony kasza wrote: > > I like the idea of logging gap ranges for a connection. Could a vector be used to store gap start and gap stop offsets? > > -AK > >> On Tue, Apr 9, 2019, 11:01 Jim Mellander wrote: >> Thanks. I was thinking of something a bit different - the total amount of the content gap is useful, but in some cases it might be useful to know where the content gaps occurred, whether in the head of the connection, which likely is impactful for protocol analysis, or in a long tail, where it probably doesn't affect analysis. >> >> Perhaps some tunable setting indicating that "I only care about content gaps in the first 10K (or whatever) of the connection" could address that... >> >>> On Tue, Apr 9, 2019 at 9:36 AM Justin Azoff wrote: >>> >>> >>>> On Mon, Apr 8, 2019 at 8:13 PM Jim Mellander wrote: >>>> It might be valuable to have some (optional) way of accessing the byte counts consisting the content gap(s). If the content gap is somewhere in a long tail, but DPD still fails, then the explanation could be something other than a content gap. >>>> >>>> On the other hand, maybe you're just thinking about content gaps at the head of a connection before it has been fully analyzed. >>> >>> >>> This is the missed_bytes field: >>> >>> missed_bytes: count &log &default = 0 &optional >>> Indicates the number of bytes missed in content gaps, which is representative of packet loss. A value other than zero will normally cause protocol analysis to fail but some analysis may have been completed prior to the packet loss. >>> >>> -- >>> Justin >> _______________________________________________ >> zeek-dev mailing list >> zeek-dev at zeek.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev > _______________________________________________ > zeek-dev mailing list > zeek-dev at zeek.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190410/a9f24201/attachment.html From kmcmahon at mitre.org Wed Apr 10 05:59:05 2019 From: kmcmahon at mitre.org (McMahon, Kevin J) Date: Wed, 10 Apr 2019 12:59:05 +0000 Subject: [Zeek-Dev] [EXT] Re: connection $history - 'g' for gap In-Reply-To: <19530_1554869914_5CAD6E9A_19530_4916_1_CAEZw2bzTef7O9jBa2f+8N4R-BTUxUjc_GWaJNTiVek6X+tbQbw@mail.gmail.com> References: <20190408200217.1C1DE74113F@rock.ICSI.Berkeley.EDU> , <19530_1554869914_5CAD6E9A_19530_4916_1_CAEZw2bzTef7O9jBa2f+8N4R-BTUxUjc_GWaJNTiVek6X+tbQbw@mail.gmail.com> Message-ID: That could get very messy in the real world. How about start of first gap, length of first gap, total number of gaps? Sent with BlackBerry Work (www.blackberry.com) From: anthony kasza > Date: Wednesday, Apr 10, 2019, 12:18 AM To: Jim Mellander > Cc: zeek-dev at zeek.org >, Vern Paxson > Subject: [EXT] Re: [Zeek-Dev] connection $history - 'g' for gap I like the idea of logging gap ranges for a connection. Could a vector be used to store gap start and gap stop offsets? -AK On Tue, Apr 9, 2019, 11:01 Jim Mellander > wrote: Thanks. I was thinking of something a bit different - the total amount of the content gap is useful, but in some cases it might be useful to know where the content gaps occurred, whether in the head of the connection, which likely is impactful for protocol analysis, or in a long tail, where it probably doesn't affect analysis. Perhaps some tunable setting indicating that "I only care about content gaps in the first 10K (or whatever) of the connection" could address that... On Tue, Apr 9, 2019 at 9:36 AM Justin Azoff > wrote: On Mon, Apr 8, 2019 at 8:13 PM Jim Mellander > wrote: It might be valuable to have some (optional) way of accessing the byte counts consisting the content gap(s). If the content gap is somewhere in a long tail, but DPD still fails, then the explanation could be something other than a content gap. On the other hand, maybe you're just thinking about content gaps at the head of a connection before it has been fully analyzed. This is the missed_bytes field: missed_bytes: count &log &default = 0 &optional Indicates the number of bytes missed in content gaps, which is representative of packet loss. A value other than zero will normally cause protocol analysis to fail but some analysis may have been completed prior to the packet loss. -- Justin _______________________________________________ zeek-dev mailing list zeek-dev at zeek.org http://mailman.icsi.berkeley.edu/mailman/listinfo/zeek-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190410/03c11e47/attachment-0001.html From vern at corelight.com Wed Apr 10 06:29:30 2019 From: vern at corelight.com (Vern Paxson) Date: Wed, 10 Apr 2019 06:29:30 -0700 Subject: [Zeek-Dev] [EXT] Re: connection $history - 'g' for gap In-Reply-To: (Wed, 10 Apr 2019 12:59:05 -0000). Message-ID: <20190410132930.D016975880C@rock.ICSI.Berkeley.EDU> > That could get very messy in the real world. How about start of first gap,= > length of first gap, total number of gaps? I think if the goal is to know whether DPD failed due to content gaps, much better than trying to infer that from a set of gap information would be for dpd.log to include "no DPD decision because ran into a content gap" or such. Vern From kmcmahon at mitre.org Wed Apr 10 12:36:21 2019 From: kmcmahon at mitre.org (McMahon, Kevin J) Date: Wed, 10 Apr 2019 19:36:21 +0000 Subject: [Zeek-Dev] [EXT] Re: connection $history - 'g' for gap In-Reply-To: <20190410132930.D016975880C@rock.ICSI.Berkeley.EDU> References: (Wed, 10 Apr 2019 12:59:05 -0000). <20190410132930.D016975880C@rock.ICSI.Berkeley.EDU> Message-ID: Agreed, but I think there is broader use for knowing, from the conn.log, that there were gaps. I think the 'g' character would address that. I've always loved the history field and have found numerous uses for it over the years. Kevin -----Original Message----- From: vern at ICIR.org On Behalf Of Vern Paxson Sent: Wednesday, April 10, 2019 9:30 AM To: McMahon, Kevin J Cc: anthony kasza ; Jim Mellander ; zeek-dev at zeek.org Subject: Re: [EXT] Re: [Zeek-Dev] connection $history - 'g' for gap > That could get very messy in the real world. How about start of first > gap,= length of first gap, total number of gaps? I think if the goal is to know whether DPD failed due to content gaps, much better than trying to infer that from a set of gap information would be for dpd.log to include "no DPD decision because ran into a content gap" or such. Vern From jmellander at lbl.gov Mon Apr 29 11:43:53 2019 From: jmellander at lbl.gov (Jim Mellander) Date: Mon, 29 Apr 2019 11:43:53 -0700 Subject: [Zeek-Dev] Sumstats undocumented feature - changing the epoch Message-ID: I've found it convenient to use an undocumented feature of Sumstats: changing the epoch. This comes particularly handy when creating statistics for human consumption, as oftentimes it is useful to synchronize to a logging interval. For example, if hourly stats are desired, it is useful to have a shorter epoch for the original sumstats to align with an hour, then to have subsequent sumstats trigger on the hour. Researching into this, I realized that the epoch variable can be changed, if the argument to *Sumstats::create* is a variable, rather than the usual style of an anonymous argument. Then, in *epoch_result*, or *epoch_finished*, the timeout for the next epoch can be recomputed on the fly using *calc_next_rotate()*. However, this fails to work as expected as the next sumstat is scheduled prior to executing *epoch_result*, and *epoch_finished*. What does work is the following hack: 1. Create the initial sumstat with a epoch that will synchronize to the logging interval 2. Immediately change the epoch to the desired interval Example: *event bro_init()* * {* * # So network_time() will be initialized...* * schedule 0 usec { setup_sumstat() };* * }* *event setup_sumstat()* * {* * ... blah ...* * local mysumstat: SumStats::SumStat;* * mysumstat = [* * $name="mysumstat",* * $epoch=calc_next_rotate(10 min) - network_time(),* * etc...* * ];* * SumStats::create(mysumstat);* * # Now SumStat has been created, and the initial epoch scheduled, change epoch to regular interval for the future* * mysumstat$epoch = 10 min;* * }* It would be convenient if the epoch could be changed in *epoch_result* or *epoch_finished*, but some internals would require a bit of change - the reschedule would need to take place after processing results, which could throw the timing off a bit - on the other hand, unless one is interested in exact statistics over a known time period (as I am), the small amount of jitter probably wouldn't be noticeable or significant. The above is horribly hackish, and a different approach for accomplishing the goal would be to allow use scripts to schedule the end of the epoch: 1. Mark *epoch* as *&optional*. 2. Expose and document *SumStats::finish_epoch* as part of the public API 3. Make the minor changes to not schedule *SumStats::finish_epoch* if *epoch* is undefined. By not defining *epoch* a script would indicate that it will manage epoch timing. The script would schedule the first epoch based on the logging interval, and in the *epoch_finished* function schedule each successive epoch to stay in sync with the logging interval. Any comments, suggestions, etc. ???? Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190429/5deca004/attachment.html