[Bro] Revisiting log rotate only

Daniel Thayer dnthayer at illinois.edu
Tue Jan 20 19:27:11 PST 2015

On 01/20/2015 04:52 PM, James Lay wrote:
> On 2015-01-20 03:17 PM, Daniel Thayer wrote:
>> On 01/20/2015 04:13 PM, James Lay wrote:
>>> On 2015-01-20 01:04 PM, Daniel Thayer wrote:
>>>> On 01/19/2015 07:57 AM, James Lay wrote:
>>>>> On Sat, 2015-01-17 at 07:37 -0700, James Lay wrote:
>>>>>> Hey all,
>>>>>> I posted about this last August here:
>>>>>> http://mailman.icsi.berkeley.edu/pipermail/bro/2014-August/007329.html
>>>>>> I also noticed someone have a disappearing log event which I have
>>>>>> seen
>>>>>> before  as well here:
>>>>>> http://mailman.icsi.berkeley.edu/pipermail/bro/2015-January/007935.html
>>>>>> I documented my process on installing bro on Ubuntu 14.04 using
>>>>>> just
>>>>>> log rotation below:
>>>>>> sudo apt-get -y install cmake
>>>>>> sudo apt-get -y install python-dev
>>>>>> sudo apt-get -y install swig
>>>>>> cp /usr/local/bro/share/bro/site
>>>>>> cp /opt/bin/startbro <- command line bro with long --filter line
>>>>>> cp /opt/bin/startbro to /etc/rc.local
>>>>>> sudo ln -s /usr/local/bro/bin/bro /usr/local/bin/
>>>>>> sudo ln -s /usr/local/bro/bin/bro-cut /usr/local/bin/
>>>>>> sudo ln -s /usr/local/bro/bin/broctl /usr/local/bin/
>>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/archive-log
>>>>>> /usr/local/bin/
>>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/broctl-config.sh
>>>>>> /usr/local/bin/
>>>>>> sudo ln -s
>>>>>> /usr/local/bro/share/broctl/scripts/create-link-for-log
>>>>>> /usr/local/bin/
>>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/make-archive-name
>>>>>> /usr/local/bin/
>>>>>> git clone https://github.com/jonschipp/mal-dnssearch.git
>>>>>> sudo make install
>>>>>> specifics on log rotate only:
>>>>>> add the below to local.bro
>>>>>> redef Log::default_rotation_interval = 86400 secs;
>>>>>> redef Log::default_rotation_postprocessor_cmd = "archive-log";
>>>>>> edit the below in broctl.cfg
>>>>>> MailTo = jlay at slave-tothe-box.net
>>>>>> <mailto:jlay at slave-tothe-box.net>
>>>>>> LogRotationInterval = 86400
>>>>>> sudo /usr/local/bro/bin/broctl install
>>>>>> Besides the edits to broctl.cfg, file locations are the default.
>>>>>> The
>>>>>> above works well usually...it's after a reboot I have found
>>>>>> things go
>>>>>> bad.  Usually logs get rotated at midnight and I get an email
>>>>>> with
>>>>>> statistics, just what I need.  I rebooted the machine on the 13,
>>>>>> and
>>>>>> that's the last email or log rotation I got....this morning I see
>>>>>> current has files and my logstash instance has data so I believe
>>>>>> the
>>>>>> rotation got..."stuck".  I'm kicking myself for not
>>>>>> heading/tailing
>>>>>> the files first, but after issuing a "sudo killall bro", those
>>>>>> file in
>>>>>> current vanished, no directory was created, and I received no
>>>>>> email,
>>>>>> that data is now gone (no big deal as this is at home).  I
>>>>>> decided to
>>>>>> run broctl install again, then start and kill bro one more time.
>>>>>> At
>>>>>> that point, I got a new directory with log rotation and an email
>>>>>> with
>>>>>> minutes or so of stats.  Please let me know if there's something
>>>>>> I can
>>>>>> do on my end to trouble shoot.  Thank you.
>>>>>> James
>>>>>> _______________________________________________
>>>>>> Bro mailing list
>>>>>> bro at bro-ids.org  <mailto:bro at bro-ids.org>
>>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>>>> Confirming that this method is no longer working.  Heading my
>>>>> connlog
>>>>> file I see:
>>>>> #open 2015-01-19-00-00-05
>>>>> my /usr/local/bro/logs is completely missing Jan 18th.  From my
>>>>> broctl.cfg:
>>>>> SpoolDir = /usr/local/bro/spool
>>>>> LogDir = /usr/local/bro/logs
>>>>> LogRotationInterval = 86400
>>>>>   From my /usr/local/bro/share/bro/site/local.bro:
>>>>> redef Log::default_rotation_interval = 86400 secs;
>>>>> redef Log::default_rotation_postprocessor_cmd = "archive-log";
>>>>> Anything else I can do to debug this?  Thank you.
>>>>> James
>>>> Are you using broctl to start and stop Bro?  What does
>>>> /opt/bin/startbro
>>>> do?
>>> Thanks for looking Daniel.  I am starting this with the below:
>>> /usr/local/bro/bin/bro --no-checksums -i eth0 -i ppp0 --filter '(
>>> large
>>> filter line here)' local "Site::local_nets += { }"
>>> I'm not using broctl.  The only small portion that I am is for the
>>> log
>>> rotation as outlined in the email thread.  After killing and
>>> starting
>>> bro yesterday, this morning at midnight logs got rotated and I got
>>> my
>>> report email.  This appears to happen after a complete reboot of the
>>> device.  It's very odd.  Thanks again.
>>> James
>> What command do you use to stop (or restart) Bro?
> The classic:  sudo killall bro :) when I have to do it manually.  Then
> start with the command line above.  Thanks again.
> James

OK, since you're not using broctl to start/stop bro, here's
what happens:

When you stop bro, bro will rotate all log files (rename them with
a timestamp).  Then, bro will spawn "archive-log" processes, one
per log file, to archive (i.e., copy or gzip to another directory)
each rotated log file.  This can take some time, depending on the
log file size, and whether you're generating connection summary
reports or not.  If the machine is rebooted while this is
happening, then one or more of the rotated logs might not get
archived (because the "archive-log" processes were killed before
they had a chance to finish).

Next time you boot your machine and start bro, the rotated logs will
still be there (unless you have some other script that removes that
directory), but they will never get archived automatically.
And, because the rotated log filenames contain a date/timestamp, they
will not be overwritten by new logs.

To avoid this issue when you want to reboot, I suggest stopping bro,
and then waiting for all the logs to finish being archived, then reboot.

More information about the Bro mailing list