[Bro] Revisiting log rotate only

[Daniel Thayer]

On 01/21/2015 05:01 AM, James Lay wrote:
> On Tue, 2015-01-20 at 21:27 -0600, Daniel Thayer wrote:
>> On 01/20/2015 04:52 PM, James Lay wrote:
>> > On 2015-01-20 03:17 PM, Daniel Thayer wrote:
>> >> On 01/20/2015 04:13 PM, James Lay wrote:
>> >>> On 2015-01-20 01:04 PM, Daniel Thayer wrote:
>> >>>> On 01/19/2015 07:57 AM, James Lay wrote:
>> >>>>> On Sat, 2015-01-17 at 07:37 -0700, James Lay wrote:
>> >>>>>> Hey all,
>> >>>>>>
>> >>>>>> I posted about this last August here:
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>http://mailman.icsi.berkeley.edu/pipermail/bro/2014-August/007329.html
>> >>>>>>
>> >>>>>> I also noticed someone have a disappearing log event which I have
>> >>>>>> seen
>> >>>>>> before  as well here:
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>http://mailman.icsi.berkeley.edu/pipermail/bro/2015-January/007935.html
>> >>>>>>
>> >>>>>> I documented my process on installing bro on Ubuntu 14.04 using
>> >>>>>> just
>> >>>>>> log rotation below:
>> >>>>>>
>> >>>>>> sudo apt-get -y install cmake
>> >>>>>> sudo apt-get -y install python-dev
>> >>>>>> sudo apt-get -y install swig
>> >>>>>> cp /usr/local/bro/share/bro/site
>> >>>>>> cp /opt/bin/startbro <- command line bro with long --filter line
>> >>>>>> cp /opt/bin/startbro to /etc/rc.local
>> >>>>>> sudo ln -s /usr/local/bro/bin/bro /usr/local/bin/
>> >>>>>> sudo ln -s /usr/local/bro/bin/bro-cut /usr/local/bin/
>> >>>>>> sudo ln -s /usr/local/bro/bin/broctl /usr/local/bin/
>> >>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/archive-log
>> >>>>>> /usr/local/bin/
>> >>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/broctl-config.sh
>> >>>>>> /usr/local/bin/
>> >>>>>> sudo ln -s
>> >>>>>> /usr/local/bro/share/broctl/scripts/create-link-for-log
>> >>>>>> /usr/local/bin/
>> >>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/make-archive-name
>> >>>>>> /usr/local/bin/
>> >>>>>> git clonehttps://github.com/jonschipp/mal-dnssearch.git
>> >>>>>> sudo make install
>> >>>>>>
>> >>>>>> specifics on log rotate only:
>> >>>>>>
>> >>>>>> add the below to local.bro
>> >>>>>> redef Log::default_rotation_interval = 86400 secs;
>> >>>>>> redef Log::default_rotation_postprocessor_cmd = "archive-log";
>> >>>>>> edit the below in broctl.cfg
>> >>>>>> MailTo =jlay at slave-tothe-box.net  <mailto:jlay at slave-tothe-box.net>
>> >>>>>> <mailto:jlay at slave-tothe-box.net>
>> >>>>>> LogRotationInterval = 86400
>> >>>>>> sudo /usr/local/bro/bin/broctl install
>> >>>>>>
>> >>>>>> Besides the edits to broctl.cfg, file locations are the default.
>> >>>>>> The
>> >>>>>> above works well usually...it's after a reboot I have found
>> >>>>>> things go
>> >>>>>> bad.  Usually logs get rotated at midnight and I get an email
>> >>>>>> with
>> >>>>>> statistics, just what I need.  I rebooted the machine on the 13,
>> >>>>>> and
>> >>>>>> that's the last email or log rotation I got....this morning I see
>> >>>>>> current has files and my logstash instance has data so I believe
>> >>>>>> the
>> >>>>>> rotation got..."stuck".  I'm kicking myself for not
>> >>>>>> heading/tailing
>> >>>>>> the files first, but after issuing a "sudo killall bro", those
>> >>>>>> file in
>> >>>>>> current vanished, no directory was created, and I received no
>> >>>>>> email,
>> >>>>>> that data is now gone (no big deal as this is at home).  I
>> >>>>>> decided to
>> >>>>>> run broctl install again, then start and kill bro one more time.
>> >>>>>> At
>> >>>>>> that point, I got a new directory with log rotation and an email
>> >>>>>> with
>> >>>>>> minutes or so of stats.  Please let me know if there's something
>> >>>>>> I can
>> >>>>>> do on my end to trouble shoot.  Thank you.
>> >>>>>>
>> >>>>>> James
>> >>>>>> _______________________________________________
>> >>>>>> Bro mailing list
>> >>>>>>bro at bro-ids.org  <mailto:bro at bro-ids.org>   <mailto:bro at bro-ids.org>
>> >>>>>>http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >>>>>
>> >>>>> Confirming that this method is no longer working.  Heading my
>> >>>>> connlog
>> >>>>> file I see:
>> >>>>>
>> >>>>> #open 2015-01-19-00-00-05
>> >>>>>
>> >>>>> my /usr/local/bro/logs is completely missing Jan 18th.  From my
>> >>>>> broctl.cfg:
>> >>>>>
>> >>>>> SpoolDir = /usr/local/bro/spool
>> >>>>> LogDir = /usr/local/bro/logs
>> >>>>> LogRotationInterval = 86400
>> >>>>>
>> >>>>>   From my /usr/local/bro/share/bro/site/local.bro:
>> >>>>>
>> >>>>> redef Log::default_rotation_interval = 86400 secs;
>> >>>>> redef Log::default_rotation_postprocessor_cmd = "archive-log";
>> >>>>>
>> >>>>> Anything else I can do to debug this?  Thank you.
>> >>>>>
>> >>>>> James
>> >>>>
>> >>>> Are you using broctl to start and stop Bro?  What does
>> >>>> /opt/bin/startbro
>> >>>> do?
>> >>>
>> >>> Thanks for looking Daniel.  I am starting this with the below:
>> >>>
>> >>> /usr/local/bro/bin/bro --no-checksums -i eth0 -i ppp0 --filter '(
>> >>> large
>> >>> filter line here)' local "Site::local_nets += { 192.168.1.0/24 }"
>> >>>
>> >>> I'm not using broctl.  The only small portion that I am is for the
>> >>> log
>> >>> rotation as outlined in the email thread.  After killing and
>> >>> starting
>> >>> bro yesterday, this morning at midnight logs got rotated and I got
>> >>> my
>> >>> report email.  This appears to happen after a complete reboot of the
>> >>> device.  It's very odd.  Thanks again.
>> >>>
>> >>> James
>> >>
>> >> What command do you use to stop (or restart) Bro?
>> >
>> > The classic:  sudo killall bro :) when I have to do it manually.  Then
>> > start with the command line above.  Thanks again.
>> >
>> > James
>>
>> OK, since you're not using broctl to start/stop bro, here's
>> what happens:
>>
>> When you stop bro, bro will rotate all log files (rename them with
>> a timestamp).  Then, bro will spawn "archive-log" processes, one
>> per log file, to archive (i.e., copy or gzip to another directory)
>> each rotated log file.  This can take some time, depending on the
>> log file size, and whether you're generating connection summary
>> reports or not.  If the machine is rebooted while this is
>> happening, then one or more of the rotated logs might not get
>> archived (because the "archive-log" processes were killed before
>> they had a chance to finish).
>>
>> Next time you boot your machine and start bro, the rotated logs will
>> still be there (unless you have some other script that removes that
>> directory), but they will never get archived automatically.
>> And, because the rotated log filenames contain a date/timestamp, they
>> will not be overwritten by new logs.
>>
>> To avoid this issue when you want to reboot, I suggest stopping bro,
>> and then waiting for all the logs to finish being archived, then reboot.
>
> Thanks Daniel,
>
> So compressed the entire directory of log files is 7.5 megs....really
> small, so I don't think it's a question of getting stuck during
> compression (truth be told the box doing the bro-ing is sitting right
> next to the box I'm typing this email on...I can hear the drive whir
> away when I stop bro and it lasts maybe 30 seconds).  Also, before
> reboot I manually stop bro...out of habit.  My only thought is that
> *maybe* the path of /usr/local/bin/ where I've symlinked the additional
> scripts aren't seen when my startbro script is run from /etc/rc.local
> file?  In any case I can reproduce the behavior on reboot, so if there's
> a way to debug this I'd love to give it a go.  I'll research the path
> thing on my end (Ubuntu 14.0.4) and I'll try a) rebooting and starting
> bro manually and b) symlinking the script files to /usr/local/sbin/.
> I'll report my findings for anyone else out there, but I kinda think
> most people are just using broctl anyways :)  Thanks again Daniel.
>
> James


One other thing to check is which directory you are starting Bro from,
because that's where Bro will create its log files (if you were
using broctl, this should be /usr/local/bro/spool/bro).

If you ever notice that you are missing logs in the archive directory
(a subdirectory of /usr/local/bro/logs), then you'll want to check
the directory where you were running Bro to see if it contains any
unarchived logs (if you were using broctl to start/stop bro, then
you'd also need to check all subdirectories of
/usr/local/bro/spool/tmp).