[Bro] Revisiting log rotate only
James Lay
jlay at slave-tothe-box.net
Wed Jan 21 03:01:41 PST 2015
On Tue, 2015-01-20 at 21:27 -0600, Daniel Thayer wrote:
> On 01/20/2015 04:52 PM, James Lay wrote:
> > On 2015-01-20 03:17 PM, Daniel Thayer wrote:
> >> On 01/20/2015 04:13 PM, James Lay wrote:
> >>> On 2015-01-20 01:04 PM, Daniel Thayer wrote:
> >>>> On 01/19/2015 07:57 AM, James Lay wrote:
> >>>>> On Sat, 2015-01-17 at 07:37 -0700, James Lay wrote:
> >>>>>> Hey all,
> >>>>>>
> >>>>>> I posted about this last August here:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> http://mailman.icsi.berkeley.edu/pipermail/bro/2014-August/007329.html
> >>>>>>
> >>>>>> I also noticed someone have a disappearing log event which I have
> >>>>>> seen
> >>>>>> before as well here:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> http://mailman.icsi.berkeley.edu/pipermail/bro/2015-January/007935.html
> >>>>>>
> >>>>>> I documented my process on installing bro on Ubuntu 14.04 using
> >>>>>> just
> >>>>>> log rotation below:
> >>>>>>
> >>>>>> sudo apt-get -y install cmake
> >>>>>> sudo apt-get -y install python-dev
> >>>>>> sudo apt-get -y install swig
> >>>>>> cp /usr/local/bro/share/bro/site
> >>>>>> cp /opt/bin/startbro <- command line bro with long --filter line
> >>>>>> cp /opt/bin/startbro to /etc/rc.local
> >>>>>> sudo ln -s /usr/local/bro/bin/bro /usr/local/bin/
> >>>>>> sudo ln -s /usr/local/bro/bin/bro-cut /usr/local/bin/
> >>>>>> sudo ln -s /usr/local/bro/bin/broctl /usr/local/bin/
> >>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/archive-log
> >>>>>> /usr/local/bin/
> >>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/broctl-config.sh
> >>>>>> /usr/local/bin/
> >>>>>> sudo ln -s
> >>>>>> /usr/local/bro/share/broctl/scripts/create-link-for-log
> >>>>>> /usr/local/bin/
> >>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/make-archive-name
> >>>>>> /usr/local/bin/
> >>>>>> git clone https://github.com/jonschipp/mal-dnssearch.git
> >>>>>> sudo make install
> >>>>>>
> >>>>>> specifics on log rotate only:
> >>>>>>
> >>>>>> add the below to local.bro
> >>>>>> redef Log::default_rotation_interval = 86400 secs;
> >>>>>> redef Log::default_rotation_postprocessor_cmd = "archive-log";
> >>>>>> edit the below in broctl.cfg
> >>>>>> MailTo = jlay at slave-tothe-box.net
> >>>>>> <mailto:jlay at slave-tothe-box.net>
> >>>>>> LogRotationInterval = 86400
> >>>>>> sudo /usr/local/bro/bin/broctl install
> >>>>>>
> >>>>>> Besides the edits to broctl.cfg, file locations are the default.
> >>>>>> The
> >>>>>> above works well usually...it's after a reboot I have found
> >>>>>> things go
> >>>>>> bad. Usually logs get rotated at midnight and I get an email
> >>>>>> with
> >>>>>> statistics, just what I need. I rebooted the machine on the 13,
> >>>>>> and
> >>>>>> that's the last email or log rotation I got....this morning I see
> >>>>>> current has files and my logstash instance has data so I believe
> >>>>>> the
> >>>>>> rotation got..."stuck". I'm kicking myself for not
> >>>>>> heading/tailing
> >>>>>> the files first, but after issuing a "sudo killall bro", those
> >>>>>> file in
> >>>>>> current vanished, no directory was created, and I received no
> >>>>>> email,
> >>>>>> that data is now gone (no big deal as this is at home). I
> >>>>>> decided to
> >>>>>> run broctl install again, then start and kill bro one more time.
> >>>>>> At
> >>>>>> that point, I got a new directory with log rotation and an email
> >>>>>> with
> >>>>>> minutes or so of stats. Please let me know if there's something
> >>>>>> I can
> >>>>>> do on my end to trouble shoot. Thank you.
> >>>>>>
> >>>>>> James
> >>>>>> _______________________________________________
> >>>>>> Bro mailing list
> >>>>>> bro at bro-ids.org <mailto:bro at bro-ids.org>
> >>>>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >>>>>
> >>>>> Confirming that this method is no longer working. Heading my
> >>>>> connlog
> >>>>> file I see:
> >>>>>
> >>>>> #open 2015-01-19-00-00-05
> >>>>>
> >>>>> my /usr/local/bro/logs is completely missing Jan 18th. From my
> >>>>> broctl.cfg:
> >>>>>
> >>>>> SpoolDir = /usr/local/bro/spool
> >>>>> LogDir = /usr/local/bro/logs
> >>>>> LogRotationInterval = 86400
> >>>>>
> >>>>> From my /usr/local/bro/share/bro/site/local.bro:
> >>>>>
> >>>>> redef Log::default_rotation_interval = 86400 secs;
> >>>>> redef Log::default_rotation_postprocessor_cmd = "archive-log";
> >>>>>
> >>>>> Anything else I can do to debug this? Thank you.
> >>>>>
> >>>>> James
> >>>>
> >>>> Are you using broctl to start and stop Bro? What does
> >>>> /opt/bin/startbro
> >>>> do?
> >>>
> >>> Thanks for looking Daniel. I am starting this with the below:
> >>>
> >>> /usr/local/bro/bin/bro --no-checksums -i eth0 -i ppp0 --filter '(
> >>> large
> >>> filter line here)' local "Site::local_nets += { 192.168.1.0/24 }"
> >>>
> >>> I'm not using broctl. The only small portion that I am is for the
> >>> log
> >>> rotation as outlined in the email thread. After killing and
> >>> starting
> >>> bro yesterday, this morning at midnight logs got rotated and I got
> >>> my
> >>> report email. This appears to happen after a complete reboot of the
> >>> device. It's very odd. Thanks again.
> >>>
> >>> James
> >>
> >> What command do you use to stop (or restart) Bro?
> >
> > The classic: sudo killall bro :) when I have to do it manually. Then
> > start with the command line above. Thanks again.
> >
> > James
>
> OK, since you're not using broctl to start/stop bro, here's
> what happens:
>
> When you stop bro, bro will rotate all log files (rename them with
> a timestamp). Then, bro will spawn "archive-log" processes, one
> per log file, to archive (i.e., copy or gzip to another directory)
> each rotated log file. This can take some time, depending on the
> log file size, and whether you're generating connection summary
> reports or not. If the machine is rebooted while this is
> happening, then one or more of the rotated logs might not get
> archived (because the "archive-log" processes were killed before
> they had a chance to finish).
>
> Next time you boot your machine and start bro, the rotated logs will
> still be there (unless you have some other script that removes that
> directory), but they will never get archived automatically.
> And, because the rotated log filenames contain a date/timestamp, they
> will not be overwritten by new logs.
>
> To avoid this issue when you want to reboot, I suggest stopping bro,
> and then waiting for all the logs to finish being archived, then reboot.
Thanks Daniel,
So compressed the entire directory of log files is 7.5 megs....really
small, so I don't think it's a question of getting stuck during
compression (truth be told the box doing the bro-ing is sitting right
next to the box I'm typing this email on...I can hear the drive whir
away when I stop bro and it lasts maybe 30 seconds). Also, before
reboot I manually stop bro...out of habit. My only thought is that
*maybe* the path of /usr/local/bin/ where I've symlinked the additional
scripts aren't seen when my startbro script is run from /etc/rc.local
file? In any case I can reproduce the behavior on reboot, so if there's
a way to debug this I'd love to give it a go. I'll research the path
thing on my end (Ubuntu 14.0.4) and I'll try a) rebooting and starting
bro manually and b) symlinking the script files to /usr/local/sbin/.
I'll report my findings for anyone else out there, but I kinda think
most people are just using broctl anyways :) Thanks again Daniel.
James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150121/cab6a6f7/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 925 bytes
Desc: not available
Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150121/cab6a6f7/attachment-0001.bin
More information about the Bro
mailing list