[Bro] Revisiting log rotate only
James Lay
jlay at slave-tothe-box.net
Sun Feb 1 07:03:57 PST 2015
On Wed, 2015-01-21 at 10:17 -0600, Daniel Thayer wrote:
> On 01/21/2015 05:01 AM, James Lay wrote:
> > On Tue, 2015-01-20 at 21:27 -0600, Daniel Thayer wrote:
> >> On 01/20/2015 04:52 PM, James Lay wrote:
> >> > On 2015-01-20 03:17 PM, Daniel Thayer wrote:
> >> >> On 01/20/2015 04:13 PM, James Lay wrote:
> >> >>> On 2015-01-20 01:04 PM, Daniel Thayer wrote:
> >> >>>> On 01/19/2015 07:57 AM, James Lay wrote:
> >> >>>>> On Sat, 2015-01-17 at 07:37 -0700, James Lay wrote:
> >> >>>>>> Hey all,
> >> >>>>>>
> >> >>>>>> I posted about this last August here:
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>http://mailman.icsi.berkeley.edu/pipermail/bro/2014-August/007329.html
> >> >>>>>>
> >> >>>>>> I also noticed someone have a disappearing log event which I have
> >> >>>>>> seen
> >> >>>>>> before as well here:
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>http://mailman.icsi.berkeley.edu/pipermail/bro/2015-January/007935.html
> >> >>>>>>
> >> >>>>>> I documented my process on installing bro on Ubuntu 14.04 using
> >> >>>>>> just
> >> >>>>>> log rotation below:
> >> >>>>>>
> >> >>>>>> sudo apt-get -y install cmake
> >> >>>>>> sudo apt-get -y install python-dev
> >> >>>>>> sudo apt-get -y install swig
> >> >>>>>> cp /usr/local/bro/share/bro/site
> >> >>>>>> cp /opt/bin/startbro <- command line bro with long --filter line
> >> >>>>>> cp /opt/bin/startbro to /etc/rc.local
> >> >>>>>> sudo ln -s /usr/local/bro/bin/bro /usr/local/bin/
> >> >>>>>> sudo ln -s /usr/local/bro/bin/bro-cut /usr/local/bin/
> >> >>>>>> sudo ln -s /usr/local/bro/bin/broctl /usr/local/bin/
> >> >>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/archive-log
> >> >>>>>> /usr/local/bin/
> >> >>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/broctl-config.sh
> >> >>>>>> /usr/local/bin/
> >> >>>>>> sudo ln -s
> >> >>>>>> /usr/local/bro/share/broctl/scripts/create-link-for-log
> >> >>>>>> /usr/local/bin/
> >> >>>>>> sudo ln -s /usr/local/bro/share/broctl/scripts/make-archive-name
> >> >>>>>> /usr/local/bin/
> >> >>>>>> git clonehttps://github.com/jonschipp/mal-dnssearch.git
> >> >>>>>> sudo make install
> >> >>>>>>
> >> >>>>>> specifics on log rotate only:
> >> >>>>>>
> >> >>>>>> add the below to local.bro
> >> >>>>>> redef Log::default_rotation_interval = 86400 secs;
> >> >>>>>> redef Log::default_rotation_postprocessor_cmd = "archive-log";
> >> >>>>>> edit the below in broctl.cfg
> >> >>>>>> MailTo =jlay at slave-tothe-box.net <mailto:jlay at slave-tothe-box.net>
> >> >>>>>> <mailto:jlay at slave-tothe-box.net>
> >> >>>>>> LogRotationInterval = 86400
> >> >>>>>> sudo /usr/local/bro/bin/broctl install
> >> >>>>>>
> >> >>>>>> Besides the edits to broctl.cfg, file locations are the default.
> >> >>>>>> The
> >> >>>>>> above works well usually...it's after a reboot I have found
> >> >>>>>> things go
> >> >>>>>> bad. Usually logs get rotated at midnight and I get an email
> >> >>>>>> with
> >> >>>>>> statistics, just what I need. I rebooted the machine on the 13,
> >> >>>>>> and
> >> >>>>>> that's the last email or log rotation I got....this morning I see
> >> >>>>>> current has files and my logstash instance has data so I believe
> >> >>>>>> the
> >> >>>>>> rotation got..."stuck". I'm kicking myself for not
> >> >>>>>> heading/tailing
> >> >>>>>> the files first, but after issuing a "sudo killall bro", those
> >> >>>>>> file in
> >> >>>>>> current vanished, no directory was created, and I received no
> >> >>>>>> email,
> >> >>>>>> that data is now gone (no big deal as this is at home). I
> >> >>>>>> decided to
> >> >>>>>> run broctl install again, then start and kill bro one more time.
> >> >>>>>> At
> >> >>>>>> that point, I got a new directory with log rotation and an email
> >> >>>>>> with
> >> >>>>>> minutes or so of stats. Please let me know if there's something
> >> >>>>>> I can
> >> >>>>>> do on my end to trouble shoot. Thank you.
> >> >>>>>>
> >> >>>>>> James
> >> >>>>>> _______________________________________________
> >> >>>>>> Bro mailing list
> >> >>>>>>bro at bro-ids.org <mailto:bro at bro-ids.org> <mailto:bro at bro-ids.org>
> >> >>>>>>http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
> >> >>>>>
> >> >>>>> Confirming that this method is no longer working. Heading my
> >> >>>>> connlog
> >> >>>>> file I see:
> >> >>>>>
> >> >>>>> #open 2015-01-19-00-00-05
> >> >>>>>
> >> >>>>> my /usr/local/bro/logs is completely missing Jan 18th. From my
> >> >>>>> broctl.cfg:
> >> >>>>>
> >> >>>>> SpoolDir = /usr/local/bro/spool
> >> >>>>> LogDir = /usr/local/bro/logs
> >> >>>>> LogRotationInterval = 86400
> >> >>>>>
> >> >>>>> From my /usr/local/bro/share/bro/site/local.bro:
> >> >>>>>
> >> >>>>> redef Log::default_rotation_interval = 86400 secs;
> >> >>>>> redef Log::default_rotation_postprocessor_cmd = "archive-log";
> >> >>>>>
> >> >>>>> Anything else I can do to debug this? Thank you.
> >> >>>>>
> >> >>>>> James
> >> >>>>
> >> >>>> Are you using broctl to start and stop Bro? What does
> >> >>>> /opt/bin/startbro
> >> >>>> do?
> >> >>>
> >> >>> Thanks for looking Daniel. I am starting this with the below:
> >> >>>
> >> >>> /usr/local/bro/bin/bro --no-checksums -i eth0 -i ppp0 --filter '(
> >> >>> large
> >> >>> filter line here)' local "Site::local_nets += { 192.168.1.0/24 }"
> >> >>>
> >> >>> I'm not using broctl. The only small portion that I am is for the
> >> >>> log
> >> >>> rotation as outlined in the email thread. After killing and
> >> >>> starting
> >> >>> bro yesterday, this morning at midnight logs got rotated and I got
> >> >>> my
> >> >>> report email. This appears to happen after a complete reboot of the
> >> >>> device. It's very odd. Thanks again.
> >> >>>
> >> >>> James
> >> >>
> >> >> What command do you use to stop (or restart) Bro?
> >> >
> >> > The classic: sudo killall bro :) when I have to do it manually. Then
> >> > start with the command line above. Thanks again.
> >> >
> >> > James
> >>
> >> OK, since you're not using broctl to start/stop bro, here's
> >> what happens:
> >>
> >> When you stop bro, bro will rotate all log files (rename them with
> >> a timestamp). Then, bro will spawn "archive-log" processes, one
> >> per log file, to archive (i.e., copy or gzip to another directory)
> >> each rotated log file. This can take some time, depending on the
> >> log file size, and whether you're generating connection summary
> >> reports or not. If the machine is rebooted while this is
> >> happening, then one or more of the rotated logs might not get
> >> archived (because the "archive-log" processes were killed before
> >> they had a chance to finish).
> >>
> >> Next time you boot your machine and start bro, the rotated logs will
> >> still be there (unless you have some other script that removes that
> >> directory), but they will never get archived automatically.
> >> And, because the rotated log filenames contain a date/timestamp, they
> >> will not be overwritten by new logs.
> >>
> >> To avoid this issue when you want to reboot, I suggest stopping bro,
> >> and then waiting for all the logs to finish being archived, then reboot.
> >
> > Thanks Daniel,
> >
> > So compressed the entire directory of log files is 7.5 megs....really
> > small, so I don't think it's a question of getting stuck during
> > compression (truth be told the box doing the bro-ing is sitting right
> > next to the box I'm typing this email on...I can hear the drive whir
> > away when I stop bro and it lasts maybe 30 seconds). Also, before
> > reboot I manually stop bro...out of habit. My only thought is that
> > *maybe* the path of /usr/local/bin/ where I've symlinked the additional
> > scripts aren't seen when my startbro script is run from /etc/rc.local
> > file? In any case I can reproduce the behavior on reboot, so if there's
> > a way to debug this I'd love to give it a go. I'll research the path
> > thing on my end (Ubuntu 14.0.4) and I'll try a) rebooting and starting
> > bro manually and b) symlinking the script files to /usr/local/sbin/.
> > I'll report my findings for anyone else out there, but I kinda think
> > most people are just using broctl anyways :) Thanks again Daniel.
> >
> > James
>
>
> One other thing to check is which directory you are starting Bro from,
> because that's where Bro will create its log files (if you were
> using broctl, this should be /usr/local/bro/spool/bro).
>
> If you ever notice that you are missing logs in the archive directory
> (a subdirectory of /usr/local/bro/logs), then you'll want to check
> the directory where you were running Bro to see if it contains any
> unarchived logs (if you were using broctl to start/stop bro, then
> you'd also need to check all subdirectories of
> /usr/local/bro/spool/tmp).
So I think I may have this resolved. Yesterday I noticed that two
symlinks were bad:
create-link-for-log
make-archive-name
I've symlinked these correctly and rebooted. I manually started bro
instead of having it start in /etc/rc.local. My last test when I need
to reboot again will be to have bro autostart. Thanks all.
James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150201/9821ed29/attachment.html
More information about the Bro
mailing list