[Bro] Revisiting log rotate only

[James Lay]

On 2015-01-20 01:04 PM, Daniel Thayer wrote:
> On 01/19/2015 07:57 AM, James Lay wrote:
>> On Sat, 2015-01-17 at 07:37 -0700, James Lay wrote:
>>> Hey all,
>>>
>>> I posted about this last August here:
>>>
>>> 
>>> http://mailman.icsi.berkeley.edu/pipermail/bro/2014-August/007329.html
>>>
>>> I also noticed someone have a disappearing log event which I have 
>>> seen
>>> before  as well here:
>>>
>>> 
>>> http://mailman.icsi.berkeley.edu/pipermail/bro/2015-January/007935.html
>>>
>>> I documented my process on installing bro on Ubuntu 14.04 using 
>>> just
>>> log rotation below:
>>>
>>> sudo apt-get -y install cmake
>>> sudo apt-get -y install python-dev
>>> sudo apt-get -y install swig
>>> cp /usr/local/bro/share/bro/site
>>> cp /opt/bin/startbro <- command line bro with long --filter line
>>> cp /opt/bin/startbro to /etc/rc.local
>>> sudo ln -s /usr/local/bro/bin/bro /usr/local/bin/
>>> sudo ln -s /usr/local/bro/bin/bro-cut /usr/local/bin/
>>> sudo ln -s /usr/local/bro/bin/broctl /usr/local/bin/
>>> sudo ln -s /usr/local/bro/share/broctl/scripts/archive-log 
>>> /usr/local/bin/
>>> sudo ln -s /usr/local/bro/share/broctl/scripts/broctl-config.sh
>>> /usr/local/bin/
>>> sudo ln -s /usr/local/bro/share/broctl/scripts/create-link-for-log
>>> /usr/local/bin/
>>> sudo ln -s /usr/local/bro/share/broctl/scripts/make-archive-name
>>> /usr/local/bin/
>>> git clone https://github.com/jonschipp/mal-dnssearch.git
>>> sudo make install
>>>
>>> specifics on log rotate only:
>>>
>>> add the below to local.bro
>>> redef Log::default_rotation_interval = 86400 secs;
>>> redef Log::default_rotation_postprocessor_cmd = "archive-log";
>>> edit the below in broctl.cfg
>>> MailTo = jlay at slave-tothe-box.net <mailto:jlay at slave-tothe-box.net>
>>> LogRotationInterval = 86400
>>> sudo /usr/local/bro/bin/broctl install
>>>
>>> Besides the edits to broctl.cfg, file locations are the default.  
>>> The
>>> above works well usually...it's after a reboot I have found things 
>>> go
>>> bad.  Usually logs get rotated at midnight and I get an email with
>>> statistics, just what I need.  I rebooted the machine on the 13, 
>>> and
>>> that's the last email or log rotation I got....this morning I see
>>> current has files and my logstash instance has data so I believe 
>>> the
>>> rotation got..."stuck".  I'm kicking myself for not heading/tailing
>>> the files first, but after issuing a "sudo killall bro", those file 
>>> in
>>> current vanished, no directory was created, and I received no 
>>> email,
>>> that data is now gone (no big deal as this is at home).  I decided 
>>> to
>>> run broctl install again, then start and kill bro one more time.  
>>> At
>>> that point, I got a new directory with log rotation and an email 
>>> with
>>> minutes or so of stats.  Please let me know if there's something I 
>>> can
>>> do on my end to trouble shoot.  Thank you.
>>>
>>> James
>>> _______________________________________________
>>> Bro mailing list
>>> bro at bro-ids.org  <mailto:bro at bro-ids.org>
>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>> Confirming that this method is no longer working.  Heading my 
>> connlog
>> file I see:
>>
>> #open 2015-01-19-00-00-05
>>
>> my /usr/local/bro/logs is completely missing Jan 18th.  From my 
>> broctl.cfg:
>>
>> SpoolDir = /usr/local/bro/spool
>> LogDir = /usr/local/bro/logs
>> LogRotationInterval = 86400
>>
>>  From my /usr/local/bro/share/bro/site/local.bro:
>>
>> redef Log::default_rotation_interval = 86400 secs;
>> redef Log::default_rotation_postprocessor_cmd = "archive-log";
>>
>> Anything else I can do to debug this?  Thank you.
>>
>> James
>
> Are you using broctl to start and stop Bro?  What does 
> /opt/bin/startbro
> do?

Thanks for looking Daniel.  I am starting this with the below:

/usr/local/bro/bin/bro --no-checksums -i eth0 -i ppp0 --filter '( large 
filter line here)' local "Site::local_nets += { 192.168.1.0/24 }"

I'm not using broctl.  The only small portion that I am is for the log 
rotation as outlined in the email thread.  After killing and starting 
bro yesterday, this morning at midnight logs got rotated and I got my 
report email.  This appears to happen after a complete reboot of the 
device.  It's very odd.  Thanks again.

James