[Bro] Stats.log Growing Out of Control!!!

[Damon Rouse]

Thanks Dan!  That worked like a charm...no emails and my
/nsm/bro/logs/stats/stats.log is no longer growing out of control.

Thanks again and I really appreciate all your help on this!

Damon

On Mon, Jan 19, 2015 at 9:20 PM, Daniel Thayer <dnthayer at illinois.edu>
wrote:

> Your spool/stats.log file became corrupt somehow, and then you started
> getting "stats-to-csv failed" emails every time cron ran.  This was
> preventing broctl from removing this file, which explains why you were
> seeing such a fast rate of growth in the size of your
> logs/stats/stats.log file (broctl cron always appends spool/stats.log
> to logs/stats/stats.log).
>
> To fix this, you could just delete the spool/stats.log file, then
> you should no longer see the "stats-to-csv failed" emails.
>
> I will improve broctl in the next release to mitigate this problem.
> Thanks for reporting this issue.
>
>
> On 01/19/2015 07:26 PM, Damon Rouse wrote:
>
>> Here's the output after patching the cron.py file
>>
>> stats-to-csv failed
>>
>> ['manager ...', 'Traceback (most recent call last):', '  File
>> "/opt/bro/share/broctl/scripts/stats-to-csv", line 134, in <module>',
>> '    processNode(stats, wwwdir, "manager", False)', ' File
>> "/opt/bro/share/broctl/scripts/stats-to-csv", line 87, in processNode',
>> '    if m[1] != node:', 'IndexError: list index out of range']
>>
>> --
>>
>> [Automatically generated.]
>>
>>
>> On Mon, Jan 19, 2015 at 3:39 PM, Daniel Thayer <dnthayer at illinois.edu
>> <mailto:dnthayer at illinois.edu>> wrote:
>>
>>     I'd like to know why the stats-to-csv script is failing.
>>     Could you apply the attached patch, and then send me
>>     the contents of the "stats-to-csv failed" email?
>>
>>     To apply the patch you'll need to change directory to (where <prefix>
>>     is the Bro install prefix directory):
>>     <prefix>/lib/broctl/BroControl
>>     In that directory you should see a file named "cron.py".
>>
>>
>>
>>     On 01/19/2015 02:16 PM, Damon Rouse wrote:
>>
>>         @Dan:  Both those files are there.
>>
>>         What my main issue seems to be is that my stats.log file is
>>         growing by
>>         20-30MB every 5 minutes when the cron runs.  I then get the
>>         email below
>>         in my original post.
>>
>>         I'm circling back here to hopefully find a resolution.  I opened a
>>         thread in the Security Onion and tried limiting these events in my
>>         broctl.cfg. doesn't seem to work.  I've stopped Bro, deleted the
>>         stats
>>         dir, did brotcl install and then start, no go there either.
>>
>>         Here's my SO thread for ref:
>>         https://groups.google.com/__forum/#!topic/security-onion/_
>> _bdmFGn3oj24
>>         <https://groups.google.com/forum/#!topic/security-onion/
>> bdmFGn3oj24>
>>
>>         If anyone has any ideas or thoughts, please let me know.  Any
>>         help is
>>         truly appreciated!
>>
>>         Thanks
>>         Damon
>>
>>         On Fri, Jan 2, 2015 at 2:16 PM, Thayer, Daniel N
>>         <dnthayer at illinois.edu <mailto:dnthayer at illinois.edu>
>>         <mailto:dnthayer at illinois.edu <mailto:dnthayer at illinois.edu>>__>
>>         wrote:
>>
>>              The stats-to-csv script creates files with a ".csv" file
>>         extension in
>>              the directory <prefix>/logs/stats/www/  (where <prefix> is
>>         the bro
>>              install directory).  In order for this script to work, it
>>         needs to
>>              read two files:  <prefix>/spool/stats.log and
>>              <prefix>/logs/stats/meta.dat
>>
>>
>>
>>
>>              From: bro-bounces at bro.org <mailto:bro-bounces at bro.org>
>>         <mailto:bro-bounces at bro.org <mailto:bro-bounces at bro.org>>
>>              [bro-bounces at bro.org <mailto:bro-bounces at bro.org>
>>         <mailto:bro-bounces at bro.org <mailto:bro-bounces at bro.org>>] on
>>         behalf of
>>              Damon Rouse [damonrouse at gmail.com
>>         <mailto:damonrouse at gmail.com> <mailto:damonrouse at gmail.com
>>         <mailto:damonrouse at gmail.com>>]
>>
>>              Sent: Friday, January 02, 2015 11:58 AM
>>
>>              To: bro at bro-ids.org <mailto:bro at bro-ids.org>
>>         <mailto:bro at bro-ids.org <mailto:bro at bro-ids.org>>
>>
>>              Subject: [Bro] (no subject)
>>
>>
>>
>>
>>
>>
>>              Happy New Year Everyone!!!
>>
>>              Has anyone ever seen the following error before?  Email
>>         alerts that
>>              come in looks like this:
>>
>>
>>
>>
>>              Subject: [Bro] cron: stats-to-csv failed
>>              Body:
>>              stats-to-csv failed
>>              --
>>              [Automatically generated.]
>>
>>              I started receiving these yesterday.  They come in every 5
>>         minutes
>>              and I've never received them before yesterday.
>>
>>              Bro is running fine, my system is completely updated and
>>         everything
>>              looks good when I run a sostat (running BRO under Security
>>         Onion).
>>
>>              Any insight is appreciated as I have no idea if they are
>>         something I
>>              should look into or not.
>>
>>              Thanks
>>              Damon
>>
>>
>>
>>
>>
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20150119/4704aa38/attachment.html