[Bro] Stats.log Growing Out of Control!!!

[Daniel Thayer]

Your spool/stats.log file became corrupt somehow, and then you started
getting "stats-to-csv failed" emails every time cron ran.  This was
preventing broctl from removing this file, which explains why you were
seeing such a fast rate of growth in the size of your
logs/stats/stats.log file (broctl cron always appends spool/stats.log
to logs/stats/stats.log).

To fix this, you could just delete the spool/stats.log file, then
you should no longer see the "stats-to-csv failed" emails.

I will improve broctl in the next release to mitigate this problem.
Thanks for reporting this issue.


On 01/19/2015 07:26 PM, Damon Rouse wrote:
> Here's the output after patching the cron.py file
>
> stats-to-csv failed
>
> ['manager ...', 'Traceback (most recent call last):', '  File
> "/opt/bro/share/broctl/scripts/stats-to-csv", line 134, in <module>',
> '    processNode(stats, wwwdir, "manager", False)', ' File
> "/opt/bro/share/broctl/scripts/stats-to-csv", line 87, in processNode',
> '    if m[1] != node:', 'IndexError: list index out of range']
>
> --
>
> [Automatically generated.]
>
>
> On Mon, Jan 19, 2015 at 3:39 PM, Daniel Thayer <dnthayer at illinois.edu
> <mailto:dnthayer at illinois.edu>> wrote:
>
>     I'd like to know why the stats-to-csv script is failing.
>     Could you apply the attached patch, and then send me
>     the contents of the "stats-to-csv failed" email?
>
>     To apply the patch you'll need to change directory to (where <prefix>
>     is the Bro install prefix directory):
>     <prefix>/lib/broctl/BroControl
>     In that directory you should see a file named "cron.py".
>
>
>
>     On 01/19/2015 02:16 PM, Damon Rouse wrote:
>
>         @Dan:  Both those files are there.
>
>         What my main issue seems to be is that my stats.log file is
>         growing by
>         20-30MB every 5 minutes when the cron runs.  I then get the
>         email below
>         in my original post.
>
>         I'm circling back here to hopefully find a resolution.  I opened a
>         thread in the Security Onion and tried limiting these events in my
>         broctl.cfg. doesn't seem to work.  I've stopped Bro, deleted the
>         stats
>         dir, did brotcl install and then start, no go there either.
>
>         Here's my SO thread for ref:
>         https://groups.google.com/__forum/#!topic/security-onion/__bdmFGn3oj24
>         <https://groups.google.com/forum/#!topic/security-onion/bdmFGn3oj24>
>
>         If anyone has any ideas or thoughts, please let me know.  Any
>         help is
>         truly appreciated!
>
>         Thanks
>         Damon
>
>         On Fri, Jan 2, 2015 at 2:16 PM, Thayer, Daniel N
>         <dnthayer at illinois.edu <mailto:dnthayer at illinois.edu>
>         <mailto:dnthayer at illinois.edu <mailto:dnthayer at illinois.edu>>__>
>         wrote:
>
>              The stats-to-csv script creates files with a ".csv" file
>         extension in
>              the directory <prefix>/logs/stats/www/  (where <prefix> is
>         the bro
>              install directory).  In order for this script to work, it
>         needs to
>              read two files:  <prefix>/spool/stats.log and
>              <prefix>/logs/stats/meta.dat
>
>
>
>
>              From: bro-bounces at bro.org <mailto:bro-bounces at bro.org>
>         <mailto:bro-bounces at bro.org <mailto:bro-bounces at bro.org>>
>              [bro-bounces at bro.org <mailto:bro-bounces at bro.org>
>         <mailto:bro-bounces at bro.org <mailto:bro-bounces at bro.org>>] on
>         behalf of
>              Damon Rouse [damonrouse at gmail.com
>         <mailto:damonrouse at gmail.com> <mailto:damonrouse at gmail.com
>         <mailto:damonrouse at gmail.com>>]
>
>              Sent: Friday, January 02, 2015 11:58 AM
>
>              To: bro at bro-ids.org <mailto:bro at bro-ids.org>
>         <mailto:bro at bro-ids.org <mailto:bro at bro-ids.org>>
>
>              Subject: [Bro] (no subject)
>
>
>
>
>
>
>              Happy New Year Everyone!!!
>
>              Has anyone ever seen the following error before?  Email
>         alerts that
>              come in looks like this:
>
>
>
>
>              Subject: [Bro] cron: stats-to-csv failed
>              Body:
>              stats-to-csv failed
>              --
>              [Automatically generated.]
>
>              I started receiving these yesterday.  They come in every 5
>         minutes
>              and I've never received them before yesterday.
>
>              Bro is running fine, my system is completely updated and
>         everything
>              looks good when I run a sostat (running BRO under Security
>         Onion).
>
>              Any insight is appreciated as I have no idea if they are
>         something I
>              should look into or not.
>
>              Thanks
>              Damon
>
>
>
>
>
>
>
>