[Bro] LogExpireInterval not respected?
Jesse Bowling
jessebowling at gmail.com
Wed Feb 13 09:30:06 PST 2013
So my solution is a two-part one...The easy part first: the cron job wasn't
running.
In the document (http://www.bro-ids.org/documentation/quickstart.html) I
used for setup, it was suggested to put the following into crontab:
0-59/5 * * * * /usr/local/bro/bin/broctl cron
However since I was putting this under /etc/cron.d/bro (on a RHEL 6 box),
what I actually need is:
0-59/5 * * * * root /usr/local/bro/bin/broctl cron
If you don't specify the user, it won't run (or complain).
Derp.
So once that was fixed, I ran into a second issue: after the first instance
ran, subsequent runs would not be able to get the lock (generating a
"cannot get lock" error email). Looking at the issue further, it appears
that there are is a capstats instance running at 99%. Killing that job
allowed the initial cron to finish, and my directories were deleted as
expected.
root 21847 21661 0 11:35 ? 00:00:00 bash
/usr/local/bro/share/broctl/scripts/helpers/run-cmd
/usr/local/bro/bin/capstats -i p2p1;p2p2;p2p3;p2p4 -I 5 -n 1
root 21848 21847 99 11:35 ? 00:11:29 /usr/local/bro/bin/capstats
-i p2p1
Seeing that the bash script
/usr/local/bro/share/broctl/scripts/helpers/run-cmd is the dead simple:
echo 0 # Fix me. Should be real return code.
eval $@ 2>&1
echo ~~~
I can surmise the problem: Because my interface specification requires the
use of ';', bash is breaking the command up before it should and capstats
doesn't know it should quit...The format I'm using (p2p1;p2p2;p2p3;p2p4) is
making use of PF_RING to listen to all these interfaces simultaneously. For
snort I have to quote it to prevent it being broken up and I suspected
something similar is required here as well.
Sure enough, patching control.py to quote the interfaces allows the job to
finish and the stats to get written, with one error message in the stats
file: "1360775469.46 proxy-1 error error proxy-1: unexpected capstats
output: capstats [Options] -i interface".
So while this apparently fixes my issue:
--- control.py 2013-02-13 12:08:00.514656601 -0500
+++ control_mod.py 2013-02-13 12:09:38.382663593 -0500
@@ -808,7 +808,7 @@
for (addr, interface) in hosts.keys():
node = hosts[addr, interface]
- capstats = [config.Config.capstatspath, "-i", interface, "-I",
str(interval), "-n", "1"]
+ capstats = [config.Config.capstatspath, "-i", '"' + interface +
'"', "-I", str(interval), "-n", "1"]
# Unfinished feature: only consider a particular MAC. Works here for
capstats
# but Bro config is not adapted currently so we disable it for now.
I cannot speak to how this might affect others, the system in general, or
where else this issue might crop up. I suspect that anywhere that involves
bash + interface names is likely to suffer unexpected results due to this
PF_RING style invocation...
I plan to let this run for now, perhaps others might comment on the
advisability of this?
Cheers,
Jesse
On Wed, Feb 13, 2013 at 10:33 AM, Jesse Bowling <jessebowling at gmail.com>wrote:
> Turned debug on, install && check'ed everything, ensured 'broctl cron' is
> in cron, ensured two jobs not running, but still my logs don't expire...
>
> More hints anyone?
>
> Cheers,
>
> Jesse
>
>
> On Tue, Feb 12, 2013 at 5:27 PM, Daniel Thayer <dnthayer at illinois.edu>wrote:
>
>> Have you tried looking at the <install_prefix>/spool/debug.**log file?
>> If that file doesn't exist, then uncomment the "Debug = 1" line
>> in your broctl.cfg file.
>>
>>
>>
>>
>> On 02/12/2013 11:18 AM, Jesse Bowling wrote:
>>
>>> Thanks again Tyler.
>>>
>>> Unfortunately, that was not the case here (although I have observed the
>>> same previously). Disabling the cron job and running it manually with
>>> strace, we end up hanging after a few minutes:
>>>
>>> read(8, "W", 1) = 1
>>> read(8, "r", 1) = 1
>>> read(8, "i", 1) = 1
>>> read(8, "t", 1) = 1
>>> read(8, "e", 1) = 1
>>> read(8, " ", 1) = 1
>>> read(8, "p", 1) = 1
>>> read(8, "a", 1) = 1
>>> read(8, "c", 1) = 1
>>> read(8, "k", 1) = 1
>>> read(8, "e", 1) = 1
>>> read(8, "t", 1) = 1
>>> read(8, "s", 1) = 1
>>> read(8, " ", 1) = 1
>>> read(8, "t", 1) = 1
>>> read(8, "o", 1) = 1
>>> read(8, " ", 1) = 1
>>> read(8, "f", 1) = 1
>>> read(8, "i", 1) = 1
>>> read(8, "l", 1) = 1
>>> read(8, "e", 1) = 1
>>> read(8, "\n", 1) = 1
>>> wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>>> read(8, "\n", 1) = 1
>>> wait4(44650, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>>> read(8, "~", 1) = 1
>>> read(8, "~", 1) = 1
>>> read(8, "~", 1) = 1
>>> read(8, "\n", 1) = 1
>>> wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>>> read(32, "0", 1) = 1
>>> read(32, "\n", 1) = 1
>>> wait4(44707, 0x7fffb630f2e4, WNOHANG, NULL) = 0
>>> read(32,
>>>
>>> Any other hints?
>>>
>>> Cheers,
>>>
>>> Jesse
>>>
>>>
>>> On Tue, Feb 12, 2013 at 9:55 AM, Tyler T. Schoenke
>>> <tyler.schoenke at colorado.edu <mailto:tyler.schoenke@**colorado.edu<tyler.schoenke at colorado.edu>>>
>>> wrote:
>>>
>>> broctl cron typically doesn't give output. If it is hanging, you
>>> should check for other instances of broctl cron running and kill
>>> them. They will sometimes log jam. I haven't figured out why that
>>> happens.
>>>
>>> Tyler
>>>
>>> --
>>> Tyler Schoenke
>>> Network Security Program Manager
>>> IT Security Office
>>> University of Colorado at Boulder
>>>
>>>
>>> On 2/11/13 9:40 PM, Jesse Bowling wrote:
>>>
>>>> Hi Tyler,
>>>>
>>>> Thanks for the response.
>>>>
>>>> Yes, I have ensured that these have been run...I've also tried
>>>> just running 'broctl cron' manually, but I get no output and it
>>>> never seems to quit (or at least, outlasts my patience)...Any
>>>> other hints?
>>>>
>>>> Cheers,
>>>>
>>>> Jesse
>>>>
>>>> On Mon, Feb 11, 2013 at 4:34 PM, Tyler T. Schoenke
>>>> <tyler.schoenke at colorado.edu <mailto:tyler.schoenke@**colorado.edu<tyler.schoenke at colorado.edu>
>>>> >>
>>>>
>>>> wrote:
>>>>
>>>> Have you run broctl install && broctl check? I always
>>>> forget to do
>>>> that after modifying LogExpireInterval.
>>>>
>>>> Tyler
>>>>
>>>> --
>>>> Tyler Schoenke
>>>> Network Security Program Manager
>>>> IT Security Office
>>>> University of Colorado at Boulder
>>>>
>>>>
>>>> On 2/11/13 1:57 PM, Jesse Bowling wrote:
>>>> > Hi,
>>>> >
>>>> > In my /usr/local/bro/etc/broctl.cfg I've specified:
>>>> >
>>>> > LogExpireInterval = 14
>>>> >
>>>> > Additionally in /etc/cron.d/bro I've specified:
>>>> >
>>>> > 0-59/5 * * * * /usr/local/bro/bin/broctl cron
>>>> >
>>>> > However I've found that I have more daily directories
>>>> present that 14
>>>> > days...What configuration options should I be checking to
>>>> troubleshoot
>>>> > this problem?
>>>> >
>>>> > Thanks,
>>>> >
>>>> > Jesse
>>>> >
>>>> > --
>>>> > Jesse Bowling
>>>> >
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Jesse Bowling
>>>>
>>>>
>>>
>>>
>>> --
>>> Jesse Bowling
>>>
>>>
>>>
>>> ______________________________**_________________
>>> Bro mailing list
>>> bro at bro-ids.org
>>> http://mailman.ICSI.Berkeley.**EDU/mailman/listinfo/bro<http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro>
>>>
>>>
>>
>
>
> --
> Jesse Bowling
>
>
--
Jesse Bowling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20130213/c7155e19/attachment.html
More information about the Bro
mailing list