[Bro] Memory Issue with Bro
Aaron Lewis
the.warl0ck.1989 at gmail.com
Sun Oct 25 22:21:03 PDT 2015
Thanks Eric,
I tried that and it's not working for me ... I even commented out
every protocol analyser. memory usage still goes up, just slower in
time
Perhaps the amount of traffic is too much for bro ids. I'm gonna send
another email
On Sun, Oct 25, 2015 at 12:36 AM, Eric Ooi <ericooi at gmail.com> wrote:
> Hey Aaron,
>
> I did it by commenting out "@load intel" in
> /opt/bro/share/bro/site/local.bro. Give that a try and see if it works for
> you.
>
> Eric
>
> On Sat, Oct 24, 2015 at 10:51 AM, Aaron Lewis <the.warl0ck.1989 at gmail.com>
> wrote:
>>
>> Hey Eric,
>>
>> How exactly did you turn off intel log? I tried to comment out this
>> line, but nothing changed on my sensor,
>>
>> @load policy/frameworks/intel/seen
>>
>> Am I wrong? I was following this link:
>> https://www.bro.org/sphinx/frameworks/intel.html
>>
>>
>>
>> On Sat, Oct 24, 2015 at 10:39 PM, Eric Ooi <ericooi at gmail.com> wrote:
>> > I have two Security Onion sensors running 2.4, one monitors a
>> > combination of 100M general office internet traffic + 10G network, the other
>> > monitors four 1G networks which includes a publicly available website and
>> > lots of syslog and SMTP traffic. I ran the default Security Onion
>> > configuration on both and noticed the sensor monitoring four 1G networks
>> > would run out of memory over the course of a few hours. I spent a day
>> > turning off various analyzers until I isolated it to the intel analyzer.
>> > Ever since I turned intel.log off for that sensor, it’s run great for weeks.
>> > I tried adding intel feeds (via CriticalStack) and using a blank intel file,
>> > with no luck. Simply having the intel analyzer on always resulted in memory
>> > loss over time. I’m guessing it has something to do with the type of
>> > traffic that particular sensor sees (more HTTP, syslog, and SMTP), but I’m
>> > not entirely sure.
>> >
>> > Don’t know if it’s related, but just thought I’d share my experience
>> > with Bro memory issues.
>> >
>> > Eric
>> >
>> >> On Oct 23, 2015, at 11:38 AM, Mike Waite <mfw113 at psu.edu> wrote:
>> >>
>> >> After 30 min things look better, I will let you know how the rest of it
>> >> makes out after a bit.
>> >>
>> >> Oct 23 12:09:52 manager child - - - info
>> >> selects=100000 canwrites=97046 pending=0
>> >> Oct 23 12:11:54 manager child - - - info
>> >> selects=200000 canwrites=97046 pending=0
>> >> Oct 23 12:14:04 manager child - - - info
>> >> selects=300000 canwrites=97046 pending=0
>> >> Oct 23 12:14:43 manager child - - - info
>> >> selects=400000 canwrites=97046 pending=0
>> >> Oct 23 12:15:20 manager child - - - info
>> >> selects=500000 canwrites=97046 pending=0
>> >> Oct 23 12:15:54 manager child - - - info
>> >> selects=600000 canwrites=97046 pending=0
>> >> Oct 23 12:16:38 manager child - - - info
>> >> selects=700000 canwrites=97046 pending=0
>> >> Oct 23 12:17:41 manager child - - - info
>> >> selects=800000 canwrites=97046 pending=0
>> >> Oct 23 12:19:03 manager child - - - info
>> >> selects=900000 canwrites=97046 pending=0
>> >> Oct 23 12:20:46 manager child - - - info
>> >> selects=1000000 canwrites=97046 pending=0
>> >> Oct 23 12:23:04 manager child - - - info
>> >> selects=1100000 canwrites=97046 pending=0
>> >> Oct 23 12:25:10 manager child - - - info
>> >> selects=1200000 canwrites=104987 pending=0
>> >> Oct 23 12:26:40 manager child - - - info
>> >> selects=1300000 canwrites=104987 pending=0
>> >> Oct 23 12:28:13 manager child - - - info
>> >> selects=1400000 canwrites=104987 pending=0
>> >> Oct 23 12:31:12 manager child - - - info
>> >> selects=1600000 canwrites=110134 pending=0
>> >> Oct 23 12:32:24 manager child - - - info
>> >> selects=1700000 canwrites=110134 pending=0
>> >> Oct 23 12:34:03 manager child - - - info
>> >> selects=1800000 canwrites=110134 pending=0
>> >> Oct 23 12:35:12 manager child - - - info
>> >> selects=1900000 canwrites=110134 pending=0
>> >> Oct 23 12:36:15 manager child - - - info
>> >> selects=2000000 canwrites=110134 pending=0
>> >> Oct 23 12:37:31 manager child - - - info
>> >> selects=2100000 canwrites=110134 pending=0
>> >>
>> >>
>> >> --
>> >> Mike Waite
>> >> CyberSecurity Intrusion Analyst
>> >> Office of Information Security
>> >> The Pennsylvania State University
>> >> ↪ 15-10-23 11:09:58, Seth Hall <seth at icir.org>:
>> >>> Mike, could you back out that patch and try my branch,
>> >>> topic/seth/remove-flare ?
>> >>>
>> >>> .Seth
>> >>>
>> >>>
>> >>>> On Oct 23, 2015, at 10:19 AM, Azoff, Justin S <jazoff at illinois.edu>
>> >>>> wrote:
>> >>>>
>> >>>> Well that doesn't look great, but could be a lot worse. Hard to say
>> >>>> without knowing what it looked like before the patch.
>> >>>>
>> >>>> The fact that pending ever goes down at all is a good sign, but
>> >>>> pending=0 is really the optimal state.
>> >>>>
>> >>>> --
>> >>>> - Justin Azoff
>> >>>>
>> >>>>> On Oct 23, 2015, at 9:21 AM, Mike Waite <mfw113 at psu.edu> wrote:
>> >>>>>
>> >>>>> Patch applied, after 15 minutes I am seeing
>> >>>>>
>> >>>>> Oct 23 09:00:43 manager child - - - info
>> >>>>> selects=300000 canwrites=216206 pending=0
>> >>>>> Oct 23 09:01:29 manager child - - - info
>> >>>>> selects=400000 canwrites=216206 pending=0
>> >>>>> Oct 23 09:02:08 manager child - - - info
>> >>>>> selects=500000 canwrites=216552 pending=0
>> >>>>> Oct 23 09:02:49 manager child - - - info
>> >>>>> selects=600000 canwrites=216557 pending=0
>> >>>>> Oct 23 09:03:34 manager child - - - info
>> >>>>> selects=700000 canwrites=216557 pending=0
>> >>>>> Oct 23 09:04:29 manager child - - - info
>> >>>>> selects=800000 canwrites=255305 pending=4007
>> >>>>> Oct 23 09:05:21 manager child - - - info
>> >>>>> selects=900000 canwrites=355305 pending=6593
>> >>>>> Oct 23 09:06:13 manager child - - - info
>> >>>>> selects=1000000 canwrites=455305 pending=6003
>> >>>>> Oct 23 09:07:04 manager child - - - info
>> >>>>> selects=1100000 canwrites=555305 pending=3077
>> >>>>> Oct 23 09:07:55 manager child - - - info
>> >>>>> selects=1200000 canwrites=640438 pending=3399
>> >>>>> Oct 23 09:08:45 manager child - - - info
>> >>>>> selects=1300000 canwrites=740438 pending=3163
>> >>>>> Oct 23 09:09:36 manager child - - - info
>> >>>>> selects=1400000 canwrites=840438 pending=5245
>> >>>>> Oct 23 09:10:25 manager child - - - info
>> >>>>> selects=1500000 canwrites=940438 pending=6027
>> >>>>> Oct 23 09:11:15 manager child - - - info
>> >>>>> selects=1600000 canwrites=1040438 pending=6713
>> >>>>> Oct 23 09:12:01 manager child - - - info
>> >>>>> selects=1700000 canwrites=1140438 pending=5713
>> >>>>> Oct 23 09:12:50 manager child - - - info
>> >>>>> selects=1800000 canwrites=1240438 pending=6747
>> >>>>> Oct 23 09:13:39 manager child - - - info
>> >>>>> selects=1900000 canwrites=1340438 pending=7417
>> >>>>> Oct 23 09:14:32 manager child - - - info
>> >>>>> selects=2000000 canwrites=1440438 pending=13117
>> >>>>> Oct 23 09:15:10 manager child - - - info
>> >>>>> selects=2100000 canwrites=1540438 pending=20825
>> >>>>> Oct 23 09:15:59 manager child - - - info
>> >>>>> selects=2200000 canwrites=1640438 pending=18539
>> >>>>> Oct 23 09:16:47 manager child - - - info
>> >>>>> selects=2300000 canwrites=1740438 pending=15881
>> >>>>> Oct 23 09:17:35 manager child - - - info
>> >>>>> selects=2400000 canwrites=1840438 pending=15389
>> >>>>> Oct 23 09:18:28 manager child - - - info
>> >>>>> selects=2500000 canwrites=1940438 pending=16685
>> >>>>> Oct 23 09:19:18 manager child - - - info
>> >>>>> selects=2600000 canwrites=2040438 pending=17031
>> >>>>>
>> >>>>>
>> >>>>> I will let you know about the mem usage after a bit
>> >>>>>
>> >>>>> --
>> >>>>> Mike Waite
>> >>>>> CyberSecurity Intrusion Analyst
>> >>>>> Office of Information Security
>> >>>>> The Pennsylvania State University
>> >>>>> ↪ 15-10-22 10:22:18, Azoff, Justin S <jazoff at illinois.edu>:
>> >>>>>>> On Oct 22, 2015, at 8:12 AM, Mike Waite <mfw113 at psu.edu> wrote:
>> >>>>>>>
>> >>>>>>> I know we are still seeing issues with the manager child proccess.
>> >>>>>>> The process will consume over 200GB of RAM in 8 hours.
>> >>>>>>>
>> >>>>>>
>> >>>>>> Give the attached patch a try.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> Monitor by using
>> >>>>>>
>> >>>>>> cat logs/current/communication.log |egrep 'manager.child'
>> >>>>>>
>> >>>>>> And check to see if pending=0 or at least not growing.
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> - Justin Azoff
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Bro mailing list
>> >>>> bro at bro-ids.org
>> >>>> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> Seth Hall
>> >>> International Computer Science Institute
>> >>> (Bro) because everyone has a network
>> >>> http://www.bro.org/
>> >>>
>> >> _______________________________________________
>> >> Bro mailing list
>> >> bro at bro-ids.org
>> >> http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>> >
>> >
>> > _______________________________________________
>> > Bro mailing list
>> > bro at bro-ids.org
>> > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro
>>
>>
>>
>> --
>> Best Regards,
>> Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/
>> Finger Print: 9F67 391B B770 8FF6 99DC D92D 87F6 2602 1371 4D33
>
>
--
Best Regards,
Aaron Lewis - PGP: 0x13714D33 - http://pgp.mit.edu/
Finger Print: 9F67 391B B770 8FF6 99DC D92D 87F6 2602 1371 4D33
More information about the Bro
mailing list