[Bro] Manager swapping..

[fatema bannatwala]

The issue got resolved. :)
I rebuilt Bro with tcmalloc, for efficiet memory usage, on the cluster and
it seems to resolve the heavy memory usage on the manager.
After that, when I disabled the scan scripts in the cluster, the memory
usage dropped down to ~5%, and
when it's enabled the memory usage toggles around ~25% (i.e ~25-28GB on
manager) and around ~31GB
on workers, so far the cluster seems to be stable with regard to memory
usage.

Thank you all for the help, for resolving the issue. Appreciate it :)

-Fatema.

On Thu, Mar 23, 2017 at 1:24 PM, fatema bannatwala <
fatema.bannatwala at gmail.com> wrote:

> Thanks Sanjay for suggestions.I already have the @load
> protocols/ssl/validate-certs disabled in local.bro. :)
>
> I was looking into the reporter logs and see some logs like this:
>
> Some INFO logs:
>
> 1490288453.884071       Reporter::INFO  Got counters:
> [new_conn_counter=4394103, is_catch_release_active=7433937,
> known_scanners_counter=0, not_scanner=2439888, darknet_counter=64358,
> not_darknet_counter=3114626, already_scanner_counter=0,
> filteration_entry=0, filteration_success=1543038,
> c_knock_filterate=3548445, c_knock_checkscan=0, c_knock_core=0,
> c_land_filterate=22317, c_land_checkscan=0, c_land_core=0,
> c_backscat_filterate=3548445, c_backscat_checkscan=0, c_backscat_core=0,
> c_addressscan_filterate=3548445, c_addressscan_checkscan=0,
> c_addressscan_core=0, check_scan_counter=0, worker_to_manager_counter=0,
> run_scan_detection=0, check_scan_cache=1543038, event_peer=worker-1-15]
>  manager
>
> 1490288454.925040       Reporter::INFO  known_scanners_inactive:
> [scanner=94.51.38.120, status=T, detection=KnockKnockScan,
> detect_ts=1490202054.11266, event_peer=manager, expire=F]   manager
> 1490288454.925040       Reporter::INFO  known_scanners_inactive:
> [scanner=171.249.5.188, status=T, detection=KnockKnockScan,
> detect_ts=1490202053.07045, event_peer=manager, expire=F]  manager
>
> Ans these error logs:
> 0.000000        Reporter::ERROR field value missing
> [Scan::geoip_info$country_code]     /usr/local/bro/2.5/share/bro/
> site/scan-NG-master/scripts/./scan-summary.bro, line 292
> 0.000000        Reporter::ERROR value used but not set
> (Scan::c_landmine_scan_summary)  /usr/local/bro/2.5/share/bro/
> site/scan-NG-master/scripts/./check-landmine.bro, line 33
> 0.000000        Reporter::ERROR value used but not set
> (Scan::c_landmine_scan_summary)  /usr/local/bro/2.5/share/bro/
> site/scan-NG-master/scripts/./check-landmine.bro, line 33
>
> Are they anywhere related to the issue?
>
> Thanks,
> Fatema.
>
> On Thu, Mar 23, 2017 at 10:56 AM, fatema bannatwala <
> fatema.bannatwala at gmail.com> wrote:
>
>> Nope, based on our previous discussion in another thread,
>> I disabled the misc/scan, and loaded scan-NG-master script.
>> I always thought that the scripts would have more load on workers than
>> manager.
>> When I was seeing memory issues on workers, I stopped using misc/scan and
>> switched to
>> the scan-NG script.
>> Didn't know that it would impact manager performance as well, hmm.
>>
>> On Thu, Mar 23, 2017 at 10:43 AM, Azoff, Justin S <jazoff at illinois.edu>
>> wrote:
>>
>>>
>>> > On Mar 23, 2017, at 7:40 AM, fatema bannatwala <
>>> fatema.bannatwala at gmail.com> wrote:
>>> >
>>> > Thanks Justin for the input :)
>>> >
>>> > I restarted Bro after disabling some of the protocols logging (like
>>> rdp, syslog, snmp etc) yesterday afternoon,
>>> > as the machine is in production and needed to be fixed kind of "ASAP".
>>> Hence couldn't get a chance to run
>>> > the broctl top while having the issue, I know you have mentioned it
>>> couple of times in past to use "broctl top"
>>> > instead of normal "top", but magically I keep forgetting to do that, I
>>> think I should come up with by BRO troubleshoot
>>> > guide, which should list some basic troubleshooting commands that you
>>> guys suggest in these emails :)
>>> >
>>> > Anyways, I did run the command today, and it looks like the manager
>>> process is overwhelmed,
>>> > hmm I thought that it might logger that might be having issues
>>> catching up on the load, but I was wrong:
>>> >
>>> > $ sudo -u bro /usr/local/bro/2.5/bin/broctl top manager logger
>>> > Name         Type    Host   Pid     Proc    VSize  Rss  Cpu   Cmd
>>> > logger       logger   IDS   60928    parent    2G    90M  17%  bro
>>> > logger       logger   IDS   60932    child   522M   246M   5%  bro
>>> > manager      manager  IDS   60990    child     1G   257M  35%  bro
>>> > manager      manager  IDS   60973    parent  222G    31G  23%  bro
>>> >
>>> > It makes me think, if there is some memory leak issue with manager.
>>>
>>> Are you loading misc/detect-traceroute or misc/scan in your local.bro?
>>>
>>> --
>>> - Justin Azoff
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170424/30a6feea/attachment.html