[Bro] intel.log file stops getting generated.

[fatema bannatwala]

It turns out to be the performance issue.
I restarted the bro cluster and it started getting generated, but have
another issue:
The bro sensors are utilizing almost 100% memory as well as some part of
swap.

We recently have upgraded the kernel and centos to 7.3 on bro cluster, as
well as using latest pfring v6.4.1
We have 4 bro sensors each with 132G of memory and 24 core cpu @ 2.50GHz
with 48 On-line CPU(s) (0-17)), and each running 22 bro processes.

The memory usage on the sensors is around ~129G
              total        used        free      shared  buff/cache
available
Mem:      131921372   129912824      801672       11224     1206876
1270972
Swap:       8388600     3378312     5010288

The peak traffic we see usually toggles around 6-7Gbps.
Don't know if this started happening after the upgrade to Bro 2.5, but the
sensors become un-responsive because of this.

I have checked that the ethtool settings on sensors are set to: rx off tx
off tso off sg off gso off gro off
Also, have commented out some scripts, that I used to run with 2.4.1, but
no luck with memory usage.

Any leads/suggestions?

Thanks,
Fatema.


On Tue, Jan 24, 2017 at 2:20 PM, fatema bannatwala <
fatema.bannatwala at gmail.com> wrote:

> Hi All,
>
> Running Bro 2.5, everything is working except intel.log file stop getting
> generated.
> Last event in that file was around 12:45pm today, and after it got rotated,
> I didn't see intel.log for 1pm hour and still no log for intel.log in the
> current log dir.
>
> Don't know why all of a sudden intel.log stopped geting generated.
>
> I checked:
> 1. The conn.log, and seeing the connections from IPs listed as bad in
> intel feed.
> $ less bad-IP.intel | grep "61.240.xx.yy"
> 61.240.xx.yy   Intel::ADDR     scanner 85      csirtg.io
>
> $ less conn.log | grep "61.240.144.65"
> 1485280794.930507       CzUCmv3TFKLcYxFps1      61.240.xx.yy   40805
> 128.4.107.206   8081    tcp     -       -       -       -       S0      F
>     T       0       S       1       40      0       0   ( empty)
>
> 2. Permissions on the intel input files are fine,i.e bro readable.
> 3. No major activity related to Bro happened during 12:45ish, that can
> impact any Bro processing.
>
> Any leads/suggestions?
>
> Thanks,
> Fatema.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20170125/2cd78461/attachment-0001.html