[Bro] Possible system tweak for reducing memory usage

[Azoff, Justin S]

TL;DR: It's possible that transparent huge pages and bro do not get along, try doing a

    # on all nodes
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
    # then
    broctl restart

There are ways to make that permanent if it helps.


I've been doing some research to try to figure out why some people have more memory issues than others.  I think the kernel feature Transparent Huge Pages (THP) and bro may not get along well.  It's supposed to help performance for memory allocations, but many services recommend disabling it (mongodb, redis, mysql).  For example:

> Transparent Huge Pages (THP) is a Linux memory management system that reduces the overhead of Translation Lookaside Buffer (TLB) lookups on machines with large amounts of memory by using larger memory pages.
> 
> However, database workloads often perform poorly with THP, because they tend to have sparse rather than contiguous memory access patterns. You should disable THP on Linux machines to ensure best performance with MongoDB.

Bro memory allocations can best described as unpredictable, especially on 'custer in a box' deployments.

On our systems, disabling it drops bro worker memory usage by 20% and manager/logger usage by even more, but since we only have one of those it's harder to compare.  For workers I disabled THP on half the nodes, and the post bro restart memory usage is consistently lower.

-- 
- Justin Azoff