[Bro-Dev] Performance Issues after the fe7e1ee commit?

Azoff, Justin S jazoff at illinois.edu
Wed Jun 6 12:03:16 PDT 2018


> On Jun 6, 2018, at 12:54 PM, McMullan, Tim <Tim.McMullan at sig.com> wrote:
> 
> We are running into performance issues (30x slower) since the Broker patch (fe7e1ee) –
>  
> We have 40G connections tapped from our storage filers feeding multiple Bro instances which analyze specifically only NFS and SMB traffic; all other analyzers are disabled.  With the broker patch we are seeing processing times for a ~1GB pcap jump from around 2 seconds to over  1 minute. Profiling Bro, it looks like the culprit is the new Actor functions --
>  
> # Before patch
> Overhead  Shared Object          Symbol
> 14.57%  [kernel]              [k] copy_user_enhanced_fast_string
>    3.20%  bro                   [.] EventHandler::operator bool
>    2.99%  bro                   [.] _siphash
>    2.89%  bro                   [.] Dictionary::Lookup
>  
> # After patch
> Overhead  Shared Object          Symbol
>    5.71%  [kernel]               [k] native_write_msr_safe
>    3.84%  libcaf_core.so.0.15.7  [.] caf::scheduler::worker<caf::policy::work_stealing>::run
>    3.71%  libcaf_core.so.0.15.7  [.] caf::detail::double_ended_queue<caf::resumable>::take_head
>    3.29%  [kernel]               [k] _raw_spin_lock
>  
> Is the Bro development team still optimizing the Broker/Actor framework? It might be helpful to have a way to disable Broker for those of us who haven’t migrated to it yet.
>  
> #  ~1GB file time (old)
> $ time /hostname/bro-devel/bin/bro -r 20180606-1049-prodfilers-truncated_00000_20180606104904.pcap  master.bro
>  
> real    0m2.294s
> user    0m1.862s
> sys     0m0.385s
>  
> #  ~1GB file time  (new)
> $ time /hostname/bro-devel/bin/bro -r 20180606-1049-prodfilers-truncated_00000_20180606104904.pcap master.bro
>  
> real    1m11.458s
> user    0m58.933s
> sys     1m34.074s

Well, I can at least reproduce that here on one of the standard test pcap files and no scripts loaded:

$ /usr/local/bin/bro --version
/usr/local/bin/bro version 2.5.2
$ time /usr/local/bin/bro -b -r ~/src/bro/testing/external/bro-testing/2009-M57-day11-18.trace

real	0m2.317s
user	0m1.874s
sys	0m0.327s

$ . ~/src/bro/build/bro-path-dev.sh
$ bro --version
bro version 2.5-648
$ time bro -b -r ~/src/bro/testing/external/bro-testing/2009-M57-day11-18.trace

real	1m5.523s
user	0m30.565s
sys	0m32.661s

and the same thing in perf:

Samples: 270K of event 'cycles:u', Event count (approx.): 131984875188
Overhead  Command  Shared Object                 Symbol
   8.95%  bro      libcaf_core.so.0.15.7         [.] caf::detail::double_ended_queue<caf::resumable>::take_tail
   5.90%  bro      libcaf_core.so.0.15.7         [.] caf::detail::double_ended_queue<caf::resumable>::take_head
   4.92%  bro      libcaf_core.so.0.15.7         [.] caf::scheduler::worker<caf::policy::work_stealing>::run
   2.67%  bro      libc-2.17.so                  [.] __GI___libc_nanosleep
   1.98%  bro      libtcmalloc.so.4.4.5          [.] operator new[]
   1.79%  bro      libc-2.17.so                  [.] __sleep
   1.68%  bro      libtcmalloc.so.4.4.5          [.] tc_deletearray_nothrow
   1.47%  bro      libc-2.17.so                  [.] __libc_disable_asynccancel
   1.28%  bro      libstdc++.so.6.0.19           [.] std::this_thread::__sleep_for
   1.10%  bro      libc-2.17.so                  [.] usleep
   1.05%  bro      bro                           [.] ones_complement_checksum



I haven't noticed a huge difference on a real multi process cluster, at least not a 30x change, so this is odd that pcap processing is so much slower.
Especially since broker should be completely disabled when pcap files are being read and caf shouldn't even be doing anything.

— 
Justin Azoff





More information about the bro-dev mailing list