From jazoff at illinois.edu Fri Jun 1 08:37:17 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 1 Jun 2018 15:37:17 +0000 Subject: [Bro-Dev] Broker has landed in master, please test In-Reply-To: <20180522142623.GE34893@icir.org> References: <20180522142623.GE34893@icir.org> Message-ID: <525697DD-8126-47CC-80C0-004845CBD8E2@illinois.edu> On May 22, 2018, at 10:26 AM, Robin Sommer > wrote: With such a large change, I'm sure there'll be some more kinks to iron out still; that's where we need everybody's help. If you have an environment where you can test drive new Bro versions, please give this a try. We're interested in any feedback you have, both specific issues you encounter (best to file tickets) and general experiences with the new version, including in particular any observations about performance (best to send to this list). Have this running on a few clusters now, so far it's been really good. This graph shows how stable it has been on one of of clusters. [cid:57EB6001-3898-4F59-A4D9-BF080307A62F at home] On that cluster manager node we were seeing a random cpu+traffic+memory spike on one of the proxies that would eventually be killed by the OOM killer.. then it would restart and get killed again shortly after that. A larger cluster would see the same spikes but it had 4x the ram and wouldn't OOM. Since switching to the broker version around 5/25, that completely stopped. The base CPU usage is a bit higher, but all the random spikes are gone. The base memory usage is also lower. I could never figure out what was causing the problem, and it's possible that &synchronized not doing anything anymore is why it's better now. I'm mostly using &synchronized for syncing input files across all the workers and one of them does have 300k entries in it. That file is fairly constant though, only a few k changes every 5 minutes and nothing that should use 20G of ram. I still need to replace all of our uses of &synchronized. The config framework may work for most cases once the cluster bits are done, but probably not for syncing the 300k item set. Another GREAT thing I noticed that we may want to add to NEWS is that it looks like the 'file descriptors used per worker' is down from 5 (1 socket + 2+2 from pipes) to just 1 socket (no more flares?). This means that even though select() is still not gone, the limitation of ~175 workers for 2.5.x will go away and people would be able to run 500+ worker clusters if they wanted to. ? Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180601/2937afef/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2018-06-01 at 10.36.25 AM.png Type: image/png Size: 216983 bytes Desc: Screen Shot 2018-06-01 at 10.36.25 AM.png Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180601/2937afef/attachment-0001.bin From johanna at icir.org Sat Jun 2 09:49:54 2018 From: johanna at icir.org (Johanna Amann) Date: Sat, 02 Jun 2018 09:49:54 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Removed a few more discovered UTF-8 characters in Bro scripts. (cd18d9620) In-Reply-To: <201806020858.w528wxib009476@bro-ids.icir.org> References: <201806020858.w528wxib009476@bro-ids.icir.org> Message-ID: <6E58E81A-5261-40DB-9ED0-6FF92089C263@icir.org> Hum. Would it make sense to introduce a test that checks all script-files for non-ascii-characters? I can so see that happening again... Johanna On 2 Jun 2018, at 1:57, Seth Hall wrote: > Repository : ssh://git at bro-ids.icir.org/bro > On branch : master > Link : > https://github.com/bro/bro/commit/cd18d96205851aa9b81bcb8f0c6960768b457f72 > >> --------------------------------------------------------------- > > commit cd18d96205851aa9b81bcb8f0c6960768b457f72 > Author: Seth Hall > Date: Sat Jun 2 04:57:48 2018 -0400 > > Removed a few more discovered UTF-8 characters in Bro scripts. > > >> --------------------------------------------------------------- > > cd18d96205851aa9b81bcb8f0c6960768b457f72 > scripts/base/protocols/smb/const-nt-status.bro | 8 ++++---- > 1 file changed, 4 insertions(+), 4 deletions(-) > > diff --git a/scripts/base/protocols/smb/const-nt-status.bro > b/scripts/base/protocols/smb/const-nt-status.bro > index ea7c7fe9e..f985e72a3 100644 > --- a/scripts/base/protocols/smb/const-nt-status.bro > +++ b/scripts/base/protocols/smb/const-nt-status.bro > @@ -494,7 +494,7 @@ redef SMB::statuses += { > [0xC0000131] = [$id="INVALID_IMAGE_WIN_16", $desc="The specified > image file did not have the correct format: it appears to be a 16-bit > Windows image."], > [0xC0000132] = [$id="LOGON_SERVER_CONFLICT", $desc="The Netlogon > service cannot start because another Netlogon service running in the > domain conflicts with the specified role."], > [0xC0000133] = [$id="TIME_DIFFERENCE_AT_DC", $desc="The time at the > primary domain controller is different from the time at the backup > domain controller or member server by too large an amount."], > - [0xC0000134] = [$id="SYNCHRONIZATION_REQUIRED", $desc="The SAM > database on a Windows?Server is significantly out of synchronization > with the copy on the domain controller. A complete synchronization is > required."], > + [0xC0000134] = [$id="SYNCHRONIZATION_REQUIRED", $desc="The SAM > database on a Windows Server is significantly out of synchronization > with the copy on the domain controller. A complete synchronization is > required."], > [0xC0000135] = [$id="DLL_NOT_FOUND", $desc="{Unable To Locate > Component} This application has failed to start because %hs was not > found. Reinstalling the application may fix this problem."], > [0xC0000136] = [$id="OPEN_FAILED", $desc="The NtCreateFile API > failed. This error should never be returned to an application; it is a > place holder for the Windows LAN Manager Redirector to use in its > internal error-mapping routines."], > [0xC0000137] = [$id="IO_PRIVILEGE_FAILED", $desc="{Privilege Failed} > The I/O permissions for the process could not be changed."], > @@ -536,7 +536,7 @@ redef SMB::statuses += { > [0xC000015B] = [$id="LOGON_TYPE_NOT_GRANTED", $desc="A user has > requested a type of logon (for example, interactive or network) that > has not been granted. An administrator has control over who may logon > interactively and through the network."], > [0xC000015C] = [$id="NOT_REGISTRY_FILE", $desc="The system has > attempted to load or restore a file into the registry, and the > specified file is not in the format of a registry file."], > [0xC000015D] = [$id="NT_CROSS_ENCRYPTION_REQUIRED", $desc="An > attempt was made to change a user password in the security account > manager without providing the necessary Windows cross-encrypted > password."], > - [0xC000015E] = [$id="DOMAIN_CTRLR_CONFIG_ERROR", $desc="A > Windows?Server has an incorrect configuration."], > + [0xC000015E] = [$id="DOMAIN_CTRLR_CONFIG_ERROR", $desc="A Windows > Server has an incorrect configuration."], > [0xC000015F] = [$id="FT_MISSING_MEMBER", $desc="An attempt was made > to explicitly access the secondary copy of information via a device > control to the fault tolerance driver and the secondary copy is not > present in the system."], > [0xC0000160] = [$id="ILL_FORMED_SERVICE_ENTRY", $desc="A > configuration registry node that represents a driver service entry was > ill-formed and did not contain the required value entries."], > [0xC0000161] = [$id="ILLEGAL_CHARACTER", $desc="An illegal character > was encountered. For a multibyte character set, this includes a lead > byte without a succeeding trail byte. For the Unicode character set > this includes the characters 0xFFFF and 0xFFFE."], > @@ -577,7 +577,7 @@ redef SMB::statuses += { > [0xC0000188] = [$id="LOG_FILE_FULL", $desc="The log file space is > insufficient to support this operation."], > [0xC0000189] = [$id="TOO_LATE", $desc="A write operation was > attempted to a volume after it was dismounted."], > [0xC000018A] = [$id="NO_TRUST_LSA_SECRET", $desc="The workstation > does not have a trust secret for the primary domain in the local LSA > database."], > - [0xC000018B] = [$id="NO_TRUST_SAM_ACCOUNT", $desc="The SAM database > on the Windows?Server does not have a computer account for this > workstation trust relationship."], > + [0xC000018B] = [$id="NO_TRUST_SAM_ACCOUNT", $desc="The SAM database > on the Windows Server does not have a computer account for this > workstation trust relationship."], > [0xC000018C] = [$id="TRUSTED_DOMAIN_FAILURE", $desc="The logon > request failed because the trust relationship between the primary > domain and the trusted domain failed."], > [0xC000018D] = [$id="TRUSTED_RELATIONSHIP_FAILURE", $desc="The logon > request failed because the trust relationship between this workstation > and the primary domain failed."], > [0xC000018E] = [$id="EVENTLOG_FILE_CORRUPT", $desc="The Eventlog log > file is corrupt."], > @@ -833,7 +833,7 @@ redef SMB::statuses += { > [0xC00002FD] = [$id="KDC_UNKNOWN_ETYPE", $desc="The encryption type > requested is not supported by the KDC."], > [0xC00002FE] = [$id="SHUTDOWN_IN_PROGRESS", $desc="A system shutdown > is in progress."], > [0xC00002FF] = [$id="SERVER_SHUTDOWN_IN_PROGRESS", $desc="The server > machine is shutting down."], > - [0xC0000300] = [$id="NOT_SUPPORTED_ON_SBS", $desc="This operation is > not supported on a computer running Windows Server?2003 for Small > Business Server."], > + [0xC0000300] = [$id="NOT_SUPPORTED_ON_SBS", $desc="This operation is > not supported on a computer running Windows Server 2003 for Small > Business Server."], > [0xC0000301] = [$id="WMI_GUID_DISCONNECTED", $desc="The WMI GUID is > no longer available."], > [0xC0000302] = [$id="WMI_ALREADY_DISABLED", $desc="Collection or > events for the WMI GUID is already disabled."], > [0xC0000303] = [$id="WMI_ALREADY_ENABLED", $desc="Collection or > events for the WMI GUID is already enabled."], > > > > _______________________________________________ > bro-commits mailing list > bro-commits at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits From Tim.McMullan at sig.com Wed Jun 6 09:54:28 2018 From: Tim.McMullan at sig.com (McMullan, Tim) Date: Wed, 6 Jun 2018 16:54:28 +0000 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? Message-ID: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> We are running into performance issues (30x slower) since the Broker patch (fe7e1ee) - We have 40G connections tapped from our storage filers feeding multiple Bro instances which analyze specifically only NFS and SMB traffic; all other analyzers are disabled. With the broker patch we are seeing processing times for a ~1GB pcap jump from around 2 seconds to over 1 minute. Profiling Bro, it looks like the culprit is the new Actor functions -- # Before patch Overhead Shared Object Symbol 14.57% [kernel] [k] copy_user_enhanced_fast_string 3.20% bro [.] EventHandler::operator bool 2.99% bro [.] _siphash 2.89% bro [.] Dictionary::Lookup # After patch Overhead Shared Object Symbol 5.71% [kernel] [k] native_write_msr_safe 3.84% libcaf_core.so.0.15.7 [.] caf::scheduler::worker::run 3.71% libcaf_core.so.0.15.7 [.] caf::detail::double_ended_queue::take_head 3.29% [kernel] [k] _raw_spin_lock Is the Bro development team still optimizing the Broker/Actor framework? It might be helpful to have a way to disable Broker for those of us who haven't migrated to it yet. # ~1GB file time (old) $ time /hostname/bro-devel/bin/bro -r 20180606-1049-prodfilers-truncated_00000_20180606104904.pcap master.bro real 0m2.294s user 0m1.862s sys 0m0.385s # ~1GB file time (new) $ time /hostname/bro-devel/bin/bro -r 20180606-1049-prodfilers-truncated_00000_20180606104904.pcap master.bro real 1m11.458s user 0m58.933s sys 1m34.074s Thanks! --Tim ________________________________ IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180606/dbb38bfb/attachment.html From jazoff at illinois.edu Wed Jun 6 12:03:16 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Wed, 6 Jun 2018 19:03:16 +0000 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> Message-ID: > On Jun 6, 2018, at 12:54 PM, McMullan, Tim wrote: > > We are running into performance issues (30x slower) since the Broker patch (fe7e1ee) ? > > We have 40G connections tapped from our storage filers feeding multiple Bro instances which analyze specifically only NFS and SMB traffic; all other analyzers are disabled. With the broker patch we are seeing processing times for a ~1GB pcap jump from around 2 seconds to over 1 minute. Profiling Bro, it looks like the culprit is the new Actor functions -- > > # Before patch > Overhead Shared Object Symbol > 14.57% [kernel] [k] copy_user_enhanced_fast_string > 3.20% bro [.] EventHandler::operator bool > 2.99% bro [.] _siphash > 2.89% bro [.] Dictionary::Lookup > > # After patch > Overhead Shared Object Symbol > 5.71% [kernel] [k] native_write_msr_safe > 3.84% libcaf_core.so.0.15.7 [.] caf::scheduler::worker::run > 3.71% libcaf_core.so.0.15.7 [.] caf::detail::double_ended_queue::take_head > 3.29% [kernel] [k] _raw_spin_lock > > Is the Bro development team still optimizing the Broker/Actor framework? It might be helpful to have a way to disable Broker for those of us who haven?t migrated to it yet. > > # ~1GB file time (old) > $ time /hostname/bro-devel/bin/bro -r 20180606-1049-prodfilers-truncated_00000_20180606104904.pcap master.bro > > real 0m2.294s > user 0m1.862s > sys 0m0.385s > > # ~1GB file time (new) > $ time /hostname/bro-devel/bin/bro -r 20180606-1049-prodfilers-truncated_00000_20180606104904.pcap master.bro > > real 1m11.458s > user 0m58.933s > sys 1m34.074s Well, I can at least reproduce that here on one of the standard test pcap files and no scripts loaded: $ /usr/local/bin/bro --version /usr/local/bin/bro version 2.5.2 $ time /usr/local/bin/bro -b -r ~/src/bro/testing/external/bro-testing/2009-M57-day11-18.trace real 0m2.317s user 0m1.874s sys 0m0.327s $ . ~/src/bro/build/bro-path-dev.sh $ bro --version bro version 2.5-648 $ time bro -b -r ~/src/bro/testing/external/bro-testing/2009-M57-day11-18.trace real 1m5.523s user 0m30.565s sys 0m32.661s and the same thing in perf: Samples: 270K of event 'cycles:u', Event count (approx.): 131984875188 Overhead Command Shared Object Symbol 8.95% bro libcaf_core.so.0.15.7 [.] caf::detail::double_ended_queue::take_tail 5.90% bro libcaf_core.so.0.15.7 [.] caf::detail::double_ended_queue::take_head 4.92% bro libcaf_core.so.0.15.7 [.] caf::scheduler::worker::run 2.67% bro libc-2.17.so [.] __GI___libc_nanosleep 1.98% bro libtcmalloc.so.4.4.5 [.] operator new[] 1.79% bro libc-2.17.so [.] __sleep 1.68% bro libtcmalloc.so.4.4.5 [.] tc_deletearray_nothrow 1.47% bro libc-2.17.so [.] __libc_disable_asynccancel 1.28% bro libstdc++.so.6.0.19 [.] std::this_thread::__sleep_for 1.10% bro libc-2.17.so [.] usleep 1.05% bro bro [.] ones_complement_checksum I haven't noticed a huge difference on a real multi process cluster, at least not a 30x change, so this is odd that pcap processing is so much slower. Especially since broker should be completely disabled when pcap files are being read and caf shouldn't even be doing anything. ? Justin Azoff From jsiwek at corelight.com Wed Jun 6 14:50:40 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Wed, 6 Jun 2018 16:50:40 -0500 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> Message-ID: On Wed, Jun 6, 2018 at 12:01 PM McMullan, Tim wrote: > Is the Bro development team still optimizing the Broker/Actor framework? Yes, in the sense that optimizations will be done according to feedback. No, in the sense that there was no one actively looking into it at the moment you asked. > It might be helpful to have a way to disable Broker for those of us who haven?t migrated to it yet. That's unlikely at this point as scripts have been completely ported to Broker now and usages of the old communication system completely removed from them. No simple switch to it back, so your feedback is especially important/helpful. > # ~1GB file time (old) > > $ time /hostname/bro-devel/bin/bro -r 20180606-1049-prodfilers-truncated_00000_20180606104904.pcap master.bro > > > > real 0m2.294s > > user 0m1.862s > > sys 0m0.385s > > > > # ~1GB file time (new) > > $ time /hostname/bro-devel/bin/bro -r 20180606-1049-prodfilers-truncated_00000_20180606104904.pcap master.bro > > > > real 1m11.458s > > user 0m58.933s > > sys 1m34.074s Try pulling in the change I just did at [1], which was a big part of the problem in my own test: # 2.5.3 $ time bro -r 2009-M57-day11-18.trace real 0m16.187s user 0m16.312s sys 0m1.865s # master before [1] $ time bro -r ../testing/external/bro-testing/Traces/2009-M57-day11-18.trace real 1m31.434s user 1m44.925s sys 1m4.947s # master after [1] $ time bro -r 2009-M57-day11-18.trace real 0m24.595s user 0m25.574s sys 0m5.816s - Jon [1] https://github.com/bro/bro/commit/9822fc252d5e92208704df4a388ea31989869499 From jsiwek at corelight.com Wed Jun 6 15:05:57 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Wed, 6 Jun 2018 17:05:57 -0500 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> Message-ID: On Wed, Jun 6, 2018 at 2:10 PM Azoff, Justin S wrote: > I haven't noticed a huge difference on a real multi process cluster, at least not a 30x change, so this is odd that pcap processing is so much slower. > Especially since broker should be completely disabled when pcap files are being read and caf shouldn't even be doing anything. It's not that simple. You could think of broker being enabled in this case of simply reading a pcap because it was querying the library for whether there were any peers (and that was actually a at least part of the perf. problem). Beyond that, you can still think of broker being enabled when you're reading a pcap and you are also using data stores (which the Known* scripts in Bro now do by default). Communication with a local master data store is still asynchronous communication (with local threads/actors) that ends up going through CAF messaging. There's also essentially a per-packet synchronization being done between Bro and Broker/CAF now to ensure that this type of asynchronous workload ends up producing the same results between any given Bro run. So there's now also that bit of extra overhead in pcap processing. - Jon From robin at icir.org Thu Jun 7 10:21:54 2018 From: robin at icir.org (Robin Sommer) Date: Thu, 7 Jun 2018 10:21:54 -0700 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> Message-ID: <20180607172154.GM19548@icir.org> Hmm, I'm still seeing much larger runtimes on that M57 trace using our test configuration, even with yesterday's change: ; Master, pre-Broker # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro 73.72user 7.90system 1:06.53elapsed 122%CPU (0avgtext+0avgdata 199092maxresident) ; Current master # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro 2191.59user 1606.69system 12:39.92elapsed 499%CPU (0avgtext+0avgdata 228400maxresident) Bro must still be blocking somewhere when reading from trace files. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at corelight.com Thu Jun 7 15:05:33 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 7 Jun 2018 17:05:33 -0500 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: <20180607172154.GM19548@icir.org> References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> <20180607172154.GM19548@icir.org> Message-ID: On Thu, Jun 7, 2018 at 12:21 PM Robin Sommer wrote: > > Hmm, I'm still seeing much larger runtimes on that M57 trace using our > test configuration, even with yesterday's change: > > ; Master, pre-Broker > # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro > 73.72user 7.90system 1:06.53elapsed 122%CPU (0avgtext+0avgdata 199092maxresident) > > ; Current master > # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro > 2191.59user 1606.69system 12:39.92elapsed 499%CPU (0avgtext+0avgdata 228400maxresident) > > Bro must still be blocking somewhere when reading from trace files. Thanks, if you pull master changes [1] again there's likely some improvement. You can also try comparing the timing of: # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro Known::use_host_store=F Known::use_service_store=F Known::use_cert_store=F With that, I get the same timings as from before pre-Broker. At least a good chunk of the difference when using data stores is that, for every query, Bro will immediately block until getting a response back from the data store thread/actor. Not doing this type of synchronous data store access when reading pcaps leads to the possibility of results differing between runs (and I recall such differences being likely from experience in running the unit test suite). - Jon [1] https://github.com/bro/broker/commit/9b56fea4999d4e11a5cd2caaafd934759015fab5 From jeffrey.bencteux at ssi.gouv.fr Fri Jun 8 05:04:25 2018 From: jeffrey.bencteux at ssi.gouv.fr (Bencteux Jeffrey) Date: Fri, 8 Jun 2018 14:04:25 +0200 Subject: [Bro-Dev] SMB Analyzer code factorization Message-ID: Hi all, As I looked into SMBv1 analyzer, I found that most of the files describing SMB messages have code duplication. According to the SMB specification ([MS-CIFS]), SMB messages are composed of a fixed-length header (defined as SMB_Header in smb1-protocol.pac for Bro) and then of two "blocks" : the parameter block (section 2.2.3.2) and the data block (section 2.2.3.3). Every message of the protocol I saw in the specification or encountered in trafic always followed that scheme. However, in Bro's SMBv1 analyzer, each .pac file describing a particular type of message reimplement the two blocks common fields. The SMB PDU is defined as such : type SMB_PDU(...) = record { ??? header: SMB_Header(...) ??? message: case msg_len of { ??? ??? 35 -> no_msg: SMB_No_Message; ??? ??? default -> msg: SMB_Message(...); ??? }; }; SMB_Message is just a switch between SMB_Message_request and SMB_Message_Response. Then these two are defined as a switch on command type : type SMB_Message_Request(...) = case command of { ??? SMB_COM_READ_ANDX -> read_andx: SMB1_read_andx_request(...); ??? ... }; And then every command is implemented like this one : type SMB_read_andx_request(...) = record { ??? word_count: uint8 ??? ... specific fields ... ??? byte_count: uint16; ??? ... specific fields ... }; I feel like it would be a good idea to abstract the two blocks : it would factorize code. Also, there is currently no check on the value of word_count field or byte_count field for the rest of the payload. This could lead to protocol violations from BinPAC if you have a byte_count set to 0 and then a following field trying to parse a SMB_String for example (see SMB1_transaction_request record in smb1-com-transaction.pac for an example of that). A solution could be to change SMB_Message_* to something closer to what the specification describe by dividing every message in two half structures : type SMB_Message_Request(...) = record { ??? parameter_block: SMB_Request_Parameters(...); ??? data_block: SMB_Request_Data(...); }; type SMB_Request_Parameters(...) = record { ??? word_count: uint8; ??? # Maybe a check on word_count here? Some messages seems to have a predefined value for this field ??? words: SMB_Request_Words(...); }; type SMB_Request_Data(...) = record { ??? byte_count: uint16; ??? data_or_not: case byte_count of { ??? ??? 0 -> none: empty; ??? ??? default -> bytes: SMB_Request_Bytes(...); ??? }; }; And then SMB_Request_Words and SMB_Request_Bytes would be switch on different messages types : type SMB_Request_Words(...) = case command of { ??? SMB_COM_READ_ANDX -> words_read_and_x: SMB1_Words_read_andx_request(...); ??? ... }; type SMB_Request_Bytes(...) = case command of { ??? SMB_COM_READ_ANDX -> bytes_read_and_x: SMB1_Bytes_read_andx_request(...); ??? ... }; Of course, it means splitting every existing message and re factoring a lot of code of the analyzer, but it would be easier then to address problems such as the one quoted as example above. What do you think of it ? Regards, -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180608/2a9c29dc/attachment.bin From robin at icir.org Fri Jun 8 08:23:47 2018 From: robin at icir.org (Robin Sommer) Date: Fri, 8 Jun 2018 08:23:47 -0700 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> <20180607172154.GM19548@icir.org> Message-ID: <20180608152347.GI75818@icir.org> On Thu, Jun 07, 2018 at 17:05 -0500, you wrote: > Thanks, if you pull master changes [1] again there's likely some improvement. Yeah, a little bit, not much though. > # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro > Known::use_host_store=F Known::use_service_store=F > Known::use_cert_store=F That indeed gets it way down, though still not back to the same level on my box: 170.49user 58.14system 1:55.94elapsed 197%CPU (pre-master: 73.72user 7.90system 1:06.53elapsed 122%CPU) Are there more places where Bro's waiting for Broker in pcap mode? > With that, I get the same timings as from before pre-Broker. At least > a good chunk of the difference when using data stores is that, for > every query, Bro will immediately block until getting a response back > from the data store thread/actor. Yeah, I remember that discussion. It's the trade-off between performance and consistency. Where's the code that's doing this blocking? Would it be possible to not block right away, but sync up with Broker when events are flushed the next time? (Like we had discussed before as a general mechanism for making async operations consistent) Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at corelight.com Fri Jun 8 10:16:41 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 8 Jun 2018 12:16:41 -0500 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: <20180608152347.GI75818@icir.org> References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> <20180607172154.GM19548@icir.org> <20180608152347.GI75818@icir.org> Message-ID: On Fri, Jun 8, 2018 at 10:23 AM Robin Sommer wrote: > > > # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro > > Known::use_host_store=F Known::use_service_store=F > > Known::use_cert_store=F > > That indeed gets it way down, though still not back to the same level > on my box: > > 170.49user 58.14system 1:55.94elapsed 197%CPU > > (pre-master: 73.72user 7.90system 1:06.53elapsed 122%CPU) Just double-checking: same configure/build flags were used between the two? Here's the more precise results I got from running on a macOS and Linux system: # Linux master (b51e6f3) --build-type=debug $ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r 2009-M57-day11-18.trace test-all-policy testing-setup real 0m32.572s user 0m40.926s sys 0m7.873s # Linux master (b51e6f3) --build-type=debug $ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r 2009-M57-day11-18.trace test-all-policy testing-setup Known::use_host_store=F Known::use_cert_store=F Known::use_service_store=F real 0m25.603s user 0m23.311s sys 0m7.952s # Linux pre-broker (7a6f502) --build-type=debug $ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r 2009-M57-day11-18.trace test-all-policy testing-setup real 0m24.785s user 0m21.230s sys 0m8.341s # macOS master (b51e6f3) --build-type=debug $ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r 2009-M57-day11-18.trace test-all-policy testing-setup.bro real 0m38.775s user 0m42.365s sys 0m8.950s # macOS master (b51e6f3) --build-type=debug $ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r 2009-M57-day11-18.trace test-all-policy testing-setup.bro Known::use_host_store=F Known::use_cert_store=F Known::use_service_store=F real 0m32.975s user 0m33.774s sys 0m4.409s # macOS pre-broker (7a6f502) --build-type=debug $ export BROPATH=$BROPATH:../testing/external/scripts; time bro -r 2009-M57-day11-18.trace test-all-policy testing-setup.bro real 0m30.785s user 0m31.840s sys 0m3.788s > Are there more places where Bro's waiting for Broker in pcap mode? Not that I can think of. > Yeah, I remember that discussion. It's the trade-off between > performance and consistency. Where's the code that's doing this > blocking? I just know that Manager::AdvanceTime() and also Manager::TrackStoreQuery() -> FlushPendingQueries() can block/spin waiting on actor/threading in pcap mode. > Would it be possible to not block right away, but sync up > with Broker when events are flushed the next time? (Like we had > discussed before as a general mechanism for making async operations > consistent) I think the way to make an async operation most consistent is to model it as a synchronous operation? That's what I'm doing now at least, and if I try moving FlushPendingQueries() to later (before the AdvanceTime() call) in an attempt to possibly have more queries in the queue/buffer, I get the external test suites failing. At least on my own test systems, I don't have much performance to recover by going this route anyway, so maybe we need to dig more into why your results are different. Only thing I'm thinking is that your test system maybe does a poorer job of scheduling the right thread to run in order for the FlushPendingQueries() spin-loop to actually finish. - Jon From jsiwek at corelight.com Fri Jun 8 10:40:52 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 8 Jun 2018 12:40:52 -0500 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> <20180607172154.GM19548@icir.org> <20180608152347.GI75818@icir.org> Message-ID: On Fri, Jun 8, 2018 at 12:16 PM Jon Siwek wrote: > Only thing I'm thinking is that your test system maybe > does a poorer job of scheduling the right thread to run in order for > the FlushPendingQueries() spin-loop to actually finish. Actually, realized you still had bad timing with data stores off, so it would more likely be a problem with the AdvanceTime() code path. There's some mutex locking going on there, but w/o data stores involved there shouldn't be anyone competing for them. - Jon From robin at icir.org Fri Jun 8 10:47:11 2018 From: robin at icir.org (Robin Sommer) Date: Fri, 8 Jun 2018 10:47:11 -0700 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> <20180607172154.GM19548@icir.org> <20180608152347.GI75818@icir.org> Message-ID: <20180608174711.GK75818@icir.org> Ok, I'll dig around a bit more and also double-check that I didn't change any settings in the meantime. Robin On Fri, Jun 08, 2018 at 12:40 -0500, you wrote: > On Fri, Jun 8, 2018 at 12:16 PM Jon Siwek wrote: > > > Only thing I'm thinking is that your test system maybe > > does a poorer job of scheduling the right thread to run in order for > > the FlushPendingQueries() spin-loop to actually finish. > > Actually, realized you still had bad timing with data stores off, so > it would more likely be a problem with the AdvanceTime() code path. > There's some mutex locking going on there, but w/o data stores > involved there shouldn't be anyone competing for them. > > - Jon > -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From Tim.McMullan at sig.com Fri Jun 8 12:18:47 2018 From: Tim.McMullan at sig.com (McMullan, Tim) Date: Fri, 8 Jun 2018 19:18:47 +0000 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> <20180607172154.GM19548@icir.org> Message-ID: Hi Jon, Thanks for looking into this! I tried this again after pulling from master and at least for our case it is looking good! ### old # /hostname/bro/bin/bro --version /hostname/bro/bin/bro version 2.5-519 # time /hostname/bro/bin/bro -r 20180606-1242-prodfilers.pcap master.bro real 0m32.772s user 0m28.821s sys 0m3.873s ### new # /hostname/bro-devel/bin/bro --version /hostname/bro-devel/bin/bro version 2.5-658 # time /hostname/bro-devel/bin/bro -r 20180606-1242-prodfilers.pcap master.bro real 0m34.684s user 0m31.502s sys 0m4.266s Thanks again! --Tim -----Original Message----- From: Jon Siwek [mailto:jsiwek at corelight.com] Sent: Thursday, June 07, 2018 6:06 PM To: Robin Sommer Cc: McMullan, Tim ; ; Wallior, Julien ; Trejo, Devin Subject: Re: [Bro-Dev] Performance Issues after the fe7e1ee commit? On Thu, Jun 7, 2018 at 12:21 PM Robin Sommer wrote: > > Hmm, I'm still seeing much larger runtimes on that M57 trace using our > test configuration, even with yesterday's change: > > ; Master, pre-Broker > # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro > 73.72user 7.90system 1:06.53elapsed 122%CPU (0avgtext+0avgdata 199092maxresident) > > ; Current master > # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro > 2191.59user 1606.69system 12:39.92elapsed 499%CPU (0avgtext+0avgdata 228400maxresident) > > Bro must still be blocking somewhere when reading from trace files. Thanks, if you pull master changes [1] again there's likely some improvement. You can also try comparing the timing of: # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro Known::use_host_store=F Known::use_service_store=F Known::use_cert_store=F With that, I get the same timings as from before pre-Broker. At least a good chunk of the difference when using data stores is that, for every query, Bro will immediately block until getting a response back from the data store thread/actor. Not doing this type of synchronous data store access when reading pcaps leads to the possibility of results differing between runs (and I recall such differences being likely from experience in running the unit test suite). - Jon [1] https://github.com/bro/broker/commit/9b56fea4999d4e11a5cd2caaafd934759015fab5 ________________________________ IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. From jmellander at lbl.gov Mon Jun 11 15:25:33 2018 From: jmellander at lbl.gov (Jim Mellander) Date: Mon, 11 Jun 2018 15:25:33 -0700 Subject: [Bro-Dev] Changing epoch timeout in SumStats Message-ID: Hi all: I'm working on a statistics gathering script using sumstats. I would dearly love to have it synchronized with the log rotation interval. This means that the first epoch interval would most likely be short, with future epochs being at the rotation interval. Prior Bros (2.5.2) had an intriguing comment about possible on-demand call for sumstats, which doesn't seem to be in the latest master. So, at the moment, what I'm looking for is one of the two options: 1. Allow for a pure 'on-demand' option for sumstats results, allowing the user script to manage the results, or 2. Allow redefining the epoch timeout on the fly. Option 3 is hacking on the sumstats code...... Any comments? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180611/7e45fea6/attachment.html From robin at icir.org Mon Jun 11 16:30:07 2018 From: robin at icir.org (Robin Sommer) Date: Mon, 11 Jun 2018 16:30:07 -0700 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: <20180608174711.GK75818@icir.org> References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> <20180607172154.GM19548@icir.org> <20180608152347.GI75818@icir.org> <20180608174711.GK75818@icir.org> Message-ID: <20180611233007.GC90763@icir.org> On Fri, Jun 08, 2018 at 10:47 -0700, I wrote: > Ok, I'll dig around a bit more and also double-check that I didn't > change any settings in the meantime. Confirmed that I'm still seeing that difference even when using the options to turn the stores. Tried it on two different Fedora 28 systems, with similar results. I haven't been able to nail down what's going on though. The AdvanceTime() method does seem to do a lot of locking in pcap mode, independent of whether there's actually any data store operations. However, I tried a quick hack to get that down and that didn't change anything. I then ran it through oprofile. Output is attached. There's indeed some locking showing up high in there, but I can't tell if that's just expected idling in CAF. I am bit surprised to see a number of std::chrono() methods showing rather prominently that I would expect to be cheap. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin -------------- next part -------------- Using /home/robin/bro/master/tmp/oprofile_data/samples/ for samples directory. WARNING! Some of the events were throttled. Throttling occurs when the initial sample rate is too high, causing an excessive number of interrupts. Decrease the sampling frequency. Check the directory /home/robin/bro/master/tmp/oprofile_data/samples/current/stats/throttled for the throttled event names. WARNING: Lost samples detected! See /home/robin/bro/master/tmp/oprofile_data/samples/operf.log for details. warning: /hpsa could not be found. warning: /kvm could not be found. warning: /nf_conntrack could not be found. warning: /nf_defrag_ipv4 could not be found. warning: /nf_nat could not be found. warning: /nf_nat_ipv4 could not be found. warning: /tg3 could not be found. CPU: Intel Sandy Bridge microarchitecture, speed 2000 MHz (estimated) Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000 samples % image name symbol name 48605 3.3738 kallsyms find_busiest_group 31108 2.1593 libtcmalloc.so.4.5.1 /usr/lib64/libtcmalloc.so.4.5.1 24088 1.6720 bro RE_Match_State::Match(unsigned char const*, int, bool, bool, bool) 22833 1.5849 kallsyms native_write_msr 20525 1.4247 kallsyms finish_task_switch 20314 1.4100 kallsyms syscall_return_via_sysret 16822 1.1677 kallsyms __softirqentry_text_start 16520 1.1467 libcaf_core.so.0.15.7 caf::detail::double_ended_queue::lock_guard::lock_guard(std::atomic_flag&) 15112 1.0490 kallsyms update_blocked_averages 12897 0.8952 kallsyms run_timer_softirq 12890 0.8947 kallsyms pick_next_task_fair 12729 0.8836 libpthread-2.27.so nanosleep 12495 0.8673 kallsyms update_curr 12209 0.8475 kallsyms _raw_spin_lock 12186 0.8459 libcaf_core.so.0.15.7 caf::resumable* caf::policy::work_stealing::dequeue >(caf::scheduler::worker*) 11979 0.8315 kallsyms __schedule 11886 0.8250 kallsyms __update_load_avg_cfs_rq.isra.34 11463 0.7957 kallsyms idle_cpu 11239 0.7801 kallsyms __update_load_avg_se.isra.33 11178 0.7759 kallsyms native_sched_clock 11010 0.7642 kallsyms update_load_avg 10854 0.7534 libcaf_core.so.0.15.7 std::atomic::node*>::load(std::memory_order) const 10770 0.7476 kallsyms load_balance 10737 0.7453 libcaf_core.so.0.15.7 decltype (({parm#1}->data)()) caf::policy::unprofiled::d >(caf::scheduler::worker*) 10582 0.7345 bro DFA_State::Xtion(int, DFA_Machine*) 10554 0.7326 libcaf_core.so.0.15.7 caf::detail::double_ended_queue::take_head() 10185 0.7070 bro RandTest::add(void const*, int) 10133 0.7034 libcaf_core.so.0.15.7 std::__uniq_ptr_impl::node, std::default_delete::node> >::_M_ptr() 9920 0.6886 libstdc++.so.6.0.25 /usr/lib64/libstdc++.so.6.0.25 9813 0.6811 kallsyms swapgs_restore_regs_and_return_to_usermode 9685 0.6723 kallsyms trigger_load_balance 9431 0.6546 libcaf_core.so.0.15.7 std::tuple_element<0ul, std::tuple::node*, std::default_delete::node> > >::type& std::get<0ul, caf::detail::double_ended_queue::node*, std::default_delete::node> >(std::tuple::node*, std::default_delete::node> >&) 9404 0.6528 libcaf_core.so.0.15.7 caf::scheduler::worker::data() 9350 0.6490 kallsyms do_syscall_64 9311 0.6463 libcaf_core.so.0.15.7 std::enable_if::node*> >, std::is_move_constructible::node*>, std::is_move_assignable::node*> >::value, void>::type std::swap::node*>(caf::detail::double_ended_queue::node*&, caf::detail::double_ended_queue::node*&) 9297 0.6453 libbroker.so.. std::operator&(std::memory_order, std::__memory_order_modifier) 9194 0.6382 bro DFA_State::Accept() const 9009 0.6253 kallsyms _raw_spin_lock_irqsave 8818 0.6121 libcaf_core.so.0.15.7 std::_Tuple_impl<0ul, caf::detail::double_ended_queue::node*, std::default_delete::node> >::_M_head(std::_Tuple_impl<0ul, caf::detail::double_ended_queue::node*, std::default_delete::node> >&) 8690 0.6032 libcaf_core.so.0.15.7 caf::detail::double_ended_queue::node*& std::__get_helper<0ul, caf::detail::double_ended_queue::node*, std::default_delete::node> >(std::_Tuple_impl<0ul, caf::detail::double_ended_queue::node*, std::default_delete::node> >&) 8643 0.5999 kallsyms queued_spin_lock_slowpath 8622 0.5985 kallsyms irq_work_interrupt 8246 0.5724 libcaf_core.so.0.15.7 std::_Head_base<0ul, caf::detail::double_ended_queue::node*, false>::_M_head(std::_Head_base<0ul, caf::detail::double_ended_queue::node*, false>&) 8095 0.5619 libcaf_core.so.0.15.7 std::unique_ptr::node, std::default_delete::node> >::reset(caf::detail::double_ended_queue::node*) 7881 0.5470 libbroker.so.. std::chrono::duration >::count() const 7790 0.5407 libcaf_core.so.0.15.7 caf::detail::double_ended_queue::lock_guard::~lock_guard() 7732 0.5367 libcaf_core.so.0.15.7 void std::this_thread::sleep_for >(std::chrono::duration > const&) 7502 0.5207 kallsyms update_cfs_group 7328 0.5087 libpthread-2.27.so __pthread_disable_asynccancel 7143 0.4958 kallsyms cpuacct_charge 6975 0.4842 libcaf_core.so.0.15.7 std::_Head_base<0ul, caf::detail::double_ended_queue::node*, false>::_M_head(std::_Head_base<0ul, caf::detail::double_ended_queue::node*, false> const&) 6972 0.4839 libc-2.27.so __GI___strlen_sse2 6606 0.4585 kallsyms enqueue_task_fair 6543 0.4542 kallsyms rebalance_domains 6239 0.4331 kallsyms irq_exit 6174 0.4286 bro ones_complement_checksum(void const*, int, unsigned int) 6160 0.4276 libcaf_core.so.0.15.7 std::unique_ptr::node, std::default_delete::node> >::operator->() const 6047 0.4197 libcaf_core.so.0.15.7 unsigned long std::uniform_int_distribution::operator() >(std::linear_congruential_engine&, std::uniform_int_distribution::param_type const&) 5997 0.4163 libc-2.27.so vfprintf 5964 0.4140 libcaf_core.so.0.15.7 std::unique_ptr::node, std::default_delete::node> >::get() const 5769 0.4004 libcaf_core.so.0.15.7 std::__uniq_ptr_impl::node, std::default_delete::node> >::_M_ptr() const 5650 0.3922 kallsyms entry_SYSCALL_64_after_hwframe 5639 0.3914 kallsyms irqtime_account_irq 5507 0.3823 libcaf_core.so.0.15.7 std::_Tuple_impl<0ul, caf::detail::double_ended_queue::node*, std::default_delete::node> >::_M_head(std::_Tuple_impl<0ul, caf::detail::double_ended_queue::node*, std::default_delete::node> > const&) 5485 0.3807 kallsyms try_to_wake_up 5388 0.3740 kallsyms find_next_and_bit 5287 0.3670 kallsyms do_nanosleep 5287 0.3670 libpthread-2.27.so pthread_mutex_lock 5264 0.3654 libcaf_core.so.0.15.7 std::__atomic_base::node*>::operator caf::detail::double_ended_queue::node*() const 5220 0.3623 kallsyms nohz_balance_exit_idle 5210 0.3616 libpthread-2.27.so pthread_cond_timedwait@@GLIBC_2.3.2 5106 0.3544 kallsyms find_next_bit 4957 0.3441 libcaf_core.so.0.15.7 std::remove_reference::node*&>::type&& std::move::node*&>(caf::detail::double_ended_queue::node*&) 4882 0.3389 [vdso] (tgid:46504 range:0x7ffe89dfa000-0x7ffe89dfbfff) [vdso] (tgid:46504 range:0x7ffe89dfa000-0x7ffe89dfbfff) 4878 0.3386 kallsyms account_entity_enqueue 4871 0.3381 kallsyms __next_timer_interrupt 4738 0.3289 libcaf_core.so.0.15.7 caf::detail::double_ended_queue::node* const& std::__get_helper<0ul, caf::detail::double_ended_queue::node*, std::default_delete::node> >(std::_Tuple_impl<0ul, caf::detail::double_ended_queue::node*, std::default_delete::node> > const&) 4729 0.3283 bro BaseList::operator[](int) const 4698 0.3261 kallsyms _raw_spin_lock_irq 4682 0.3250 kallsyms timerqueue_add 4535 0.3148 kallsyms enqueue_entity 4500 0.3124 libcaf_core.so.0.15.7 std::uniform_int_distribution::param_type::b() const 4431 0.3076 kallsyms update_rq_clock 4341 0.3013 kallsyms check_preempt_wakeup 4338 0.3011 kallsyms hrtimer_nanosleep 4319 0.2998 bro threading::Queue::Get() 4280 0.2971 kallsyms __hrtimer_run_queues 4255 0.2954 kallsyms _raw_spin_trylock 4180 0.2901 libcaf_core.so.0.15.7 std::__uniq_ptr_impl, std::default_delete > >::_M_ptr() const 4118 0.2858 kallsyms __calc_delta 4111 0.2854 kallsyms __radix_tree_lookup 4110 0.2853 bro std::chrono::duration >::count() const 4102 0.2847 libcaf_core.so.0.15.7 std::unique_ptr::node, std::default_delete::node> >::release() 4078 0.2831 libcaf_core.so.0.15.7 std::__detail::_Mod::__calc(unsigned long) 3918 0.2720 libcaf_core.so.0.15.7 std::tuple_element<0ul, std::tuple::node*, std::default_delete::node> > >::type const& std::get<0ul, caf::detail::double_ended_queue::node*, std::default_delete::node> >(std::tuple::node*, std::default_delete::node> > const&) 3916 0.2718 libbroker.so.. std::chrono::duration >::duration(long const&) 3880 0.2693 bro std::chrono::duration_values::zero() 3854 0.2675 bro std::chrono::duration >::duration(long const&) 3852 0.2674 kallsyms ktime_get 3837 0.2663 libcaf_core.so.0.15.7 caf::scheduler::abstract_coordinator::num_workers() const 3802 0.2639 kallsyms account_entity_dequeue 3798 0.2636 kallsyms sys_nanosleep 3758 0.2609 libcaf_core.so.0.15.7 std::unique_ptr::node, std::default_delete::node> >::~unique_ptr() 3699 0.2568 libcaf_core.so.0.15.7 unsigned long std::uniform_int_distribution::operator() >(std::linear_congruential_engine&) 3634 0.2522 libcaf_core.so.0.15.7 std::atomic::node*>::operator caf::detail::double_ended_queue::node*() const 3612 0.2507 libpthread-2.27.so __pthread_mutex_unlock_usercnt 3609 0.2505 libcaf_core.so.0.15.7 caf::resumable* caf::policy::work_stealing::try_steal >(caf::scheduler::worker*) 3563 0.2473 libcaf_core.so.0.15.7 unsigned long std::__detail::__mod(unsigned long) 3531 0.2451 kallsyms _raw_spin_unlock_irqrestore 3492 0.2424 kallsyms read_tsc 3466 0.2406 bro Unref(BroObj*) 3427 0.2379 libcaf_core.so.0.15.7 std::chrono::duration >::zero() 3417 0.2372 kallsyms __queue_work 3414 0.2370 kallsyms cyc2ns_read_begin.part.2 3359 0.2332 libcaf_core.so.0.15.7 caf::scheduler::worker* const& std::__get_helper<0ul, caf::scheduler::worker*, std::default_delete > >(std::_Tuple_impl<0ul, caf::scheduler::worker*, std::default_delete > > const&) 3350 0.2325 kallsyms run_rebalance_domains 3308 0.2296 kallsyms insert_work 3295 0.2287 kallsyms copy_user_generic_string 3283 0.2279 libcrypto.so.1.0.2o /usr/lib64/libcrypto.so.1.0.2o 3264 0.2266 kallsyms irq_work_run_list 3242 0.2250 libcaf_core.so.0.15.7 caf::scheduler::coordinator::worker_by_id(unsigned long) 3218 0.2234 libbroker.so.. std::chrono::duration > std::chrono::__duration_cast_impl >, std::ratio<1000l, 1l>, long, false, true>::__cast >(std::chrono::duration > const&) 3173 0.2202 kallsyms hrtimer_active 3160 0.2193 kallsyms task_tick_fair 3128 0.2171 libc-2.27.so clock_gettime 3097 0.2150 kallsyms hrtimer_start_range_ns 3061 0.2125 libcaf_core.so.0.15.7 std::_Tuple_impl<0ul, caf::scheduler::worker*, std::default_delete > >::_M_head(std::_Tuple_impl<0ul, caf::scheduler::worker*, std::default_delete > > const&) 3036 0.2107 bro _siphash 3006 0.2087 kallsyms reweight_entity 2989 0.2075 kallsyms apic_timer_interrupt 2944 0.2044 kallsyms dequeue_task_fair 2928 0.2032 libcaf_core.so.0.15.7 caf::scheduler::worker::parent() 2921 0.2028 bro std::chrono::duration >::count() const 2907 0.2018 bro std::chrono::duration >::duration(long const&) 2852 0.1980 libcaf_core.so.0.15.7 std::linear_congruential_engine::operator()() 2801 0.1944 bro BaseList::length() const 2773 0.1925 libc-2.27.so _IO_default_xsputn 2765 0.1919 kallsyms rcu_process_callbacks 2764 0.1919 libcaf_core.so.0.15.7 std::_Head_base<0ul, caf::scheduler::worker*, false>::_M_head(std::_Head_base<0ul, caf::scheduler::worker*, false> const&) 2726 0.1892 kallsyms kthread_data 2715 0.1885 libcaf_core.so.0.15.7 std::vector, std::default_delete > >, std::allocator, std::default_delete > > > >::operator[](unsigned long) 2713 0.1883 libcaf_core.so.0.15.7 bool std::chrono::operator< , long, std::ratio<1l, 1000000l> >(std::chrono::duration > const&, std::chrono::duration > const&) 2704 0.1877 kallsyms ret_from_intr 2697 0.1872 kallsyms lapic_next_deadline 2662 0.1848 kallsyms decay_load 2619 0.1818 bro Ref(BroObj*) 2603 0.1807 kallsyms do_futex 2548 0.1769 kallsyms __hrtimer_next_event_base 2540 0.1763 kallsyms put_prev_entity 2520 0.1749 kallsyms ctx_sched_out 2516 0.1746 libcaf_core.so.0.15.7 std::unique_ptr, std::default_delete > >::get() const 2449 0.1700 libcaf_core.so.0.15.7 std::chrono::duration > std::chrono::__duration_cast_impl >, std::ratio<1l, 1000000l>, long, true, false>::__cast >(std::chrono::duration > const&) 2448 0.1699 libcaf_core.so.0.15.7 std::uniform_int_distribution::param_type::a() const 2432 0.1688 kallsyms __indirect_thunk_start 2426 0.1684 libcaf_core.so.0.15.7 std::enable_if > >::value, std::chrono::duration > >::type std::chrono::duration_cast >, long, std::ratio<1l, 1000000l> >(std::chrono::duration > const&) 2393 0.1661 libcaf_core.so.0.15.7 bool std::chrono::operator<=, long, std::ratio<1l, 1000000l> >(std::chrono::duration > const&, std::chrono::duration > const&) 2391 0.1660 kallsyms perf_event_task_tick 2385 0.1655 libpthread-2.27.so pthread_cond_signal@@GLIBC_2.3.2 2374 0.1648 kallsyms interrupt_entry 2360 0.1638 bro threading::Queue::Put(threading::BasicOutputMessage*) 2335 0.1621 libc-2.27.so __strchrnul_sse2 2334 0.1620 libcaf_core.so.0.15.7 caf::detail::double_ended_queue::take_tail() 2326 0.1615 bro BroType::Tag() const 2312 0.1605 kallsyms dequeue_entity 2294 0.1592 kallsyms hrtimer_interrupt 2272 0.1577 kallsyms __hrtimer_init 2264 0.1572 kallsyms get_nohz_timer_target 2251 0.1562 kallsyms _copy_from_user 2247 0.1560 kallsyms schedule 2244 0.1558 libpthread-2.27.so __pthread_enable_asynccancel 2226 0.1545 kallsyms futex_wait 2224 0.1544 kallsyms __enqueue_entity 2188 0.1519 kallsyms dbs_update_util_handler 2178 0.1512 kallsyms update_min_vruntime 2175 0.1510 libc-2.27.so vsnprintf 2166 0.1503 kallsyms __perf_event_task_sched_out 2164 0.1502 libc-2.27.so __strcpy_sse2_unaligned 2153 0.1494 kallsyms sys_futex 2134 0.1481 libcaf_core.so.0.15.7 std::tuple_element<0ul, std::tuple*, std::default_delete > > >::type const& std::get<0ul, caf::scheduler::worker*, std::default_delete > >(std::tuple*, std::default_delete > > const&) 2133 0.1481 bro threading::Manager::Process() 2109 0.1464 kallsyms rb_insert_color 2094 0.1454 kallsyms note_gp_changes 2077 0.1442 kallsyms rcu_check_callbacks 2071 0.1438 kallsyms activate_task 2058 0.1429 libcaf_core.so.0.15.7 std::enable_if > >::value, std::chrono::duration > >::type std::chrono::duration_cast >, long, std::ratio<1l, 1l> >(std::chrono::duration > const&) 2036 0.1413 kallsyms resched_curr 2017 0.1400 bro ODesc::StartsWithEscapeSequence(char const*, char const*) 2003 0.1390 kallsyms clear_buddies 2002 0.1390 kallsyms pick_next_entity 1980 0.1374 kallsyms sched_clock_cpu 1970 0.1367 kallsyms cpu_load_update 1931 0.1340 kallsyms __perf_event_task_sched_in 1929 0.1339 libcaf_core.so.0.15.7 std::common_type >, std::chrono::duration > >::type std::chrono::operator-, long, std::ratio<1l, 1l> >(std::chrono::duration > const&, std::chrono::duration > const&) 1896 0.1316 libc-2.27.so memcpy at GLIBC_2.2.5 1813 0.1258 kallsyms scheduler_tick 1812 0.1258 kallsyms __cgroup_account_cputime 1805 0.1253 bro Dictionary::NextEntry(HashKey*&, IterCookie*&, int) const 1805 0.1253 libpthread-2.27.so __pthread_mutex_cond_lock 1773 0.1231 kallsyms deactivate_task 1768 0.1227 bro iosource::Manager::FindSoonest(double*) 1738 0.1206 kallsyms queue_work_on 1733 0.1203 kallsyms futex_wake 1692 0.1174 kallsyms rb_insert_color_cached 1679 0.1165 kallsyms smp_apic_timer_interrupt 1674 0.1162 kallsyms wakeup_preempt_entity.isra.59 1667 0.1157 libpthread-2.27.so __condvar_dec_grefs 1663 0.1154 libc-2.27.so __memcpy_sse2_unaligned_erms 1649 0.1145 kallsyms prepare_exit_to_usermode 1625 0.1128 kallsyms cpuacct_account_field 1600 0.1111 kallsyms clockevents_program_event 1589 0.1103 kallsyms entry_SYSCALL_64_stage2 1586 0.1101 libcaf_core.so.0.15.7 std::chrono::duration >::duration, void>(std::chrono::duration > const&) 1585 0.1100 kallsyms cpumask_next_and 1581 0.1097 libpthread-2.27.so pthread_mutex_unlock 1555 0.1079 kallsyms get_futex_key 1536 0.1066 bro threading::Queue::MaybeReady() 1526 0.1059 kallsyms futex_wait_setup 1525 0.1059 libbroker.so.. std::enable_if > >::value, std::chrono::duration > >::type std::chrono::duration_cast >, long, std::ratio<1l, 1000000l> >(std::chrono::duration > const&) 1517 0.1053 kallsyms enqueue_hrtimer 1490 0.1034 bro threading::MsgThread::RetrieveIn() 1462 0.1015 kallsyms x86_pmu_disable 1451 0.1007 kallsyms __accumulate_pelt_segments 1448 0.1005 kallsyms cpu_needs_another_gp 1443 0.1002 bro __gthread_active_p() 1439 0.0999 kallsyms rb_erase_cached 1421 0.0986 libcaf_core.so.0.15.7 caf::scheduler::worker::id() const 1400 0.0972 kallsyms tick_sched_timer 1399 0.0971 kallsyms set_next_buddy 1398 0.0970 kallsyms check_preempt_curr 1397 0.0970 kallsyms rcu_note_context_switch 1379 0.0957 bro std::chrono::time_point > >::time_since_epoch() const 1372 0.0952 kallsyms __list_del_entry_valid 1366 0.0948 kallsyms sched_avg_update 1353 0.0939 kallsyms __hrtimer_get_next_event 1339 0.0929 kallsyms __list_add_valid 1334 0.0926 kallsyms profile_tick 1313 0.0911 bro threading::Manager::NextTimestamp(double*) 1303 0.0904 bro RecordVal::RecordVal(RecordType*) 1275 0.0885 libcaf_core.so.0.15.7 std::chrono::duration > std::chrono::__duration_cast_impl >, std::ratio<1000000l, 1l>, long, false, true>::__cast >(std::chrono::duration > const&) 1271 0.0882 bro bool std::operator==(std::_Deque_iterator const&, std::_Deque_iterator const&) 1271 0.0882 kallsyms rcu_irq_exit 1262 0.0876 kallsyms hrtimer_try_to_cancel 1253 0.0870 libc-2.27.so __memcmp_sse4_1 1236 0.0858 kallsyms __intel_pmu_enable_all.constprop.17 1210 0.0840 libpcap.so.1.8.1 /usr/lib64/libpcap.so.1.8.1 1195 0.0829 kallsyms select_idle_sibling 1190 0.0826 kallsyms run_posix_cpu_timers 1181 0.0820 bro threading::Queue::Get() 1181 0.0820 kallsyms __bitmap_and 1180 0.0819 bro std::unique_lock::~unique_lock() 1177 0.0817 kallsyms futex_wait_queue_me 1174 0.0815 kallsyms account_system_index_time 1169 0.0811 bro threading::BasicThread::Fmt(char const*, ...) 1155 0.0802 bro bool std::operator==(std::_Deque_iterator const&, std::_Deque_iterator const&) 1139 0.0791 kallsyms update_group_capacity 1136 0.0789 kallsyms tick_program_event 1133 0.0786 kallsyms select_task_rq_fair 1117 0.0775 bro BroObj::BroObj() 1112 0.0772 bro copy_string(char const*) 1100 0.0764 bro Attributes::FindAttr(attr_tag) const 1093 0.0759 bro logging::WriterBackend::OnHeartbeat(double, double) 1080 0.0750 kallsyms can_migrate_task 1058 0.0734 bro bool std::chrono::operator< >, std::chrono::duration > >(std::chrono::time_point > > const&, std::chrono::time_point > > const&) 1057 0.0734 bro ODesc::FirstEscapeLoc(char const*, unsigned long) 1055 0.0732 bro StmtList::Exec(Frame*, stmt_flow_type&) const 1054 0.0732 bro threading::MsgThread::Run() 1048 0.0727 bro std::unique_lock::unique_lock(std::mutex&) 1037 0.0720 kallsyms exit_to_usermode_loop 1033 0.0717 kallsyms rb_next 1031 0.0716 bro ExprStmt::Exec(Frame*, stmt_flow_type&) const 1019 0.0707 bro BroFunc::Call(ValPList*, Frame*) const 1016 0.0705 kallsyms __update_idle_core 999 0.0693 kallsyms dbs_irq_work 998 0.0693 bro std::vector >::size() const 997 0.0692 bro DFA_State::Stats(unsigned int*, unsigned int*) 997 0.0692 bro analyzer::tcp::TCP_Analyzer::DeliverPacket(int, unsigned char const*, bool, unsigned long, IP_Hdr const*, int) 983 0.0682 kallsyms retint_kernel 976 0.0677 bro void __gnu_cxx::new_allocator::construct(threading::BasicOutputMessage**, threading::BasicOutputMessage* const&) 965 0.0670 kallsyms hash_futex 965 0.0670 kallsyms hrtimer_forward 962 0.0668 kallsyms rcu_nmi_exit 929 0.0645 libc-2.27.so _IO_no_init 918 0.0637 bro std::cv_status std::condition_variable::wait_for >(std::unique_lock&, std::chrono::duration > const&) 914 0.0634 kallsyms get_futex_key_refs.isra.13 910 0.0632 bro Dictionary::Lookup(void const*, int, unsigned long) const 901 0.0625 kallsyms smp_irq_work_interrupt 888 0.0616 kallsyms irq_enter 882 0.0612 bro safe_vsnprintf(char*, unsigned long, char const*, __va_list_tag*) 876 0.0608 libc-2.27.so _IO_str_init_static_internal 871 0.0605 bro operator new(unsigned long, void*) 862 0.0598 bro DebugLogger::IsEnabled(DebugStream) const 859 0.0596 kallsyms ttwu_do_activate 843 0.0585 bro std::unique_lock::unlock() 841 0.0584 bro std::cv_status std::condition_variable::__wait_until_impl > >(std::unique_lock&, std::chrono::time_point > > const&) 841 0.0584 kallsyms get_timespec64 839 0.0582 kallsyms rcu_bh_qs 836 0.0580 kallsyms native_apic_mem_write 836 0.0580 kallsyms update_cfs_rq_h_load 827 0.0574 bro std::_Rb_tree_const_iterator, std::allocator > >::_Rb_tree_const_iterator(std::_Rb_tree_node_base const*) 822 0.0571 kallsyms get_pwq.isra.20 821 0.0570 bro threading::HeartbeatMessage::~HeartbeatMessage() 820 0.0569 bro __gthread_cond_timedwait(pthread_cond_t*, pthread_mutex_t*, timespec const*) 818 0.0568 bro EventHandler::operator bool() const 815 0.0566 kallsyms ktime_get_update_offsets_now 799 0.0555 kallsyms raise_softirq 779 0.0541 bro std::_Rb_tree, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::_Identity, std::allocator > >, std::less, std::allocator > >, std::allocator, std::allocator > > >::end() const 778 0.0540 bro threading::HeartbeatMessage::Process() 778 0.0540 kallsyms ttwu_do_wakeup 778 0.0540 libpthread-2.27.so __lll_unlock_wake 772 0.0536 libc-2.27.so __mempcpy_sse2_unaligned 770 0.0534 bro threading::Queue::Ready() 760 0.0528 bro UnaryExpr::Eval(Frame*) const 750 0.0521 kallsyms set_next_entity 749 0.0520 kallsyms intel_pmu_disable_all 746 0.0518 bro logging::WriterBackend::Write(int, int, threading::Value***) 745 0.0517 kallsyms task_tick_numa 738 0.0512 bro threading::formatter::Ascii::Describe(ODesc*, threading::Value*, std::__cxx11::basic_string, std::allocator > const&) const 734 0.0509 bro __gthread_mutex_unlock(pthread_mutex_t*) 725 0.0503 bro __gthread_mutex_lock(pthread_mutex_t*) 716 0.0497 bro std::_Rb_tree_const_iterator, std::allocator > >::operator!=(std::_Rb_tree_const_iterator, std::allocator > > const&) const 708 0.0491 kallsyms cpu_load_update_active 705 0.0489 bro EventHandlerPtr::operator bool() const 703 0.0488 kallsyms rcu_irq_enter 703 0.0488 libc-2.27.so sched_yield 695 0.0482 bro threading::BasicThread::Name() const 687 0.0477 bro NameExpr::Eval(Frame*) const 687 0.0477 kallsyms wq_worker_waking_up 681 0.0473 bro NetSessions::DoNextPacket(double, Packet const*, IP_Hdr const*, EncapsulationStack const*) 681 0.0473 kallsyms __cgroup_account_cputime_field 679 0.0471 kallsyms idr_find 677 0.0470 bro threading::BasicThread::Killed() const 677 0.0470 kallsyms lock_hrtimer_base.isra.21 677 0.0470 libc-2.27.so _IO_old_init 663 0.0460 bro threading::Message::Name() const 659 0.0457 bro std::_Deque_iterator::difference_type std::operator-(std::_Deque_iterator const&, std::_Deque_iterator const&) 655 0.0455 bro RuleMatcher::Match(RuleEndpointState*, Rule::PatternType, unsigned char const*, int, bool, bool, bool) 621 0.0431 bro std::unique_lock::lock() 619 0.0430 bro BroObj::~BroObj() 614 0.0426 bro logging::writer::Ascii::DoHeartbeat(double, double) 614 0.0426 bro std::mutex::lock() 611 0.0424 bro ID::AsType() 605 0.0420 kallsyms __intel_pmu_disable_all 603 0.0419 bro threading::MsgThread::SendOut(threading::BasicOutputMessage*, bool) 599 0.0416 bro std::cv_status std::condition_variable::wait_until > >(std::unique_lock&, std::chrono::time_point > > const&) 597 0.0414 kallsyms sched_clock 593 0.0412 kallsyms rcu_all_qs 592 0.0411 bro Val::~Val() 588 0.0408 kallsyms put_prev_task_fair 586 0.0407 bro threading::Message::Message(char const*) 577 0.0401 kallsyms irqtime_account_process_tick.isra.10 563 0.0391 kallsyms rb_erase 560 0.0389 bro plugin::Manager::HavePluginForHook(plugin::HookType) const 558 0.0387 bro iosource::IOSource::IsOpen() const 551 0.0382 bro std::_Rb_tree, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::_Identity, std::allocator > >, std::less, std::allocator > >, std::allocator, std::allocator > > >::begin() const 533 0.0370 bro Attr::Tag() const 521 0.0362 libc-2.27.so isprint 520 0.0361 bro std::deque >::empty() const 518 0.0360 kallsyms find_first_bit 517 0.0359 kallsyms set_task_cpu 513 0.0356 bro BaseList::append(void*) 513 0.0356 bro SerialObj::SerialObj() 512 0.0355 bro threading::MsgThread::Debug(DebugStream, char const*) 512 0.0355 libz.so.1.2.11 /usr/lib64/libz.so.1.2.11 509 0.0353 bro base_type_no_ref(TypeTag) 509 0.0353 bro std::mutex::unlock() 506 0.0351 bro threading::acquire_lock(std::mutex&) 505 0.0351 bro std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >::end() const 495 0.0344 kallsyms plist_add 493 0.0342 bro Val::AsRecord() const 488 0.0339 kallsyms rcu_segcblist_ready_cbs 484 0.0336 kallsyms pick_next_task_idle 483 0.0335 bro SegmentProfiler::SegmentProfiler(SegmentStatsReporter*, char const*) 483 0.0335 bro threading::MsgThread::Failed() const 475 0.0330 bro IfStmt::DoExec(Frame*, Val*, stmt_flow_type&) const 467 0.0324 kallsyms irq_work_queue 465 0.0323 kallsyms __msecs_to_jiffies 464 0.0322 kallsyms native_apic_mem_read 463 0.0321 bro ValPList::operator[](int) const 460 0.0319 bro TransientID::TransientID() 455 0.0316 bro BaseList::BaseList(int) 451 0.0313 bro BroType::InternalType() const 449 0.0312 bro std::chrono::duration > std::chrono::__duration_cast_impl >, std::ratio<1000000000l, 1l>, long, false, true>::__cast >(std::chrono::duration > const&) 447 0.0310 bro threading::formatter::Ascii::Describe(ODesc*, int, threading::Field const* const*, threading::Value**) const 439 0.0305 kallsyms __remove_hrtimer 433 0.0301 bro std::__deque_buf_size(unsigned long) 427 0.0296 bro RecordVal::Assign(int, Val*, Opcode) 423 0.0294 bro BinaryExpr::Eval(Frame*) const 420 0.0292 bro std::_List_iterator::operator++(int) 420 0.0292 bro threading::BasicOutputMessage::BasicOutputMessage(char const*) 418 0.0290 bro std::mutex* std::__addressof(std::mutex&) 415 0.0288 kallsyms cgroup_cpu_stat_updated 412 0.0286 bro BaseList::clear() 412 0.0286 kallsyms get_futex_value_locked 409 0.0284 bro ODesc::AddBytesRaw(void const*, unsigned int) 404 0.0280 kallsyms rcu_nmi_enter 401 0.0278 bro analyzer::tcp::ContentLine_Analyzer::DoDeliverOnce(int, unsigned char const*) 400 0.0278 libc-2.27.so _IO_setb 397 0.0276 bro threading::Message::~Message() 396 0.0275 bro Stmt::RegisterAccess() const 387 0.0269 bro Packet::Init(int, timeval*, unsigned int, unsigned int, unsigned char const*, int, std::__cxx11::basic_string, std::allocator >) 372 0.0258 bro std::__detail::_List_node_header::_M_init() 371 0.0258 bro __gnu_cxx::__aligned_membuf, std::allocator > >::_M_ptr() const 370 0.0257 libcaf_core.so.0.15.7 std::this_thread::yield() 369 0.0256 libcaf_core.so.0.15.7 __gthread_yield() 368 0.0255 bro CallExpr::Eval(Frame*) const 367 0.0255 kallsyms irq_work_run 365 0.0253 bro void __gnu_cxx::new_allocator::destroy(threading::BasicInputMessage**) 361 0.0251 bro std::_Rb_tree_node, std::allocator > >::_M_valptr() const 361 0.0251 bro threading::DebugMessage::DebugMessage(DebugStream, threading::MsgThread*, char const*) 359 0.0249 bro bool std::chrono::operator< , long, std::ratio<1l, 1000000000l> >(std::chrono::duration > const&, std::chrono::duration > const&) 359 0.0249 kallsyms find_get_entry 358 0.0248 bro std::deque >::push_back(threading::BasicOutputMessage* const&) 358 0.0248 bro threading::MsgThread::SendOut(threading::BasicOutputMessage*) 355 0.0246 bro starts_with(char const*, char const*, unsigned long) 353 0.0245 bro threading::BasicThread::Terminating() const 352 0.0244 bro NetSessions::NextPacket(double, Packet const*) 350 0.0243 bro Val::Ref() 349 0.0242 kallsyms get_work_pool 345 0.0239 bro BaseList::replace(int, void*) 340 0.0236 libpthread-2.27.so _pthread_cleanup_push 339 0.0235 kallsyms _cond_resched 338 0.0235 bro net_run() 332 0.0230 bro analyzer::Analyzer::NextPacket(int, unsigned char const*, bool, unsigned long, IP_Hdr const*, int) 330 0.0229 bro AssignExpr::Eval(Frame*) const 329 0.0228 bro iosource::pcap::PcapSource::ExtractNextPacket(Packet*) 326 0.0226 bro EventMgr::Drain() 326 0.0226 bro iosource::PktSrc::Process() 326 0.0226 kallsyms retint_user 326 0.0226 kallsyms task_ctx_sched_out 320 0.0222 libc-2.27.so __strcmp_sse2_unaligned 319 0.0221 bro Val::Type() 318 0.0221 bro Expr::IsError() const 318 0.0221 kallsyms ext4_mark_iloc_dirty 316 0.0219 libc-2.27.so __fread_chk 315 0.0219 bro DictEntryPList::operator[](int) const 313 0.0217 bro std::_Rb_tree_const_iterator, std::allocator > >::operator++() 313 0.0217 bro std::__shared_ptr::get() const 313 0.0217 kallsyms migrate_task_rq_fair 312 0.0217 bro std::enable_if > >::value, std::chrono::time_point > > >::type std::chrono::time_point_cast >, std::chrono::_V2::system_clock, std::chrono::duration > >(std::chrono::time_point > > const&) 310 0.0215 libc-2.27.so __mpn_divrem 309 0.0214 bro logging::WriterFrontend::FlushWriteBuffer() 308 0.0214 bro std::_Rb_tree_const_iterator, std::allocator > >::operator*() const 307 0.0213 bro std::enable_if > >::value, std::chrono::duration > >::type std::chrono::duration_cast >, long, std::ratio<1l, 1l> >(std::chrono::duration > const&) 306 0.0212 bro threading::Queue::Put(threading::BasicInputMessage*) 304 0.0211 bro __gnu_cxx::__aligned_membuf::_M_ptr() 304 0.0211 kallsyms change_protection 303 0.0210 bro PQ_Element::Time() const 303 0.0210 libc-2.27.so tolower 302 0.0210 bro Trigger::EvaluatePending() 302 0.0210 kallsyms radix_tree_lookup 299 0.0208 bro void std::__cxx11::basic_string, std::allocator >::_M_construct(char*, char*, std::forward_iterator_tag) 298 0.0207 bro std::_Rb_tree, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::_Identity, std::allocator > >, std::less, std::allocator > >, std::allocator, std::allocator > > >::empty() const 296 0.0205 bro Frame::NthElement(int) 296 0.0205 kallsyms check_cfs_rq_runtime 295 0.0205 bro iosource::PktSrc::ExtractNextPacketInternal() 292 0.0203 bro Val::~Val() 292 0.0203 bro std::unique_lock::mutex() const 292 0.0203 libbroker.so.. bool std::chrono::operator> >, std::chrono::duration > >(std::chrono::time_point > > const&, std::chrono::time_point > > const&) 290 0.0201 bro Specific_RE_Matcher::LongestMatch(unsigned char const*, int) 288 0.0200 bro ListExpr::Eval(Frame*) const 288 0.0200 bro TableVal::MemoryAllocation() const 288 0.0200 kallsyms detach_entity_cfs_rq 288 0.0200 kallsyms timerqueue_del 287 0.0199 libc-2.27.so fread 286 0.0199 bro std::chrono::time_point >, std::chrono::duration > >::type> std::chrono::operator+ >, long, std::ratio<1l, 1000000000l> >(std::chrono::time_point > > const&, std::chrono::duration > const&) 285 0.0198 kallsyms tick_sched_handle 284 0.0197 kallsyms perf_pmu_disable.part.92 282 0.0196 bro Connection::IsReuse(double, unsigned char const*) 282 0.0196 bro std::unordered_set, std::allocator >, std::hash, std::allocator > >, std::equal_to, std::allocator > >, std::allocator, std::allocator > > >::empty() const 281 0.0195 bro analyzer::Analyzer::FirstSupportAnalyzer(bool) 279 0.0194 bro RecordType::FieldDecl(int) 274 0.0190 bro pad_size(unsigned long) 273 0.0189 bro net_packet_dispatch(double, Packet const*, iosource::PktSrc*) 273 0.0189 bro threading::HeartbeatMessage::~HeartbeatMessage() 272 0.0189 bro analyzer::Analyzer::Conn() const 271 0.0188 bro StmtPList::operator[](int) const 271 0.0188 bro std::common_type >, std::chrono::duration > >::type std::chrono::operator- >, std::chrono::duration > >(std::chrono::time_point > > const&, std::chrono::time_point > > const&) 271 0.0188 bro std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >::begin() const 270 0.0187 bro std::_List_iterator::_List_iterator(std::__detail::_List_node_base*) 269 0.0187 bro Func::HasBodies() const 269 0.0187 bro eval_list(Frame*, ListExpr const*) 269 0.0187 bro std::_Rb_tree_const_iterator, std::allocator > >::_Rb_tree_const_iterator() 266 0.0185 bro __gnu_cxx::__aligned_membuf, std::allocator > >::_M_addr() const 266 0.0185 bro __gnu_cxx::__aligned_membuf::_M_ptr() 263 0.0183 bro threading::OutputMessage::OutputMessage(char const*, threading::MsgThread*) 263 0.0183 libbroker.so.. broker::endpoint::advance_time(std::chrono::time_point > >) 261 0.0181 libc-2.27.so __GI___printf_fp_l 260 0.0180 bro analyzer::Analyzer::ForwardStream(int, unsigned char const*, bool) 259 0.0180 bro threading::DebugMessage::Process() 258 0.0179 kallsyms sys_sched_yield 258 0.0179 kallsyms wake_q_add 257 0.0178 bro std::__cxx11::list >::end() 256 0.0178 bro RecordVal::Lookup(int) const 255 0.0177 kallsyms __update_load_avg_blocked_se.isra.32 253 0.0176 libc-2.27.so __memset_sse2_unaligned 252 0.0175 bro std::chrono::duration >::duration, void>(std::chrono::duration > const&) 250 0.0174 bro RecordType::FieldType(int) const 250 0.0174 bro std::_List_node::_M_valptr() 248 0.0172 bro rules_lex() 248 0.0172 bro std::_Deque_iterator::_Deque_iterator(std::_Deque_iterator const&) 248 0.0172 bro std::_List_iterator::operator*() const 247 0.0171 bro Reassembler::NewBlock(double, unsigned long, unsigned long, unsigned char const*) 247 0.0171 kallsyms calc_global_load_tick 247 0.0171 kallsyms native_apic_wait_icr_idle 247 0.0171 kallsyms rcu_segcblist_future_gp_needed 246 0.0171 bro BroString::Reset() 246 0.0171 bro threading::MsgThread::MightHaveOut() 246 0.0171 kallsyms __switch_to 246 0.0171 kallsyms decay_load_missed 243 0.0169 bro threading::InputMessage::Object() const 242 0.0168 bro TypeDeclPList::operator[](int) const 241 0.0167 bro ODesc::AddBytes(void const*, unsigned int) 241 0.0167 bro TableEntryVal::ExpireAccessTime() const 240 0.0167 bro safe_malloc(unsigned long) 238 0.0165 kallsyms update_process_times 237 0.0165 bro threading::MsgThread::SendIn(threading::BasicInputMessage*, bool) 236 0.0164 kallsyms cpumask_next_wrap 235 0.0163 bro std::__cxx11::list >::begin() 232 0.0161 bro std::_Deque_iterator::operator*() const 232 0.0161 bro std::__cxx11::list >::begin() 232 0.0161 libpthread-2.27.so _pthread_cleanup_pop 231 0.0160 bro std::_Hashtable, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::allocator, std::allocator > >, std::__detail::_Identity, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::empty() const 230 0.0160 bro iosource::PktSrc::NextTimestamp(double*) 228 0.0158 bro IP_Hdr::TotalLen() const 227 0.0158 bro RecordVal::MemoryAllocation() const 227 0.0158 bro Val::Val(int, TypeTag) 227 0.0158 bro std::chrono::time_point > >::time_since_epoch() const 224 0.0155 bro std::_List_iterator::operator*() const 223 0.0155 kallsyms propagate_entity_cfs_rq.isra.70 221 0.0153 bro analyzer::http::HTTP_Analyzer::DeliverStream(int, unsigned char const*, bool) 220 0.0153 bro threading::Value::~Value() 220 0.0153 kallsyms __handle_mm_fault 219 0.0152 bro DebugLogger::IsVerbose() const 218 0.0151 bro analyzer::tcp::TCP_Analyzer::ExtractTCP_Header(unsigned char const*&, int&, int&) 217 0.0151 bro Connection::NextPacket(double, int, IP_Hdr const*, int, int, unsigned char const*&, int&, int&, Packet const*) 217 0.0151 bro modp_dtoa 217 0.0151 kallsyms attach_entity_load_avg 216 0.0150 bro std::_List_iterator::operator!=(std::_List_iterator const&) const 216 0.0150 libc-2.27.so htons 215 0.0149 bro Reassembler::CheckOverlap(DataBlock*, DataBlock*, unsigned long, unsigned long, unsigned char const*) 215 0.0149 bro analyzer::http::HTTP_Entity::Deliver(int, char const*, int) 215 0.0149 kallsyms ttwu_stat 215 0.0149 libbroker.so.. std::multimap > >, std::pair, std::less > > >, std::allocator > > const, std::pair > > >::empty() const 214 0.0149 bro Val::AsRecordVal() 213 0.0148 bro Reassembler::AddAndCheck(DataBlock*, unsigned long, unsigned long, unsigned char const*) 213 0.0148 bro post_execute_stmt(Stmt*, Frame*, Val*, stmt_flow_type*) 213 0.0148 bro std::enable_if > >::value, std::chrono::duration > >::type std::chrono::duration_cast >, long, std::ratio<1l, 1000000000l> >(std::chrono::duration > const&) 212 0.0147 bro std::__shared_ptr_access::operator->() const 212 0.0147 bro std::queue > >::empty() const 210 0.0146 bro bool std::chrono::operator< , long, std::ratio<1l, 1l> >(std::chrono::duration > const&, std::chrono::duration > const&) 209 0.0145 bro Dictionary::Lookup(HashKey const*) const 209 0.0145 bro __gnu_cxx::__aligned_membuf::_M_addr() 209 0.0145 bro file_analysis::File::DeliverStream(unsigned char const*, unsigned long) 209 0.0145 bro std::_List_iterator::_List_iterator(std::__detail::_List_node_base*) 209 0.0145 bro std::_List_node::_M_valptr() 209 0.0145 libpthread-2.27.so __condvar_confirm_wakeup 208 0.0144 bro Packet::ProcessLayer2() 208 0.0144 bro std::_List_iterator::_List_iterator(std::__detail::_List_node_base*) 208 0.0144 kallsyms account_process_tick 207 0.0144 bro SerialObj::~SerialObj() 207 0.0144 libbroker.so.. std::_Rb_tree > >, std::pair > > const, std::pair >, std::_Select1st > > const, std::pair > >, std::less > > >, std::allocator > > const, std::pair > > >::empty() const 206 0.0143 bro BinaryExpr::Fold(Val*, Val*) const 206 0.0143 bro ValPList::append(Val*) 206 0.0143 bro broker::endpoint::is_shutdown() const 206 0.0143 kallsyms __default_send_IPI_shortcut 205 0.0142 bro std::common_type >, std::chrono::duration > >::type std::chrono::operator+, long, std::ratio<1l, 1000000000l> >(std::chrono::duration > const&, std::chrono::duration > const&) 205 0.0142 libc-2.27.so __strncpy_sse2_unaligned 203 0.0141 bro std::__cxx11::list >::begin() 203 0.0141 kallsyms irq_work_tick 202 0.0140 kallsyms hrtimer_reprogram 202 0.0140 ld-2.27.so do_lookup_x 201 0.0140 bro Val::AsNonConstRecord() 201 0.0140 bro analyzer::tcp::TCP_Analyzer::IsReuse(double, unsigned char const*) 201 0.0140 bro std::_List_iterator::operator++() 200 0.0139 bro RemoteSerializer::NextTimestamp(double*) 199 0.0138 kallsyms tick_nohz_tick_stopped 198 0.0137 bro std::deque >::size() const 196 0.0136 kallsyms cpus_share_cache 194 0.0135 bro logging::FlushWriteBufferMessage::Process() 194 0.0135 kallsyms yield_task_fair 193 0.0134 bro analyzer::tcp::ContentLine_Analyzer::DoDeliver(int, unsigned char const*) 193 0.0134 bro std::deque >::begin() 192 0.0133 bro ValPList::replace(int, Val*) 191 0.0133 kallsyms page_fault 191 0.0133 kallsyms switch_mm_irqs_off 190 0.0132 bro std::deque >::empty() const 189 0.0131 bro PriorityQueue::BubbleDown(int) 189 0.0131 kallsyms hrtimer_init 188 0.0130 bro analyzer::conn_size::ConnSize_Analyzer::DeliverPacket(int, unsigned char const*, bool, unsigned long, IP_Hdr const*, int) 188 0.0130 bro threading::acquire_lock(std::mutex&) 185 0.0128 bro BaseList::~BaseList() 185 0.0128 bro threading::BasicOutputMessage* const& std::forward(std::remove_reference::type&) 185 0.0128 libc-2.27.so _IO_sgetn 184 0.0128 bro BoolExpr::Eval(Frame*) const 184 0.0128 bro BroType::BroType(TypeTag, bool) 184 0.0128 bro BuildConnIDHashKey(ConnID const&) 184 0.0128 bro Frame::SetNextStmt(Stmt*) 184 0.0128 bro pre_execute_stmt(Stmt*, Frame*) 182 0.0126 bro std::deque >::pop_front() 181 0.0126 bro brolex() 178 0.0124 bro BroType::Ref() 177 0.0123 bro RuleMatcherState::Match(Rule::PatternType, unsigned char const*, int, bool, bool, bool, bool) 177 0.0123 bro analyzer::http::HTTP_Entity::SubmitData(int, char const*) 175 0.0121 bro analyzer::tcp::TCP_Reassembler::AckReceived(unsigned long) 174 0.0121 bro threading::MsgThread::HasOut() 174 0.0121 kallsyms native_iret 173 0.0120 kallsyms llist_add_batch 172 0.0119 bro std::_Hashtable, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::allocator, std::allocator > >, std::__detail::_Identity, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::size() const 170 0.0118 bro BroString::Len() const 170 0.0118 bro std::chrono::duration > std::chrono::__duration_cast_impl >, std::ratio<1l, 1000000000l>, long, true, false>::__cast >(std::chrono::duration > const&) 168 0.0117 bro PriorityQueue::Top() const 168 0.0117 bro iosource::IOSource::IsIdle() const 167 0.0116 bro Func::Flavor() const 167 0.0116 kallsyms wake_up_process 165 0.0115 bro TimerMgr::Advance(double, int) 164 0.0114 bro is_vector(Val*) 163 0.0113 bro DebugLogger::Log(DebugStream, char const*, ...) 163 0.0113 bro Frame::Reset(int) 163 0.0113 bro std::common_type >, std::chrono::duration > >::type std::chrono::operator-, long, std::ratio<1l, 1l> >(std::chrono::duration > const&, std::chrono::duration > const&) 162 0.0112 bro analyzer::tcp::TCP_Reassembler::DataSent(double, unsigned long, int, unsigned char const*, analyzer::tcp::TCP_Flags, bool) 162 0.0112 bro std::__cxx11::_List_base >::_M_clear() 161 0.0112 bro Dictionary::Init(int) 161 0.0112 bro analyzer::http::HTTP_Entity::DeliverBodyClear(int, char const*, int) 161 0.0112 bro std::__cxx11::list >::begin() 161 0.0112 bro std::chrono::time_point > >::time_point(std::chrono::duration > const&) 161 0.0112 kallsyms __acct_update_integrals 161 0.0112 libc-2.27.so _IO_file_xsgetn 160 0.0111 bro Val::Val(bool, TypeTag) 159 0.0110 bro FieldExpr::Fold(Val*) const 159 0.0110 bro std::_List_iterator::_List_iterator(std::__detail::_List_node_base*) 159 0.0110 bro std::__cxx11::list >::end() 158 0.0110 bro threading::InputMessage::~InputMessage() 158 0.0110 bro void std::vector, std::allocator > >::_M_realloc_insert >(__gnu_cxx::__normal_iterator*, std::vector, std::allocator > > >, std::unique_lock&&) 157 0.0109 bro std::__cxx11::list >::end() 156 0.0108 bro RecordType::FieldOffset(char const*) const 156 0.0108 bro analyzer::mime::MIME_Entity::DecodeDataLine(int, char const*, int) 156 0.0108 bro current_time(bool) 155 0.0108 bro SegmentProfiler::~SegmentProfiler() 155 0.0108 bro file_analysis::File::DeliverChunk(unsigned char const*, unsigned long, unsigned long) 155 0.0108 bro std::_List_iterator::operator!=(std::_List_iterator const&) const 154 0.0107 bro Frame::Release() 154 0.0107 bro threading::OutputMessage::OutputMessage(char const*, logging::WriterFrontend*) 154 0.0107 libm-2.27.so __ieee754_log_avx 153 0.0106 bro Connection::CheckHistory(unsigned int, char) 153 0.0106 bro __gnu_cxx::__aligned_membuf::_M_addr() 153 0.0106 bro delete_vals(ValPList*) 153 0.0106 bro std::_Deque_iterator::_Deque_iterator(std::_Deque_iterator const&) 153 0.0106 bro threading::MsgThread::RetrieveOut() 151 0.0105 bro std::__cxx11::_List_base >::_M_get_size() const 150 0.0104 bro ID::HasVal() const 150 0.0104 bro std::insert_iterator > > std::__niter_base > > >(std::insert_iterator > >) 149 0.0103 bro BroType::AsFuncType() 149 0.0103 bro RandTest::end(double*, double*, double*, double*, double*) 149 0.0103 bro analyzer::mime::MIME_Entity::Deliver(int, char const*, int) 149 0.0103 bro analyzer::tcp::TCP_Flags::SYN() const 148 0.0103 bro Val::Val(BroType*) 147 0.0102 bro ID::IsGlobal() const 147 0.0102 kallsyms hrtick_update 146 0.0101 bro non-virtual thunk to RemoteSerializer::NextTimestamp(double*) 146 0.0101 bro std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > >::empty() const 145 0.0101 bro analyzer::conn_size::ConnSize_Analyzer::CheckSizes(bool) 145 0.0101 bro analyzer::tcp::ContentLine_Analyzer::DeliverStream(int, unsigned char const*, bool) 145 0.0101 bro analyzer::tcp::TCP_Endpoint::DataSent(double, unsigned long, int, int, unsigned char const*, IP_Hdr const*, tcphdr const*) 145 0.0101 kallsyms __fget 144 0.0100 bro Frame::Clear() 144 0.0100 kallsyms start_this_handle 143 0.0099 bro Dictionary::DeInit() 143 0.0099 bro analyzer::tcp::TCP_Flags::RST() const 143 0.0099 bro file_analysis::Manager::DataIn(unsigned char const*, unsigned long, analyzer::Tag, Connection*, bool, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&) 143 0.0099 bro void std::allocator_traits >::construct(std::allocator&, threading::BasicOutputMessage**, threading::BasicOutputMessage* const&) 143 0.0099 kallsyms ktime_add_safe 143 0.0099 libc-2.27.so _itoa_word 142 0.0099 bro BuiltinFunc::Call(ValPList*, Frame*) const 142 0.0099 bro RuleEndpointState::MatcherPList::operator[](int) const 141 0.0098 bro analyzer::mime::MIME_Message::Deliver(int, char const*, int) 141 0.0098 kallsyms __sb_start_write 139 0.0096 bro HashKey::~HashKey() 139 0.0096 bro NullStmt::Exec(Frame*, stmt_flow_type&) const 139 0.0096 kallsyms __x86_indirect_thunk_r14 138 0.0096 bro BroString::MemoryAllocation() const 138 0.0096 bro NetSessions::CheckHeaderTrunc(int, unsigned int, unsigned int, Packet const*, EncapsulationStack const*) 138 0.0096 bro base_type(TypeTag) 138 0.0096 kallsyms __fget_light 137 0.0095 bro bro_broker::Manager::Active() 137 0.0095 bro logging::writer::Ascii::DoWrite(int, threading::Field const* const*, threading::Value**) 137 0.0095 bro std::chrono::time_point > >::time_point(std::chrono::duration > const&) 135 0.0094 bro analyzer::pia::PIA::Match(Rule::PatternType, unsigned char const*, int, bool, bool, bool, bool) 135 0.0094 bro std::unique_lock::unique_lock(std::unique_lock&&) 134 0.0093 bro SegmentProfiler::SegmentProfiler(SegmentStatsReporter*, Location const*) 134 0.0093 bro analyzer::Analyzer::NextStream(int, unsigned char const*, bool) 134 0.0093 bro analyzer::Analyzer::UpdateConnVal(RecordVal*) 134 0.0093 kallsyms clear_page_rep 133 0.0092 bro MutableVal::MutableVal(BroType*) 133 0.0092 bro Val::Val(unsigned long, TypeTag) 133 0.0092 libc-2.27.so _IO_vfscanf 132 0.0092 bro IP_Hdr::HdrLen() const 132 0.0092 bro RecordCoerceExpr::Fold(Val*) const 132 0.0092 bro TypeList::~TypeList() 131 0.0091 bro Dictionary::MemoryAllocation() const 131 0.0091 bro std::__detail::_List_node_header::_List_node_header() 131 0.0091 bro threading::BasicInputMessage::~BasicInputMessage() 131 0.0091 kallsyms tick_sched_do_timer 130 0.0090 bro ODesc::Grow(unsigned int) 130 0.0090 bro std::chrono::duration > std::chrono::__duration_cast_impl >, std::ratio<1000000000l, 1l>, double, false, true>::__cast >(std::chrono::duration > const&) 130 0.0090 kallsyms jbd2_journal_stop 129 0.0090 bro CompositeHash::ComputeSingletonHash(Val const*, int) const 129 0.0090 libc-2.27.so hack_digit 128 0.0089 bro NetSessions::DumpPacket(Packet const*, int) 128 0.0089 bro TableVal::DoExpire(double) 128 0.0089 bro analyzer::tcp::TCP_Analyzer::UpdateStateMachine(double, analyzer::tcp::TCP_Endpoint*, analyzer::tcp::TCP_Endpoint*, unsigned int, unsigned int, int, int, int, analyzer::tcp::TCP_Flags, int&, int&) 127 0.0088 bro analyzer::tcp::TCP_Reassembler::BlockInserted(DataBlock*) 126 0.0087 bro Connection::CheckFlowLabel(bool, unsigned int) 126 0.0087 bro RuleMatcherState::MatcherInitialized(bool) 126 0.0087 bro TableEntryVal::Value() 126 0.0087 bro analyzer::SupportAnalyzer::ForwardStream(int, unsigned char const*, bool) 126 0.0087 bro std::pair::pair(char const* const&, int&&) 126 0.0087 bro std::queue > >::empty() const 125 0.0087 bro TableVal::Lookup(Val*, bool) 125 0.0087 bro analyzer::tcp::TCP_Endpoint::NoDataAcked() const 124 0.0086 bro seq_delta(unsigned int, unsigned int) 124 0.0086 bro std::deque >::front() 124 0.0086 kallsyms fsnotify 123 0.0085 bro CompositeHash::ComputeHash(Val const*, int) const 123 0.0085 bro analyzer::tcp::TCP_Reassembler::DeliverBlock(unsigned long, int, unsigned char const*) 123 0.0085 bro iosource::Manager::Size() const 123 0.0085 bro std::enable_if > >::value, std::chrono::duration > >::type std::chrono::duration_cast >, long, std::ratio<1l, 1000000000l> >(std::chrono::duration > const&) 122 0.0085 bro IPAddr::IPAddr(in_addr const&) 122 0.0085 bro std::__shared_ptr_access::_M_get() const 121 0.0084 bro std::mutex::native_handle() 120 0.0083 bro TimerMgr::Time() const 120 0.0083 bro Val::AsCount() const 120 0.0083 kallsyms account_user_time 119 0.0083 bro Val::IsZero() const 119 0.0083 kallsyms arch_irq_work_raise 118 0.0082 bro Expr::Type() const 118 0.0082 bro MutableVal::LoggingAccess() const 117 0.0081 bro Connection::CheckEncapsulation(EncapsulationStack const*) 117 0.0081 bro IPAddr::operator=(IPAddr const&) 116 0.0081 bro Frame::Frame(int, BroFunc const*, ValPList const*) 116 0.0081 bro MutableVal::~MutableVal() 116 0.0081 bro Val::Val(TypeTag) 115 0.0080 bro Connection::BuildConnVal() 115 0.0080 bro analyzer::tcp::TCP_Analyzer::DeliverData(double, unsigned char const*, int, int, IP_Hdr const*, tcphdr const*, analyzer::tcp::TCP_Endpoint*, unsigned long, int, analyzer::tcp::TCP_Flags) 114 0.0079 bro ID::IsInternalGlobal() const 114 0.0079 bro analyzer::tcp::TCP_Endpoint::IsOrig() const 114 0.0079 bro strreverse 113 0.0078 bro MutableVal::Modified() 113 0.0078 bro analyzer::tcp::TCP_Reassembler::Deliver(unsigned long, int, unsigned char const*) 113 0.0078 kallsyms __ext4_get_inode_loc 112 0.0078 bro Connection::GetTimerMgr() const 112 0.0078 bro ODesc::AddRaw(std::__cxx11::basic_string, std::allocator > const&) 112 0.0078 bro std::chrono::duration > std::chrono::__duration_cast_impl >, std::ratio<1l, 1l>, long, true, true>::__cast >(std::chrono::duration > const&) 110 0.0076 bro ODesc::IsBinary() const 109 0.0076 bro Connection::Match(Rule::PatternType, unsigned char const*, int, bool, bool, bool, bool) 109 0.0076 bro Frame::HasDelayed() const 109 0.0076 bro InitStmt::Exec(Frame*, stmt_flow_type&) const 109 0.0076 bro SerialObj::IncreaseTimeCounter() 109 0.0076 bro init_endpoint(analyzer::tcp::TCP_Endpoint*, analyzer::tcp::TCP_Flags, unsigned int, unsigned int, double) 109 0.0076 bro logging::Manager::RecordToFilterVals(logging::Manager::Stream*, logging::Manager::Filter*, RecordVal*) 109 0.0076 kallsyms wake_up_q 108 0.0075 bro DFA_Machine::StartState() const 108 0.0075 bro modp_ulitoa10 108 0.0075 kallsyms sched_slice.isra.57 107 0.0074 bro analyzer::tcp::TCP_ApplicationAnalyzer::TCP() 107 0.0074 bro file_analysis::File::UpdateLastActivityTime() 107 0.0074 bro std::chrono::duration >::duration(double const&) 107 0.0074 kallsyms __mark_inode_dirty 106 0.0074 bro ListVal::ListVal(TypeTag) 106 0.0074 bro RefExpr::Assign(Frame*, Val*, Opcode) 106 0.0074 bro ValPList::~ValPList() 106 0.0074 bro std::_List_iterator::operator!=(std::_List_iterator const&) const 106 0.0074 bro threading::MsgThread::Heartbeat() 106 0.0074 kallsyms kmem_cache_alloc 105 0.0073 bro DFA_State_Cache::GetStats(DFA_State_Cache::Stats*) 105 0.0073 bro RemoteSerializer::Poll(bool) 105 0.0073 bro logging::FlushWriteBufferMessage::FlushWriteBufferMessage(logging::WriterFrontend*) 105 0.0073 kallsyms drop_futex_key_refs.isra.15 105 0.0073 kallsyms error_entry 104 0.0072 bro AttrPList::operator[](int) const 104 0.0072 bro IP_Hdr::SrcAddr() const 104 0.0072 bro ListVal::~ListVal() 104 0.0072 bro siphash(unsigned long*, unsigned char const*, unsigned long, unsigned char const*) 104 0.0072 bro std::queue > >::size() const 103 0.0071 bro ConstExpr::Eval(Frame*) const 103 0.0071 bro ProfileLogger::Log() 103 0.0071 bro void __gnu_cxx::new_allocator::construct(threading::BasicInputMessage**, threading::BasicInputMessage* const&) 102 0.0071 bro DataBlock::DataBlock(Reassembler*, unsigned char const*, unsigned long, unsigned long, DataBlock*, DataBlock*, ReassemblerType) 102 0.0071 bro HashKey::HashBytes(void const*, int) 101 0.0070 bro ConstExpr::Value() const 101 0.0070 bro EventHandler::Call(ValPList*, bool) 101 0.0070 bro FragReassemblerTracker::~FragReassemblerTracker() 101 0.0070 bro IP_Hdr::IP_Hdr(ip const*, bool) 101 0.0070 bro iosource::IOSource::SetIdle(bool) 101 0.0070 bro std::__cxx11::list >::clear() 101 0.0070 bro std::chrono::duration >::count() const 101 0.0070 bro std::vector >::size() const 100 0.0069 bro HasFieldExpr::Fold(Val*) const 100 0.0069 bro NetSessions::LookupTimerMgr(std::__cxx11::basic_string, std::allocator > const*, bool) 100 0.0069 bro ODesc::Add(char const*, int) 100 0.0069 bro PQ_TimerMgr::Top() 100 0.0069 bro analyzer::tcp::TCP_Flags::FIN() const 99 0.0069 bro Frame::SetElement(int, Val*) 99 0.0069 bro std::_List_iterator::operator!=(std::_List_iterator const&) const 98 0.0068 bro HashKey::HashKey(void const*, int) 98 0.0068 bro update_window(analyzer::tcp::TCP_Endpoint*, unsigned int, unsigned int, unsigned int, analyzer::tcp::TCP_Flags) 97 0.0067 bro EquivClass::Size() const 97 0.0067 bro Frame::SetCall(CallExpr const*) 97 0.0067 bro ReturnStmt::Exec(Frame*, stmt_flow_type&) const 96 0.0067 bro BaseList::remove(void*) 96 0.0067 bro ID::Offset() const 96 0.0067 bro std::chrono::duration >::duration(int const&) 95 0.0066 bro Func::FType() const 95 0.0066 bro HashKey::HashKey(char const*) 95 0.0066 bro get_relative_seq(analyzer::tcp::TCP_Endpoint const*, unsigned int, unsigned int, unsigned int, bool*) 94 0.0065 bro FragReassemblerTracker::FragReassemblerTracker(NetSessions*, FragReassembler*) 94 0.0065 bro IPAddr::~IPAddr() 94 0.0065 kallsyms __find_get_block 92 0.0064 bro BroTypePList::append(BroType*) 92 0.0064 bro ListVal::Append(Val*) 92 0.0064 bro ODesc::Add(std::__cxx11::basic_string, std::allocator > const&) 92 0.0064 libc-2.27.so read 91 0.0063 bro NotExpr::Fold(Val*) const 91 0.0063 bro std::_List_iterator::operator!=(std::_List_iterator const&) const 91 0.0063 bro std::__cxx11::_List_base >::_List_impl::_List_impl() 91 0.0063 ld-2.27.so _dl_lookup_symbol_x 90 0.0062 bro IP_Hdr::~IP_Hdr() 90 0.0062 bro IndexExpr::Eval(Frame*) const 90 0.0062 bro PriorityQueue::SetElement(int, PQ_Element*) 90 0.0062 bro RecordType::NumFields() const 90 0.0062 bro StringVal::~StringVal() 90 0.0062 bro Val*&& std::forward(std::remove_reference::type&) 90 0.0062 bro analyzer::Analyzer::AppendNewChildren() 90 0.0062 bro std::iterator_traits::difference_type std::__distance(char*, char*, std::random_access_iterator_tag) 90 0.0062 kallsyms ext4_block_write_begin 90 0.0062 kallsyms ext4_da_write_begin 89 0.0062 bro Connection::OrigAddr() const 89 0.0062 bro InExpr::Fold(Val*, Val*) const 89 0.0062 bro Tag::Tag(Tag const&) 89 0.0062 bro TypeList::Append(BroType*) 89 0.0062 bro logging::Manager::ValToLogVal(Val*, BroType*) 89 0.0062 bro std::queue > >::push(threading::BasicOutputMessage* const&) 88 0.0061 bro Specific_RE_Matcher::Match(unsigned char const*, int) 88 0.0061 bro std::__cxx11::list >::list() 88 0.0061 kallsyms sched_clock_tick 87 0.0060 libc-2.27.so __rawmemchr_sse2 86 0.0060 bro PQ_TimerMgr::DoAdvance(double, int) 86 0.0060 bro analyzer::mime::MIME_Entity::NewDataLine(int, char const*, int) 86 0.0060 bro int const& std::min(int const&, int const&) 86 0.0060 bro std::insert_iterator > > std::__copy_move::__copy_m, std::insert_iterator > > >(std::_List_iterator, std::_List_iterator, std::insert_iterator > >) 86 0.0060 kallsyms vfs_write 85 0.0059 bro BroType::AsTypeList() 85 0.0059 bro Val::AsString() const 85 0.0059 kallsyms __do_page_fault 85 0.0059 libc-2.27.so snprintf 84 0.0058 bro DFA_State::NFAStateNum() const 84 0.0058 bro analyzer::http::HTTP_Entity::DeliverBody(int, char const*, int) 84 0.0058 bro file_analysis::AnalyzerSet::DrainModifications() 84 0.0058 kallsyms hrtimer_cancel 84 0.0058 kallsyms kmem_cache_free 83 0.0058 bro analyzer::tcp::TCP_Endpoint::AckWraps() const 83 0.0058 kallsyms handle_mm_fault 82 0.0057 bro operator==(IPAddr const&, IPAddr const&) 82 0.0057 bro std::__cxx11::list >::end() 82 0.0057 kallsyms ext4_da_write_end 81 0.0056 bro HashKey::HashKey(void const*, int, unsigned long) 81 0.0056 bro analyzer::tcp::TCP_Analyzer::CheckRecording(int, analyzer::tcp::TCP_Flags) 81 0.0056 bro int&& std::forward(std::remove_reference::type&) 81 0.0056 bro std::allocator >::allocator() 81 0.0056 bro std::pair::operator=(std::pair&&) 81 0.0056 bro std::vector >::pop_back() 81 0.0056 kallsyms __vfs_read 81 0.0056 tg3 /tg3 80 0.0056 bro file_analysis::Hash::DeliverStream(unsigned char const*, unsigned long) 79 0.0055 bro EqExpr::Fold(Val*, Val*) const 79 0.0055 bro analyzer::tcp::TCP_Endpoint::CheckHistory(unsigned int, char) 79 0.0055 bro std::__cxx11::_List_base >::_M_clear() 79 0.0055 kallsyms reschedule_interrupt 78 0.0054 bro TypeList::TypeList(BroType*) 78 0.0054 bro std::queue > >::front() 78 0.0054 kallsyms hrtimer_wakeup 78 0.0054 kallsyms security_file_permission 77 0.0053 bro BinaryExpr::StringFold(Val*, Val*) const 77 0.0053 bro Reassembler::TrimToSeq(unsigned long) 77 0.0053 bro Tag::~Tag() 77 0.0053 bro analyzer::Tag::~Tag() 77 0.0053 bro analyzer::tcp::ContentLine_Analyzer::SkipDeliveries() const 77 0.0053 bro operator<(IPAddr const&, IPAddr const&) 76 0.0053 bro FieldExpr::Assign(Frame*, Val*, Opcode) 76 0.0053 bro ListVal::~ListVal() 76 0.0053 bro void __gnu_cxx::new_allocator::destroy(threading::BasicOutputMessage**) 76 0.0053 kallsyms mark_page_accessed 75 0.0052 bro TraceState::DoTrace() const 75 0.0052 bro analyzer::tcp::TCP_Flags::TCP_Flags(tcphdr const*) 75 0.0052 bro std::deque >::empty() const 75 0.0052 bro update_history(analyzer::tcp::TCP_Flags, analyzer::tcp::TCP_Endpoint*, unsigned long, int) 75 0.0052 bro void std::allocator_traits >::destroy(std::allocator&, threading::BasicInputMessage**) 75 0.0052 kallsyms apic_send_IPI_self 75 0.0052 kallsyms run_local_timers 74 0.0051 bro RuleMatcher::InitEndpoint(analyzer::Analyzer*, IP_Hdr const*, int, RuleEndpointState*, bool, analyzer::pia::PIA*) 74 0.0051 kallsyms current_kernel_time64 74 0.0051 libc-2.27.so htonl 73 0.0051 bro Event::~Event() 73 0.0051 bro std::_Deque_iterator::_S_buffer_size() 73 0.0051 bro std::insert_iterator > >::insert_iterator(std::__cxx11::list >&, std::_List_iterator) 73 0.0051 bro std::queue > >::pop() 73 0.0051 bro std::unique_lock* std::__uninitialized_copy::__uninit_copy*>, std::unique_lock*>(std::move_iterator*>, std::move_iterator*>, std::unique_lock*) 73 0.0051 libc-2.27.so fwrite 72 0.0050 bro Event::Dispatch(bool) 72 0.0050 bro analyzer::http::HTTP_Message::IsOrig() 72 0.0050 bro analyzer::tcp::TCP_Endpoint::ValidChecksum(tcphdr const*, int) const 72 0.0050 bro file_analysis::Manager::GetFile(std::__cxx11::basic_string, std::allocator > const&, Connection*, analyzer::Tag, bool, bool, char const*) 72 0.0050 bro std::deque >::pop_front() 72 0.0050 bro std::move_iterator*>::base() const 72 0.0050 bro void std::vector, std::allocator > >::emplace_back >(std::unique_lock&&) 72 0.0050 kallsyms memset 72 0.0050 kallsyms smp_call_function_many 72 0.0050 ld-2.27.so _dl_fixup 72 0.0050 libc-2.27.so isupper 71 0.0049 bro Connection::SetLastTime(double) 71 0.0049 bro FuncType::Flavor() const 71 0.0049 bro addr_port_canon_lt(IPAddr const&, unsigned int, IPAddr const&, unsigned int) 71 0.0049 bro bool __gnu_cxx::__is_null_pointer(char*) 71 0.0049 bro std::_List_node::_M_valptr() 71 0.0049 bro threading::DebugMessage::~DebugMessage() 70 0.0049 bro Dictionary::InitForIteration() const 70 0.0049 bro HashKey::Size() const 70 0.0049 bro Val::AsTable() const 70 0.0049 bro std::vector, std::allocator > >::size() const 70 0.0049 kallsyms cpumask_next 69 0.0048 bro BroType::~BroType() 69 0.0048 bro analyzer::tcp::TCP_Endpoint::SeqWraps() const 69 0.0048 bro escapeReservedContent(ODesc*, std::__cxx11::basic_string, std::allocator > const&, char const*, int) 69 0.0048 bro std::insert_iterator > > std::__copy_move_a2, std::insert_iterator > > >(std::_List_iterator, std::_List_iterator, std::insert_iterator > >) 69 0.0048 kallsyms add_transaction_credits 68 0.0047 bro ConnID::~ConnID() 68 0.0047 bro PriorityQueue::Swap(int, int) 68 0.0047 bro RecordVal::~RecordVal() 68 0.0047 bro VectorVal::Assign(unsigned int, Val*, Opcode) 68 0.0047 bro analyzer::Analyzer::DeliverPacket(int, unsigned char const*, bool, unsigned long, IP_Hdr const*, int) 68 0.0047 bro analyzer::tcp::TCP_Analyzer::UpdateConnVal(RecordVal*) 68 0.0047 bro net_is_processing_suspended() 67 0.0047 bro BifFunc::bro_to_lower(Frame*, ValPList*) 67 0.0047 bro CompositeHash::SingleValHash(int, char*, BroType*, Val*, bool) const 67 0.0047 bro HashKey::Hash() const 67 0.0047 bro Val::InternalInt() const 67 0.0047 bro __gnu_cxx::__aligned_membuf::_M_ptr() 67 0.0047 bro long const& std::min(long const&, long const&) 67 0.0047 bro std::_List_iterator::_List_iterator(std::__detail::_List_node_base*) 67 0.0047 libc-2.27.so __libc_enable_asynccancel 66 0.0046 bro IP_Hdr::IsFragment() const 66 0.0046 bro __gnu_cxx::__aligned_membuf::_M_ptr() 66 0.0046 bro std::__cxx11::list >::_M_node_count() const 66 0.0046 bro std::deque >::begin() 66 0.0046 bro streq(char const*, char const*) 66 0.0046 bro void __gnu_cxx::new_allocator::construct(CallInfo*, CallInfo&&) 66 0.0046 kallsyms unlock_page 65 0.0045 bro analyzer::http::HTTP_Message::RequestBuffer(int*, char**) 65 0.0045 bro threading::BasicOutputMessage::~BasicOutputMessage() 64 0.0044 bro BoolExpr::DoSingleEval(Frame*, Val*, Expr*) const 64 0.0044 bro IPAddr::IPAddr() 64 0.0044 bro expire_timers(iosource::PktSrc*) 64 0.0044 bro net_update_time(double) 64 0.0044 bro std::vector >::_M_default_append(unsigned long) 64 0.0044 bro yyparse() 64 0.0044 kallsyms do_select 64 0.0044 kallsyms jbd2_journal_dirty_metadata 63 0.0044 bro Dictionary::Dictionary(dict_order, int) 63 0.0044 bro Dictionary::Insert(DictEntry*, int) 63 0.0044 bro bool std::operator==(std::_Deque_iterator const&, std::_Deque_iterator const&) 63 0.0044 bro file_analysis::File::DataIn(unsigned char const*, unsigned long) 63 0.0044 bro std::__cxx11::_List_base >::_M_init() 63 0.0044 bro void std::_Destroy_aux::__destroy*>(std::unique_lock*, std::unique_lock*) 63 0.0044 bro void std::vector >::emplace_back(CallInfo&&) 63 0.0044 kallsyms ext4_file_write_iter 62 0.0043 bro Dictionary::Length() const 62 0.0043 bro HashKey::CopyKey(void const*, int) const 62 0.0043 bro analyzer::Analyzer::GetAnalyzerTag() const 62 0.0043 bro analyzer::mime::MIME_Entity::FlushData() 62 0.0043 bro std::_List_iterator::operator*() const 62 0.0043 bro unsigned long const& std::max(unsigned long const&, unsigned long const&) 62 0.0043 kallsyms shmem_getpage_gfp 62 0.0043 libc-2.27.so _IO_file_underflow@@GLIBC_2.2.5 61 0.0042 bro CompositeHash::CompositeHash(TypeList*) 61 0.0042 bro Event::Event(EventHandlerPtr, ValPList*, unsigned long, unsigned int, TimerMgr*, BroObj*) 61 0.0042 bro FuncType::YieldType() 61 0.0042 bro IterCookie::IterCookie(int, int) 61 0.0042 bro do_split_string(StringVal*, RE_Matcher*, int, int) 61 0.0042 bro rules_parse() 61 0.0042 bro std::_List_iterator::operator++(int) 61 0.0042 bro std::vector >::end() const 61 0.0042 kallsyms __generic_file_write_iter 61 0.0042 kallsyms copy_page_to_iter 61 0.0042 kallsyms sys_read 60 0.0042 bro Bstr_cmp(BroString const*, BroString const*) 60 0.0042 bro analyzer::conn_size::ConnSize_Analyzer::UpdateConnVal(RecordVal*) 60 0.0042 bro file_analysis::boolPDict::Lookup(char const*) const 60 0.0042 bro std::__cxx11::list >::size() const From jsiwek at corelight.com Tue Jun 12 12:16:22 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Tue, 12 Jun 2018 14:16:22 -0500 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: <20180611233007.GC90763@icir.org> References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> <20180607172154.GM19548@icir.org> <20180608152347.GI75818@icir.org> <20180608174711.GK75818@icir.org> <20180611233007.GC90763@icir.org> Message-ID: On Mon, Jun 11, 2018 at 6:30 PM Robin Sommer wrote: > Confirmed that I'm still seeing that difference even when using the > options to turn the stores. Tried it on two different Fedora 28 > systems, with similar results. I had previously tried on a Fedora 28 VM w/ 4 CPUs and didn't reproduce it, but trying again on a system with 32 CPUs, I can reproduce your results. This has lead to the fix/workaround at [1], now in master, which should bring things back to baseline when not using data stores and more sane timings when using them. General changes/improvements in CAF itself may be warranted here because I can recreate the same performance overhead if I artificially remove all broker functionality from Bro, but then simply create and never use a caf::actor_system via the default configuration. - Jon [1] https://github.com/bro/bro/commit/c9fe9a943c4d78b18ffbae13c562b93349a5f951 From robin at icir.org Tue Jun 12 14:26:19 2018 From: robin at icir.org (Robin Sommer) Date: Tue, 12 Jun 2018 14:26:19 -0700 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> <20180607172154.GM19548@icir.org> <20180608152347.GI75818@icir.org> <20180608174711.GK75818@icir.org> <20180611233007.GC90763@icir.org> Message-ID: <20180612212619.GC22221@icir.org> On Tue, Jun 12, 2018 at 14:16 -0500, you wrote: > This has lead to the fix/workaround at [1], now in master, which Cool, that indeed solved it! It also helps significantly when data stores *are* being used; that still takes about 2x the time, but that's much less than before. Now I'm wondering if we can get that back down to normal, too ... One question about Broker's endpoint::advance_time(): that's locking on each call when in pcap mode, i.e., once per packet. Could we limit that to cases where something actually needs to be done? Quick idea: use an atomic<> for current_time plus another atomic<> counter tracking if there are any pending message. And go into the locked case only if the latter is non-zero? > General changes/improvements in CAF itself may be warranted here Yeah, sounds like it. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at corelight.com Wed Jun 13 12:50:08 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Wed, 13 Jun 2018 14:50:08 -0500 Subject: [Bro-Dev] Performance Issues after the fe7e1ee commit? In-Reply-To: <20180612212619.GC22221@icir.org> References: <0a56732d4f5541f09647afd0199a1293@msx.bala.susq.com> <20180607172154.GM19548@icir.org> <20180608152347.GI75818@icir.org> <20180608174711.GK75818@icir.org> <20180611233007.GC90763@icir.org> <20180612212619.GC22221@icir.org> Message-ID: On Tue, Jun 12, 2018 at 4:26 PM Robin Sommer wrote: > One question about Broker's endpoint::advance_time(): that's locking > on each call when in pcap mode, i.e., once per packet. Could we limit > that to cases where something actually needs to be done? Quick idea: > use an atomic<> for current_time plus another atomic<> counter > tracking if there are any pending message. And go into the locked case > only if the latter is non-zero? Yes, I've done that in master, but didn't measure a notable difference. > > General changes/improvements in CAF itself may be warranted here Made an issue for that here: https://github.com/actor-framework/actor-framework/issues/699 - Jon From dopheide at es.net Thu Jun 14 21:30:28 2018 From: dopheide at es.net (Michael Dopheide) Date: Thu, 14 Jun 2018 23:30:28 -0500 Subject: [Bro-Dev] DHCP event removal Message-ID: While testing the new Broker code in master I came across this a bit unexpectedly when trying to run our full production policy stack: 2.5-544 | 2018-05-01 17:57:15 -0500 * Rewrite the DHCP analyzer and accompanying script-layer API. I'm all for analyzer updates and improvements, but what I'm honestly not sure about is this: * Reduced all DHCP events into a single dhcp_message event. (removed legacy events since they weren't widely used anyway) How was the determination made that it's not widely used? I don't recall a survey on the bro/bro-dev lists and there's clearly instances of it's use when searching github. -Dop -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180614/4bd82b2b/attachment.html From vern at corelight.com Fri Jun 15 10:41:40 2018 From: vern at corelight.com (Vern Paxson) Date: Fri, 15 Jun 2018 10:41:40 -0700 Subject: [Bro-Dev] $history extensions - zero windows, logarithmic counts Message-ID: <201806151741.w5FHfeYk013203@fruitcake.ICSI.Berkeley.EDU> I'm working on two enhancements to the $history tracking for connections that thought I'd tee them up for comments. (1) A new history element, 'W'/'w', which means that a TCP receiver advertised a zero window, indicating that the corresponding process was unable to keep up with the incoming data. (This element is omitted in cases where zero windows aren't problematic: initial SYNs, and after FINs or RSTs.) (2) A notion of "logarithmic counts" for history events: for certain events ('C' = checksum, 'T' = retransmission, and 'W' = zero window) the count is repeated on the 10th/100th/1000th/etc. occurrence. So a history value of 'ttt' means that the responder sent somewhere between 100 and 999 retransmissions. This is useful because for large connections, a single checksum error, retransmission, or zero window is much less significant for analyzing performance issues than a whole bunch of these. Comments? Vern From michalpurzynski1 at gmail.com Fri Jun 15 11:15:59 2018 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Fri, 15 Jun 2018 11:15:59 -0700 Subject: [Bro-Dev] $history extensions - zero windows, logarithmic counts In-Reply-To: <201806151741.w5FHfeYk013203@fruitcake.ICSI.Berkeley.EDU> References: <201806151741.w5FHfeYk013203@fruitcake.ICSI.Berkeley.EDU> Message-ID: I really like those ideas, especially the logarithmic count. How much would it cost to have an event fired when those thresholds are crossed? > On Jun 15, 2018, at 10:41 AM, Vern Paxson wrote: > > I'm working on two enhancements to the $history tracking for connections > that thought I'd tee them up for comments. > > (1) A new history element, 'W'/'w', which means that a TCP receiver > advertised a zero window, indicating that the corresponding process > was unable to keep up with the incoming data. (This element is omitted > in cases where zero windows aren't problematic: initial SYNs, and after > FINs or RSTs.) > > (2) A notion of "logarithmic counts" for history events: for certain > events ('C' = checksum, 'T' = retransmission, and 'W' = zero window) > the count is repeated on the 10th/100th/1000th/etc. occurrence. So a > history value of 'ttt' means that the responder sent somewhere between > 100 and 999 retransmissions. This is useful because for large > connections, a single checksum error, retransmission, or zero window > is much less significant for analyzing performance issues than a whole > bunch of these. > > Comments? > > Vern > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From vern at corelight.com Fri Jun 15 11:24:08 2018 From: vern at corelight.com (Vern Paxson) Date: Fri, 15 Jun 2018 11:24:08 -0700 Subject: [Bro-Dev] $history extensions - zero windows, logarithmic counts In-Reply-To: (Fri, 15 Jun 2018 11:15:59 PDT). Message-ID: <201806151824.w5FIO8jn019104@fruitcake.ICSI.Berkeley.EDU> > I really like those ideas, especially the logarithmic count. Cool :-). > How much would it cost to have an event fired when those thresholds are crossed? Nice thought. I think it would be too expensive if done for the first instance, but for each of the backed-off instances it ought to be rare enough that it's not a problem. So maybe something like: ## Generated each time a reporting threshold (10, 100, 1000, ...) ## is crossed, starting with 10. event multiple_tcp_zero_windows(c: connection, is_orig: bool, threshold: count); event multiple_tcp_checksum_errors(c: connection, is_orig: bool, threshold: count); event multiple_tcp_retransmissions(c: connection, is_orig: bool, threshold: count); ? Vern From jazoff at illinois.edu Fri Jun 15 11:25:17 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 15 Jun 2018 18:25:17 +0000 Subject: [Bro-Dev] Broker has landed in master, please test In-Reply-To: <525697DD-8126-47CC-80C0-004845CBD8E2@illinois.edu> References: <20180522142623.GE34893@icir.org> <525697DD-8126-47CC-80C0-004845CBD8E2@illinois.edu> Message-ID: <48289E28-650E-4068-BB7E-83BC1B147645@illinois.edu> > On Jun 1, 2018, at 11:37 AM, Azoff, Justin S wrote: > > I could never figure out what was causing the problem, and it's possible that &synchronized not doing anything anymore is why it's better now. I'm mostly using &synchronized for syncing input files across all the workers and one of them does have 300k entries in it. That file is fairly constant though, only a few k changes every 5 minutes and nothing that should use 20G of ram. FWIW, I figured out what was causing this problem. While the file wasn't changing that much, I was using something like curl -o file.new $URL && mv file.new file.csv to download the file, and apparently unless you pass -f to curl, it doesn't actually exit with a non-zero status code on server errors. This was causing a server error page to be written to the csv file every now and then. When this happened: * the input reader would throw a warning that the file couldn't be parsed, and clear out the set * bro would then clear the set, triggering a removal of 300k items across all nodes (56 in the case of the test cluster) * 5 minutes later the next download would work * bro would then fill back in the set, and trigger 300k items to be synced to all 56 nodes again. so within 5 minutes, 300,000*56*2 updates would be kicked off, which is 33million updates. This seemed to max out the proxies for 30 minutes. The raw size of the data is only ~4M, or 261M total, which makes it a little crazy that memory usage would blow up by dozens of gigabytes of ram. &synchronized not having an effect in master made this problem go away, and adding a -f to curl on our pre-broker clusters fixed those too. All the more reason to port the method of distributing the data off of &synchronized. I think I will just run the curl command on the worker nodes too, effectively replacing &synchronized with curl. ? Justin Azoff From johanna at icir.org Fri Jun 15 11:42:54 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 15 Jun 2018 11:42:54 -0700 Subject: [Bro-Dev] $history extensions - zero windows, logarithmic counts In-Reply-To: <201806151741.w5FHfeYk013203@fruitcake.ICSI.Berkeley.EDU> References: <201806151741.w5FHfeYk013203@fruitcake.ICSI.Berkeley.EDU> Message-ID: <20180615184254.iv3tffapga2bi44v@user157.sys.ICSI.Berkeley.EDU> I think I like these, the only small concern I have is... > (2) A notion of "logarithmic counts" for history events: for certain > events ('C' = checksum, 'T' = retransmission, and 'W' = zero window) > the count is repeated on the 10th/100th/1000th/etc. occurrence. So a > history value of 'ttt' means that the responder sent somewhere between > 100 and 999 retransmissions. This is useful because for large > connections, a single checksum error, retransmission, or zero window > is much less significant for analyzing performance issues than a whole > bunch of these. Here we will not have cases where some repetitions are logarithmic, and some (like for R) are not. I guess that makes sense, but I can see it potentially being confusing. Johanna From vern at corelight.com Fri Jun 15 11:51:00 2018 From: vern at corelight.com (Vern Paxson) Date: Fri, 15 Jun 2018 11:51:00 -0700 Subject: [Bro-Dev] $history extensions - zero windows, logarithmic counts In-Reply-To: <20180615184254.iv3tffapga2bi44v@user157.sys.ICSI.Berkeley.EDU> (Fri, 15 Jun 2018 11:42:54 PDT). Message-ID: <201806151851.w5FIp0fh022106@fruitcake.ICSI.Berkeley.EDU> > Here we will not have cases where some repetitions are logarithmic, and > some (like for R) are not. I guess that makes sense, but I can see it > potentially being confusing. Yeah, I chewed on that too, but I don't see a better solution. The semantics of repeated R are different, too (per the recent $history thread, it entails differing sequence numbers), so I think once that's the case, then it's not all that much more confusing if the significance of a repetition has different semantics too. Vern From vlad at es.net Fri Jun 15 13:20:43 2018 From: vlad at es.net (Vlad Grigorescu) Date: Fri, 15 Jun 2018 20:20:43 +0000 Subject: [Bro-Dev] $history extensions - zero windows, logarithmic counts In-Reply-To: <201806151851.w5FIp0fh022106@fruitcake.ICSI.Berkeley.EDU> References: <20180615184254.iv3tffapga2bi44v@user157.sys.ICSI.Berkeley.EDU> <201806151851.w5FIp0fh022106@fruitcake.ICSI.Berkeley.EDU> Message-ID: I think this is a useful feature. I'm a bit unclear on the logarithmic counts. Take, for instance SaDtTtT. If I'm reading this correctly, I think that means 10-99 retransmissions from orig, followed by 10-99 from resp, then more retransmissions from orig (enough to reach a total of 100-999), and similarly more from resp. However, I could also interpret it as 10-99 from orig, 10-99 from resp, 10-99 from orig, 10-99 from resp. Another question I had was that most of these are TCP-specific. Would checksum apply to UDP as well? One downside of the logarithmic approach is that it makes it hard to search for, since searching for 't.*t' means one thing for small conns, and another for large conns. As you say, if what I care about is the overall number compared to the number of packets, that feels more like a percentage. To me, it'd seem more natural to use something like "0t" means "of the total number of packets from the originator, 0-9% were retransmissions," "1t" means 10-19%, etc. What I'm left debating is whether adding numerical data to history is the right approach, though. missed_bytes is a separate field, but it feels similar. If we did something like the log approach for that, we'd lose exact counts, but we'd have granularity on the direction. Maybe we add the new letters, but don't repeat them and also add new fields for exact bytecounts? --Vlad On Fri, Jun 15, 2018 at 6:51 PM, Vern Paxson wrote: > > Here we will not have cases where some repetitions are logarithmic, and > > some (like for R) are not. I guess that makes sense, but I can see it > > potentially being confusing. > > Yeah, I chewed on that too, but I don't see a better solution. The > semantics > of repeated R are different, too (per the recent $history thread, it > entails > differing sequence numbers), so I think once that's the case, then it's > not all that much more confusing if the significance of a repetition has > different semantics too. > > Vern > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180615/c35fbc94/attachment.html From seth at corelight.com Fri Jun 15 14:18:00 2018 From: seth at corelight.com (Seth Hall) Date: Fri, 15 Jun 2018 17:18:00 -0400 Subject: [Bro-Dev] DHCP event removal In-Reply-To: References: Message-ID: <48118758-BE99-43C1-A7EC-730B488DE1EE@corelight.com> On 15 Jun 2018, at 0:30, Michael Dopheide wrote: > How was the determination made that it's not widely used?? I don't > recall a survey on the bro/bro-dev lists and there's clearly instances > of it's use when searching github. I didn't dig around much which I made that change. Generally the DHCP analyzer hasn't been something that I've heard of much use (although Vlad did point out that I broke one of my own scripts!). How many scripts do you think it impacted? I suspect it's fairly few and we probably just need to put a change note in the release to say that developers will need to handle a new event to get the data. On the upside, you can handle both the old events and the new and they shouldn't impact each other (if you want to make a script work on multiple releases). .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From jazoff at illinois.edu Fri Jun 15 14:22:45 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 15 Jun 2018 21:22:45 +0000 Subject: [Bro-Dev] DHCP event removal In-Reply-To: <48118758-BE99-43C1-A7EC-730B488DE1EE@corelight.com> References: <48118758-BE99-43C1-A7EC-730B488DE1EE@corelight.com> Message-ID: <191AF065-64A3-41E5-9AC8-CC39714B450C@illinois.edu> > On Jun 15, 2018, at 5:18 PM, Seth Hall wrote: > > On the > upside, you can handle both the old events and the new and they > shouldn't impact each other (if you want to make a script work on > multiple releases). I ran into this on a script I got from somewhere, bash-cve-2014-6271.bro The fix is a little trickier, you can't handle both events because the DHCP::Msg type no longer exists and you need to wrap the old event with @ifdef (DHCP::Msg) @endif So for that script I ended up with @ifdef (DHCP::Msg) event dhcp_message(c: connection, is_orig: bool, msg: DHCP::Msg, options: DHCP::Options) { if ( options?$host_name && shellshock in options$host_name ) NOTICE([$note=Bash::DHCP_hostname_Attack, $conn=c, $msg=fmt("%s may have attempted to exploit CVE-2014-6271, bash environment variable attack, via dhcp hostname against %s submitting \"hostname\"=\"%s\"",c$id$orig_h, c$id$resp_h, options$host_name), $identifier=c$uid]); } @else event dhcp_offer(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) { if ( shellshock in host_name ) NOTICE([$note=Bash::DHCP_hostname_Attack, $conn=c, $msg=fmt("%s may have attempted to exploit CVE-2014-6271, bash environment variable attack, via dhcp hostname against %s submitting \"hostname\"=\"%s\"",c$id$orig_h, c$id$resp_h, host_name), $identifier=c$uid]); } @endif ? Justin Azoff From vlad at es.net Fri Jun 15 14:38:36 2018 From: vlad at es.net (Vlad Grigorescu) Date: Fri, 15 Jun 2018 21:38:36 +0000 Subject: [Bro-Dev] DHCP event removal In-Reply-To: <191AF065-64A3-41E5-9AC8-CC39714B450C@illinois.edu> References: <48118758-BE99-43C1-A7EC-730B488DE1EE@corelight.com> <191AF065-64A3-41E5-9AC8-CC39714B450C@illinois.edu> Message-ID: Yeah, I've mainly seen it used for shellshock. On top of that, I saw some scripts in GitHub that used it from: - Michal: https://github.com/michalpurzynski/bro-gramming/blob/master/dhcpr.bro - Matthias: https://github.com/bro/bro-scripts/blob/master/roam.bro - Grant Stavely: https://github.com/evernote/bro-scripts/blob/master/bolo/scripts/main.bro - Anthony: https://github.com/anthonykasza/users/blob/master/users.bro (There were a few others, like IVRE, but they've already updated). Even if it's not widely used, I think it'd be a nicer user experience if we were to ship a script that handled dhcp_message, and raised the old events. We could mark the old events as deprecated, and remove them in the next version. That way, people have at least one cycle to upgrade. Hopefully, as we see more published Bro packages, we have a better idea of which events are/aren't being used. --Vlad On Fri, Jun 15, 2018 at 9:22 PM, Azoff, Justin S wrote: > > On Jun 15, 2018, at 5:18 PM, Seth Hall wrote: > > > > On the > > upside, you can handle both the old events and the new and they > > shouldn't impact each other (if you want to make a script work on > > multiple releases). > > I ran into this on a script I got from somewhere, bash-cve-2014-6271.bro > > The fix is a little trickier, you can't handle both events because the > DHCP::Msg type no longer exists and you need to wrap the old event with > > @ifdef (DHCP::Msg) > @endif > > So for that script I ended up with > > @ifdef (DHCP::Msg) > event dhcp_message(c: connection, is_orig: bool, msg: DHCP::Msg, options: > DHCP::Options) > { > if ( options?$host_name && shellshock in options$host_name ) > NOTICE([$note=Bash::DHCP_hostname_Attack, > $conn=c, > $msg=fmt("%s may have attempted to exploit > CVE-2014-6271, bash environment variable attack, via dhcp hostname against > %s submitting \"hostname\"=\"%s\"",c$id$orig_h, c$id$resp_h, > options$host_name), > $identifier=c$uid]); > } > @else > event dhcp_offer(c: connection, msg: dhcp_msg, mask: addr, router: > dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) > { > if ( shellshock in host_name ) > NOTICE([$note=Bash::DHCP_hostname_Attack, > $conn=c, > $msg=fmt("%s may have attempted to exploit > CVE-2014-6271, bash environment variable attack, via dhcp hostname against > %s submitting \"hostname\"=\"%s\"",c$id$orig_h, c$id$resp_h, host_name), > $identifier=c$uid]); > } > @endif > > > > ? > Justin Azoff > > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180615/7e457cce/attachment-0001.html From jazoff at illinois.edu Fri Jun 15 14:40:11 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 15 Jun 2018 21:40:11 +0000 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Removed a few more discovered UTF-8 characters in Bro scripts. (cd18d9620) In-Reply-To: <6E58E81A-5261-40DB-9ED0-6FF92089C263@icir.org> References: <201806020858.w528wxib009476@bro-ids.icir.org> <6E58E81A-5261-40DB-9ED0-6FF92089C263@icir.org> Message-ID: <7FF5B84D-82BB-447A-9EA4-68D7F6DCF529@illinois.edu> > On Jun 2, 2018, at 12:49 PM, Johanna Amann wrote: > > Hum. Would it make sense to introduce a test that checks all > script-files for non-ascii-characters? > > I can so see that happening again... > > Johanna > I added this as one of the checks that packages.bro.org does, corelight/bro-hardware is the only one I found that has non-ascii in it.. but all in the auto generated data.bro. ? Justin Azoff From vern at corelight.com Fri Jun 15 14:54:31 2018 From: vern at corelight.com (Vern Paxson) Date: Fri, 15 Jun 2018 14:54:31 -0700 Subject: [Bro-Dev] $history extensions - zero windows, logarithmic counts In-Reply-To: (Fri, 15 Jun 2018 20:20:43 -0000). Message-ID: <201806152154.w5FLsVBH015894@fruitcake.ICSI.Berkeley.EDU> > it unclear on the logarithmic > counts. Take, for instance SaDtTtT. If I'm reading this correctly, I think > that means 10-99 retransmissions from orig, followed by 10-99 from resp, > then more retransmissions from orig (enough to reach a total of 100-999), > and similarly more from resp. Correct in principle. (1) These would be 1-9 followed by enough to get to 10-99, since a single retransmission is already a 't' / 'T', and (2) lower letters are responders rther than originators. > However, I could also interpret it as 10-99 > from orig, 10-99 from resp, 10-99 from orig, 10-99 from resp. Nope. The counter doesn't reset at any point, it's cumulative. > Another question I had was that most of these are TCP-specific. Would > checksum apply to UDP as well? Right, it would apply to UDP too, just like is done presently for the boolean indicator. > As you say, if what I care about is the overall > number compared to the number of packets, that feels more like a > percentage. Well, I think this is yes-and-no. For one, the overall percentage might be quite small and still have a large impact on what's supposed to be a high-speed transfer - particularly if it means that a connection entered an extented timeout-and-back-off - so I don't know if there would be a natural point of inspection for it. (It could also quite large but no big deal because the connection is a runt.) > To me, it'd seem more natural to use something like "0t" means > "of the total number of packets from the originator, 0-9% were > retransmissions," "1t" means 10-19%, etc. I'm inclined to wait on refinements like this. Let's first see whether having log-counter-style histories leaves people wanting more before qualitatively changing the nature of the history field, or adding new fields. > Maybe we add the > new letters, but don't repeat them and also add new fields for exact > bytecounts? I'm not following this. If we add new letters that don't repeat *and* we add new fields, why do we need the letters given that the fields are there? Vern From johanna at icir.org Fri Jun 15 15:13:01 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 15 Jun 2018 15:13:01 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Removed a few more discovered UTF-8 characters in Bro scripts. (cd18d9620) In-Reply-To: <7FF5B84D-82BB-447A-9EA4-68D7F6DCF529@illinois.edu> References: <201806020858.w528wxib009476@bro-ids.icir.org> <6E58E81A-5261-40DB-9ED0-6FF92089C263@icir.org> <7FF5B84D-82BB-447A-9EA4-68D7F6DCF529@illinois.edu> Message-ID: On 15 Jun 2018, at 14:40, Azoff, Justin S wrote: >> On Jun 2, 2018, at 12:49 PM, Johanna Amann wrote: >> >> Hum. Would it make sense to introduce a test that checks all >> script-files for non-ascii-characters? >> >> I can so see that happening again... >> >> Johanna >> > > > I added this as one of the checks that packages.bro.org does, > > corelight/bro-hardware is the only one I found that has non-ascii in > it.. but all in the auto generated data.bro. I actually am more concerned about the scripts that we ship with Bro itself :). Though it is nice to have for packages too... Johanna From vlad at es.net Fri Jun 15 15:16:21 2018 From: vlad at es.net (Vlad Grigorescu) Date: Fri, 15 Jun 2018 22:16:21 +0000 Subject: [Bro-Dev] $history extensions - zero windows, logarithmic counts In-Reply-To: <201806152154.w5FLsVBH015894@fruitcake.ICSI.Berkeley.EDU> References: <201806152154.w5FLsVBH015894@fruitcake.ICSI.Berkeley.EDU> Message-ID: On Fri, Jun 15, 2018 at 9:54 PM, Vern Paxson wrote: > > it unclear on the logarithmic > > counts. Take, for instance SaDtTtT. If I'm reading this correctly, I > think > > that means 10-99 retransmissions from orig, followed by 10-99 from resp, > > then more retransmissions from orig (enough to reach a total of 100-999), > > and similarly more from resp. > > Correct in principle. (1) These would be 1-9 followed by enough to > get to 10-99, since a single retransmission is already a 't' / 'T', and > (2) lower letters are responders rther than originators. > Ah, right. Thanks for clearing that up. > > Maybe we add the > > new letters, but don't repeat them and also add new fields for exact > > bytecounts? > > I'm not following this. If we add new letters that don't repeat *and* we > add new fields, why do we need the letters given that the fields are there? > My thought for this was simply if it mattered *where* in the state history the trouble occurred. For instance, if I'm seeing retransmissions at the very end of a connection, that might indicate that one side abruptly terminated the connection (I'd see this with things like fail2ban inserting an iptables rule to block a brute-forcer). Similarly, if I see a zero window at the start of a connection, that would tell me that the buffer was full due to another connection or connections, as opposed to filling up due the connection I'm looking at. I'm having a tough time thinking up additional use-cases without having some sample data, so perhaps the best course is to add what you proposed, and then revisit it if we feel like anything's missing. --Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180615/6cac959d/attachment.html From jazoff at illinois.edu Fri Jun 15 15:29:19 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 15 Jun 2018 22:29:19 +0000 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Removed a few more discovered UTF-8 characters in Bro scripts. (cd18d9620) In-Reply-To: References: <201806020858.w528wxib009476@bro-ids.icir.org> <6E58E81A-5261-40DB-9ED0-6FF92089C263@icir.org> <7FF5B84D-82BB-447A-9EA4-68D7F6DCF529@illinois.edu> Message-ID: > On Jun 15, 2018, at 6:13 PM, Johanna Amann wrote: > > I actually am more concerned about the scripts that we ship with Bro itself :). Though it is nice to have for packages too... > > Johanna Oh :-) I hadn't ran it on the scripts that come with bro.. it finds one non-ascii char: base/protocols/rdp/consts.bro:230 non ascii characters which is [1044] = "Norwegian (Bokm?l)", ? Justin Azoff From michalpurzynski1 at gmail.com Fri Jun 15 17:02:49 2018 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Fri, 15 Jun 2018 17:02:49 -0700 Subject: [Bro-Dev] DHCP event removal In-Reply-To: References: <48118758-BE99-43C1-A7EC-730B488DE1EE@corelight.com> <191AF065-64A3-41E5-9AC8-CC39714B450C@illinois.edu> Message-ID: Hey, I use the dhcp analyzer because i cannot count on our dhcp logs. Not just that, I do some detection around it. > On Jun 15, 2018, at 2:38 PM, Vlad Grigorescu wrote: > > Yeah, I've mainly seen it used for shellshock. On top of that, I saw some scripts in GitHub that used it from: > > - Michal: https://github.com/michalpurzynski/bro-gramming/blob/master/dhcpr.bro > - Matthias: https://github.com/bro/bro-scripts/blob/master/roam.bro > - Grant Stavely: https://github.com/evernote/bro-scripts/blob/master/bolo/scripts/main.bro > - Anthony: https://github.com/anthonykasza/users/blob/master/users.bro > > (There were a few others, like IVRE, but they've already updated). > > Even if it's not widely used, I think it'd be a nicer user experience if we were to ship a script that handled dhcp_message, and raised the old events. We could mark the old events as deprecated, and remove them in the next version. That way, people have at least one cycle to upgrade. > > Hopefully, as we see more published Bro packages, we have a better idea of which events are/aren't being used. > > --Vlad > >> On Fri, Jun 15, 2018 at 9:22 PM, Azoff, Justin S wrote: >> > On Jun 15, 2018, at 5:18 PM, Seth Hall wrote: >> > >> > On the >> > upside, you can handle both the old events and the new and they >> > shouldn't impact each other (if you want to make a script work on >> > multiple releases). >> >> I ran into this on a script I got from somewhere, bash-cve-2014-6271.bro >> >> The fix is a little trickier, you can't handle both events because the DHCP::Msg type no longer exists and you need to wrap the old event with >> >> @ifdef (DHCP::Msg) >> @endif >> >> So for that script I ended up with >> >> @ifdef (DHCP::Msg) >> event dhcp_message(c: connection, is_orig: bool, msg: DHCP::Msg, options: DHCP::Options) >> { >> if ( options?$host_name && shellshock in options$host_name ) >> NOTICE([$note=Bash::DHCP_hostname_Attack, >> $conn=c, >> $msg=fmt("%s may have attempted to exploit CVE-2014-6271, bash environment variable attack, via dhcp hostname against %s submitting \"hostname\"=\"%s\"",c$id$orig_h, c$id$resp_h, options$host_name), >> $identifier=c$uid]); >> } >> @else >> event dhcp_offer(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) >> { >> if ( shellshock in host_name ) >> NOTICE([$note=Bash::DHCP_hostname_Attack, >> $conn=c, >> $msg=fmt("%s may have attempted to exploit CVE-2014-6271, bash environment variable attack, via dhcp hostname against %s submitting \"hostname\"=\"%s\"",c$id$orig_h, c$id$resp_h, host_name), >> $identifier=c$uid]); >> } >> @endif >> >> >> >> ? >> Justin Azoff >> >> >> >> _______________________________________________ >> bro-dev mailing list >> bro-dev at bro.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180615/53737777/attachment-0001.html From vern at corelight.com Fri Jun 15 17:33:38 2018 From: vern at corelight.com (Vern Paxson) Date: Fri, 15 Jun 2018 17:33:38 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <20180430153420.GA70745@icir.org> (Mon, 30 Apr 2018 08:34:20 PDT). Message-ID: <201806160033.w5G0Xcwv029825@fruitcake.ICSI.Berkeley.EDU> 1.5 months ago we haggled over adding these and came close to converging, and then I dropped the ball on trying to ice the deal :-(. (Well, I also needed some time to lick my wounds, since I didn't get my way much on syntax preferences! :-P) Here's where I believe we wound up: (1) Add bitwise operators on "count" variables for &, | and ~. (2) Deprecate element-wise arithmetic operations on vectors, such as "v * 3" meaning "multiply each element of v by 3". Perhaps down the road we'll introduce syntax that flags things like "for this expression, vectorize it". (3) Implement "v += e" to mean "append the element e to the vector v". (4) Wait on whether "v + e" should mean "return a vector that is v with the element e appended". (And indeed we can't do this right now if we're #2.) (5) Keep "add" and "delete" for manipulating sets in terms of individual elements. (6) Add "s1 & s2", "s1 | s2", and "s1 - s2" as intersection, union, and set difference. I'm not clear whether we reached agreement on: (7?) Add "s1 &= s2" etc. to mean "s1 = s1 & s2". The advantage of having this as an operator is it might more easily enable efficient implementation of some set operations for big sets. I suppose if we have it then we'd be expected to also have: (7') "c1 &= c2" etc., i.e., bitwise assignments for "count" variables. Please chime in if you remember where we wound up differently, and also whether you disagree with #7. Again, my apologies for the lengthy lull in attending to this thread. Vern From seth at corelight.com Fri Jun 15 23:24:34 2018 From: seth at corelight.com (Seth Hall) Date: Sat, 16 Jun 2018 02:24:34 -0400 Subject: [Bro-Dev] DHCP event removal In-Reply-To: References: <48118758-BE99-43C1-A7EC-730B488DE1EE@corelight.com> <191AF065-64A3-41E5-9AC8-CC39714B450C@illinois.edu> Message-ID: <1A828126-59D0-47AB-A9E7-E821D6257B37@corelight.com> On 15 Jun 2018, at 20:02, Micha? Purzy?ski wrote: > Hey, I use the dhcp analyzer because i cannot count on our dhcp logs. > Not just that, I do some detection around it. How much trouble is it to migrate your scripts to what's in Bro master? .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From seth at corelight.com Fri Jun 15 23:26:44 2018 From: seth at corelight.com (Seth Hall) Date: Sat, 16 Jun 2018 02:26:44 -0400 Subject: [Bro-Dev] DHCP event removal In-Reply-To: <191AF065-64A3-41E5-9AC8-CC39714B450C@illinois.edu> References: <48118758-BE99-43C1-A7EC-730B488DE1EE@corelight.com> <191AF065-64A3-41E5-9AC8-CC39714B450C@illinois.edu> Message-ID: <801C8BCA-F86A-4EA2-98A1-E30B3C91F9C1@corelight.com> On 15 Jun 2018, at 17:22, Azoff, Justin S wrote: > The fix is a little trickier, you can't handle both events because the > DHCP::Msg type no longer exists and you need to wrap the old event > with > > @ifdef (DHCP::Msg) > @endif Ugh, I didn't consider that. Bro has re-reached the point where touching any number of things can set off an avalanche of problems like this. Anyone on this thread up for submitting a patch which makes Bro cope with the changes automatically? You can then even mark the old events as deprecated. :) .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From vlad at es.net Sat Jun 16 06:07:42 2018 From: vlad at es.net (Vlad Grigorescu) Date: Sat, 16 Jun 2018 13:07:42 +0000 Subject: [Bro-Dev] DHCP event removal In-Reply-To: <801C8BCA-F86A-4EA2-98A1-E30B3C91F9C1@corelight.com> References: <48118758-BE99-43C1-A7EC-730B488DE1EE@corelight.com> <191AF065-64A3-41E5-9AC8-CC39714B450C@illinois.edu> <801C8BCA-F86A-4EA2-98A1-E30B3C91F9C1@corelight.com> Message-ID: Yep, already working on it. :-) On Sat, Jun 16, 2018 at 6:26 AM, Seth Hall wrote: > > On 15 Jun 2018, at 17:22, Azoff, Justin S wrote: > > > The fix is a little trickier, you can't handle both events because the > > DHCP::Msg type no longer exists and you need to wrap the old event > > with > > > > @ifdef (DHCP::Msg) > > @endif > > Ugh, I didn't consider that. Bro has re-reached the point where > touching any number of things can set off an avalanche of problems like > this. > > Anyone on this thread up for submitting a patch which makes Bro cope > with the changes automatically? You can then even mark the old events > as deprecated. :) > > .Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180616/26821022/attachment.html From michalpurzynski1 at gmail.com Sun Jun 17 12:48:46 2018 From: michalpurzynski1 at gmail.com (=?utf-8?Q?Micha=C5=82_Purzy=C5=84ski?=) Date: Sun, 17 Jun 2018 12:48:46 -0700 Subject: [Bro-Dev] DHCP event removal In-Reply-To: References: <48118758-BE99-43C1-A7EC-730B488DE1EE@corelight.com> <191AF065-64A3-41E5-9AC8-CC39714B450C@illinois.edu> <801C8BCA-F86A-4EA2-98A1-E30B3C91F9C1@corelight.com> Message-ID: No idea what?s in master. It should be trivial if you don?t remove the dhcp analyzer completely. > On Jun 16, 2018, at 6:07 AM, Vlad Grigorescu wrote: > > Yep, already working on it. :-) > >> On Sat, Jun 16, 2018 at 6:26 AM, Seth Hall wrote: >> >> On 15 Jun 2018, at 17:22, Azoff, Justin S wrote: >> >> > The fix is a little trickier, you can't handle both events because the >> > DHCP::Msg type no longer exists and you need to wrap the old event >> > with >> > >> > @ifdef (DHCP::Msg) >> > @endif >> >> Ugh, I didn't consider that. Bro has re-reached the point where >> touching any number of things can set off an avalanche of problems like >> this. >> >> Anyone on this thread up for submitting a patch which makes Bro cope >> with the changes automatically? You can then even mark the old events >> as deprecated. :) >> >> .Seth >> >> -- >> Seth Hall * Corelight, Inc * www.corelight.com >> _______________________________________________ >> bro-dev mailing list >> bro-dev at bro.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180617/ee88e7e1/attachment.html From seth at corelight.com Mon Jun 18 06:38:47 2018 From: seth at corelight.com (Seth Hall) Date: Mon, 18 Jun 2018 09:38:47 -0400 Subject: [Bro-Dev] DHCP event removal In-Reply-To: References: <48118758-BE99-43C1-A7EC-730B488DE1EE@corelight.com> <191AF065-64A3-41E5-9AC8-CC39714B450C@illinois.edu> <801C8BCA-F86A-4EA2-98A1-E30B3C91F9C1@corelight.com> Message-ID: An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180618/c622652b/attachment.html From commike at reservoir.com Mon Jun 18 12:09:41 2018 From: commike at reservoir.com (Alan Commike) Date: Mon, 18 Jun 2018 21:09:41 +0200 Subject: [Bro-Dev] DHCP event removal In-Reply-To: <801C8BCA-F86A-4EA2-98A1-E30B3C91F9C1@corelight.com> References: <48118758-BE99-43C1-A7EC-730B488DE1EE@corelight.com> <191AF065-64A3-41E5-9AC8-CC39714B450C@illinois.edu> <801C8BCA-F86A-4EA2-98A1-E30B3C91F9C1@corelight.com> Message-ID: On Fri, Jun 15, 2018 at 11:38 PM Seth Hall wrote: > > Ugh, I didn't consider that. Bro has re-reached the point where > touching any number of things can set off an avalanche of problems like > this. > The other aspect of Bro changes is in the logging format. This comment isn't specifically tied the DHCP change, but since we're talking about breaking things.... With the default TSV, any change can break export into the various back-end log stores and SIEMs. When adding new fields, it would be nice to see them added to the end of the Info structure. ...alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20180618/42550a5e/attachment.html From vern at corelight.com Mon Jun 18 18:18:58 2018 From: vern at corelight.com (Vern Paxson) Date: Mon, 18 Jun 2018 18:18:58 -0700 Subject: [Bro-Dev] $history extensions - zero windows, logarithmic counts In-Reply-To: (Fri, 15 Jun 2018 22:16:21 -0000). Message-ID: <201806190118.w5J1Iwsj013077@fruitcake.ICSI.Berkeley.EDU> > My thought for this was simply if it mattered *where* in the state history > the trouble occurred. I agree that it could ... but I think for at least some situations where it does, for the logs to help in diagnosing them will require something well beyond indicator flags. It's interesting to consider what these might look like, but for now I'd like to get this simpler additional functionality implemented, as I think it'll already be handy - not pointwise for diagnosing specific connections, but as manifest more in aggregate, such as "gee when we talk with a.b.0.0/16 we sure to rack up the checksum errors" or such. > I'm having a tough time thinking up additional use-cases without having > some sample data, so perhaps the best course is to add what you proposed, > and then revisit it if we feel like anything's missing. Sounds good. I'll aim to have a branch that people can try out ready in a bit. Vern From seth at corelight.com Tue Jun 19 10:55:47 2018 From: seth at corelight.com (Seth Hall) Date: Tue, 19 Jun 2018 13:55:47 -0400 Subject: [Bro-Dev] DHCP event removal In-Reply-To: References: <48118758-BE99-43C1-A7EC-730B488DE1EE@corelight.com> <191AF065-64A3-41E5-9AC8-CC39714B450C@illinois.edu> <801C8BCA-F86A-4EA2-98A1-E30B3C91F9C1@corelight.com> Message-ID: On 18 Jun 2018, at 15:09, Alan Commike wrote: > With the default TSV, any change can break export into the various > back-end log stores and SIEMs. When adding new fields, it would be > nice to see them added to the end of the Info structure.? This was a complete rework on the logs and scripts so the structure is completely different. Unfortunately it wasn't just one of the cases where a field or two was added. I don't think that assuming the order of fields is ever a safe assumption. It's why we shipped a version of bro-cut with Bro 2.0. We wanted to encourage people to refer to fields by the field name rather than the ordinal position of the field. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From vern at corelight.com Tue Jun 19 11:21:10 2018 From: vern at corelight.com (Vern Paxson) Date: Tue, 19 Jun 2018 11:21:10 -0700 Subject: [Bro-Dev] patterns and &&/|| vs. &/| operators Message-ID: <201806191821.w5JILAxt014214@fruitcake.ICSI.Berkeley.EDU> In working on adding bitwise &/| operators for counts, I've come across apparently undocumented && and || operators for patterns: p1 && p2 yields a pattern that matches a p1 followed by a p2 p1 || p2 yields a pattern that matches either p1 or p2 Confusingly, Bro also supports "p1 | p2", which means the same as "p1 || p2" above, but *only* if p1 and p2 are literal patterns, not if they are variables of type "pattern". (This functionality is in common use.) It doesn't support "p1 & p2" in any form. I searched a large corpus of scripts and didn't find any instances of "p1 && p2" or "p1 || p2" for literal patterns, so I suspect the current feature is basically unused. Proposal: as part of adding bitwise &/| operators for counts, I'll also implement &/| operators for patterns, and remove the current &&/|| functionality. This seems pretty straightforward to me - but I've mistakenly thought that about other things before! :-P So if anyone has comments, plz speak up ... Vern % bro -e 'print (/foo/ && /bar/) in "xfoobary"' T From johanna at icir.org Wed Jun 20 11:22:51 2018 From: johanna at icir.org (Johanna Amann) Date: Wed, 20 Jun 2018 11:22:51 -0700 Subject: [Bro-Dev] patterns and &&/|| vs. &/| operators In-Reply-To: <201806191821.w5JILAxt014214@fruitcake.ICSI.Berkeley.EDU> References: <201806191821.w5JILAxt014214@fruitcake.ICSI.Berkeley.EDU> Message-ID: <20180620182251.qrestag7krnbltwr@Beezling.local> On Tue, Jun 19, 2018 at 11:21:10AM -0700, Vern Paxson wrote: > In working on adding bitwise &/| operators for counts, I've come across > apparently undocumented && and || operators for patterns: > > p1 && p2 yields a pattern that matches a p1 followed by a p2 > p1 || p2 yields a pattern that matches either p1 or p2 > > Confusingly, Bro also supports "p1 | p2", which means the same as "p1 || p2" > above, but *only* if p1 and p2 are literal patterns, not if they are > variables of type "pattern". (This functionality is in common use.) > It doesn't support "p1 & p2" in any form. > > I searched a large corpus of scripts and didn't find any instances of > "p1 && p2" or "p1 || p2" for literal patterns, so I suspect the current > feature is basically unused. > > Proposal: as part of adding bitwise &/| operators for counts, I'll > also implement &/| operators for patterns, and remove the current > &&/|| functionality. > > This seems pretty straightforward to me - but I've mistakenly thought > that about other things before! :-P So if anyone has comments, plz > speak up ... Sounds good to me... Johanna From johanna at icir.org Wed Jun 20 11:34:39 2018 From: johanna at icir.org (Johanna Amann) Date: Wed, 20 Jun 2018 11:34:39 -0700 Subject: [Bro-Dev] set and vector operators In-Reply-To: <201806160033.w5G0Xcwv029825@fruitcake.ICSI.Berkeley.EDU> References: <20180430153420.GA70745@icir.org> <201806160033.w5G0Xcwv029825@fruitcake.ICSI.Berkeley.EDU> Message-ID: <20180620183439.c7xjps7rnxb36m2s@Beezling.local> > (3) Implement "v += e" to mean "append the element e to the vector v". Do we want to do this now, or should we potentially wait a release-cycle with it (to prevent the situation where v + e and v+= e means something different). Looking at the emails I am generally not 100% sure if we reached consensus on this one. > (4) Wait on whether "v + e" should mean "return a vector that is v with > the element e appended". (And indeed we can't do this right now if > we're #2.) Yup. > I'm not clear whether we reached agreement on: > > (7?) Add "s1 &= s2" etc. to mean "s1 = s1 & s2". The advantage of > having this as an operator is it might more easily enable efficient > implementation of some set operations for big sets. I suppose > if we have it then we'd be expected to also have: > (7') "c1 &= c2" etc., i.e., bitwise assignments for "count" > variables. They were contained in the "minimal" list that Jon came up with. I think I would be ok with them, but they did not really get any discussion afterwards that I can see. Johanna From jsiwek at corelight.com Thu Jun 21 11:32:52 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 21 Jun 2018 13:32:52 -0500 Subject: [Bro-Dev] patterns and &&/|| vs. &/| operators In-Reply-To: <201806191821.w5JILAxt014214@fruitcake.ICSI.Berkeley.EDU> References: <201806191821.w5JILAxt014214@fruitcake.ICSI.Berkeley.EDU> Message-ID: On Tue, Jun 19, 2018 at 1:21 PM Vern Paxson wrote: > Proposal: as part of adding bitwise &/| operators for counts, I'll > also implement &/| operators for patterns, and remove the current > &&/|| functionality. The meaning of "p1 & p2" would be "yields a pattern that matches both p1 and p2" versus the meaning of "p1 && p2" currently being "yields a pattern that matches a p1 followed by a p2" ? I'd generally say that deprecating (emit a warning message pointing to each usage) for a time period is a more cautious approach. - Jon From vern at corelight.com Thu Jun 21 11:34:45 2018 From: vern at corelight.com (Vern Paxson) Date: Thu, 21 Jun 2018 11:34:45 -0700 Subject: [Bro-Dev] patterns and &&/|| vs. &/| operators In-Reply-To: (Thu, 21 Jun 2018 13:32:52 CDT). Message-ID: <201806211834.w5LIYjoI018873@fruitcake.ICSI.Berkeley.EDU> > The meaning of "p1 & p2" would be "yields a pattern that matches both > p1 and p2" versus the meaning of "p1 && p2" currently being "yields a > pattern that matches a p1 followed by a p2" ? No, p1 & p2 would be the new way to express p1 && p2. > I'd generally say that deprecating (emit a warning message pointing to > each usage) for a time period is a more cautious approach. Easy 'nuf, though I'd be amazed if anyone is using p1 && p2 given it's not documented and not intuitive! Vern From jsiwek at corelight.com Thu Jun 21 12:12:28 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 21 Jun 2018 14:12:28 -0500 Subject: [Bro-Dev] set and vector operators In-Reply-To: <201806160033.w5G0Xcwv029825@fruitcake.ICSI.Berkeley.EDU> References: <20180430153420.GA70745@icir.org> <201806160033.w5G0Xcwv029825@fruitcake.ICSI.Berkeley.EDU> Message-ID: On Fri, Jun 15, 2018 at 7:40 PM Vern Paxson wrote: > (1) Add bitwise operators on "count" variables for &, | and ~. Yeah, looks like everyone was in favor of those. > (2) Deprecate element-wise arithmetic operations on vectors, such > as "v * 3" meaning "multiply each element of v by 3". Perhaps down > the road we'll introduce syntax that flags things like "for this > expression, vectorize it". > > (3) Implement "v += e" to mean "append the element e to the vector v". > > (4) Wait on whether "v + e" should mean "return a vector that is v with > the element e appended". (And indeed we can't do this right now if > we're #2.) This sounds like what everyone was thinking. My suggestion for the timing of these would be to implement the deprecation (2) by itself ASAP, at least before the 2.6 release, and shortly after 2.6 it can be removed in git/master and both (3) and (4) done. Hopefully 2.6 is not terribly far away. > (5) Keep "add" and "delete" for manipulating sets in terms of individual > elements. > > (6) Add "s1 & s2", "s1 | s2", and "s1 - s2" as intersection, union, > and set difference. Yeah, sounds right. > I'm not clear whether we reached agreement on: > > (7?) Add "s1 &= s2" etc. to mean "s1 = s1 & s2". The advantage of > having this as an operator is it might more easily enable efficient > implementation of some set operations for big sets. I suppose > if we have it then we'd be expected to also have: > (7') "c1 &= c2" etc., i.e., bitwise assignments for "count" > variables. > > Please chime in if you remember where we wound up differently, and also > whether you disagree with #7. I don't see much previous discussion around these, though they all make sense to me. - Jon From jsiwek at corelight.com Thu Jun 21 12:33:39 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 21 Jun 2018 14:33:39 -0500 Subject: [Bro-Dev] patterns and &&/|| vs. &/| operators In-Reply-To: <201806211834.w5LIYjoI018873@fruitcake.ICSI.Berkeley.EDU> References: <201806211834.w5LIYjoI018873@fruitcake.ICSI.Berkeley.EDU> Message-ID: On Thu, Jun 21, 2018 at 1:34 PM Vern Paxson wrote: > > > The meaning of "p1 & p2" would be "yields a pattern that matches both > > p1 and p2" versus the meaning of "p1 && p2" currently being "yields a > > pattern that matches a p1 followed by a p2" ? > > No, p1 & p2 would be the new way to express p1 && p2. > > > I'd generally say that deprecating (emit a warning message pointing to > > each usage) for a time period is a more cautious approach. > > Easy 'nuf, though I'd be amazed if anyone is using p1 && p2 given it's > not documented and not intuitive! Ok, I see now, yeah & would be better than && for that semantic, though maybe p1 + p2 would be even better at expressing that concatenation is happening? I also notice from [1]: ?r/s?: an ?r? but only if it is followed by an ?s? ... Maybe another option? Just making suggestions since I didn't quite get what p1 & p2 would do at first. - Jon [1] http://westes.github.io/flex/manual/Patterns.html From vern at corelight.com Thu Jun 21 14:12:08 2018 From: vern at corelight.com (Vern Paxson) Date: Thu, 21 Jun 2018 14:12:08 -0700 Subject: [Bro-Dev] v += e (Re: set and vector operators) In-Reply-To: <20180620183439.c7xjps7rnxb36m2s@Beezling.local> (Wed, 20 Jun 2018 11:34:39 PDT). Message-ID: <201806212112.w5LLC8ZG005686@fruitcake.ICSI.Berkeley.EDU> > > (3) Implement "v += e" to mean "append the element e to the vector v". > > Do we want to do this now, or should we potentially wait a release-cycle > with it (to prevent the situation where v + e and v+= e means something > different). Turns out that "v += e" currently generates an error ("requires two arithmetic or two string operands"), so it seems safe to me to introduce it as append-to-vector concurrent with deprecating "v + e" (which from the discussion on the list it seems likely current has little or no use). BTW, "v1 += v2" likewise generates an error, even when "v1 + v2" works. Vern From vern at corelight.com Thu Jun 21 14:25:48 2018 From: vern at corelight.com (Vern Paxson) Date: Thu, 21 Jun 2018 14:25:48 -0700 Subject: [Bro-Dev] patterns and &&/|| vs. &/| operators In-Reply-To: (Thu, 21 Jun 2018 14:33:39 CDT). Message-ID: <201806212125.w5LLPmDT008224@fruitcake.ICSI.Berkeley.EDU> > though maybe p1 + p2 would be even better at expressing that > concatenation is happening? I think this is somewhat problematic, since '+' already has a regular-expression meaning which is different. In addition, '&' is a more natural dual to '|' than '+' is. Indeed, in some contexts '|' and '+' are synonyms (e.g., I originally wanted them both for set union). > I also notice from [1]: > > `r/s': an `r' but only if it is followed by an `s' ... > > Maybe another option? Note that Bro's REs don't support that ... and in general that operator is a PITA to support correctly+efficiently. It would also step on the current syntax of '/'s being used to express /re/ 's. > Just making suggestions since I didn't quite get what p1 & p2 would do at first. Interestingly, I discovered that we have a BIF merge_pattern(p1, p2) which does the same thing as "p1 & p2" (in the new syntax). As best as I can tell it's not used anywhere - plus it's funky (only allows itself to be called if Bro isn't processing traffic yet). Perhaps we can deprecate it, too? Vern From jsiwek at corelight.com Thu Jun 21 14:58:42 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 21 Jun 2018 16:58:42 -0500 Subject: [Bro-Dev] v += e (Re: set and vector operators) In-Reply-To: <201806212112.w5LLC8ZG005686@fruitcake.ICSI.Berkeley.EDU> References: <20180620183439.c7xjps7rnxb36m2s@Beezling.local> <201806212112.w5LLC8ZG005686@fruitcake.ICSI.Berkeley.EDU> Message-ID: On Thu, Jun 21, 2018 at 4:23 PM Vern Paxson wrote: > > > > (3) Implement "v += e" to mean "append the element e to the vector v". > > > > Do we want to do this now, or should we potentially wait a release-cycle > > with it (to prevent the situation where v + e and v+= e means something > > different). > > Turns out that "v += e" currently generates an error ("requires two > arithmetic or two string operands"), so it seems safe to me to introduce > it as append-to-vector concurrent with deprecating "v + e" (which from the > discussion on the list it seems likely current has little or no use). Yeah, seems fine for v += e and v + e to temporarily have different meanings given that the later will, for some time, have a clear deprecation/warning message for each use (in any rare cases where it might actually be used). So my current POV is everything you outlined looks good to do whenever. - Jon From jsiwek at corelight.com Thu Jun 21 15:19:38 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Thu, 21 Jun 2018 17:19:38 -0500 Subject: [Bro-Dev] patterns and &&/|| vs. &/| operators In-Reply-To: <201806212125.w5LLPmDT008224@fruitcake.ICSI.Berkeley.EDU> References: <201806212125.w5LLPmDT008224@fruitcake.ICSI.Berkeley.EDU> Message-ID: On Thu, Jun 21, 2018 at 4:25 PM Vern Paxson wrote: > > > though maybe p1 + p2 would be even better at expressing that > > concatenation is happening? > > I think this is somewhat problematic, since '+' already has a > regular-expression meaning which is different. In addition, '&' is > a more natural dual to '|' than '+' is. Yeah, agree w/ that. > Interestingly, I discovered that we have a BIF merge_pattern(p1, p2) which > does the same thing as "p1 & p2" (in the new syntax). As best as I can > tell it's not used anywhere - plus it's funky (only allows itself to be > called if Bro isn't processing traffic yet). Perhaps we can deprecate it, too? If there actually is no (longer) problems with concatenating patterns at run-time, I'd agree to deprecate. I'm imagine it existed because there was such a problem with dynamically creating patterns at run-time, but don't know/remember what it was. - Jon From johanna at icir.org Thu Jun 21 15:35:33 2018 From: johanna at icir.org (Johanna Amann) Date: Thu, 21 Jun 2018 15:35:33 -0700 Subject: [Bro-Dev] patterns and &&/|| vs. &/| operators In-Reply-To: References: <201806212125.w5LLPmDT008224@fruitcake.ICSI.Berkeley.EDU> Message-ID: <20180621223533.yrbfwrn65bbq2w27@user157.sys.ICSI.Berkeley.EDU> > If there actually is no (longer) problems with concatenating patterns > at run-time, I'd agree to deprecate. > > I'm imagine it existed because there was such a problem with > dynamically creating patterns at run-time, but don't know/remember > what it was. > Now that you mention it - yes, there still is a problem as far as I know. https://bro-tracker.atlassian.net/browse/BIT-328 seems to be the relevant ticket. This probably means that we will either have to limit p1 & p2 to only be allowed when Bro is not processing traffic yet too - or fix the cleanup of the DFA data structures. Johanna From vern at corelight.com Sun Jun 24 19:07:06 2018 From: vern at corelight.com (Vern Paxson) Date: Sun, 24 Jun 2018 19:07:06 -0700 Subject: [Bro-Dev] more set operators? (equality/subset) Message-ID: <201806250207.w5P276gK023110@fruitcake.ICSI.Berkeley.EDU> I've implemented the set operators we discussed, and am now developing test cases. In the process, I'm finding that it would be handy to have set equality and subset/superset testing. These would be (for type equivalence sets): s1 == s2 iff both sets have exactly the same members s1 < s2 iff every element in s1 is in s2, but s2 has some elements not in s1 analogous <=, >, >=, != operators These can already be implemented in terms of existing operators (if I haven't screwed something up): s1 == s2 <=> |s1 & s2| == |s1 | s2| s1 < s2 <=> |s1 & s2| == |s1| && |s1| < |s2| Any concerns with adding these too? Vern From johanna at icir.org Fri Jun 29 08:52:01 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 29 Jun 2018 08:52:01 -0700 Subject: [Bro-Dev] more set operators? (equality/subset) In-Reply-To: <201806250207.w5P276gK023110@fruitcake.ICSI.Berkeley.EDU> References: <201806250207.w5P276gK023110@fruitcake.ICSI.Berkeley.EDU> Message-ID: <20180629155201.sf3u7zipzxaa4oq3@Trafalgar.local> On Sun, Jun 24, 2018 at 07:07:06PM -0700, Vern Paxson wrote: > s1 == s2 iff both sets have exactly the same members > > s1 < s2 iff every element in s1 is in s2, but s2 has some > elements not in s1 [...] > Any concerns with adding these too? I actually have a small question when thinking about these - which I should already have raised about the intersect operators. What happens when sets contain records or other complex types in these cases? >From what I can tell, Bro so far refuses to compare records - the reason being that (I think) we do not have properly implemented comparison operators internally. For example, the following script: type A: record { a: string; }; event bro_init() { local i = A($a="a"); local j = A($a="a"); print i == j; } outputs: $ bro test.bro error in ./test.bro, line 9: illegal comparison (i == j) I assume what will at the moment happen with sets is that the pointers of records are checked for equality - not the content. Which might arguably be a bit non-intuitive. As I said this is more of a concern about the already added operators - in principle I don't have a problem with ==, but I think it should work for other complex datatypes too. Johanna From vern at corelight.com Fri Jun 29 08:58:25 2018 From: vern at corelight.com (Vern Paxson) Date: Fri, 29 Jun 2018 08:58:25 -0700 Subject: [Bro-Dev] more set operators? (equality/subset) In-Reply-To: <20180629155201.sf3u7zipzxaa4oq3@Trafalgar.local> (Fri, 29 Jun 2018 08:52:01 PDT). Message-ID: <201806291558.w5TFwPQY013337@fruitcake.ICSI.Berkeley.EDU> > I assume what will at the moment happen with sets is that the pointers of > records are checked for equality Specifically, in my branch they are checked for whether the composite hash index matches. Happily, this works: type A: record { a: string; }; event bro_init() { local i = A($a="a"); local j = A($a="a"); print set(i) | set(j); } when run prints { [a=a] } and if you change j to be $a="b" then you get: { [a=b], [a=a] } This in fact suggests we could implement record equality by converting the records to hash indices and then comparing those. Vern From johanna at icir.org Fri Jun 29 09:08:58 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 29 Jun 2018 09:08:58 -0700 Subject: [Bro-Dev] more set operators? (equality/subset) In-Reply-To: <201806291558.w5TFwPQY013337@fruitcake.ICSI.Berkeley.EDU> References: <20180629155201.sf3u7zipzxaa4oq3@Trafalgar.local> <201806291558.w5TFwPQY013337@fruitcake.ICSI.Berkeley.EDU> Message-ID: <20180629160858.cju2kq22aess2a75@Trafalgar.local> On Fri, Jun 29, 2018 at 08:58:25AM -0700, Vern Paxson wrote: > This in fact suggests we could implement record equality by converting the > records to hash indices and then comparing those. Oh, neat. If that actually works in all cases (so also with records of records, etc) I would be totally on board with this. Hash equality was something that I missed a few times :). Johanna From vern at corelight.com Fri Jun 29 09:55:19 2018 From: vern at corelight.com (Vern Paxson) Date: Fri, 29 Jun 2018 09:55:19 -0700 Subject: [Bro-Dev] more set operators? (equality/subset) In-Reply-To: <20180629160858.cju2kq22aess2a75@Trafalgar.local> (Fri, 29 Jun 2018 09:08:58 PDT). Message-ID: <201806291655.w5TGtJ1x021130@fruitcake.ICSI.Berkeley.EDU> > Oh, neat. If that actually works in all cases (so also with records of > records, etc) Well, it almost does. I tried it with records that contain records and that's fine. For records that contain sets, it often works in my testing, but not always, evidently due to the randomized hash keying, since I can make it go away by always loading the same seeds. The same problem occurs with set deletion: deleting from a set of records-containing-sets sometimes fails to delete an element that's indeed in the set. (Hmmm and we also don't support sets of sets, which seems like a natural.) I think the right answer for this is to have some sort of canonical ordering for hash keys. Seems like a pain given the need to also randomize hash keys. I'll file a ticket, but won't aim to fix it this go-around. Vern From vern at corelight.com Fri Jun 29 12:00:30 2018 From: vern at corelight.com (Vern Paxson) Date: Fri, 29 Jun 2018 12:00:30 -0700 Subject: [Bro-Dev] case-insensitive patterns Message-ID: <201806291900.w5TJ0VZD018815@fruitcake.ICSI.Berkeley.EDU> Once I wound up monkeying around with the internals of the pattern-matching code (to fix leaks, because Johanna [correctly] pushed back on adding the &/| operators for general use if they leaked, which an old ticket indicated they would) ... I thought what-the-heck, it's time for supporting case-insensitive patterns. This turned out to be tricky to implement, as I gleaned from talking with Seth about an approach he had tried a while back but abandoned. But I now have it working. Here's the blurb from the NEWS entry in the topics/vern/case-insensitive-patterns branch: - You can now specify that a pattern matches in a case-insensitive fashion by adding 'i' to the end of its specification. So for example /fOO/i == "Foo" yields T, as does /fOO/i in "xFoObar". Characters enclosed in quotes however keep their casing, so /"fOO"/i in "xFoObar" yields F, though it yields T for "xfOObar". You can achieve the same functionality for a subpattern enclosed in parentheses by adding "+i" to the open parenthesis, optionally followed by whitespace. So for example "/foo|(+i bar)/" will match "BaR", but not "FoO". For both ways of specifying case-insensitivity, characters enclosed in double quotes maintain their case-sensitivity. So for example /"foo"/i will not match "Foo", but it will match "foo". The funky (+i ...) syntax isn't meant for general user consumption (though it's okay if a user wants to use it directly), but rather is how I implemented /pattern/i functionality. Basically, /pattern/i turns into /(+i pattern)/. That switch is necessary because the robust way to implement case-insensitive patterns, such that they can be composed with the & and | operators and behave as expected, is to modify the parsing of REs to turn any instance of a letter into a character class (so that /foo/ becomes /[Ff][Oo]Oo]/, just like people have been doing by hand for years), and also to modify the parsing of character classes. That requires alerting the RE scanner that it's doing a case-insensitive (sub)pattern, which in turn requires a prefix operator that specifies case-insensitivity. Let me know if you have any concerns. Otherwise, I'll tee this up for merging early next week. Vern From johanna at icir.org Fri Jun 29 12:14:21 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 29 Jun 2018 12:14:21 -0700 Subject: [Bro-Dev] case-insensitive patterns In-Reply-To: <201806291900.w5TJ0VZD018815@fruitcake.ICSI.Berkeley.EDU> References: <201806291900.w5TJ0VZD018815@fruitcake.ICSI.Berkeley.EDU> Message-ID: <20180629191421.ui463xvw6fanmjdv@Trafalgar.local> On Fri, Jun 29, 2018 at 12:00:30PM -0700, Vern Paxson wrote: > Once I wound up monkeying around with the internals of the pattern-matching > code (to fix leaks, because Johanna [correctly] pushed back on adding the > &/| operators for general use if they leaked, which an old ticket indicated > they would) ... I thought what-the-heck, it's time for supporting > case-insensitive patterns. Thanks a lot for searching the memory leaks - I know that has been a pain. > This turned out to be tricky to implement, as I gleaned from talking with > Seth about an approach he had tried a while back but abandoned. But I now > have it working. This is great - case-insensitive pattern have been something that I wanted to have for a long time. > You can achieve the same functionality for a subpattern enclosed in > parentheses by adding "+i" to the open parenthesis, optionally followed > by whitespace. So for example "/foo|(+i bar)/" will match "BaR", but > not "FoO". Hum. Is there a reason why we come up with our own syntax for this? Other implementations already have this using a just slightly different syntax. To do the same in perl, you would use "/foo|(?i:bar)/". It also supports turning off case insensitivity for part of a pattern by doing "/foo|(?-i:bar)/". Furthermore you can also switch it on for the rest of the pattern by doing (?i) - after that everything is insensitive. https://perldoc.perl.org/perlre.html#Extended-Patterns has more details Python supports the exact same syntax. And - to make things easier for users I think it would be way nicer if we just also would do this. > The funky (+i ...) syntax isn't meant for general user consumption (though > it's okay if a user wants to use it directly), but rather is how I implemented > /pattern/i functionality. And this is fine - but if we support it I would actually prefer just making it explicit and doing it like everyone else :) Johanna From vern at corelight.com Fri Jun 29 12:23:26 2018 From: vern at corelight.com (Vern Paxson) Date: Fri, 29 Jun 2018 12:23:26 -0700 Subject: [Bro-Dev] case-insensitive patterns In-Reply-To: <20180629191421.ui463xvw6fanmjdv@Trafalgar.local> (Fri, 29 Jun 2018 12:14:21 PDT). Message-ID: <201806291923.w5TJNRd0023391@fruitcake.ICSI.Berkeley.EDU> > Hum. Is there a reason why we come up with our own syntax for this? No, just that I didn't have the other syntax on my radar. I was looking at Snort & Suricata and didn't find this; I'm not a PCRE user myself. It's simple to change, will do so now (though I think mine is slightly more cool ;-). > Python supports the exact same syntax. And - to make things easier for > users I think it would be way nicer if we just also would do this. Sure. Just so I have this right: it looks like the preferred would not be /(?i foo)/ but rather /(?i)foo/, yes? Vern From vern at corelight.com Fri Jun 29 12:26:34 2018 From: vern at corelight.com (Vern Paxson) Date: Fri, 29 Jun 2018 12:26:34 -0700 Subject: [Bro-Dev] case-insensitive patterns In-Reply-To: (Fri, 29 Jun 2018 12:23:26 PDT). Message-ID: <201806291926.w5TJQYIt023942@fruitcake.ICSI.Berkeley.EDU> > Just so I have this right: it looks like the preferred would not be > /(?i foo)/ but rather /(?i)foo/, yes? Oh and to follow up on this, so in PCRE does /x((?i)bar)foo/ make the "foo" part case-insensitive too, or not? It's not obvious to me from the page you pointed me at, and I don't have an environment set up to definitively test this. Vern From johanna at icir.org Fri Jun 29 12:38:25 2018 From: johanna at icir.org (Johanna Amann) Date: Fri, 29 Jun 2018 12:38:25 -0700 Subject: [Bro-Dev] case-insensitive patterns In-Reply-To: <201806291926.w5TJQYIt023942@fruitcake.ICSI.Berkeley.EDU> References: <201806291926.w5TJQYIt023942@fruitcake.ICSI.Berkeley.EDU> Message-ID: <20180629193825.w7syr4dwvpinlgnz@Trafalgar.local> On Fri, Jun 29, 2018 at 12:26:34PM -0700, Vern Paxson wrote: > > Just so I have this right: it looks like the preferred would not be > > /(?i foo)/ but rather /(?i)foo/, yes? > > Oh and to follow up on this, so in PCRE does /x((?i)bar)foo/ make the "foo" > part case-insensitive too, or not? It's not obvious to me from the page > you pointed me at, and I don't have an environment set up to definitively > test this. It does not. So: /x((?i)bar)foo/ is (kind-of [1]) equivalent to /x(?i:bar)foo/ and only makes the bar part case insensitive. /x(?i)barfoo/ would make barfoo case insensitive. I am fine with implementing either (?[flags]) or (?[flags]:[pattern) or both. I hope this makes sense. Johanna [1]: Kind-of, because (?i:bar) is non-capturing, whereas ((?i)bar) captures "bar" in the first capture-variable. Since we don't support capturing at the moment that does not really matter for us - it might be of importance in the future though. From vern at corelight.com Fri Jun 29 12:49:57 2018 From: vern at corelight.com (Vern Paxson) Date: Fri, 29 Jun 2018 12:49:57 -0700 Subject: [Bro-Dev] case-insensitive patterns In-Reply-To: <20180629193825.w7syr4dwvpinlgnz@Trafalgar.local> (Fri, 29 Jun 2018 12:38:25 PDT). Message-ID: <201806291949.w5TJnvKr027383@fruitcake.ICSI.Berkeley.EDU> > I am fine with implementing either (?[flags]) or (?[flags]:[pattern) or > both. I'm going to just do the latter then, as it's a simple syntax change from what I currently have, whereas the other is more involved. Vern From vern at corelight.com Fri Jun 29 13:17:26 2018 From: vern at corelight.com (Vern Paxson) Date: Fri, 29 Jun 2018 13:17:26 -0700 Subject: [Bro-Dev] case-insensitive patterns In-Reply-To: (Fri, 29 Jun 2018 12:49:57 PDT). Message-ID: <201806292017.w5TKHQ22004196@fruitcake.ICSI.Berkeley.EDU> > > I am fine with implementing either (?[flags]) or (?[flags]:[pattern) or > > both. > > I'm going to just do the latter then, as it's a simple syntax change from > what I currently have, whereas the other is more involved. (this change is now pushed)