From ottobackwards at gmail.com Thu May 9 07:08:04 2019 From: ottobackwards at gmail.com (Otto Fowler) Date: Thu, 9 May 2019 10:08:04 -0400 Subject: [Zeek-Dev] History or explaination of default analyzer handling of isPartial and hasGap Message-ID: Hi everyone, The analyzers, from what I can see, all have pretty much the same handling for isPartial() and hasGap in DeliverStream(). I was wondering if anyone could explain or give some history behind why that became the default and what the issues were before, or point to some documentation. Thank you ottO -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190509/fcba8748/attachment.html From seth at corelight.com Thu May 9 08:53:04 2019 From: seth at corelight.com (Seth Hall) Date: Thu, 09 May 2019 11:53:04 -0400 Subject: [Zeek-Dev] History or explaination of default analyzer handling of isPartial and hasGap In-Reply-To: References: Message-ID: On 9 May 2019, at 10:08, Otto Fowler wrote: > The analyzers, from what I can see, all have pretty much the same > handling > for isPartial() and hasGap in DeliverStream(). > > I was wondering if anyone could explain or give some history behind > why > that became the default and what the issues were before, or point to > some > documentation. It mostly became that way due to copying code from older analyzers as new ones were written, but there are some reasons why it still makes sense. Most of the protocol analyzers today can't resync if they miss any data and they have the related issue that they can't synchronize to the stream correctly if they didn't see the beginning. As analyzers begin to acquire the ability to sychronize to streams, many of these conditionals that stop analysis will begin to disappear because there won't be a point anymore where irregularities like packet loss need to lead to stopping analysis. We've been thinking about this for a long time and there are some analyzers (the SMB analyzer being the biggest example) that deals just fine with stream resychronization if you'd like to see something that doesn't have the same handling. We're hoping to have a broader and easier approach at this problem eventually, just nothing ready yet. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com From jmellander at lbl.gov Wed May 29 10:23:32 2019 From: jmellander at lbl.gov (Jim Mellander) Date: Wed, 29 May 2019 10:23:32 -0700 Subject: [Zeek-Dev] CFLAGS/CXXFLAGS bug with configure Message-ID: Hi all: I've been tinkering with optimizing zeek for performance, and came across a problem with CFLAGS/CXXFLAGS specified to the configure script. Lets say I want to try the -O3 flag for more aggressive optimization than the default -O2: $ CFLAGS='-O3' CXXFLAGS='-O3' ./configure ... ====================| Bro Build Summary |===================== ... CFLAGS: -O3 -Wall -Wno-unused -O2 -g -DNDEBUG ... CXXFLAGS: -O3 -Wall -Wno-unused -std=c++11 -O2 -g -DNDEBUG ... ================================================================ Note that the supplied CFLAGS/CXXFLAGS is before the default options for the build. From the GCC man page: *If you use multiple -O options, with or without level numbers, the last such option is the one that is effective.* This, of course, means that the -O3 specified will not take effect. Running make -in confirms that those flags are being passed to the compiler. There appear to be several alternatives to allow supplied flags to override the defaults, the most reasonable appearing to be modifying the build commands to apply CFLAGS/CXXFLAGS after the defaults. I'm not a cmake maven, but it seems likely that lines like: set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${bro_cache_CMAKE_C_FLAGS}") could be reversed ala: set(CMAKE_C_FLAGS "${bro_cache_CMAKE_C_FLAGS} ${CMAKE_C_FLAGS}") to achieve that effect. (Yes, I know --build-type=Release compiles with -O3, but the point remains that overriding flags via environment variables is generally expected to be operative) Comments? Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190529/0b75ff2c/attachment.html From jsiwek at corelight.com Wed May 29 14:21:22 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Wed, 29 May 2019 14:21:22 -0700 Subject: [Zeek-Dev] CFLAGS/CXXFLAGS bug with configure In-Reply-To: References: Message-ID: On Wed, May 29, 2019 at 10:30 AM Jim Mellander wrote: > There appear to be several alternatives to allow supplied flags to override the defaults, the most reasonable appearing to be modifying the build commands to apply CFLAGS/CXXFLAGS after the defaults. > > I'm not a cmake maven, but it seems likely that lines like: > set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${bro_cache_CMAKE_C_FLAGS}") > > could be reversed ala: > > set(CMAKE_C_FLAGS "${bro_cache_CMAKE_C_FLAGS} ${CMAKE_C_FLAGS}") > > to achieve that effect. The couple places where we set CMAKE_C{XX}_FLAGS don't seem to matter: the flags associated with CMAKE_BUILD_TYPE always come last in the generated Makefile and I think that's just what CMake does, not something we can control. However, you should be able to bypass CMake from inserting build-type flags entirely if you supply an unknown/bogus build type: CFLAGS='-O3' CXXFLAGS='-O3' ./configure --build-type=none - Jon From jmellander at lbl.gov Thu May 30 10:04:27 2019 From: jmellander at lbl.gov (Jim Mellander) Date: Thu, 30 May 2019 10:04:27 -0700 Subject: [Zeek-Dev] CFLAGS/CXXFLAGS bug with configure In-Reply-To: References: Message-ID: Thanks, sounds like a CMake bug, or at least a violation of the Principle of Least Surprise - explicit settings are always expected to override defaults.... In any event, in the Zeek context, this thread should suffice to document this (admittedly obscure) issue, and mitigation. On Wed, May 29, 2019 at 2:21 PM Jon Siwek wrote: > On Wed, May 29, 2019 at 10:30 AM Jim Mellander wrote: > > > There appear to be several alternatives to allow supplied flags to > override the defaults, the most reasonable appearing to be modifying the > build commands to apply CFLAGS/CXXFLAGS after the defaults. > > > > I'm not a cmake maven, but it seems likely that lines like: > > set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${bro_cache_CMAKE_C_FLAGS}") > > > > could be reversed ala: > > > > set(CMAKE_C_FLAGS "${bro_cache_CMAKE_C_FLAGS} ${CMAKE_C_FLAGS}") > > > > to achieve that effect. > > The couple places where we set CMAKE_C{XX}_FLAGS don't seem to matter: > the flags associated with CMAKE_BUILD_TYPE always come last in the > generated Makefile and I think that's just what CMake does, not > something we can control. > > However, you should be able to bypass CMake from inserting build-type > flags entirely if you supply an unknown/bogus build type: > > CFLAGS='-O3' CXXFLAGS='-O3' ./configure --build-type=none > > - Jon > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20190530/1a603ad5/attachment.html