From jsiwek at corelight.com Mon Dec 9 14:44:58 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 9 Dec 2019 14:44:58 -0800 Subject: [Zeek-Dev] maybe migrating Zeek CI from Travis to Cirrus Message-ID: I experimented last week moving the CI config to Cirrus and found several benefits to making that switch. Example build in my Zeek fork: https://cirrus-ci.com/build/6738081318699008 Notable differences from the Travis config: * Free for open-source projects, can allocate up to 8 cpus (we get 2 cpus on Travis) * Concurrent tasks up to 8 Linux, 2 FreeBSD, and 1 macOS (Travis is 8 total, doesn't offer FreeBSD, and we haven't configured it for macOS). This is also limited per _user_ not per repo, so misc. PRs will no longer backup the CI queue like they do on Travis. E.g. to quote docs: "For example, if you have 10 active contributors to a repository then you can end up with 130 tasks running in parallel for the repository" * I've set up pre-built Dockerfiles for all Linux systems to make it faster to get those up and running rather than wait to download dependencies every time (and potentially encounter transient network issues as we see sometimes on Travis). This also makes it just as easy as Travis to reproduce the CI environment locally (Travis also used docker, but didn't organize things as standard Dockerfiles) * I've set the external pcap files (~850mb) to get cached after the first successful build (or auto-reuploaded if they ever change later) and downloading from the cache takes around 1 minute rather than sometimes 15-20 on Travis * The above differences amount to getting CI results in about 12-30 minutes (macOS and memleak task are still a bit slow) rather than 50-60 minutes on Travis. So my idea going forward (let me know if there's other thoughts): I'll set up the main Zeek GitHub organization/repo to use Cirrus in parallel with existing Travis CI. If it performs well for several months, we can think about completely migrating off Travis. Don't necessarily need to move other projects to it, but might anyway in spare time. Also definitely would need to move the Coverity Scan cron somewhere else. IMO, this doesn't fit in with the CI config, it just needs to run regularly somewhere, so GitHub Actions may be a good fit for that. Other caveats: * I haven't configured Cirrus to access the external private test suite. IMO, we might just stop using that, since it's awkward for an open-source project: we don't give access to private pcaps from arbitrary PRs that trigger CI, but then what happens if the merger finds failures in the private test suite (or forget to run it before merging) ? Either need to give access to the private pcaps or else the core team basically becomes responsible (and I can imagine some new features/PRs being non-trivial to hunt down diffs ourselves). * Haven't figured out how to configure email notifications on Cirrus CI failures, but can figure that out once it's up and running. - Jon From tim at corelight.com Mon Dec 9 15:05:12 2019 From: tim at corelight.com (Tim Wojtulewicz) Date: Mon, 9 Dec 2019 16:05:12 -0700 Subject: [Zeek-Dev] maybe migrating Zeek CI from Travis to Cirrus In-Reply-To: References: Message-ID: I?m pretty impressed by the build time improvement. A few questions: 1. Can we build arbitrary branches on Cirrus? To do that on Travis you either had to make a draft PR off the branch, or push a commit to your branch that whitelisted the branch name and then manually make a build with a cron job. Neither were great solutions. 2. Why move the Coverity build? Does it not offer a cron-like interface for running regular builds? 3. Can you move the travis-job script up into the ci directory instead of leaving it in testing? Tim > On Dec 9, 2019, at 3:44 PM, Jon Siwek wrote: > > The above differences amount to getting CI results in about 12-30 > minutes (macOS and memleak task are still a bit slow) rather than > 50-60 minutes on Travis. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/zeek-dev/attachments/20191209/336a534f/attachment.html From jsiwek at corelight.com Mon Dec 9 17:15:10 2019 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 9 Dec 2019 17:15:10 -0800 Subject: [Zeek-Dev] maybe migrating Zeek CI from Travis to Cirrus In-Reply-To: References: Message-ID: On Mon, Dec 9, 2019 at 3:05 PM Tim Wojtulewicz wrote: > 1. Can we build arbitrary branches on Cirrus? Yes, it's a bit awkward how you do it, but I guess better than Travis. In Cirrus, you can manually trigger an arbitrary branch, but like Travis, the default is to reject branches that are not whitelisted: https://github.com/jsiwek/zeek/blob/94c4790610cf279157ecd536df144fadb83835c1/.cirrus.yml#L13-L16 But in Cirrus, you also get the option to override the .cirrus.yml directly in the web interface for manually triggering a branch. So basically you can copy-paste the .cirrus.yml and remove the "only_if" lines, no temporary hacks needed to be committed into the branch itself. > 2. Why move the Coverity build? Does it not offer a cron-like interface for running regular builds? Didn't look like Cirrus has cron-like functionality yet, see https://github.com/cirruslabs/cirrus-ci-docs/issues/39 But also the Coverity build doesn't exactly fit the same mold as the other tasks. If we didn't have any Coverity usage limitations (think it's 1 build per day?) and it was required to run on every new commit in `master`, then I'd say it would be ideal to have that ported into Cirrus, too, but since that's not the case, I don't have much preference where it runs except doesn't seem efficient to pay to bypass the 50 minute Travis time limit anymore just for doing the Coverity build if GitHub Actions has a free 6 hour time limit and also supports cron workflows. > 3. Can you move the travis-job script up into the ci directory instead of leaving it in testing? Just to keep things a bit more organized while Travis is still being used in parallel with Cirrus? Sure, but eventually travis-job can get removed: rewrote/factored bits of that into the other ci/*.sh scripts for Cirrus to use. - Jon From tim at corelight.com Mon Dec 9 17:25:53 2019 From: tim at corelight.com (Tim Wojtulewicz) Date: Mon, 9 Dec 2019 18:25:53 -0700 Subject: [Zeek-Dev] maybe migrating Zeek CI from Travis to Cirrus In-Reply-To: References: Message-ID: > On Dec 9, 2019, at 6:15 PM, Jon Siwek wrote: > > On Mon, Dec 9, 2019 at 3:05 PM Tim Wojtulewicz wrote: > >> 1. Can we build arbitrary branches on Cirrus? > > Yes, it's a bit awkward how you do it, but I guess better than Travis. > In Cirrus, you can manually trigger an arbitrary branch, but like > Travis, the default is to reject branches that are not whitelisted: > > https://github.com/jsiwek/zeek/blob/94c4790610cf279157ecd536df144fadb83835c1/.cirrus.yml#L13-L16 > > But in Cirrus, you also get the option to override the .cirrus.yml > directly in the web interface for manually triggering a branch. So > basically you can copy-paste the .cirrus.yml and remove the "only_if" > lines, no temporary hacks needed to be committed into the branch > itself. That sounds reasonable enough. It just always bothered me having to do weird tricks with the actual branch to make it build. > >> 2. Why move the Coverity build? Does it not offer a cron-like interface for running regular builds? > > Didn't look like Cirrus has cron-like functionality yet, see > https://github.com/cirruslabs/cirrus-ci-docs/issues/39 > > But also the Coverity build doesn't exactly fit the same mold as the > other tasks. If we didn't have any Coverity usage limitations (think > it's 1 build per day?) and it was required to run on every new commit > in `master`, then I'd say it would be ideal to have that ported into > Cirrus, too, but since that's not the case, I don't have much > preference where it runs except doesn't seem efficient to pay to > bypass the 50 minute Travis time limit anymore just for doing the > Coverity build if GitHub Actions has a free 6 hour time limit and also > supports cron workflows. I see your point there. A single GitHub action would be enough to do the single-platform build that we need for Coverity. > >> 3. Can you move the travis-job script up into the ci directory instead of leaving it in testing? > > Just to keep things a bit more organized while Travis is still being > used in parallel with Cirrus? Sure, but eventually travis-job can get > removed: rewrote/factored bits of that into the other ci/*.sh scripts > for Cirrus to use. That?s what I was thinking. It seems strange to have it out on an island by itself, even if it?s going to get removed eventually. From johanna at corelight.com Tue Dec 17 06:24:10 2019 From: johanna at corelight.com (Johanna Amann) Date: Tue, 17 Dec 2019 15:24:10 +0100 Subject: [Zeek-Dev] Question about IntrusivePtr and incomplete types Message-ID: Hi, I just tried to use our new IntrusivePtr type for the first time - and encountered a (for me) unexpected problem. In my specific case, I want to introduce a new member variable for TableVal. The type of it is: IntrusivePtr change_func; Compiling Zeek with this addition to the TableVal class will yield in the following compilation error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -DDOCTEST_CONFIG_DISABLE -Isrc/zeekygen -I../src/zeekygen -Isrc -I../src -I/opt/local/include -Iaux/binpac/lib -I../aux/binpac/lib -I../aux/paraglob/include -I../aux/broker/3rdparty/caf/libcaf_test -I../aux/broker/3rdparty/caf/libcaf_openssl -I../aux/broker/3rdparty/caf/libcaf_io -Iaux/broker/caf-build/libcaf_core -I../aux/broker/3rdparty/caf/libcaf_core -Iaux/broker/include -I../aux/broker/include -I. -Wall -Wno-unused -g -DDEBUG -DBRO_DEBUG -Wno-register -std=c++17 -g -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -mmacosx-version-min=10.14 -MD -MT src/zeekygen/CMakeFiles/bro_zeekygen.dir/ScriptInfo.cc.o -MF src/zeekygen/CMakeFiles/bro_zeekygen.dir/ScriptInfo.cc.o.d -o src/zeekygen/CMakeFiles/bro_zeekygen.dir/ScriptInfo.cc.o -c ../src/zeekygen/ScriptInfo.cc In file included from ../src/zeekygen/ScriptInfo.cc:7: In file included from ../src/zeekygen/Manager.h:14: In file included from ../src/Val.h:23: ../src/IntrusivePtr.h:88:4: error: no matching function for call to 'Unref' Unref(ptr_); ^~~~~ ../src/Val.h:903:2: note: in instantiation of member function 'IntrusivePtr::~IntrusivePtr' requested here TableVal() {} ^ ../src/Obj.h:199:13: note: candidate function not viable: cannot convert argument of incomplete type 'IntrusivePtr::pointer' (aka 'Expr *') to 'BroObj *' for 1st argument inline void Unref(BroObj* o) ^ Including Expr.h in Val.h is not trivially possible due to other ordering conflicts breaking the build. Is there some way to use an IntrusivePtr in this case - or is is just not possible to use it in cases where it is not possible to use complete types? Thanks, Johanna From dominik.charousset at corelight.com Tue Dec 17 06:29:03 2019 From: dominik.charousset at corelight.com (Dominik Charousset) Date: Tue, 17 Dec 2019 15:29:03 +0100 Subject: [Zeek-Dev] Question about IntrusivePtr and incomplete types In-Reply-To: References: Message-ID: <97DD599C-D7CA-4F5E-89F2-411EFF919BB9@corelight.com> > I just tried to use our new IntrusivePtr type for the first time - and > encountered a (for me) unexpected problem. > > In my specific case, I want to introduce a new member variable for > TableVal. The type of it is: > > IntrusivePtr change_func; > > Compiling Zeek with this addition to the TableVal class will yield in > the following compilation error: > > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ > -DDOCTEST_CONFIG_DISABLE -Isrc/zeekygen -I../src/zeekygen -Isrc > -I../src -I/opt/local/include -Iaux/binpac/lib -I../aux/binpac/lib > -I../aux/paraglob/include -I../aux/broker/3rdparty/caf/libcaf_test > -I../aux/broker/3rdparty/caf/libcaf_openssl > -I../aux/broker/3rdparty/caf/libcaf_io > -Iaux/broker/caf-build/libcaf_core > -I../aux/broker/3rdparty/caf/libcaf_core -Iaux/broker/include > -I../aux/broker/include -I. -Wall -Wno-unused -g -DDEBUG -DBRO_DEBUG > -Wno-register -std=c++17 -g -isysroot > /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk > -mmacosx-version-min=10.14 -MD -MT > src/zeekygen/CMakeFiles/bro_zeekygen.dir/ScriptInfo.cc.o -MF > src/zeekygen/CMakeFiles/bro_zeekygen.dir/ScriptInfo.cc.o.d -o > src/zeekygen/CMakeFiles/bro_zeekygen.dir/ScriptInfo.cc.o -c > ../src/zeekygen/ScriptInfo.cc > In file included from ../src/zeekygen/ScriptInfo.cc:7: > In file included from ../src/zeekygen/Manager.h:14: > In file included from ../src/Val.h:23: > ../src/IntrusivePtr.h:88:4: error: no matching function for call to > 'Unref' > Unref(ptr_); > ^~~~~ > ../src/Val.h:903:2: note: in instantiation of member function > 'IntrusivePtr::~IntrusivePtr' requested here > TableVal() {} > ^ > ../src/Obj.h:199:13: note: candidate function not viable: cannot convert > argument of incomplete type 'IntrusivePtr::pointer' (aka 'Expr *') > to 'BroObj *' for 1st argument > inline void Unref(BroObj* o) > ^ > Including Expr.h in Val.h is not trivially possible due to other > ordering conflicts breaking the build. > > Is there some way to use an IntrusivePtr in this case - or is is just > not possible to use it in cases where it is not possible to use complete > types? You should be able to get away with forward declarations as long as you provide matching prototypes for Ref and Unref. From johanna at corelight.com Tue Dec 17 07:04:20 2019 From: johanna at corelight.com (Johanna Amann) Date: Tue, 17 Dec 2019 16:04:20 +0100 Subject: [Zeek-Dev] Question about IntrusivePtr and incomplete types In-Reply-To: <97DD599C-D7CA-4F5E-89F2-411EFF919BB9@corelight.com> References: <97DD599C-D7CA-4F5E-89F2-411EFF919BB9@corelight.com> Message-ID: <129864EE-D951-4512-951A-A3910D7EDF88@corelight.com> >> Is there some way to use an IntrusivePtr in this case - or is is just >> not possible to use it in cases where it is not possible to use >> complete >> types? > > You should be able to get away with forward declarations as long as > you provide matching prototypes for Ref and Unref. If I do that I get linking errors: "Ref(Expr*)", referenced from: Expr::Ref() in IdentifierInfo.cc.o Expr::Ref() in Expr.cc.o Expr::Ref() in ID.cc.o WhenStmt::Exec(Frame*, stmt_flow_type&) const in Stmt.cc.o Expr::Ref() in Type.cc.o Expr::Ref() in Val.cc.o make_var(ID*, BroType*, init_class, Expr*, List*, decl_type, int) in Var.cc.o ... "Unref(Expr*)", referenced from: zeekygen::IdentifierInfo::Redefinition::~Redefinition() in IdentifierInfo.cc.o Attr::~Attr() in Attr.cc.o UnaryExpr::~UnaryExpr() in Expr.cc.o BinaryExpr::~BinaryExpr() in Expr.cc.o CondExpr::~CondExpr() in Expr.cc.o ScheduleExpr::~ScheduleExpr() in Expr.cc.o CallExpr::~CallExpr() in Expr.cc.o ... Which makes sense, because Ref/Unref(Expr*) never exist - they only exist for BroObj*. I could implement a forward function that does nothing besides calling Ref(BroObj) in Expr.cc - but that seems? not really elegant/worth using anymore in this case. Unless I an declare them in another way that I am missing. Johanna From johanna at corelight.com Tue Dec 17 07:06:22 2019 From: johanna at corelight.com (Johanna Amann) Date: Tue, 17 Dec 2019 16:06:22 +0100 Subject: [Zeek-Dev] Question about IntrusivePtr and incomplete types In-Reply-To: <129864EE-D951-4512-951A-A3910D7EDF88@corelight.com> References: <97DD599C-D7CA-4F5E-89F2-411EFF919BB9@corelight.com> <129864EE-D951-4512-951A-A3910D7EDF88@corelight.com> Message-ID: On 17 Dec 2019, at 16:04, Johanna Amann wrote: >>> Is there some way to use an IntrusivePtr in this case - or is is >>> just >>> not possible to use it in cases where it is not possible to use >>> complete >>> types? >> >> You should be able to get away with forward declarations as long as >> you provide matching prototypes for Ref and Unref. > > If I do that I get linking errors: > > "Ref(Expr*)", referenced from: > Expr::Ref() in IdentifierInfo.cc.o > Expr::Ref() in Expr.cc.o > Expr::Ref() in ID.cc.o > WhenStmt::Exec(Frame*, stmt_flow_type&) const in Stmt.cc.o > Expr::Ref() in Type.cc.o > Expr::Ref() in Val.cc.o > make_var(ID*, BroType*, init_class, Expr*, List*, > decl_type, int) in Var.cc.o > ... > "Unref(Expr*)", referenced from: > zeekygen::IdentifierInfo::Redefinition::~Redefinition() in > IdentifierInfo.cc.o > Attr::~Attr() in Attr.cc.o > UnaryExpr::~UnaryExpr() in Expr.cc.o > BinaryExpr::~BinaryExpr() in Expr.cc.o > CondExpr::~CondExpr() in Expr.cc.o > ScheduleExpr::~ScheduleExpr() in Expr.cc.o > CallExpr::~CallExpr() in Expr.cc.o > ... > > Which makes sense, because Ref/Unref(Expr*) never exist - they only > exist for BroObj*. I could implement a forward function that does > nothing besides calling Ref(BroObj) in Expr.cc - but that seems? not > really elegant/worth using anymore in this case. Unless I an declare > them in another way that I am missing. For completeness - the forward declaration/prototype in Val.h is: class Expr; void Ref(Expr*); void Unref(Expr*); Johanna From dominik.charousset at corelight.com Tue Dec 17 07:28:28 2019 From: dominik.charousset at corelight.com (Dominik Charousset) Date: Tue, 17 Dec 2019 16:28:28 +0100 Subject: [Zeek-Dev] Question about IntrusivePtr and incomplete types In-Reply-To: <129864EE-D951-4512-951A-A3910D7EDF88@corelight.com> References: <97DD599C-D7CA-4F5E-89F2-411EFF919BB9@corelight.com> <129864EE-D951-4512-951A-A3910D7EDF88@corelight.com> Message-ID: >>> Is there some way to use an IntrusivePtr in this case - or is is just >>> not possible to use it in cases where it is not possible to use complete >>> types? >> >> You should be able to get away with forward declarations as long as you provide matching prototypes for Ref and Unref. > > If I do that I get linking errors: > > "Ref(Expr*)", referenced from: > Expr::Ref() in IdentifierInfo.cc.o > Expr::Ref() in Expr.cc.o > Expr::Ref() in ID.cc.o > WhenStmt::Exec(Frame*, stmt_flow_type&) const in Stmt.cc.o > Expr::Ref() in Type.cc.o > Expr::Ref() in Val.cc.o > make_var(ID*, BroType*, init_class, Expr*, List*, decl_type, int) in Var.cc.o > ... > "Unref(Expr*)", referenced from: > zeekygen::IdentifierInfo::Redefinition::~Redefinition() in IdentifierInfo.cc.o > Attr::~Attr() in Attr.cc.o > UnaryExpr::~UnaryExpr() in Expr.cc.o > BinaryExpr::~BinaryExpr() in Expr.cc.o > CondExpr::~CondExpr() in Expr.cc.o > ScheduleExpr::~ScheduleExpr() in Expr.cc.o > CallExpr::~CallExpr() in Expr.cc.o > ... > > Which makes sense, because Ref/Unref(Expr*) never exist - they only exist for BroObj*. I could implement a forward function that does nothing besides calling Ref(BroObj) in Expr.cc - but that seems? not really elegant/worth using anymore in this case. Unless I an declare them in another way that I am missing. I agree, that extra bit of boilerplate is not very elegant. However, the compiler has no knowledge of the class hierarchy of Expr and thus doesn?t know that Expr* is convertible to BroObj*. Hence, we need the one extra layer of indirection via Ref/Unref overloads for Expr. Implementing the functions like this should make the linker happy as well: void Ref(Expr* ptr) { Ref(static_cast(ptr); } void Unref(Expr* ptr) { Unref(static_cast(ptr); } There is one alternative: The compiler only needs to see Ref/Unref overloads when *instantiating* any member function of IntrusivePtr. You probably have some inline constructor or destructor that causes the compiler to instantiate the template. Moving *all* member function that operate on the IntrusivePtr member out of the .h file should work as well. For example: ``` struct foo; struct foobar { foobar(); ~foobar(); IntrusivePtr f; }; ``` should compile, whereas ``` struct foo; struct foobar { foobar(); ~foobar() {} IntrusivePtr f; }; ``` won?t. In the latter case, providing the destructor inline causes the compiler to instantiate the destructor of `IntrusivePtr`, which relies on an Unref function it can?t find at this point.