From asharma at lbl.gov Tue Jul 10 12:03:12 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 10 Jul 2018 12:03:12 -0700 Subject: [Bro-Dev] error compiling master Message-ID: <20180710190310.GA53027@MacPro-2331.local> Probably obvious but I am not very sure so asking here. I see this error trying to build current master - Thoughts what am I missing ? (trying the build on: FreeBSD 10.3-STABLE) I am building as: ./configure --prefix=/usr/local/bro-master && make .. .. gmake[6]: Leaving directory '/usr/local/src/master/build/aux/broker/caf-build' gmake[6]: Entering directory '/usr/local/src/master/build/aux/broker/caf-build' [ 96%] Building CXX object libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/src/manager.cpp.o clang: warning: argument unused during compilation: '-L/lib' /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:100:8: error: use of undeclared identifier 'get_or' if (!get_or(config(), "middleman.attach-utility-actors", false)) ^ /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:102:14: error: calling a private constructor of class 'caf::typed_actor, unsigned short, caf::intrusive_ptr, std::__1::set, std::__1::less >, std::__1::allocator > >, std::__1::basic_string, bool>, caf::output_tuple >, caf::typed_mpi, unsigned short, std::__1::basic_string, bool>, caf::output_tuple >, caf::typed_mpi, std::__1::basic_string, unsigned short>, caf::output_tuple, std::__1::set, std::__1::less >, std::__1::allocator > > > >, caf::typed_mpi, caf::actor_addr, unsigned short>, caf::output_tuple >, caf::typed_mpi, unsigned short>, caf::output_tuple >, caf::typed_mpi, caf::node_id, std::__1::basic_string, caf::message, std::__1::set, std::__1::less >, std::__1::allocator > > >, caf::output_tuple > >, caf::typed_mpi, caf::node_id>, caf::output_tuple, unsigned short> > >' manager_ = nullptr; ^ /usr/local/include/caf/typed_actor.hpp:270:3: note: declared private here typed_actor(actor_control_block* ptr) : ptr_(ptr) { ^ /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:112:27: error: no member named 'openssl_certificate' in 'caf::actor_system_config' if (system().config().openssl_certificate.size() == 0) ~~~~~~~~~~~~~~~~~ ^ /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:114:27: error: no member named 'openssl_key' in 'caf::actor_system_config' if (system().config().openssl_key.size() == 0) ~~~~~~~~~~~~~~~~~ ^ /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:131:10: error: use of undeclared identifier 'openssl_manager'; did you mean 'opencl_manager'? return openssl_manager; ^~~~~~~~~~~~~~~ opencl_manager /usr/local/include/caf/actor_system.hpp:140:7: note: 'opencl_manager' declared here opencl_manager, ^ /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:140:14: error: no member named 'openssl_certificate' in 'caf::actor_system_config' return cfg.openssl_certificate.size() > 0 || cfg.openssl_key.size() > 0 ~~~ ^ /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:140:52: error: no member named 'openssl_key' in 'caf::actor_system_config' return cfg.openssl_certificate.size() > 0 || cfg.openssl_key.size() > 0 ~~~ ^ /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:141:17: error: no member named 'openssl_passphrase' in 'caf::actor_system_config' || cfg.openssl_passphrase.size() > 0 || cfg.openssl_capath.size() > 0 ~~~ ^ /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:141:54: error: no member named 'openssl_capath' in 'caf::actor_system_config' || cfg.openssl_passphrase.size() > 0 || cfg.openssl_capath.size() > 0 ~~~ ^ /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:142:17: error: no member named 'openssl_cafile' in 'caf::actor_system_config' || cfg.openssl_cafile.size() > 0; ~~~ ^ 10 errors generated. gmake[6]: *** [libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/build.make:63: libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/src/manager.cpp.o] Error 1 gmake[6]: Leaving directory '/usr/local/src/master/build/aux/broker/caf-build' gmake[5]: *** [CMakeFiles/Makefile2:394: libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/all] Error 2 gmake[5]: Leaving directory '/usr/local/src/master/build/aux/broker/caf-build' gmake[4]: *** [Makefile:128: all] Error 2 gmake[4]: Leaving directory '/usr/local/src/master/build/aux/broker/caf-build' *** Error code 2 Stop. make[3]: stopped in /usr/local/src/master/build *** Error code 1 Stop. make[2]: stopped in /usr/local/src/master/build *** Error code 1 Stop. make[1]: stopped in /usr/local/src/master/build *** Error code 1 Stop. make: stopped in /usr/local/src/master From dnthayer at illinois.edu Tue Jul 10 13:24:34 2018 From: dnthayer at illinois.edu (Daniel Thayer) Date: Tue, 10 Jul 2018 15:24:34 -0500 Subject: [Bro-Dev] error compiling master In-Reply-To: <20180710190310.GA53027@MacPro-2331.local> References: <20180710190310.GA53027@MacPro-2331.local> Message-ID: <9b303e0d-c85b-19da-3090-de177f547120@illinois.edu> Was master (after the broker merge) previously working on this same machine? It works for me on 10.4-RELEASE. Maybe you could try "make distclean" and "git pull" and try again. On 7/10/18 2:03 PM, Aashish Sharma wrote: > Probably obvious but I am not very sure so asking here. > > I see this error trying to build current master - Thoughts what am I missing ? > > (trying the build on: FreeBSD 10.3-STABLE) > > I am building as: > > ./configure --prefix=/usr/local/bro-master && make > > > > .. > .. > > gmake[6]: Leaving directory '/usr/local/src/master/build/aux/broker/caf-build' > gmake[6]: Entering directory '/usr/local/src/master/build/aux/broker/caf-build' > [ 96%] Building CXX object libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/src/manager.cpp.o > clang: warning: argument unused during compilation: '-L/lib' > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:100:8: error: use of undeclared identifier 'get_or' > if (!get_or(config(), "middleman.attach-utility-actors", false)) > ^ > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:102:14: error: calling a private constructor of class 'caf::typed_actor, unsigned short, caf::intrusive_ptr, > std::__1::set, std::__1::less >, std::__1::allocator > >, std::__1::basic_string, bool>, caf::output_tuple >, caf::typed_mpi, unsigned short, > std::__1::basic_string, bool>, caf::output_tuple >, caf::typed_mpi, std::__1::basic_string, unsigned short>, caf::output_tuple, > std::__1::set, std::__1::less >, std::__1::allocator > > > >, caf::typed_mpi, caf::actor_addr, unsigned short>, caf::output_tuple >, > caf::typed_mpi, unsigned short>, caf::output_tuple >, caf::typed_mpi, caf::node_id, std::__1::basic_string, caf::message, std::__1::set, > std::__1::less >, std::__1::allocator > > >, caf::output_tuple > >, caf::typed_mpi, caf::node_id>, caf::output_tuple std::__1::basic_string, unsigned short> > >' > manager_ = nullptr; > ^ > /usr/local/include/caf/typed_actor.hpp:270:3: note: declared private here > typed_actor(actor_control_block* ptr) : ptr_(ptr) { > ^ > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:112:27: error: no member named 'openssl_certificate' in 'caf::actor_system_config' > if (system().config().openssl_certificate.size() == 0) > ~~~~~~~~~~~~~~~~~ ^ > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:114:27: error: no member named 'openssl_key' in 'caf::actor_system_config' > if (system().config().openssl_key.size() == 0) > ~~~~~~~~~~~~~~~~~ ^ > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:131:10: error: use of undeclared identifier 'openssl_manager'; did you mean 'opencl_manager'? > return openssl_manager; > ^~~~~~~~~~~~~~~ > opencl_manager > /usr/local/include/caf/actor_system.hpp:140:7: note: 'opencl_manager' declared here > opencl_manager, > ^ > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:140:14: error: no member named 'openssl_certificate' in 'caf::actor_system_config' > return cfg.openssl_certificate.size() > 0 || cfg.openssl_key.size() > 0 > ~~~ ^ > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:140:52: error: no member named 'openssl_key' in 'caf::actor_system_config' > return cfg.openssl_certificate.size() > 0 || cfg.openssl_key.size() > 0 > ~~~ ^ > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:141:17: error: no member named 'openssl_passphrase' in 'caf::actor_system_config' > || cfg.openssl_passphrase.size() > 0 || cfg.openssl_capath.size() > 0 > ~~~ ^ > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:141:54: error: no member named 'openssl_capath' in 'caf::actor_system_config' > || cfg.openssl_passphrase.size() > 0 || cfg.openssl_capath.size() > 0 > ~~~ ^ > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:142:17: error: no member named 'openssl_cafile' in 'caf::actor_system_config' > || cfg.openssl_cafile.size() > 0; > ~~~ ^ > 10 errors generated. > gmake[6]: *** [libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/build.make:63: libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/src/manager.cpp.o] Error 1 > gmake[6]: Leaving directory '/usr/local/src/master/build/aux/broker/caf-build' > gmake[5]: *** [CMakeFiles/Makefile2:394: libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/all] Error 2 > gmake[5]: Leaving directory '/usr/local/src/master/build/aux/broker/caf-build' > gmake[4]: *** [Makefile:128: all] Error 2 > gmake[4]: Leaving directory '/usr/local/src/master/build/aux/broker/caf-build' > *** Error code 2 > > Stop. > make[3]: stopped in /usr/local/src/master/build > *** Error code 1 > > Stop. > make[2]: stopped in /usr/local/src/master/build > *** Error code 1 > > Stop. > make[1]: stopped in /usr/local/src/master/build > *** Error code 1 > > Stop. > make: stopped in /usr/local/src/master > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From asharma at lbl.gov Tue Jul 10 13:33:32 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 10 Jul 2018 13:33:32 -0700 Subject: [Bro-Dev] error compiling master In-Reply-To: <9b303e0d-c85b-19da-3090-de177f547120@illinois.edu> References: <20180710190310.GA53027@MacPro-2331.local> <9b303e0d-c85b-19da-3090-de177f547120@illinois.edu> Message-ID: <20180710203332.GD53027@MacPro-2331.local> HI Daniel, > Was master (after the broker merge) previously working on this same machine? Nah! just building from scratch first time. Previously I was/am running bro-2.5.3 on this host. Same error when tried on another 10.3-STABLE release. Aashish On Tue, Jul 10, 2018 at 03:24:34PM -0500, Daniel Thayer wrote: > Was master (after the broker merge) previously working on this same machine? > > It works for me on 10.4-RELEASE. Maybe you could try "make distclean" > and "git pull" and try again. > > > On 7/10/18 2:03 PM, Aashish Sharma wrote: > > Probably obvious but I am not very sure so asking here. > > > > I see this error trying to build current master - Thoughts what am I missing ? > > > > (trying the build on: FreeBSD 10.3-STABLE) > > > > I am building as: > > > > ./configure --prefix=/usr/local/bro-master && make > > > > > > > > .. > > .. > > > > gmake[6]: Leaving directory '/usr/local/src/master/build/aux/broker/caf-build' > > gmake[6]: Entering directory '/usr/local/src/master/build/aux/broker/caf-build' > > [ 96%] Building CXX object libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/src/manager.cpp.o > > clang: warning: argument unused during compilation: '-L/lib' > > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:100:8: error: use of undeclared identifier 'get_or' > > if (!get_or(config(), "middleman.attach-utility-actors", false)) > > ^ > > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:102:14: error: calling a private constructor of class 'caf::typed_actor, unsigned short, caf::intrusive_ptr, > > std::__1::set, std::__1::less >, std::__1::allocator > >, std::__1::basic_string, bool>, caf::output_tuple >, caf::typed_mpi, unsigned short, > > std::__1::basic_string, bool>, caf::output_tuple >, caf::typed_mpi, std::__1::basic_string, unsigned short>, caf::output_tuple, > > std::__1::set, std::__1::less >, std::__1::allocator > > > >, caf::typed_mpi, caf::actor_addr, unsigned short>, caf::output_tuple >, > > caf::typed_mpi, unsigned short>, caf::output_tuple >, caf::typed_mpi, caf::node_id, std::__1::basic_string, caf::message, std::__1::set, > > std::__1::less >, std::__1::allocator > > >, caf::output_tuple > >, caf::typed_mpi, caf::node_id>, caf::output_tuple > std::__1::basic_string, unsigned short> > >' > > manager_ = nullptr; > > ^ > > /usr/local/include/caf/typed_actor.hpp:270:3: note: declared private here > > typed_actor(actor_control_block* ptr) : ptr_(ptr) { > > ^ > > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:112:27: error: no member named 'openssl_certificate' in 'caf::actor_system_config' > > if (system().config().openssl_certificate.size() == 0) > > ~~~~~~~~~~~~~~~~~ ^ > > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:114:27: error: no member named 'openssl_key' in 'caf::actor_system_config' > > if (system().config().openssl_key.size() == 0) > > ~~~~~~~~~~~~~~~~~ ^ > > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:131:10: error: use of undeclared identifier 'openssl_manager'; did you mean 'opencl_manager'? > > return openssl_manager; > > ^~~~~~~~~~~~~~~ > > opencl_manager > > /usr/local/include/caf/actor_system.hpp:140:7: note: 'opencl_manager' declared here > > opencl_manager, > > ^ > > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:140:14: error: no member named 'openssl_certificate' in 'caf::actor_system_config' > > return cfg.openssl_certificate.size() > 0 || cfg.openssl_key.size() > 0 > > ~~~ ^ > > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:140:52: error: no member named 'openssl_key' in 'caf::actor_system_config' > > return cfg.openssl_certificate.size() > 0 || cfg.openssl_key.size() > 0 > > ~~~ ^ > > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:141:17: error: no member named 'openssl_passphrase' in 'caf::actor_system_config' > > || cfg.openssl_passphrase.size() > 0 || cfg.openssl_capath.size() > 0 > > ~~~ ^ > > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:141:54: error: no member named 'openssl_capath' in 'caf::actor_system_config' > > || cfg.openssl_passphrase.size() > 0 || cfg.openssl_capath.size() > 0 > > ~~~ ^ > > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:142:17: error: no member named 'openssl_cafile' in 'caf::actor_system_config' > > || cfg.openssl_cafile.size() > 0; > > ~~~ ^ > > 10 errors generated. > > gmake[6]: *** [libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/build.make:63: libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/src/manager.cpp.o] Error 1 > > gmake[6]: Leaving directory '/usr/local/src/master/build/aux/broker/caf-build' > > gmake[5]: *** [CMakeFiles/Makefile2:394: libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/all] Error 2 > > gmake[5]: Leaving directory '/usr/local/src/master/build/aux/broker/caf-build' > > gmake[4]: *** [Makefile:128: all] Error 2 > > gmake[4]: Leaving directory '/usr/local/src/master/build/aux/broker/caf-build' > > *** Error code 2 > > > > Stop. > > make[3]: stopped in /usr/local/src/master/build > > *** Error code 1 > > > > Stop. > > make[2]: stopped in /usr/local/src/master/build > > *** Error code 1 > > > > Stop. > > make[1]: stopped in /usr/local/src/master/build > > *** Error code 1 > > > > Stop. > > make: stopped in /usr/local/src/master > > _______________________________________________ > > bro-dev mailing list > > bro-dev at bro.org > > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > > > From jsiwek at corelight.com Tue Jul 10 16:02:51 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Tue, 10 Jul 2018 18:02:51 -0500 Subject: [Bro-Dev] error compiling master In-Reply-To: <20180710190310.GA53027@MacPro-2331.local> References: <20180710190310.GA53027@MacPro-2331.local> Message-ID: On Tue, Jul 10, 2018 at 2:10 PM Aashish Sharma wrote: > [ 96%] Building CXX object libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/src/manager.cpp.o > clang: warning: argument unused during compilation: '-L/lib' > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:100:8: error: use of undeclared identifier 'get_or' > if (!get_or(config(), "middleman.attach-utility-actors", false)) > ^ This is at least an odd error if it were actually finding the CAF headers that are embedded in the Bro source tree, so a guess is that you have an old version of CAF (AKA actor-framework or libcaf) installed in a system directory and it is finding those instead. Can you check? A specific header file in question here would be "caf/config_value.hpp". And for reference, building on a fresh FreeBSD 10.3 system + bro/master checkout worked for me just now, which may also suggest a problem in the system setup (or else an inconsistent Bro checkout/configuration, but it seems like we are both starting from a fresh clone). - Jon From asharma at lbl.gov Tue Jul 10 16:10:41 2018 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 10 Jul 2018 16:10:41 -0700 Subject: [Bro-Dev] error compiling master In-Reply-To: References: <20180710190310.GA53027@MacPro-2331.local> Message-ID: <20180710231040.GU53027@MacPro-2331.local> Jon, > you have an old version of CAF (AKA actor-framework or libcaf) Yes, that was the issue. I didn't realize I already had caf on this from Jan 2017. Thank you. root at box:# pkg info |fgrep caf caf-0.15.3 C++ actor framework Removing this, lets me build bro now. Thanks for the pointer. Aashish On Tue, Jul 10, 2018 at 06:02:51PM -0500, Jon Siwek wrote: > On Tue, Jul 10, 2018 at 2:10 PM Aashish Sharma wrote: > > > [ 96%] Building CXX object libcaf_openssl/CMakeFiles/libcaf_openssl_shared.dir/src/manager.cpp.o > > clang: warning: argument unused during compilation: '-L/lib' > > /usr/local/src/master/aux/broker/3rdparty/caf/libcaf_openssl/src/manager.cpp:100:8: error: use of undeclared identifier 'get_or' > > if (!get_or(config(), "middleman.attach-utility-actors", false)) > > ^ > > This is at least an odd error if it were actually finding the CAF > headers that are embedded in the Bro source tree, so a guess is that > you have an old version of CAF (AKA actor-framework or libcaf) > installed in a system directory and it is finding those instead. Can > you check? A specific header file in question here would be > "caf/config_value.hpp". > > And for reference, building on a fresh FreeBSD 10.3 system + > bro/master checkout worked for me just now, which may also suggest a > problem in the system setup (or else an inconsistent Bro > checkout/configuration, but it seems like we are both starting from a > fresh clone). > > - Jon From robin at corelight.com Fri Jul 27 10:39:29 2018 From: robin at corelight.com (Robin Sommer) Date: Fri, 27 Jul 2018 10:39:29 -0700 Subject: [Bro-Dev] Broker::publish API Message-ID: <20180727173929.GA6404@corelight.com> The other day when merging Johanna's code to clusterize the configuration framework, I noticed this code in there: # [Send id=val to everyone else] Broker::publish(change_topic, Config::cluster_set_option, ID, val, location); if ( Cluster::local_node_type() != Cluster::MANAGER ) Broker::relay(change_topic, change_topic, Config::cluster_set_option, ID, val, location); It took me a bit to understand that ... The goal here is that a change in a configuration value gets propagated out to all nodes in the cluster. The Broker::publish() sends it to a node's immediate neighbors, but not further. That means that for workers it goes (only) to their manager; for the manager it means, it goes to all workers. If we're not a manager, we then separately (through Broker::relay()) ask our neighbors (that's the manager) to forward the change to *their* neighbors (that's the other workers), without reraising it locally. I remember we have discussed this API before, but I wanted to bring it up again as I keep finding it confusing. I believe the code above could be simplified by using the newer Broker::publish_and_relay(), which was added to combine the two operations. Still, I'm realizing now that I don't like thinking about this in terms of separate publishing and relaying operations. It all won't become easier once we add multi-hop routing to the mix (which is in the works). And on top of all that, we also have Cluster::publish_rr, Cluster::publish_hew, Cluster::relay_rr, and Cluster::relay_hew -- another set of separate publishing & relay options. I'm wondering if we should give it another try to simply this API while we still can (i.e., before 2.6 goes out). To me, the most intuitive publish operation is "send to topic T and propagate to everybody subscribed to that topic". I'd structure the API around that, making that the main publish function for that simply: Broker::publish(topic, args); That would send to all neighbors, which then process locally and relay to their neighbors. Right now, that would propagate just across one hop but once we have multihop that'd start being broadcasted out broadly. To support the other use cases, we can then add modifiers & functions to tweak this default, e.g.: - Give publish() another argument "relay: bool &default=T" to prevent it from going beyond the immediate receiver. Or maybe instead: "relay_hops: int &default=-1" to specify the max number of hops to relay across, with -1 meaning no limit. (I recall concerns about loops being too easy to create; we could set the default here to F/0 to default to no forwarding, although conceptually I don't really like that :-) - Give publish() another argument "relay_topic: string &default="" to change the topic when relaying on the 1st hop. - Give publish() another argument "process_on_relays: bool &default=T" to change whether a relaying hop also sees the event locally. - Add a second function publish_pool() that has all the same options, but receives a pool type instead of a topic (just an enum: RR, HRW). What I'm not quite sure about is if some of these modifiers are better to leave for the receiver to specify (e.g., whether to raise events received on a given topic locally, or just forward). I think I can see that either way. Robin -- Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com From jsiwek at corelight.com Fri Jul 27 12:47:46 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 27 Jul 2018 14:47:46 -0500 Subject: [Bro-Dev] Broker::publish API In-Reply-To: <20180727173929.GA6404@corelight.com> References: <20180727173929.GA6404@corelight.com> Message-ID: On Fri, Jul 27, 2018 at 12:40 PM Robin Sommer wrote: > I'm wondering if we should give it another try to simply this API > while we still can (i.e., before 2.6 goes out). To me, the most > intuitive publish operation is "send to topic T and propagate to > everybody subscribed to that topic". I'd structure the API around > that, making that the main publish function for that simply: > > Broker::publish(topic, args); > > That would send to all neighbors, which then process locally and relay > to their neighbors. Right now, that would propagate just across one > hop but once we have multihop that'd start being broadcasted out > broadly. Can you remind/clarify what's meant by "multihop" ? I thought: Broker already has manual multihop if you set up subscriptions on all relevant nodes on the path yourself. Bro doesn't use it right now. Broker does not yet have automatic multihop where subscriptions are globally flooded automatically. A difference between "manual multihop" and "automatic multihop" would be that in the later, some relaying nodes may not actually hold a subscription to the message they are relaying and so, in the case of Bro events, I think they would not process them locally. > - Give publish() another argument "relay: bool &default=T" to prevent > it from going beyond the immediate receiver. Or maybe instead: > "relay_hops: int &default=-1" to specify the max number of hops > to relay across, with -1 meaning no limit. Going with the generalized approach of configurable number of hops per message via "relay_hops" from the start would be better than finding out we need it later. Possibly a downside is now you need to store original hop limit in addition to current TTL in each message if you want to detect the "is 1st hop" condition for the "relay_topic" option below. > . (I recall concerns > about loops being too easy to create; we could set the default > here to F/0 to default to no forwarding, although conceptually I > don't really like that :-) It's maybe both a concern and a reality -- Bro clusters currently contain cycles (e.g. worker -> manager -> proxy -> worker) > - Give publish() another argument "relay_topic: string &default="" > to change the topic when relaying on the 1st hop. > > - Give publish() another argument "process_on_relays: bool &default=T" > to change whether a relaying hop also sees the event locally. Those seem fine to me. > - Add a second function publish_pool() that has all the same > options, but receives a pool type instead of a topic (just an > enum: RR, HRW). What's the goal of the enums instead of just publish_hrw() and publish_rr() ? Instead of the API being 2 functions, it then seems like 2 enums that are never used elsewhere + 1 function that now always branches internally. - Jon From jazoff at illinois.edu Fri Jul 27 13:48:21 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 27 Jul 2018 20:48:21 +0000 Subject: [Bro-Dev] Broker::publish API In-Reply-To: <20180727173929.GA6404@corelight.com> References: <20180727173929.GA6404@corelight.com> Message-ID: > On Jul 27, 2018, at 1:39 PM, Robin Sommer wrote: > > I'm wondering if we should give it another try to simply this API > while we still can (i.e., before 2.6 goes out). To me, the most > intuitive publish operation is "send to topic T and propagate to > everybody subscribed to that topic". I'd structure the API around > that, making that the main publish function for that simply: > > Broker::publish(topic, args); > > That would send to all neighbors, which then process locally and relay > to their neighbors. Right now, that would propagate just across one > hop but once we have multihop that'd start being broadcasted out > broadly. This would do weird things on workers, since they connect to both the manager and proxies. Worker 1 would send to it's neighbors [manager, proxy1, proxy2] but then those 3 nodes would relay to all of the other workers. The TTL would stop the propagation, but you'd still end up sending 3 copies of the same message to each worker. I do agree that there's room for a lot of simplification, for example a worker broadcasting a message efficiently to all other workers needs to do something like this from the docs: Cluster::relay_rr(Cluster::proxy_pool, "example_key", Cluster::worker_topic, worker_to_workers, Cluster::node + " (via a proxy)"); But a lot of that could have defaults: Most use cases would want to relay through the default proxy pool Since round robin is in use, they key shouldn't matter. The round robin part itself is really an implementation detail for proxy load balancing and maybe not something that should be exposed in the API. Now that I think of it I'm not sure why one would ever use relay_hrw over relay_rr. Removing a lot of that gets close to what you are suggesting: Cluster::relay(Cluster::worker_topic, worker_to_workers, Cluster::node + " (via a proxy)"); which is I guess just Cluster::relay(topic, args) like you said. ? Justin Azoff From jsiwek at corelight.com Fri Jul 27 15:10:40 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Fri, 27 Jul 2018 17:10:40 -0500 Subject: [Bro-Dev] Broker::publish API In-Reply-To: References: <20180727173929.GA6404@corelight.com> Message-ID: On Fri, Jul 27, 2018 at 3:55 PM Azoff, Justin S wrote: > I do agree that there's room for a lot of simplification, for example a worker broadcasting a message efficiently to all > other workers needs to do something like this from the docs: > > Cluster::relay_rr(Cluster::proxy_pool, "example_key", > Cluster::worker_topic, worker_to_workers, > Cluster::node + " (via a proxy)"); > > But a lot of that could have defaults: > > Most use cases would want to relay through the default proxy pool > Since round robin is in use, they key shouldn't matter. At the moment, one could write their own wrapper function around that if they find it too verbose and always want to use certain defaults? > The round robin part itself is really an implementation detail for proxy load balancing and maybe not something that > should be exposed in the API. Now that I think of it I'm not sure why one would ever use relay_hrw over relay_rr. Theoretically, a more favorable load distribution that's consistent over time? e.g. if you do RR of the same messaging pattern from multiple nodes, you could have waves of "randomly" overlapping loads on the relayer-node since everyone is cycling through all the proxies at their own rate when choosing the relayer. With HRW, you'd stick with the same relayer over time and only change on outages, but everyone should have chosen their relayer in a uniformly distributed fashion. - Jon From jazoff at illinois.edu Fri Jul 27 17:30:13 2018 From: jazoff at illinois.edu (Azoff, Justin S) Date: Sat, 28 Jul 2018 00:30:13 +0000 Subject: [Bro-Dev] Broker::publish API In-Reply-To: References: <20180727173929.GA6404@corelight.com> Message-ID: <073C00E9-8696-4387-A640-36A708A27EF6@illinois.edu> > On Jul 27, 2018, at 6:10 PM, Jon Siwek wrote: > > On Fri, Jul 27, 2018 at 3:55 PM Azoff, Justin S wrote: > >> I do agree that there's room for a lot of simplification, for example a worker broadcasting a message efficiently to all >> other workers needs to do something like this from the docs: >> >> Cluster::relay_rr(Cluster::proxy_pool, "example_key", >> Cluster::worker_topic, worker_to_workers, >> Cluster::node + " (via a proxy)"); >> >> But a lot of that could have defaults: >> >> Most use cases would want to relay through the default proxy pool >> Since round robin is in use, they key shouldn't matter. > > At the moment, one could write their own wrapper function around that > if they find it too verbose and always want to use certain defaults? Yeah.. The wrapper would be trivial.. Should bro include it so that the API scripts use is simpler? >> The round robin part itself is really an implementation detail for proxy load balancing and maybe not something that >> should be exposed in the API. Now that I think of it I'm not sure why one would ever use relay_hrw over relay_rr. > > Theoretically, a more favorable load distribution that's consistent > over time? e.g. if you do RR of the same messaging pattern from > multiple nodes, you could have waves of "randomly" overlapping loads > on the relayer-node since everyone is cycling through all the proxies > at their own rate when choosing the relayer. With HRW, you'd stick > with the same relayer over time and only change on outages, but > everyone should have chosen their relayer in a uniformly distributed > fashion. > > - Jon I'd expect that round robin would give the most uniform load distribution, for N proxies each proxy would see 1/N relay messages, but I guess in general round robin isn't the best load balance mechanism since it doesn't take into account the responsiveness of each proxy. With some of the information CAF provides it may be possible to also support weighted round robin. That way if a proxy node doesn't die outright but starts having issues for one reason or another, relay_rr could avoid sending it messages. ? Justin Azoff From robin at corelight.com Mon Jul 30 09:01:01 2018 From: robin at corelight.com (Robin Sommer) Date: Mon, 30 Jul 2018 09:01:01 -0700 Subject: [Bro-Dev] Broker::publish API In-Reply-To: Message-ID: <20180730160101.GE43154@corelight.com> On Fri, Jul 27, 2018 at 14:47 -0500, Jonathan Siwek wrote: > Broker does not yet have automatic multihop where subscriptions are > globally flooded automatically. Yep, that's what I meant: dynamic multihop where each node tracks what its peers are subscribing to, and forwards messages independent of its own subscriptions. > Possibly a downside is now you need to store original hop limit in > addition to current TTL in each message if you want to detect the "is > 1st hop" condition for the "relay_topic" option below. Yeah, that's right. Actually I think ideally the 1st hop wouldn't have any special role anyways if we didn't need that "relay_topic". > It's maybe both a concern and a reality -- Bro clusters currently > contain cycles (e.g. worker -> manager -> proxy -> worker) True, although it's not cycles in the connection topology that matter, it's cycles in topic subscriptions. I need to think about this a bit more (and I need to remind myself how our topics currently look like) but could we set up topics so that even in a cluster, messages don't go into a cycle? Is there a summary somewhere of what events & topics the cluster nodes are currently exchanging? > > - Add a second function publish_pool() that has all the same > > options, but receives a pool type instead of a topic (just an > > enum: RR, HRW). > > What's the goal of the enums instead of just publish_hrw() and publish_rr() ? Similar to what Justin wrote, it would more directly express the intent, with less emphasis on the mechanism; we could set a default to whatever we recommend people normally use; and it'd be more extensible. > At the moment, one could write their own wrapper function around that > if they find it too verbose and always want to use certain defaults? They could, but my general point is that it'd be nice to have a simple API that covers the most common uses cases directly and intuitively. Then let people change defaults if they have to and know what they are doing. Robin -- Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com From jsiwek at corelight.com Mon Jul 30 09:01:51 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 30 Jul 2018 11:01:51 -0500 Subject: [Bro-Dev] Broker::publish API In-Reply-To: <073C00E9-8696-4387-A640-36A708A27EF6@illinois.edu> References: <20180727173929.GA6404@corelight.com> <073C00E9-8696-4387-A640-36A708A27EF6@illinois.edu> Message-ID: On Fri, Jul 27, 2018 at 7:30 PM Azoff, Justin S wrote: > > > > On Jul 27, 2018, at 6:10 PM, Jon Siwek wrote: > > > > On Fri, Jul 27, 2018 at 3:55 PM Azoff, Justin S wrote: > > > >> I do agree that there's room for a lot of simplification, for example a worker broadcasting a message efficiently to all > >> other workers needs to do something like this from the docs: > >> > >> Cluster::relay_rr(Cluster::proxy_pool, "example_key", > >> Cluster::worker_topic, worker_to_workers, > >> Cluster::node + " (via a proxy)"); > >> > >> But a lot of that could have defaults: > >> > >> Most use cases would want to relay through the default proxy pool > >> Since round robin is in use, they key shouldn't matter. > > > > At the moment, one could write their own wrapper function around that > > if they find it too verbose and always want to use certain defaults? > > Yeah.. The wrapper would be trivial.. Should bro include it so that the API scripts use is simpler? Maybe. We can see how it fits in the mix of what Robin suggested: # Supports variadic args in place of Broker::Event. Broker::publish(topic: string, args: Broker::Event, relay_topic: string &default="", process_on_relayer: bool &default=F) # Supports variadic args in place of Broker::Event. Cluster::publish(pool: Cluster::pool, key: any, strategy: enum, args: Broker::Event, relay_topic: string &default="", process_on_relayer: bool &default=F) # Supports variadic args in place of Broker::Event. Use proxy pool and RR method w/ arbitrary, internal key by default. Cluster::publish_via_proxy(relay_topic: string, args: Broker::Event) That last one being the wrapper you're asking for. Also, I compressed the ideas of having a separate "relay: bool" / "relay_hops: int" and "relay_topic: string" args -- a non-empty relay topic implicitly means you want to enable relaying on the receiving node. Thinking more about original idea of giving the number of relay hops: it may be better to leave that until Broker multihop is more robust and allow it's automatic forwarding mechanisms to take care of those scenarios whereas a "relay" is a simple mechanism at the Bro application level (has it's own unique message format) that serves to aid load-balancing use-cases (rather than routing use-cases). - Jon From jsiwek at corelight.com Mon Jul 30 09:15:13 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 30 Jul 2018 11:15:13 -0500 Subject: [Bro-Dev] Broker::publish API In-Reply-To: <20180730160101.GE43154@corelight.com> References: <20180730160101.GE43154@corelight.com> Message-ID: On Mon, Jul 30, 2018 at 11:02 AM Robin Sommer wrote: > True, although it's not cycles in the connection topology that matter, > it's cycles in topic subscriptions. Right, good point. > I need to think about this a bit > more (and I need to remind myself how our topics currently look like) I think we just have the "broadcast_topic" to which all nodes subscribe, but not sure if there's more. > but could we set up topics so that even in a cluster, messages don't > go into a cycle? I don't see why not, but it takes planning and prudence on everyone's part (including users) to not break that rule. I'd be more comfortable if one could automate answering the question: "if I add a subscription to a given node in the network, will I create a cycle?". - Jon From robin at corelight.com Mon Jul 30 10:16:21 2018 From: robin at corelight.com (Robin Sommer) Date: Mon, 30 Jul 2018 10:16:21 -0700 Subject: [Bro-Dev] Broker::publish API In-Reply-To: References: <20180730160101.GE43154@corelight.com> Message-ID: <20180730171621.GC12319@corelight.com> On Mon, Jul 30, 2018 at 11:15 -0500, Jonathan Siwek wrote: > I don't see why not, but it takes planning and prudence on everyone's > part (including users) to not break that rule. Yeah, question is we can pre-configure the cluster so that user's don't need to worry about it most of the time. > I'd be more comfortable if one could automate answering the question: > "if I add a subscription to a given node in the network, will I create > a cycle?". Hmm ... What about a test mode where we'd spin up a dummy cluster (similar to what the bests do), have each node send a message to all subscribed topics, and watch for TTL violations? Robin -- Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com From jsiwek at corelight.com Mon Jul 30 11:30:30 2018 From: jsiwek at corelight.com (Jon Siwek) Date: Mon, 30 Jul 2018 13:30:30 -0500 Subject: [Bro-Dev] Broker::publish API In-Reply-To: <20180730171621.GC12319@corelight.com> References: <20180730160101.GE43154@corelight.com> <20180730171621.GC12319@corelight.com> Message-ID: On Mon, Jul 30, 2018 at 12:16 PM Robin Sommer wrote: > > I'd be more comfortable if one could automate answering the question: > > "if I add a subscription to a given node in the network, will I create > > a cycle?". > > Hmm ... What about a test mode where we'd spin up a dummy cluster > (similar to what the bests do), have each node send a message to all > subscribed topics, and watch for TTL violations? Seems clunky and could get dicey -- subscriptions that transient/dynamic may not be well-tested and you'd probably want to guarantee that sending such a dummy message actually does not result in any side-effects at the Bro-layer. If nodes start raising random events at unusual/unintended times I start to doubt the stability of things. - Jon From robin at corelight.com Mon Jul 30 11:35:26 2018 From: robin at corelight.com (Robin Sommer) Date: Mon, 30 Jul 2018 11:35:26 -0700 Subject: [Bro-Dev] Broker::publish API In-Reply-To: References: <20180730160101.GE43154@corelight.com> <20180730171621.GC12319@corelight.com> Message-ID: <20180730183526.GE12319@corelight.com> On Mon, Jul 30, 2018 at 13:30 -0500, Jonathan Siwek wrote: > Seems clunky and could get dicey Agreed. :) It'd just be a heuristic to catch some obvious errors. I don't think there's more we can do, we can't really catch loops statically by looking at the code. Robin -- Robin Sommer * Corelight, Inc. * robin at corelight.com * www.corelight.com