From noreply at bro.org Sun May 1 00:00:18 2016 From: noreply at bro.org (Merge Tracker) Date: Sun, 1 May 2016 00:00:18 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201605010700.u4170I4w018266@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1579 [1] Bro Johanna Amann - 2016-04-29 2.5 Normal Please merge topic/johanna/xmpp-starttls BIT-1571 [2] BroControl Adam Slagell - 2016-04-28 2.5 Low Connection summaries w/ IPv6 have poor readabiity BIT-1510 [3] BroControl Seth Hall Justin Azoff 2016-04-07 2.5 Normal Crash reports when no crash happened BIT-1507 [4] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ------------------------------------------------------- 373c872 [5] bro Daniel Thayer 2016-04-29 Fix a few incorrect type tags in Bro broker source code 362bf7a [6] bro Daniel Thayer 2016-04-27 Update docs and tests of the fmt() function 23d2562 [7] bro Seth Hall 2016-04-13 Revert "Fix RFB analyzer to build on FreeBSD" 16c0707 [8] bro Daniel Thayer 2016-04-13 Fix RFB analyzer to build on FreeBSD Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ----------------------------------------------------------------------- #52 [9] bro J-Gras [10] 2016-04-07 Fixed matching mail address intel [11] #22 [12] bro-plugins nickwallen [13] 2016-04-11 BIT-1559 Bro-Plugins Send each log stream to different kafka topic [14] #18 [15] bro-plugins jshlbrd [16] 2016-03-03 SSDP analyzer [17] [1] BIT-1579 https://bro-tracker.atlassian.net/browse/BIT-1579 [2] BIT-1571 https://bro-tracker.atlassian.net/browse/BIT-1571 [3] BIT-1510 https://bro-tracker.atlassian.net/browse/BIT-1510 [4] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [5] 373c872 https://github.com/bro/bro/commit/373c872e939f97c498b029cd08d4b24c0ab71c70 [6] 362bf7a https://github.com/bro/bro/commit/362bf7aee12814781ef97242accb176423cd2a64 [7] 23d2562 https://github.com/bro/bro/commit/23d25628ad9473f2a0faecafb1d6eb157a141673 [8] 16c0707 https://github.com/bro/bro/commit/16c0707b1d804ccfcc671fb9642a0c21ffd7219f [9] Pull Request #52 https://github.com/bro/bro/pull/52 [10] J-Gras https://github.com/J-Gras [11] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [12] Pull Request #22 https://github.com/bro/bro-plugins/pull/22 [13] nickwallen https://github.com/nickwallen [14] Merge Pull Request #22 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git support-many-kafka-topics [15] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [16] jshlbrd https://github.com/jshlbrd [17] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From noreply at bro.org Mon May 2 00:00:21 2016 From: noreply at bro.org (Merge Tracker) Date: Mon, 2 May 2016 00:00:21 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201605020700.u4270Lv8001762@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1579 [1] Bro Johanna Amann - 2016-04-29 2.5 Normal Please merge topic/johanna/xmpp-starttls BIT-1571 [2] BroControl Adam Slagell - 2016-04-28 2.5 Low Connection summaries w/ IPv6 have poor readabiity BIT-1510 [3] BroControl Seth Hall Justin Azoff 2016-04-07 2.5 Normal Crash reports when no crash happened BIT-1507 [4] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- ------------- ---------- ------------------------------------------------------- 373c872 [5] bro Daniel Thayer 2016-04-29 Fix a few incorrect type tags in Bro broker source code 362bf7a [6] bro Daniel Thayer 2016-04-27 Update docs and tests of the fmt() function 23d2562 [7] bro Seth Hall 2016-04-13 Revert "Fix RFB analyzer to build on FreeBSD" 16c0707 [8] bro Daniel Thayer 2016-04-13 Fix RFB analyzer to build on FreeBSD Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ----------------------------------------------------------------------- #52 [9] bro J-Gras [10] 2016-04-07 Fixed matching mail address intel [11] #22 [12] bro-plugins nickwallen [13] 2016-04-11 BIT-1559 Bro-Plugins Send each log stream to different kafka topic [14] #18 [15] bro-plugins jshlbrd [16] 2016-03-03 SSDP analyzer [17] [1] BIT-1579 https://bro-tracker.atlassian.net/browse/BIT-1579 [2] BIT-1571 https://bro-tracker.atlassian.net/browse/BIT-1571 [3] BIT-1510 https://bro-tracker.atlassian.net/browse/BIT-1510 [4] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [5] 373c872 https://github.com/bro/bro/commit/373c872e939f97c498b029cd08d4b24c0ab71c70 [6] 362bf7a https://github.com/bro/bro/commit/362bf7aee12814781ef97242accb176423cd2a64 [7] 23d2562 https://github.com/bro/bro/commit/23d25628ad9473f2a0faecafb1d6eb157a141673 [8] 16c0707 https://github.com/bro/bro/commit/16c0707b1d804ccfcc671fb9642a0c21ffd7219f [9] Pull Request #52 https://github.com/bro/bro/pull/52 [10] J-Gras https://github.com/J-Gras [11] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [12] Pull Request #22 https://github.com/bro/bro-plugins/pull/22 [13] nickwallen https://github.com/nickwallen [14] Merge Pull Request #22 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git support-many-kafka-topics [15] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [16] jshlbrd https://github.com/jshlbrd [17] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From jan.grashoefer at gmail.com Mon May 2 06:39:35 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Mon, 2 May 2016 15:39:35 +0200 Subject: [Bro-Dev] Opaque type in plugin In-Reply-To: <20160428184905.GO50101@icir.org> References: <16497b8e-6c14-554f-a940-093ade9bb326@gmail.com> <20160428184905.GO50101@icir.org> Message-ID: <04da9cda-c398-00ae-1051-cb4b5e9012f5@gmail.com> >> Is it possible to create a new opaque type for Bro using a dynamic >> plugin without touching Bro sources? > > Yes, it should. I believe if you implement the logic in your plugin's > InitPreScript() method, it should work. Meanwhile I gave it a try: I implemented a class inheriting from OpaqueVal and declared a global OpaqueType variable as extern. I initialized the type variable inside InitPreScript(). That compiled just fine but executing Bro failed due to dynamic linking issues (http://stackoverflow.com/questions/19373061 seems to explain that). I circumvented this issue by using a static variable (= limited scope) that I initialized in place. This went well and the new opaque value seems to behave as expected. However, I wonder if I am missing anything that might cause problems? One thing I am aware of is that I did not declare the value type as serializable. I think that is not needed for my use case. Besides it would require to define a corresponding identifier (see SerialTypes.h), whose uniqueness cannot be guaranteed for a plugin, right? Best regards Jan From robin at icir.org Mon May 2 07:51:18 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 2 May 2016 07:51:18 -0700 Subject: [Bro-Dev] Opaque type in plugin In-Reply-To: <04da9cda-c398-00ae-1051-cb4b5e9012f5@gmail.com> References: <16497b8e-6c14-554f-a940-093ade9bb326@gmail.com> <20160428184905.GO50101@icir.org> <04da9cda-c398-00ae-1051-cb4b5e9012f5@gmail.com> Message-ID: <20160502145118.GC92343@icir.org> On Mon, May 02, 2016 at 15:39 +0200, you wrote: > that I initialized in place. This went well and the new opaque value > seems to behave as expected. However, I wonder if I am missing anything > that might cause problems? Mind sharing your code (both before/after the change, and also the error message you got)? > One thing I am aware of is that I did not declare the value type as > serializable. Yeah, fine as long as you don't try to send it around or store on disk. > Besides it would require to define a corresponding identifier (see > SerialTypes.h), whose uniqueness cannot be guaranteed for a plugin, > right? Yeah, good point, there's no mechanism for that. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jan.grashoefer at gmail.com Mon May 2 10:01:34 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Mon, 2 May 2016 19:01:34 +0200 Subject: [Bro-Dev] Opaque type in plugin In-Reply-To: <20160502145118.GC92343@icir.org> References: <16497b8e-6c14-554f-a940-093ade9bb326@gmail.com> <20160428184905.GO50101@icir.org> <04da9cda-c398-00ae-1051-cb4b5e9012f5@gmail.com> <20160502145118.GC92343@icir.org> Message-ID: <49d0147c-0b79-e29c-8b59-b77eb385a5b2@gmail.com> >> that I initialized in place. This went well and the new opaque value >> seems to behave as expected. However, I wonder if I am missing anything >> that might cause problems? > > Mind sharing your code (both before/after the change, and also the > error message you got)? Sure. The code is available at https://github.com/J-Gras/bro-plugins/tree/topic/jgras/liblognorm/liblognorm. Unfortunately I don't have commits for both versions but the changes are minimal. The relevant files are src/LogNormalizer*. Currently LogNormalizer.cc:11 declares the type using static. That seems to work (as it is local). If I remove "static" and use the "extern" declaration commented in LogNormalizer.h:78 (like its done in Bro), loading the plugin fails with: fatal error in /home/jgras/devel/bro/scripts/base/init-bare.bro, line 1: cannot load plugin library /home/jgras/devel/bro/aux/plugins/liblognorm/build//lib/Bro-Lognorm.linux-x86_64.so: /home/jgras/devel/bro/aux/plugins/liblognorm/build//lib/Bro-Lognorm.linux-x86_64.so: undefined symbol: _ZN6plugin11Bro_Lognorm18lognormalizer_typeE >From what I understand this is expected as a dynamically linked global variable would need special treatment. But as I am not used to C++ and my understanding of Bro's type system is limited, I am not sure I missed important details in my solution. Best regards, Jan From jira at bro-tracker.atlassian.net Mon May 2 13:48:00 2016 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 2 May 2016 15:48:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1581) topic/seth/stats-improvement In-Reply-To: References: Message-ID: Seth Hall created BIT-1581: ------------------------------ Summary: topic/seth/stats-improvement Key: BIT-1581 URL: https://bro-tracker.atlassian.net/browse/BIT-1581 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Seth Hall This branch is ready to be merged. It makes the "misc/stats.bro" much more useful by expanding the data exposed by it to include a lot more data. It still writes to stats.log by default and is a normal Bro log (to avoid the problems traditionally encountered with the misc/profiling script). The focus is to only include data from simple counters to prevent this script from blocking the main thread since simple counters are quick and easy to collect. Notes: - The "resource_usage" bif is removed in favor of a larger number of bifs for collecting data. -- This message was sent by Atlassian JIRA (v1000.5.0#72002) From jira at bro-tracker.atlassian.net Mon May 2 13:48:00 2016 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 2 May 2016 15:48:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1581) topic/seth/stats-improvement In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1581: --------------------------- Status: Merge Request (was: Open) > topic/seth/stats-improvement > ---------------------------- > > Key: BIT-1581 > URL: https://bro-tracker.atlassian.net/browse/BIT-1581 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > > This branch is ready to be merged. It makes the "misc/stats.bro" much more useful by expanding the data exposed by it to include a lot more data. It still writes to stats.log by default and is a normal Bro log (to avoid the problems traditionally encountered with the misc/profiling script). The focus is to only include data from simple counters to prevent this script from blocking the main thread since simple counters are quick and easy to collect. > Notes: > - The "resource_usage" bif is removed in favor of a larger number of bifs for collecting data. -- This message was sent by Atlassian JIRA (v1000.5.0#72002) From jira at bro-tracker.atlassian.net Mon May 2 13:56:00 2016 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Mon, 2 May 2016 15:56:00 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1581) topic/seth/stats-improvement In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1581: --------------------------- Description: This branch is ready to be merged. It makes the "misc/stats.bro" much more useful by expanding the data exposed. It still writes to stats.log by default and is a normal Bro log (to avoid the problems traditionally encountered with the misc/profiling script). The focus is to only include data from simple counters to prevent this script from blocking the main thread since simple counters are quick and easy to collect. Notes: - The "resource_usage" bif is removed in favor of a larger number of bifs for collecting data. was: This branch is ready to be merged. It makes the "misc/stats.bro" much more useful by expanding the data exposed by it to include a lot more data. It still writes to stats.log by default and is a normal Bro log (to avoid the problems traditionally encountered with the misc/profiling script). The focus is to only include data from simple counters to prevent this script from blocking the main thread since simple counters are quick and easy to collect. Notes: - The "resource_usage" bif is removed in favor of a larger number of bifs for collecting data. > topic/seth/stats-improvement > ---------------------------- > > Key: BIT-1581 > URL: https://bro-tracker.atlassian.net/browse/BIT-1581 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: git/master > Reporter: Seth Hall > > This branch is ready to be merged. It makes the "misc/stats.bro" much more useful by expanding the data exposed. It still writes to stats.log by default and is a normal Bro log (to avoid the problems traditionally encountered with the misc/profiling script). The focus is to only include data from simple counters to prevent this script from blocking the main thread since simple counters are quick and easy to collect. > Notes: > - The "resource_usage" bif is removed in favor of a larger number of bifs for collecting data. -- This message was sent by Atlassian JIRA (v1000.5.0#72002) From vlad at grigorescu.org Mon May 2 14:03:41 2016 From: vlad at grigorescu.org (Vlad Grigorescu) Date: Mon, 2 May 2016 16:03:41 -0500 Subject: [Bro-Dev] which of these Lintian error messages need tickets? In-Reply-To: <7EFD7D614A2BB84ABEA19B2CEDD246580A73778A@CITESMBX5.ad.uillinois.edu> References: <7EFD7D614A2BB84ABEA19B2CEDD246580A73778A@CITESMBX5.ad.uillinois.edu> Message-ID: I'll take a shot: > *1. binary file built without LFS support* > binpac: > binary-file-built-without-LFS-support > > usr/bin/binpac > > bro (2.4.1+dfsg-2+b3; main): > binary-file-built-without-LFS-support > > usr/bin/bro > > bro-aux (0.35-1): > binary-file-built-without-LFS-support > > usr/bin/nfcollector > binpac - probably not. We should check to see what files bro-aux and bro are accessing with support for large files. > *2. binary without manpage* > binpac (0.44-1): > binary-without-manpage > > usr/bin/binpac > usr/bin/binpac > > btest (0.54-1): > binary-without-manpage > > usr/bin/btest > usr/bin/btest-ask-update > usr/bin/btest-bg-run > usr/bin/btest-bg-run-helper > usr/bin/btest-bg-wait > usr/bin/btest-diff > usr/bin/btest-diff-rst > usr/bin/btest-rst-cmd > usr/bin/btest-rst-include > usr/bin/btest-rst-pipe > usr/bin/btest-setsid > I think that these should be fixed. We really shouldn't be installing stuff in /usr/bin without manpages. I think we have most of this already documented, it'd just be a matter for formatting it the right way. > *3. hardening no bindnow* > binpac (0.44-1): > hardening-no-bindnow > > usr/bin/binpac > usr/bin/binpac > > bro (2.4.1+dfsg-2+b3; main): > hardening-no-bindnow > > usr/bin/bro > usr/bin/bro > > bro-aux (0.35-1): > hardening-no-bindnow > > usr/bin/adtrace > usr/bin/adtrace > usr/bin/bro-cut > usr/bin/bro-cut > usr/bin/ftwire2bro > usr/bin/ftwire2bro > usr/bin/nfcollector > usr/bin/nfcollector > usr/bin/rst > usr/bin/rst > > capstats (0.22-1): > hardening-no-bindnow > > usr/bin/capstats > usr/bin/capstats > This would probably be easy enough to add, though I'm not sure how useful it is. > *4. hardening no pie* > binpac (0.44-1): > hardening-no-pie > usr/bin/binpac > usr/bin/binpac > > bro (2.4.1+dfsg-2+b3; main): > hardening-no-pie > usr/bin/bro > usr/bin/bro > > bro-aux (0.35-1): > hardening-no-pie > usr/bin/adtrace > usr/bin/adtrace > usr/bin/bro-cut > usr/bin/bro-cut > usr/bin/ftwire2bro > usr/bin/ftwire2bro > usr/bin/nfcollector > usr/bin/nfcollector > usr/bin/rst > usr/bin/rst > > capstats (0.22-1): > hardening-no-pie > usr/bin/capstats > usr/bin/capstats > We have had a ticket about this, so it'd be nice to support ASLR with a configure option rather than forcing the user to override CFLAGS. > *5. no ctrl scripts* > binpac (0.44-1): > no-ctrl-scripts > > bro (2.4.1+dfsg-2+b3; main): > no-ctrl-scripts > > bro-common: > no-ctrl-scripts > > bro-aux (0.35-1): > no-ctrl-scripts > > capstats (0.22-1): > no-ctrl-scripts > I don't really understand this. > *6. static library has unneeded section* > binpac (0.44-1): > static-library-has-unneeded-section > > usr/lib/libbinpac.a(binpac_buffer.cc.o) .comment > usr/lib/libbinpac.a(binpac_buffer.cc.o) .comment > usr/lib/libbinpac.a(binpac_bytestring.cc.o) .comment > usr/lib/libbinpac.a(binpac_bytestring.cc.o) .comment > usr/lib/libbinpac.a(binpac_regex.cc.o) .comment > usr/lib/libbinpac.a(binpac_regex.cc.o) .comment > Probably would be easy enough to remove. > *7. unused override* > bro (2.4.1+dfsg-2+b3; main): > unused-override > description-starts-with-package-name > I think this is on the maintainer of the package. > *8. extended description is probably too short* > bro-common: > extended-description-is-probably-too-short > > > *9. ctrl script* (is this really an error? it doesn't seem like one) > broctl (1.4-1): > ctrl-script > postinst > prerm > > btest (0.54-1): > ctrl-script > postinst > prerm > > *10. vcs field uses insecure uri* > trace-summary (0.84-1): > vcs-field-uses-insecure-uri > > vcs-browser http://anonscm.debian.org/cgit/collab-maint/trace-summary.git > vcs-git git://anonscm.debian.org/collab-maint/trace-summary.git > These are out of our control, I believe. --Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160502/dcd6f1cc/attachment-0001.html From noreply at bro.org Tue May 3 00:00:19 2016 From: noreply at bro.org (Merge Tracker) Date: Tue, 3 May 2016 00:00:19 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201605030700.u4370JNW002614@bro-ids.icir.org> Open Merge Requests =================== ID Component Reporter Assignee Updated For Version Priority Summary ------------ ----------- -------------- ------------ ---------- ------------- ---------- ------------------------------------------------------ BIT-1581 [1] Bro Seth Hall - 2016-05-02 - Normal topic/seth/stats-improvement [2] BIT-1579 [3] Bro Johanna Amann - 2016-04-29 2.5 Normal Please merge topic/johanna/xmpp-starttls BIT-1571 [4] BroControl Adam Slagell - 2016-04-28 2.5 Low Connection summaries w/ IPv6 have poor readabiity BIT-1510 [5] BroControl Seth Hall Justin Azoff 2016-04-07 2.5 Normal Crash reports when no crash happened BIT-1507 [6] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open Fastpath Commits ====================== Commit Component Author Date Summary ------------ ----------- ------------- ---------- ------------------------------------------------------- 373c872 [7] bro Daniel Thayer 2016-04-29 Fix a few incorrect type tags in Bro broker source code 362bf7a [8] bro Daniel Thayer 2016-04-27 Update docs and tests of the fmt() function 23d2562 [9] bro Seth Hall 2016-04-13 Revert "Fix RFB analyzer to build on FreeBSD" 16c0707 [10] bro Daniel Thayer 2016-04-13 Fix RFB analyzer to build on FreeBSD Open GitHub Pull Requests ========================= Issue Component User Updated Title -------- ----------- --------------- ---------- ----------------------------------------------------------------------- #52 [11] bro J-Gras [12] 2016-04-07 Fixed matching mail address intel [13] #22 [14] bro-plugins nickwallen [15] 2016-04-11 BIT-1559 Bro-Plugins Send each log stream to different kafka topic [16] #18 [17] bro-plugins jshlbrd [18] 2016-03-03 SSDP analyzer [19] [1] BIT-1581 https://bro-tracker.atlassian.net/browse/BIT-1581 [2] stats-improvement https://github.com/bro/bro/tree/topic/seth/stats-improvement [3] BIT-1579 https://bro-tracker.atlassian.net/browse/BIT-1579 [4] BIT-1571 https://bro-tracker.atlassian.net/browse/BIT-1571 [5] BIT-1510 https://bro-tracker.atlassian.net/browse/BIT-1510 [6] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [7] 373c872 https://github.com/bro/bro/commit/373c872e939f97c498b029cd08d4b24c0ab71c70 [8] 362bf7a https://github.com/bro/bro/commit/362bf7aee12814781ef97242accb176423cd2a64 [9] 23d2562 https://github.com/bro/bro/commit/23d25628ad9473f2a0faecafb1d6eb157a141673 [10] 16c0707 https://github.com/bro/bro/commit/16c0707b1d804ccfcc671fb9642a0c21ffd7219f [11] Pull Request #52 https://github.com/bro/bro/pull/52 [12] J-Gras https://github.com/J-Gras [13] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [14] Pull Request #22 https://github.com/bro/bro-plugins/pull/22 [15] nickwallen https://github.com/nickwallen [16] Merge Pull Request #22 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git support-many-kafka-topics [17] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [18] jshlbrd https://github.com/jshlbrd [19] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp From asharma at lbl.gov Tue May 3 00:25:57 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Tue, 3 May 2016 00:25:57 -0700 Subject: [Bro-Dev] bloomfilter_counting_init parameterization ? In-Reply-To: <20160411152637.GH1342@shogun> References: <20160411064721.GA74290@mac.local> <20160411152637.GH1342@shogun> Message-ID: <20160503072555.GF84792@mac.local> So I am trying to use bloomfilter_counting_init for keeping a count of uniq IPs seen within a subnet and instead of relying on a table or a set, I was toying with an idea of using bloomfilter_counting_init. However, I am not clear on the parameterization below: global bloomfilter_counting_init: function(k: count , cells: count , max: count , name: string &default=""): opaque of bloomfilter ; What should be the length of the cells for storing 65536 IPs ? Is k=3 a good value or I need something else ? Could someone elaborate on how to decide these parameters. I looked at /btest/bifs/bloomfilter.bro but not quite clear. thanks, Aashish On Mon, Apr 11, 2016 at 08:26:37AM -0700, Matthias Vallentin wrote: From vallentin at icir.org Tue May 3 07:18:24 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 3 May 2016 07:18:24 -0700 Subject: [Bro-Dev] bloomfilter_counting_init parameterization ? In-Reply-To: <20160503072555.GF84792@mac.local> References: <20160411064721.GA74290@mac.local> <20160411152637.GH1342@shogun> <20160503072555.GF84792@mac.local> Message-ID: <20160503141824.GI72836@shogun> > global bloomfilter_counting_init: function(k: count , cells: count , > max: count , name: string &default=""): opaque of bloomfilter ; The counting Bloomfilter is very similar to a regular Bloom filter, except that the underlying bit vector now consists of "cells," i.e., sequences of bits that represents a counter. With 4 bits per cells, you can count from 0 to 2^4-1 = 15. Consider this scenario: +----+----+----+----+----+ |0101|0000|0010|0000|0010| +----+----+----+----+----+ In terms of variable names of the function signature, we have cells=5 and max=15. The number of hash functions to use, k, is independent of the cell sizing. The first cell has counter value 5, the third and fifth a value of 2. If you have k=3 hash functions and your hash function yields those indexes, the counting Bloom filter computes the minimum over all 3 cell values (=2) and returns that as the counter value. Unlike for the basic Bloom filter, there is no closed-form solution to the Bloom Error, i.e., the probability of a false positive occurring. That's why the interface is a bit more low-level. > What should be the length of the cells for storing 65536 IPs ? This really depends on two things: (1) the maximum counter value you need, and (2) your memory constrains. The counting Bloom filter is a bit more wasteful, especially for heavy-tailed distributions, which are very common in network traffic. Matthias From robin at icir.org Tue May 3 09:28:50 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 3 May 2016 09:28:50 -0700 Subject: [Bro-Dev] Less noise on bro-dev Message-ID: <20160503162850.GB83074@icir.org> In the interest of removing noise on this list, we've changed the destination for issue tracker notifications: they now to go to the bro-commits list. Feel free to subscribe at http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits With that change, this list will focus on discussion of development topics; no more automated postings here. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Fri May 6 08:30:22 2016 From: robin at icir.org (Robin Sommer) Date: Fri, 6 May 2016 08:30:22 -0700 Subject: [Bro-Dev] Opaque type in plugin In-Reply-To: <49d0147c-0b79-e29c-8b59-b77eb385a5b2@gmail.com> References: <16497b8e-6c14-554f-a940-093ade9bb326@gmail.com> <20160428184905.GO50101@icir.org> <04da9cda-c398-00ae-1051-cb4b5e9012f5@gmail.com> <20160502145118.GC92343@icir.org> <49d0147c-0b79-e29c-8b59-b77eb385a5b2@gmail.com> Message-ID: <20160506153022.GD61305@icir.org> On Mon, May 02, 2016 at 19:01 +0200, you wrote: > Currently LogNormalizer.cc:11 declares the type using static. That seems > to work (as it is local). If I remove "static" and use the "extern" > declaration commented in LogNormalizer.h:78 (like its done in Bro), > loading the plugin fails with: Took me a bit to look at this, but I've tried it now. My guess is that it was a namespace issue. Look at the attached patch, that version works for me. That said, your version using "static" is completely fine. Indeed, it's even the better one for the plugin case: defining it globally non-static is useful only if other object files need to access it. Inside your plugin, that's not the case; and outside the plugin nobody can (because nobody can rely on the plugin being present). So I think your solution is just fine. Good to see you got the type working from a plugin. Would be even nicer if Bro actually showed the new type in the output of "-NN" but we don't have the support in place yet for a plugin to register it accordingly. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin -------------- next part -------------- diff --git a/liblognorm/src/LogNormalizer.cc b/liblognorm/src/LogNormalizer.cc index d713640..4e9029e 100644 --- a/liblognorm/src/LogNormalizer.cc +++ b/liblognorm/src/LogNormalizer.cc @@ -8,7 +8,7 @@ extern "C" { using namespace plugin::Bro_Lognorm; -static OpaqueType* lognormalizer_type = new OpaqueType("lognormalizer"); +OpaqueType* plugin::Bro_Lognorm::lognormalizer_type = new OpaqueType("lognormalizer"); LogNormalizer::LogNormalizer(EventHandlerPtr evt_unparsed) : evt_unparsed(evt_unparsed) { diff --git a/liblognorm/src/LogNormalizer.h b/liblognorm/src/LogNormalizer.h index 0361f1f..a9b6fa4 100644 --- a/liblognorm/src/LogNormalizer.h +++ b/liblognorm/src/LogNormalizer.h @@ -13,6 +13,7 @@ extern "C" { namespace plugin { namespace Bro_Lognorm { +extern OpaqueType* lognormalizer_type; typedef std::map FieldList; From jan.grashoefer at gmail.com Fri May 6 15:39:54 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Sat, 7 May 2016 00:39:54 +0200 Subject: [Bro-Dev] Opaque type in plugin In-Reply-To: <20160506153022.GD61305@icir.org> References: <16497b8e-6c14-554f-a940-093ade9bb326@gmail.com> <20160428184905.GO50101@icir.org> <04da9cda-c398-00ae-1051-cb4b5e9012f5@gmail.com> <20160502145118.GC92343@icir.org> <49d0147c-0b79-e29c-8b59-b77eb385a5b2@gmail.com> <20160506153022.GD61305@icir.org> Message-ID: <0b156bd3-64eb-a767-a0bb-f7f835f5108e@gmail.com> > Took me a bit to look at this, but I've tried it now. My guess is that > it was a namespace issue. Look at the attached patch, that version > works for me. Thanks a lot for taking a look! I missed that "using" does not do the trick for a definition. I guess its time to read a C++ book :) > That said, your version using "static" is completely fine. Indeed, > it's even the better one for the plugin case: defining it globally > non-static is useful only if other object files need to access it. > Inside your plugin, that's not the case; and outside the plugin nobody > can (because nobody can rely on the plugin being present). So I think > your solution is just fine. I am happy to hear that this is a proper solution. My fear was that I was missing a mechanism, which announces the type to Bro. My thought was that in this case the limited visibility might become an issue. Big thanks for your support! Jan From robin at icir.org Fri May 6 17:56:24 2016 From: robin at icir.org (Robin Sommer) Date: Fri, 6 May 2016 17:56:24 -0700 Subject: [Bro-Dev] 2.5 Roadmap Message-ID: <20160507005624.GI61305@icir.org> To summarize a team discussion today, we decided that we'll start wrapping up development for Bro 2.5. There's a lot of new functionality in git master now, and it will be good to get that all into general use. The main piece missing from our original 2.5 roadmap is porting the Bro Cluster to Broker. While much of that work is in place already in a development branch, we are going to postpone integration for now and will instead merge it at the beginning of the upcoming 2.6 development cycle. That way we will have more time for testing stability and performance of that new cluster setup. For 2.5, we're still targetting a few more pieces that aren't yet in master, including: - Fixing the flare code to improve performance. - New SMB and NTP analyzers. - A simpler Broker API. Furthermore, if there are any tickets pending that you'd like Bro 2.5 to address, now would be a good time to point them out. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jan.grashoefer at gmail.com Sat May 7 01:36:46 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Sat, 7 May 2016 10:36:46 +0200 Subject: [Bro-Dev] 2.5 Roadmap In-Reply-To: <20160507005624.GI61305@icir.org> References: <20160507005624.GI61305@icir.org> Message-ID: <327e82ad-e5b7-153d-3738-360aeb809954@gmail.com> > Furthermore, if there are any tickets pending that you'd like Bro 2.5 > to address, now would be a good time to point them out. Coordinated with Seth I have done some refactoring of the intel-framework (https://github.com/J-Gras/bro/tree/topic/jgras/intel-update). There are only two changes left to do: - adding Seth's extension mechanism - adding expiration for items I would be happy to finish that and open a pull request if you consider this as relevant for 2.5. In case, what would be the deadline for the PR? Best regards, Jan From slagell at illinois.edu Sun May 8 09:57:40 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Sun, 8 May 2016 16:57:40 +0000 Subject: [Bro-Dev] Detecting protocols without full analyzers Message-ID: Some of the core developers of Bro have been having this discussion internally, and I?d like to bring it to the broader community. It has been recognized that there are a lot of protocols for which we don?t have full analyzers that some would still like to detect in our conn.logs via simple signatures. A full analyzer is much harder to write and to do well. This creates a barrier to entry. Further, some protocols would not benefit much from deeper analysis because of encryption or other issues. However, it is still desirable to notice that such protocols and applications are used on your network. I don?t think anyone disagreed that this could be useful, but the question would be how to do it in a maintainable way and where to put it. For example, would this be another field in the conn.log? Would this be turned on in Bro by default, would it be in the policy directory and not base, or would it be a separate plugin people could download if they want. I?m not going to repeat all the arguments for or against different positions here; I?ll let people do that for themselves. I just want to start the conversation within the broader community. :Adam ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From asharma at lbl.gov Mon May 9 02:29:49 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 9 May 2016 02:29:49 -0700 Subject: [Bro-Dev] bloomfilter_counting_init parameterization ? In-Reply-To: <20160503141824.GI72836@shogun> References: <20160411064721.GA74290@mac.local> <20160411152637.GH1342@shogun> <20160503072555.GF84792@mac.local> <20160503141824.GI72836@shogun> Message-ID: <20160509092946.GD79714@mac.local> Matthias, I am encountering some big tables in my scan-detection heuristics and which grow due to scanners: So was thinking of this possibility to use counting bloomfilters instead of tables and sets. After-all we are still looking for cardinality of tables and sets for identifying scanners. for example: 1) global distinct_peers: table[addr] of set[addr] then .... ..... if ( orig !in distinct_peers ) distinct_peers[orig] = set() &mergeable; if ( resp !in distinct_peers[orig] ) add distinct_peers[orig][resp]; local n = |distinct_peers[orig]|; and if n > N - its a scanner !!! SO I was wondering can the following be somehow represented as combinations of counting bloomfilters: 1) global distinct_peers: table[addr] of set[addr] and/or 2) global distinct_backscatter_peers: table[addr] of table[port] of set[addr] Aashish Here is an example proof-of-concept policy of what I am tryig to explore: ======================= bloom-scan.bro ========== module Scan; global src: opaque of bloomfilter ; global dst_port: opaque of bloomfilter ; event bro_init() { src = bloomfilter_counting_init(3, 128, 100000000); dst_port = bloomfilter_counting_init(3, 128, 100000000); } function check_bloom (c: connection) { local orig = c$id$orig_h; local resp = c$id$resp_h ; local resp_p = c$id$resp_p ; if (resp_p == 40884/tcp || resp_p == 40876/tcp) return ; bloomfilter_add (src, orig); bloomfilter_add (dst_port, fmt("%s%s", resp, resp_p)); local src_counts = bloomfilter_lookup(src, orig) ; local dst_counts = bloomfilter_lookup(dst_port, fmt("%s%s", resp, resp_p)) ; #### idea here is that a remote scanner is going to be hitting a lot of local hosts #### so footprint (conn counts of the remote scanner) is going to be dis-propotionate to ### footprint of local host+port if (src_counts > 30 && dst_counts < 5) print fmt ("possible_scanner: %s -> %s on %s ( counts: %s, %s)", orig, resp, resp_p, src_counts, dst_counts); } event partial_connection(c: connection) { Scan::check_bloom(c); } event connection_attempt(c: connection) { Scan::check_bloom(c); } event connection_half_finished(c: connection) { # Half connections never were "established", so do scan-checking here. Scan::check_bloom(c); } event connection_rejected(c: connection) { Scan::check_bloom(c); } event connection_reset(c: connection) { Scan::check_bloom(c); } event connection_pending(c: connection) { if ( c$orig$state == TCP_PARTIAL && c$resp$state == TCP_INACTIVE ) Scan::check_bloom(c); } From robin at icir.org Mon May 9 09:20:02 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 9 May 2016 09:20:02 -0700 Subject: [Bro-Dev] Detecting protocols without full analyzers In-Reply-To: References: Message-ID: <20160509162002.GD54345@icir.org> On Sun, May 08, 2016 at 16:57 +0000, you wrote: > I don?t think anyone disagreed that this could be useful, but the > question would be how to do it in a maintainable way and where to put > it. I agree that detecting more protocols would certainly be useful, but I remain skeptical of the mechanism: the proposal is to detect protocols by relying only on signatures looking for characteristic byte sequences; in contrast to the current DPD approach actually attempting to parse the protocol. I am concerned about reliability with any signatures-only approach. Actually I would propose something else: we recently added minimal analyzers for IMAP and XMPP that parse just the beginning of a session---just enough to confirm the protocol and, in these cases, also use of SSL. That's an approach that I think could work more generally as well: even if a full analyzer isn't feasible, doing just the standard DPD confirmation for a protocol should usually be pretty straight-forward. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From slagell at illinois.edu Mon May 9 09:22:45 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Mon, 9 May 2016 16:22:45 +0000 Subject: [Bro-Dev] Detecting protocols without full analyzers In-Reply-To: <20160509162002.GD54345@icir.org> References: <20160509162002.GD54345@icir.org> Message-ID: > On May 9, 2016, at 11:20 AM, Robin Sommer wrote: > > Actually I would propose something else: we recently added minimal > analyzers for IMAP and XMPP that parse just the beginning of a > session---just enough to confirm the protocol and, in these cases, > also use of SSL. That's an approach that I think could work more > generally as well: even if a full analyzer isn't feasible, doing just > the standard DPD confirmation for a protocol should usually be pretty > straight-forward. Is this what Justin did for RDP, because I don?t think that was much effort, was it Justin? :Adam ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From robin at icir.org Mon May 9 09:48:58 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 9 May 2016 09:48:58 -0700 Subject: [Bro-Dev] 2.5 Roadmap In-Reply-To: <327e82ad-e5b7-153d-3738-360aeb809954@gmail.com> References: <20160507005624.GI61305@icir.org> <327e82ad-e5b7-153d-3738-360aeb809954@gmail.com> Message-ID: <20160509164858.GF54345@icir.org> On Sat, May 07, 2016 at 10:36 +0200, you wrote: > I would be happy to finish that and open a pull request if you consider > this as relevant for 2.5. In case, what would be the deadline for the PR? We don't have a concrete deadline yet, it'll most likely take a few more weeks until we call a feature freeze. I'll let Seth comment on content but generally merging your changes in for 2.5 certainly sounds good to me. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From asharma at lbl.gov Mon May 9 11:26:24 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 9 May 2016 11:26:24 -0700 Subject: [Bro-Dev] bloomfilter_counting_init parameterization ? In-Reply-To: <20160509092946.GD79714@mac.local> References: <20160411064721.GA74290@mac.local> <20160411152637.GH1342@shogun> <20160503072555.GF84792@mac.local> <20160503141824.GI72836@shogun> <20160509092946.GD79714@mac.local> Message-ID: Nevermind my email! I found: src/probabilistic/cardinality-counter.bif Thanks, Aashish On Mon, May 9, 2016 at 2:29 AM, Aashish Sharma wrote: > Matthias, > > I am encountering some big tables in my scan-detection heuristics and which grow due to scanners: > > So was thinking of this possibility to use counting bloomfilters instead of tables and sets. After-all we are still looking for cardinality of tables and sets for identifying scanners. > > for example: > > 1) global distinct_peers: table[addr] of set[addr] > > then .... > ..... > > if ( orig !in distinct_peers ) > distinct_peers[orig] = set() &mergeable; > > if ( resp !in distinct_peers[orig] ) > add distinct_peers[orig][resp]; > > local n = |distinct_peers[orig]|; > > > and if n > N - its a scanner !!! > > > SO I was wondering can the following be somehow represented as combinations of counting bloomfilters: > > 1) global distinct_peers: table[addr] of set[addr] > > and/or > > 2) global distinct_backscatter_peers: table[addr] of table[port] of set[addr] > > Aashish > > > Here is an example proof-of-concept policy of what I am tryig to explore: > > ======================= bloom-scan.bro ========== > > module Scan; > > global src: opaque of bloomfilter ; > global dst_port: opaque of bloomfilter ; > > > event bro_init() > { > > src = bloomfilter_counting_init(3, 128, 100000000); > dst_port = bloomfilter_counting_init(3, 128, 100000000); > } > > > function check_bloom (c: connection) > { > > local orig = c$id$orig_h; > local resp = c$id$resp_h ; > local resp_p = c$id$resp_p ; > > > if (resp_p == 40884/tcp || resp_p == 40876/tcp) > return ; > > bloomfilter_add (src, orig); > bloomfilter_add (dst_port, fmt("%s%s", resp, resp_p)); > > > local src_counts = bloomfilter_lookup(src, orig) ; > local dst_counts = bloomfilter_lookup(dst_port, fmt("%s%s", resp, resp_p)) ; > > #### idea here is that a remote scanner is going to be hitting a lot of local hosts > #### so footprint (conn counts of the remote scanner) is going to be dis-propotionate to > ### footprint of local host+port > > if (src_counts > 30 && dst_counts < 5) > print fmt ("possible_scanner: %s -> %s on %s ( counts: %s, %s)", orig, resp, resp_p, src_counts, dst_counts); > > > } > > > event partial_connection(c: connection) > { > Scan::check_bloom(c); > } > > event connection_attempt(c: connection) > { > Scan::check_bloom(c); > } > > event connection_half_finished(c: connection) > { > # Half connections never were "established", so do scan-checking here. > Scan::check_bloom(c); > } > > event connection_rejected(c: connection) > { > Scan::check_bloom(c); > } > > event connection_reset(c: connection) > { > Scan::check_bloom(c); > } > > event connection_pending(c: connection) > { > if ( c$orig$state == TCP_PARTIAL && c$resp$state == TCP_INACTIVE ) > Scan::check_bloom(c); > } > > From seth at icir.org Tue May 10 10:22:37 2016 From: seth at icir.org (Seth Hall) Date: Tue, 10 May 2016 13:22:37 -0400 Subject: [Bro-Dev] 2.5 Roadmap In-Reply-To: <20160509164858.GF54345@icir.org> References: <20160507005624.GI61305@icir.org> <327e82ad-e5b7-153d-3738-360aeb809954@gmail.com> <20160509164858.GF54345@icir.org> Message-ID: > On May 9, 2016, at 12:48 PM, Robin Sommer wrote: > > We don't have a concrete deadline yet, it'll most likely take a few > more weeks until we call a feature freeze. I'll let Seth comment on > content but generally merging your changes in for 2.5 certainly sounds > good to me. Agreed, looking forward to the changes Jan! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From vallentin at icir.org Wed May 11 18:07:32 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Wed, 11 May 2016 18:07:32 -0700 Subject: [Bro-Dev] Broker C++ namespace: broker -> bro? Message-ID: <20160512010732.GO2796@samurai.ICIR.org> I'm considering renaming the C++ broker namespace from "broker" to "bro". After all, it's *Bro's* communication library. This would reduce noise in the source code (3 less characters for those who cannot or don't want to import the entire namespace) and also reduce ambiguity (we can now use bro::broker as a class name). The only point where I could see this problematic is if we ever wanted to distribute Bro itself as shared library. Personally, I don't see this a realistic outcome though. The overwhelming amount of global state and program structure has not been designed to allow for distribution as (shared) library. Thoughts? Matthias From asharma at lbl.gov Thu May 12 01:44:58 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Thu, 12 May 2016 01:44:58 -0700 Subject: [Bro-Dev] declaration error: &default function type clash Message-ID: <20160512084455.GG2221@mac.local> So I am trying to convert tables into using opaque of cardinality since thats more memory efficient (or counting bloomfilters for that matter): works: if table (0) converted to (1) errors: if table (2) converted to (3) Details: I am trying the following, original table (0) converted to (1): (0) global likely_scanner: table[addr,port] of set[addr] &read_expire=1 day &synchronized ; (1) global c_likely_scanner: table[addr] of opaque of cardinality &default = function(n: any): opaque of cardinality { return hll_cardinality_init(0.1, 0.95); } &read_expire=1 day ; ERRORS: (2) global likely_scanner: table[addr,port] of set[addr] &read_expire=1 day &synchronized ; Converted table: (3) global c_likely_scanner: table[addr,port] of opaque of cardinality &default = function(n: any): opaque of cardinality { return hll_cardinality_init(0.1, 0.95); } &read_expire=1 day ; I get this error: check-knock.bro, line 58: &default function type clash (&default=anonymous-function{ return (hll_cardinality_init(0.1, 0.95))}) Question: how do I declare (3) so that I can avoid the "&default function type clash" error above. I am not sure what am I doing wrong in the declaration. Any thoughts/advice how to get past this issue ? Thanks, Aashish From jan.grashoefer at gmail.com Thu May 12 07:36:36 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Thu, 12 May 2016 16:36:36 +0200 Subject: [Bro-Dev] declaration error: &default function type clash In-Reply-To: <20160512084455.GG2221@mac.local> References: <20160512084455.GG2221@mac.local> Message-ID: <9ba24a5c-f069-fa75-f284-75b75b9965d9@gmail.com> > how do I declare (3) so that I can avoid the "&default function type clash" error above. I guess the function for initialization receives the index that should be initialized. In this case the index consists of two values. I tried the following and Bro did not complain: global c_likely_scanner: table[addr,port] of opaque of cardinality &default = function(a: addr, p: port): opaque of cardinality { return hll_cardinality_init(0.1, 0.95); } &read_expire=1 day; Hope that works for you! Regards, Jan From asharma at lbl.gov Thu May 12 10:12:41 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Thu, 12 May 2016 10:12:41 -0700 Subject: [Bro-Dev] declaration error: &default function type clash In-Reply-To: <9ba24a5c-f069-fa75-f284-75b75b9965d9@gmail.com> References: <20160512084455.GG2221@mac.local> <9ba24a5c-f069-fa75-f284-75b75b9965d9@gmail.com> Message-ID: <20160512171238.GA22306@mac.local> Jan, > I guess the function for initialization receives the index that should > be initialized. Thank you. This works! For future reference: I also needed to convert the following table to use opaque of cardinality for this table grows reasonably big: global distinct_backscatter_peers: table[addr] of table[port] of set[addr] &read_expire=1 day; Here is how I did this one: type bs: table[port] of opaque of cardinality &default=function(p:port): opaque of cardinality {return hll_cardinality_init(0.1, 0.95); }; global c_distinct_backscatter_peers: table[addr] of bs &read_expire=1 day ; and to access: if ( orig !in c_distinct_backscatter_peers) c_distinct_backscatter_peers[orig] = table() ; if (s_port !in c_distinct_backscatter_peers[orig]) { local cp: opaque of cardinality = hll_cardinality_init(0.1, 0.95); c_distinct_backscatter_peers[orig][s_port]=cp ; } hll_cardinality_add(c_distinct_backscatter_peers[orig][s_port], resp); local d_val = double_to_count(hll_cardinality_estimate(c_distinct_backscatter_peers[orig][s_port])); ### use d_val here .... Now may be there is a better more elegant way to do this, but above seems to work for me. Again, thanks Jan!! Aashish On Thu, May 12, 2016 at 04:36:36PM +0200, Jan Grash?fer wrote: > > how do I declare (3) so that I can avoid the "&default function type clash" error above. > > I guess the function for initialization receives the index that should > be initialized. In this case the index consists of two values. I tried > the following and Bro did not complain: > > global c_likely_scanner: table[addr,port] of opaque of cardinality > &default = function(a: addr, p: port): opaque of cardinality { > return hll_cardinality_init(0.1, 0.95); } > &read_expire=1 day; > > Hope that works for you! > > Regards, > Jan > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From robin at icir.org Thu May 12 11:59:11 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 12 May 2016 11:59:11 -0700 Subject: [Bro-Dev] Broker C++ namespace: broker -> bro? In-Reply-To: <20160512010732.GO2796@samurai.ICIR.org> References: <20160512010732.GO2796@samurai.ICIR.org> Message-ID: <20160512185911.GE10312@icir.org> On Wed, May 11, 2016 at 18:07 -0700, you wrote: > I'm considering renaming the C++ broker namespace from "broker" to > "bro". After all, it's *Bro's* communication library. Not sure I like that. The library is called Broker, so "broker" seems the natural namespace to me. And it's not really just for Bro either, people might just as well use it outside of Bro (and I hope they eventually will! :) > This would reduce noise in the source code (3 less characters for > those who cannot or don't want to import the entire namespace) and > also reduce ambiguity (we can now use bro::broker as a class name). I don't think the first one is a problem, and the latter is a self-made one: I think the better solution would be finding a different name for those things now called "broker", rather than changing the entire namespace. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From vallentin at icir.org Thu May 12 12:42:07 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 12 May 2016 12:42:07 -0700 Subject: [Bro-Dev] Broker C++ namespace: broker -> bro? In-Reply-To: <20160512185911.GE10312@icir.org> References: <20160512010732.GO2796@samurai.ICIR.org> <20160512185911.GE10312@icir.org> Message-ID: <20160512194207.GT72836@shogun> > Not sure I like that. The library is called Broker, so "broker" seems > the natural namespace to me. All right, let's keep it as is. This is also aligns with the sentiment at our Gitter room. > I think the better solution would be finding a different name for > those things now called "broker" [..] Speaking of which: let me summarize the issue. We need a good name for something that is a message broker, an entity that communicates in the publish/subscribe paradigm. These message brokers can communicate via TCP or shared memory, and any program can have a large number of them. In the current Broker code, we use the term "endpoint," but I find it implies too little pub/sub. I'm currently using the term "broker" instead. However, this is also the library name and could cause some confusion. Internally, a "broker" maps to an actor (as in the actor model). If anyone has suggestions on how to better name these abstract pub/sub entities, please chime in. Matthias From jsiwek at illinois.edu Fri May 13 09:30:29 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 13 May 2016 16:30:29 +0000 Subject: [Bro-Dev] Broker C++ namespace: broker -> bro? In-Reply-To: <20160512194207.GT72836@shogun> References: <20160512010732.GO2796@samurai.ICIR.org> <20160512185911.GE10312@icir.org> <20160512194207.GT72836@shogun> Message-ID: <5FC0DC78-7010-4B94-91BA-219C57390E72@illinois.edu> > On May 12, 2016, at 2:42 PM, Matthias Vallentin wrote: > > In the current Broker code, we use the term "endpoint," but I find it > implies too little pub/sub. I'm currently using the term "broker" > instead. However, this is also the library name and could cause some > confusion. Internally, a "broker" maps to an actor (as in the actor > model). If anyone has suggestions on how to better name these abstract > pub/sub entities, please chime in. To me a ?broker? in a pub/sub system describes a thing that sits between publisher and subscribers (distinct from either) and manages the flow/filtering of messages. The internals of the library itself, already named Broker, do seem to serve that role, and so naming another thing ?broker? adds confusion. Ideas fort improving the name of ?endpoint?: (1) Make up a name that looks/sounds like pub/sub. e.g. call it a ?pubsub? or a ?pubsubber? (2) Try to decouple the publisher and subscriber features into two separate classes and just literally name them ?publisher? and ?subscriber?. - Jon From robin at icir.org Fri May 20 12:32:53 2016 From: robin at icir.org (Robin Sommer) Date: Fri, 20 May 2016 12:32:53 -0700 Subject: [Bro-Dev] 2.5 Roadmap In-Reply-To: <20160507005624.GI61305@icir.org> References: <20160507005624.GI61305@icir.org> Message-ID: <20160520193253.GA66696@icir.org> Updates from today's meeting regarding 2.5 progress: - Seth will send summary of the "flare situation" to bro-dev. - NTP analyzer is in progress. - SMB analyzer almost ready for merge. We'll not load it by default. We will put it into policy instead and leave it commented out in local.bro, marked as experimental. - Intel updates are in the queue. - New Broker API is in progress. Also, everybody, please look at tickets: - Check any 2.5 tickets assigned to you; if you don't think you'll make it, bump to 2.6. - Assign further tickets to 2.5 that should go in but aren't marked so yet. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Fri May 20 13:16:33 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 20 May 2016 20:16:33 +0000 Subject: [Bro-Dev] CBAN design proposal Message-ID: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> Here?s an updated design doc for CBAN, a plugin manager for Bro, with some new ideas on how it could work: https://www.bro.org/development/projects/cban2.html Eventually it may be good to ask for feedback on bro-users to make sure we?re not missing important features the community would want, but I thought I?d start w/ bro-dev in the interest of wasting fewer people?s time in the case the design is way off base. - Jon From seth at icir.org Fri May 20 13:52:17 2016 From: seth at icir.org (Seth Hall) Date: Fri, 20 May 2016 16:52:17 -0400 Subject: [Bro-Dev] Flare removal Message-ID: For the 2.5 release, we were hoping to understand why the topic/seth/remove-flare fixes some issues that people have been seeing with the communication code. Perhaps even more to the point we are aiming to understand why that branch fixes the problem, but Robin's branch topic/robin/no-flares-2.4.1 doesn't work. The problem that we've seen will exhibit on Linux (for some reason FreeBSD doesn't seem to be affected) and you will see high memory use on the child of your manager process. People will tend to notice it in two ways. 1. Memory exhaustion 2. Logs being written that are seconds to minutes old. This isn't exactly a request for anyone to do anything, but more a call for anyone that would like to dig around in the core to figure out what is going on here so we can get a fix merged into master. Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jazoff at illinois.edu Fri May 20 15:02:14 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 20 May 2016 22:02:14 +0000 Subject: [Bro-Dev] Flare removal In-Reply-To: References: Message-ID: <02BF1382-0E82-4C1D-8CA2-F2C470CB5FF7@illinois.edu> > On May 20, 2016, at 4:52 PM, Seth Hall wrote: > > For the 2.5 release, we were hoping to understand why the topic/seth/remove-flare fixes some issues that people have been seeing with the communication code. Perhaps even more to the point we are aiming to understand why that branch fixes the problem, but Robin's branch topic/robin/no-flares-2.4.1 doesn't work. > > The problem that we've seen will exhibit on Linux (for some reason FreeBSD doesn't seem to be affected) and you will see high memory use on the child of your manager process. People will tend to notice it in two ways. > 1. Memory exhaustion > 2. Logs being written that are seconds to minutes old. > > This isn't exactly a request for anyone to do anything, but more a call for anyone that would like to dig around in the core to figure out what is going on here so we can get a fix merged into master. > > Thanks! > .Seth I had looked into it a while ago.. I don't think the differences in your branches has anything to do with flares... $ git diff origin/topic/robin/no-flares-2.4.1 origin/topic/seth/remove-flare src/iosource/Manager.cc diff --git a/src/iosource/Manager.cc b/src/iosource/Manager.cc index 80fa5fe..5ad8cca 100644 --- a/src/iosource/Manager.cc +++ b/src/iosource/Manager.cc @@ -96,8 +96,8 @@ IOSource* Manager::FindSoonest(double* ts) // return it. int maxx = 0; - if ( soonest_src && (call_count % SELECT_FREQUENCY) != 0 ) - goto finished; +// if ( soonest_src && (call_count % SELECT_FREQUENCY) != 0 ) +// goto finished; // Select on the join of all file descriptors. fd_set fd_read, fd_write, fd_except; $ git diff origin/topic/robin/no-flares-2.4.1 origin/topic/seth/remove-flare src/RemoteSerializer.cc [snip] - // FIXME: Fine-tune this (timeouts, flush, etc.) - struct timeval small_timeout; - small_timeout.tv_sec = 0; - small_timeout.tv_usec = - io->CanWrite() || io->CanRead() ? 1 : 10; - -#if 0 - if ( ! io->CanWrite() ) - usleep(10); -#endif - - int a = select(max_fd + 1, &fd_read, &fd_write, &fd_except, - &small_timeout); + struct timeval timeout; + timeout.tv_sec = 1; + timeout.tv_usec = 0; - if ( a == 0 ) - ++timeouts; + int a = select(max_fd + 1, &fd_read, &fd_write, &fd_except, &timeout); Seths branch removes the SELECT_FREQUENCY check and defaults the serializer 'small timeout' to 1 full second. Robins branch still has the SELECT_FREQUENCY check and has the small timeout set to 1 or 10 microseconds. I think the two extra changes in Seths branch combine to make bro spend more time in the RemoteSerializer code. When I was trying to figure some of this out I believed that many of these constants were part of the issue. All the different places calling select with different timeouts and different frequencies causing bro to spend more time calling select than it was actually moving bytes around. The only thing I ever really found wrong with the flare code was that repeated fire/extinguishes were not No-Ops and I had a small patch that improved that without changing anything else (attached). I think Robins branch doesn't fix the problem because I don't think the flares were really the issue.. I think bro started having issues because between 2.3 and 2.4 traffic volumes increased, cluster sizes increased, and we added a ton of new analyzers and log files which put even more strain on the communication system. -- - Justin Azoff -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160520/ca61cdc5/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: flare_fix.patch Type: application/octet-stream Size: 839 bytes Desc: flare_fix.patch Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160520/ca61cdc5/attachment.obj From vallentin at icir.org Sat May 21 09:01:32 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Sat, 21 May 2016 09:01:32 -0700 Subject: [Bro-Dev] Broker status update Message-ID: <20160521160132.GA89196@shogun> This is just a brief summary of where the Broker overhaul is currently at. The new Broker API will come two types of endpoints: blocking and nonblocking. The former provides a synchronous API to receive messages, whereas the latter operates fully asynchronous. Performance-wise, the asynchronous version has a major advantage because it does not introduce buffering. The callback provided upon endpoint construction will be immediately executed by the Broker runtime, for each message as it arrives. For a blocking endpoint, it's the users responsibility to extract messages, otherwise they queue up. Endpoints can only be constructed through a broker::context object: // Encapsulates global states, such as announced types, thread pool, // CAF scheduler, etc. context ctx; // Creates a synchronous endpoint. auto be = ctx.spawn(); auto msg = be.receive(); // Block and wait for next message. be.receive( [&](const topic&, const message&) { // As above, but use a lambda to handle the topic-message pair. } ); // Block and wait for next message. // Creates an asynchronous endpoint. auto ne = ctx.spawn( [=](const topic&, const message&) { // invoked for every subscribed message } ); Other than that, both brokers have a publish/subscribe interface, e.g.: be.subscribe("foo"); be.subscribe("bar"); auto t = topic{"bar"}; auto msg = make_message(42, "string"); be.publish(t, msg); I'll post more updates as the development progresses, but all of the above examples are now working. The TODO list currently includes: - Peerings: managing subscriptions across multiple endpoints. Here's an example: // Make two endpoints. auto x = ctx.spawn<...>(...); auto y = ctx.spawn<...>(...); I'm torn between two interfaces. This is the first: peer(x, y); // Create a peering between the two endpoints. peer(y, x); // Idempotent. Peerings are symmetric. // Peer with a remote endpoint. Requires that the remote side // called e.listen(addr, port) beforehand. auto r = ctx.spawn("1.2.3.4", 42000); peer(y, r); // Undo a peering. unpeer(x, y); unpeer(r, y); The second is: x.peer(y); // Create a peering between the two endpoints. y.peer(x); // Idempotent. Peerings are symmetric. x.peer("1.2.3.4", 42000); // Peer with a remote endpoint. x.unpeer(y); x.unpeer("1.2.3.4", 42000); // Unpeer from a remote endpoint, must match previous peering data. Personally, I'm favoring the first version, as the functional API makes the symmetry of the peering relationship clearer. - Data store: integrate the new "main API" with the existing data stores. My plan is to use as much as possible from the existing data store API. - Bindings: For Python, I'm considering switching to pybind11 [1], which provides a much more convenient API than SWIG and supports modern C++11. Please chime in if you have questions/comment or see opportunities for improvement. Matthias [1] http://pybind11.readthedocs.io/en/latest/classes.html From robin at icir.org Sat May 21 15:44:01 2016 From: robin at icir.org (Robin Sommer) Date: Sat, 21 May 2016 15:44:01 -0700 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> Message-ID: <20160521224401.GL41563@icir.org> On Fri, May 20, 2016 at 20:16 +0000, Jon wrote: > Here?s an updated design doc for CBAN, a plugin manager for Bro, with > some new ideas on how it could work: Cool, thanks. I'm going to send some feedback but first I wanted to bring something up that might be controversial. :-) As I read through the design doc, I started questioning our plan of curating CBAN content. I know that's what we've been intending to do, but is that really the best approach? I don't know of script repositories for other languages that enforce quality control on submissions beyond checking technical conventions like certain meta data being there. I'm getting concerned that we're creating an unncessary bottleneck by imposing the Bro Team into the critical path for submissions and updates. Why not let people control their stuff themselves? They'd register things with CBAN to make it available to everybody, but we'd stay out of the loop for anything further. We may still want to keep a local copy of the code to protect against stuff going offline, but that could be automated. Or we could even move that burden to the authors as well and just keep a reference where to find the code, without a local copy; if it disappears, so be it. In other words, my proposal is to put authors into control of their code, and make them fully responsible for it too --- not us. We'd just connect authors with users, with as little friction as possible. If we want some kind of quality measure, we could introduce stars for modules that user assign if they like something; or even some facility to leave comments or other feedback that's visible to everybody. We could also verify if a plugins builds and loads correctly, or if tests pass. But we wouldn't block it if something fails, just mark it (say, with a banner saying "tests pass", "tests fail", "no tests"). Thoughts? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From asharma at lbl.gov Sat May 21 15:52:00 2016 From: asharma at lbl.gov (Aashish Sharma) Date: Sat, 21 May 2016 15:52:00 -0700 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160521224401.GL41563@icir.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> Message-ID: <20160521225158.GF53321@mac.local> > In other words, my proposal is to put authors into control of their > code, and make them fully responsible for it too --- not us. We'd just > connect authors with users, with as little friction as possible. > I support this completely. > If we want some kind of quality measure, we could introduce stars for > modules that user assign if they like something; or even some facility > to leave comments or other feedback that's visible to everybody. We I think community vetting and feedback (and experience stories) is the right way to go here. Bro team vetting something will be very hard. My personal experience says there are times when scripts bring surprises weeks and months down the line - so testing isn't merely run a few days and give an OK. Aashish On Sat, May 21, 2016 at 03:44:01PM -0700, Robin Sommer wrote: > > > On Fri, May 20, 2016 at 20:16 +0000, Jon wrote: > > > Here?s an updated design doc for CBAN, a plugin manager for Bro, with > > some new ideas on how it could work: > > Cool, thanks. I'm going to send some feedback but first I wanted to > bring something up that might be controversial. :-) > > As I read through the design doc, I started questioning our plan of > curating CBAN content. I know that's what we've been intending to do, > but is that really the best approach? I don't know of script > repositories for other languages that enforce quality control on > submissions beyond checking technical conventions like certain meta > data being there. > > I'm getting concerned that we're creating an unncessary bottleneck by > imposing the Bro Team into the critical path for submissions and > updates. Why not let people control their stuff themselves? They'd > register things with CBAN to make it available to everybody, but we'd > stay out of the loop for anything further. We may still want to keep a > local copy of the code to protect against stuff going offline, but > that could be automated. Or we could even move that burden to the > authors as well and just keep a reference where to find the code, > without a local copy; if it disappears, so be it. > > In other words, my proposal is to put authors into control of their > code, and make them fully responsible for it too --- not us. We'd just > connect authors with users, with as little friction as possible. > > If we want some kind of quality measure, we could introduce stars for > modules that user assign if they like something; or even some facility > to leave comments or other feedback that's visible to everybody. We > could also verify if a plugins builds and loads correctly, or if tests > pass. But we wouldn't block it if something fails, just mark it (say, > with a banner saying "tests pass", "tests fail", "no tests"). > > Thoughts? > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From slagell at illinois.edu Sat May 21 16:16:14 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Sat, 21 May 2016 23:16:14 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160521224401.GL41563@icir.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu>, <20160521224401.GL41563@icir.org> Message-ID: <3603B060-4909-4D25-888D-1A9B782AF5B7@illinois.edu> > On May 21, 2016, at 5:44 PM, Robin Sommer wrote: > > As I read through the design doc, I started questioning our plan of > curating CBAN content. I know that's what we've been intending to do, > but is that really the best approach? I don't know of script > repositories for other languages that enforce quality control on > submissions beyond checking technical conventions like certain meta > data being there. I think there is a broad spectrum from no interaction to vetting and pulling into the main repository. It is about finding the right balance. I agree with minimal restrictions that block submissions. There needs to be some basic quality control and standardization there. For example, do you have all the right pieces. I do think there is another level of non blocking checks and quality control we can provide. For example, we can do checks for exec calls and give warnings to users. I think Puppet Forge has a nice model here. We won't block a submission, but these checks encourage better development and help new users trust submissions. That said, I think these must be automated. They can't block on a human reviewing them. Finally, I think we need a way to let the whole community endorse scripts or script authors. From seth at icir.org Sat May 21 19:01:01 2016 From: seth at icir.org (Seth Hall) Date: Sat, 21 May 2016 22:01:01 -0400 Subject: [Bro-Dev] Flare removal In-Reply-To: <02BF1382-0E82-4C1D-8CA2-F2C470CB5FF7@illinois.edu> References: <02BF1382-0E82-4C1D-8CA2-F2C470CB5FF7@illinois.edu> Message-ID: > On May 20, 2016, at 6:02 PM, Azoff, Justin S wrote: > > I think Robins branch doesn't fix the problem because I don't think the flares were really the issue.. I think bro started having issues because between 2.3 and 2.4 traffic volumes increased, cluster sizes increased, and we added a ton of new analyzers and log files which put even more strain on the communication system. That's an interesting idea and at least sounds as plausible as anything else I've heard. Have you tried making those small changes you outlined by themselves on the NCSA dev cluster yet? I'd be curious to hear if that fixes the problem. Or do you even see this issue on that cluster? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From robin at icir.org Mon May 23 08:16:07 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 23 May 2016 08:16:07 -0700 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <3603B060-4909-4D25-888D-1A9B782AF5B7@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <3603B060-4909-4D25-888D-1A9B782AF5B7@illinois.edu> Message-ID: <20160523151607.GE57255@icir.org> On Sat, May 21, 2016 at 23:16 +0000, you wrote: > I think there is a broad spectrum from no interaction to vetting and > pulling into the main repository. It is about finding the right > balance. Yep, and I'm arguing for going very far towards the "no interaction" side, with just some automatic checks for the most crucial things. Maybe the initial pull request for integration could be merged manually to avoid any spamming, but any updates for example should not require any interaction from us. > I do think there is another level of non blocking checks and quality > control we can provide. Yes, indeed, I like this. With things already merged in, we can in parallel, in the background, build up a toolbox of things to check for and mark a module accordingly. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Mon May 23 08:23:26 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 23 May 2016 08:23:26 -0700 Subject: [Bro-Dev] Broker status update In-Reply-To: <20160521160132.GA89196@shogun> References: <20160521160132.GA89196@shogun> Message-ID: <20160523152326.GG57255@icir.org> On Sat, May 21, 2016 at 09:01 -0700, you wrote: > peer(x, y); // Create a peering between the two endpoints. > peer(y, x); // Idempotent. Peerings are symmetric. > x.peer(y); // Create a peering between the two endpoints. > y.peer(x); // Idempotent. Peerings are symmetric. I would prefer the 2nd way for consistenct, as all the other operations use the method-based scheme. The idempotency seems secondary to that I would say. Related question: what exactly are the semantics if only one side of the peering is set up? > - Bindings: For Python, I'm considering switching to pybind11 [1], > which provides a much more convenient API than SWIG and supports > modern C++11. Hmm ... I see the appeal but it would introduce a new dependency and its Python-specific (I assume), whereas with SWIG it's easier to add more languages later. Is that worth the benefit of switching? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From vlad at grigorescu.org Mon May 23 09:22:37 2016 From: vlad at grigorescu.org (Vlad Grigorescu) Date: Mon, 23 May 2016 11:22:37 -0500 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160523151607.GE57255@icir.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <3603B060-4909-4D25-888D-1A9B782AF5B7@illinois.edu> <20160523151607.GE57255@icir.org> Message-ID: I think we're generally on the same page, but I wanted to elaborate a bit on what I envisioned with the plugin submission process. When a contributor submits a new script, there would be some mandatory checks that would need to pass for the script to be included: * Is the plugin structure valid? * Is there a valid metadata file? This file could list things like dependencies, what version of Bro the plugin works on, etc. I think the bare minimum of what needs to be in the plugin is: 1) version number and 2) license information. It's possible that we wouldn't even need the license if the submission process puts a default license on any plugin that doesn't specify otherwise. * If there are dependencies, are those already in the CBAN repository? * Is Bro able to load this plugin with --parse-only, or does it generate a parse error? * Is there already a plugin with that name and version number? If so, the author would need to increment the version. I think this is a nice balance between some bare minimum checks designed to ensure that the plugin is actually installable and not putting too much of a burden on contributors. Once a plugin has been submitted, if it passes those checks, I think it should be automatically pulled into a repo. I don't think that we need manual intervention for this. At this point, though, I think we could run some "quality tests" and give the plugin a quality score. This might be things like: * Is there documentation? (Both a README and checking to see if, for example, external redef-able consts are documented). * Are there btests? * Do the tests pass? * Does the plugin load in bare mode? These quality scores would be strictly informative, and wouldn't prevent or modify the acceptance of the plugin. What I'm imagining is something like this: https://forge.puppet.com/vshn/gitlab/scores#module-release-info --Vlad On Mon, May 23, 2016 at 10:16 AM, Robin Sommer wrote: > > > On Sat, May 21, 2016 at 23:16 +0000, you wrote: > > > I think there is a broad spectrum from no interaction to vetting and > > pulling into the main repository. It is about finding the right > > balance. > > Yep, and I'm arguing for going very far towards the "no interaction" > side, with just some automatic checks for the most crucial things. > Maybe the initial pull request for integration could be merged > manually to avoid any spamming, but any updates for example should not > require any interaction from us. > > > I do think there is another level of non blocking checks and quality > > control we can provide. > > Yes, indeed, I like this. With things already merged in, we can in > parallel, in the background, build up a toolbox of things to check for > and mark a module accordingly. > > Robin > > -- > Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160523/b9d28063/attachment.html From robin at icir.org Mon May 23 09:44:04 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 23 May 2016 09:44:04 -0700 Subject: [Bro-Dev] Coverity update Message-ID: <20160523164404.GJ57255@icir.org> I went through the pending Converity alerts, fixed a few with my commits this morning, and marked the others as minor. I didn't see anything severe in there, most were expected. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Mon May 23 10:26:28 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Mon, 23 May 2016 17:26:28 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160521224401.GL41563@icir.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> Message-ID: > On May 21, 2016, at 5:44 PM, Robin Sommer wrote: > > I'm getting concerned that we're creating an unncessary bottleneck by > imposing the Bro Team into the critical path for submissions and > updates. Why not let people control their stuff themselves? They'd > register things with CBAN to make it available to everybody, but we'd > stay out of the loop for anything further. To clarify, is the concern w/ the amount of non-automatable tasks or with the model of requiring authors to ?ping? some middle-man in order for updates to their plugin to become visible to all CBAN users? Because most what I had outlined could be automated (apart from the suggestion that initial submissions have someone from the Bro Team quickly review whether it looks legit). Though I am on board for trying to draw up a contrasting design where the focus is on directly connecting users w/ plugin authors without any blocking processes or bureaucracy in the middle-man. That would make life easier for authors, and that?s maybe even a higher priority than maximizing the quality/consistency of user experience because, without authors, there won?t be much for users to experience in the first place. - Jon From robin at icir.org Mon May 23 11:59:01 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 23 May 2016 11:59:01 -0700 Subject: [Bro-Dev] Option -z Message-ID: <20160523185901.GQ57255@icir.org> Does anybody remember what Bro's option -z is for? -z|--analyze | run the specified policy file analysis Turns out the only supported "analysis" is "notice": # bro -r x.pcap -z notice Found NOTICE: PacketFilter::Dropped_Packets Found NOTICE: PacketFilter::Install_Failure Found NOTICE: Signatures::Signature_Summary Found NOTICE: PacketFilter::Compile_Failure Found NOTICE: Signatures::Multiple_Sig_Responders Found NOTICE: Signatures::Sensitive_Signature Found NOTICE: Signatures::Count_Signature Found NOTICE: PacketFilter::Too_Long_To_Compile_Filter Found NOTICE: Signatures::Multiple_Signatures This looks very specific for hard-coded event-engine functionality. I propose to remove unless somebody still sees a use for this? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Mon May 23 14:59:43 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 23 May 2016 14:59:43 -0700 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> Message-ID: <20160523215943.GR57255@icir.org> On Mon, May 23, 2016 at 17:26 +0000, you wrote: > To clarify, is the concern w/ the amount of non-automatable tasks or > with the model of requiring authors to ?ping? some middle-man in order > for updates to their plugin to become visible to all CBAN users? Both actually. Putting us into the path introduces delay and requires somebody making time available. This is already not working well for the current plugin repository, where things stall because we would like to provide feedback and help guide the author along, but then don't have the cycles for actually doing that. The delay/effort will be shorter/less the more tasks can be automated, but at the beginning we won't have much automation in place I imagine and even long-term certain stuff could never be automated. So I'm wondering if we really need to be in the path at all, seems that can only cause friction that would be better to avoid in particular as we kick things off. > Because most what I had outlined could be automated Yep, understood, my mail was not directly targetting your proposal; that just triggered me thinking about this again. My comments were meant more broadly, we've been discussing different notions of vetting over time with various subsets of people. > That would make life easier for authors, and that?s maybe even a > higher priority than maximizing the quality/consistency of user > experience because, without authors, there won?t be much for users to > experience in the first place. Yeah, that's exactly what I'm advocating: making it easy should really be priority number one, with everything else coming second. If you see ways to adapt the design to target that specifically, I'm all for it. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Mon May 23 15:40:09 2016 From: robin at icir.org (Robin Sommer) Date: Mon, 23 May 2016 15:40:09 -0700 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <3603B060-4909-4D25-888D-1A9B782AF5B7@illinois.edu> <20160523151607.GE57255@icir.org> Message-ID: <20160523224009.GS57255@icir.org> On Mon, May 23, 2016 at 11:22 -0500, you wrote: > When a contributor submits a new script, there would be some mandatory > checks that would need to pass for the script to be included: The "mandatory" is where I disagree. I believe there's just to much involved with any initial vetting, even if conceptually simply, that it will create a bottleneck. Take your first question as an example: "Is the plugin structure valid?" We wouldn't get very far with a simple yes/no answer. We'd have to explain what exactly is not valid, with some of that being just convention, or maybe something that matters in general for plugins but for some reason not the particular one. Or what if the plugin works with some version of Bro, but not another. Are we going say it needs to work with the current release? What if a new release comes out and breaks all existing plugins? What if then an update for a plugin comes in that doesn't move to the new version yet? > At this point, though, I think we could run some "quality tests" and > give the plugin a quality score. This might be things like: Yep, I'm fully on board with this part, that's good information we can provide to users about the state of something. And that state could be "totally broken". :-) Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From slagell at illinois.edu Mon May 23 16:30:44 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Mon, 23 May 2016 23:30:44 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160523224009.GS57255@icir.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <3603B060-4909-4D25-888D-1A9B782AF5B7@illinois.edu> <20160523151607.GE57255@icir.org> <20160523224009.GS57255@icir.org> Message-ID: <0FF909E5-D212-4FD4-A454-AE5117A9AA7D@illinois.edu> > On May 23, 2016, at 5:40 PM, Robin Sommer wrote: > >> >> When a contributor submits a new script, there would be some mandatory >> checks that would need to pass for the script to be included: > > The "mandatory" is where I disagree. I believe there's just to much > involved with any initial vetting, even if conceptually simply, that > it will create a bottleneck. I guess there is a balance here. If we do no mandatory checks and you could submit something that isn?t even a Bro plugin, the repository could become cluttered with junk. Do we really want things that don?t even ?compile?? I guess we can wait and see for some of these decisions, meaning start with optional and decide to make them mandatory if it becomes a problem. However, where we can?t do that is with the metadata we collect. If we don?t require what we think is important metadata in the beginning, then we will have a gap if we decide it was important all along. So there I would err towards overcorrecting in the beginning, and make things optional in the future if it turns out not to be important. From jsiwek at illinois.edu Tue May 24 09:21:02 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 24 May 2016 16:21:02 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160523215943.GR57255@icir.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <20160523215943.GR57255@icir.org> Message-ID: <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> > On May 23, 2016, at 4:59 PM, Robin Sommer wrote: > >> That would make life easier for authors, and that?s maybe even a >> higher priority than maximizing the quality/consistency of user >> experience because, without authors, there won?t be much for users to >> experience in the first place. > > Yeah, that's exactly what I'm advocating: making it easy should really > be priority number one, with everything else coming second. If you see > ways to adapt the design to target that specifically, I'm all for it. Yeah, I have ideas, but seems like there might be another day of some discussion before I try to formally reframe a design doc. Here?s the direction I'm thinking: - still have a "bro/cban" github repository, but it now contains git submodules that point directly to other github accounts/repos - the initial submission process involves doing a pull request on the bro/cban repo where the only change made is the addition of a submodule. These merges probably can be automated, but if a human were to do it, I?d expect it wouldn?t be a time-consuming task ? just check if the change is adding a submodule and then click a button to merge (don?t even have to look at the contents of the submodule). - a person that has submitted something to cban needs no further interaction with it and they resume their typical development workflow ? cban client's ?update? command will fetch/pull directly from their git repo. - all submodules get scanned by an out-of-band nightly process which checks for brokenness and other quality metrics - submodules that are found to have never been in a working state are auto-removed (or could initially be a task that?s not a big deal for a human to do every so often if metrics of brokenness are consistently available) - it?s not a big deal for a submodule to temporarily enter in to a broken state ? cban users can always roll back to a previous version or just uninstall it. It?s up to the community to communicate/collaborate directly w/ the author here to get things fixed. - metadata associated w/ plugins is all optional, but its existence contributes to some arbitrary ?quality? rating/metrics - metadata logically can be categorized in two types, one type is related to discoverability (tags, author, license, etc) and one type is related to interoperability (version number, dependencies) - discoverability metadata is aggregated during the nightly quality check processes and automatically commits that information to the ?bro/cban? repo. Without doing this, I think cban clients would have an incredibly slow ?search? command that goes out to each submodule individually and gathers metadata. (features related to discoverability might be lower priority in general) - interoperability metadata can also be aggregated nightly along the discoverability metadata, but when the cban client is actually going to perform specific operations on a particular submodule, it gets this data directly from the cloned submodule(s) to make sure the info is up-to-date. Version numbering can probably be done via git tags, but dependency info stored in a canonically named text file. I think all those points make things easy on contributors, minimize direct involvement of the Bro Team in sorting out problems related to particular plugins, and provide a useful way for users to discover and maintain Bro plugins. There?s more potential for users to encounter broken/bad plugins, but maybe that also encourages stronger community involvement w/ users more likely to try and help get problems resolved. - Jon From slagell at illinois.edu Tue May 24 09:49:22 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Tue, 24 May 2016 16:49:22 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <20160523215943.GR57255@icir.org> <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> Message-ID: <0A2DDFEE-245B-4E50-9D50-AB250277E89D@illinois.edu> > On May 24, 2016, at 11:21 AM, Siwek, Jon wrote: > > I think all those points make things easy on contributors, minimize direct involvement of the Bro Team in sorting out problems related to particular plugins, and provide a useful way for users to discover and maintain Bro plugins. There?s more potential for users to encounter broken/bad plugins, but maybe that also encourages stronger community involvement w/ users more likely to try and help get problems resolved. I don?t feel like we have converged on agreement regarding the balance of mandatory vs. optional checks. I think we need to define some basic metadata as a requirement for interoperability and discovery. Otherwise, what do we really end up providing above and beyond GitHub. Other quality checks can be optional, as long as we can change that in the future. I still think we should do do some basic checks to avoid completely broken stuff. It might mean more work for us in making sure we have good feedback and documentation. In general we all want to avoid human interaction becoming a bottleneck to submissions. I propose that we keep mandatory checks minimal, but not non-existent, and then we reevaluate when we have real data about how well this works. But I would really like more feedback from the community. Maybe I am an outlier here? ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From jsiwek at illinois.edu Tue May 24 10:07:11 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 24 May 2016 17:07:11 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <0FF909E5-D212-4FD4-A454-AE5117A9AA7D@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <3603B060-4909-4D25-888D-1A9B782AF5B7@illinois.edu> <20160523151607.GE57255@icir.org> <20160523224009.GS57255@icir.org> <0FF909E5-D212-4FD4-A454-AE5117A9AA7D@illinois.edu> Message-ID: <9E2F07E7-046E-4D40-8B91-B63AFD3CFD80@illinois.edu> > On May 23, 2016, at 6:30 PM, Slagell, Adam J wrote: > > I guess there is a balance here. If we do no mandatory checks and you could submit something that isn?t even a Bro plugin, the repository could become cluttered with junk. Do we really want things that don?t even ?compile?? The clutter could still be removed by an out-of-band process. e.g. there?s no initial check for whether a submission actually works, but after X days of a nightly process finding it is broken, it gets auto-removed. > However, where we can?t do that is with the metadata we collect. If we don?t require what we think is important metadata in the beginning, then we will have a gap if we decide it was important all along. So there I would err towards overcorrecting in the beginning, and make things optional in the future if it turns out not to be important. I think the most important metadata has to do w/ plugin interoperability (versioning and dependency info) and metadata that improves discoverability and search features is less important. For one reason, the former has a more objective correctness to it and the later is more subjective. Having wrong or missing discoverability metadata is also not going to cause things to break this missing interoperability data would. But even though I think interoperability metadata is important, I?m also not sure it needs to be collected/aggregated before plugin submissions are accepted ? it might be something the client can collect ?just in time? directly from a clone of the plugin itself. Even if a plugin doesn?t initially include this, the expected behavior could be for the cban client to use the plugin?s master branch and assume it will work w/ everything. If the user finds that not to be true, then they just uninstall it and ask the author to add proper versioning/dependency info or they might even try to add it themselves and submit the fixes back to the author. Metadata that helps improve discoverability and search features (topics/tags/keywords, author, license, etc) I don?t see becoming so important but underused to the point that you?d wish it were a requirement for submissions to be accepted. I don't expect adding metadata to be so much a burden that people avoid it entirely. Were there other reasons to think people won?t eventually add metadata info even if none is initially required? - Jon From slagell at illinois.edu Tue May 24 10:11:24 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Tue, 24 May 2016 17:11:24 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <9E2F07E7-046E-4D40-8B91-B63AFD3CFD80@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <3603B060-4909-4D25-888D-1A9B782AF5B7@illinois.edu> <20160523151607.GE57255@icir.org> <20160523224009.GS57255@icir.org> <0FF909E5-D212-4FD4-A454-AE5117A9AA7D@illinois.edu> <9E2F07E7-046E-4D40-8B91-B63AFD3CFD80@illinois.edu> Message-ID: <2C11724C-CE6B-4CAF-A8E4-38B976E43BBE@illinois.edu> > On May 24, 2016, at 12:07 PM, Siwek, Jon wrote: > >> >> On May 23, 2016, at 6:30 PM, Slagell, Adam J wrote: >> >> I guess there is a balance here. If we do no mandatory checks and you could submit something that isn?t even a Bro plugin, the repository could become cluttered with junk. Do we really want things that don?t even ?compile?? > > The clutter could still be removed by an out-of-band process. e.g. there?s no initial check for whether a submission actually works, but after X days of a nightly process finding it is broken, it gets auto-removed. That is a good point. I am more concerned about accumulating clutter. ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From jsiwek at illinois.edu Tue May 24 10:40:52 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 24 May 2016 17:40:52 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <0A2DDFEE-245B-4E50-9D50-AB250277E89D@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <20160523215943.GR57255@icir.org> <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> <0A2DDFEE-245B-4E50-9D50-AB250277E89D@illinois.edu> Message-ID: > On May 24, 2016, at 11:49 AM, Slagell, Adam J wrote: > > I propose that we keep mandatory checks minimal, but not non-existent, and then we reevaluate when we have real data about how well this works. But I would really like more feedback from the community. Maybe I am an outlier here? I think starting w/ either approach could end up evolving/devolving in to the other? If you had no checks in place, but then later instituted mandatory checks, you might be able to have the cban client not remove things a user has already checked out. So you can delist plugins if they fail the new checks, but users would still have the local version they can use (if somehow they?ve got it in a configuration that?s usable to them, but that doesn?t pass the new mandatory quality checks). I lean toward starting w/ the most streamlined and least complicated approach and seeing what quality control checks you need to layer on top of it because we might just expend a lot of effort planning for problems that don?t actual ever pop up in practice. But as a person that has to do development work on cban I might be biased toward doing what seems easier for me, so I?m fine not having a vote. - Jon From slagell at illinois.edu Tue May 24 11:11:13 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Tue, 24 May 2016 18:11:13 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <20160523215943.GR57255@icir.org> <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> <0A2DDFEE-245B-4E50-9D50-AB250277E89D@illinois.edu> Message-ID: <8C21701B-FF81-4487-8FA7-DC40E0D90119@illinois.edu> > On May 24, 2016, at 12:40 PM, Siwek, Jon wrote: > > I lean toward starting w/ the most streamlined and least complicated approach and seeing what quality control checks you need to layer on top of it because we might just expend a lot of effort planning for problems that don?t actual ever pop up in practice. But as a person that has to do development work on cban I might be biased toward doing what seems easier for me, so I?m fine not having a vote. I guess I?m not seeing more vs. less complicated; I see mandatory vs. optional being the difference. The important design goals here I think are: 1. Not block anything on a human if possible. 2. Be extensible so we can add future checks and change things between optional and mandatory. 3. Require as little information as needed, but no less. And these goals are really in service of the broader goals of having a useful repository with low barrier to entry. It is is these two goals that are in tension a bit. I?d prefer not to have anything in there that we know is broken, and I believe Robin is concerned that any blocks are going to require interaction on our part. I don?t think that is the case, but both of us are speculating with out data. I think compromises can be made as long as we have the flexibility to change approaches as things progress and we get data back. ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From jan.grashoefer at gmail.com Tue May 24 12:52:14 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Tue, 24 May 2016 21:52:14 +0200 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <20160523215943.GR57255@icir.org> <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> Message-ID: <89123223-d8e6-01e4-87ee-f7ffec57f490@gmail.com> > - it?s not a big deal for a submodule to temporarily enter in to a broken state ? cban users can always roll back to a previous version or just uninstall it. It?s up to the community to communicate/collaborate directly w/ the author here to get things fixed. I really like the community-centric approach. Regarding the discussion about checks and consistency I think that basically all conventions, that could be enforced automatically, will make the archive easier to work with (for authors and for end-users). But there is another thing that came to my mind: How are situations handled in which the author becomes the bottleneck? Imagine there was someone who published an awesome script but a new version of Bro breaks it. Another one patched the script and sends the patch to the original author. What will happen, in case he does not respond? Personally I don't like repositories which end up with entries like: "awesome-script", "awesome-script v2", "awesome-script by Jan" ... To avoid this one might consider to support forking plugins or organize the plugins user-centered ("jan/awesome-script", "anna/awesome-script"). Best regards, Jan From vallentin at icir.org Tue May 24 14:46:20 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 24 May 2016 14:46:20 -0700 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <2C11724C-CE6B-4CAF-A8E4-38B976E43BBE@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <3603B060-4909-4D25-888D-1A9B782AF5B7@illinois.edu> <20160523151607.GE57255@icir.org> <20160523224009.GS57255@icir.org> <0FF909E5-D212-4FD4-A454-AE5117A9AA7D@illinois.edu> <9E2F07E7-046E-4D40-8B91-B63AFD3CFD80@illinois.edu> <2C11724C-CE6B-4CAF-A8E4-38B976E43BBE@illinois.edu> Message-ID: <20160524214620.GX63455@shogun> (I will respond to the actual proposal in more depth later.) > That is a good point. I am more concerned about accumulating clutter. If I understold it correctly, I don't think that the central CBAN repository will accumulate clutter. The automatic checks will help simply age out repos that do not comply with the minimal standards. It's up to the devs to comply if the want to be integrated. More generally, there will presumably some functionality to add "remotes" to one's configuration, allowing plugin writers to point to experimental code if they wish. Then they can still hack out code and mix it with existing CBAN plugins, at their own risk. With a small linter in place, we would also lower the bar for devs to comply. Homebrew has a nice checker, for example. On a cosmetic note, will thing be called CBAN? I find it a very cryptic name, often confused it with BPAN (even though it doesn't make sense), and was wondering whether we should converge on some more pronounceable candidates. Matthias From slagell at illinois.edu Tue May 24 16:45:37 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Tue, 24 May 2016 23:45:37 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160524214620.GX63455@shogun> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <3603B060-4909-4D25-888D-1A9B782AF5B7@illinois.edu> <20160523151607.GE57255@icir.org> <20160523224009.GS57255@icir.org> <0FF909E5-D212-4FD4-A454-AE5117A9AA7D@illinois.edu> <9E2F07E7-046E-4D40-8B91-B63AFD3CFD80@illinois.edu> <2C11724C-CE6B-4CAF-A8E4-38B976E43BBE@illinois.edu> <20160524214620.GX63455@shogun> Message-ID: <93313B80-1B5E-4AA3-B302-C713CD3D6D59@illinois.edu> > On May 24, 2016, at 4:46 PM, Matthias Vallentin wrote: > > If I understold it correctly, I don't think that the central CBAN > repository will accumulate clutter. The automatic checks will help > simply age out repos that do not comply with the minimal standards. It's > up to the devs to comply if the want to be integrated. I think that?s fine, but that isn?t what I thought Robin was saying. I thought he did not want minimal standards. From robin at icir.org Tue May 24 22:12:13 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 24 May 2016 22:12:13 -0700 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <20160523215943.GR57255@icir.org> <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> Message-ID: <20160525051213.GA57255@icir.org> On Tue, May 24, 2016 at 16:21 +0000, you wrote: > Yeah, I have ideas, but seems like there might be another day of some > discussion before I try to formally reframe a design doc. Here?s the > direction I'm thinking: I like the process you sketch, that sounds like the right way to go to me. A few notes, also trying to address some of Adam's comments: > - the initial submission process involves doing a pull request on the > bro/cban repo where the only change made is the addition of a > submodule. These merges probably can be automated, but if a human > were to do it, I?d expect it wouldn?t be a time-consuming task Yeah, maybe that initial merge is one task we leave to a human, who could then actually take a quick 30sec look at the module to see if it's not totally off the mark. That would address Adam's point about what if somebody submits something that's not even a Bro thing (but I wouldn't go further; e.g., don't try to compile, etc.. Everything looking roughly right gets in at this time.) > - submodules that are found to have never been in a working state are > auto-removed (or could initially be a task that?s not a big deal for a > human to do every so often if metrics of brokenness are consistently > available) Auto-removal sounds dangerous to me; there may be different reasons why something's not in a good state. I'd leave cleanup to humans too: if there's a module that's consistently flagged as broken, that's when we can send a mail to the author and remove it manually if no improvement is in sight. I'd rather err on the side of having a broken module than remove something that could actually still be useful. > - metadata logically can be categorized in two types, one type is > related to discoverability (tags, author, license, etc) and one type > is related to interoperability (version number, dependencies) I wouldn't object to making some meta-data mandatory, per Adam's comments. For example enforcing having an author and a license would be useful I think. Author gets us contact information and license is always worth clarifying. > - discoverability metadata is aggregated during the nightly quality > check processes and automatically commits that information to the > ?bro/cban? repo. Would it be better to maintain this information outside of git in a state file that clients download? Otherwise the repository will clutter up quite a bit over time with tons of automatic commits. Overall, I agree that we can always add more restrictions later if it turns out necessary. It's not that we'll have 1000s of Bro modules in there within the first two weeks (as long as we prevent somebody spamming us). Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From slagell at illinois.edu Wed May 25 07:49:48 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Wed, 25 May 2016 14:49:48 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160525051213.GA57255@icir.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <20160523215943.GR57255@icir.org> <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu>, <20160525051213.GA57255@icir.org> Message-ID: > On May 25, 2016, at 12:12 AM, Robin Sommer wrote: > > Overall, I agree that we can always add more restrictions later if it > turns out necessary. It's not that we'll have 1000s of Bro modules in > there within the first two weeks (as long as we prevent somebody > spamming us So this has become an involved thread. Do we need a call, or do you think you can pull out enough to get started Jon? From jsiwek at illinois.edu Wed May 25 08:37:54 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 25 May 2016 15:37:54 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <89123223-d8e6-01e4-87ee-f7ffec57f490@gmail.com> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <20160523215943.GR57255@icir.org> <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> <89123223-d8e6-01e4-87ee-f7ffec57f490@gmail.com> Message-ID: > On May 24, 2016, at 2:52 PM, Jan Grash?fer wrote: > > > Imagine there was someone who published an awesome script but a new > version of Bro breaks it. Another one patched the script and sends the > patch to the original author. What will happen, in case he does not > respond? Personally I don't like repositories which end up with entries > like: "awesome-script", "awesome-script v2", "awesome-script by Jan" ... > To avoid this one might consider to support forking plugins or organize > the plugins user-centered ("jan/awesome-script", "anna/awesome-script"). I also think going with a hierarchical organization like that would help the situation you describe ? if an original author of a plugin becomes unresponsive then another person who wants to start maintaining it could ?fork? and submit that to cban, Then hopefully when a cban user is searching around it would be apparent that a submission got forked by other users in the community and the answer to why and which version they should use is a few ?cban info ? commands away. - Jon From jsiwek at illinois.edu Wed May 25 08:48:41 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 25 May 2016 15:48:41 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160524214620.GX63455@shogun> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <3603B060-4909-4D25-888D-1A9B782AF5B7@illinois.edu> <20160523151607.GE57255@icir.org> <20160523224009.GS57255@icir.org> <0FF909E5-D212-4FD4-A454-AE5117A9AA7D@illinois.edu> <9E2F07E7-046E-4D40-8B91-B63AFD3CFD80@illinois.edu> <2C11724C-CE6B-4CAF-A8E4-38B976E43BBE@illinois.edu> <20160524214620.GX63455@shogun> Message-ID: <2E278A1B-5427-4A07-ACEB-5C41DA07D6CA@illinois.edu> > On May 24, 2016, at 4:46 PM, Matthias Vallentin wrote: > > More generally, there will presumably some functionality to add > "remotes" to one's configuration, allowing plugin writers to point to > experimental code if they wish. Then they can still hack out code and > mix it with existing CBAN plugins, at their own risk. Yes, part of the plan is to allow one to add arbitrary git repos. i.e. repos that haven?t formally been submitted to CBAN, private ones, etc. > On a cosmetic note, will thing be called CBAN? I find it a very cryptic > name, often confused it with BPAN (even though it doesn't make sense), > and was wondering whether we should converge on some more pronounceable > candidates. I?m just using CBAN since that seems like the name that stuck around longest without anything better being proposed, so I think a rename isn?t out of the question if there?s something else everyone comes to like. - Jon From jsiwek at illinois.edu Wed May 25 09:03:29 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 25 May 2016 16:03:29 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160525051213.GA57255@icir.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <20160523215943.GR57255@icir.org> <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> <20160525051213.GA57255@icir.org> Message-ID: <10DF0D28-B10F-4A35-9374-02400A5C1A8D@illinois.edu> > On May 25, 2016, at 12:12 AM, Robin Sommer wrote: > >> - discoverability metadata is aggregated during the nightly quality >> check processes and automatically commits that information to the >> ?bro/cban? repo. > > Would it be better to maintain this information outside of git in a > state file that clients download? Otherwise the repository will > clutter up quite a bit over time with tons of automatic commits. Yeah, I like that idea. But where to host it? Pulling it from bro.org means an additional point of failure, so maybe it just all gets shoved into a separate gist file that lives on github? - Jon From jsiwek at illinois.edu Wed May 25 09:25:39 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 25 May 2016 16:25:39 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <20160523215943.GR57255@icir.org> <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> <20160525051213.GA57255@icir.org> Message-ID: > On May 25, 2016, at 9:49 AM, Slagell, Adam J wrote: > > So this has become an involved thread. Do we need a call, or do you think you can pull out enough to get started Jon? Yes I can reorganize all latest ideas into an alternate design document. Or if you meant get started working on the client, the open question about what metadata will be required has the most impact on how the client works, but like I said, I think things might also work if I start w/ the assumption that plugins are not required to present any metadata. - Jon From slagell at illinois.edu Wed May 25 10:04:24 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Wed, 25 May 2016 17:04:24 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <20160521224401.GL41563@icir.org> <20160523215943.GR57255@icir.org> <7141AE28-E709-4856-BC60-FD3FF65C333A@illinois.edu> <20160525051213.GA57255@icir.org> Message-ID: <6208B655-C297-4F76-847D-AE5EE040CBF8@illinois.edu> > On May 25, 2016, at 11:25 AM, Siwek, Jon wrote: > >> >> On May 25, 2016, at 9:49 AM, Slagell, Adam J wrote: >> >> So this has become an involved thread. Do we need a call, or do you think you can pull out enough to get started Jon? > > Yes I can reorganize all latest ideas into an alternate design document. That?s all I meant. Thanks. ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From jsiwek at illinois.edu Wed May 25 10:59:36 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Wed, 25 May 2016 17:59:36 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> Message-ID: <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> Here?s a revised/alternate design based on feedback so far: https://www.bro.org/development/projects/cban3.html Note that I put these at unique URLs and didn?t update in place since they?re different enough that, if someone tried to follow along from the start of the thread, I didn?t think it would make sense to them. - Jon From vern at icir.org Wed May 25 20:56:49 2016 From: vern at icir.org (Vern Paxson) Date: Wed, 25 May 2016 20:56:49 -0700 Subject: [Bro-Dev] Option -z In-Reply-To: <20160523185901.GQ57255@icir.org> (Mon, 23 May 2016 11:59:01 PDT). Message-ID: <20160526035649.83A462C41C0@rock.ICSI.Berkeley.EDU> > Does anybody remember what Bro's option -z is for? Well it's there in CHANGES, per the appended. But yeah looks like it never went anywhere beyond the original instigation, so I think removing it is okay. OTOH, it's a pretty handy general notion, so instead pushing it further strikes me as also reasonable. Vern 0.9a8 Wed Feb 16 17:09:34 PST 2005 .... - Bro now has a geneal mechanism internal for traversing policy scripts (Umesh Shankar). Various script analyses can be specified using the new -z flag. Currently, the one supported form of analysis is "-z notice", which prints all of the different types of notices that the script you've loaded can generate. For example, "bro -z notice ftp" will generate: Found NOTICE: BackscatterSeen Found NOTICE: FTP_PrivPort Found NOTICE: FTP_BadPort Found NOTICE: PortScan Found NOTICE: FTP_ExcessiveFilename Found NOTICE: ScanSummary Found NOTICE: AddressDropped Found NOTICE: DroppedPackets Found NOTICE: SensitiveConnection Found NOTICE: FTP_UnexpectedConn Found NOTICE: SSH_Overflow Found NOTICE: FTP_Sensitive Found NOTICE: TerminatingConnection Found NOTICE: PasswordGuessing Found NOTICE: AddressDropIgnored Found NOTICE: AddressScan From robin at icir.org Thu May 26 07:15:39 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 26 May 2016 07:15:39 -0700 Subject: [Bro-Dev] Option -z In-Reply-To: <20160526035649.83A462C41C0@rock.ICSI.Berkeley.EDU> References: <20160523185901.GQ57255@icir.org> <20160526035649.83A462C41C0@rock.ICSI.Berkeley.EDU> Message-ID: <20160526141539.GF66040@icir.org> On Wed, May 25, 2016 at 20:56 -0700, you wrote: > Well it's there in CHANGES, per the appended. But yeah looks like it never > went anywhere beyond the original instigation, so I think removing it is okay. Ah, I didn't realize this is what originally introduced the whole traversal machinery. That infrastructure is used in a few places now, and I'm not planing on touching that. Just removing this specific use of finding NOTICEs, which doesn't seem anybody has been using in a long time. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jazoff at illinois.edu Thu May 26 07:22:55 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Thu, 26 May 2016 14:22:55 +0000 Subject: [Bro-Dev] Option -z In-Reply-To: <20160526141539.GF66040@icir.org> References: <20160523185901.GQ57255@icir.org> <20160526035649.83A462C41C0@rock.ICSI.Berkeley.EDU> <20160526141539.GF66040@icir.org> Message-ID: > On May 26, 2016, at 10:15 AM, Robin Sommer wrote: > > > > On Wed, May 25, 2016 at 20:56 -0700, you wrote: > >> Well it's there in CHANGES, per the appended. But yeah looks like it never >> went anywhere beyond the original instigation, so I think removing it is okay. > > Ah, I didn't realize this is what originally introduced the whole > traversal machinery. That infrastructure is used in a few places now, > and I'm not planing on touching that. Just removing this specific use > of finding NOTICEs, which doesn't seem anybody has been using in a > long time. > > Robin It also has a minor issue that prevents it from being more useful, it outputs AddressScan instead of the fully namespaced Scan::AddressScan -- - Justin Azoff From vern at icir.org Thu May 26 07:41:34 2016 From: vern at icir.org (Vern Paxson) Date: Thu, 26 May 2016 07:41:34 -0700 Subject: [Bro-Dev] Option -z In-Reply-To: <20160526141539.GF66040@icir.org> (Thu, 26 May 2016 07:15:39 PDT). Message-ID: <20160526144134.56C812C4016@rock.ICSI.Berkeley.EDU> > Just removing this specific use > of finding NOTICEs, which doesn't seem anybody has been using in a > long time. I wonder if they don't use it because it's not on their radar. It's actually pretty handy, a way of telling when you think the set of NOTICEs should be X, but it's actually X'. Can help with writing documentation or finding dead code (of a form), or telling just what happens due to the hierarchy of @load's that a script pulls in. Vern From robin at icir.org Thu May 26 07:50:02 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 26 May 2016 07:50:02 -0700 Subject: [Bro-Dev] Option -z In-Reply-To: <20160526144134.56C812C4016@rock.ICSI.Berkeley.EDU> References: <20160526141539.GF66040@icir.org> <20160526144134.56C812C4016@rock.ICSI.Berkeley.EDU> Message-ID: <20160526145002.GL66040@icir.org> On Thu, May 26, 2016 at 07:41 -0700, you wrote: > I wonder if they don't use it because it's not on their radar. It's > actually pretty handy, I see that in principle but hardcoding the functionality in C++-land doesn't seem to be the ideal way to go about things like this. If one could express such analyses easily with a few lines of script code, that would be quite powerful for doing script inspection that's also easy to customize. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From vern at icir.org Thu May 26 07:56:57 2016 From: vern at icir.org (Vern Paxson) Date: Thu, 26 May 2016 07:56:57 -0700 Subject: [Bro-Dev] Option -z In-Reply-To: <20160526145002.GL66040@icir.org> (Thu, 26 May 2016 07:50:02 PDT). Message-ID: <20160526145657.D0AB52C40B1@rock.ICSI.Berkeley.EDU> > If one > could express such analyses easily with a few lines of script code, > that would be quite powerful for doing script inspection that's also > easy to customize. Well sure, but it's not clear one can get to that point without some significant work under the hood anyway in terms of the features needed to make the script-level expression a few lines of code. Vern From robin at icir.org Thu May 26 08:46:13 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 26 May 2016 08:46:13 -0700 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> Message-ID: <20160526154613.GN66040@icir.org> On Wed, May 25, 2016 at 17:59 +0000, you wrote: > https://www.bro.org/development/projects/cban3.html Looks great to me. Unsorted thoughts/notes: - Let's keep BroControl integration in mind at leat. I agree that a standalone client makes most sense right now, but if there's something we can do that will make it easier for BroControl later to integrate, that's worth preparing for. - One idea along those integration lines (BroControl and even elsewhere): would be nice to structure the client so that the main functionality resides in a Python module with a well-defined API. The client itself would then become just a small command-line frontend to that library. - Where's that backend-side "out-of-band process" located? Is that part of the client as well? Or maybe at least part of that Python module? Would be nice to keep it easy to operate for CBAN-like repositories that people maintain externally. - having both "upgrade" vs "update" commands sounds confusing, I'm sure I would constantly mix up the two. Suggest to rename one of them. - What's the policy for namespaces (both script-land and plugin-land), and what's the relationship to the CBAN path? Should people use their GitHub name as namespace for their module? On the one hand that would be an easy way to avoid conflicts. On the other hand that makes forking difficult. - Originally I wanted to suggest allowing more than one plugin per git repository but I really like the idea to leverage submodules, so I'm skipping that suggestion. :-) - How do updating a plugin works now? If the author just updates the remote code, the submodules won't move, so is there an additional step there? - Using a gist for meta data sounds good. We should then also have a nice permanent *.bro.org URL redirecting there so that we hide the actual location a bit for easier relocation. - Any thoughts on distributing precompiled plugins? It would be nice if, say, I could just install the Mac version of plugin XYZ without having to compile it locally. The authors could optionally offer selected prebuilds for whatever platforms they want to support. Probably more a a feature for CBAN v2, but would be wort keeping mind how binaries would fit in. - Another maybe-v2 idea: if a plugin listed its configuration options in a well-defined manner, tools like BroControl could pick up on that and offer a configuration UI without peopling having to write script code. - Terminology 1: agree that we should find a better name for CBAN. - Terminology 2: Using "plugin" as the entity name for everything is confusing I think, as right now I think most people understand it as something that gets compiled; nobody calls a script a plugin (unless it's a plugin for a specific script-framework; but that's even more confusing then :). The best alternative names I can come up with are "module" or "package", which are also overloaded already, but at least more generic. Other suggestions? - We could create a public mailing list for CBAN for all discussions of individual plugins/modules, including in particular for decisions to remove something. Would be good to keep this all open and transparent. - I'm offended that my plugin is just "nice", but Seth's is *cool*! Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From slagell at illinois.edu Thu May 26 08:57:02 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Thu, 26 May 2016 15:57:02 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160526154613.GN66040@icir.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> <20160526154613.GN66040@icir.org> Message-ID: <90C28A1A-F119-4999-84B1-F27E6681A801@illinois.edu> > On May 26, 2016, at 10:46 AM, Robin Sommer wrote: > > - Terminology 1: agree that we should find a better name for CBAN. BroForge? > > - Terminology 2: Using "plugin" as the entity name for everything is > confusing I think, as right now I think most people understand it as > something that gets compiled; nobody calls a script a plugin (unless > it's a plugin for a specific script-framework; but that's even more > confusing then :). The best alternative names I can come up with are > "module" or "package", which are also overloaded already, but at > least more generic. Other suggestions? > But we are wrapping everything in the Bro plugin architecture, right? I guess I I don?t think of plugins as necessarily binary, but maybe we shift nomenclature completely and start calling plugins packages across our documentation. But if we refer to the same thing as a plugin sometimes and a package other times, I think it gets even more confusing. > - We could create a public mailing list for CBAN for all discussions > of individual plugins/modules, including in particular for decisions > to remove something. Would be good to keep this all open and > transparent. I like the idea of a CBAN list. ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From jdopheid at illinois.edu Thu May 26 09:50:42 2016 From: jdopheid at illinois.edu (Dopheide, Jeannette M) Date: Thu, 26 May 2016 16:50:42 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <90C28A1A-F119-4999-84B1-F27E6681A801@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> <20160526154613.GN66040@icir.org> <90C28A1A-F119-4999-84B1-F27E6681A801@illinois.edu> Message-ID: CBAN is a good name. It?s short, easy to say, and what the acronym means, the ?Comprehensive Bro Archive Network?, makes sense. But I don?t really have a dog in this fight. ------ Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign On 5/26/16, 10:57 AM, "bro-dev-bounces at bro.org on behalf of Slagell, Adam J" wrote: > On May 26, 2016, at 10:46 AM, Robin Sommer wrote: > > - Terminology 1: agree that we should find a better name for CBAN. BroForge? From jsiwek at illinois.edu Thu May 26 10:41:47 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Thu, 26 May 2016 17:41:47 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160526154613.GN66040@icir.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> <20160526154613.GN66040@icir.org> Message-ID: <8280EF72-472C-411B-877C-A8C1ADBA6671@illinois.edu> > On May 26, 2016, at 10:46 AM, Robin Sommer wrote: > > - Let's keep BroControl integration in mind at leat. I agree that a > standalone client makes most sense right now, but if there's > something we can do that will make it easier for BroControl later to > integrate, that's worth preparing for. > > - One idea along those integration lines (BroControl and even > elsewhere): would be nice to structure the client so that the main > functionality resides in a Python module with a well-defined API. > The client itself would then become just a small command-line > frontend to that library. Ack. > - Where's that backend-side "out-of-band process" located? Is that > part of the client as well? Or maybe at least part of that Python > module? Would be nice to keep it easy to operate for CBAN-like > repositories that people maintain externally. I imagined it being part of the nightly process that does quality/metric gathering. The code to do it could be part of the client, but I expect the aggregation process could be time consuming enough that a user wouldn?t be happy w/ it. Especially if CBAN grows to contain many submodules, I don?t think it would scale well for each client to have to do the metadata aggregation themselves. Is it an important use case for the client to be able to interact w/ other repos that are structured like the one the Bro Team will be hosting? Seems less likely that people will want to do that especially if the CBAN repository is easy enough for people to submit to and the client can handle the case of adding individual external repos just fine. The metadata from individual, externally added repos would be searchable along with the bulk/aggregated metadata. e.g. all the submodules a user has locally installed have the latest metadata available, but all the submodules the user has not installed will fall back on the aggregated metadata that was fetched the last time they ran an ?update? command. > - having both "upgrade" vs "update" commands sounds confusing, I'm > sure I would constantly mix up the two. Suggest to rename one of > them. I do confuse them too, but it?s also what some other package managers use (e.g. home-brew and apt-get). I?ll look at others package managers for more examples to see if I find something. > - What's the policy for namespaces (both script-land and plugin-land), > and what's the relationship to the CBAN path? Should people use > their GitHub name as namespace for their module? On the one hand > that would be an easy way to avoid conflicts. On the other hand that > makes forking difficult. Didn?t really have plans/ideas for a namespace policy, but checking for conflicts could be added to the list of things the automated nightly quality/metrics thing checks for. But maybe there?s a slicker thing you could add to the Bro language itself to let users specify a custom namespace mapping for certain file paths allowing them to resolve conflicts themselves. > - How do updating a plugin works now? If the author just updates the > remote code, the submodules won't move, so is there an additional > step there? The client's ?update? command will do a ?git pull? in the parent repo and then a ?git fetch? in any installed submodules. Any submodules that are not installed are a no-op because they are just pointers that got updated by the ?git pull?. This means the client should now be aware of updates to all the plugins they have installed, but are still using the old version until they choose to perform the ?upgrade? command which then moves the installed submodules to the latest version. Or at least it will look something like that. I haven?t had a chance to test if there will need to be a separate ?working copy? of submodules the user chooses to install. > - Any thoughts on distributing precompiled plugins? It would be nice > if, say, I could just install the Mac version of plugin XYZ without > having to compile it locally. The authors could optionally offer > selected prebuilds for whatever platforms they want to support. > Probably more a a feature for CBAN v2, but would be wort keeping > mind how binaries would fit in. Didn?t think much about it yet, but I expect a plugin to be able to define its own build steps. For precompiled plugins they could likely just do a no-op for the build step. Maybe there?s other install paths that need to get set up that I?m forgetting, but that shouldn't be too much extra to get working. Let me know if you had other thoughts that would make this more complicated than I?m expecting. > - Terminology 2: Using "plugin" as the entity name for everything is > confusing I think, as right now I think most people understand it as > something that gets compiled; nobody calls a script a plugin (unless > it's a plugin for a specific script-framework; but that's even more > confusing then :). The best alternative names I can come up with are > "module" or "package", which are also overloaded already, but at > least more generic. Other suggestions? Don?t have other suggestions, but ?package? seems fitting because the concept of ?package management? is an established term that people are familiar with. > - I'm offended that my plugin is just "nice", but Seth's is *cool*! Maybe some people prefer ?nice? to ?cool?. BTW, originally you had just ?some-plugin?, but I changed it. - Jon From vallentin at icir.org Thu May 26 14:07:16 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Thu, 26 May 2016 14:07:16 -0700 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160526154613.GN66040@icir.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> <20160526154613.GN66040@icir.org> Message-ID: <20160526210716.GD2796@samurai.ICIR.org> > - having both "upgrade" vs "update" commands sounds confusing, I'm > sure I would constantly mix up the two. Suggest to rename one of > them. I think this comes from Homebrew (and maybe other package managers as well). I kinda got used to it in this context, but occasionally still trip over it. So yeah, I also think we should use either "update" or "upgrade," but not both. > - What's the policy for namespaces (both script-land and plugin-land), > and what's the relationship to the CBAN path? Should people use > their GitHub name as namespace for their module? On the one hand > that would be an easy way to avoid conflicts. On the other hand that > makes forking difficult. Is it necessary to enforce this? Intuitively, I would just leave namespacing to the user. Clearly, to be interoperable, a convention could be using your github handle, but I don't see it being a necessity. If multiple devs want to collaborate and enhance the same namespace, we should not make this a cumbersome due to namespace hassles. > - Originally I wanted to suggest allowing more than one plugin per git > repository but I really like the idea to leverage submodules, so I'm > skipping that suggestion. :-) How do submodules scale? Or asked differently, what are the number (order of magnitude) of submodules we're expecting? > - Using a gist for meta data sounds good. We should then also have a > nice permanent *.bro.org URL redirecting there so that we hide the > actual location a bit for easier relocation. Is the idea that all meta data will reside in a single file? What format would that file be, YAML? If each user repo has 20 lines of meta data, then the file would be 2KB to track 100 repositories. Seems fine to me, but I wonder how much will end up in there to get a tractable upper bound. > - Any thoughts on distributing precompiled plugins? It would be nice > if, say, I could just install the Mac version of plugin XYZ without > having to compile it locally. The authors could optionally offer > selected prebuilds for whatever platforms they want to support. > Probably more a a feature for CBAN v2, but would be wort keeping > mind how binaries would fit in. Binaries are a big project, I believe. Not only do we need to support differently platforms, but also different OS versions and distributions. It would probably mean we have to invest into a bigger Jenkins setup, and then push the binaries into a CDN after a successful build. > - Terminology 1: agree that we should find a better name for CBAN. Here are some bro'ish suggestions (not all are serious ones): - brow (part of the logo already) - broom (sweep new code into bro) - broil (Bake your enhancements into bro) - broad (extends Bro in a broader sense) - broem (pleasing in a literary manner) - bropane (hot stuff for your bro) - brorito (original scripts from Mexico!) - brotein (makes you grow like a bro) - hebro (how about catering to Isreal?) - bronx (or to NYC?) And some random word forging: - bkg (instead of FreeBSD's pkg) - lion (animal I like) - crank (crank up that Bro) - scrit (tweak on "script") - berk (a Bro perk...or short for Berkeley) - bip (pip for Bro) - bang (starts with a "b" and sounds hip) - beco (Bro ecosystem) > The best alternative names I can come up with are "module" or > "package", which are also overloaded already, but at least more > generic. Of the two, I prefer "module" because "package" to me reminds me of a compiled thing, which is not always the case. Other than that, "extension" comes to mind, or maybe "bundle." > - We could create a public mailing list for CBAN for all discussions > of individual plugins/modules, including in particular for decisions > to remove something. Would be good to keep this all open and > transparent. ...and a Gitter channel, once we have converged on a name :-). Matthias From jan.grashoefer at gmail.com Thu May 26 14:42:20 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Thu, 26 May 2016 23:42:20 +0200 Subject: [Bro-Dev] Configurable &write_expire interval In-Reply-To: <57050AFC.1000609@gmail.com> References: <57050AFC.1000609@gmail.com> Message-ID: Hi, I ran into an issue while trying to make the &write_expire interval configurable: Using a redefable constant does not work here, as the expression only gets evaluated when the table is initialized and thus later redefs do not influence the value. I thought about circumventing this by setting the value to 0 and maintain an extra variable to check against in my expire_func and return the right value. Unfortunately this won't work with write/read_expire as a write or read will reset the expiration to the initial value of 0. A solution could be to evaluate the interval expression every time it is used inside the table implementation. The drawback would be that there is no fixed value for serialization (I am not sure about the effects here). Another solution would be to provide a bif (or implement a language feature) to change the expire_time value from inside the expire_func. There was a somehow similar discussion about per item expiration (see http://mailman.icsi.berkeley.edu/pipermail/bro-dev/2016-April/011731.html) in which Robin came up with the solution of multiple tables with different expiration values. Again this would be a solution but doesn't feel right (duplicate code, static and somehow counterintuitive for the user). By the way: This is again about intel expiration. This time I thought I'll keep it simple and just have one user-defined expiration value for all items :D Maybe I am missing something regarding the loading sequence of scripts and this problem could be solved easier. So I am open for any suggestions or feedback! Best regards, Jan From jan.grashoefer at gmail.com Thu May 26 14:51:42 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Thu, 26 May 2016 23:51:42 +0200 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160526210716.GD2796@samurai.ICIR.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> <20160526154613.GN66040@icir.org> <20160526210716.GD2796@samurai.ICIR.org> Message-ID: <833a58a7-fccf-b014-104c-658d9200258f@gmail.com> > Here are some bro'ish suggestions (not all are serious ones): > > - brow (part of the logo already) > - broom (sweep new code into bro) > - broil (Bake your enhancements into bro) > - broad (extends Bro in a broader sense) > - broem (pleasing in a literary manner) > - bropane (hot stuff for your bro) > - brorito (original scripts from Mexico!) > - brotein (makes you grow like a bro) > - hebro (how about catering to Isreal?) > - bronx (or to NYC?) > > And some random word forging: > > - bkg (instead of FreeBSD's pkg) > - lion (animal I like) > - crank (crank up that Bro) > - scrit (tweak on "script") > - berk (a Bro perk...or short for Berkeley) > - bip (pip for Bro) > - bang (starts with a "b" and sounds hip) > - beco (Bro ecosystem) I like bkg, broil and brow. What about bob (helps you building up you Bro ;)? From slagell at illinois.edu Thu May 26 17:39:59 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Fri, 27 May 2016 00:39:59 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <833a58a7-fccf-b014-104c-658d9200258f@gmail.com> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> <20160526154613.GN66040@icir.org> <20160526210716.GD2796@samurai.ICIR.org>, <833a58a7-fccf-b014-104c-658d9200258f@gmail.com> Message-ID: <0C1F9A9E-FE3B-4048-AC9C-0BF783BAC6C8@illinois.edu> Or just bro-ports On May 26, 2016, at 4:52 PM, Jan Grash?fer wrote: >> Here are some bro'ish suggestions (not all are serious ones): >> >> - brow (part of the logo already) >> - broom (sweep new code into bro) >> - broil (Bake your enhancements into bro) >> - broad (extends Bro in a broader sense) >> - broem (pleasing in a literary manner) >> - bropane (hot stuff for your bro) >> - brorito (original scripts from Mexico!) >> - brotein (makes you grow like a bro) >> - hebro (how about catering to Isreal?) >> - bronx (or to NYC?) >> >> And some random word forging: >> >> - bkg (instead of FreeBSD's pkg) >> - lion (animal I like) >> - crank (crank up that Bro) >> - scrit (tweak on "script") >> - berk (a Bro perk...or short for Berkeley) >> - bip (pip for Bro) >> - bang (starts with a "b" and sounds hip) >> - beco (Bro ecosystem) > > I like bkg, broil and brow. What about bob (helps you building up you > Bro ;)? > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev From robin at icir.org Thu May 26 20:54:26 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 26 May 2016 20:54:26 -0700 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160526210716.GD2796@samurai.ICIR.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> <20160526154613.GN66040@icir.org> <20160526210716.GD2796@samurai.ICIR.org> Message-ID: <20160527035426.GZ66040@icir.org> On Thu, May 26, 2016 at 14:07 -0700, you wrote: > Is it necessary to enforce this? Intuitively, I would just leave > namespacing to the user. No need to enforce, but it would be good to have some guidelines at least. And part of the guidelines should be keeping the "Bro" namespace reserved. > "extension" comes to mind, or maybe "bundle." I like "bundle". Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Thu May 26 21:01:06 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 26 May 2016 21:01:06 -0700 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <90C28A1A-F119-4999-84B1-F27E6681A801@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> <20160526154613.GN66040@icir.org> <90C28A1A-F119-4999-84B1-F27E6681A801@illinois.edu> Message-ID: <20160527040106.GA66040@icir.org> On Thu, May 26, 2016 at 15:57 +0000, you wrote: > But we are wrapping everything in the Bro plugin architecture, right? I think we'll need to play with that a bit still to see how exactly a minimal script module would be distributed. The binary plugins require a bit more structure, and are loaded a bit differently, than something that has just scripts would need. Maybe we can simplify the plugin layout a bit to support that case better, not sure. That said, even if using all the same structure, in my head I still associate "plugin" with one of the binary extensions (log writer, packet source) etc. and that's also how the documentation uses that term currently. Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From robin at icir.org Thu May 26 21:14:05 2016 From: robin at icir.org (Robin Sommer) Date: Thu, 26 May 2016 21:14:05 -0700 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <8280EF72-472C-411B-877C-A8C1ADBA6671@illinois.edu> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> <20160526154613.GN66040@icir.org> <8280EF72-472C-411B-877C-A8C1ADBA6671@illinois.edu> Message-ID: <20160527041405.GB66040@icir.org> On Thu, May 26, 2016 at 17:41 +0000, you wrote: > I imagined it being part of the nightly process that does quality/metric gathering. Yeah, makes sense. > Is it an important use case for the client to be able to interact w/ > other repos that are structured like the one the Bro Team will be > hosting? Seems less likely that people will want to do that > especially if the CBAN repository is easy enough for people to submit Yeah, agree it's less important with that liberal model. I would still like to support it though unless it's a major pain. > But maybe there?s a slicker thing you could add to the Bro language > itself to let users specify a custom namespace mapping for certain > file paths allowing them to resolve conflicts themselves. Conflict reporting sounds good, renaming could get confusing. > The client's ?update? command will do a ?git pull? in the parent repo > and then a ?git fetch? in any installed submodules. So does that mean the client ignores the version that the central CBAN parent repository points to? Say Seth finished version 1.0 of his cool-plugin and files a merge request. We add the submodule to CBAN; it will point to 1.0. Then Seth updates to 1.1 on his end. CBAN would still point to 1.0, but clients would just move their local clones to 1.1, ignoring the parent repository's state. Is that what you have in mind? Robin -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin From jsiwek at illinois.edu Fri May 27 08:58:44 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 27 May 2016 15:58:44 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160526210716.GD2796@samurai.ICIR.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> <20160526154613.GN66040@icir.org> <20160526210716.GD2796@samurai.ICIR.org> Message-ID: <15DF89C5-F80F-4B42-ABE3-3F92ACFB08A4@illinois.edu> > On May 26, 2016, at 4:07 PM, Matthias Vallentin wrote: > > How do submodules scale? Or asked differently, what are the number > (order of magnitude) of submodules we're expecting? I imagine it will scale because when a user clones the main/parent CBAN repo, we?re telling them to just clone that and *not* recursively clone all the submodules (unlike the process for the ?bro? repo that you?re familiar with). Then the user will pick and choose the individual submodules they want. I don?t know much the average user will end up installing, but I?m guessing in the order of 10s and not 100s. Does that make sense or is there something more to your concern I might be missing or misunderstood? > Is the idea that all meta data will reside in a single file? What format > would that file be, YAML? If each user repo has 20 lines of meta data, > then the file would be 2KB to track 100 repositories. Seems fine to me, > but I wonder how much will end up in there to get a tractable upper > bound. I didn?t have a standardized format in mind to begin with, I just planned to do ?the simplest thing that works? first and then let it evolve from there (clients can be updated in a way that?s resilient to changes in the format), but if enough people want to start w/ a particular format to begin with that?s fine too. > Binaries are a big project, I believe. Not only do we need to support > differently platforms, but also different OS versions and distributions. > It would probably mean we have to invest into a bigger Jenkins setup, > and then push the binaries into a CDN after a successful build. I imagined it would still be left up to individual authors to make precompiled binaries available and support/maintain them. It would also be optional for them to make binaries available, not required. - Jon From jsiwek at illinois.edu Fri May 27 09:20:28 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Fri, 27 May 2016 16:20:28 +0000 Subject: [Bro-Dev] CBAN design proposal In-Reply-To: <20160527041405.GB66040@icir.org> References: <4B35669A-863B-45F6-9C0B-8D909E1082AE@illinois.edu> <4276CDDC-47A9-412C-A05A-2F6C9877B163@illinois.edu> <20160526154613.GN66040@icir.org> <8280EF72-472C-411B-877C-A8C1ADBA6671@illinois.edu> <20160527041405.GB66040@icir.org> Message-ID: > On May 26, 2016, at 11:14 PM, Robin Sommer wrote: > >> Is it an important use case for the client to be able to interact w/ >> other repos that are structured like the one the Bro Team will be >> hosting? Seems less likely that people will want to do that >> especially if the CBAN repository is easy enough for people to submit > > Yeah, agree it's less important with that liberal model. I would still > like to support it though unless it's a major pain. Ok, I?ll plan from the beginning to be able point to multiple CBAN-like repository structures and then complain if I find out it?s complicated to support that. >> The client's ?update? command will do a ?git pull? in the parent repo >> and then a ?git fetch? in any installed submodules. > > So does that mean the client ignores the version that the central CBAN > parent repository points to? Say Seth finished version 1.0 of his > cool-plugin and files a merge request. We add the submodule to CBAN; > it will point to 1.0. Then Seth updates to 1.1 on his end. CBAN would > still point to 1.0, but clients would just move their local clones to > 1.1, ignoring the parent repository's state. Is that what you have in > mind? Yeah, the parent repo would always point to the first version that was submitted, but when a user chooses to install something, that submodule gets initialized/cloned and the user can choose to use whatever version of it they want. And it won?t be the common case for a user to be doing any actual work inside the parent repo (unless they?re working on the cban client code), so it shouldn?t be a big deal if things like `git status` in the parent repo show a bunch of the ?new commits? messages for each installed submodule. But I?ll also probably structure it so all submodules get cloned in to a subdirectory and then list that subdirectory in .gitignore to limit the noise. - Jon From vallentin at icir.org Fri May 27 10:00:45 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 27 May 2016 10:00:45 -0700 Subject: [Bro-Dev] CBAN naming Message-ID: <20160527170045.GG2796@samurai.ICIR.org> To find the new name for our CBAN project, it probably make sense to brainstorm separately from the existing technical thread. I'd say let's collect some candidates and then create survey to vote on them. Here are some ideas from the existing thread: - brow - broil - broom - bpk - berk - bob - bip Looking forward to hear your ideas. Matthias From vallentin at icir.org Fri May 27 10:35:23 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 27 May 2016 10:35:23 -0700 Subject: [Bro-Dev] Broker status update In-Reply-To: <20160523152326.GG57255@icir.org> References: <20160521160132.GA89196@shogun> <20160523152326.GG57255@icir.org> Message-ID: <20160527173523.GJ2796@samurai.ICIR.org> > > peer(x, y); // Create a peering between the two endpoints. > > peer(y, x); // Idempotent. Peerings are symmetric. > > > x.peer(y); // Create a peering between the two endpoints. > > y.peer(x); // Idempotent. Peerings are symmetric. > > I would prefer the 2nd way for consistenct, as all the other > operations use the method-based scheme. The idempotency seems > secondary to that I would say. Okay, after mulling over this a bit more, I think the second version makes sense also for another reason: overloads. Both of these statements should be valid: x.peer("127.0.0.1", 42000); x.peer(y); With a free function, this becomes not as clear anymore. > Related question: what exactly are the semantics if only one side of > the peering is set up? The peering occurs asynchronously. The function endpoint::peer returns void, and when the peering result (failure/success) becomes available, a dedicated status message will be enqueued in the endpoint's mailbox. This message has to be extracted via endpoint::receive, or in the asynchronous case, there needs to be handler for the status present (otherwise the message gets dropped). > Hmm ... I see the appeal but it would introduce a new dependency and > its Python-specific (I assume), whereas with SWIG it's easier to add > more languages later. Is that worth the benefit of switching? We already need SWIG and its Python-specific bindings, which are two separate packages. Since every language has its idiosyncrasies, it's quite unwieldy to have one framework to rule them all. At least SWIG feels not like a productive toolkit for modern C++. Also, since pybind11 is a small header-only package, we can easily ship it directly with Broker, which would render the existing package dependencies obsolete. That is, we would not introducing a new dependency, but remove two instead ;-). That said, this is my impression after just briefly looking at the bindings space. I'll perform a more thorough evaluation once I'll actually start delving into writing the bindings. Matthias From jazoff at illinois.edu Fri May 27 10:37:46 2016 From: jazoff at illinois.edu (Azoff, Justin S) Date: Fri, 27 May 2016 17:37:46 +0000 Subject: [Bro-Dev] CBAN naming In-Reply-To: <20160527170045.GG2796@samurai.ICIR.org> References: <20160527170045.GG2796@samurai.ICIR.org> Message-ID: <6E58CD20-0905-4468-B79A-B68665C05F8E@illinois.edu> > On May 27, 2016, at 1:00 PM, Matthias Vallentin wrote: > > To find the new name for our CBAN project, it probably make sense to > brainstorm separately from the existing technical thread. I'd say let's > collect some candidates and then create survey to vote on them. > > Here are some ideas from the existing thread: > > - brow > - broil > - broom > - bpk > - berk > - bob > - bip I don't have much of an opinion in the naming, just that we should avoid using a name that already exists. I checked for collisions in debian(package names and filenames under *bin/) the only tool that currently exists is 'bip' which is an irc proxy. There is a chef related tool called berkshelf: https://github.com/berkshelf/berkshelf which installs a 'berks' command One thing that did come up was this: https://github.com/CompEvol/CBAN: Comprehensive BEAST Archive Network which doesn't seem terribly popular, but is actively in use. bpkg is another obvious name that is already in use: https://github.com/bpkg -- - Justin Azoff From anthony.kasza at gmail.com Fri May 27 13:50:12 2016 From: anthony.kasza at gmail.com (anthony kasza) Date: Fri, 27 May 2016 13:50:12 -0700 Subject: [Bro-Dev] CBAN naming In-Reply-To: <6E58CD20-0905-4468-B79A-B68665C05F8E@illinois.edu> References: <20160527170045.GG2796@samurai.ICIR.org> <6E58CD20-0905-4468-B79A-B68665C05F8E@illinois.edu> Message-ID: Just my opinion... I like how Spicy was named, by choosing something completely different and unrelated to the "bro" theme. Or call it frat house. -AK On May 27, 2016 10:38 AM, "Azoff, Justin S" wrote: > > > On May 27, 2016, at 1:00 PM, Matthias Vallentin > wrote: > > > > To find the new name for our CBAN project, it probably make sense to > > brainstorm separately from the existing technical thread. I'd say let's > > collect some candidates and then create survey to vote on them. > > > > Here are some ideas from the existing thread: > > > > - brow > > - broil > > - broom > > - bpk > > - berk > > - bob > > - bip > > I don't have much of an opinion in the naming, just that we should avoid > using a name that already exists. > > I checked for collisions in debian(package names and filenames under > *bin/) the only tool that currently exists is 'bip' which is an irc proxy. > > There is a chef related tool called berkshelf: > https://github.com/berkshelf/berkshelf which installs a 'berks' command > > One thing that did come up was this: > > https://github.com/CompEvol/CBAN: Comprehensive BEAST Archive Network > > which doesn't seem terribly popular, but is actively in use. > > bpkg is another obvious name that is already in use: > https://github.com/bpkg > > > > -- > - Justin Azoff > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160527/fa863f8c/attachment.html From vallentin at icir.org Fri May 27 17:24:18 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Fri, 27 May 2016 17:24:18 -0700 Subject: [Bro-Dev] CBAN naming In-Reply-To: References: <20160527170045.GG2796@samurai.ICIR.org> <6E58CD20-0905-4468-B79A-B68665C05F8E@illinois.edu> Message-ID: <20160528002418.GM2796@samurai.ICIR.org> > I like how Spicy was named, by choosing something completely different and > unrelated to the "bro" theme. I like that too. Do you have a suggestion? Matthias From anthony.kasza at gmail.com Fri May 27 17:38:32 2016 From: anthony.kasza at gmail.com (anthony kasza) Date: Fri, 27 May 2016 17:38:32 -0700 Subject: [Bro-Dev] CBAN naming In-Reply-To: <20160528002418.GM2796@samurai.ICIR.org> References: <20160527170045.GG2796@samurai.ICIR.org> <6E58CD20-0905-4468-B79A-B68665C05F8E@illinois.edu> <20160528002418.GM2796@samurai.ICIR.org> Message-ID: Oh man. I don't know. Call it the colossal. -AK On May 27, 2016 5:24 PM, "Matthias Vallentin" wrote: > > I like how Spicy was named, by choosing something completely different > and > > unrelated to the "bro" theme. > > I like that too. Do you have a suggestion? > > Matthias > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160527/e26413c2/attachment.html From anthony.kasza at gmail.com Fri May 27 17:46:08 2016 From: anthony.kasza at gmail.com (anthony kasza) Date: Fri, 27 May 2016 17:46:08 -0700 Subject: [Bro-Dev] CBAN naming In-Reply-To: References: <20160527170045.GG2796@samurai.ICIR.org> <6E58CD20-0905-4468-B79A-B68665C05F8E@illinois.edu> <20160528002418.GM2796@samurai.ICIR.org> Message-ID: Besides naming and skipping ahead to implementation, I highly recommend looking at how Rust manages crates. They are a pleasure to use and work with. -AK On May 27, 2016 5:38 PM, "anthony kasza" wrote: > Oh man. I don't know. Call it the colossal. > > -AK > On May 27, 2016 5:24 PM, "Matthias Vallentin" wrote: > >> > I like how Spicy was named, by choosing something completely different >> and >> > unrelated to the "bro" theme. >> >> I like that too. Do you have a suggestion? >> >> Matthias >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160527/80710c86/attachment.html From slagell at illinois.edu Fri May 27 18:17:51 2016 From: slagell at illinois.edu (Slagell, Adam J) Date: Sat, 28 May 2016 01:17:51 +0000 Subject: [Bro-Dev] CBAN naming In-Reply-To: <20160527170045.GG2796@samurai.ICIR.org> References: <20160527170045.GG2796@samurai.ICIR.org> Message-ID: <6BB7568F-3B00-40C8-A4DD-E681B6464060@illinois.edu> Think the current name is fine, but if you change it I think it helps if it relates to things people know. So names like bpkg, bro-ports, or bro-brew would give the immediate analogy through the name. > On May 27, 2016, at 12:00 PM, Matthias Vallentin wrote: > > To find the new name for our CBAN project, it probably make sense to > brainstorm separately from the existing technical thread. I'd say let's > collect some candidates and then create survey to vote on them. > > Here are some ideas from the existing thread: > > - brow > - broil > - broom > - bpk > - berk > - bob > - bip > > Looking forward to hear your ideas. > > Matthias > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev ------ Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." From jan.grashoefer at gmail.com Sat May 28 02:39:30 2016 From: jan.grashoefer at gmail.com (=?UTF-8?Q?Jan_Grash=c3=b6fer?=) Date: Sat, 28 May 2016 11:39:30 +0200 Subject: [Bro-Dev] CBAN naming In-Reply-To: <6BB7568F-3B00-40C8-A4DD-E681B6464060@illinois.edu> References: <20160527170045.GG2796@samurai.ICIR.org> <6BB7568F-3B00-40C8-A4DD-E681B6464060@illinois.edu> Message-ID: <43f3ce86-2869-877f-ae66-5040f145694d@gmail.com> > Think the current name is fine, but if you change it I think it helps if it relates to things people know. So names like bpkg, bro-ports, or bro-brew would give the immediate analogy through the name. True. That's why I somehow like brow. It relates to the logo and is memorable but could also be seen as a merge of bro-brew. By the way: Are we talking about renaming the whole project? I already get used to CBAN and calling the project CBAN does not necessarily imply that e.g. the CBAN-client could be named brow (although that might make things not that intuitive). Best regards, Jan From vallentin at icir.org Mon May 30 19:39:03 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 30 May 2016 19:39:03 -0700 Subject: [Bro-Dev] CBAN naming In-Reply-To: References: <20160527170045.GG2796@samurai.ICIR.org> <6E58CD20-0905-4468-B79A-B68665C05F8E@illinois.edu> <20160528002418.GM2796@samurai.ICIR.org> Message-ID: <20160531023903.GA1312@shogun.local> > Besides naming and skipping ahead to implementation, I highly recommend > looking at how Rust manages crates. They are a pleasure to use and work > with. Rust's cargo [1] is indeed well thought through (I like the dependency specification especially [2]) and we should look at it in depth during the design. We already have a very detailed technical thread, that's why I'd like this thread to remain on the naming aspect. The rust folks chose cargo/crate for their package manager vocabulary. I think we should also put some effort in it, although I don't see the need to find synonyms or other fancy words for package management. Matthias [1] http://doc.crates.io/guide.html [2] http://doc.crates.io/specifying-dependencies.html From vallentin at icir.org Mon May 30 19:44:30 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Mon, 30 May 2016 19:44:30 -0700 Subject: [Bro-Dev] CBAN naming In-Reply-To: <43f3ce86-2869-877f-ae66-5040f145694d@gmail.com> References: <20160527170045.GG2796@samurai.ICIR.org> <6BB7568F-3B00-40C8-A4DD-E681B6464060@illinois.edu> <43f3ce86-2869-877f-ae66-5040f145694d@gmail.com> Message-ID: <20160531024430.GB1312@shogun.local> > By the way: Are we talking about renaming the whole project? I don't know :-). We need a command line client and a project name. When they are the same, it's certainly easier to remember. My favorite name so far is also "brow." The eye brow of the Bro logo makes for a nice association. It didn't occur to me until you mentioned it, but it's also only edit distance one away from "brew," which helps in creating a package manager association. Matthias From vlad at grigorescu.org Tue May 31 08:00:19 2016 From: vlad at grigorescu.org (Vlad Grigorescu) Date: Tue, 31 May 2016 10:00:19 -0500 Subject: [Bro-Dev] CBAN naming In-Reply-To: <20160531024430.GB1312@shogun.local> References: <20160527170045.GG2796@samurai.ICIR.org> <6BB7568F-3B00-40C8-A4DD-E681B6464060@illinois.edu> <43f3ce86-2869-877f-ae66-5040f145694d@gmail.com> <20160531024430.GB1312@shogun.local> Message-ID: I don't like the name CBAN, for a number of reasons. The are a number of C?AN's out there. I believe that CTAN was the first one (TeX), and then CPAN (Perl), CRAN (R), and CEAN (Erlang) followed. The architecture of CPAN was also ported for CCAN (C) and JSAN (Javascript). Because these all share a family tree, much of the nomenclature is the same. Terms such as "packages," "modules," and "distributions" mean something specific in these cases, and those don't map well to Bro terminology. The other consistency is that most C?AN's don't actually have a command line client - Perl seems to be the notable exception, and its command line client is awful. I think one of the main reasons people like CBAN is because they assume people will liken it to CPAN and immediately understand what it does. That might be the case for people who have Perl experience (although I'll argue that the number of those people is declining), but users with CTAN, CRAN, or CCAN experience will come in with completely different assumptions. Even people who do liken it to CPAN will need to reframe their assumptions, since the similarities don't extend very far. So essentially, the impact of the name CBAN on people who are unfamiliar with it would be one of: 1) haven't heard of C?AN, don't understand what it does, 2) have heard of CPAN, assume a similar model and a similar (awful) command line client, or 3) have heard of a different C?AN, assume they have to download "packages" from the website manually, or 4) have heard of CPAN and some other C?AN, have no idea what to assume. I also don't think the acronym CBAN represents the goals of the project: Comprehensive - I don't think we can come close to claiming that. If the project was hunting for Bro scripts that others have shared and aggregating them somehow, then I think we could claim that it'd be comprehensive, but that seems like a weird claim when submissions are voluntary. Bro - No arguments here. :-) Archive - I'm not sure what this means in this context. Does it mean that plugins are distributed as archives? I'm not sure this makes sense to do. I think this is a holdover from the days before distributed VCS and GitHub, when you could download an archive of the packages. While I think we will offer that functionality, I don't think that is or should be a key feature. Network - Again, not really sure what this means. To me, this implies a distributed model, which I don't think this project will offer -- at least not for a while. Finally, from the strictly practical perspective: * cban.org is taken * github.com/cban is taken * There seems to be a conflicting use of the acronym ("Comprehensive BEAST Archive Network"), though admittedly I'm not sure how widespread that usage is. --Vlad On Mon, May 30, 2016 at 9:44 PM, Matthias Vallentin wrote: > > By the way: Are we talking about renaming the whole project? > > I don't know :-). We need a command line client and a project name. When > they are the same, it's certainly easier to remember. > > My favorite name so far is also "brow." The eye brow of the Bro logo > makes for a nice association. It didn't occur to me until you mentioned > it, but it's also only edit distance one away from "brew," which helps > in creating a package manager association. > > Matthias > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20160531/ccca1bdf/attachment.html From jsiwek at illinois.edu Tue May 31 09:38:01 2016 From: jsiwek at illinois.edu (Siwek, Jon) Date: Tue, 31 May 2016 16:38:01 +0000 Subject: [Bro-Dev] CBAN naming In-Reply-To: <20160531024430.GB1312@shogun.local> References: <20160527170045.GG2796@samurai.ICIR.org> <6BB7568F-3B00-40C8-A4DD-E681B6464060@illinois.edu> <43f3ce86-2869-877f-ae66-5040f145694d@gmail.com> <20160531024430.GB1312@shogun.local> Message-ID: <944E9525-D2EE-4E4A-8398-63DC795505F0@illinois.edu> > On May 30, 2016, at 9:44 PM, Matthias Vallentin wrote: > > I don't know :-). We need a command line client and a project name. When > they are the same, it's certainly easier to remember. Yes, I think the client and project name should be the same. From the other thread, I take it we also need a term to use as a substitute for ?package? ? Misc. ideas for both client/project name as well as term to use for ?package?: - boxer, boxes (sort of already taken by Broala's BroBox) - bundler, bundles (though seems like Ruby has taken this name) - bagger/bagboy, bags (also has the association w/ eye bags) - tempest, drops (eye of the storm, rain drops, eye drops... the tears of those trapped in dependency-hell?) I favor ?bagger? or ?bagboy? along with ?bags?. It?s charming/amusing, easily logo-able, and is a nice change from away from the usual motifs related to eyes or puns related to ?what words are there that start with b, r, o??. They also work as analogies for the process the command line client will perform and for indicating that bags are not just filled with one type of thing. e.g. bags may contain any combination of scripts, BIFs, or native plugins/code, like grocery bags may contain various types of food. Should also consider how easy it is to stick the word Bro in front of it when trying to associate it w/ the main Bro project. i.e. ?Bro?s Bagboy?, ?Bro Bagger? and ?Bro Bag? I find easy enough to say. > My favorite name so far is also "brow." The eye brow of the Bro logo > makes for a nice association. It didn't occur to me until you mentioned > it, but it's also only edit distance one away from "brew," which helps > in creating a package manager association. I also like ?brow? the best (of other people?s ideas), but still don?t like it that much. While it is clever that ?brow" both starts with ?bro? and is related to eyes, it also feels predictable. - Jon From vallentin at icir.org Tue May 31 10:54:13 2016 From: vallentin at icir.org (Matthias Vallentin) Date: Tue, 31 May 2016 10:54:13 -0700 Subject: [Bro-Dev] CBAN naming In-Reply-To: <944E9525-D2EE-4E4A-8398-63DC795505F0@illinois.edu> References: <20160527170045.GG2796@samurai.ICIR.org> <6BB7568F-3B00-40C8-A4DD-E681B6464060@illinois.edu> <43f3ce86-2869-877f-ae66-5040f145694d@gmail.com> <20160531024430.GB1312@shogun.local> <944E9525-D2EE-4E4A-8398-63DC795505F0@illinois.edu> Message-ID: <20160531175413.GO2796@samurai.ICIR.org> > - boxer, boxes (sort of already taken by Broala's BroBox) > - bundler, bundles (though seems like Ruby has taken this name) > - bagger/bagboy, bags (also has the association w/ eye bags) > - tempest, drops (eye of the storm, rain drops, eye drops... the tears of those trapped in dependency-hell?) Of those, I like "bag" the best to represent a package that can be a "mixed bag" of scripts and compiled code. My second vote goes for "bundle," which is also something quite intuitive for a package. The remaining ones take a bit more think time (at least for me) to map to something package related. > I favor ?bagger? or ?bagboy? along with ?bags?. I did not get the "bagger" and "bagboy" variations until I googled it, probably because I'm not a native speaker. However, I like the grocery bag association, that one stuck immediately. We could have a shopping "cart" as well, to represent a collection of bags. The grocery store theme works really well, in my opinion. Matthias From robin at icir.org Tue May 31 14:06:40 2016 From: robin at icir.org (Robin Sommer) Date: Tue, 31 May 2016 14:06:40 -0700 Subject: [Bro-Dev] CBAN naming In-Reply-To: <20160531175413.GO2796@samurai.ICIR.org> References: <20160527170045.GG2796@samurai.ICIR.org> <6BB7568F-3B00-40C8-A4DD-E681B6464060@illinois.edu> <43f3ce86-2869-877f-ae66-5040f145694d@gmail.com> <20160531024430.GB1312@shogun.local> <944E9525-D2EE-4E4A-8398-63DC795505F0@illinois.edu> <20160531175413.GO2796@samurai.ICIR.org> Message-ID: <20160531210640.GQ45278@icir.org> I like the bagging theme, too. As one more thought, maybe the client could just be "bag" as well? It's a verb too. :) On Tue, May 31, 2016 at 10:54 -0700, you wrote: > > - boxer, boxes (sort of already taken by Broala's BroBox) > > - bundler, bundles (though seems like Ruby has taken this name) > > - bagger/bagboy, bags (also has the association w/ eye bags) > > - tempest, drops (eye of the storm, rain drops, eye drops... the tears of those trapped in dependency-hell?) > > Of those, I like "bag" the best to represent a package that can be a > "mixed bag" of scripts and compiled code. My second vote goes for > "bundle," which is also something quite intuitive for a package. The > remaining ones take a bit more think time (at least for me) to map to > something package related. > > > I favor ?bagger? or ?bagboy? along with ?bags?. > > I did not get the "bagger" and "bagboy" variations until I googled it, > probably because I'm not a native speaker. However, I like the grocery > bag association, that one stuck immediately. We could have a shopping > "cart" as well, to represent a collection of bags. The grocery store > theme works really well, in my opinion. > > Matthias > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -- Robin Sommer * ICSI/LBNL * robin at icir.org * www.icir.org/robin