From noreply at bro.org Mon Jul 1 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Mon, 1 Jul 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307010700.r61703BL003884@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] Bro | 1024 [7] | dmandelb | | Medium | make it possible to use redef to append to vectors Bro | 1027 [8] | seth | | Medium | topic/seth/ssl-remove-log-queue [9] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [7] #1024: http://tracker.bro.org/bro/ticket/1024 [8] #1027: http://tracker.bro.org/bro/ticket/1027 [9] ssl-remove-log-queue: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/ssl-remove-log-queue From noreply at bro.org Tue Jul 2 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Tue, 2 Jul 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307020700.r62702Vx001203@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1021 [5] | amannb | | Medium | merge topic/bernhard/input-update [6] Bro | 1024 [7] | dmandelb | | Medium | make it possible to use redef to append to vectors Bro | 1027 [8] | seth | | Medium | topic/seth/ssl-remove-log-queue [9] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1021: http://tracker.bro.org/bro/ticket/1021 [6] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [7] #1024: http://tracker.bro.org/bro/ticket/1024 [8] #1027: http://tracker.bro.org/bro/ticket/1027 [9] ssl-remove-log-queue: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/ssl-remove-log-queue From bro at tracker.bro.org Tue Jul 2 11:54:16 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 02 Jul 2013 18:54:16 -0000 Subject: [Bro-Dev] #1020: TLSv1.2 Support In-Reply-To: <049.6691fc15a068a27b0f83b980af157585@tracker.bro.org> References: <049.6691fc15a068a27b0f83b980af157585@tracker.bro.org> Message-ID: <064.c26480a4a0227e602bec7d2dbed70a83@tracker.bro.org> #1020: TLSv1.2 Support ----------------------------+------------------------ Reporter: liamrandall | Owner: robin Type: Merge Request | Status: assigned Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by seth): * owner: => robin * status: new => assigned * type: Feature Request => Merge Request Comment: The contributed patch is incorrect. The correct fix is in the topic/seth/tls-1.2-fix branch. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Jul 2 11:55:02 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 02 Jul 2013 18:55:02 -0000 Subject: [Bro-Dev] #1020: TLSv1.2 Support In-Reply-To: <049.6691fc15a068a27b0f83b980af157585@tracker.bro.org> References: <049.6691fc15a068a27b0f83b980af157585@tracker.bro.org> Message-ID: <064.7bc6f4f6bcf92abd679ae0be929fe908@tracker.bro.org> #1020: TLSv1.2 Support ----------------------------+------------------------ Reporter: liamrandall | Owner: robin Type: Merge Request | Status: assigned Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by seth): Also, I forgot to mention the TLS fix branch does not address record fragmentation. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Jul 2 13:08:47 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 02 Jul 2013 20:08:47 -0000 Subject: [Bro-Dev] #1028: topic/dnthayer/broctl-testing-fixes Message-ID: <046.0a9375cec5ec1c9b84d40d814b47c873@tracker.bro.org> #1028: topic/dnthayer/broctl-testing-fixes ---------------------------+------------------------ Reporter: dnthayer | Owner: Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: BroControl | Version: git/master Keywords: | ---------------------------+------------------------ This branch contains a few fixes for things that were changed when the broctl-testing branch was merged into master. Also fixed a few test cases that could fail, and cleaned up the broctl test build script. Here are the one-line summaries of all commits in this branch: Replace realpath command with python equivalent Remove the "-j" option to make Remove unused Makefile variable Minor cleanup of the build script Remove "make quick" from the README Update baselines for recent changes to crash-diag Fix a canonifier script -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Wed Jul 3 00:00:04 2013 From: noreply at bro.org (Merge Tracker) Date: Wed, 3 Jul 2013 00:00:04 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307030700.r63704pw020879@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1013 [2] | dmandelb | | Medium | redef'ing tables overwrites unrelated values Bro | 1019 [3] | jsiwek | | Medium | topic/jsiwek/plugin-docs [4] Bro | 1020 [5] | liamrandall | robin | Medium | TLSv1.2 Support Bro | 1021 [6] | amannb | | Medium | merge topic/bernhard/input-update [7] Bro | 1024 [8] | dmandelb | | Medium | make it possible to use redef to append to vectors Bro | 1027 [9] | seth | | Medium | topic/seth/ssl-remove-log-queue [10] BroControl | 1028 [11] | dnthayer | | Medium | topic/dnthayer/broctl-testing-fixes [12] [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1013: http://tracker.bro.org/bro/ticket/1013 [3] #1019: http://tracker.bro.org/bro/ticket/1019 [4] plugin-docs: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/jsiwek/plugin-docs [5] #1020: http://tracker.bro.org/bro/ticket/1020 [6] #1021: http://tracker.bro.org/bro/ticket/1021 [7] input-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/input-update [8] #1024: http://tracker.bro.org/bro/ticket/1024 [9] #1027: http://tracker.bro.org/bro/ticket/1027 [10] ssl-remove-log-queue: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/ssl-remove-log-queue [11] #1028: http://tracker.bro.org/bro/ticket/1028 [12] broctl-testing-fixes: http://tracker.bro.org/bro/changeset?old_path=%2Fbrocontrol&old=master&new_path=%2Fbrocontrol&new=topic/dnthayer/broctl-testing-fixes From bro at tracker.bro.org Wed Jul 3 13:05:29 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 03 Jul 2013 20:05:29 -0000 Subject: [Bro-Dev] #1010: BroControl plugin for adding environment variables In-Reply-To: <042.6f4e2d24401d8353a46d8eb4e4a37862@tracker.bro.org> References: <042.6f4e2d24401d8353a46d8eb4e4a37862@tracker.bro.org> Message-ID: <057.6124fa4281acf6aa83717b54919fb233@tracker.bro.org> #1010: BroControl plugin for adding environment variables ------------------------------+------------------------ Reporter: seth | Owner: dnthayer Type: Feature Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: | Keywords: ------------------------------+------------------------ Comment (by Daniel Thayer ): In [changeset:d9323aaa5c1cd79b782d20d8bfa94632389db0dc/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="d9323aaa5c1cd79b782d20d8bfa94632389db0dc" Add ability to set environment variables in node.cfg Added the ability to specify environment variables in node.cfg to be set when Bro is started (these can override values set by the load-balancing plugins). This uses the existing "env_vars" node attribute, but now allows a user to specify it in the node.cfg file. One limitation is that the value of any such environment variable must be a constant. If a value can be determined only at run-time, then a user would need to write their own plugin to set such a variable (similar to the load-balancing plugins). Addresses #1010. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From vallentin at icir.org Wed Jul 3 13:53:29 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Wed, 3 Jul 2013 22:53:29 +0200 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/seth/libinjection: Integration with libinjection (https://github.com/client9/libinjection) (9ee6dff) In-Reply-To: <201307031802.r63I2Cjc008434@bro-ids.icir.org> References: <201307031802.r63I2Cjc008434@bro-ids.icir.org> Message-ID: > - This is only for show. I did a tiny bit of testing with real > network traffic and there were way too many false positives for > this to be really useful. I'm not going to be filing a merge > request for this. Very useful to know! I was about to offer a student to investigate the efficacy of libinjection, but given the high FPs, I am less excited about it. Do you think it's possible to improve on the FP rate or is the "model" hardcoded in the library? Matthias From seth at icir.org Wed Jul 3 14:25:00 2013 From: seth at icir.org (Seth Hall) Date: Wed, 3 Jul 2013 17:25:00 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/seth/libinjection: Integration with libinjection (https://github.com/client9/libinjection) (9ee6dff) In-Reply-To: References: <201307031802.r63I2Cjc008434@bro-ids.icir.org> Message-ID: <4EA7F1EE-A7CC-4A47-A231-B297BBE2AFFD@icir.org> On Jul 3, 2013, at 4:53 PM, Matthias Vallentin wrote: > Very useful to know! I was about to offer a student to investigate the > efficacy of libinjection, but given the high FPs, I am less excited > about it. Do you think it's possible to improve on the FP rate or is > the "model" hardcoded in the library? I don't know, the false hits make me believe it's a failing in their model, but I don't really understand how it works so I can't say that authoritatively. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From bro at tracker.bro.org Wed Jul 3 17:28:00 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Jul 2013 00:28:00 -0000 Subject: [Bro-Dev] #1013: redef'ing tables overwrites unrelated values In-Reply-To: <046.70c89aaf401c58227cd7d8cc363e89c1@tracker.bro.org> References: <046.70c89aaf401c58227cd7d8cc363e89c1@tracker.bro.org> Message-ID: <061.f49cddc8c94210ffd517c1186920c1a6@tracker.bro.org> #1013: redef'ing tables overwrites unrelated values ----------------------------+------------------------ Reporter: dmandelb | Owner: robin Type: Merge Request | Status: closed Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:ed45a6ea6095a7dd054e6aedd6401adb30d3f088/bro]: {{{ #!CommitTicketReference repository="bro" revision="ed45a6ea6095a7dd054e6aedd6401adb30d3f088" Merge remote-tracking branch 'origin/topic/jsiwek/1013' Closes #1013. * origin/topic/jsiwek/1013: Fix redef of table index from clearing table. Addresses #1013. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jul 3 17:28:00 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Jul 2013 00:28:00 -0000 Subject: [Bro-Dev] #1019: topic/jsiwek/plugin-docs In-Reply-To: <044.37226bc69187528df953997207fe87e7@tracker.bro.org> References: <044.37226bc69187528df953997207fe87e7@tracker.bro.org> Message-ID: <059.f3326001e5b55e411a0dc28444fbccbd@tracker.bro.org> #1019: topic/jsiwek/plugin-docs ----------------------------+------------------------ Reporter: jsiwek | Owner: robin Type: Merge Request | Status: closed Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:a329c3e7c31808d408b12a686985fd40c4fbb4e1/bro]: {{{ #!CommitTicketReference repository="bro" revision="a329c3e7c31808d408b12a686985fd40c4fbb4e1" Merge remote-tracking branch 'origin/topic/jsiwek/plugin-docs' Closes #1019. * origin/topic/jsiwek/plugin-docs: Teach broxygen to generate protocol analyzer plugin reference. const adjustments }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jul 3 17:28:00 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Jul 2013 00:28:00 -0000 Subject: [Bro-Dev] #1020: TLSv1.2 Support In-Reply-To: <049.6691fc15a068a27b0f83b980af157585@tracker.bro.org> References: <049.6691fc15a068a27b0f83b980af157585@tracker.bro.org> Message-ID: <064.76d7e5466c491cbbd8ac2a9e2193ca93@tracker.bro.org> #1020: TLSv1.2 Support ----------------------------+------------------------ Reporter: liamrandall | Owner: robin Type: Merge Request | Status: closed Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * status: assigned => closed * resolution: => fixed Comment: In [changeset:ba4f03bc98263d1efa6685c63d659a9c26016515/bro]: {{{ #!CommitTicketReference repository="bro" revision="ba4f03bc98263d1efa6685c63d659a9c26016515" Merge remote-tracking branch 'origin/topic/seth/tls-1.2-fix' Closes #1020. * origin/topic/seth/tls-1.2-fix: Single character fix to correct support for TLS 1.2 (my bad). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jul 3 17:28:00 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Jul 2013 00:28:00 -0000 Subject: [Bro-Dev] #1002: Merge new thread cleanup code In-Reply-To: <044.aeacefd901bb69438d337af478584646@tracker.bro.org> References: <044.aeacefd901bb69438d337af478584646@tracker.bro.org> Message-ID: <059.660e7db7fd0c54329e3ddbae034334a8@tracker.bro.org> #1002: Merge new thread cleanup code ----------------------------+------------------------ Reporter: amannb | Owner: robin Type: Merge Request | Status: closed Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * resolution: Solved/Applied => fixed Comment: In [changeset:d8b05af7e535dc39e111079325f6331262b4bfe9/bro]: {{{ #!CommitTicketReference repository="bro" revision="d8b05af7e535dc39e111079325f6331262b4bfe9" Merge remote-tracking branch 'origin/topic/jsiwek/faf-cleanup' Closes #1002. * origin/topic/jsiwek/faf-cleanup: Move file analyzers to new plugin infrastructure. Add a general file analysis overview/how-to document. Improve file analysis doxygen comments. Improve tracking of HTTP file extraction (addresses #988). Fix HTTP multipart body file analysis. Remove logging of analyzers field of FileAnalysis::Info. Remove extraction counter in default file extraction scripts. Remove FileAnalysis::postpone_timeout. Make default get_file_handle handlers &priority=5. Add input interface to forward data for file analysis. File analysis framework interface simplifications. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jul 3 17:28:00 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Jul 2013 00:28:00 -0000 Subject: [Bro-Dev] #1021: merge topic/bernhard/input-update In-Reply-To: <044.9e7ba994ecc2870deb40b091909ee1d1@tracker.bro.org> References: <044.9e7ba994ecc2870deb40b091909ee1d1@tracker.bro.org> Message-ID: <059.bf63b97501e8aca0b601c9860c7a84d8@tracker.bro.org> #1021: merge topic/bernhard/input-update ----------------------------+------------------------ Reporter: amannb | Owner: robin Type: Merge Request | Status: closed Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:96fe05633a5376626333bafee4ca03534cd8e937/bro]: {{{ #!CommitTicketReference repository="bro" revision="96fe05633a5376626333bafee4ca03534cd8e937" Merge remote-tracking branch 'origin/topic/bernhard/input-update' Closes #1021. * origin/topic/bernhard/input-update: this event handler fails the unused-event-handlers test because it is a bit of a special case. ...and fix the event ordering issue. Dispatch != QueueEvent add Terminate to input framework to prevent potential shutdown race- conditions. fix warning. fix stderr test. ls behaves differently on errors on linux... small fixes. linux does not have strnstr and close only fds that are currently open (the logging framework really did not like that :) ) A bunch of more changes for the raw reader make reading from stdout and stderr simultaneously work. allow sending data to stdin of child process Streaming reads from external commands work without blocking anything. replace popen with fork and exec. change raw reader to use basic c io instead of fdstream encapsulation class. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jul 3 17:28:00 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Jul 2013 00:28:00 -0000 Subject: [Bro-Dev] #1027: topic/seth/ssl-remove-log-queue In-Reply-To: <042.68150ac2b7b87dea8be623c2e7907f6b@tracker.bro.org> References: <042.68150ac2b7b87dea8be623c2e7907f6b@tracker.bro.org> Message-ID: <057.1db782a98b4ed06ef0280c4e775588b0@tracker.bro.org> #1027: topic/seth/ssl-remove-log-queue ----------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:fa8777cbd2c15c855cf74ebcb922ff63caa164fd/bro]: {{{ #!CommitTicketReference repository="bro" revision="fa8777cbd2c15c855cf74ebcb922ff63caa164fd" Merge remote-tracking branch 'origin/topic/seth/ssl-remove-log-queue' Closes #1027. * origin/topic/seth/ssl-remove-log-queue: Remove the log queueing mechanism that was included with the SSL log delay mechanism. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jul 3 17:28:00 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Jul 2013 00:28:00 -0000 Subject: [Bro-Dev] #988: Bug in HTTP body extraction In-Reply-To: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> References: <046.e98622bfacb442ab1adf8a4c4fe0b381@tracker.bro.org> Message-ID: <061.9fc737ed1a3b87ce4e8d36830bb1210b@tracker.bro.org> #988: Bug in HTTP body extraction -----------------------+--------------------------- Reporter: matthias | Owner: seth Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: file-analysis -----------------------+--------------------------- Comment (by robin): In [changeset:d8b05af7e535dc39e111079325f6331262b4bfe9/bro]: {{{ #!CommitTicketReference repository="bro" revision="d8b05af7e535dc39e111079325f6331262b4bfe9" Merge remote-tracking branch 'origin/topic/jsiwek/faf-cleanup' Closes #1002. * origin/topic/jsiwek/faf-cleanup: Move file analyzers to new plugin infrastructure. Add a general file analysis overview/how-to document. Improve file analysis doxygen comments. Improve tracking of HTTP file extraction (addresses #988). Fix HTTP multipart body file analysis. Remove logging of analyzers field of FileAnalysis::Info. Remove extraction counter in default file extraction scripts. Remove FileAnalysis::postpone_timeout. Make default get_file_handle handlers &priority=5. Add input interface to forward data for file analysis. File analysis framework interface simplifications. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Wed Jul 3 17:28:30 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Jul 2013 00:28:30 -0000 Subject: [Bro-Dev] #1028: topic/dnthayer/broctl-testing-fixes In-Reply-To: <046.0a9375cec5ec1c9b84d40d814b47c873@tracker.bro.org> References: <046.0a9375cec5ec1c9b84d40d814b47c873@tracker.bro.org> Message-ID: <061.bd3b568f6c60975a7b449260547b99ba@tracker.bro.org> #1028: topic/dnthayer/broctl-testing-fixes ----------------------------+------------------------ Reporter: dnthayer | Owner: robin Type: Merge Request | Status: closed Priority: Medium | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:017e7732446b36af935c26834394b51829335e7c/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="017e7732446b36af935c26834394b51829335e7c" Merge remote-tracking branch 'origin/topic/dnthayer/broctl-testing-fixes' Closes #1028. * origin/topic/dnthayer/broctl-testing-fixes: Fix a canonifier script Update baselines for recent changes to crash-diag Remove "make quick" from the README Minor cleanup of the build script Remove unused Makefile variable Remove the "-j" option to make Replace realpath command with python equivalent }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Thu Jul 4 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Thu, 4 Jul 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307040700.r647038R015202@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1001 [1] | robin | jsiwek | Low | File analysis framework tasks Bro | 1024 [2] | dmandelb | | Medium | make it possible to use redef to append to vectors [1] #1001: http://tracker.bro.org/bro/ticket/1001 [2] #1024: http://tracker.bro.org/bro/ticket/1024 From bro at tracker.bro.org Thu Jul 4 08:19:16 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Jul 2013 15:19:16 -0000 Subject: [Bro-Dev] #1001: File analysis framework tasks In-Reply-To: <043.9c013e245588fe246d6f8a529a01a055@tracker.bro.org> References: <043.9c013e245588fe246d6f8a529a01a055@tracker.bro.org> Message-ID: <058.0ef84828acd824f71e8ab0623de14032@tracker.bro.org> #1001: File analysis framework tasks -----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: closed Priority: Low | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Jul 4 08:19:25 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Jul 2013 15:19:25 -0000 Subject: [Bro-Dev] #1024: make it possible to use redef to append to vectors In-Reply-To: <046.0bbc08c176f214d1179ab50d7527ead3@tracker.bro.org> References: <046.0bbc08c176f214d1179ab50d7527ead3@tracker.bro.org> Message-ID: <061.099e614687ab32f135d92c2e1d2df347@tracker.bro.org> #1024: make it possible to use redef to append to vectors -----------------------------+------------------------ Reporter: dmandelb | Owner: Type: Merge Request | Status: closed Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From david at mandelberg.org Thu Jul 4 08:56:26 2013 From: david at mandelberg.org (David Mandelberg) Date: Thu, 04 Jul 2013 11:56:26 -0400 Subject: [Bro-Dev] &synchronized on manager Message-ID: <9a0fb79322e8fa4291a9b34fc7060988@mail.mandelberg.org> Hi, My &synchronized variables seem to be synchronized among workers and the proxy, but not the manager. Is that by design, should I file a bug, or is &synchronized just not worth fixing? -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ From bro at tracker.bro.org Thu Jul 4 10:16:09 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Thu, 04 Jul 2013 17:16:09 -0000 Subject: [Bro-Dev] #1029: support printing arbitrary expressions Message-ID: <046.498195d3db7d30fa2c8a5c32c155057c@tracker.bro.org> #1029: support printing arbitrary expressions ------------------------+----------------------------- Reporter: dmandelb | Type: Feature Request Status: new | Priority: Low Milestone: Bro2.2 | Component: BroControl Version: git/master | Keywords: ------------------------+----------------------------- `broctl`'s print command can be very verbose for large tables. It would be nice if it could support at least the below two styles of commands, but ideally it could support any Bro Scripting Language expression. {{{ [BroControl] > print BBNHostPeering::host_peers[127.0.0.1] [BroControl] > print |BBNHostPeering::host_peers| }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Jul 4 22:43:13 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Fri, 05 Jul 2013 05:43:13 -0000 Subject: [Bro-Dev] #1030: topic/seth/packet-filter-updates Message-ID: <042.7222b9fa8c601237762ebab430e04d82@tracker.bro.org> #1030: topic/seth/packet-filter-updates ---------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ Finally, the packet filter framework updates. This provides a much easier and modern mechanism for choosing which traffic to view and which traffic to exlude from analysis. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Thu Jul 4 22:43:34 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Fri, 05 Jul 2013 05:43:34 -0000 Subject: [Bro-Dev] #1030: topic/seth/packet-filter-updates In-Reply-To: <042.7222b9fa8c601237762ebab430e04d82@tracker.bro.org> References: <042.7222b9fa8c601237762ebab430e04d82@tracker.bro.org> Message-ID: <057.ca838dd29c81ae6b637f0ca4249ca50c@tracker.bro.org> #1030: topic/seth/packet-filter-updates ----------------------------+------------------------ Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Description changed by seth: Old description: > Finally, the packet filter framework updates. This provides a much > easier and modern mechanism for choosing which traffic to view and which > traffic to exlude from analysis. New description: Finally, the packet filter framework updates. This provides a much easier and modern mechanism for choosing which traffic to view and which traffic to exlude from analysis. There is an equivalently named branch in the private test suite. -- -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Fri Jul 5 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Fri, 5 Jul 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307050700.r65703bB022652@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1030 [1] | seth | | Medium | topic/seth/packet-filter-updates [2] [1] #1030: http://tracker.bro.org/bro/ticket/1030 [2] packet-filter-updates: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/packet-filter-updates From noreply at bro.org Sat Jul 6 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Sat, 6 Jul 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307060700.r667033i014171@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1030 [1] | seth | | Medium | topic/seth/packet-filter-updates [2] [1] #1030: http://tracker.bro.org/bro/ticket/1030 [2] packet-filter-updates: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/packet-filter-updates From noreply at bro.org Sun Jul 7 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Sun, 7 Jul 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307070700.r677033l022232@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1030 [1] | seth | | Medium | topic/seth/packet-filter-updates [2] [1] #1030: http://tracker.bro.org/bro/ticket/1030 [2] packet-filter-updates: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/packet-filter-updates From noreply at bro.org Mon Jul 8 00:00:04 2013 From: noreply at bro.org (Merge Tracker) Date: Mon, 8 Jul 2013 00:00:04 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307080700.r68704Uc005660@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1030 [1] | seth | | Medium | topic/seth/packet-filter-updates [2] [1] #1030: http://tracker.bro.org/bro/ticket/1030 [2] packet-filter-updates: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/packet-filter-updates From robin at icir.org Mon Jul 8 11:28:54 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 8 Jul 2013 11:28:54 -0700 Subject: [Bro-Dev] &synchronized on manager In-Reply-To: <9a0fb79322e8fa4291a9b34fc7060988@mail.mandelberg.org> References: <9a0fb79322e8fa4291a9b34fc7060988@mail.mandelberg.org> Message-ID: <20130708182854.GV40990@icir.org> That's by design, state is shared across workers and proxies only. This is not a limitation of synchronized though, it's just how it is configured by frameworks/cluster/setup-connections.bro Robin On Thu, Jul 04, 2013 at 11:56 -0400, you wrote: > Hi, > > My &synchronized variables seem to be synchronized among workers and > the proxy, but not the manager. Is that by design, should I file a bug, > or is &synchronized just not worth fixing? > -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From david at mandelberg.org Mon Jul 8 13:02:42 2013 From: david at mandelberg.org (David Mandelberg) Date: Mon, 08 Jul 2013 16:02:42 -0400 Subject: [Bro-Dev] adding boost as a dependency Message-ID: <6f139c97ba3f117cfdda908bdac6c9b4@mail.mandelberg.org> Hi, How do people feel about adding Boost as a dependency? I'd like to make some of boost::math's statistical functions available to the scripting language. -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ From bro at tracker.bro.org Mon Jul 8 13:07:21 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 08 Jul 2013 20:07:21 -0000 Subject: [Bro-Dev] #1031: add script based on BBN's Flow Analyzer Message-ID: <046.e17841c0d6a57732dbee0abdb0e64ca7@tracker.bro.org> #1031: add script based on BBN's Flow Analyzer ------------------------+-------------------- Reporter: dmandelb | Type: Patch Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+-------------------- BBN's RePS team wrote this script that might be useful to the Bro community if it were added to the bro-scripts repository. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon Jul 8 13:08:51 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 8 Jul 2013 13:08:51 -0700 Subject: [Bro-Dev] adding boost as a dependency In-Reply-To: <6f139c97ba3f117cfdda908bdac6c9b4@mail.mandelberg.org> References: <6f139c97ba3f117cfdda908bdac6c9b4@mail.mandelberg.org> Message-ID: <20130708200851.GI40990@icir.org> On Mon, Jul 08, 2013 at 16:02 -0400, you wrote: > How do people feel about adding Boost as a dependency? Not good. :-) > I'd like to make some of boost::math's statistical functions > available to the scripting language. In the future we'll have support for dynamically loaded plugins. You could then write one that links with Boost for providing that functionality, without that we'd need to link Bro itself against it. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From bro at tracker.bro.org Mon Jul 8 13:13:38 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 08 Jul 2013 20:13:38 -0000 Subject: [Bro-Dev] #1032: add script based on BBN's Host Characterization Message-ID: <046.fd0b2e7b7fe0bf9a7ca44d92066c6766@tracker.bro.org> #1032: add script based on BBN's Host Characterization ------------------------+-------------------- Reporter: dmandelb | Type: Patch Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+-------------------- BBN's RePS team wrote this script that might be useful to the Bro community if it were added to the bro-scripts repository. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Jul 8 13:18:54 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 08 Jul 2013 20:18:54 -0000 Subject: [Bro-Dev] #1033: add script based on BBN's ICMP analyzer Message-ID: <046.dd83591fca824320a4a0a48d45a68c49@tracker.bro.org> #1033: add script based on BBN's ICMP analyzer ------------------------+-------------------- Reporter: dmandelb | Type: Patch Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+-------------------- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Jul 8 13:23:48 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 08 Jul 2013 20:23:48 -0000 Subject: [Bro-Dev] #1030: topic/seth/packet-filter-updates In-Reply-To: <042.7222b9fa8c601237762ebab430e04d82@tracker.bro.org> References: <042.7222b9fa8c601237762ebab430e04d82@tracker.bro.org> Message-ID: <057.a02f900b91af45661b5bad305cbfd477@tracker.bro.org> #1030: topic/seth/packet-filter-updates ----------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:b62927e9de56489a73ac205b41760a41f849d77d/bro]: {{{ #!CommitTicketReference repository="bro" revision="b62927e9de56489a73ac205b41760a41f849d77d" Merge remote-tracking branch 'origin/topic/seth/packet-filter-updates' Closes #1030. * origin/topic/seth/packet-filter-updates: Missed a test fix. Updating test baselines. Updates for the PacketFilter framework to simplify it. Last test update for PacketFilter framework. Several final fixes for PacketFilter framework. Packet filter framework checkpoint. Checkpoint on the packet filter framework. Initial rework of packet filter framework. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Jul 8 13:35:06 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 08 Jul 2013 20:35:06 -0000 Subject: [Bro-Dev] #1034: add scripts that provide simple correlation Message-ID: <046.6f92d0b6c6a26a0db54874348e076197@tracker.bro.org> #1034: add scripts that provide simple correlation ------------------------+-------------------- Reporter: dmandelb | Type: Patch Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+-------------------- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Jul 9 20:44:03 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 10 Jul 2013 03:44:03 -0000 Subject: [Bro-Dev] #1035: topic/seth/bittorrent-fix-and-dpd-sig-breakout Message-ID: <042.2d94bead28dade791edcc021fccdd67b@tracker.bro.org> #1035: topic/seth/bittorrent-fix-and-dpd-sig-breakout ---------------------+------------------------ Reporter: seth | Owner: robin Type: Problem | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ This branch has some fixes for the Bittorrent analyzer plugin and the DPD sigs are broken out into their separate protocol script directories. There is an equivalently named branch in the private test suite repository. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Jul 9 20:44:11 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 10 Jul 2013 03:44:11 -0000 Subject: [Bro-Dev] #1035: topic/seth/bittorrent-fix-and-dpd-sig-breakout In-Reply-To: <042.2d94bead28dade791edcc021fccdd67b@tracker.bro.org> References: <042.2d94bead28dade791edcc021fccdd67b@tracker.bro.org> Message-ID: <057.43f39771470301a47e0094b0fb666b87@tracker.bro.org> #1035: topic/seth/bittorrent-fix-and-dpd-sig-breakout ----------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by seth): * type: Problem => Merge Request -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Wed Jul 10 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Wed, 10 Jul 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307100700.r6A702UZ023482@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1035 [1] | seth | robin | Medium | topic/seth/bittorrent-fix-and-dpd-sig-breakout [2] [1] #1035: http://tracker.bro.org/bro/ticket/1035 [2] bittorrent-fix-and-dpd-sig-breakout: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/seth/bittorrent-fix-and-dpd-sig-breakout From vallentin at icir.org Wed Jul 10 01:27:36 2013 From: vallentin at icir.org (Matthias Vallentin) Date: Wed, 10 Jul 2013 10:27:36 +0200 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/matthias/bloom-filter: Fixing for unserializion error. (40201a1) In-Reply-To: <201307100406.r6A466cl015781@bro-ids.icir.org> References: <201307100406.r6A466cl015781@bro-ids.icir.org> Message-ID: > Fixing for unserializion error. Thanks for delving into it! > Because BloomFilter is a base class, with other classes derived from > it, it needs special treatment. I assume this is only necessary when serializing through the base class? For example, we have another serializable base class HashVal with derived classes for MD5, SHA1, etc., but never use the class instances in a polymorphic context. Seems like this class would need a similar fix if we start using it like Bloom filters. Matthias From david at mandelberg.org Wed Jul 10 07:32:07 2013 From: david at mandelberg.org (David Mandelberg) Date: Wed, 10 Jul 2013 10:32:07 -0400 Subject: [Bro-Dev] adding boost as a dependency In-Reply-To: <20130708200851.GI40990@icir.org> References: <6f139c97ba3f117cfdda908bdac6c9b4@mail.mandelberg.org> <20130708200851.GI40990@icir.org> Message-ID: <656c65c78afc200bba311ba364b46ba5@mail.mandelberg.org> On 2013-07-08 16:08, Robin Sommer wrote: > On Mon, Jul 08, 2013 at 16:02 -0400, you wrote: >> I'd like to make some of boost::math's statistical functions >> available to the scripting language. > > In the future we'll have support for dynamically loaded plugins. You > could then write one that links with Boost for providing that > functionality, without that we'd need to link Bro itself against it. Cool, that sounds like a good idea, thanks. -- David Eric Mandelberg / dseomn http://david.mandelberg.org/ From robin at icir.org Wed Jul 10 07:34:35 2013 From: robin at icir.org (Robin Sommer) Date: Wed, 10 Jul 2013 07:34:35 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/matthias/bloom-filter: Fixing for unserializion error. (40201a1) In-Reply-To: References: <201307100406.r6A466cl015781@bro-ids.icir.org> Message-ID: <20130710143435.GC9917@icir.org> On Wed, Jul 10, 2013 at 10:27 +0200, you wrote: > I assume this is only necessary when serializing through the base > class? Indeed. > For example, we have another serializable base class HashVal with > derived classes for MD5, SHA1, etc., but never use the class > instances in a polymorphic context. HashVal is derived from Val, so that should be fine as long as unserializing is triggered via Val::Unserialize(). A good rule of thumb for this treatment is base classes that come with their own Unserialize() method. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From bro at tracker.bro.org Wed Jul 10 07:45:56 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 10 Jul 2013 14:45:56 -0000 Subject: [Bro-Dev] #1036: add script based on BBN's Host Peering Message-ID: <046.490cd0119e209335aebe62aa7e1e2ba1@tracker.bro.org> #1036: add script based on BBN's Host Peering ------------------------+-------------------- Reporter: dmandelb | Type: Patch Status: new | Priority: Medium Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+-------------------- -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Jul 10 12:32:33 2013 From: seth at icir.org (Seth Hall) Date: Wed, 10 Jul 2013 15:32:33 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/faf-updates: Make the custom libmagic database a git submodule. (99d604c) In-Reply-To: <201307101913.r6AJDOUq020126@bro-ids.icir.org> References: <201307101913.r6AJDOUq020126@bro-ids.icir.org> Message-ID: On Jul 10, 2013, at 3:13 PM, Jonathan Siwek wrote: > Make the custom libmagic database a git submodule. Thanks Jon! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Wed Jul 10 14:00:17 2013 From: seth at icir.org (Seth Hall) Date: Wed, 10 Jul 2013 17:00:17 -0400 Subject: [Bro-Dev] custom magic database Message-ID: If anyone feels like putting some TLC into our custom magic database that Jon just created (git://git.bro.org/bromagic), here is a file that is identified incorrectly? http://maps.google.com/intl/en_us/mapfiles/openhand.cur It gets identified as application/x-123 right now, but it seems to be a cursor or something. I'll give short instructions for testing? git clone git://git.bro.org/bromagic cd bromagic wget http://maps.google.com/intl/en_us/mapfiles/openhand.cur file -i -m database/ openhand.cur Now someone just needs to figure out what type of file that actually is and work on the files in database/ directory. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 495 bytes Desc: Message signed with OpenPGP using GPGMail Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130710/3204c37a/attachment.bin From bro at tracker.bro.org Wed Jul 10 14:30:33 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Wed, 10 Jul 2013 21:30:33 -0000 Subject: [Bro-Dev] #1035: topic/seth/bittorrent-fix-and-dpd-sig-breakout In-Reply-To: <042.2d94bead28dade791edcc021fccdd67b@tracker.bro.org> References: <042.2d94bead28dade791edcc021fccdd67b@tracker.bro.org> Message-ID: <057.b0fdc6cabbbb6f9fd07348e861778d04@tracker.bro.org> #1035: topic/seth/bittorrent-fix-and-dpd-sig-breakout ----------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => fixed Comment: In [changeset:cb09bd635865d16a9e81cf422ed0b9273208980c/bro]: {{{ #!CommitTicketReference repository="bro" revision="cb09bd635865d16a9e81cf422ed0b9273208980c" Merge remote-tracking branch 'origin/topic/seth/bittorrent-fix-and-dpd- sig-breakout' Closes #1035. * origin/topic/seth/bittorrent-fix-and-dpd-sig-breakout: Small test fixes. Added a missing curly brace in smtp/dpd.sig Fix a bug where the same analyzer tag was reused for two different analyzers. Moved DPD signatures into script specific directories. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Fri Jul 12 00:00:05 2013 From: noreply at bro.org (Merge Tracker) Date: Fri, 12 Jul 2013 00:00:05 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307120700.r6C705nq024013@bro-ids.icir.org> > Unmerged Fastpath Commits > ========================= Component | Revision | Committer | Date | Summary ------------------------------------------------------------------------------------------------------------------ broctl | 69c76b4 | Bernhard Amann | 2013-07-11 | fix broken link (thanks kraigu) [1] [1] fastpath: http://tracker.bro.org/bro/changeset/69c76b4200265418bde748c5252b404a4951dc13/broctl From bro at tracker.bro.org Fri Jul 12 17:45:12 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Sat, 13 Jul 2013 00:45:12 -0000 Subject: [Bro-Dev] #1037: merge topic/bernhard/sqlite-update Message-ID: <044.bc4375d8bcf0037883139b6a787e893a@tracker.bro.org> #1037: merge topic/bernhard/sqlite-update ---------------------------+------------------------ Reporter: amannb | Owner: Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ Update sqlite to 3.7.17 -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Sat Jul 13 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Sat, 13 Jul 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307130700.r6D703Us012723@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1037 [1] | amannb | | Medium | merge topic/bernhard/sqlite-update [2] > Unmerged Fastpath Commits > ========================= Component | Revision | Committer | Date | Summary ------------------------------------------------------------------------------------------------------------------ broctl | 69c76b4 | Bernhard Amann | 2013-07-11 | fix broken link (thanks kraigu) [3] [1] #1037: http://tracker.bro.org/bro/ticket/1037 [2] sqlite-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/sqlite-update [3] fastpath: http://tracker.bro.org/bro/changeset/69c76b4200265418bde748c5252b404a4951dc13/broctl From noreply at bro.org Sun Jul 14 00:00:06 2013 From: noreply at bro.org (Merge Tracker) Date: Sun, 14 Jul 2013 00:00:06 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307140700.r6E706Yi023045@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1037 [1] | amannb | | Medium | merge topic/bernhard/sqlite-update [2] > Unmerged Fastpath Commits > ========================= Component | Revision | Committer | Date | Summary ------------------------------------------------------------------------------------------------------------------ broctl | 69c76b4 | Bernhard Amann | 2013-07-11 | fix broken link (thanks kraigu) [3] [1] #1037: http://tracker.bro.org/bro/ticket/1037 [2] sqlite-update: http://tracker.bro.org/bro/changeset?old_path=%2Fbro&old=master&new_path=%2Fbro&new=topic/bernhard/sqlite-update [3] fastpath: http://tracker.bro.org/bro/changeset/69c76b4200265418bde748c5252b404a4951dc13/broctl From bro at tracker.bro.org Sun Jul 14 08:45:16 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Sun, 14 Jul 2013 15:45:16 -0000 Subject: [Bro-Dev] #1037: merge topic/bernhard/sqlite-update In-Reply-To: <044.bc4375d8bcf0037883139b6a787e893a@tracker.bro.org> References: <044.bc4375d8bcf0037883139b6a787e893a@tracker.bro.org> Message-ID: <059.ab5bb186933bb377cd016244e2cc7336@tracker.bro.org> #1037: merge topic/bernhard/sqlite-update ----------------------------+------------------------ Reporter: amannb | Owner: robin Type: Merge Request | Status: closed Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [changeset:50357ec47a33f4b817bdb89025ff697caf74043a/bro]: {{{ #!CommitTicketReference repository="bro" revision="50357ec47a33f4b817bdb89025ff697caf74043a" Merge remote-tracking branch 'origin/topic/bernhard/sqlite-update' * origin/topic/bernhard/sqlite-update: yep, freebsd still needs this fix bump sqlite to 3.7.17. Closes #1037. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Jul 15 11:09:24 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 15 Jul 2013 18:09:24 -0000 Subject: [Bro-Dev] #1038: memory leak in input framework Message-ID: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> #1038: memory leak in input framework ------------------------+--------------------- Reporter: scampbell | Type: Problem Status: new | Priority: High Milestone: Bro2.2 | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- When using the input framework to read a data file and convert into an event stream, it seems that none of the objects instantiated for event generation are freed up. For a file of ~ 5.25 M lines, I am seeing > 4 GB memory used. Version is 2.1-798 A quick demo script looks like: event bro_init() { # input stream setup Input::add_event([$source=data_file, $reader=Input::READER_RAW, $mode=Input::STREAM, $name="issh", $fields=lineVals, $ev=sshLine]); } event sshLine(description: Input::EventDescription, tpe: Input::Event, LV: lineVals) { return; } thanks! scott -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Jul 15 11:11:40 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 15 Jul 2013 18:11:40 -0000 Subject: [Bro-Dev] #1038: memory leak in input framework In-Reply-To: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> References: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> Message-ID: <062.380b0f6c1bc6a4f561d768c1e61cb840@tracker.bro.org> #1038: memory leak in input framework ------------------------+------------------------ Reporter: scampbell | Owner: amannb Type: Problem | Status: accepted Priority: High | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------+------------------------ Changes (by amannb): * owner: => amannb * status: new => accepted Comment: Thanks, I will take a look at this. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Jul 15 13:54:17 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 15 Jul 2013 20:54:17 -0000 Subject: [Bro-Dev] #1038: memory leak in input framework In-Reply-To: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> References: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> Message-ID: <062.4b05b7f804ca533b04aa50fb19ee411f@tracker.bro.org> #1038: memory leak in input framework ------------------------+------------------------ Reporter: scampbell | Owner: amannb Type: Problem | Status: accepted Priority: High | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------+------------------------ Comment (by amannb): In [changeset:7427ce511b78c8ae5656762ad8c229976dd33fd3/bro]: {{{ #!CommitTicketReference repository="bro" revision="7427ce511b78c8ae5656762ad8c229976dd33fd3" Small raw reader fixes * crash when accessing nonexistant file. * memory leak when reading from file. Addresses #1038. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Jul 15 14:00:16 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 15 Jul 2013 21:00:16 -0000 Subject: [Bro-Dev] #1038: Input framework memory usage issue with huge files (was: memory leak in input framework) In-Reply-To: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> References: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> Message-ID: <062.70d79cc23bc9fa5314b47adcafbb549e@tracker.bro.org> #1038: Input framework memory usage issue with huge files ------------------------+------------------------ Reporter: scampbell | Owner: amannb Type: Problem | Status: accepted Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------+------------------------ Changes (by amannb): * priority: High => Medium Comment: Fastpath contains a fix for the memory leak -- could you check if that fixes your problem? However, there is another issue with the input framework at the moment. While testing the memory leak, I noticed that Bro memory usage will spike while reading huge files. The reason for this is, that the input framework will add data to the child-thread->main-thread queue much faster than it is being read, which can mean that for a file of several hundred Megabytes, several GB of main memory are being held. Robin and I think that solution for this is probably to make the child- thread->main-thread queue blocking once it reaches a certain size. -- Ticket URL: Bro Tracker Bro Issue Tracker From scampbell at lbl.gov Mon Jul 15 14:12:27 2013 From: scampbell at lbl.gov (Scott Campbell) Date: Mon, 15 Jul 2013 16:12:27 -0500 Subject: [Bro-Dev] #1038: Input framework memory usage issue with huge files In-Reply-To: <062.70d79cc23bc9fa5314b47adcafbb549e@tracker.bro.org> References: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> <062.70d79cc23bc9fa5314b47adcafbb549e@tracker.bro.org> Message-ID: <51E465BB.8000104@lbl.gov> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thanks - I will try to get at it tonight! The workaround to the "reading huge files" problem seems like a good one. Now that you mention it, I did notice that before as well and the description also explains a number of peculiar behaviors that I have been seeing with the input framework. cheers, scott On 7/15/13 4:00 PM, Bro Tracker wrote: > #1038: Input framework memory usage issue with huge files > ------------------------+------------------------ Reporter: > scampbell | Owner: amannb Type: Problem | Status: > accepted Priority: Medium | Milestone: Bro2.2 Component: > Bro | Version: git/master Resolution: | > Keywords: ------------------------+------------------------ Changes > (by amannb): > > * priority: High => Medium > > > Comment: > > Fastpath contains a fix for the memory leak -- could you check if > that fixes your problem? > > However, there is another issue with the input framework at the > moment. While testing the memory leak, I noticed that Bro memory > usage will spike while reading huge files. The reason for this is, > that the input framework will add data to the > child-thread->main-thread queue much faster than it is being read, > which can mean that for a file of several hundred Megabytes, > several GB of main memory are being held. > > Robin and I think that solution for this is probably to make the > child- thread->main-thread queue blocking once it reaches a certain > size. > -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHkZbsACgkQK2Plq8B7ZBwnAwCgkBY5pbIeZSRLjH5+b9RnLkTD 8rAAn1SXnTYipsAsdMX870e5TAX8OqrJ =J563 -----END PGP SIGNATURE----- From bro at tracker.bro.org Mon Jul 15 14:15:57 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 15 Jul 2013 21:15:57 -0000 Subject: [Bro-Dev] #1038: Input framework memory usage issue with huge files In-Reply-To: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> References: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> Message-ID: <062.7ebcf3ae58d86300a9f2f67235a444a4@tracker.bro.org> #1038: Input framework memory usage issue with huge files ------------------------+------------------------ Reporter: scampbell | Owner: amannb Type: Problem | Status: accepted Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------+------------------------ Comment (by scampbell): -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Thanks - I will try to get at it tonight! The workaround to the "reading huge files" problem seems like a good one. Now that you mention it, I did notice that before as well and the description also explains a number of peculiar behaviors that I have been seeing with the input framework. cheers, scott On 7/15/13 4:00 PM, Bro Tracker wrote: > #1038: Input framework memory usage issue with huge files > ------------------------+------------------------ Reporter: > scampbell | Owner: amannb Type: Problem | Status: > accepted Priority: Medium | Milestone: Bro2.2 Component: > Bro | Version: git/master Resolution: | > Keywords: ------------------------+------------------------ Changes > (by amannb): > > * priority: High => Medium > > > Comment: > > Fastpath contains a fix for the memory leak -- could you check if > that fixes your problem? > > However, there is another issue with the input framework at the > moment. While testing the memory leak, I noticed that Bro memory > usage will spike while reading huge files. The reason for this is, > that the input framework will add data to the > child-thread->main-thread queue much faster than it is being read, > which can mean that for a file of several hundred Megabytes, > several GB of main memory are being held. > > Robin and I think that solution for this is probably to make the > child- thread->main-thread queue blocking once it reaches a certain > size. > -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHkZbsACgkQK2Plq8B7ZBwnAwCgkBY5pbIeZSRLjH5+b9RnLkTD 8rAAn1SXnTYipsAsdMX870e5TAX8OqrJ =J563 -----END PGP SIGNATURE----- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Tue Jul 16 11:31:18 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 16 Jul 2013 18:31:18 -0000 Subject: [Bro-Dev] #1038: Input framework memory usage issue with huge files In-Reply-To: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> References: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> Message-ID: <062.9ad1246e4a21fade0fb4a288060e6810@tracker.bro.org> #1038: Input framework memory usage issue with huge files ------------------------+------------------------ Reporter: scampbell | Owner: amannb Type: Problem | Status: accepted Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------+------------------------ Comment (by scampbell): -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 7/15/13 4:00 PM, Bro Tracker wrote: > #1038: Input framework memory usage issue with huge files > ------------------------+------------------------ Reporter: > scampbell | Owner: amannb Type: Problem | Status: > accepted Priority: Medium | Milestone: Bro2.2 Component: > Bro | Version: git/master Resolution: | > Keywords: ------------------------+------------------------ Changes > (by amannb): > > * priority: High => Medium > > > Comment: > > Fastpath contains a fix for the memory leak -- could you check if > that fixes your problem? > > However, there is another issue with the input framework at the > moment. While testing the memory leak, I noticed that Bro memory > usage will spike while reading huge files. The reason for this is, > that the input framework will add data to the > child-thread->main-thread queue much faster than it is being read, > which can mean that for a file of several hundred Megabytes, > several GB of main memory are being held. > > Robin and I think that solution for this is probably to make the > child- thread->main-thread queue blocking once it reaches a certain > size. > I am seeing some significant issues with this run (2.1-814). When I start with a new install (no session state) with an empty but existing data file the exec size rapidly goes to 9-16G, the CPU runs at 100% and any additional data added to the data file after start is ignored. The test is being done on OSX 10.8.4 . Sorry for the bad news, scott -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHlkKcACgkQK2Plq8B7ZBy91gCgtxlreSjH10IqzFz50f5Fs1L8 qLgAoLR36AfLizWI8VZdLr4pr4xLxyCO =GXur -----END PGP SIGNATURE----- -- Ticket URL: Bro Tracker Bro Issue Tracker From scampbell at lbl.gov Tue Jul 16 11:27:51 2013 From: scampbell at lbl.gov (Scott Campbell) Date: Tue, 16 Jul 2013 13:27:51 -0500 Subject: [Bro-Dev] #1038: Input framework memory usage issue with huge files In-Reply-To: <062.70d79cc23bc9fa5314b47adcafbb549e@tracker.bro.org> References: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> <062.70d79cc23bc9fa5314b47adcafbb549e@tracker.bro.org> Message-ID: <51E590A7.5040707@lbl.gov> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 7/15/13 4:00 PM, Bro Tracker wrote: > #1038: Input framework memory usage issue with huge files > ------------------------+------------------------ Reporter: > scampbell | Owner: amannb Type: Problem | Status: > accepted Priority: Medium | Milestone: Bro2.2 Component: > Bro | Version: git/master Resolution: | > Keywords: ------------------------+------------------------ Changes > (by amannb): > > * priority: High => Medium > > > Comment: > > Fastpath contains a fix for the memory leak -- could you check if > that fixes your problem? > > However, there is another issue with the input framework at the > moment. While testing the memory leak, I noticed that Bro memory > usage will spike while reading huge files. The reason for this is, > that the input framework will add data to the > child-thread->main-thread queue much faster than it is being read, > which can mean that for a file of several hundred Megabytes, > several GB of main memory are being held. > > Robin and I think that solution for this is probably to make the > child- thread->main-thread queue blocking once it reaches a certain > size. > I am seeing some significant issues with this run (2.1-814). When I start with a new install (no session state) with an empty but existing data file the exec size rapidly goes to 9-16G, the CPU runs at 100% and any additional data added to the data file after start is ignored. The test is being done on OSX 10.8.4 . Sorry for the bad news, scott -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHlkKcACgkQK2Plq8B7ZBy91gCgtxlreSjH10IqzFz50f5Fs1L8 qLgAoLR36AfLizWI8VZdLr4pr4xLxyCO =GXur -----END PGP SIGNATURE----- From bro at tracker.bro.org Tue Jul 16 11:33:46 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 16 Jul 2013 18:33:46 -0000 Subject: [Bro-Dev] #1038: Input framework memory usage issue with huge files In-Reply-To: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> References: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> Message-ID: <062.432a9e59f38cac2e460e542173deb625@tracker.bro.org> #1038: Input framework memory usage issue with huge files ------------------------+------------------------ Reporter: scampbell | Owner: amannb Type: Problem | Status: accepted Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------+------------------------ Comment (by amannb): Just to check - memory use does not go down after a while? And also - is this a new problem with the re-write of the raw reader, did it also occur before or did you just not use as much data before? -- Ticket URL: Bro Tracker Bro Issue Tracker From scampbell at lbl.gov Tue Jul 16 14:19:56 2013 From: scampbell at lbl.gov (Scott Campbell) Date: Tue, 16 Jul 2013 16:19:56 -0500 Subject: [Bro-Dev] #1038: Input framework memory usage issue with huge files In-Reply-To: <062.432a9e59f38cac2e460e542173deb625@tracker.bro.org> References: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> <062.432a9e59f38cac2e460e542173deb625@tracker.bro.org> Message-ID: <51E5B8FC.7090507@lbl.gov> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 7/16/13 1:33 PM, Bro Tracker wrote: > #1038: Input framework memory usage issue with huge files > ------------------------+------------------------ Reporter: > scampbell | Owner: amannb Type: Problem | Status: > accepted Priority: Medium | Milestone: Bro2.2 Component: > Bro | Version: git/master Resolution: | > Keywords: ------------------------+------------------------ > > Comment (by amannb): > > Just to check - memory use does not go down after a while? > > And also - is this a new problem with the re-write of the raw > reader, did it also occur before or did you just not use as much > data before? > Memory and cpu usage seem to stay constant. The memory gain from an empty file seems to be new behavior as it did not exhibit in 2.1-798 . Data use has been constant - I have been using the same 5M line. thanks! scott -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHluPwACgkQK2Plq8B7ZBw5dgCfcUjxDgweFMYHrlJ79Oj1WYJX GmIAoMu+S/SHvyAJLMwvftuHkmZFmiDu =YqI9 -----END PGP SIGNATURE----- From bro at tracker.bro.org Tue Jul 16 14:23:27 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Tue, 16 Jul 2013 21:23:27 -0000 Subject: [Bro-Dev] #1038: Input framework memory usage issue with huge files In-Reply-To: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> References: <047.82c5cdf0516fa06e3b05d65a26a4e7d1@tracker.bro.org> Message-ID: <062.89b9189e7837921baaadfe9a805c226e@tracker.bro.org> #1038: Input framework memory usage issue with huge files ------------------------+------------------------ Reporter: scampbell | Owner: amannb Type: Problem | Status: accepted Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ------------------------+------------------------ Comment (by scampbell): -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 7/16/13 1:33 PM, Bro Tracker wrote: > #1038: Input framework memory usage issue with huge files > ------------------------+------------------------ Reporter: > scampbell | Owner: amannb Type: Problem | Status: > accepted Priority: Medium | Milestone: Bro2.2 Component: > Bro | Version: git/master Resolution: | > Keywords: ------------------------+------------------------ > > Comment (by amannb): > > Just to check - memory use does not go down after a while? > > And also - is this a new problem with the re-write of the raw > reader, did it also occur before or did you just not use as much > data before? > Memory and cpu usage seem to stay constant. The memory gain from an empty file seems to be new behavior as it did not exhibit in 2.1-798 . Data use has been constant - I have been using the same 5M line. thanks! scott -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.19 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHluPwACgkQK2Plq8B7ZBw5dgCfcUjxDgweFMYHrlJ79Oj1WYJX GmIAoMu+S/SHvyAJLMwvftuHkmZFmiDu =YqI9 -----END PGP SIGNATURE----- -- Ticket URL: Bro Tracker Bro Issue Tracker From mthompson at hexwave.com Thu Jul 18 14:24:06 2013 From: mthompson at hexwave.com (Matt Thompson) Date: Thu, 18 Jul 2013 16:24:06 -0500 Subject: [Bro-Dev] Infinite loop with corrupt pcap Message-ID: Hi, I came across a case where reading a corrupt pcap file resulted in pcap_next() to return !NULL, with hdr.len == 0 and hdr.caplen == 0. This seems to cause Bro to enter an infinite loop consuming 100% CPU. Following patch has fixed the problem, but I'm not sure it's the best approach. diff --git a/src/PktSrc.cc b/src/PktSrc.cc index 105dc90..de048cc 100644 --- a/src/PktSrc.cc +++ b/src/PktSrc.cc @@ -77,6 +77,9 @@ int PktSrc::ExtractNextPacket() data = last_data = pcap_next(pd, &hdr); + if(hdr.len == 0 || hdr.caplen == 0) + return 0; + if ( data ) next_timestamp = hdr.ts.tv_sec + double(hdr.ts.tv_usec) / 1e6; Cheers, Matt Thompson From robin at icir.org Fri Jul 19 08:12:08 2013 From: robin at icir.org (Robin Sommer) Date: Fri, 19 Jul 2013 08:12:08 -0700 Subject: [Bro-Dev] Infinite loop with corrupt pcap In-Reply-To: References: Message-ID: <20130719151208.GC5724@icir.org> Is this something you can reproduce with a small subset of the pcap file that we could include into our test suite? Robin On Thu, Jul 18, 2013 at 16:24 -0500, you wrote: > Hi, > > I came across a case where reading a corrupt pcap file resulted in pcap_next() to return !NULL, with hdr.len == 0 and hdr.caplen == 0. > > This seems to cause Bro to enter an infinite loop consuming 100% CPU. Following patch has fixed the problem, but I'm not sure it's the best approach. > > diff --git a/src/PktSrc.cc b/src/PktSrc.cc > index 105dc90..de048cc 100644 > --- a/src/PktSrc.cc > +++ b/src/PktSrc.cc > @@ -77,6 +77,9 @@ int PktSrc::ExtractNextPacket() > > data = last_data = pcap_next(pd, &hdr); > > + if(hdr.len == 0 || hdr.caplen == 0) > + return 0; > + > if ( data ) > next_timestamp = hdr.ts.tv_sec + double(hdr.ts.tv_usec) / 1e6; > > > Cheers, > Matt Thompson > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From bro at tracker.bro.org Fri Jul 19 22:39:02 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Sat, 20 Jul 2013 05:39:02 -0000 Subject: [Bro-Dev] #1010: BroControl plugin for adding environment variables In-Reply-To: <042.6f4e2d24401d8353a46d8eb4e4a37862@tracker.bro.org> References: <042.6f4e2d24401d8353a46d8eb4e4a37862@tracker.bro.org> Message-ID: <057.3e1eb6837c6790edbf2838b28163298a@tracker.bro.org> #1010: BroControl plugin for adding environment variables ----------------------------+------------------------ Reporter: seth | Owner: robin Type: Merge Request | Status: assigned Priority: Medium | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by seth): * owner: dnthayer => robin * status: new => assigned * type: Feature Request => Merge Request Comment: Several people are using this now, let's get it merged. It seems to work fine. -- Ticket URL: Bro Tracker Bro Issue Tracker From noreply at bro.org Sat Jul 20 00:00:05 2013 From: noreply at bro.org (Merge Tracker) Date: Sat, 20 Jul 2013 00:00:05 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307200700.r6K705ej014422@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ BroControl | 1010 [1] | seth | robin | Medium | BroControl plugin for adding environment variables [1] #1010: http://tracker.bro.org/bro/ticket/1010 From noreply at bro.org Sun Jul 21 00:00:05 2013 From: noreply at bro.org (Merge Tracker) Date: Sun, 21 Jul 2013 00:00:05 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307210700.r6L705D6014551@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ BroControl | 1010 [1] | seth | robin | Medium | BroControl plugin for adding environment variables [1] #1010: http://tracker.bro.org/bro/ticket/1010 From noreply at bro.org Mon Jul 22 00:00:04 2013 From: noreply at bro.org (Merge Tracker) Date: Mon, 22 Jul 2013 00:00:04 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307220700.r6M704gY025345@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ BroControl | 1010 [1] | seth | robin | Medium | BroControl plugin for adding environment variables [1] #1010: http://tracker.bro.org/bro/ticket/1010 From jsiwek at illinois.edu Mon Jul 22 11:21:46 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Mon, 22 Jul 2013 18:21:46 +0000 Subject: [Bro-Dev] [Bro-Commits] [git/bro] fastpath: Fixed a scriptland state issue that manifested especially badly on proxies. (5c3bf14) In-Reply-To: <201307221806.r6MI6Joa031438@bro-ids.icir.org> References: <201307221806.r6MI6Joa031438@bro-ids.icir.org> Message-ID: What was the issue? Maybe I'm reading the diff wrong, but the membership check should be implicit in the delete statement, so this change shouldn't function differently? Or if it actually does, then there's probably still a bug in the core code. - Jon On Jul 22, 2013, at 1:06 PM, Seth Hall wrote: > Repository : ssh://git at bro-ids.icir.org/bro > > On branch : fastpath > Link : http://tracker.bro-ids.org/bro/changeset/5c3bf14d168cca9af75e0ac642de8049f89cf525/bro > >> --------------------------------------------------------------- > > commit 5c3bf14d168cca9af75e0ac642de8049f89cf525 > Author: Seth Hall > Date: Mon Jul 22 14:02:56 2013 -0400 > > Fixed a scriptland state issue that manifested especially badly on proxies. > > >> --------------------------------------------------------------- > > 5c3bf14d168cca9af75e0ac642de8049f89cf525 > scripts/base/protocols/irc/dcc-send.bro | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/scripts/base/protocols/irc/dcc-send.bro b/scripts/base/protocols/irc/dcc-send.bro > index 0a7f27e..3194766 100644 > --- a/scripts/base/protocols/irc/dcc-send.bro > +++ b/scripts/base/protocols/irc/dcc-send.bro > @@ -185,5 +185,6 @@ event expected_connection_seen(c: connection, a: Analyzer::Tag) &priority=10 > > event connection_state_remove(c: connection) &priority=-5 > { > - delete dcc_expected_transfers[c$id$resp_h, c$id$resp_p]; > + if ( [c$id$resp_h, c$id$resp_p] in dcc_expected_transfers ) > + delete dcc_expected_transfers[c$id$resp_h, c$id$resp_p]; > } > > _______________________________________________ > bro-commits mailing list > bro-commits at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits > From seth at icir.org Mon Jul 22 11:28:28 2013 From: seth at icir.org (Seth Hall) Date: Mon, 22 Jul 2013 14:28:28 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] fastpath: Fixed a scriptland state issue that manifested especially badly on proxies. (5c3bf14) In-Reply-To: References: <201307221806.r6MI6Joa031438@bro-ids.icir.org> Message-ID: <9908CA2F-3218-4E9D-B95D-46DB37152424@icir.org> On Jul 22, 2013, at 2:21 PM, "Siwek, Jonathan Luke" wrote: > What was the issue? Maybe I'm reading the diff wrong, but the membership check should be implicit in the delete statement, so this change shouldn't function differently? Or if it actually does, then there's probably still a bug in the core code. That variable has &synchronized applied to it. It causes tons of del messages to be sent over the communication code on clusters. Yet another *very* difficult to find problem due to ambiguities brought up from the &synchronized attribute. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From robin at icir.org Mon Jul 22 11:47:16 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 22 Jul 2013 11:47:16 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] fastpath: Fixed a scriptland state issue that manifested especially badly on proxies. (5c3bf14) In-Reply-To: <9908CA2F-3218-4E9D-B95D-46DB37152424@icir.org> References: <201307221806.r6MI6Joa031438@bro-ids.icir.org> <9908CA2F-3218-4E9D-B95D-46DB37152424@icir.org> Message-ID: <20130722184716.GA68130@icir.org> On Mon, Jul 22, 2013 at 14:28 -0400, you wrote: > That variable has &synchronized applied to it. It causes tons of del messages to be sent over the communication code on clusters. I think the core should be able to do that check generally: send out updates only if the element is indeed part of the set. As the assumption is that all nodes have the same version of the set, that shouldn't break anything (though it could because in practice that assumption does not always hold ...) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From seth at icir.org Mon Jul 22 11:54:18 2013 From: seth at icir.org (Seth Hall) Date: Mon, 22 Jul 2013 14:54:18 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] fastpath: Fixed a scriptland state issue that manifested especially badly on proxies. (5c3bf14) In-Reply-To: <20130722184716.GA68130@icir.org> References: <201307221806.r6MI6Joa031438@bro-ids.icir.org> <9908CA2F-3218-4E9D-B95D-46DB37152424@icir.org> <20130722184716.GA68130@icir.org> Message-ID: <2A2A0969-827A-498B-A13D-3570BE7AB3F3@icir.org> On Jul 22, 2013, at 2:47 PM, Robin Sommer wrote: > assumption is that all nodes have the same version of the set, that > shouldn't break anything (though it could because in practice that > assumption does not always hold ...) Anyway, Jon, if you're interested in making that change I think the door is open. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From bro at tracker.bro.org Mon Jul 22 12:20:29 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 22 Jul 2013 19:20:29 -0000 Subject: [Bro-Dev] #1010: BroControl plugin for adding environment variables In-Reply-To: <042.6f4e2d24401d8353a46d8eb4e4a37862@tracker.bro.org> References: <042.6f4e2d24401d8353a46d8eb4e4a37862@tracker.bro.org> Message-ID: <057.0fbb9bf52a53e562f3dc1997e68d51f6@tracker.bro.org> #1010: BroControl plugin for adding environment variables -------------------------+------------------------ Reporter: seth | Owner: dnthayer Type: Task | Status: assigned Priority: Medium | Milestone: Bro2.2 Component: BroControl | Version: git/master Resolution: | Keywords: -------------------------+------------------------ Changes (by robin): * owner: robin => dnthayer * type: Merge Request => Task Comment: Questions/requests: - there's code to remove quotes, but that seems to work only if there's just a single variable in the env_vars (i.e., not if individual variables are quoted). Is that intended? What's the situation here that's addressed, i.e, why would there be quotes in the first place? - I suggest changing the node's {{{env_vars}}} to a dictionary internally. Maintaining the space-separated list seems error-prone, in particular when plugins modify it. {{{_makeEnvParams()}}} would then build the string. That could also solve the quotation problem. - Actually another way to do this would be not passing all the variables on the command line, but installing them into the process environment before we spawn the child process. Not a must for this patch, but something to consider. - the Myricom plugin is overriding the existing variables, that should probably be adding to them instead? - I'm not seeing the {{{broctl.cfg}}} option? - The option documentation should specify the format. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Jul 22 13:57:13 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 22 Jul 2013 20:57:13 -0000 Subject: [Bro-Dev] #1039: Merge request for Bloom filters Message-ID: <046.61f14594469161914eaee0db0a0d055e@tracker.bro.org> #1039: Merge request for Bloom filters ---------------------------+------------------------ Reporter: matthias | Owner: Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ The Bloom filter implementation in `topic/matthias/bloom-filter` is ready to merge into master. Have a look at the very end of `bro.bif` for the script-land interface. Internally, we have a new `BloomFilterVal`, which is serializable and mergeable and thus ready for cluster use. This `Val` contains a polymorphic Bloom filter instance, which hides the concrete Bloom filter type (currently only basic and counting). Moreover, this branch introduces the notion of ''hashers'', which are parameterizable (i.e., seedable) structures for hashing values ''k'' times. I recall that Bernhard waits for this feature. See `Hasher.h` for the documented interface. In the future, we need to rethink how to construct hash functions which only depend on a seed given at script land. This will be important when sharing Bloom filters across organizational boundaries. At this point, the implementation relies on `CompHash` (at least for composite values, such as records) which itself depends on the initial Bro seed generated at startup time or when the user specifies the environment variable `$BRO_SEED`. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro.org Mon Jul 22 15:47:44 2013 From: bro at tracker.bro.org (Bro Tracker) Date: Mon, 22 Jul 2013 22:47:44 -0000 Subject: [Bro-Dev] #1039: Merge request for Bloom filters In-Reply-To: <046.61f14594469161914eaee0db0a0d055e@tracker.bro.org> References: <046.61f14594469161914eaee0db0a0d055e@tracker.bro.org> Message-ID: <061.0d0ea7e70357018ef41bcd5624840d30@tracker.bro.org> #1039: Merge request for Bloom filters ----------------------------+------------------------ Reporter: matthias | Owner: Type: Merge Request | Status: new Priority: Medium | Milestone: Bro2.2 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by aashish): None Filename None could not be saved, problem: [Errno 13] Permission denied: '/da/trac/bro/attachments/ticket/1039'\Very very cool Matthias! Thanks for your work. I cannot wait to try this. Aashish On Mon, Jul 22, 2013 at 08:57:13PM -0000, Bro Tracker wrote: > #1039: Merge request for Bloom filters > ---------------------------+------------------------ > Reporter: matthias | Owner: > Type: Merge Request | Status: new > Priority: Medium | Milestone: Bro2.2 > Component: Bro | Version: git/master > Keywords: | > ---------------------------+------------------------ > The Bloom filter implementation in `topic/matthias/bloom-filter` is ready > to merge into master. Have a look at the very end of `bro.bif` for the > script-land interface. > > Internally, we have a new `BloomFilterVal`, which is serializable and > mergeable and thus ready for cluster use. This `Val` contains a > polymorphic Bloom filter instance, which hides the concrete Bloom filter > type (currently only basic and counting). Moreover, this branch introduces > the notion of ''hashers'', which are parameterizable (i.e., seedable) > structures for hashing values ''k'' times. I recall that Bernhard waits > for this feature. See `Hasher.h` for the documented interface. > > In the future, we need to rethink how to construct hash functions which > only depend on a seed given at script land. This will be important when > sharing Bloom filters across organizational boundaries. At this point, the > implementation relies on `CompHash` (at least for composite values, such > as records) which itself depends on the initial Bro seed generated at > startup time or when the user specifies the environment variable > `$BRO_SEED`. > > -- > Ticket URL: > Bro Tracker > Bro Issue Tracker > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev [attachment:"None"] -- Ticket URL: Bro Tracker Bro Issue Tracker From asharma at lbl.gov Mon Jul 22 15:44:10 2013 From: asharma at lbl.gov (Aashish Sharma) Date: Mon, 22 Jul 2013 15:44:10 -0700 Subject: [Bro-Dev] #1039: Merge request for Bloom filters In-Reply-To: <046.61f14594469161914eaee0db0a0d055e@tracker.bro.org> References: <046.61f14594469161914eaee0db0a0d055e@tracker.bro.org> Message-ID: <20130722224359.GL2940@yaksha.lbl.gov> Very very cool Matthias! Thanks for your work. I cannot wait to try this. Aashish On Mon, Jul 22, 2013 at 08:57:13PM -0000, Bro Tracker wrote: > #1039: Merge request for Bloom filters > ---------------------------+------------------------ > Reporter: matthias | Owner: > Type: Merge Request | Status: new > Priority: Medium | Milestone: Bro2.2 > Component: Bro | Version: git/master > Keywords: | > ---------------------------+------------------------ > The Bloom filter implementation in `topic/matthias/bloom-filter` is ready > to merge into master. Have a look at the very end of `bro.bif` for the > script-land interface. > > Internally, we have a new `BloomFilterVal`, which is serializable and > mergeable and thus ready for cluster use. This `Val` contains a > polymorphic Bloom filter instance, which hides the concrete Bloom filter > type (currently only basic and counting). Moreover, this branch introduces > the notion of ''hashers'', which are parameterizable (i.e., seedable) > structures for hashing values ''k'' times. I recall that Bernhard waits > for this feature. See `Hasher.h` for the documented interface. > > In the future, we need to rethink how to construct hash functions which > only depend on a seed given at script land. This will be important when > sharing Bloom filters across organizational boundaries. At this point, the > implementation relies on `CompHash` (at least for composite values, such > as records) which itself depends on the initial Bro seed generated at > startup time or when the user specifies the environment variable > `$BRO_SEED`. > > -- > Ticket URL: > Bro Tracker > Bro Issue Tracker > > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Aashish Sharma (asharma at lbl.gov) Cyber Security, Information Technology Division Lawrence Berkeley National Laboratory http://go.lbl.gov/pgp-aashish Office: (510)-495-2680 Cell: (510)-457-1525 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130722/d68caf01/attachment.bin From noreply at bro.org Tue Jul 23 00:00:03 2013 From: noreply at bro.org (Merge Tracker) Date: Tue, 23 Jul 2013 00:00:03 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307230700.r6N703L0019607@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ Bro | 1039 [1] | matthias | | Medium | Merge request for Bloom filters [1] #1039: http://tracker.bro.org/bro/ticket/1039 From seth at icir.org Tue Jul 23 07:19:25 2013 From: seth at icir.org (Seth Hall) Date: Tue, 23 Jul 2013 10:19:25 -0400 Subject: [Bro-Dev] super-merge Message-ID: For everyone paying attention, I created a topic/seth/super-merge branch last night. There is no new code going into that branch, it's merely a way for some people to be able to run a bunch of development branches more easily. It's like pre-pre-pre-alpha code or something. :) I'll be deleting once all of the relevant code is merged into master. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From seth at icir.org Tue Jul 23 21:09:17 2013 From: seth at icir.org (Seth Hall) Date: Wed, 24 Jul 2013 00:09:17 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] branch 'topic/vlad/dhcp' created In-Reply-To: <201307240403.r6O43L8l017852@bro-ids.icir.org> References: <201307240403.r6O43L8l017852@bro-ids.icir.org> Message-ID: On Jul 24, 2013, at 12:03 AM, vlad at ICSI.Berkeley.EDU wrote: > New branch : topic/vlad/dhcp Vlad's first commit before I fixed his sender settings in gitolite. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From noreply at bro.org Wed Jul 24 00:00:02 2013 From: noreply at bro.org (Merge Tracker) Date: Wed, 24 Jul 2013 00:00:02 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307240700.r6O702J9021644@bro-ids.icir.org> > Open Merge Requests for Bro2.2 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ | [1] | | | | 302 Found | [2] | | | | | [3] | | | |

Found

| [4] | | | |

The document has moved here.

| [5] | | | | | [6] | | | | > Open Merge Requests for Bro2.1 > ============================== Component | Id | Reporter | Owner | Prio | Summary ------------------------------------------------------------------------------------------------------------------ | [7] | | | | 302 Found | [8] | | | | | [9] | | | |

Found

| [10] | | | |

The document has moved here.

| [11] | | | | | [12] | | | | [1] #0: http://tracker.bro.org/bro/ticket/ [2] #0: http://tracker.bro.org/bro/ticket/ [3] #0: http://tracker.bro.org/bro/ticket/ [4] #0: http://tracker.bro.org/bro/ticket/ [5] #0: http://tracker.bro.org/bro/ticket/ [6] #0: http://tracker.bro.org/bro/ticket/ [7] #0: http://tracker.bro.org/bro/ticket/ [8] #0: http://tracker.bro.org/bro/ticket/ [9] #0: http://tracker.bro.org/bro/ticket/ [10] #0: http://tracker.bro.org/bro/ticket/ [11] #0: http://tracker.bro.org/bro/ticket/ [12] #0: http://tracker.bro.org/bro/ticket/ From jira at bro-tracker.atlassian.net Wed Jul 24 12:14:04 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Wed, 24 Jul 2013 14:14:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-950) Add client/server random to SSL hello events In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130724/c12dcc0c/attachment-0001.html From jira at bro-tracker.atlassian.net Wed Jul 24 12:14:04 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Wed, 24 Jul 2013 14:14:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-950) Add client/server random to SSL hello events In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130724/e17f0187/attachment-0001.html From jira at bro-tracker.atlassian.net Wed Jul 24 12:50:59 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 24 Jul 2013 14:50:59 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1040) Test In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130724/cb155f6f/attachment-0001.html From jira at bro-tracker.atlassian.net Wed Jul 24 13:00:54 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 24 Jul 2013 15:00:54 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1040) Test In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130724/316c0f94/attachment-0001.html From jira at bro-tracker.atlassian.net Wed Jul 24 13:01:54 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 24 Jul 2013 15:01:54 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1040) Test In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130724/47cd864c/attachment.html From jira at bro-tracker.atlassian.net Wed Jul 24 13:45:04 2013 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Wed, 24 Jul 2013 15:45:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1010) BroControl plugin for adding environment variables In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20130724/da7c7dc4/attachment.html From jira at bro-tracker.atlassian.net Wed Jul 24 13:51:56 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 24 Jul 2013 15:51:56 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1041) Another Test Ticket In-Reply-To: References: Message-ID: Jon Siwek created BIT-1041: ------------------------------ Summary: Another Test Ticket Key: BIT-1041 URL: https://bro-tracker.atlassian.net/browse/BIT-1041 Project: Bro Issue Tracker Issue Type: Problem Reporter: Jon Siwek More notification and git integration testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Wed Jul 24 13:58:04 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 24 Jul 2013 15:58:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1041) Another Test Ticket In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1041: --------------------------- Resolution: Fixed Status: Closed (was: Open) > Another Test Ticket > ------------------- > > Key: BIT-1041 > URL: https://bro-tracker.atlassian.net/browse/BIT-1041 > Project: Bro Issue Tracker > Issue Type: Problem > Reporter: Jon Siwek > > More notification and git integration testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Wed Jul 24 13:58:04 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 24 Jul 2013 15:58:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1041) Another Test Ticket In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13311#comment-13311 ] Jon Siwek commented on BIT-1041: -------------------------------- It works. > Another Test Ticket > ------------------- > > Key: BIT-1041 > URL: https://bro-tracker.atlassian.net/browse/BIT-1041 > Project: Bro Issue Tracker > Issue Type: Problem > Reporter: Jon Siwek > > More notification and git integration testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Thu Jul 25 02:05:04 2013 From: jira at bro-tracker.atlassian.net (Brian Little (JIRA)) Date: Thu, 25 Jul 2013 04:05:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1042) Caret in regex pattern doesn't match start of line In-Reply-To: References: Message-ID: Brian Little created BIT-1042: --------------------------------- Summary: Caret in regex pattern doesn't match start of line Key: BIT-1042 URL: https://bro-tracker.atlassian.net/browse/BIT-1042 Project: Bro Issue Tracker Issue Type: Bug Components: Bro Affects Versions: 2.1 Environment: Ubuntu 12.04 and 13.04. Dependencies from apt-get ubuntu repos only. Reporter: Brian Little Priority: Normal Attachments: caret.bro print split_all("some string", /^t/); I would expect it to not match ^t, but it matches any t in the string. output: { [1] = some s, [3] = ring, [2] = t } expected: { [1] = some string } tested on bro 2.1 and github master 2.1-824 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Thu Jul 25 05:38:04 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 25 Jul 2013 07:38:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1042) Caret in regex pattern doesn't match start of line In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13312#comment-13312 ] Seth Hall commented on BIT-1042: -------------------------------- The caret matches beginning of string. If you want to match newlines, use /\n/. > Caret in regex pattern doesn't match start of line > -------------------------------------------------- > > Key: BIT-1042 > URL: https://bro-tracker.atlassian.net/browse/BIT-1042 > Project: Bro Issue Tracker > Issue Type: Bug > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu 12.04 and 13.04. Dependencies from apt-get ubuntu repos only. > Reporter: Brian Little > Priority: Normal > Labels: pattern, regex > Attachments: caret.bro > > > print split_all("some string", /^t/); > I would expect it to not match ^t, but it matches any t in the string. > output: > { > [1] = some s, > [3] = ring, > [2] = t > } > expected: > { > [1] = some string > } > tested on bro 2.1 and github master 2.1-824 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Thu Jul 25 05:38:04 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Thu, 25 Jul 2013 07:38:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1042) Caret in regex pattern doesn't match start of line In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1042: --------------------------- Resolution: Won't Fix Status: Closed (was: Open) > Caret in regex pattern doesn't match start of line > -------------------------------------------------- > > Key: BIT-1042 > URL: https://bro-tracker.atlassian.net/browse/BIT-1042 > Project: Bro Issue Tracker > Issue Type: Bug > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu 12.04 and 13.04. Dependencies from apt-get ubuntu repos only. > Reporter: Brian Little > Priority: Normal > Labels: pattern, regex > Attachments: caret.bro > > > print split_all("some string", /^t/); > I would expect it to not match ^t, but it matches any t in the string. > output: > { > [1] = some s, > [3] = ring, > [2] = t > } > expected: > { > [1] = some string > } > tested on bro 2.1 and github master 2.1-824 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Thu Jul 25 06:37:04 2013 From: jira at bro-tracker.atlassian.net (Brian Little (JIRA)) Date: Thu, 25 Jul 2013 08:37:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1042) Caret in regex pattern doesn't match start of line In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Little updated BIT-1042: ------------------------------ Status: Reopened (was: Closed) > Caret in regex pattern doesn't match start of line > -------------------------------------------------- > > Key: BIT-1042 > URL: https://bro-tracker.atlassian.net/browse/BIT-1042 > Project: Bro Issue Tracker > Issue Type: Bug > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu 12.04 and 13.04. Dependencies from apt-get ubuntu repos only. > Reporter: Brian Little > Priority: Normal > Labels: pattern, regex > Attachments: caret.bro > > > print split_all("some string", /^t/); > I would expect it to not match ^t, but it matches any t in the string. > output: > { > [1] = some s, > [3] = ring, > [2] = t > } > expected: > { > [1] = some string > } > tested on bro 2.1 and github master 2.1-824 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Thu Jul 25 06:39:04 2013 From: jira at bro-tracker.atlassian.net (Brian Little (JIRA)) Date: Thu, 25 Jul 2013 08:39:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1042) Caret in regex pattern doesn't match start of line In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13313#comment-13313 ] Brian Little commented on BIT-1042: ----------------------------------- Sorry, I didn't mean "start of line" in the title, but start of string. Does this work as expected? print split_all("some string", /^t/); > Caret in regex pattern doesn't match start of line > -------------------------------------------------- > > Key: BIT-1042 > URL: https://bro-tracker.atlassian.net/browse/BIT-1042 > Project: Bro Issue Tracker > Issue Type: Bug > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu 12.04 and 13.04. Dependencies from apt-get ubuntu repos only. > Reporter: Brian Little > Priority: Normal > Labels: pattern, regex > Attachments: caret.bro > > > print split_all("some string", /^t/); > I would expect it to not match ^t, but it matches any t in the string. > output: > { > [1] = some s, > [3] = ring, > [2] = t > } > expected: > { > [1] = some string > } > tested on bro 2.1 and github master 2.1-824 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Thu Jul 25 07:50:04 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 25 Jul 2013 09:50:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1042) Caret in regex pattern doesn't match start of line In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1042: -------------------------------- Status: Closed (was: Reopened) split_all basi > Caret in regex pattern doesn't match start of line > -------------------------------------------------- > > Key: BIT-1042 > URL: https://bro-tracker.atlassian.net/browse/BIT-1042 > Project: Bro Issue Tracker > Issue Type: Bug > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu 12.04 and 13.04. Dependencies from apt-get ubuntu repos only. > Reporter: Brian Little > Priority: Normal > Labels: pattern, regex > Attachments: caret.bro > > > print split_all("some string", /^t/); > I would expect it to not match ^t, but it matches any t in the string. > output: > { > [1] = some s, > [3] = ring, > [2] = t > } > expected: > { > [1] = some string > } > tested on bro 2.1 and github master 2.1-824 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Thu Jul 25 07:52:04 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Thu, 25 Jul 2013 09:52:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1042) Caret in regex pattern doesn't match start of line In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13314#comment-13314 ] Bernhard Amann edited comment on BIT-1042 at 7/25/13 9:50 AM: -------------------------------------------------------------- split_all basically applies the regular expression multiple times on smaller-growing portions of the string until there is nothing left to match. So it works as expected. To just get the first match use split1 was (Author: amannb): split_all basi > Caret in regex pattern doesn't match start of line > -------------------------------------------------- > > Key: BIT-1042 > URL: https://bro-tracker.atlassian.net/browse/BIT-1042 > Project: Bro Issue Tracker > Issue Type: Bug > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu 12.04 and 13.04. Dependencies from apt-get ubuntu repos only. > Reporter: Brian Little > Priority: Normal > Labels: pattern, regex > Attachments: caret.bro > > > print split_all("some string", /^t/); > I would expect it to not match ^t, but it matches any t in the string. > output: > { > [1] = some s, > [3] = ring, > [2] = t > } > expected: > { > [1] = some string > } > tested on bro 2.1 and github master 2.1-824 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Thu Jul 25 09:21:04 2013 From: jira at bro-tracker.atlassian.net (Brian Little (JIRA)) Date: Thu, 25 Jul 2013 11:21:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1042) Caret in regex pattern doesn't match start of line In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13315#comment-13315 ] Brian Little commented on BIT-1042: ----------------------------------- I was using that as an simple example, but it shouldn't match at all. The specific problem I have it trying to remove the first . in a domain, using print sub(".github.com", /^\./, ""); #github.com This works, however that specific pattern /^\./ also matches this: print sub("github.com", /^\./, ""); #githubcom (note missing .) Is "sub" the wrong tool to use for this? It seems to work for other patterns fine, including the $ (end of string), but not for the start of string > Caret in regex pattern doesn't match start of line > -------------------------------------------------- > > Key: BIT-1042 > URL: https://bro-tracker.atlassian.net/browse/BIT-1042 > Project: Bro Issue Tracker > Issue Type: Bug > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu 12.04 and 13.04. Dependencies from apt-get ubuntu repos only. > Reporter: Brian Little > Priority: Normal > Labels: pattern, regex > Attachments: caret.bro > > > print split_all("some string", /^t/); > I would expect it to not match ^t, but it matches any t in the string. > output: > { > [1] = some s, > [3] = ring, > [2] = t > } > expected: > { > [1] = some string > } > tested on bro 2.1 and github master 2.1-824 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Thu Jul 25 12:44:04 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 25 Jul 2013 14:44:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1039) Merge request for Bloom filters In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13316#comment-13316 ] Robin Sommer commented on BIT-1039: ----------------------------------- Merged, thanks. However, we need to address the hash problem to support merging across Bro instances, I'm leaving the ticket open for that. Here's a proposal what to do (after talking to Bernhard): 1. Change {{CompositeHash}} to optionally use a custom {{H3}} instance. 2. Extend the {{Hasher}} to take both a name and an optional seed value (probably another string). Internally, it combines the two into the seed for the {{Hasher's}} internal H3, i.e., same name+seed means same hash functions. If the optional seed value is not given, take it from a global script level variable {{GLOBAL_INSTALLATION_SEED}} (or so :). 3. Change BloomFilterVal to pass to {{CompositeHash}} a custom {{H3}}. I believe this could be the same instance that {{Hasher}} is using internally, so that we get the same consistency guarantees. Indeed, hashing of {{Val}} should then probably move into {{Hasher}}. 3. Along the same lines as (2), extend the bloom filter interface to take both a name and the optional seed. Same name+seed means filters can be merged. 4. Change BroControl to redefine {{GLOBAL_INSTALLATION_SEED}} to a non-predictable value that will remain consistent across {{install}}. I believe that with this we can support two use cases: (1) in a cluster, all bloom filters created with the same name but without any further seed value will be compatible (because they'll use {{GLOBAL_INSTALLATION_SEED}}); and (2) externally provided Bloom filters can specify their own seed so that any Bro installation can pull them in. Does this make sense? > Merge request for Bloom filters > ------------------------------- > > Key: BIT-1039 > URL: https://bro-tracker.atlassian.net/browse/BIT-1039 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Reporter: Matthias Vallentin > Priority: Medium > Fix For: 2.2 > > > The Bloom filter implementation in `topic/matthias/bloom-filter` is ready to merge into master. Have a look at the very end of `bro.bif` for the script-land interface. > Internally, we have a new `BloomFilterVal`, which is serializable and mergeable and thus ready for cluster use. This `Val` contains a polymorphic Bloom filter instance, which hides the concrete Bloom filter type (currently only basic and counting). Moreover, this branch introduces the notion of ''hashers'', which are parameterizable (i.e., seedable) structures for hashing values ''k'' times. I recall that Bernhard waits for this feature. See `Hasher.h` for the documented interface. > In the future, we need to rethink how to construct hash functions which only depend on a seed given at script land. This will be important when sharing Bloom filters across organizational boundaries. At this point, the implementation relies on `CompHash` (at least for composite values, such as records) which itself depends on the initial Bro seed generated at startup time or when the user specifies the environment variable `$BRO_SEED`. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Thu Jul 25 12:51:04 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 25 Jul 2013 14:51:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1039) Merge request for Bloom filters In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1039: ------------------------------ Status: Open (was: Merge Request) > Merge request for Bloom filters > ------------------------------- > > Key: BIT-1039 > URL: https://bro-tracker.atlassian.net/browse/BIT-1039 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Reporter: Matthias Vallentin > Priority: Medium > Fix For: 2.2 > > > The Bloom filter implementation in `topic/matthias/bloom-filter` is ready to merge into master. Have a look at the very end of `bro.bif` for the script-land interface. > Internally, we have a new `BloomFilterVal`, which is serializable and mergeable and thus ready for cluster use. This `Val` contains a polymorphic Bloom filter instance, which hides the concrete Bloom filter type (currently only basic and counting). Moreover, this branch introduces the notion of ''hashers'', which are parameterizable (i.e., seedable) structures for hashing values ''k'' times. I recall that Bernhard waits for this feature. See `Hasher.h` for the documented interface. > In the future, we need to rethink how to construct hash functions which only depend on a seed given at script land. This will be important when sharing Bloom filters across organizational boundaries. At this point, the implementation relies on `CompHash` (at least for composite values, such as records) which itself depends on the initial Bro seed generated at startup time or when the user specifies the environment variable `$BRO_SEED`. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From robin at icir.org Thu Jul 25 14:40:51 2013 From: robin at icir.org (Robin Sommer) Date: Thu, 25 Jul 2013 14:40:51 -0700 Subject: [Bro-Dev] [JIRA] (BIT-1042) Caret in regex pattern doesn't match start of line In-Reply-To: References: Message-ID: <20130725214051.GA95898@icir.org> Yeah, that looks like a bug. Bro's treatment of "^/$" is a bit, let's say, complex unfortunately. Robin On Thu, Jul 25, 2013 at 11:21 -0500, you wrote: > > [ https://bro-tracker.atlassian.net/browse/BIT-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13315#comment-13315 ] > > Brian Little commented on BIT-1042: > ----------------------------------- > > I was using that as an simple example, but it shouldn't match at all. > > The specific problem I have it trying to remove the first . in a domain, using > > print sub(".github.com", /^\./, ""); > #github.com > > This works, however that specific pattern /^\./ also matches this: > > print sub("github.com", /^\./, ""); > #githubcom (note missing .) > > Is "sub" the wrong tool to use for this? It seems to work for other patterns fine, including the $ (end of string), but not for the start of string > > > Caret in regex pattern doesn't match start of line > > -------------------------------------------------- > > > > Key: BIT-1042 > > URL: https://bro-tracker.atlassian.net/browse/BIT-1042 > > Project: Bro Issue Tracker > > Issue Type: Bug > > Components: Bro > > Affects Versions: 2.1 > > Environment: Ubuntu 12.04 and 13.04. Dependencies from apt-get ubuntu repos only. > > Reporter: Brian Little > > Priority: Normal > > Labels: pattern, regex > > Attachments: caret.bro > > > > > > print split_all("some string", /^t/); > > I would expect it to not match ^t, but it matches any t in the string. > > output: > > { > > [1] = some s, > > [3] = ring, > > [2] = t > > } > > expected: > > { > > [1] = some string > > } > > tested on bro 2.1 and github master 2.1-824 > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa > For more information on JIRA, see: http://www.atlassian.com/software/jira > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From jira at bro-tracker.atlassian.net Thu Jul 25 14:43:04 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Thu, 25 Jul 2013 16:43:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1042) Caret in regex pattern doesn't match start of line In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13317#comment-13317 ] Robin Sommer commented on BIT-1042: ----------------------------------- Yeah, that looks like a bug. Bro's treatment of "^/$" is a bit, let's say, complex unfortunately. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin > Caret in regex pattern doesn't match start of line > -------------------------------------------------- > > Key: BIT-1042 > URL: https://bro-tracker.atlassian.net/browse/BIT-1042 > Project: Bro Issue Tracker > Issue Type: Bug > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu 12.04 and 13.04. Dependencies from apt-get ubuntu repos only. > Reporter: Brian Little > Priority: Normal > Labels: pattern, regex > Attachments: caret.bro > > > print split_all("some string", /^t/); > I would expect it to not match ^t, but it matches any t in the string. > output: > { > [1] = some s, > [3] = ring, > [2] = t > } > expected: > { > [1] = some string > } > tested on bro 2.1 and github master 2.1-824 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Fri Jul 26 02:12:04 2013 From: jira at bro-tracker.atlassian.net (Brian Little (JIRA)) Date: Fri, 26 Jul 2013 04:12:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1042) Caret in regex pattern doesn't match start of line In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Little updated BIT-1042: ------------------------------ Status: Reopened (was: Closed) > Caret in regex pattern doesn't match start of line > -------------------------------------------------- > > Key: BIT-1042 > URL: https://bro-tracker.atlassian.net/browse/BIT-1042 > Project: Bro Issue Tracker > Issue Type: Bug > Components: Bro > Affects Versions: 2.1 > Environment: Ubuntu 12.04 and 13.04. Dependencies from apt-get ubuntu repos only. > Reporter: Brian Little > Priority: Normal > Labels: pattern, regex > Attachments: caret.bro > > > print split_all("some string", /^t/); > I would expect it to not match ^t, but it matches any t in the string. > output: > { > [1] = some s, > [3] = ring, > [2] = t > } > expected: > { > [1] = some string > } > tested on bro 2.1 and github master 2.1-824 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Fri Jul 26 09:11:04 2013 From: jira at bro-tracker.atlassian.net (Jordi Ros-Giralt (JIRA)) Date: Fri, 26 Jul 2013 11:11:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1043) LRU Table implementation In-Reply-To: References: Message-ID: Jordi Ros-Giralt created BIT-1043: ------------------------------------- Summary: LRU Table implementation Key: BIT-1043 URL: https://bro-tracker.atlassian.net/browse/BIT-1043 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: 2.1 Reporter: Jordi Ros-Giralt Priority: Medium Attaching below the email description i exchanged with Seth and Robin describing this work. ------ Hi Seth and Robin, We got the repo up, you can get to our branch as follows: git clone --recursive https://github.com/giralt/bro.git cd bro/ git checkout lru-table We would be happy to contribute this code to the Bro community. This is what it does: - It implements LRU tables for Bro - A Bro table can be enhanced with the LRU functionality with the following new table attributes: &lru_table: enhance the table with LRU functionality &size_limit=n: if adding an element to the table makes the size of the table larger than n, then drop the LRU element from that table before inserting the new element. n=0 means table size can be infinite (so don't drop elements from it) &drop_func=callback_func: defines a programmable callback function that gets called automatically every time an element from the LRU table is dropped due to hitting the size_limit. The prototype of this callback must be as follows: function callback_func(t: table[keytype] of valuetype, key: keytype, val: valuetype): count - It adds the following bif functions: function get_lru%(v: any%): any function get_mru%(v: any%): any function get_lru_key%(v: any%): any function get_mru_key%(v: any%): any - Example: function freed(t: table[port] of string, key: port, val: string): count { print "Dropped"; } local port_names: table[port] of string &lru &size_limit=2 &drop_func=freed; In terms of applications, we are currently using this feature for the chimera-to-bro compiler we are working on: http://www.chimera-query.org/index.html We thought that we could also use this feature to provide a sort of memory management facility for Bro. I had a talk with Seth some weeks ago about this. Something like the LRU implementation allows programmers to put bounds on the size of tables and prioritize which elements can be dropped first upon memory exhaustion scenarios. Perhaps an idea here would be to develop a garbage collector (could be done using Bro language itself, perhaps as a framework) which would be run upon hitting a certain memory usage watermark and which would mainly run through the set of tables marked as "garbage collectable" dropping LRU elements from them to help reduce/eliminate the risk of running out of memory. Should this be something interesting, what are the steps we would need to do to open source the LRU code into Bro? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Fri Jul 26 09:29:04 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Fri, 26 Jul 2013 11:29:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1043) LRU Table implementation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1043: --------------------------- Affects Version/s: (was: 2.1) 2.3 > LRU Table implementation > ------------------------ > > Key: BIT-1043 > URL: https://bro-tracker.atlassian.net/browse/BIT-1043 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.3 > Reporter: Jordi Ros-Giralt > Priority: Medium > > Attaching below the email description i exchanged with Seth and Robin describing this work. > ------ > Hi Seth and Robin, > We got the repo up, you can get to our branch as follows: > git clone --recursive https://github.com/giralt/bro.git > cd bro/ > git checkout lru-table > We would be happy to contribute this code to the Bro community. This is what it does: > - It implements LRU tables for Bro > - A Bro table can be enhanced with the LRU functionality with the following new table attributes: > &lru_table: enhance the table with LRU functionality > &size_limit=n: if adding an element to the table makes the size of the table larger than n, then drop the LRU element from that table before inserting the new element. n=0 means table size can be infinite (so don't drop elements from it) > &drop_func=callback_func: defines a programmable callback function that gets called automatically every time an element from the LRU table is dropped due to hitting the size_limit. The prototype of this callback must be as follows: > function callback_func(t: table[keytype] of valuetype, key: keytype, val: valuetype): count > - It adds the following bif functions: > function get_lru%(v: any%): any > function get_mru%(v: any%): any > function get_lru_key%(v: any%): any > function get_mru_key%(v: any%): any > - Example: > function freed(t: table[port] of string, key: port, val: string): count { print "Dropped"; } > local port_names: table[port] of string &lru &size_limit=2 &drop_func=freed; > In terms of applications, we are currently using this feature for the chimera-to-bro compiler we are working on: http://www.chimera-query.org/index.html > We thought that we could also use this feature to provide a sort of memory management facility for Bro. I had a talk with Seth some weeks ago about this. Something like the LRU implementation allows programmers to put bounds on the size of tables and prioritize which elements can be dropped first upon memory exhaustion scenarios. Perhaps an idea here would be to develop a garbage collector (could be done using Bro language itself, perhaps as a framework) which would be run upon hitting a certain memory usage watermark and which would mainly run through the set of tables marked as "garbage collectable" dropping LRU elements from them to help reduce/eliminate the risk of running out of memory. > Should this be something interesting, what are the steps we would need to do to open source the LRU code into Bro? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From seth at icir.org Fri Jul 26 13:26:36 2013 From: seth at icir.org (Seth Hall) Date: Fri, 26 Jul 2013 16:26:36 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/exec-module: Exec module changes/fixes. (73eb87a) In-Reply-To: <201307231931.r6NJVYhH009889@bro-ids.icir.org> References: <201307231931.r6NJVYhH009889@bro-ids.icir.org> Message-ID: <854475B7-2F76-449F-A38C-B12173E42D5E@icir.org> On Jul 23, 2013, at 3:31 PM, Jonathan Siwek wrote: > Exec module changes/fixes. Do you feel comfortable with this being merged into master now? I'm cool with it if you are. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From jira at bro-tracker.atlassian.net Fri Jul 26 20:30:04 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Fri, 26 Jul 2013 22:30:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1044) topic/seth/faf-updates ready for merge In-Reply-To: References: Message-ID: Seth Hall created BIT-1044: ------------------------------ Summary: topic/seth/faf-updates ready for merge Key: BIT-1044 URL: https://bro-tracker.atlassian.net/browse/BIT-1044 Project: Bro Issue Tracker Issue Type: Task Reporter: Seth Hall Assignee: Robin Sommer Priority: Normal Big updates to logs and functionality of the files framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Fri Jul 26 20:30:04 2013 From: jira at bro-tracker.atlassian.net (Seth Hall (JIRA)) Date: Fri, 26 Jul 2013 22:30:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1044) topic/seth/faf-updates ready for merge In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1044: --------------------------- Status: Merge Request (was: Open) > topic/seth/faf-updates ready for merge > -------------------------------------- > > Key: BIT-1044 > URL: https://bro-tracker.atlassian.net/browse/BIT-1044 > Project: Bro Issue Tracker > Issue Type: Task > Reporter: Seth Hall > Assignee: Robin Sommer > Priority: Normal > > Big updates to logs and functionality of the files framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From seth at icir.org Fri Jul 26 20:31:44 2013 From: seth at icir.org (Seth Hall) Date: Fri, 26 Jul 2013 23:31:44 -0400 Subject: [Bro-Dev] Merge requests? Message-ID: I suspect that our nightly merge request emails aren't working yet? Anyway, the faf-updates branch is ready to be merged and has equivalently named branches in the public and private test suites. https://bro-tracker.atlassian.net/browse/BIT-1044 .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From robin at icir.org Fri Jul 26 21:46:05 2013 From: robin at icir.org (Robin Sommer) Date: Fri, 26 Jul 2013 21:46:05 -0700 Subject: [Bro-Dev] Merge requests? In-Reply-To: References: Message-ID: <20130727044605.GA69707@icir.org> Ok. The nightly script is almost ready for JIRA, will make it live soon. Robin On Fri, Jul 26, 2013 at 23:31 -0400, you wrote: > I suspect that our nightly merge request emails aren't working yet? Anyway, the faf-updates branch is ready to be merged and has equivalently named branches in the public and private test suites. > > https://bro-tracker.atlassian.net/browse/BIT-1044 > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > > -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From jira at bro-tracker.atlassian.net Sun Jul 28 07:11:04 2013 From: jira at bro-tracker.atlassian.net (Vlad Grigorescu (JIRA)) Date: Sun, 28 Jul 2013 09:11:04 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1045) Review usage of InternalError when parsing network traffic In-Reply-To: References: Message-ID: Vlad Grigorescu created BIT-1045: ------------------------------------ Summary: Review usage of InternalError when parsing network traffic Key: BIT-1045 URL: https://bro-tracker.atlassian.net/browse/BIT-1045 Project: Bro Issue Tracker Issue Type: Task Components: Bro Affects Versions: 2.1, git/master Reporter: Vlad Grigorescu Creating issue for tracking purposes. Reporter->InternalError denotes a fatal error, and will cause Bro to stop. Calling this function when parsing network traffic creates the possibility for an attacker using a "packet of death," which could stop Bro. I suspect that in most cases, a weird should be generated instead, and Bro should just move on to the next packet. A quick grep shows some likely candidates for incorrect use of InternalError: src/Sessions.cc: reporter->InternalError("Bad IP protocol version in DoNextInnerPacket"); src/Sessions.cc: reporter->InternalError("fragment block not in dictionary"); src/Sessions.cc: reporter->InternalError("fragment block missing"); src/Sessions.cc: reporter->InternalError("unknown transport protocol"); src/Frag.cc: reporter->InternalError("bad IP version in fragment reassembly"); src/IP.cc: reporter->InternalError("IPv6_HdrChain::Init with truncated IP header"); src/IP.cc: reporter->InternalError("IPv6_Hdr_Chain bad header %d", type); src/IP.h: reporter->InternalError("bad IP version in IP_Hdr ctor"); src/RSH.cc: reporter->InternalError("multiple rsh client names"); src/RSH.cc: reporter->InternalError("multiple rsh initial client names"); src/POP3.cc: reporter->InternalError("command not known"); src/Rlogin.cc: reporter->InternalError("multiple rlogin client names"); src/ICMP.cc: reporter->InternalError("unexpected IP proto in ICMP analyzer: %d", src/ICMP.cc: reporter->InternalError("unexpected next protocol in ICMP::DeliverPacket()"); src/SMB.cc: reporter->InternalError("command mismatch for ParseTransaction"); src/HTTP.cc: reporter->InternalError("unrecognized HTTP message event"); src/HTTP.cc: reporter->InternalError("HTTP ParseRequest failed"); src/DPM.cc: reporter->InternalError("unknown protocol"); src/RPC.cc: reporter->InternalError("RPC underflow"); src/RPC.cc: reporter->InternalError("RPC resync: skipping over data failed"); src/RPC.cc: reporter->InternalError("inconsistent RPC record marker extraction"); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Mon Jul 29 07:46:06 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 29 Jul 2013 09:46:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1043) LRU Table implementation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1043: --------------------------- Affects Version/s: (was: 2.3) 2.1 > LRU Table implementation > ------------------------ > > Key: BIT-1043 > URL: https://bro-tracker.atlassian.net/browse/BIT-1043 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.1 > Reporter: Jordi Ros-Giralt > Fix For: 2.3 > > > Attaching below the email description i exchanged with Seth and Robin describing this work. > ------ > Hi Seth and Robin, > We got the repo up, you can get to our branch as follows: > git clone --recursive https://github.com/giralt/bro.git > cd bro/ > git checkout lru-table > We would be happy to contribute this code to the Bro community. This is what it does: > - It implements LRU tables for Bro > - A Bro table can be enhanced with the LRU functionality with the following new table attributes: > &lru_table: enhance the table with LRU functionality > &size_limit=n: if adding an element to the table makes the size of the table larger than n, then drop the LRU element from that table before inserting the new element. n=0 means table size can be infinite (so don't drop elements from it) > &drop_func=callback_func: defines a programmable callback function that gets called automatically every time an element from the LRU table is dropped due to hitting the size_limit. The prototype of this callback must be as follows: > function callback_func(t: table[keytype] of valuetype, key: keytype, val: valuetype): count > - It adds the following bif functions: > function get_lru%(v: any%): any > function get_mru%(v: any%): any > function get_lru_key%(v: any%): any > function get_mru_key%(v: any%): any > - Example: > function freed(t: table[port] of string, key: port, val: string): count { print "Dropped"; } > local port_names: table[port] of string &lru &size_limit=2 &drop_func=freed; > In terms of applications, we are currently using this feature for the chimera-to-bro compiler we are working on: http://www.chimera-query.org/index.html > We thought that we could also use this feature to provide a sort of memory management facility for Bro. I had a talk with Seth some weeks ago about this. Something like the LRU implementation allows programmers to put bounds on the size of tables and prioritize which elements can be dropped first upon memory exhaustion scenarios. Perhaps an idea here would be to develop a garbage collector (could be done using Bro language itself, perhaps as a framework) which would be run upon hitting a certain memory usage watermark and which would mainly run through the set of tables marked as "garbage collectable" dropping LRU elements from them to help reduce/eliminate the risk of running out of memory. > Should this be something interesting, what are the steps we would need to do to open source the LRU code into Bro? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Mon Jul 29 07:46:06 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 29 Jul 2013 09:46:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1043) LRU Table implementation In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1043: --------------------------- Fix Version/s: 2.3 > LRU Table implementation > ------------------------ > > Key: BIT-1043 > URL: https://bro-tracker.atlassian.net/browse/BIT-1043 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.1 > Reporter: Jordi Ros-Giralt > Fix For: 2.3 > > > Attaching below the email description i exchanged with Seth and Robin describing this work. > ------ > Hi Seth and Robin, > We got the repo up, you can get to our branch as follows: > git clone --recursive https://github.com/giralt/bro.git > cd bro/ > git checkout lru-table > We would be happy to contribute this code to the Bro community. This is what it does: > - It implements LRU tables for Bro > - A Bro table can be enhanced with the LRU functionality with the following new table attributes: > &lru_table: enhance the table with LRU functionality > &size_limit=n: if adding an element to the table makes the size of the table larger than n, then drop the LRU element from that table before inserting the new element. n=0 means table size can be infinite (so don't drop elements from it) > &drop_func=callback_func: defines a programmable callback function that gets called automatically every time an element from the LRU table is dropped due to hitting the size_limit. The prototype of this callback must be as follows: > function callback_func(t: table[keytype] of valuetype, key: keytype, val: valuetype): count > - It adds the following bif functions: > function get_lru%(v: any%): any > function get_mru%(v: any%): any > function get_lru_key%(v: any%): any > function get_mru_key%(v: any%): any > - Example: > function freed(t: table[port] of string, key: port, val: string): count { print "Dropped"; } > local port_names: table[port] of string &lru &size_limit=2 &drop_func=freed; > In terms of applications, we are currently using this feature for the chimera-to-bro compiler we are working on: http://www.chimera-query.org/index.html > We thought that we could also use this feature to provide a sort of memory management facility for Bro. I had a talk with Seth some weeks ago about this. Something like the LRU implementation allows programmers to put bounds on the size of tables and prioritize which elements can be dropped first upon memory exhaustion scenarios. Perhaps an idea here would be to develop a garbage collector (could be done using Bro language itself, perhaps as a framework) which would be run upon hitting a certain memory usage watermark and which would mainly run through the set of tables marked as "garbage collectable" dropping LRU elements from them to help reduce/eliminate the risk of running out of memory. > Should this be something interesting, what are the steps we would need to do to open source the LRU code into Bro? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jsiwek at illinois.edu Mon Jul 29 07:49:28 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Mon, 29 Jul 2013 14:49:28 +0000 Subject: [Bro-Dev] [JIRA] (BIT-1043) LRU Table implementation In-Reply-To: References: Message-ID: "Fix Version" is what's used to put something on the "roadmap." i.e. it can be used like "milestones" were w/ Trac. (I think that's what was intended by Seth's "Affects Version change.) - Jon On Jul 29, 2013, at 9:46 AM, "Jon Siwek (JIRA)" wrote: > > [ https://bro-tracker.atlassian.net/browse/BIT-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] > > Jon Siwek updated BIT-1043: > --------------------------- > > Fix Version/s: 2.3 > >> LRU Table implementation >> ------------------------ >> >> Key: BIT-1043 >> URL: https://bro-tracker.atlassian.net/browse/BIT-1043 >> Project: Bro Issue Tracker >> Issue Type: Improvement >> Components: Bro >> Affects Versions: 2.1 >> Reporter: Jordi Ros-Giralt >> Fix For: 2.3 >> >> >> Attaching below the email description i exchanged with Seth and Robin describing this work. >> ------ >> Hi Seth and Robin, >> We got the repo up, you can get to our branch as follows: >> git clone --recursive https://github.com/giralt/bro.git >> cd bro/ >> git checkout lru-table >> We would be happy to contribute this code to the Bro community. This is what it does: >> - It implements LRU tables for Bro >> - A Bro table can be enhanced with the LRU functionality with the following new table attributes: >> &lru_table: enhance the table with LRU functionality >> &size_limit=n: if adding an element to the table makes the size of the table larger than n, then drop the LRU element from that table before inserting the new element. n=0 means table size can be infinite (so don't drop elements from it) >> &drop_func=callback_func: defines a programmable callback function that gets called automatically every time an element from the LRU table is dropped due to hitting the size_limit. The prototype of this callback must be as follows: >> function callback_func(t: table[keytype] of valuetype, key: keytype, val: valuetype): count >> - It adds the following bif functions: >> function get_lru%(v: any%): any >> function get_mru%(v: any%): any >> function get_lru_key%(v: any%): any >> function get_mru_key%(v: any%): any >> - Example: >> function freed(t: table[port] of string, key: port, val: string): count { print "Dropped"; } >> local port_names: table[port] of string &lru &size_limit=2 &drop_func=freed; >> In terms of applications, we are currently using this feature for the chimera-to-bro compiler we are working on: http://www.chimera-query.org/index.html >> We thought that we could also use this feature to provide a sort of memory management facility for Bro. I had a talk with Seth some weeks ago about this. Something like the LRU implementation allows programmers to put bounds on the size of tables and prioritize which elements can be dropped first upon memory exhaustion scenarios. Perhaps an idea here would be to develop a garbage collector (could be done using Bro language itself, perhaps as a framework) which would be run upon hitting a certain memory usage watermark and which would mainly run through the set of tables marked as "garbage collectable" dropping LRU elements from them to help reduce/eliminate the risk of running out of memory. >> Should this be something interesting, what are the steps we would need to do to open source the LRU code into Bro? > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa > For more information on JIRA, see: http://www.atlassian.com/software/jira > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From jira at bro-tracker.atlassian.net Mon Jul 29 07:57:06 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 29 Jul 2013 09:57:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1046) topic/jsiwek/exec-module In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1046: --------------------------- Status: Merge Request (was: Open) > topic/jsiwek/exec-module > ------------------------ > > Key: BIT-1046 > URL: https://bro-tracker.atlassian.net/browse/BIT-1046 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.1 > Reporter: Jon Siwek > Fix For: 2.2 > > > Some scripts for executing system commands and getting the results (stderr/stdout, exit code, file output) back in to Bro. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Mon Jul 29 07:57:06 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 29 Jul 2013 09:57:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1046) topic/jsiwek/exec-module In-Reply-To: References: Message-ID: Jon Siwek created BIT-1046: ------------------------------ Summary: topic/jsiwek/exec-module Key: BIT-1046 URL: https://bro-tracker.atlassian.net/browse/BIT-1046 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: 2.1 Reporter: Jon Siwek Fix For: 2.2 Some scripts for executing system commands and getting the results (stderr/stdout, exit code, file output) back in to Bro. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Mon Jul 29 07:57:06 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Mon, 29 Jul 2013 09:57:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1044) topic/seth/faf-updates ready for merge In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-1044: --------------------------- Fix Version/s: 2.2 > topic/seth/faf-updates ready for merge > -------------------------------------- > > Key: BIT-1044 > URL: https://bro-tracker.atlassian.net/browse/BIT-1044 > Project: Bro Issue Tracker > Issue Type: Task > Reporter: Seth Hall > Assignee: Robin Sommer > Fix For: 2.2 > > > Big updates to logs and functionality of the files framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jsiwek at illinois.edu Mon Jul 29 07:56:18 2013 From: jsiwek at illinois.edu (Siwek, Jonathan Luke) Date: Mon, 29 Jul 2013 14:56:18 +0000 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/exec-module: Exec module changes/fixes. (73eb87a) In-Reply-To: <854475B7-2F76-449F-A38C-B12173E42D5E@icir.org> References: <201307231931.r6NJVYhH009889@bro-ids.icir.org> <854475B7-2F76-449F-A38C-B12173E42D5E@icir.org> Message-ID: >> Exec module changes/fixes. > > Do you feel comfortable with this being merged into master now? I'm not aware of any outstanding problems w/ it, so yes. I made a merge request ticket. - Jon From seth at icir.org Mon Jul 29 08:08:12 2013 From: seth at icir.org (Seth Hall) Date: Mon, 29 Jul 2013 11:08:12 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/jsiwek/exec-module: Exec module changes/fixes. (73eb87a) In-Reply-To: References: <201307231931.r6NJVYhH009889@bro-ids.icir.org> <854475B7-2F76-449F-A38C-B12173E42D5E@icir.org> Message-ID: <6BECF662-08BC-4DEF-BB87-AAB6C965D6DE@icir.org> On Jul 29, 2013, at 10:56 AM, "Siwek, Jonathan Luke" wrote: > >>> Exec module changes/fixes. >> >> Do you feel comfortable with this being merged into master now? > > I'm not aware of any outstanding problems w/ it, so yes. I made a merge request ticket. Cool, thanks. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ From robin at icir.org Mon Jul 29 09:56:28 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 29 Jul 2013 09:56:28 -0700 Subject: [Bro-Dev] [JIRA] (BIT-1045) Review usage of InternalError when parsing network traffic In-Reply-To: References: Message-ID: <20130729165628.GO36425@icir.org> On Sun, Jul 28, 2013 at 09:11 -0500, you wrote: > Reporter->InternalError denotes a fatal error, and will cause Bro to > stop. Calling this function when parsing network traffic creates the > possibility for an attacker using a "packet of death," which could > stop Bro. Ack, InternalError() is not something that external input should be able to trigger. I already removed a number of these over time, but never looked systematically for them. > I suspect that in most cases, a weird should be generated instead, and > Bro should just move on to the next packet. Agreed, though sometimes they aren't about the traffic but about a logic error in decoding it; it would be good to still differentiate those cases from a broken packet, however indeed without aborting. From jira at bro-tracker.atlassian.net Mon Jul 29 09:58:06 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 29 Jul 2013 11:58:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1045) Review usage of InternalError when parsing network traffic In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400#comment-13400 ] Robin Sommer commented on BIT-1045: ----------------------------------- Ack, InternalError() is not something that external input should be able to trigger. I already removed a number of these over time, but never looked systematically for them. Agreed, though sometimes they aren't about the traffic but about a logic error in decoding it; it would be good to still differentiate those cases from a broken packet, however indeed without aborting. > Review usage of InternalError when parsing network traffic > ---------------------------------------------------------- > > Key: BIT-1045 > URL: https://bro-tracker.atlassian.net/browse/BIT-1045 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Affects Versions: git/master, 2.1 > Reporter: Vlad Grigorescu > > Creating issue for tracking purposes. > Reporter->InternalError denotes a fatal error, and will cause Bro to stop. Calling this function when parsing network traffic creates the possibility for an attacker using a "packet of death," which could stop Bro. > I suspect that in most cases, a weird should be generated instead, and Bro should just move on to the next packet. A quick grep shows some likely candidates for incorrect use of InternalError: > src/Sessions.cc: reporter->InternalError("Bad IP protocol version in DoNextInnerPacket"); > src/Sessions.cc: reporter->InternalError("fragment block not in dictionary"); > src/Sessions.cc: reporter->InternalError("fragment block missing"); > src/Sessions.cc: reporter->InternalError("unknown transport protocol"); > src/Frag.cc: reporter->InternalError("bad IP version in fragment reassembly"); > src/IP.cc: reporter->InternalError("IPv6_HdrChain::Init with truncated IP header"); > src/IP.cc: reporter->InternalError("IPv6_Hdr_Chain bad header %d", type); > src/IP.h: reporter->InternalError("bad IP version in IP_Hdr ctor"); > src/RSH.cc: reporter->InternalError("multiple rsh client names"); > src/RSH.cc: reporter->InternalError("multiple rsh initial client names"); > src/POP3.cc: reporter->InternalError("command not known"); > src/Rlogin.cc: reporter->InternalError("multiple rlogin client names"); > src/ICMP.cc: reporter->InternalError("unexpected IP proto in ICMP analyzer: %d", > src/ICMP.cc: reporter->InternalError("unexpected next protocol in ICMP::DeliverPacket()"); > src/SMB.cc: reporter->InternalError("command mismatch for ParseTransaction"); > src/HTTP.cc: reporter->InternalError("unrecognized HTTP message event"); > src/HTTP.cc: reporter->InternalError("HTTP ParseRequest failed"); > src/DPM.cc: reporter->InternalError("unknown protocol"); > src/RPC.cc: reporter->InternalError("RPC underflow"); > src/RPC.cc: reporter->InternalError("RPC resync: skipping over data failed"); > src/RPC.cc: reporter->InternalError("inconsistent RPC record marker extraction"); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Mon Jul 29 15:50:06 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 29 Jul 2013 17:50:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: Robin Sommer created BIT-1047: --------------------------------- Summary: Delete old scripts before installing new ones Key: BIT-1047 URL: https://bro-tracker.atlassian.net/browse/BIT-1047 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Reporter: Robin Sommer Fix For: 2.2 People keep having problems when they install a new Bro version over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From vern at icir.org Mon Jul 29 20:08:30 2013 From: vern at icir.org (Vern Paxson) Date: Mon, 29 Jul 2013 20:08:30 -0700 Subject: [Bro-Dev] [JIRA] (BIT-1045) Review usage of InternalError when parsing network traffic In-Reply-To: <20130729165628.GO36425@icir.org> (Mon, 29 Jul 2013 09:56:28 PDT). Message-ID: <20130730030830.3C8AC2C4002@rock.ICSI.Berkeley.EDU> > > I suspect that in most cases, a weird should be generated instead, and > > Bro should just move on to the next packet. > > Agreed, though sometimes they aren't about the traffic but about a > logic error in decoding it; it would be good to still differentiate > those cases from a broken packet, however indeed without aborting. In line with what you frame, the history behind these is that they're meant to reflect should-never-happen situations; not "weird" activity, but apparent internal inconsistencies in Bro's processing/execution. So they don't really fit with the notion of "weird". (Of course it's definitely possible there's some mission-creep and InternalError is misused when Weird really is the right notion.) That said, for sure I agree that we don't want to give adversaries a way to tickle a Bro crash. So ideally the solution here would be to come up with something similar to the weird/notice framework, but that expicitly captures the notion that Bro-is-confused rather than something-happened-on-the-network. Vern From vern at icir.org Mon Jul 29 20:08:36 2013 From: vern at icir.org (Vern Paxson) Date: Mon, 29 Jul 2013 20:08:36 -0700 Subject: [Bro-Dev] viability of stand-alone (not installed) Bro Message-ID: <20130730030836.E8DDB2C4002@rock.ICSI.Berkeley.EDU> For various reasons I sometimes like to run Bro out of the directory I used to build it, rather than installing it. With a fresh git pull, when doing this I get: % build/src/bro internal error: can't load magic file /usr/local/bro/share/bro/magic: could not find any valid magic files! Abort Well harumph. However, the reason for this note (rather than a shiny-new-tracker ticket) is I'm wondering whether filing the ticket is a lost cause (i.e., the current philosophy is it's okay if things only work post "make install") ... ? Vern From jira at bro-tracker.atlassian.net Mon Jul 29 20:10:06 2013 From: jira at bro-tracker.atlassian.net (Vern Paxson (JIRA)) Date: Mon, 29 Jul 2013 22:10:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1045) Review usage of InternalError when parsing network traffic In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401#comment-13401 ] Vern Paxson commented on BIT-1045: ---------------------------------- In line with what you frame, the history behind these is that they're meant to reflect should-never-happen situations; not "weird" activity, but apparent internal inconsistencies in Bro's processing/execution. So they don't really fit with the notion of "weird". (Of course it's definitely possible there's some mission-creep and InternalError is misused when Weird really is the right notion.) That said, for sure I agree that we don't want to give adversaries a way to tickle a Bro crash. So ideally the solution here would be to come up with something similar to the weird/notice framework, but that expicitly captures the notion that Bro-is-confused rather than something-happened-on-the-network. Vern > Review usage of InternalError when parsing network traffic > ---------------------------------------------------------- > > Key: BIT-1045 > URL: https://bro-tracker.atlassian.net/browse/BIT-1045 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro > Affects Versions: git/master, 2.1 > Reporter: Vlad Grigorescu > > Creating issue for tracking purposes. > Reporter->InternalError denotes a fatal error, and will cause Bro to stop. Calling this function when parsing network traffic creates the possibility for an attacker using a "packet of death," which could stop Bro. > I suspect that in most cases, a weird should be generated instead, and Bro should just move on to the next packet. A quick grep shows some likely candidates for incorrect use of InternalError: > src/Sessions.cc: reporter->InternalError("Bad IP protocol version in DoNextInnerPacket"); > src/Sessions.cc: reporter->InternalError("fragment block not in dictionary"); > src/Sessions.cc: reporter->InternalError("fragment block missing"); > src/Sessions.cc: reporter->InternalError("unknown transport protocol"); > src/Frag.cc: reporter->InternalError("bad IP version in fragment reassembly"); > src/IP.cc: reporter->InternalError("IPv6_HdrChain::Init with truncated IP header"); > src/IP.cc: reporter->InternalError("IPv6_Hdr_Chain bad header %d", type); > src/IP.h: reporter->InternalError("bad IP version in IP_Hdr ctor"); > src/RSH.cc: reporter->InternalError("multiple rsh client names"); > src/RSH.cc: reporter->InternalError("multiple rsh initial client names"); > src/POP3.cc: reporter->InternalError("command not known"); > src/Rlogin.cc: reporter->InternalError("multiple rlogin client names"); > src/ICMP.cc: reporter->InternalError("unexpected IP proto in ICMP analyzer: %d", > src/ICMP.cc: reporter->InternalError("unexpected next protocol in ICMP::DeliverPacket()"); > src/SMB.cc: reporter->InternalError("command mismatch for ParseTransaction"); > src/HTTP.cc: reporter->InternalError("unrecognized HTTP message event"); > src/HTTP.cc: reporter->InternalError("HTTP ParseRequest failed"); > src/DPM.cc: reporter->InternalError("unknown protocol"); > src/RPC.cc: reporter->InternalError("RPC underflow"); > src/RPC.cc: reporter->InternalError("RPC resync: skipping over data failed"); > src/RPC.cc: reporter->InternalError("inconsistent RPC record marker extraction"); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From slagell at illinois.edu Mon Jul 29 20:13:29 2013 From: slagell at illinois.edu (Slagell, Adam J) Date: Tue, 30 Jul 2013 03:13:29 +0000 Subject: [Bro-Dev] viability of stand-alone (not installed) Bro In-Reply-To: <20130730030836.E8DDB2C4002@rock.ICSI.Berkeley.EDU> References: <20130730030836.E8DDB2C4002@rock.ICSI.Berkeley.EDU> Message-ID: <3EE576D3-1FE7-4E1D-91A9-8AC730D77455@illinois.edu> I don't think that is intended behavior, but rather an unintended consequence of some of the work on the file analysis framework and shipping with our own magic DB. Perhaps Jon can elaborate more on what it would take to fix this? On Jul 29, 2013, at 10:09 PM, "Vern Paxson" wrote: > For various reasons I sometimes like to run Bro out of the directory > I used to build it, rather than installing it. With a fresh git pull, > when doing this I get: > > % build/src/bro > internal error: can't load magic file /usr/local/bro/share/bro/magic: could not find any valid magic files! > Abort > > Well harumph. > > However, the reason for this note (rather than a shiny-new-tracker ticket) > is I'm wondering whether filing the ticket is a lost cause (i.e., the > current philosophy is it's okay if things only work post "make install") ... ? > > Vern > _______________________________________________ > bro-dev mailing list > bro-dev at bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From robin at icir.org Mon Jul 29 20:23:49 2013 From: robin at icir.org (Robin Sommer) Date: Mon, 29 Jul 2013 20:23:49 -0700 Subject: [Bro-Dev] viability of stand-alone (not installed) Bro In-Reply-To: <20130730030836.E8DDB2C4002@rock.ICSI.Berkeley.EDU> References: <20130730030836.E8DDB2C4002@rock.ICSI.Berkeley.EDU> Message-ID: <20130730032349.GD4913@icir.org> On Mon, Jul 29, 2013 at 20:08 -0700, you wrote: > I'm wondering whether filing the ticket is a lost cause (i.e., the > current philosophy is it's okay if things only work post "make > install") ... ? Quite the opposite: I'm sure most of us run it right out of the source tree more often than not. Try "source build/bro-path-dev.sh". Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org/robin From vern at icir.org Mon Jul 29 20:30:25 2013 From: vern at icir.org (Vern Paxson) Date: Mon, 29 Jul 2013 20:30:25 -0700 Subject: [Bro-Dev] viability of stand-alone (not installed) Bro In-Reply-To: <20130730032349.GD4913@icir.org> (Mon, 29 Jul 2013 20:23:49 PDT). Message-ID: <20130730033025.99AC02C4002@rock.ICSI.Berkeley.EDU> > Try "source build/bro-path-dev.sh". Cool, that does it. The only problem is surely I'm going to fail to remember that bit of voodoo, and bug you with similar questions in the future at least three more times :-P. Vern From jira at bro-tracker.atlassian.net Mon Jul 29 20:36:06 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 29 Jul 2013 22:36:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1044) topic/seth/faf-updates ready for merge In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1044: ------------------------------ Resolution: Merged Status: Closed (was: Merge Request) > topic/seth/faf-updates ready for merge > -------------------------------------- > > Key: BIT-1044 > URL: https://bro-tracker.atlassian.net/browse/BIT-1044 > Project: Bro Issue Tracker > Issue Type: Task > Reporter: Seth Hall > Assignee: Robin Sommer > Fix For: 2.2 > > > Big updates to logs and functionality of the files framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Mon Jul 29 20:36:06 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Mon, 29 Jul 2013 22:36:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1046) topic/jsiwek/exec-module In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1046: ------------------------------ Resolution: Merged Status: Closed (was: Merge Request) > topic/jsiwek/exec-module > ------------------------ > > Key: BIT-1046 > URL: https://bro-tracker.atlassian.net/browse/BIT-1046 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro > Affects Versions: 2.1 > Reporter: Jon Siwek > Fix For: 2.2 > > > Some scripts for executing system commands and getting the results (stderr/stdout, exit code, file output) back in to Bro. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Tue Jul 30 07:51:06 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 30 Jul 2013 09:51:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402#comment-13402 ] Jon Siwek commented on BIT-1047: -------------------------------- {quote} People keep having problems when they install a new Bro version over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. {quote} Is the problem that the stale scripts can actually get loaded or something else? I didn't think that they could be loaded unless new scripts actually @load old ones (bug in new scripts). Is there a specific example to help me understand? > Delete old scripts before installing new ones > --------------------------------------------- > > Key: BIT-1047 > URL: https://bro-tracker.atlassian.net/browse/BIT-1047 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Reporter: Robin Sommer > Fix For: 2.2 > > > People keep having problems when they install a new Bro version > over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. > We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Tue Jul 30 08:12:06 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 30 Jul 2013 10:12:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403#comment-13403 ] Robin Sommer commented on BIT-1047: ----------------------------------- Seth had one: somebody was loading one of the old scripts in local.bro. However, the error message was suggesting that the old script itslef had a problem because that's where teh error eas encountered (a record field had went away so that script didn't work anymore). If we had deleted the old script, it would have been clear that local.bro needs updating. More generally, not cleaning up leaves the install directory quite messy over time. If somebody wanted to understand what scripts Bro uses (or just what's available, sya in policy/), that could get quite confusing. Robin > Delete old scripts before installing new ones > --------------------------------------------- > > Key: BIT-1047 > URL: https://bro-tracker.atlassian.net/browse/BIT-1047 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Reporter: Robin Sommer > Fix For: 2.2 > > > People keep having problems when they install a new Bro version > over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. > We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Tue Jul 30 09:24:06 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Tue, 30 Jul 2013 11:24:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404#comment-13404 ] Bernhard Amann commented on BIT-1047: ------------------------------------- Would it perhaps be possible to only delete files that were present in the original Bro installation at some point of time? I know people are not supposed to put stuff into base - but I am kind of afraid that some people still do and that we could potentially delete things they do not want to have deleted. Perhaps we could check if there is stuff present in the directory besides the stuff that is being installed at the moment and warn? Bernhard > Delete old scripts before installing new ones > --------------------------------------------- > > Key: BIT-1047 > URL: https://bro-tracker.atlassian.net/browse/BIT-1047 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Reporter: Robin Sommer > Fix For: 2.2 > > > People keep having problems when they install a new Bro version > over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. > We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Tue Jul 30 09:34:06 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 30 Jul 2013 11:34:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405#comment-13405 ] Robin Sommer commented on BIT-1047: ----------------------------------- That sounds quite cumbersome to track. Also people already risk their stuff being overwritten whenever there's something with the same name in the distribution (i.e., this is only about someone adding news files; if he's editing existing files, he'll already loose it). Also note that this applies only to source installs; people installing from packages (including security onion) won't be affected. One precaution we could take is making the installation directory read-only so that if somebody started editing stuff in there, he'd at least need to override that. I think overall my preference here is making sure "make install" works reliably and produces consistent results over saving people who edited stuff they shouldn't ... > Delete old scripts before installing new ones > --------------------------------------------- > > Key: BIT-1047 > URL: https://bro-tracker.atlassian.net/browse/BIT-1047 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Reporter: Robin Sommer > Fix For: 2.2 > > > People keep having problems when they install a new Bro version > over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. > We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Tue Jul 30 09:57:06 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Tue, 30 Jul 2013 11:57:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1010) BroControl plugin for adding environment variables In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406#comment-13406 ] Robin Sommer commented on BIT-1010: ----------------------------------- I like the idea of specifying the variables separately, putting it all in one line seems very fragile. How about this syntax: {code} env:VAR1=1 env:VAR2=2 {code} I believe the parser should support that, no? We would just iterate through all entries and pick out those starting with {{env:}}. Same for {{broctl.cfg}}. Regarding the Myricom, I didn't compare with PF_RING, I just noticed that it seems to override any previously set value rather than adding to it. If node.cfg has priority, that's the most important thing, but still it seems fragile because if other parts of the code had set a (different) environment variable before, the plugin would override that. So all I suggest is making that {{+=}}, > BroControl plugin for adding environment variables > -------------------------------------------------- > > Key: BIT-1010 > URL: https://bro-tracker.atlassian.net/browse/BIT-1010 > Project: Bro Issue Tracker > Issue Type: Task > Components: BroControl > Affects Versions: git/master > Reporter: Seth Hall > Assignee: Daniel Thayer > Fix For: 2.2 > > Attachments: env_vars.py > > > We should have the ability to add environment variables to Bro at start up time. The option should be available globally in broctl.cfg and per-node in node.cfg. The environments variables should be applied to the process with priority based on how specific the variable is applied (per-node variables defined after global variables so that the per-node variable is used). > As a name suggestion for the configuration option: env_vars (same name in node.cfg and broctl.cfg). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Tue Jul 30 10:50:06 2013 From: jira at bro-tracker.atlassian.net (liamrandall (JIRA)) Date: Tue, 30 Jul 2013 12:50:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13407#comment-13407 ] liamrandall commented on BIT-1047: ---------------------------------- Does it make sense to support a versioning w/i the code tree- perhaps a root directory w/ a distribution release? It would be nice to have multiple versions installed side by side so you could roll back if for some reason you have trouble. Somewhere in a script definition, perhaps in a __load__ could there be a supported version feature? Specify minimum or precise version of support for a script. > Delete old scripts before installing new ones > --------------------------------------------- > > Key: BIT-1047 > URL: https://bro-tracker.atlassian.net/browse/BIT-1047 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Reporter: Robin Sommer > Fix For: 2.2 > > > People keep having problems when they install a new Bro version > over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. > We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Tue Jul 30 11:46:06 2013 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Tue, 30 Jul 2013 13:46:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-920) Have broctl return useful exit codes In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408#comment-13408 ] Daniel Thayer commented on BIT-920: ----------------------------------- In branch topic/dnthayer/bug920, I've made some small fixes to the patch, and added support to broctl commands that were missed by the original patch. The only remaining issues I see are: 1) "broctl cron" always returns 0 (this might not matter, because the return value doesn't affect the emails that cron sends), 2) "broctl status" and "broctl top" return 0 if all bro nodes are running, and returns 1 otherwise (to me, this seems inconsistent with the other commands, where a non-zero return value indicates some error occurred), 3) there is no mechanism to return non-zero for a command provided by a broctl plugin (such as "broctl ps.bro") > Have broctl return useful exit codes > ------------------------------------ > > Key: BIT-920 > URL: https://bro-tracker.atlassian.net/browse/BIT-920 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl > Affects Versions: git/master > Reporter: grigorescu > Assignee: Daniel Thayer > Fix For: 2.2 > > > I've got a broctl branch here: https://github.com/grigorescu/broctl which aims to have it return a 0 or 1 exit code for most execution paths. My dive down this particular rabbit hole started when I wanted to have status return a non-zero exit code if a node had failed, but I tried to cover everything else while I was at it. > If someone could double-check it, to make sure that I didn't miss anything, it'd be much appreciated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Tue Jul 30 12:33:06 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Tue, 30 Jul 2013 14:33:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409#comment-13409 ] Jon Siwek commented on BIT-1047: -------------------------------- So more generally the problem is "the Bro distribution may update default settings in config files from version to version, but in the case a user has modified a config file, they're also responsible for checking the diff and doing a merge (against the distribution's new file named ".example") after they've done `make install`". Should we make the output of `make install` more clearly indicate warnings for such cases? I'm just thinking that deleting old scripts during the `make install` process resolves the particular issue mentioned here, but there's still possibilities for other user-modified config files that are *not* bro scripts (e.g. broctl.cfg) to cause a similar issue if/when a config option were to be removed in a future version? > Delete old scripts before installing new ones > --------------------------------------------- > > Key: BIT-1047 > URL: https://bro-tracker.atlassian.net/browse/BIT-1047 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Reporter: Robin Sommer > Fix For: 2.2 > > > People keep having problems when they install a new Bro version > over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. > We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Tue Jul 30 16:33:06 2013 From: jira at bro-tracker.atlassian.net (Daniel Thayer (JIRA)) Date: Tue, 30 Jul 2013 18:33:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1010) BroControl plugin for adding environment variables In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410#comment-13410 ] Daniel Thayer commented on BIT-1010: ------------------------------------ We are using Python's ConfigParser to read the node.cfg file. It doesn't support the notion of having multiple instances of a key in the same section (it ignores duplicates). However, I've just made a change (in branch topic/dnthayer/ticket1010) to make env_vars stored as a dictionary internally and I've added more error-checking of the user-supplied list of environment variables, so that should help improve the situation. Regarding the Myricom plugin, I didn't use "+=" because that would override the user's values from node.cfg. Instead, I put the plugin's environment variables at the beginning of the list (so that the user's values can override the plugin's values). However, env_vars has now been re-implemented as a dictionary, so hopefully the code is easier to read. > BroControl plugin for adding environment variables > -------------------------------------------------- > > Key: BIT-1010 > URL: https://bro-tracker.atlassian.net/browse/BIT-1010 > Project: Bro Issue Tracker > Issue Type: Task > Components: BroControl > Affects Versions: git/master > Reporter: Seth Hall > Assignee: Daniel Thayer > Fix For: 2.2 > > Attachments: env_vars.py > > > We should have the ability to add environment variables to Bro at start up time. The option should be available globally in broctl.cfg and per-node in node.cfg. The environments variables should be applied to the process with priority based on how specific the variable is applied (per-node variables defined after global variables so that the per-node variable is used). > As a name suggestion for the configuration option: env_vars (same name in node.cfg and broctl.cfg). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From noreply at bro.org Wed Jul 31 07:57:05 2013 From: noreply at bro.org (Merge Tracker) Date: Wed, 31 Jul 2013 07:57:05 -0700 Subject: [Bro-Dev] [Auto] Merge Status Message-ID: <201307311457.r6VEv5p5011714@bro-ids.icir.org> Open Fastpath Commits ====================== Commit Component Author Date Summary ----------- ----------- -------------- ---------- ------------------------------------------------------------ edb04e6 [1] bro Bernhard Amann 2013-07-30 fix segfault that could be caused by merging an empty bloom- [1] edb04e6 https://github.com/bro/bro/commit/edb04e6d8bfe68dddf6968ec37cf39ea3a47feab From jira at bro-tracker.atlassian.net Wed Jul 31 10:28:06 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 31 Jul 2013 12:28:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411#comment-13411 ] Robin Sommer commented on BIT-1047: ----------------------------------- While I'd answer "yes" to that question, I think it's a different case: people are supposed to edit these config files and we aren't touching them (including local.bro). The scripts are different: it's all "ours" and edits aren't considered safe and hence, I claim, we can just as well delete them. That argument does extend to some other stuff as well though, like the new magic database. > Delete old scripts before installing new ones > --------------------------------------------- > > Key: BIT-1047 > URL: https://bro-tracker.atlassian.net/browse/BIT-1047 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Reporter: Robin Sommer > Fix For: 2.2 > > > People keep having problems when they install a new Bro version > over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. > We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Wed Jul 31 10:33:06 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 31 Jul 2013 12:33:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1010) BroControl plugin for adding environment variables In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412#comment-13412 ] Robin Sommer commented on BIT-1010: ----------------------------------- Right that's why I was wondering if {{env:VAR1=1}} would work: i.e., the key would be {{env:VAR1}} hoping the parsers splits simply at the {{=}} and accepts the colon as part of the key. But not worth discussing further, please file a merge request once ready. > BroControl plugin for adding environment variables > -------------------------------------------------- > > Key: BIT-1010 > URL: https://bro-tracker.atlassian.net/browse/BIT-1010 > Project: Bro Issue Tracker > Issue Type: Task > Components: BroControl > Affects Versions: git/master > Reporter: Seth Hall > Assignee: Daniel Thayer > Fix For: 2.2 > > Attachments: env_vars.py > > > We should have the ability to add environment variables to Bro at start up time. The option should be available globally in broctl.cfg and per-node in node.cfg. The environments variables should be applied to the process with priority based on how specific the variable is applied (per-node variables defined after global variables so that the per-node variable is used). > As a name suggestion for the configuration option: env_vars (same name in node.cfg and broctl.cfg). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Wed Jul 31 14:09:06 2013 From: jira at bro-tracker.atlassian.net (Jon Siwek (JIRA)) Date: Wed, 31 Jul 2013 16:09:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413#comment-13413 ] Jon Siwek commented on BIT-1047: -------------------------------- Yeah, I'm not too nervous about deleting previously installed scripts (ones that aren't intended to be user-modified), but I'm thinking that doesn't entirely solve things for the user -- they'll probably still get an error, it's just this time it's something like "in local.bro: can't load X: no such file" which will be more obvious than whatever it was before, but they'll still have to know that means to edit their local.bro to fix the problem. So I've not got an argument against cleaning up stale scripts, just saying this is a more general thing that can happen with config files that users edit while meanwhile the upstream version of it changes. More clearly indicating such situations at `make install` might help. I'd prefer if everything were designed so configuration/customization weren't done at all by editing installed files in place, rather through additional files that may be created at predetermined locations (but which may override default settings in one of the installed config files). > Delete old scripts before installing new ones > --------------------------------------------- > > Key: BIT-1047 > URL: https://bro-tracker.atlassian.net/browse/BIT-1047 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Reporter: Robin Sommer > Fix For: 2.2 > > > People keep having problems when they install a new Bro version > over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. > We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From bernhard at ICSI.Berkeley.EDU Wed Jul 31 16:25:31 2013 From: bernhard at ICSI.Berkeley.EDU (Bernhard Amann) Date: Wed, 31 Jul 2013 16:25:31 -0700 Subject: [Bro-Dev] Move 3rdparty into a separate submodule Message-ID: Hi, I just wanted to ask what you think about moving the 3rdparty stuff under src (which at the moment only contains sqlite) into a separate git submodule? That way we keep the 3rdparty sources and their change history out of Bro history. SQLite alone already has more than 140k lines - and I kind of suspect that we might other 3rdparty sources in the future. The disadvantage of this approach is, that the bro git repository does no longer contain all source-files necessary to compile bro. On the other hand - stuff like cmake already is in submodules - so even though the bro git contains all source files at the moment, you still cannot compile it without checking out the submodules. Bernhard From jira at bro-tracker.atlassian.net Wed Jul 31 16:32:06 2013 From: jira at bro-tracker.atlassian.net (Robin Sommer (JIRA)) Date: Wed, 31 Jul 2013 18:32:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414#comment-13414 ] Robin Sommer commented on BIT-1047: ----------------------------------- Agreed, but I don't really see how we could do that for, say, local.bro or broctl.cfg where we want to give users templates to work from. Ideas? > Delete old scripts before installing new ones > --------------------------------------------- > > Key: BIT-1047 > URL: https://bro-tracker.atlassian.net/browse/BIT-1047 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Reporter: Robin Sommer > Fix For: 2.2 > > > People keep having problems when they install a new Bro version > over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. > We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Wed Jul 31 16:36:06 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Wed, 31 Jul 2013 18:36:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1047) Delete old scripts before installing new ones In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415#comment-13415 ] Bernhard Amann commented on BIT-1047: ------------------------------------- I always liked the FreeBSD approach to config-files, where they show you a diff between your version and the current version, if you did change the config file. But that is interactive and probably too much work to be worth it. > Delete old scripts before installing new ones > --------------------------------------------- > > Key: BIT-1047 > URL: https://bro-tracker.atlassian.net/browse/BIT-1047 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Reporter: Robin Sommer > Fix For: 2.2 > > > People keep having problems when they install a new Bro version > over the installation of an old one because scripts that have disappeared in the new version will keep sticking around from the previous installation. > We should simply remove the old scripts/base and scripts/policy before installing anything new. People aren't supposed to edit in there so that should be safe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Wed Jul 31 16:52:06 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Wed, 31 Jul 2013 18:52:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1048) merge topic/bernhard/topk In-Reply-To: References: Message-ID: Bernhard Amann created BIT-1048: ----------------------------------- Summary: merge topic/bernhard/topk Key: BIT-1048 URL: https://bro-tracker.atlassian.net/browse/BIT-1048 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: 2.2 Reporter: Bernhard Amann merge topic/bernhard/topk. The branch should work and is integrated with sumstats. There is a small upcoming interface change - however this will just add new functionality and just change the bifs. For users of sumstats, the integration should not change at all, for others it might take minimal changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira From jira at bro-tracker.atlassian.net Wed Jul 31 16:52:06 2013 From: jira at bro-tracker.atlassian.net (Bernhard Amann (JIRA)) Date: Wed, 31 Jul 2013 18:52:06 -0500 (CDT) Subject: [Bro-Dev] [JIRA] (BIT-1048) merge topic/bernhard/topk In-Reply-To: References: Message-ID: [ https://bro-tracker.atlassian.net/browse/BIT-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1048: -------------------------------- Status: Merge Request (was: Open) > merge topic/bernhard/topk > ------------------------- > > Key: BIT-1048 > URL: https://bro-tracker.atlassian.net/browse/BIT-1048 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro > Affects Versions: 2.2 > Reporter: Bernhard Amann > Labels: sumstats, topk > > merge topic/bernhard/topk. The branch should work and is integrated with sumstats. There is a small upcoming interface change - however this will just add new functionality and just change the bifs. For users of sumstats, the integration should not change at all, for others it might take minimal changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://bro-tracker.atlassian.net/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira