From bro at tracker.bro-ids.org Thu Sep 1 07:45:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 01 Sep 2011 14:45:30 -0000 Subject: [Bro-Dev] #581: Add body length from http_stat to http.log Message-ID: <048.563f43cc45822dc0b2b6386e3aa94732@tracker.bro-ids.org> #581: Add body length from http_stat to http.log ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ Add the body length from the http_stat record to http.log (you get this record in http_message_done). The Content-Length is often incorrect (interrupted connections, broken servers). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 1 08:52:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 01 Sep 2011 15:52:30 -0000 Subject: [Bro-Dev] #581: Add body length from http_stat to http.log In-Reply-To: <048.563f43cc45822dc0b2b6386e3aa94732@tracker.bro-ids.org> References: <048.563f43cc45822dc0b2b6386e3aa94732@tracker.bro-ids.org> Message-ID: <063.f857e5fe65f4a9bca69e39a27cd6138f@tracker.bro-ids.org> #581: Add body length from http_stat to http.log ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by gregor): might also want to add the stat$interrupted flag (Note that both the body_length and the interrupted flag will need to be logged for request and response) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 1 09:39:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 01 Sep 2011 16:39:43 -0000 Subject: [Bro-Dev] #582: HTTP scripts don't handle temporary (1xx) response codes Message-ID: <048.1b597c8ecc43f41458dc3b044bf3b0cb@tracker.bro-ids.org> #582: HTTP scripts don't handle temporary (1xx) response codes ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ The HTTP scripts don't (and never have) handle temporary 1xx response code. If a 1xx response is received, the scripts assume that the request/reply pair is over. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 1 09:56:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 01 Sep 2011 16:56:36 -0000 Subject: [Bro-Dev] #582: HTTP scripts don't handle temporary (1xx) response codes In-Reply-To: <048.1b597c8ecc43f41458dc3b044bf3b0cb@tracker.bro-ids.org> References: <048.1b597c8ecc43f41458dc3b044bf3b0cb@tracker.bro-ids.org> Message-ID: <063.d64b5f8f2bde8df4fe21fbf45c42feb9@tracker.bro-ids.org> #582: HTTP scripts don't handle temporary (1xx) response codes ------------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Resolution: Duplicate | Keywords: ------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Duplicate Comment: This is a duplicate of #411. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 1 09:57:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 01 Sep 2011 16:57:23 -0000 Subject: [Bro-Dev] #411: Non-binpac HTTP analyzer fails on 1xx response codes (was: Non-binpac analyzer generates incorrect weird) In-Reply-To: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> References: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> Message-ID: <061.1351abb4b8b754eb9cb7bfd49f19e8de@tracker.bro-ids.org> #411: Non-binpac HTTP analyzer fails on 1xx response codes ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 1 10:28:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 01 Sep 2011 17:28:48 -0000 Subject: [Bro-Dev] #583: system function with feedback Message-ID: <046.d03023985c47130a6af60bec0359f9f5@tracker.bro-ids.org> #583: system function with feedback -----------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Keywords: language | -----------------------------+-------------------- Jon and I just came up with a proposal for talking to external processes and getting results. Our idea is to center it around running the external process in the background as it's done now then redirecting stdin and stdout to temporary files and to have a file handle where the exit code will be available. Bro could poll using timers for the exit code to indicate the external process has completed then an event would be generated. Here is an example of using this function/event combination: {{{ global called_procs: table[string] of string = {}; # These are the prototypes for the function and event that need to be added # to the core. The names are both terrible. global piped_exec_with_result: function(uid: string, program: string, to_write: string); global system_result: event(uid: string, exit_code: int, stdout: string, stderr: string); event bro_init() { local uid = unique_id(""); called_procs[uid] = "date"; piped_exec_with_result(uid, "date", ""); } event system_result(uid: string, exit_code: int, stdout: string, stderr: string) { print fmt("The command line %s was run and exited with %d.", called_procs[uid], exit_code); print fmt(" On stdout: %s", stdout); print fmt(" On stderr: %s", stderr); } }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From mcholste at gmail.com Thu Sep 1 10:35:15 2011 From: mcholste at gmail.com (Martin Holste) Date: Thu, 1 Sep 2011 12:35:15 -0500 Subject: [Bro-Dev] Error compiling aux/broctl Message-ID: I had to add this to aux/broct/aux/capstats/CMakeLists.txt because I have a custom pcap dir and there is no --with-PCAP=: set(PCAP_ROOT_DIR "/usr/local/pfring") find_path(PCAP_ROOT_DIR NAMES include/pcap.h ) find_path(PCAP_INCLUDE_DIR NAMES pcap.h HINTS ${PCAP_ROOT_DIR}/include ) find_library(PCAP_LIBRARY NAMES pcap HINTS ${PCAP_ROOT_DIR}/lib ) include(FindPackageHandleStandardArgs) find_package_handle_standard_args(PCAP DEFAULT_MSG PCAP_LIBRARY PCAP_INCLUDE_DIR ) mark_as_advanced( PCAP_ROOT_DIR PCAP_INCLUDE_DIR PCAP_LIBRARY ) From jsiwek at ncsa.illinois.edu Thu Sep 1 11:11:46 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Thu, 1 Sep 2011 13:11:46 -0500 Subject: [Bro-Dev] Error compiling aux/broctl In-Reply-To: References: Message-ID: <4B5AD35E-431A-4404-B8E5-6BFEADA3FEB0@ncsa.illinois.edu> > I had to add this to aux/broct/aux/capstats/CMakeLists.txt because I > have a custom pcap dir and there is no --with-PCAP=: Thanks, I added the missing --with-pcap option to the 'broctl' repo's fastpath branch. On a different note, I'm curious if there's a special reason you're building broctl separate from bro? I know in the past versions there's been an odd integration where the install directions said to install bro first and then install broctl over it, but now installing from the main 'bro' repo should automatically also install all the aux/ submodules, including broctl. - Jon From gc355804 at ohio.edu Thu Sep 1 11:24:14 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 01 Sep 2011 14:24:14 -0400 Subject: [Bro-Dev] #583: system function with feedback In-Reply-To: <046.d03023985c47130a6af60bec0359f9f5@tracker.bro-ids.org> References: <046.d03023985c47130a6af60bec0359f9f5@tracker.bro-ids.org> Message-ID: <4E5FCDCE.4050401@ohio.edu> I think I'd be worried about spawning too many long-running background processes and hitting the process fd limit (default is 1024 on Linux, I think?). Counter-proposal: fork(), configure pipes (dup2, I think?), exec(), and then wait for SIGCHLD. Alternatively, since the timer would allow for a poll every so often, could possibly mask SIGCHLD away and just do a waitpid with WNOHANG in the fired timing event to loop over all the bg processes and report on the ones that are done. I can't remember enough about fork and SIGCHLD to know exactly how kosher the second option is, though, so YMMV. --Gilbert On 9/1/2011 1:28 PM, Bro Tracker wrote: > #583: system function with feedback > -----------------------------+-------------------- > Reporter: seth | Owner: > Type: Feature Request | Status: new > Priority: Normal | Milestone: Bro1.7 > Component: Bro | Version: > Keywords: language | > -----------------------------+-------------------- > Jon and I just came up with a proposal for talking to external processes > and getting results. Our idea is to center it around running the external > process in the background as it's done now then redirecting stdin and > stdout to temporary files and to have a file handle where the exit code > will be available. Bro could poll using timers for the exit code to > indicate the external process has completed then an event would be > generated. Here is an example of using this function/event combination: > > {{{ > global called_procs: table[string] of string = {}; > > # These are the prototypes for the function and event that need to be > added > # to the core. The names are both terrible. > global piped_exec_with_result: function(uid: string, program: string, > to_write: string); > global system_result: event(uid: string, exit_code: int, stdout: string, > stderr: string); > > event bro_init() > { > local uid = unique_id(""); > called_procs[uid] = "date"; > piped_exec_with_result(uid, "date", ""); > } > > event system_result(uid: string, exit_code: int, stdout: string, stderr: > string) > { > print fmt("The command line %s was run and exited with %d.", > called_procs[uid], exit_code); > print fmt(" On stdout: %s", stdout); > print fmt(" On stderr: %s", stderr); > } > }}} > From bro at tracker.bro-ids.org Thu Sep 1 12:02:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 01 Sep 2011 19:02:38 -0000 Subject: [Bro-Dev] #564: Bro silently quicks when filter for -f does not compile In-Reply-To: <048.2e1452a19b89aeb611c2b003cd56db4f@tracker.bro-ids.org> References: <048.2e1452a19b89aeb611c2b003cd56db4f@tracker.bro-ids.org> Message-ID: <063.df862337d8c030b8c9c3093eb9c29464@tracker.bro-ids.org> #564: Bro silently quicks when filter for -f does not compile -----------------------------+------------------------ Reporter: gregor | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: BETA -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Merged. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 1 12:10:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 01 Sep 2011 19:10:25 -0000 Subject: [Bro-Dev] #584: DNS TXT record lookup bif Message-ID: <046.2077446f247be33289c55cd0258699d3@tracker.bro-ids.org> #584: DNS TXT record lookup bif -----------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Keywords: | -----------------------------+-------------------- We need a lookup_txt bif like the lookup_name and lookup_host bifs. It would make two things possible: 1. Improved integration with Team Cymru's malware hash registry. 2. Integration with Google's Certificate Catalog to find "bad" certs. -- Ticket URL: Bro Tracker Bro Issue Tracker From mcholste at gmail.com Thu Sep 1 12:12:50 2011 From: mcholste at gmail.com (Martin Holste) Date: Thu, 1 Sep 2011 14:12:50 -0500 Subject: [Bro-Dev] Error compiling aux/broctl In-Reply-To: <4B5AD35E-431A-4404-B8E5-6BFEADA3FEB0@ncsa.illinois.edu> References: <4B5AD35E-431A-4404-B8E5-6BFEADA3FEB0@ncsa.illinois.edu> Message-ID: The server I was building on didn't have access to the git port, so I couldn't do a recursive, and none of the aux stuff is in the bro git snapshot tarball. On Thu, Sep 1, 2011 at 1:11 PM, Jonathan Siwek wrote: >> I had to add this to aux/broct/aux/capstats/CMakeLists.txt because I >> have a custom pcap dir and there is no --with-PCAP=: > > Thanks, I added the missing --with-pcap option to the 'broctl' repo's fastpath branch. > > On a different note, I'm curious if there's a special reason you're building broctl separate from bro? ?I know in the past versions there's been an odd integration where the install directions said to install bro first and then install broctl over it, but now installing from the main 'bro' repo should automatically also install all the aux/ submodules, including broctl. > > - Jon From bro at tracker.bro-ids.org Thu Sep 1 15:08:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 01 Sep 2011 22:08:23 -0000 Subject: [Bro-Dev] #585: LogWriterAscii can write NULL bytes Message-ID: <048.d5296b9ea016f9129205b1fc4b0eccbe@tracker.bro-ids.org> #585: LogWriterAscii can write NULL bytes ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: BETA | ---------------------+------------------------ The LogWriterAscii doesn't properly escape NULL bytes and thus NULL bytes can appear in the log output. In my particular case it happened in the http.log for the URI column. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 1 16:18:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 01 Sep 2011 23:18:48 -0000 Subject: [Bro-Dev] #585: LogWriterAscii can write NULL bytes In-Reply-To: <048.d5296b9ea016f9129205b1fc4b0eccbe@tracker.bro-ids.org> References: <048.d5296b9ea016f9129205b1fc4b0eccbe@tracker.bro-ids.org> Message-ID: <063.b18604496d298851dd805be40bf8b0f8@tracker.bro-ids.org> #585: LogWriterAscii can write NULL bytes ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: BETA ----------------------+------------------------ Comment (by gregor): it appears the problem is more general: the LogWriteAscii doesn't quote special characters. I got a newline as well. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 2 07:17:25 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 02 Sep 2011 14:17:25 -0000 Subject: [Bro-Dev] #586: make install fails in distribution root Message-ID: <051.af745a099f450a673a9da6ce4662ee16@tracker.bro-ids.org> #586: make install fails in distribution root -----------------------+--------------------- Reporter: wurbanski | Type: Problem Status: new | Priority: Normal Milestone: | Component: Bro Version: | Keywords: make -----------------------+--------------------- in the bro beta 'make install' or 'sudo make install' fails in the distribution root, and says that 'build/' does not exist even though it does. If you CD to build and run make install it installs correctly. sensor at trix ~/bro]$ sudo make install Error: No build/ directory found. Did you run configure? *** Error code 1 Stop in /usr/home/sensor/bro. [sensor at trix ~/bro]$ cd build/ [sensor at trix ~/bro/build]$ sudo make install [ 1%] Built target binpac_lib [ 13%] Built target binpac [ 16%] Built target bifcl [ 76%] Built target bro [ 76%] Built target _SubnetTree [ 76%] Built target capstats [ 76%] Built target adtrace [ 76%] Built target bdcat [ 77%] Built target ftwire2bro [ 77%] Built target nfcollector [ 79%] Built target rst [ 87%] Built target broccoli [ 95%] Built target broccoliStatic [ 97%] Built target broconftest [ 97%] Built target broconn [ 97%] Built target broenum [ 97%] Built target brohose [ 98%] Built target broping [ 98%] Built target brotable [100%] Built target _broccoli_intern Install the project... -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 2 07:28:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 02 Sep 2011 14:28:20 -0000 Subject: [Bro-Dev] #586: make install fails in distribution root In-Reply-To: <051.af745a099f450a673a9da6ce4662ee16@tracker.bro-ids.org> References: <051.af745a099f450a673a9da6ce4662ee16@tracker.bro-ids.org> Message-ID: <066.b08be6d0552c91985bf85c5c49dcc1f0@tracker.bro-ids.org> #586: make install fails in distribution root -----------------------------+-------------------- Reporter: wurbanski | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: Resolution: Solved/Applied | Keywords: make -----------------------------+-------------------- Changes (by jsiwek): * status: new => closed * resolution: => Solved/Applied Comment: This is fixed on the fastpath branch [78512569e96bf0d4b3b9b2e66bf6c6cc0dc54ba5/bro] -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 2 09:27:26 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 02 Sep 2011 16:27:26 -0000 Subject: [Bro-Dev] #587: topic/jsiwek/cmake-rpath Message-ID: <048.aee02a39621ffba5ac8cc08b74fbd84e@tracker.bro-ids.org> #587: topic/jsiwek/cmake-rpath ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch is in the `bro` repo and all submodules, recursively, as well as the `broccoli-ruby` repo. By default, CMake builds all binaries with an RPATH such that all dynamic libraries linked against them (either internal/private within the build directory or external/explicitly specified ones) can be found if they're used from that build directory. But the other CMake default is to strip the RPATH on installation, which can cause non-obvious problems for users because the dynamically linked libraries might change after install in some situations. The change in this branch makes it so that a user building/installing from source will generate binaries that link against exactly the libraries they asked for at configure-time. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 2 10:14:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 02 Sep 2011 17:14:27 -0000 Subject: [Bro-Dev] #588: Segfault in rotation code for logging framework Message-ID: <046.8c03ccb146ba869387339739e5ce79a3@tracker.bro-ids.org> #588: Segfault in rotation code for logging framework ---------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: beta | ---------------------+------------------------ This seems to be cropping up everywhere that Bro is being run and rotating logs. With the base rotation time set at "00:00", I've seen this fairly regularly at the "01:00" rotation time. {{{ bro.core Core was generated by `bro'. Program terminated with signal 11, Segmentation fault. #0 LogMgr::FindWriter (this=0x7cb410, writer=0x355fd70) at /usr/home/sensor/bro/src/LogMgr.cc:447 447 if ( winfo->writer == writer ) #0 LogMgr::FindWriter (this=0x7cb410, writer=0x355fd70) at /usr/home/sensor/bro/src/LogMgr.cc:447 #1 0x000000000053da1d in LogMgr::FinishedRotation (this=0x7cb410, writer=) at /usr/home/sensor/bro/src/LogMgr.cc:1508 #2 0x00000000005459e4 in LogWriter::FinishedRotation (this=0x355fd70, new_name=@0x7fffffffe140, old_name=) at /usr/home/sensor/bro/src/LogWriter.cc:157 #3 0x0000000000546fa0 in LogWriterAscii::DoRotate (this=0x355fd70, rotated_path=) at /usr/home/sensor/bro/src/LogWriterAscii.cc:257 #4 0x0000000000545d03 in LogWriter::Rotate (this=0x355fd70, rotated_path=) at /usr/home/sensor/bro/src/LogWriter.cc:95 #5 0x000000000053e3d8 in LogMgr::Rotate (this=) at /usr/home/sensor/bro/src/LogMgr.cc:1499 #6 0x000000000053ec6d in RotationTimer::Dispatch (this=0x865d30, t=) at /usr/home/sensor/bro/src/LogMgr.cc:1409 #7 0x00000000005b820f in PQ_TimerMgr::Expire (this=0x7c29b8) at /usr/home/sensor/bro/src/Timer.cc:142 #8 0x0000000000466955 in done_with_network () at /usr/home/sensor/bro/src/main.cc:231 #9 0x000000000046709b in termination_signal () at /usr/home/sensor/bro/src/main.cc:303 #10 0x000000000055690a in net_run () at /usr/home/sensor/bro/src/Net.cc:511 #11 0x00000000004688ff in main (argc=22323816, argv=) at /usr/home/sensor/bro/src/main.cc:1005 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 2 11:10:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 02 Sep 2011 18:10:14 -0000 Subject: [Bro-Dev] #589: topic/jsiwek/findpcap_pfring Message-ID: <048.c2b93c46d6cd70b08e9e467495b07cac@tracker.bro-ids.org> #589: topic/jsiwek/findpcap_pfring ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ This branch is in the `bro`, `broccoli`, `bro-aux`, and `capstats` repos and makes it easier to link against the libpcap provided by PF_RING (which, by default, is linked statically against libpfring which may not automatically pull in the system's shared thread library). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 2 11:14:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 02 Sep 2011 18:14:12 -0000 Subject: [Bro-Dev] #590: Compile errors on FreeBSD 7.2-RELEASE-p4 Message-ID: <049.d3d434aad51771079b9552f746be9ea4@tracker.bro-ids.org> #590: Compile errors on FreeBSD 7.2-RELEASE-p4 ------------------------+--------------------- Reporter: aashish | Type: Problem Status: new | Priority: Normal Milestone: | Component: Bro Version: git/master | Keywords: ------------------------+--------------------- [ 98%] Building C object aux/broccoli/test/CMakeFiles/broping.dir/broping.c.o /home/users/bro/bro-2.0/aux/broccoli/test/broping.c:49: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'seq' /home/users/bro/bro-2.0/aux/broccoli/test/broping.c:68: error: expected declaration specifiers or '...' before 'uint64' /home/users/bro/bro-2.0/aux/broccoli/test/broping.c: In function 'bro_pong': /home/users/bro/bro-2.0/aux/broccoli/test/broping.c:73: error: 'seq' undeclared (first use in this function) /home/users/bro/bro-2.0/aux/broccoli/test/broping.c:73: error: (Each undeclared identifier is reported only once /home/users/bro/bro-2.0/aux/broccoli/test/broping.c:73: error: for each function it appears in.) /home/users/bro/bro-2.0/aux/broccoli/test/broping.c: In function 'bro_pong_record': /home/users/bro/bro-2.0/aux/broccoli/test/broping.c:85: error: 'uint64' undeclared (first use in this function) /home/users/bro/bro-2.0/aux/broccoli/test/broping.c:85: error: 'seq' undeclared (first use in this function) /home/users/bro/bro-2.0/aux/broccoli/test/broping.c: In function 'bro_pong_compact': /home/users/bro/bro-2.0/aux/broccoli/test/broping.c:123: error: 'uint64' undeclared (first use in this function) /home/users/bro/bro-2.0/aux/broccoli/test/broping.c:123: error: 'seq' undeclared (first use in this function) /home/users/bro/bro-2.0/aux/broccoli/test/broping.c:164: error: expected expression before ')' token /home/users/bro/bro-2.0/aux/broccoli/test/broping.c:166: error: too many arguments to function 'bro_pong' /home/users/bro/bro-2.0/aux/broccoli/test/broping.c: In function 'main': /home/users/bro/bro-2.0/aux/broccoli/test/broping.c:407: error: 'seq' undeclared (first use in this function) *** Error code 1 Stop in /home/users/bro/bro-2.0/build. *** Error code 1 Stop in /home/users/bro/bro-2.0/build. *** Error code 1 Stop in /home/users/bro/bro-2.0/build. *** Error code 1 Stop in /home/users/bro/bro-2.0. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 2 12:09:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 02 Sep 2011 19:09:01 -0000 Subject: [Bro-Dev] #590: Compile errors on FreeBSD 7.2-RELEASE-p4 In-Reply-To: <049.d3d434aad51771079b9552f746be9ea4@tracker.bro-ids.org> References: <049.d3d434aad51771079b9552f746be9ea4@tracker.bro-ids.org> Message-ID: <064.4b91481be50a232e7ae01c99e3c3acbf@tracker.bro-ids.org> #590: Compile errors on FreeBSD 7.2-RELEASE-p4 ----------------------+------------------------ Reporter: aashish | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): In [aa525b97ede84cd9dd35007126a98221c5566f02/broccoli]: {{{ #!CommitTicketReference repository="broccoli" revision="aa525b97ede84cd9dd35007126a98221c5566f02" Compiling test programs should first search for broccoli.h in build directory. Addresses #590 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 2 12:17:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 02 Sep 2011 19:17:07 -0000 Subject: [Bro-Dev] #590: Compile errors on FreeBSD 7.2-RELEASE-p4 In-Reply-To: <049.d3d434aad51771079b9552f746be9ea4@tracker.bro-ids.org> References: <049.d3d434aad51771079b9552f746be9ea4@tracker.bro-ids.org> Message-ID: <064.603969d2d37a40f17e588f2320688c4d@tracker.bro-ids.org> #590: Compile errors on FreeBSD 7.2-RELEASE-p4 ----------------------+------------------------ Reporter: aashish | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): I didn't reproduce this exactly, but I saw a possible problem if there exists an old broccoli.h somewhere in a standard place the compiler searches. I made an attempt to fix it on the broccoli repo's fastpath branch which you can checkout like: {{{ cd aux/broccoli git checkout fastpath git pull }}} If you try building with that, let me know if you still get a build error. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 2 12:49:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 02 Sep 2011 19:49:56 -0000 Subject: [Bro-Dev] #590: Compile errors on FreeBSD 7.2-RELEASE-p4 In-Reply-To: <049.d3d434aad51771079b9552f746be9ea4@tracker.bro-ids.org> References: <049.d3d434aad51771079b9552f746be9ea4@tracker.bro-ids.org> Message-ID: <064.2dc4062155d32106c9c98b4677539aeb@tracker.bro-ids.org> #590: Compile errors on FreeBSD 7.2-RELEASE-p4 ----------------------+------------------------ Reporter: aashish | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by aashish): Issue has been fixed. Thanks a lot Jon. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 2 12:53:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 02 Sep 2011 19:53:22 -0000 Subject: [Bro-Dev] #590: Compile errors on FreeBSD 7.2-RELEASE-p4 In-Reply-To: <049.d3d434aad51771079b9552f746be9ea4@tracker.bro-ids.org> References: <049.d3d434aad51771079b9552f746be9ea4@tracker.bro-ids.org> Message-ID: <064.316f87200a889646fc915d2b9aca4092@tracker.bro-ids.org> #590: Compile errors on FreeBSD 7.2-RELEASE-p4 -----------------------------+------------------------ Reporter: aashish | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by jsiwek): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 2 22:08:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 03 Sep 2011 05:08:21 -0000 Subject: [Bro-Dev] #591: Time to finish collecting stats Message-ID: <046.79f8f007d391e0035e2e0757bc57d907@tracker.bro-ids.org> #591: Time to finish collecting stats ------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Keywords: beta | ------------------------+-------------------- I've run into several cases now where the stats part of broctl's cron command either takes too long to process or gets wedged somehow. We should probably add a timer to the stats collection so that if it takes longer than some configurable default (1 minute maybe?) everything is killed off and the stats collection is given up on. What I ran into just now is that capstats may not be compatible with pf_ring's libpcap wrapper. The process just sits and never finishes. This results in the cron command never being able to do it's job and makes it nearly impossible to use broctl at all since the never-dying cron command is holding onto a lock forever. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Sun Sep 4 12:29:45 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 4 Sep 2011 12:29:45 -0700 Subject: [Bro-Dev] Impact of timers/scheduled events? In-Reply-To: References: Message-ID: <20110904192944.GR89721@icir.org> On Tue, Aug 30, 2011 at 10:19 -0400, you wrote: > I'm not exactly sure of all the impacts of scheduling events. Are > there any known issues with scheduling very large numbers of them? Generally, timers are quite efficient, note that there's already at least one per connection internally. Never tried to create huge numbers on the scripting layer but I would think it should work reasonably well. I think more important than the sheer number is the churn rate: if you reinstall zillions of timers every few seconds, it gets expensive. Also, Bro limits the numbers of timers it expires at once, so there could be a backlog accumulating if too many fire simultaniously. Standard disclaimer: I don't dare to make performance predictions about Bro anymore without actually trying it. :) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Sun Sep 4 10:56:09 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 4 Sep 2011 10:56:09 -0700 Subject: [Bro-Dev] #580: pf_ring support In-Reply-To: <4E5EAA6B.70801@ohio.edu> References: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> <4E5EAA6B.70801@ohio.edu> Message-ID: <20110904175609.GK89721@icir.org> On Wed, Aug 31, 2011 at 14:40 -0700, you wrote: > The PF_RING user land library seems to be LGPL; the code itself seems to > disagree with itself as to which version applies (pfring.h says 2 or > later, pfring.c says 3 or later). COPYING is LGPL v3, so I'd assume > that was the intended license. (Please make sure to add such things to the tracker ticket, that's very valuable information. If your email address matches your tracker account, you should be able to email it there directly). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Sun Sep 4 19:22:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 05 Sep 2011 02:22:29 -0000 Subject: [Bro-Dev] #510: Upgrade How-To In-Reply-To: <047.e793bc8325e1f7ba35cd0d2c4f23760e@tracker.bro-ids.org> References: <047.e793bc8325e1f7ba35cd0d2c4f23760e@tracker.bro-ids.org> Message-ID: <062.f93b20dff84348a1b74c0981408e263a@tracker.bro-ids.org> #510: Upgrade How-To ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: preview ----------------------+------------------------ Comment (by robin): We should briefly mention the new snaplen default and its potential impact on performance. Also, adding a note about that to FAQ would be good. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 4 19:22:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 05 Sep 2011 02:22:32 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.c8d6dac8c000b7ebbf262cb7fbfe0cfb@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): On Fri, Aug 19, 2011 at 19:21 -0000, you wrote: > Fixing the Broccoli API Reference stuff still seems like a task for the > release after next since that depends on extending doc extraction to C/C++ > sources. Yeah, but it would be good if we integrated the two (rst manual and current API docs) in some form at least. I don't know how the API docs are generated. Can we remove the old SGML docs but keep the current API doc generation for now? Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 4 19:22:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 05 Sep 2011 02:22:31 -0000 Subject: [Bro-Dev] #585: LogWriterAscii can write NULL bytes In-Reply-To: <048.d5296b9ea016f9129205b1fc4b0eccbe@tracker.bro-ids.org> References: <048.d5296b9ea016f9129205b1fc4b0eccbe@tracker.bro-ids.org> Message-ID: <063.0e3f9f43f6903a95dec44fe1d29f448c@tracker.bro-ids.org> #585: LogWriterAscii can write NULL bytes ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: BETA ----------------------+------------------------ Comment (by robin): On Thu, Sep 01, 2011 at 23:18 -0000, you wrote: > it appears the problem is more general: the LogWriteAscii doesn't quote > special characters. I got a newline as well. That surprises me, we even have test cases checking that it does. Do you have an example that doesn't work? Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 4 19:22:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 05 Sep 2011 02:22:29 -0000 Subject: [Bro-Dev] #578: Add ICMPv6 support to Bro In-Reply-To: <048.1e6e618d9d8a166299c6c8582e9c8511@tracker.bro-ids.org> References: <048.1e6e618d9d8a166299c6c8582e9c8511@tracker.bro-ids.org> Message-ID: <063.7dc682b12f079310362a3f84ec0cfd6a@tracker.bro-ids.org> #578: Add ICMPv6 support to Bro ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Resolution: | Keywords: IPv6 ----------------------+------------------------ Comment (by robin): On Wed, Aug 31, 2011 at 17:03 -0000, you wrote: > What is the status of Matti's branch that he was working on? It's pretty much ready iirc. I cleaned it up a bit and was waiting for Matti's feedback. Will ping him on that, but in any case it's something to include into the upcoming larger batch of IPv6 work. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 4 19:22:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 05 Sep 2011 02:22:32 -0000 Subject: [Bro-Dev] #580: pf_ring support In-Reply-To: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> References: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> Message-ID: <061.d163532b22796e97d22c2fd793058008@tracker.bro-ids.org> #580: pf_ring support ------------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by robin): On Wed, Aug 31, 2011 at 21:02 -0000, you wrote: > Like the syslog writer, I'm not sure of a timeline on this either. I'm > setting it to the release after the next, but if we could squeeze it into > the next release I'm sure that everyone would benefit. I would prefer to not add any further extensions to the core at this point so that we get the release out. But it sounds good for the one after that. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 4 19:22:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 05 Sep 2011 02:22:34 -0000 Subject: [Bro-Dev] #572: topic/jsiwek/filter-rotation In-Reply-To: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> References: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> Message-ID: <063.143cd9e582a9f23b74113f98b9838e45@tracker.bro-ids.org> #572: topic/jsiwek/filter-rotation ----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): On Fri, Aug 26, 2011 at 19:48 -0000, you wrote: > This branch adds an optional `Log::RotationControl` to `Log::Filter` > records so that defining a filter's rotation behavior can be done at the > time of the filter creation/addition instead of separately redef'ing > `Log::rotation_control`. Hmm... Would it be better to get rid of the rotation_control table and control rotation just via the Filter record? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 5 04:11:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 05 Sep 2011 11:11:05 -0000 Subject: [Bro-Dev] #580: pf_ring support In-Reply-To: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> References: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> Message-ID: <061.df0691095d991dbdac496778612383ef@tracker.bro-ids.org> #580: pf_ring support ------------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by seth): > I would prefer to not add any further extensions to the core at this > point so that we get the release out. But it sounds good for the one > after that. There is actually no core support for pf_ring. All we needed was some slight modifications to the cmake scripts and a BiF to set environment variables. The actual support to "turn on" pf_ring load balancing is done at the scripting layer. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 5 08:26:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 05 Sep 2011 15:26:13 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.3e88b53a232d195510b15826bbd737be@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header ---------------------+---------------------- Reporter: gclark | Owner: robin Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: logging ---------------------+---------------------- Comment (by robin): In [630c256a72ef1b997d7d440be122803e3be5e029/bro]: {{{ #!CommitTicketReference repository="bro" revision="630c256a72ef1b997d7d440be122803e3be5e029" Merge remote branch 'origin/topic/gilbert/ascii-header' * origin/topic/gilbert/ascii-header: Updated tests; removed net type from type conversion code. Updated header format (see #558) Header modification to LogWriterAscii to make it easier for scripts to understand bro log files. Notes: - I've refactored the code a bit, also adapting the style a bit. Also edited the header format slightly. - I'm skipping the testing/btest/profiles directory, which seems unrelated. - I'm also skipping the baseline updates as they weren't up-to-date anymore. Will update them in a subsequent commit. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 5 11:36:41 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 05 Sep 2011 18:36:41 -0000 Subject: [Bro-Dev] #592: topic/gilbert/profiles Message-ID: <048.50f5253509e35787f64b7ef3adde24a0@tracker.bro-ids.org> #592: topic/gilbert/profiles ---------------------------+----------------- Reporter: gclark | Owner: Type: Merge Request | Status: new Priority: Low | Milestone: Component: bro-aux | Version: Keywords: | ---------------------------+----------------- topic/gilbert/profiles in the 'btest' project should be ready to merge. Doesn't seem to be a 'btest' component option, so chose bro-aux instead. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 5 12:09:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 05 Sep 2011 19:09:32 -0000 Subject: [Bro-Dev] #593: topic/gilbert/rand-pool Message-ID: <048.552e03f58ca192dae7ba2307bac44e74@tracker.bro-ids.org> #593: topic/gilbert/rand-pool ---------------------------+------------------- Reporter: gclark | Owner: robin Type: Merge Request | Status: new Priority: Low | Milestone: Component: Bro | Version: Keywords: | ---------------------------+------------------- This branch separates the bro core UID generator from that of the scripts. It additionally offers a mechanism to request different generators for different scripts (so that the UIDs generated by one script will not affect the UIDs generated by another). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 5 21:22:59 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 06 Sep 2011 04:22:59 -0000 Subject: [Bro-Dev] #580: pf_ring support In-Reply-To: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> References: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> Message-ID: <061.03ceddb4a399c5edfe347520534a1067@tracker.bro-ids.org> #580: pf_ring support ------------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: ------------------------------+-------------------- Comment (by robin): On Mon, Sep 05, 2011 at 11:11 -0000, you wrote: > There is actually no core support for pf_ring. Ah, then I misunderstood. It sounded to me that there were different ways of doing this, with one requiring core support. Great. :) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 05:50:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 06 Sep 2011 12:50:54 -0000 Subject: [Bro-Dev] #594: Control via git not reliable Message-ID: <047.30f313cbc7651ae0c84ce513ff7a48c6@tracker.bro-ids.org> #594: Control via git not reliable ---------------------------+-------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Component: TicketTracker | ---------------------------+-------------------- Adding "Closes #xxx" to commit messages works sometimes, but not always. See ccad24b68584aabecaf0af69f2202914221be9e4 for a commit where it didn't. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 05:51:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 06 Sep 2011 12:51:31 -0000 Subject: [Bro-Dev] #595: Batch edit plugin missing Message-ID: <047.84fa27bb78663227cbc2cbb0778ce3bf@tracker.bro-ids.org> #595: Batch edit plugin missing ---------------------------+-------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Component: TicketTracker | ---------------------------+-------------------- The batch edit plugin for Trac has disappaered. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 05:51:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 06 Sep 2011 12:51:40 -0000 Subject: [Bro-Dev] #447: Bro snaplen In-Reply-To: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> References: <046.4db53ae499264619cad7da2294ca2be3@tracker.bro-ids.org> Message-ID: <061.7b5fa12f9ccaebb370b6408b0220f2c1@tracker.bro-ids.org> #447: Bro snaplen -----------------------------+-------------------- Reporter: vern | Owner: Type: Merge Request | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 05:52:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 06 Sep 2011 12:52:09 -0000 Subject: [Bro-Dev] #570: topic/jsiwek/ascii-log-rotate-fix In-Reply-To: <048.7a1abf54719746c4b4bb2aaf41579a96@tracker.bro-ids.org> References: <048.7a1abf54719746c4b4bb2aaf41579a96@tracker.bro-ids.org> Message-ID: <063.7f9f183031d55280cfb737e09e022206@tracker.bro-ids.org> #570: topic/jsiwek/ascii-log-rotate-fix -----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 05:52:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 06 Sep 2011 12:52:23 -0000 Subject: [Bro-Dev] #571: topic/jsiwek/nested-record-coerce-fix In-Reply-To: <048.5755952d6bb1d678d97b2365edf175e2@tracker.bro-ids.org> References: <048.5755952d6bb1d678d97b2365edf175e2@tracker.bro-ids.org> Message-ID: <063.ba1de43b7f8c32957096372a45432495@tracker.bro-ids.org> #571: topic/jsiwek/nested-record-coerce-fix -----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 05:55:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 06 Sep 2011 12:55:16 -0000 Subject: [Bro-Dev] #587: topic/jsiwek/cmake-rpath In-Reply-To: <048.aee02a39621ffba5ac8cc08b74fbd84e@tracker.bro-ids.org> References: <048.aee02a39621ffba5ac8cc08b74fbd84e@tracker.bro-ids.org> Message-ID: <063.d4e6efacf0dc17e6480ee78db7d7918a@tracker.bro-ids.org> #587: topic/jsiwek/cmake-rpath ----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): Applied to all, except the broccoli-ruby repository. What's the state for that anyway? Should it move into `aux/broccoli/bindings`? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 05:55:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 06 Sep 2011 12:55:37 -0000 Subject: [Bro-Dev] #589: topic/jsiwek/findpcap_pfring In-Reply-To: <048.c2b93c46d6cd70b08e9e467495b07cac@tracker.bro-ids.org> References: <048.c2b93c46d6cd70b08e9e467495b07cac@tracker.bro-ids.org> Message-ID: <063.3eee1d72ebe734520a4f7e2aabe5af32@tracker.bro-ids.org> #589: topic/jsiwek/findpcap_pfring -----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Tue Sep 6 07:48:30 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 6 Sep 2011 09:48:30 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] fastpath, master: Install binaries with an RPATH. (fbe6356) In-Reply-To: <201109051526.p85FQREM007609@bro-ids.icir.org> References: <201109051526.p85FQREM007609@bro-ids.icir.org> Message-ID: <884043D2-D8BD-40E0-9561-61ECE71136DD@ncsa.illinois.edu> > Note, I didn't see the topic/jsiwek/cmake-rpath branch in the Bro > repository so I copied the change from the submodules, assuming it > applies in the same way. Yep, thanks for catching that. - Jon From bro at tracker.bro-ids.org Tue Sep 6 08:59:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 06 Sep 2011 15:59:01 -0000 Subject: [Bro-Dev] #587: topic/jsiwek/cmake-rpath In-Reply-To: <048.aee02a39621ffba5ac8cc08b74fbd84e@tracker.bro-ids.org> References: <048.aee02a39621ffba5ac8cc08b74fbd84e@tracker.bro-ids.org> Message-ID: <063.d1e42fe9895f5540d6659f9bfafd6ccd@tracker.bro-ids.org> #587: topic/jsiwek/cmake-rpath ----------------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * owner: => seth * status: new => assigned Comment: Replying to [comment:1 robin]: > Applied to all, except the broccoli-ruby repository. What's the state for that anyway? Should it move into `aux/broccoli/bindings`? Last I checked, that repo still built fine, but I didn't try using it at all. Were these bindings started by you, Seth? If so, can you merge this branch into the repository, close this ticket, and then if you think it's ready to be included as a submodule, start a new ticket for me to integrate it? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 10:07:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 06 Sep 2011 17:07:45 -0000 Subject: [Bro-Dev] #587: topic/jsiwek/cmake-rpath In-Reply-To: <048.aee02a39621ffba5ac8cc08b74fbd84e@tracker.bro-ids.org> References: <048.aee02a39621ffba5ac8cc08b74fbd84e@tracker.bro-ids.org> Message-ID: <063.5901b4e56ba58af98c724d64f42684df@tracker.bro-ids.org> #587: topic/jsiwek/cmake-rpath ----------------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by seth): > Applied to all, except the broccoli-ruby repository. What's the state for > that anyway? Should it move into `aux/broccoli/bindings`? Let's wait a bit before we move it. It's not quite in a shape to ship it with Bro yet. I've been meaning to update it to the Ruby 1.9 API for a long time but I haven't yet. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 11:53:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 06 Sep 2011 18:53:29 -0000 Subject: [Bro-Dev] #587: topic/jsiwek/cmake-rpath In-Reply-To: <048.aee02a39621ffba5ac8cc08b74fbd84e@tracker.bro-ids.org> References: <048.aee02a39621ffba5ac8cc08b74fbd84e@tracker.bro-ids.org> Message-ID: <063.8e080163fddb26c874e5d22a9fff7bac@tracker.bro-ids.org> #587: topic/jsiwek/cmake-rpath ----------------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by seth): > Last I checked, that repo still built fine, but I didn't try using it at > all. Were these bindings started by you, Seth? If so, can you merge this > branch into the repository, close this ticket, and then if you think it's > ready to be included as a submodule, start a new ticket for me to > integrate it? Ok, I'll check it out with ruby 1.8.x soon. We can document that the bindings don't currently work with 1.9.x. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 12:42:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 06 Sep 2011 19:42:43 -0000 Subject: [Bro-Dev] #580: pf_ring support In-Reply-To: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> References: <046.86c1866acdc92a7dc2ecdcca2b1a14ad@tracker.bro-ids.org> Message-ID: <061.d05601d05006d0016d5ca41d679dbfe2@tracker.bro-ids.org> #580: pf_ring support ------------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: closed Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Resolution: Solved/Applied | Keywords: ------------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: The final support piece for this was just merged. If Bro is linked with the pf_ring libpcap wrapper and the misc/pf-ring-load-balancing script is loaded, multiple instances of Bro sniffing the same interface will result in the traffic being flow based load balanced across the Bro processes. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 21:14:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 04:14:24 -0000 Subject: [Bro-Dev] #557: Remove malloc.c In-Reply-To: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> References: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> Message-ID: <062.aab8f67bb8d7f6c60e10fc670c9fd910@tracker.bro-ids.org> #557: Remove malloc.c ----------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by robin): So results are a bit inconclusive because there are so many factors going in. But still, let's go ahead and remove the custom malloc, I don't think there's a good reason to keep it. What we could do is add a configure option to link with tcmalloc if available. But I'm fine leaving that to later. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 21:18:59 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 04:18:59 -0000 Subject: [Bro-Dev] #591: Time to finish collecting stats In-Reply-To: <046.79f8f007d391e0035e2e0757bc57d907@tracker.bro-ids.org> References: <046.79f8f007d391e0035e2e0757bc57d907@tracker.bro-ids.org> Message-ID: <061.4f8684557505afb12d783e377078ac05@tracker.bro-ids.org> #591: Time to finish collecting stats -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta -------------------------+-------------------- Comment (by robin): This isn't that easy to do as we'd need to run the stats code in its own process or thread. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 6 21:20:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 04:20:32 -0000 Subject: [Bro-Dev] #558: /topic/gilbert/ascii-header In-Reply-To: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> References: <048.edf92bb6c4c9f2d2ba6b1c56b76ebabb@tracker.bro-ids.org> Message-ID: <063.07990b47e41602a36c89c196c9fc2dd3@tracker.bro-ids.org> #558: /topic/gilbert/ascii-header -----------------------------+--------------------- Reporter: gclark | Owner: robin Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: logging -----------------------------+--------------------- Changes (by robin): * status: assigned => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 7 07:30:18 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 14:30:18 -0000 Subject: [Bro-Dev] #596: broctl "restart" commands is reversed Message-ID: <046.ff424b35e40b3078b7bbffd077d23722@tracker.bro-ids.org> #596: broctl "restart" commands is reversed ------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Keywords: beta | ------------------------+-------------------- When restarting a cluster, the workers should be shutdown first, but the manager is currently being shutdown first. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 7 08:57:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 15:57:32 -0000 Subject: [Bro-Dev] #597: net_init and bro_init Message-ID: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> #597: net_init and bro_init ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: | ---------------------+-------------------- In main.cc, the net_init function is called before the bro_init handlers are called. Could we either move the bro_init handlers call up or the net_init function call down? This would give us the ability to set environment variables in bro_init handlers before net_init is called and generally can make things more flexible at the scripting layer. I'm currently testing on live traffic with moving the net_init function call down to after the bro_init handlers are done. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 7 09:44:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 16:44:13 -0000 Subject: [Bro-Dev] #598: topic/jsiwek/misc-doc-fixes Message-ID: <048.9c484d4c829b091b2b97e02b8847b5da@tracker.bro-ids.org> #598: topic/jsiwek/misc-doc-fixes ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ See [80e154ba3c63bf39c16ce3d66bc8c73f8e515a4b/bro] and [95ed1920885e3b3975b2e9eb248182a1fc9709d7/bro] for changes. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 7 10:01:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 17:01:06 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.0cef03c1e2910f29cd59f1c3ca427355@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): In [9d0cef8e10720993a25e23972c07712ca021f52a/broccoli]: {{{ #!CommitTicketReference repository="broccoli" revision="9d0cef8e10720993a25e23972c07712ca021f52a" Broccoli API docs now generated via Doxygen. Overhauled the Broccoli API doc generation process to use Doxygen instead of GTK-Doc. Some comment-style changes were needed. The ``make doc`` target will generate the API docs. Addresses #563. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From mcholste at gmail.com Wed Sep 7 10:03:07 2011 From: mcholste at gmail.com (Martin Holste) Date: Wed, 7 Sep 2011 12:03:07 -0500 Subject: [Bro-Dev] Memory leaks on git eds2245 Message-ID: I'm finding that Bro leaks memory heavily whenever it drops packets, such that if overwhelmed, it will consume all memory on a beefy box in minutes. Even under a slight load, there appears to be memory leakage. Here's the tail end of a valgrind: ==4477== 479,944 (128 direct, 479,816 indirect) bytes in 1 blocks are definitely lost in loss record 5,304 of 5,306 ==4477== at 0x4C274A8: malloc (vg_replace_malloc.c:236) ==4477== by 0x554FF21: CRYPTO_malloc (in /lib/libcrypto.so.0.9.8) ==4477== by 0x55E3D96: X509_STORE_new (in /lib/libcrypto.so.0.9.8) ==4477== by 0x5AAED4: BifFunc::bro_x509_verify(Frame*, ValPList*) (bro.bif:3449) ==4477== by 0x59B88E: BuiltinFunc::Call(ValPList*, Frame*) const (Func.cc:463) ==4477== by 0x5888BD: CallExpr::Eval(Frame*) const (Expr.cc:4649) ==4477== by 0x578169: AssignExpr::Eval(Frame*) const (Expr.cc:2598) ==4477== by 0x63AFAF: ExprStmt::Exec(Frame*, stmt_flow_type&) const (Stmt.cc:369) ==4477== by 0x633E00: StmtList::Exec(Frame*, stmt_flow_type&) const (Stmt.cc:1404) ==4477== by 0x59C9A0: BroFunc::Call(ValPList*, Frame*) const (Func.cc:320) ==4477== by 0x553A45: EventHandler::Call(ValPList*, bool) (EventHandler.cc:73) ==4477== by 0x5531B4: EventMgr::Dispatch() (Event.h:46) ==4477== ==4477== 974,513 (5,760 direct, 968,753 indirect) bytes in 72 blocks are definitely lost in loss record 5,306 of 5,306 ==4477== at 0x4C27CC1: operator new(unsigned long) (vg_replace_malloc.c:261) ==4477== by 0x4FD9AD: yyparse() (parse.y:610) ==4477== by 0x50D437: main (main.cc:745) ==4477== ==4477== LEAK SUMMARY: ==4477== definitely lost: 285,262 bytes in 14,318 blocks ==4477== indirectly lost: 2,282,276 bytes in 39,468 blocks ==4477== possibly lost: 139,163 bytes in 2,955 blocks ==4477== still reachable: 9,741,975 bytes in 133,386 blocks ==4477== suppressed: 0 bytes in 0 blocks ==4477== Reachable blocks (those to which a pointer was found) are not shown. ==4477== To see them, rerun with: --leak-check=full --show-reachable=yes ==4477== ==4477== For counts of detected and suppressed errors, rerun with: -v ==4477== Use --track-origins=yes to see where uninitialised values come from ==4477== ERROR SUMMARY: 222225 errors from 373 contexts (suppressed: 4 from 4) From mcholste at gmail.com Wed Sep 7 10:06:13 2011 From: mcholste at gmail.com (Martin Holste) Date: Wed, 7 Sep 2011 12:06:13 -0500 Subject: [Bro-Dev] validate-certs.bro error In-Reply-To: References: Message-ID: Looks like this fix didn't make it into the current trunk: 1315414623.768835 internal error in /usr/local/bro-eds2245/share/bro/policy/protocols/ssl/validate-certs.bro, line 20: field value missing (SSL::c$ssl) Line 20: if ( !c$ssl?$cert || !c$ssl?$cert_chain ) On Wed, Aug 31, 2011 at 10:54 AM, Seth Hall wrote: > > On Aug 31, 2011, at 11:40 AM, Martin Holste wrote: > >> Fatal from git rev 005b150: >> 1314803689.614709 internal error in >> /usr/local/bro-005b150/share/bro/policy/protocols/ssl/validate-certs.bro, >> line 20: field value missing (SSL::c$ssl) > > > Sorry about that, I've had it fixed locally for a few days but I'm trying to finish up some automatic notice deduplication code before I commit it because I'm modifying some other SSL scripts to use the notice dedup-ing. > > ?.Seth > > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > From bro at tracker.bro-ids.org Wed Sep 7 10:08:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 17:08:37 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.c2f3f4aadd0a849742c9dad38b92e50f@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): > Yeah, but it would be good if we integrated the two (rst manual and > current API docs) in some form at least. I don't know how the API docs > are generated. Can we remove the old SGML docs but keep the current > API doc generation for now? The old API doc generation was never integrated with CMake and seems harder to repair than to just start from scratch and use Doxygen instead, so I've done that. So now there's two components under that `doc/` directory: 1) the reST manual and 2) the framework to generate HTML API docs via Doxygen The best way of integrating those into the `www` is still an open question (I'm trying to figure out a good scheme to collect/group documentation according to Bro version), but I think you can consider merging this branch as that gets sorted out separately. -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Wed Sep 7 10:52:48 2011 From: seth at icir.org (Seth Hall) Date: Wed, 7 Sep 2011 13:52:48 -0400 Subject: [Bro-Dev] validate-certs.bro error In-Reply-To: References: Message-ID: <66CFEADA-6F12-42E0-9287-29DA5461B7D5@icir.org> On Sep 7, 2011, at 1:06 PM, Martin Holste wrote: > Looks like this fix didn't make it into the current trunk: > > 1315414623.768835 internal error in > /usr/local/bro-eds2245/share/bro/policy/protocols/ssl/validate-certs.bro, > line 20: field value missing (SSL::c$ssl) I implemented it as part of a larger change that hasn't been committed yet. I'll commit it separately right now. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Wed Sep 7 12:07:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 19:07:00 -0000 Subject: [Bro-Dev] #599: Capstats and pf_ring Message-ID: <046.c573030a1b3b313b9ea9fd908254c7b1@tracker.bro-ids.org> #599: Capstats and pf_ring ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Capstats | Version: Keywords: beta | ----------------------+-------------------- Capstats never returns when linked again pf_ring's libpcap. This causes severe problems for broctl's cron command. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 7 16:11:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 23:11:14 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.5ab90524d462a83c578a652cd49ef543@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): On Wed, Sep 07, 2011 at 17:08 -0000, you wrote: > The old API doc generation was never integrated with CMake and seems > harder to repair than to just start from scratch and use Doxygen instead, > so I've done that. Makes sense, thanks! > The best way of integrating those into the `www` is still an open question > (I'm trying to figure out a good scheme to collect/group documentation > according to Bro version) I have put a bit of machinery in place for that already, via checking out specific tags. But I'm not really happy with it yet. > but I think you can consider merging this branch as that gets sorted > out separately. Will do. Are there links from the rest docs to the Doxygen docs that we will need to fix later? Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 7 16:19:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 23:19:49 -0000 Subject: [Bro-Dev] #600: Coercion problem with log funcs Message-ID: <047.51c0e18a7925a83f11c8e3221c3807d2@tracker.bro-ids.org> #600: Coercion problem with log funcs ------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Version: git/master | Keywords: beta ------------------------+--------------------- The following code: {{{ function split_log(id: Log::ID, path: string, rec: record { id: conn_id; } ) : string ????????{ ????????return Site::is_local_addr(rec$id$orig_h) ? "local" : "remote"; ????????} event bro_init() ????????{ ????????# Add a new filter to the Conn::LOG stream that logs only ????????# timestamp and originator address. ????????local filter: Log::Filter = [$name="dst-only", $path_func=split_log, $include=set("ts", "id.orig_h")]; ????????Log::add_filter(Conn::LOG, filter); ????????} }}} produces {{{ 1144876588.568110 fatal error in : Val::CONVERTER (time/record) (1144876583.36149) }}} That looks like the record coercsion doesn't work right. -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 7 16:21:41 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 23:21:41 -0000 Subject: [Bro-Dev] #597: net_init and bro_init In-Reply-To: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> References: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> Message-ID: <061.62dfc85ca737dfde8c031744ee096667@tracker.bro-ids.org> #597: net_init and bro_init ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by robin): On Wed, Sep 07, 2011 at 15:57 -0000, you wrote: > I'm currently testing on live traffic with moving the net_init function > call down to after the bro_init handlers are done. I can't say immediately whether there are things that depend on order, but if your testing is going fine, moving should indeed not be a problem. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 7 16:28:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 23:28:56 -0000 Subject: [Bro-Dev] #597: net_init and bro_init In-Reply-To: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> References: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> Message-ID: <061.9d5c924a4890478a6fcae343fe1d228a@tracker.bro-ids.org> #597: net_init and bro_init ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by gregor): > > I'm currently testing on live traffic with moving the net_init function > > call down to after the bro_init handlers are done. > > I can't say immediately whether there are things that depend on order, > but if your testing is going fine, moving should indeed not be a > problem. > One potential problem might be writing pcap files, since you need to have the pcap handle from your capture to know the linktype for the savefile. However, I don't know whether that's done in bro_init() or not. cu gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 7 16:29:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 23:29:20 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.6fe7e55bc5646f7fb6dfdf58b510193c@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): > > The best way of integrating those into the `www` is still an open question > > (I'm trying to figure out a good scheme to collect/group documentation > > according to Bro version) > > I have put a bit of machinery in place for that already, via checking > out specific tags. But I'm not really happy with it yet. Yeah, the current problem I was running into is that grabbing something from a given tag of a given repo isn't enough -- it seems like we'd want to create self-contained bundles of documentation per Bro release, which means we'd need to be able to grab submodule documentation according to the commit that was used for a given Bro tag. Gitweb doesn't understand what a submodule is so you can't get the submodule commit hash that way. I think it needs to change to just using a local clone of the Bro repository to pull files into the `www` repo for later rendering. > > but I think you can consider merging this branch as that gets sorted > > out separately. > > Will do. Are there links from the rest docs to the Doxygen docs that > we will need to fix later? There's not any links between the two at the moment, we'll have to add them. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 7 16:45:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 07 Sep 2011 23:45:42 -0000 Subject: [Bro-Dev] #597: net_init and bro_init In-Reply-To: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> References: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> Message-ID: <061.8908472cf6a672575796b2576c8bb221@tracker.bro-ids.org> #597: net_init and bro_init ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by robin): On Wed, Sep 07, 2011 at 23:28 -0000, you wrote: > However, I don't know whether that's done in bro_init() or not. No, that's done internally. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Wed Sep 7 17:10:04 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 7 Sep 2011 17:10:04 -0700 Subject: [Bro-Dev] Memory leaks on git eds2245 In-Reply-To: References: Message-ID: <20110908001004.GC48266@icir.org> On Wed, Sep 07, 2011 at 12:03 -0500, you wrote: > I'm finding that Bro leaks memory heavily whenever it drops packets, Seth/Gregor, could this still be SSL-related? Martin, any chance you could try without SSL to see if it shows the same behaviour? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gregor at icir.org Wed Sep 7 22:25:37 2011 From: gregor at icir.org (Gregor Maier) Date: Wed, 07 Sep 2011 22:25:37 -0700 Subject: [Bro-Dev] Memory leaks on git eds2245 In-Reply-To: <20110908001004.GC48266@icir.org> References: <20110908001004.GC48266@icir.org> Message-ID: <4E6851D1.6000307@icir.org> On 9/7/11 17:10 , Robin Sommer wrote: > > On Wed, Sep 07, 2011 at 12:03 -0500, you wrote: > >> I'm finding that Bro leaks memory heavily whenever it drops packets, > > Seth/Gregor, could this still be SSL-related? Martin, any chance you > could try without SSL to see if it shows the same behaviour? I would guess so. The valgrind output definitely hints at that. I just don't know SSL or its analyzer at all, so I'm afraid I might not be much help :-( cu Gregor (BTW, the memory problems I have/had weren't "real" leaks. One a SSL connection was done Bro would free the memory for it again. The problem is that many SSL connections can live for days and thus they ultimately consume memory like a "real" leak would). -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From seth at icir.org Wed Sep 7 22:37:56 2011 From: seth at icir.org (Seth Hall) Date: Thu, 8 Sep 2011 01:37:56 -0400 Subject: [Bro-Dev] Memory leaks on git eds2245 In-Reply-To: <4E6851D1.6000307@icir.org> References: <20110908001004.GC48266@icir.org> <4E6851D1.6000307@icir.org> Message-ID: On Sep 8, 2011, at 1:25 AM, Gregor Maier wrote: > (BTW, the memory problems I have/had weren't "real" leaks. One a SSL > connection was done Bro would free the memory for it again. The problem > is that many SSL connections can live for days and thus they ultimately > consume memory like a "real" leak would). I implemented the code yesterday to stop analyzing connections with the skip_further_processing bif and it caused Bro to peak using more memory on the tracefile I was using it with than not stopping analysis of connections. One thing the SSL scripts are currently doing that I probably need to change is after logging the SSL log, I should probably do "delete c$ssl". The certificate and certificate chain are stored in there. Actually, as I think about it more that's probably most of the problem. We may want to look into the real traffic implications of calling the skip_further_processing bif eventually too though. I was pretty disheartened to see more memory used from calling that than not calling it. Perhaps it results in more memory use to remember which connections to ignore? I suppose I wasn't checking completion time which is probably where the savings should mostly come from. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Wed Sep 7 23:56:59 2011 From: seth at icir.org (Seth Hall) Date: Thu, 8 Sep 2011 02:56:59 -0400 Subject: [Bro-Dev] Memory leaks on git eds2245 In-Reply-To: References: <20110908001004.GC48266@icir.org> <4E6851D1.6000307@icir.org> Message-ID: <117E0DFF-56BE-4786-B550-BA2E7C1FA340@icir.org> On Sep 8, 2011, at 1:37 AM, Seth Hall wrote: > One thing the SSL scripts are currently doing that I probably need to change is after logging the SSL log, I should probably do "delete c$ssl". The certificate and certificate chain are stored in there. Actually, as I think about it more that's probably most of the problem. I implemented this and started running it on live traffic on a cluster and so far it seems to be holding up much better than it was previously. I'll have a better feel about it tomorrow but the initial indications seem to be that this is fixing most of the problem. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Thu Sep 8 00:24:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 07:24:56 -0000 Subject: [Bro-Dev] #329: Optimizing detect-protocols-http.bro In-Reply-To: <046.fc3e396e80d9cd99cc21e00aef84ff98@tracker.bro-ids.org> References: <046.fc3e396e80d9cd99cc21e00aef84ff98@tracker.bro-ids.org> Message-ID: <061.71797808f96e41d41b88f9faf9c30616@tracker.bro-ids.org> #329: Optimizing detect-protocols-http.bro ---------------------+---------------------- Reporter: seth | Owner: Type: Task | Status: assigned Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: Resolution: | Keywords: sprint ---------------------+---------------------- Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: Bumping this ticket back, it's not overly crucial right now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 00:26:30 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 07:26:30 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.578591a375705963b99cd74e2550f833@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): This may be fixed now with commit 4931aa815f5de7f0dac31e4ee526c69c56b4961d. Feedback regarding memory stability would be greatly appreciated. Hopefully we can get this ticket closed now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 00:28:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 07:28:21 -0000 Subject: [Bro-Dev] #588: Segfault in rotation code for logging framework In-Reply-To: <046.8c03ccb146ba869387339739e5ce79a3@tracker.bro-ids.org> References: <046.8c03ccb146ba869387339739e5ce79a3@tracker.bro-ids.org> Message-ID: <061.4aa89cc5bb9c3e6171d157acc51f383b@tracker.bro-ids.org> #588: Segfault in rotation code for logging framework ----------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by seth): I haven't seen this crash at places I would expect to see it since commit f868af101cd4e789094984fa0950d26822defcdc. I'll keep an eye on it for another day or two and close this ticket if I don't see the crashes anymore. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 00:30:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 07:30:07 -0000 Subject: [Bro-Dev] #599: Capstats and pf_ring In-Reply-To: <046.c573030a1b3b313b9ea9fd908254c7b1@tracker.bro-ids.org> References: <046.c573030a1b3b313b9ea9fd908254c7b1@tracker.bro-ids.org> Message-ID: <061.458bef18bc443bc520cfa7af87f16895@tracker.bro-ids.org> #599: Capstats and pf_ring -----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Capstats | Version: Resolution: | Keywords: beta -----------------------+-------------------- Comment (by seth): To avoid the issue on pf_ring enabled installations and make the cron command work, add the following line to /etc/broctl.cfg: {{{ CapstatsPath = }}} That will cause broctl to not attempt to use capstats. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 00:31:26 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 07:31:26 -0000 Subject: [Bro-Dev] #591: Time to finish collecting stats In-Reply-To: <046.79f8f007d391e0035e2e0757bc57d907@tracker.bro-ids.org> References: <046.79f8f007d391e0035e2e0757bc57d907@tracker.bro-ids.org> Message-ID: <061.ed687fa6cdf6892fabf2a245b06581e0@tracker.bro-ids.org> #591: Time to finish collecting stats -------------------------+------------------ Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: BroControl | Version: Resolution: | Keywords: beta -------------------------+------------------ Changes (by seth): * milestone: Bro1.6 => Comment: Removing milestone from this and leaving it as a possible reminder for the future. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 00:32:18 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 07:32:18 -0000 Subject: [Bro-Dev] #567: Protect/secure format strings in Reporter.cc In-Reply-To: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> References: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> Message-ID: <063.2a3990796c8b961b12834b9b43439a91@tracker.bro-ids.org> #567: Protect/secure format strings in Reporter.cc ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): Is this fully addressed through the fastpath commits that were merged into master recently? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 00:35:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 07:35:37 -0000 Subject: [Bro-Dev] #390: Problem with _logStats method in BroControl In-Reply-To: <046.ce93434f327b649ec398f3396d9295e6@tracker.bro-ids.org> References: <046.ce93434f327b649ec398f3396d9295e6@tracker.bro-ids.org> Message-ID: <061.51f54f61ee1ba1a4dff076a316e06520@tracker.bro-ids.org> #390: Problem with _logStats method in BroControl -------------------------------+------------------------ Reporter: seth | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: Feedback Missing | Keywords: -------------------------------+------------------------ Changes (by seth): * status: needs information => closed * resolution: => Feedback Missing Comment: I'm going to close this ticket since I haven't seen evidence of this problem in a while and I don't have further information. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 00:44:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 07:44:42 -0000 Subject: [Bro-Dev] #600: Coercion problem with log funcs In-Reply-To: <047.51c0e18a7925a83f11c8e3221c3807d2@tracker.bro-ids.org> References: <047.51c0e18a7925a83f11c8e3221c3807d2@tracker.bro-ids.org> Message-ID: <062.7dba9bee59172dfe4c58a5a9212824c8@tracker.bro-ids.org> #600: Coercion problem with log funcs ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by seth): Yep, this is the case I was having trouble with on the mailing list. Sorry I never followed up on that. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 07:02:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 14:02:56 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.f1336b9143db503fa5a9d240a51e2219@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Changes (by robin): * keywords: => beta -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 07:06:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 14:06:00 -0000 Subject: [Bro-Dev] #322: Update Checklist-for-Release In-Reply-To: <047.f46acab040dd9987da860d0b0051af15@tracker.bro-ids.org> References: <047.f46acab040dd9987da860d0b0051af15@tracker.bro-ids.org> Message-ID: <062.874886e82dbda3916a5948480941a582@tracker.bro-ids.org> #322: Update Checklist-for-Release -----------------------------+------------------------ Reporter: robin | Owner: Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied Comment: File is gone already. Will put something on the web pages later. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 07:26:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 14:26:28 -0000 Subject: [Bro-Dev] #378: Test for problem with optional record fields. In-Reply-To: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> References: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> Message-ID: <062.04beea9e2ed958961a19954e3de6fa7e@tracker.bro-ids.org> #378: Test for problem with optional record fields. --------------------------------+---------------------- Reporter: robin | Owner: robin Type: Test Case Missing | Status: accepted Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta --------------------------------+---------------------- Changes (by robin): * keywords: => beta Comment: This still crashes. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 07:45:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 14:45:12 -0000 Subject: [Bro-Dev] #498: Efficiency problem with remote log flushing In-Reply-To: <047.837129432a53b3b6b387597dd321a7c9@tracker.bro-ids.org> References: <047.837129432a53b3b6b387597dd321a7c9@tracker.bro-ids.org> Message-ID: <062.8e7fc1e1dc322ec8e5599082e42bd061@tracker.bro-ids.org> #498: Efficiency problem with remote log flushing ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Changes (by robin): * keywords: preview => beta -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 07:45:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 14:45:44 -0000 Subject: [Bro-Dev] #510: Upgrade How-To In-Reply-To: <047.e793bc8325e1f7ba35cd0d2c4f23760e@tracker.bro-ids.org> References: <047.e793bc8325e1f7ba35cd0d2c4f23760e@tracker.bro-ids.org> Message-ID: <062.b6e8ed0cb7b3f63edcd79d85fc402059@tracker.bro-ids.org> #510: Upgrade How-To ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Changes (by robin): * keywords: preview => beta -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 07:46:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 14:46:23 -0000 Subject: [Bro-Dev] #511: Misc distribution cleanup In-Reply-To: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> References: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> Message-ID: <062.9167c02faf5da93f7f58b100d4f79205@tracker.bro-ids.org> #511: Misc distribution cleanup ---------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ---------------------+-------------------- Changes (by robin): * keywords: preview => beta -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 07:46:57 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 14:46:57 -0000 Subject: [Bro-Dev] #519: policy/protocols/http/headers.bro only logs client headers In-Reply-To: <046.5777f3c379efa3c38e8a08a6979a91db@tracker.bro-ids.org> References: <046.5777f3c379efa3c38e8a08a6979a91db@tracker.bro-ids.org> Message-ID: <061.6f4b011da9b0bc33f0cf95104fb304c1@tracker.bro-ids.org> #519: policy/protocols/http/headers.bro only logs client headers ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Changes (by robin): * keywords: => beta -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 07:48:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 14:48:51 -0000 Subject: [Bro-Dev] #567: Protect/secure format strings in Reporter.cc In-Reply-To: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> References: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> Message-ID: <063.37fea340b03376ec9e80f0a29273ca21@tracker.bro-ids.org> #567: Protect/secure format strings in Reporter.cc ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by robin): We should do another double-check that all the messages are right. But I prefer to keep the functionality for the future. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 07:49:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 14:49:06 -0000 Subject: [Bro-Dev] #567: Protect/secure format strings in Reporter.cc In-Reply-To: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> References: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> Message-ID: <063.43ad81940754762c0ccb0455f055916f@tracker.bro-ids.org> #567: Protect/secure format strings in Reporter.cc ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Changes (by robin): * keywords: => beta -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 07:50:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 14:50:22 -0000 Subject: [Bro-Dev] #572: topic/jsiwek/filter-rotation In-Reply-To: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> References: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> Message-ID: <063.b646f8e793ebeea0ab826daf9afe82e3@tracker.bro-ids.org> #572: topic/jsiwek/filter-rotation ----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------------+------------------------ Changes (by robin): * keywords: => beta Comment: I guess I'll just merge it in for now and we can change that later. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 07:51:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 14:51:40 -0000 Subject: [Bro-Dev] #581: Add body length from http_stat to http.log In-Reply-To: <048.563f43cc45822dc0b2b6386e3aa94732@tracker.bro-ids.org> References: <048.563f43cc45822dc0b2b6386e3aa94732@tracker.bro-ids.org> Message-ID: <063.c48dc151485b071e9aa4fe9bddd72d87@tracker.bro-ids.org> #581: Add body length from http_stat to http.log ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Changes (by robin): * keywords: => beta Comment: If we want to do this, we should do it for the beta. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 07:56:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 14:56:35 -0000 Subject: [Bro-Dev] #572: topic/jsiwek/filter-rotation In-Reply-To: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> References: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> Message-ID: <063.6ca4946a6f2a1e55467f0857934fbde2@tracker.bro-ids.org> #572: topic/jsiwek/filter-rotation ----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------------+------------------------ Comment (by jsiwek): Replying to [comment:1 robin]: > I guess I'll just merge it in for now and we can change that later. Sorry, I was investigating how to remove the rotation_control and ran into some trickiness with how WriterInfo's only know about the path after it's been altered by the new default_path_func so that it became harder to go back and look up the corresponding Filter for it's rotation information. I might have something that works later today, but if I get stuck for too long, I'll let you know and we can revisit it later. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 08:03:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 15:03:20 -0000 Subject: [Bro-Dev] #572: topic/jsiwek/filter-rotation In-Reply-To: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> References: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> Message-ID: <063.0147db775df4afb469c946434095e521@tracker.bro-ids.org> #572: topic/jsiwek/filter-rotation ---------------------+------------------------ Reporter: jsiwek | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ---------------------+------------------------ Changes (by jsiwek): * type: Merge Request => Task -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 08:06:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 15:06:44 -0000 Subject: [Bro-Dev] #597: net_init and bro_init In-Reply-To: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> References: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> Message-ID: <061.bf73cfb89f954a003c70a3c56bbeffbe@tracker.bro-ids.org> #597: net_init and bro_init ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Changes (by robin): * keywords: => beta -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 08:17:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 15:17:24 -0000 Subject: [Bro-Dev] #601: Beta documentation Message-ID: <047.b53a75afaafe0544bba44907e863138d@tracker.bro-ids.org> #601: Beta documentation ------------------------+-------------------- Reporter: robin | Type: Task Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Version: git/master | Keywords: beta ------------------------+-------------------- This ticket collects what we should have in place before releasing the beta: - Quickstart guide - Upgrade guide - Complete script reference - How-To for the new logging framework - How-To for the new notice framework - Updated BroCtl README - reST-based Broccoli docs - doxygen-based Broccoli API reference - Updates READMEs for the other submodules. Optional: - Guide on the new model for extending scripts. I'll create a documentation-2.0 directory in www where we can built the new documentation page. We can then switch over when we do the release (not the beta). -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 08:17:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 15:17:54 -0000 Subject: [Bro-Dev] #503: Update broctl README In-Reply-To: <047.1a0db4ec338e9735ce78c840cd80bbc3@tracker.bro-ids.org> References: <047.1a0db4ec338e9735ce78c840cd80bbc3@tracker.bro-ids.org> Message-ID: <062.692f3b6123b14c3404b6993a9d6946af@tracker.bro-ids.org> #503: Update broctl README -----------------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: Solved/Applied | Keywords: preview -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied Comment: Superseded by doc-meta ticket #601. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 08:34:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 15:34:27 -0000 Subject: [Bro-Dev] #549: Expose broccoli file handle through pybroccoli In-Reply-To: <046.1adc4393e81172e99fdad54fd5737163@tracker.bro-ids.org> References: <046.1adc4393e81172e99fdad54fd5737163@tracker.bro-ids.org> Message-ID: <061.1b9cc1a5fe795442efdb25d0ad68b9f0@tracker.bro-ids.org> #549: Expose broccoli file handle through pybroccoli ------------------------------+-------------------- Reporter: seth | Owner: robin Type: Feature Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: broccoli-python | Version: Resolution: Solved/Applied | Keywords: ------------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied Comment: It's already there: {{{ int bro_conn_get_fd(BroConn *bc); }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 08:37:08 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 15:37:08 -0000 Subject: [Bro-Dev] #593: topic/gilbert/rand-pool In-Reply-To: <048.552e03f58ca192dae7ba2307bac44e74@tracker.bro-ids.org> References: <048.552e03f58ca192dae7ba2307bac44e74@tracker.bro-ids.org> Message-ID: <063.fc820361d809d6e1c0df9584547bb761@tracker.bro-ids.org> #593: topic/gilbert/rand-pool -----------------------------+-------------------- Reporter: gclark | Owner: robin Type: Merge Request | Status: closed Priority: Low | Milestone: Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Thu Sep 8 08:41:21 2011 From: gregor at icir.org (Gregor Maier) Date: Thu, 08 Sep 2011 08:41:21 -0700 Subject: [Bro-Dev] Memory leaks on git eds2245 In-Reply-To: References: <20110908001004.GC48266@icir.org> <4E6851D1.6000307@icir.org> Message-ID: <4E68E221.5040402@icir.org> On 9/7/11 22:37 , Seth Hall wrote: > On Sep 8, 2011, at 1:25 AM, Gregor Maier wrote: > >> (BTW, the memory problems I have/had weren't "real" leaks. One a SSL >> connection was done Bro would free the memory for it again. The problem >> is that many SSL connections can live for days and thus they ultimately >> consume memory like a "real" leak would). > > > I implemented the code yesterday to stop analyzing connections with the skip_further_processing bif and it caused Bro to peak using more memory on the tracefile I was using it with than not stopping analysis of connections. One thing the SSL scripts are currently doing that I probably need to change is after logging the SSL log, I should probably do "delete c$ssl". The certificate and certificate chain are stored in there. Actually, as I think about it more that's probably most of the problem. skip_further_processing() actually just sets the skip flag which means that no further data is delivered to the analyzers, but the analyzers aren't removed. The disable_analyzer() bif will actually remove the analyzer, however it needs an analyzer_id, so the SSL analyzer would somehow need to add it's analyzer_id to one of it's events. > We may want to look into the real traffic implications of calling the skip_further_processing bif eventually too though. I was pretty disheartened to see more memory used from calling that than not calling it. Perhaps it results in more memory use to remember which connections to ignore? I suppose I wasn't checking completion time which is probably where the savings should mostly come from. How much difference in memory usage did you see? And how much memory usage do you see in general? My Bros usually need about 100-300MB and with SSL I will eventually get to 1GB or more (not on all nodes and it will often take hours until it starts get there). (Disabling SSL altogether will reduce the "baseline" usage by about 25%--50%) However, it still puzzles me that skip_further_processing didn't help. In my case memory consumption was ramping up over several minutes so either data is still delivered to the analyzer during this ramp-up or there are additional reasons for the memory consumption? cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Thu Sep 8 09:02:08 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 16:02:08 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.381a45acabe51a47c6fa1000381f2091@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): Tried to merge this, but some issues: - I get this error with `make doc`: {{{ Generating file member index... finalizing index lists... finished... [100%] Built target broccolidoc make[4]: *** No rule to make target `broccolidoc', needed by `doc/CMakeFiles/doc'. Stop. make[3]: *** [doc/CMakeFiles/doc.dir/all] Error 2 make[2]: *** [doc/CMakeFiles/doc.dir/rule] Error 2 make[1]: *** [doc] Error 2 make: *** [doc] Error 2 }}} - The generated Doxygen docs show the data structures on the overview page but not the more important functions. Also, they don't seem to pick up some (all?) of the comments. This is Doxygen 1.7.2 if it matters. - Generally I think we don't need/want to document all the internal stuff. An equivalent of this page would be good: http://www.icir.org/christian/broccoli/manual/broccoli-broccoli.html I believe that's just `broccoli.h`. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 09:05:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 16:05:24 -0000 Subject: [Bro-Dev] #592: topic/gilbert/profiles In-Reply-To: <048.50f5253509e35787f64b7ef3adde24a0@tracker.bro-ids.org> References: <048.50f5253509e35787f64b7ef3adde24a0@tracker.bro-ids.org> Message-ID: <063.c88265a2851f23f288e350b5f433c8e7@tracker.bro-ids.org> #592: topic/gilbert/profiles ----------------------------+---------------------- Reporter: gclark | Owner: gclark Type: Merge Request | Status: assigned Priority: Low | Milestone: Component: BTest | Version: Resolution: | Keywords: ----------------------------+---------------------- Changes (by robin): * owner: => gclark * status: new => assigned * component: bro-aux => BTest Comment: Please add documentation to the README as discussed. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 09:08:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 16:08:43 -0000 Subject: [Bro-Dev] #598: topic/jsiwek/misc-doc-fixes In-Reply-To: <048.9c484d4c829b091b2b97e02b8847b5da@tracker.bro-ids.org> References: <048.9c484d4c829b091b2b97e02b8847b5da@tracker.bro-ids.org> Message-ID: <063.4f94af51d258c4c6c6375e41ea00cf51@tracker.bro-ids.org> #598: topic/jsiwek/misc-doc-fixes ----------------------------+------------------------ Reporter: jsiwek | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [db8ab89c3a76f1acdfb9b5049d663cca31cd88a0/bro]: {{{ #!CommitTicketReference repository="bro" revision="db8ab89c3a76f1acdfb9b5049d663cca31cd88a0" Merge remote branch 'origin/topic/jsiwek/misc-doc-fixes' * origin/topic/jsiwek/misc-doc-fixes: More tweaks to generated script docs. Various changes to documentation framework. Closes #598. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 09:24:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 16:24:55 -0000 Subject: [Bro-Dev] #567: Protect/secure format strings in Reporter.cc In-Reply-To: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> References: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> Message-ID: <063.d06aad43a9f00d30eb9fabe470d29aad@tracker.bro-ids.org> #567: Protect/secure format strings in Reporter.cc ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by gregor): {{{ #!rst The fastpath commit found the one instance were I experienced crashes. I would still strongly opt to change the way the format strings are done currently. Either by a) requiring developers to explicitly use the fmt() function and removing format string capabilities from the reporter. b) Having two sets of methods in Reporter. E.g., Info() and InfoFmt() of which the first one only accepts fixed strings and the second accepts a format string. This way a developer needs to make a conscious decision about using format strings. Why do I really want this? * Such bug are quite dangerous because they could ultimate lead to arbitrary code execution. * They are hard to find, since it's likely that the bug will only be triggered in very rare cases. * It's easy to make this error. E.g., a developer wants to print a warning and sees another piece of code that uses reporter->Warning(foo) (which might be perfectly save if foo is guaranteed to not have format strings) and assumes that Warning() just takes a fixed string as argument. }} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 09:29:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 16:29:01 -0000 Subject: [Bro-Dev] #601: Beta documentation In-Reply-To: <047.b53a75afaafe0544bba44907e863138d@tracker.bro-ids.org> References: <047.b53a75afaafe0544bba44907e863138d@tracker.bro-ids.org> Message-ID: <062.58b1ec05a5a4d10aa1d8c4399862640a@tracker.bro-ids.org> #601: Beta documentation ---------------------+------------------------ Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ---------------------+------------------------ Comment (by gregor): > - reST-based Broccoli docs > - doxygen-based Broccoli API reference What's the reason why those aren't optional? Won't most users never see the the broccoli API directly anyways? cu Gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 09:49:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 16:49:03 -0000 Subject: [Bro-Dev] #597: net_init and bro_init In-Reply-To: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> References: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> Message-ID: <061.6d953bba06f3e873b9b3807d08b6d09d@tracker.bro-ids.org> #597: net_init and bro_init ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by seth): I played with this for a while last night and I was unable to move things around successfully without causing crashes so I did the integration in broctl instead. We can either bump this ticket back or close it. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 10:13:04 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 17:13:04 -0000 Subject: [Bro-Dev] #567: Protect/secure format strings in Reporter.cc In-Reply-To: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> References: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> Message-ID: <063.53b30887408ac20332b4e228dba24412@tracker.bro-ids.org> #567: Protect/secure format strings in Reporter.cc ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by robin): On Thu, Sep 08, 2011 at 16:24 -0000, you wrote: > b) Having two sets of methods in Reporter. E.g., Info() and InfoFmt() of Can we achieve this with overloading the function? Not quite sure whether the compiler would find it ambigious though. > Why do I really want this? I understandn why you want this. But having to use fmt() pretty much everytime isn't great either, and the case where one passes in a variable as the first argument seems rare. And we already have other functions than fmt() as well that take var args. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 10:14:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 17:14:15 -0000 Subject: [Bro-Dev] #601: Beta documentation In-Reply-To: <047.b53a75afaafe0544bba44907e863138d@tracker.bro-ids.org> References: <047.b53a75afaafe0544bba44907e863138d@tracker.bro-ids.org> Message-ID: <062.2a8dd981c52b87336e04fa7ab36bfbe9@tracker.bro-ids.org> #601: Beta documentation ---------------------+------------------------ Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ---------------------+------------------------ Comment (by robin): On Thu, Sep 08, 2011 at 16:29 -0000, you wrote: > What's the reason why those aren't optional? These docs already exist, we just need to get them into shape. I don't really want to make a step backwards in terms of available docs. Robin -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Sep 8 10:53:16 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 8 Sep 2011 10:53:16 -0700 Subject: [Bro-Dev] Updating testing/external Message-ID: <20110908175316.GO71984@icir.org> I'm going to do some work on the testing/external infrastructure, which meanss the testing repositories may not be available for a bit (and will likely need a fresh clone afterwards). This is to remove the big trace files from the repositories; wasn't a good idea to store them in git. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Sep 8 11:11:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 18:11:40 -0000 Subject: [Bro-Dev] #567: Protect/secure format strings in Reporter.cc In-Reply-To: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> References: <048.2977795f08466e96df0bed12ea39f590@tracker.bro-ids.org> Message-ID: <063.fb1d5009e2de043c0d25edc57fcdb85c@tracker.bro-ids.org> #567: Protect/secure format strings in Reporter.cc ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by gregor): > > b) Having two sets of methods in Reporter. E.g., Info() and InfoFmt() > of > > Can we achieve this with overloading the function? Not quite sure > whether the compiler would find it ambigious though. Unfortunately not, since the compiler doesn't know which function to use :-( I tried that when I did the fix for the segfault. cu Gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 12:40:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 19:40:28 -0000 Subject: [Bro-Dev] #602: trace-summary doesn't parse new log headers Message-ID: <048.ed1ed25efc27f66c8e7e9430c52b0ecd@tracker.bro-ids.org> #602: trace-summary doesn't parse new log headers ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: trace-summary | Version: git/master Keywords: beta | ---------------------------+------------------------ e.g. here's what broctl emailed me after the usual rotating of logs and trying to build the connection summary: {{{ Ignoring corrupt line: '#separator \x09' Ignoring corrupt line: '#path conn' Traceback (most recent call last): File "/home/jsiwek/tmp/bro/build/root/bin/trace-summary", line 854, in readConnSummaries(file) File "/home/jsiwek/tmp/bro/build/root/bin/trace-summary", line 451, in readConnSummaries parseConnLine(line) File "/home/jsiwek/tmp/bro/build/root/bin/trace-summary", line 512, in parseConnLine time = float(f[ts_idx]) UnboundLocalError: local variable 'ts_idx' referenced before assignment Command exited with non-zero status 1 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 12:53:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 19:53:06 -0000 Subject: [Bro-Dev] #602: trace-summary doesn't parse new log headers In-Reply-To: <048.ed1ed25efc27f66c8e7e9430c52b0ecd@tracker.bro-ids.org> References: <048.ed1ed25efc27f66c8e7e9430c52b0ecd@tracker.bro-ids.org> Message-ID: <063.da4ec32762e4528e0e299dde7053d7d5@tracker.bro-ids.org> #602: trace-summary doesn't parse new log headers ----------------------------+------------------------ Reporter: jsiwek | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: trace-summary | Version: git/master Resolution: fixed | Keywords: beta ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [fb8f25a0f73fee40cd2403ee39fdf18c350e012a/trace-summary]: {{{ #!CommitTicketReference repository="trace-summary" revision="fb8f25a0f73fee40cd2403ee39fdf18c350e012a" Now ignoring all lines starting with a pound. Closes #602. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 13:01:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 20:01:17 -0000 Subject: [Bro-Dev] #572: topic/jsiwek/filter-rotation In-Reply-To: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> References: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> Message-ID: <063.02633a68e9e1d0356e2657e63d83d158@tracker.bro-ids.org> #572: topic/jsiwek/filter-rotation ---------------------+------------------------ Reporter: jsiwek | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ---------------------+------------------------ Comment (by jsiwek): In [fe38c22d2bd84f9ca132f7a216d71aebeea4d5df/bro]: {{{ #!CommitTicketReference repository="bro" revision="fe38c22d2bd84f9ca132f7a216d71aebeea4d5df" Remove Log::rotation_control (addresses #572). Log rotation is now controlled directly through Filter records. Also addressed a TODO in the default_path_func regarding the LogMgr::AddFilter function generating internal filter path suggestions/fallbacks. Now, if the user doesn't explicitly set a filter path, the filter's path will be the result of the first call to default_path_func (happens during the first write to the log). And in that case the path suggestion argument to the path_func is an empty string. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 13:07:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 20:07:51 -0000 Subject: [Bro-Dev] #572: topic/jsiwek/filter-rotation In-Reply-To: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> References: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> Message-ID: <063.0c3506f889c09814bf51512239485028@tracker.bro-ids.org> #572: topic/jsiwek/filter-rotation ----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------------+------------------------ Changes (by jsiwek): * type: Task => Merge Request Comment: > ran into some trickiness with how WriterInfo's only know about the path after it's been altered by the new default_path_func so that it became harder to go back and look up the corresponding Filter for it's rotation information. Actually the problem seemed to be just the TODO regarding the internal generation of the filter's optional path field if it wasn't explicitly set at the time the filter is added to a stream. After fixing that it, seems to work fine now in absence of `Log::rotation_control`. -- Ticket URL: Bro Tracker Bro Issue Tracker From gc355804 at ohio.edu Thu Sep 8 13:33:41 2011 From: gc355804 at ohio.edu (G. Clark) Date: Thu, 08 Sep 2011 13:33:41 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/parallel-btest: Adding group "comm" to communication tests for parallelizing execution with new btest version. (32cb6d6) In-Reply-To: <201109081613.p88GD4m4024259@bro-ids.icir.org> References: <201109081613.p88GD4m4024259@bro-ids.icir.org> Message-ID: <4E6926A5.9090006@ohio.edu> It seems like TEST-GROUP is very similar to TEST-PROFILE (in the btest profile branch I'm working on). . . any way to merge these? --Gilbert On 9/8/11 9:13 AM, Robin Sommer wrote: > Repository : ssh://git at bro-ids.icir.org/bro > > On branch : topic/robin/parallel-btest > Link : http://tracker.bro-ids.org/bro/changeset/32cb6d6bc31c03d746bb0b72b012dd6730308c54/bro > >> --------------------------------------------------------------- > > commit 32cb6d6bc31c03d746bb0b72b012dd6730308c54 > Author: Robin Sommer > Date: Fri Sep 2 23:55:32 2011 -0700 > > Adding group "comm" to communication tests for parallelizing execution > with new btest version. > > >> --------------------------------------------------------------- > > 32cb6d6bc31c03d746bb0b72b012dd6730308c54 > testing/btest/istate/broccoli.bro | 1 + > testing/btest/istate/events-ssl.bro | 2 +- > testing/btest/istate/events.bro | 2 +- > testing/btest/istate/pybroccoli.py | 1 + > testing/btest/istate/sync.bro | 2 +- > .../policy/frameworks/cluster/start-it-up.bro | 1 + > .../frameworks/control/configuration_update.bro | 1 + > .../btest/policy/frameworks/control/id_value.bro | 1 + > .../btest/policy/frameworks/control/shutdown.bro | 1 + > .../policy/frameworks/logging/remote-types.bro | 2 +- > testing/btest/policy/frameworks/logging/remote.bro | 2 +- > .../policy/frameworks/metrics/basic-cluster.bro | 1 + > 12 files changed, 12 insertions(+), 5 deletions(-) > > diff --git a/testing/btest/istate/broccoli.bro b/testing/btest/istate/broccoli.bro > index 7f97f40..def9a78 100644 > --- a/testing/btest/istate/broccoli.bro > +++ b/testing/btest/istate/broccoli.bro > @@ -1,6 +1,7 @@ > # @TEST-REQUIRES: grep -vq '#define BROv6' $BUILD/config.h > # @TEST-REQUIRES: test -e $BUILD/aux/broccoli/src/libbroccoli.so || test -e $BUILD/aux/broccoli/src/libbroccoli.dylib > # > +# @TEST-GROUP: comm > # @TEST-EXEC: btest-bg-run bro bro %INPUT $DIST/aux/broccoli/test/broping-record.bro > # @TEST-EXEC: btest-bg-run broccoli $BUILD/aux/broccoli/test/broping -r -c 3 127.0.0.1 > # @TEST-EXEC: btest-bg-wait -k 20 > diff --git a/testing/btest/istate/events-ssl.bro b/testing/btest/istate/events-ssl.bro > index cfacae9..02fd957 100644 > --- a/testing/btest/istate/events-ssl.bro > +++ b/testing/btest/istate/events-ssl.bro > @@ -1,4 +1,4 @@ > -# > +# @TEST-GROUP: comm > # @TEST-EXEC: btest-bg-run sender bro -C -r $TRACES/web.trace --pseudo-realtime ../sender.bro > # @TEST-EXEC: btest-bg-run receiver bro ../receiver.bro > # @TEST-EXEC: btest-bg-wait -k 20 > diff --git a/testing/btest/istate/events.bro b/testing/btest/istate/events.bro > index ecf2f2e..d2dfc63 100644 > --- a/testing/btest/istate/events.bro > +++ b/testing/btest/istate/events.bro > @@ -1,4 +1,4 @@ > -# > +# @TEST-GROUP: comm > # @TEST-EXEC: btest-bg-run sender bro -C -r $TRACES/web.trace --pseudo-realtime ../sender.bro > # @TEST-EXEC: btest-bg-run receiver bro ../receiver.bro > # @TEST-EXEC: btest-bg-wait -k 20 > diff --git a/testing/btest/istate/pybroccoli.py b/testing/btest/istate/pybroccoli.py > index b7fb53a..ec4097a 100644 > --- a/testing/btest/istate/pybroccoli.py > +++ b/testing/btest/istate/pybroccoli.py > @@ -1,3 +1,4 @@ > +# @TEST-GROUP: comm > # @TEST-REQUIRES: grep -vq '#define BROv6' $BUILD/config.h > # @TEST-REQUIRES: test -e $BUILD/aux/broccoli/src/libbroccoli.so || test -e $BUILD/aux/broccoli/src/libbroccoli.dylib > # @TEST-REQUIRES: test -e $BUILD/aux/broccoli/bindings/broccoli-python/_broccoli_intern.so > diff --git a/testing/btest/istate/sync.bro b/testing/btest/istate/sync.bro > index 567bbf2..c41542d 100644 > --- a/testing/btest/istate/sync.bro > +++ b/testing/btest/istate/sync.bro > @@ -1,4 +1,4 @@ > -# > +# @TEST-GROUP: comm > # @TEST-EXEC: btest-bg-run sender bro %INPUT ../sender.bro > # @TEST-EXEC: btest-bg-run receiver bro %INPUT ../receiver.bro > # @TEST-EXEC: btest-bg-wait 20 > diff --git a/testing/btest/policy/frameworks/cluster/start-it-up.bro b/testing/btest/policy/frameworks/cluster/start-it-up.bro > index d1eb94d..c7d6c47 100644 > --- a/testing/btest/policy/frameworks/cluster/start-it-up.bro > +++ b/testing/btest/policy/frameworks/cluster/start-it-up.bro > @@ -1,3 +1,4 @@ > +# @TEST-GROUP: comm > # @TEST-EXEC: btest-bg-run manager-1 BROPATH=$BROPATH:.. CLUSTER_NODE=manager-1 bro %INPUT > # @TEST-EXEC: btest-bg-run proxy-1 BROPATH=$BROPATH:.. CLUSTER_NODE=proxy-1 bro %INPUT > # @TEST-EXEC: btest-bg-run proxy-2 BROPATH=$BROPATH:.. CLUSTER_NODE=proxy-2 bro %INPUT > diff --git a/testing/btest/policy/frameworks/control/configuration_update.bro b/testing/btest/policy/frameworks/control/configuration_update.bro > index 23b4998..6d0f886 100644 > --- a/testing/btest/policy/frameworks/control/configuration_update.bro > +++ b/testing/btest/policy/frameworks/control/configuration_update.bro > @@ -1,3 +1,4 @@ > +# @TEST-GROUP: comm > # @TEST-EXEC: btest-bg-run controllee BROPATH=$BROPATH:.. bro %INPUT frameworks/control/controllee Communication::listen_port_clear=65531/tcp > # @TEST-EXEC: btest-bg-run controller BROPATH=$BROPATH:.. bro %INPUT test-redef frameworks/control/controller Control::host=127.0.0.1 Control::host_port=65531/tcp Control::cmd=configuration_update > # @TEST-EXEC: btest-bg-run controller2 BROPATH=$BROPATH:.. bro %INPUT frameworks/control/controller Control::host=127.0.0.1 Control::host_port=65531/tcp Control::cmd=shutdown > diff --git a/testing/btest/policy/frameworks/control/id_value.bro b/testing/btest/policy/frameworks/control/id_value.bro > index 9f0cb76..c751928 100644 > --- a/testing/btest/policy/frameworks/control/id_value.bro > +++ b/testing/btest/policy/frameworks/control/id_value.bro > @@ -1,3 +1,4 @@ > +# @TEST-GROUP: comm > # @TEST-EXEC: btest-bg-run controllee BROPATH=$BROPATH:.. bro %INPUT only-for-controllee frameworks/control/controllee Communication::listen_port_clear=65532/tcp > # @TEST-EXEC: btest-bg-run controller BROPATH=$BROPATH:.. bro %INPUT frameworks/control/controller Control::host=127.0.0.1 Control::host_port=65532/tcp Control::cmd=id_value Control::arg=test_var > # @TEST-EXEC: btest-bg-wait -k 1 > diff --git a/testing/btest/policy/frameworks/control/shutdown.bro b/testing/btest/policy/frameworks/control/shutdown.bro > index 55af973..e7aabea 100644 > --- a/testing/btest/policy/frameworks/control/shutdown.bro > +++ b/testing/btest/policy/frameworks/control/shutdown.bro > @@ -1,3 +1,4 @@ > +# @TEST-GROUP: comm > # @TEST-EXEC: btest-bg-run controllee BROPATH=$BROPATH:.. bro %INPUT frameworks/control/controllee Communication::listen_port_clear=65530/tcp > # @TEST-EXEC: btest-bg-run controller BROPATH=$BROPATH:.. bro %INPUT frameworks/control/controller Control::host=127.0.0.1 Control::host_port=65530/tcp Control::cmd=shutdown > # @TEST-EXEC: btest-bg-wait 1 > diff --git a/testing/btest/policy/frameworks/logging/remote-types.bro b/testing/btest/policy/frameworks/logging/remote-types.bro > index 08960eb..164e7f3 100644 > --- a/testing/btest/policy/frameworks/logging/remote-types.bro > +++ b/testing/btest/policy/frameworks/logging/remote-types.bro > @@ -1,4 +1,4 @@ > -# > +# @TEST-GROUP: comm > # @TEST-EXEC: btest-bg-run sender bro --pseudo-realtime %INPUT ../sender.bro > # @TEST-EXEC: btest-bg-run receiver bro --pseudo-realtime %INPUT ../receiver.bro > # @TEST-EXEC: btest-bg-wait -k 1 > diff --git a/testing/btest/policy/frameworks/logging/remote.bro b/testing/btest/policy/frameworks/logging/remote.bro > index 00d7d69..34cf714 100644 > --- a/testing/btest/policy/frameworks/logging/remote.bro > +++ b/testing/btest/policy/frameworks/logging/remote.bro > @@ -1,4 +1,4 @@ > -# > +# @TEST-GROUP: comm > # @TEST-EXEC: btest-bg-run sender bro --pseudo-realtime %INPUT ../sender.bro > # @TEST-EXEC: sleep 1 > # @TEST-EXEC: btest-bg-run receiver bro --pseudo-realtime %INPUT ../receiver.bro > diff --git a/testing/btest/policy/frameworks/metrics/basic-cluster.bro b/testing/btest/policy/frameworks/metrics/basic-cluster.bro > index eda41c3..a282a96 100644 > --- a/testing/btest/policy/frameworks/metrics/basic-cluster.bro > +++ b/testing/btest/policy/frameworks/metrics/basic-cluster.bro > @@ -1,3 +1,4 @@ > +# @TEST-GROUP: comm > # @TEST-EXEC: btest-bg-run manager-1 BROPATH=$BROPATH:.. CLUSTER_NODE=manager-1 bro %INPUT > # @TEST-EXEC: btest-bg-run proxy-1 BROPATH=$BROPATH:.. CLUSTER_NODE=proxy-1 bro %INPUT > # @TEST-EXEC: sleep 1 > > _______________________________________________ > bro-commits mailing list > bro-commits at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6010 bytes Desc: S/MIME Cryptographic Signature Url : http://mailman.icsi.berkeley.edu/pipermail/bro-dev/attachments/20110908/04bcc941/attachment-0001.bin From bro at tracker.bro-ids.org Thu Sep 8 14:00:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 21:00:11 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.ec29395b308e7ea94b42b65e0a6e7c8a@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): In [e34cdb437d8820aa3db345c787573817735e6959/broccoli]: {{{ #!CommitTicketReference repository="broccoli" revision="e34cdb437d8820aa3db345c787573817735e6959" Broccoli API Doxygen doc tweaks (addresses #563) - Broccoli.h is now the only source file for which docs are generated. - Changing some comment styles and setting an AUTOBRIEF option so Doxygen will render more of what's in comments. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 8 14:08:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 08 Sep 2011 21:08:58 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.9295dc44ab8ca5b68d92589df63936ef@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): > - I get this error with `make doc`: > > {{{ > Generating file member index... > finalizing index lists... > finished... > [100%] Built target broccolidoc > make[4]: *** No rule to make target `broccolidoc', needed by `doc/CMakeFiles/doc'. Stop. > make[3]: *** [doc/CMakeFiles/doc.dir/all] Error 2 > make[2]: *** [doc/CMakeFiles/doc.dir/rule] Error 2 > make[1]: *** [doc] Error 2 > make: *** [doc] Error 2 > > }}} Do you still see this? If so, what directory are you running `make doc` from and what version of CMake are you using? > - The generated Doxygen docs show the data structures on the overview page but not the more important functions. Now sure exactly what you mean, but maybe this is addressed by the other fixes. Not sure what to do right now about the main index being so empty, but it can be extended with the `@mainpage` section I added at the top of `broccoli.h.in` > Also, they don't seem to pick up some (all?) of the comments Think I got this by updating some comment styles and setting an AUTOBRIEF to the Doxyfile config. > - Generally I think we don't need/want to document all the internal stuff. An equivalent of this page would be good: http://www.icir.org/christian/broccoli/manual/broccoli-broccoli.html I believe that's just `broccoli.h`. Changed it to just render docs for broccoli.h. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Fri Sep 9 05:32:44 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 9 Sep 2011 05:32:44 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/parallel-btest: Adding group "comm" to communication tests for parallelizing execution with new btest version. (32cb6d6) In-Reply-To: <4E6926A5.9090006@ohio.edu> References: <201109081613.p88GD4m4024259@bro-ids.icir.org> <4E6926A5.9090006@ohio.edu> Message-ID: <20110909123244.GH9465@icir.org> On Thu, Sep 08, 2011 at 13:33 -0700, you wrote: > It seems like TEST-GROUP is very similar to TEST-PROFILE (in the btest > profile branch I'm working on). . Does TEST-PROFILE limit a test to run only when a specific profile has been selected, and be skipped if not? What's a use case for using it? Btw, another question: the "transform" pass runs after the test. Can we add another similar one that runs before the test executes? We could then remove BTest's filters and use profiles instead I believe. And: your README says "four scripts need to be defined": please make them optional so that one can skip scripts that aren't needed for a profile. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Fri Sep 9 07:08:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 09 Sep 2011 14:08:58 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.fe90a43c24b0e2302ad4f4092bde825f@tracker.bro-ids.org> #535: Get rid of the "net" type. ----------------------------+-------------------- Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: ----------------------------+-------------------- Comment (by jsiwek): In [5fbe406ecd4b52f1458a87581fe5087030231360/broccoli-ruby]: {{{ #!CommitTicketReference repository="broccoli-ruby" revision="5fbe406ecd4b52f1458a87581fe5087030231360" Remove the 'net' type from Broccoli ruby bindings (addresses #535). Also removed swig-generated files from git control. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 9 07:08:58 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 09 Sep 2011 14:08:58 -0000 Subject: [Bro-Dev] #535: Get rid of the "net" type. In-Reply-To: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> References: <046.e9aae531beed1e354d3444fdf028f3d3@tracker.bro-ids.org> Message-ID: <061.8b45f5543aa88e741dc82b3871572ceb@tracker.bro-ids.org> #535: Get rid of the "net" type. ----------------------------+-------------------- Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: ----------------------------+-------------------- Comment (by seth): In [cfc1d6e1ef8fc432933bff861bd9d938fa929cf5/broccoli-ruby]: {{{ #!CommitTicketReference repository="broccoli-ruby" revision="cfc1d6e1ef8fc432933bff861bd9d938fa929cf5" Merge remote-tracking branch 'origin/topic/jsiwek/remove-net-type' * origin/topic/jsiwek/remove-net-type: Remove the 'net' type from Broccoli ruby bindings (addresses #535). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 9 07:18:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 09 Sep 2011 14:18:54 -0000 Subject: [Bro-Dev] #603: Checking correctness of logs Message-ID: <047.5d54b4bcd94fce279ea5de274dc02946@tracker.bro-ids.org> #603: Checking correctness of logs ------------------------+-------------------- Reporter: robin | Type: Task Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Version: git/master | ------------------------+-------------------- Before we release the final 2.0, we really need to do a rather thorough check of the logs to make sure they are correct. The way I picture doing that is that everybody picks connections at random and manually checks that the logs report what he'd expect from examining the raw payload with tcdpump/wireshark/strings/whatever. That's pretty painful but I don't really see a better way. Thoughts welcome. -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Fri Sep 9 07:43:44 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 9 Sep 2011 07:43:44 -0700 Subject: [Bro-Dev] Updating testing/external In-Reply-To: <20110908175316.GO71984@icir.org> References: <20110908175316.GO71984@icir.org> Message-ID: <20110909144344.GA16506@icir.org> On Thu, Sep 08, 2011 at 10:53 -0700, I wrote: > I'm going to do some work on the testing/external infrastructure, Done with this. I've recreated the bro-testing repository so if you have checked out the old one into testing/external, first remove it and then follow the (updated) README again. It will now pull the traces via curl. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Fri Sep 9 07:49:51 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 9 Sep 2011 07:49:51 -0700 Subject: [Bro-Dev] tests and libmagic (Re: Updating testing/external) In-Reply-To: <20110909144344.GA16506@icir.org> References: <20110908175316.GO71984@icir.org> <20110909144344.GA16506@icir.org> Message-ID: <20110909144951.GO9465@icir.org> One problem I'm running into is that different libmagic setups classify data in different ways. For example, I see a number of HTTP entities classified as text/html on one machine yet as text/plain on another. Not sure how to deal with that for test baselines. I'm thinking to preprocess logs to just have a boolean flag indicating whether there is a mime type at all before diffing but otherwise make the comparision oblivious to the actual value. Better ideas? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Fri Sep 9 07:52:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 09 Sep 2011 14:52:15 -0000 Subject: [Bro-Dev] #603: Checking correctness of logs In-Reply-To: <047.5d54b4bcd94fce279ea5de274dc02946@tracker.bro-ids.org> References: <047.5d54b4bcd94fce279ea5de274dc02946@tracker.bro-ids.org> Message-ID: <062.8ed1a5f808136dd6c6fb59713fd6247c@tracker.bro-ids.org> #603: Checking correctness of logs ---------------------+------------------------ Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ---------------------+------------------------ Comment (by gregor): On 9/9/11 7:18 , Bro Tracker wrote: > Before we release the final 2.0, we really need to do a rather > thorough check of the logs to make sure they are correct. The way I > picture doing that is that everybody picks connections at random and > manually checks that the logs report what he'd expect from examining > the raw payload with tcdpump/wireshark/strings/whatever. That's pretty > painful but I don't really see a better way. Thoughts welcome. Additionally comparing to the output of Bro-1.5 is also a (less painful) option. cu Gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Fri Sep 9 07:58:22 2011 From: seth at icir.org (Seth Hall) Date: Fri, 9 Sep 2011 10:58:22 -0400 Subject: [Bro-Dev] tests and libmagic (Re: Updating testing/external) In-Reply-To: <20110909144951.GO9465@icir.org> References: <20110908175316.GO71984@icir.org> <20110909144344.GA16506@icir.org> <20110909144951.GO9465@icir.org> Message-ID: <80FD231E-1F61-4067-A974-237BF5B5EA38@icir.org> On Sep 9, 2011, at 10:49 AM, Robin Sommer wrote: > Not sure how to deal with that for test baselines. I'm thinking to > preprocess logs to just have a boolean flag indicating whether there > is a mime type at all before diffing but otherwise make the > comparision oblivious to the actual value. > > Better ideas? Jon handled this in several places are there some he missed? Which tests are you having trouble with? Part of the general file analysis work will be completely not relying on libmagic for file type identification anymore, it's way too annoying. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Fri Sep 9 08:06:49 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 9 Sep 2011 08:06:49 -0700 Subject: [Bro-Dev] tests and libmagic (Re: Updating testing/external) In-Reply-To: <80FD231E-1F61-4067-A974-237BF5B5EA38@icir.org> References: <20110908175316.GO71984@icir.org> <20110909144344.GA16506@icir.org> <20110909144951.GO9465@icir.org> <80FD231E-1F61-4067-A974-237BF5B5EA38@icir.org> Message-ID: <20110909150649.GP9465@icir.org> On Fri, Sep 09, 2011 at 10:58 -0400, you wrote: > Jon handled this in several places are there some he missed? Which > tests are you having trouble with? The tests running on traces in external/*. What's the trick to make them ignore the differences? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Fri Sep 9 08:09:20 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Fri, 9 Sep 2011 10:09:20 -0500 Subject: [Bro-Dev] tests and libmagic (Re: Updating testing/external) In-Reply-To: <80FD231E-1F61-4067-A974-237BF5B5EA38@icir.org> References: <20110908175316.GO71984@icir.org> <20110909144344.GA16506@icir.org> <20110909144951.GO9465@icir.org> <80FD231E-1F61-4067-A974-237BF5B5EA38@icir.org> Message-ID: <4BEF5536-A176-432D-9726-FC6F4E06A50A@ncsa.illinois.edu> > Jon handled this in several places are there some he missed? Which tests are you having trouble with? Could be; I just tried to make it work for some of the testing/btest unit tests that I caught using it, but I didn't try to address the problem for testing/external. The way I did it for the testing/btest tests was to use filters to either filter mime types out completely if the test doesn't depend on it, or if it does, to again use a filter to normalize mime types to some constant dummy value. That approach might not be so scalable to do in a generalized way for testing/external. - Jon From jsiwek at ncsa.illinois.edu Fri Sep 9 08:15:31 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Fri, 9 Sep 2011 10:15:31 -0500 Subject: [Bro-Dev] tests and libmagic (Re: Updating testing/external) In-Reply-To: <20110909150649.GP9465@icir.org> References: <20110908175316.GO71984@icir.org> <20110909144344.GA16506@icir.org> <20110909144951.GO9465@icir.org> <80FD231E-1F61-4067-A974-237BF5B5EA38@icir.org> <20110909150649.GP9465@icir.org> Message-ID: <22FA6127-9FAA-4866-8BB1-22B0BA83A19B@ncsa.illinois.edu> > The tests running on traces in external/*. What's the trick to make > them ignore the differences? testing/btest/scripts/base/protocols/irc/dcc-extract.test has an example of what I did to normalize mime type for unit tests. Maybe it's easy enough to brute-force the same filtering approach for now if the number of logs/fields that depend on libmagic is small. - Jon From bro at tracker.bro-ids.org Fri Sep 9 08:18:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 09 Sep 2011 15:18:37 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.e6ff0af0efe0f8b69f6279e5444c07e3@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by gregor): > This may be fixed now with commit > 4931aa815f5de7f0dac31e4ee526c69c56b4961d. Feedback regarding memory > stability would be greatly appreciated. Hopefully we can get this ticket > closed now. Even with the current master the problem persists. cu gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From gc355804 at ohio.edu Fri Sep 9 09:56:21 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Fri, 09 Sep 2011 12:56:21 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/parallel-btest: Adding group "comm" to communication tests for parallelizing execution with new btest version. (32cb6d6) In-Reply-To: <20110909123244.GH9465@icir.org> References: <201109081613.p88GD4m4024259@bro-ids.icir.org> <4E6926A5.9090006@ohio.edu> <20110909123244.GH9465@icir.org> Message-ID: <4E6A4535.7060709@ohio.edu> On 9/9/2011 8:32 AM, Robin Sommer wrote: > On Thu, Sep 08, 2011 at 13:33 -0700, you wrote: > >> It seems like TEST-GROUP is very similar to TEST-PROFILE (in the btest >> profile branch I'm working on). . > Does TEST-PROFILE limit a test to run only when a specific profile has > been selected, and be skipped if not? At the moment, no. It doesn't seem like something that would be hard to do, though. > What's a use case for using it? The primary use-case at the moment is log testing. For example, if there were a 'dataseries' profile and an 'sql' profile, like so: @TEST-PROFILE dataseries sql the test containing this directive would run three times: * once under the 'default' profile * once under the 'dataseries' profile * once under the 'sql' profile This way, we get to re-use a subset of existing tests to exercise alternative logging formats. > Btw, another question: the "transform" pass runs after the test. Can > we add another similar one that runs before the test executes? We > could then remove BTest's filters and use profiles instead I believe. Sure. > And: your README says "four scripts need to be defined": please make > them optional so that one can skip scripts that aren't needed for a > profile. > Okay; in the absence of the 'supported' script, would it make sense to assume that the profile should always run? --Gilbert From robin at icir.org Fri Sep 9 10:07:58 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 9 Sep 2011 10:07:58 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/parallel-btest: Adding group "comm" to communication tests for parallelizing execution with new btest version. (32cb6d6) In-Reply-To: <4E6A4535.7060709@ohio.edu> References: <201109081613.p88GD4m4024259@bro-ids.icir.org> <4E6926A5.9090006@ohio.edu> <20110909123244.GH9465@icir.org> <4E6A4535.7060709@ohio.edu> Message-ID: <20110909170758.GQ9465@icir.org> On Fri, Sep 09, 2011 at 12:56 -0400, you wrote: > @TEST-PROFILE dataseries sql Ah, then I misunderstood how you trigger activating a profile. I was assuming there would be something like a command line argument to run all tests with, say, profile "dataseries". Wouldn't that be better? Then one wouldn't need to add a line to pretty much all tests (nor touch them all when adding a new profile). > Okay; in the absence of the 'supported' script, would it make sense to > assume that the profile should always run? Yes, I think so. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From gc355804 at ohio.edu Fri Sep 9 10:32:18 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Fri, 09 Sep 2011 13:32:18 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/parallel-btest: Adding group "comm" to communication tests for parallelizing execution with new btest version. (32cb6d6) In-Reply-To: <20110909170758.GQ9465@icir.org> References: <201109081613.p88GD4m4024259@bro-ids.icir.org> <4E6926A5.9090006@ohio.edu> <20110909123244.GH9465@icir.org> <4E6A4535.7060709@ohio.edu> <20110909170758.GQ9465@icir.org> Message-ID: <4E6A4DA2.4070309@ohio.edu> On 9/9/2011 1:07 PM, Robin Sommer wrote: > On Fri, Sep 09, 2011 at 12:56 -0400, you wrote: > >> @TEST-PROFILE dataseries sql > Ah, then I misunderstood how you trigger activating a profile. I was > assuming there would be something like a command line argument to run > all tests with, say, profile "dataseries". Wouldn't that be better? > Then one wouldn't need to add a line to pretty much all tests (nor > touch them all when adding a new profile). > Yeah, but a lot of alternative logging targets would only really use a subset of the tests. Testing log rotation, for example, wouldn't make sense when dealing with an SQL backend. We could use groups to specify different classes of tests. . . but I'm afraid we'd run into granularity issues if we went that route (which is the reason I went with tags instead). So, how about this: in each profile directory, we add a 'tests' file. This file contains a list of all the tests corresponding to a given profile. The exception here would be the default profile, for which the tests file would list all tests that *would not* run (since e.g. some SQL-specific tests might not make sense when dealing with vanilla log files, but it would be tedious to manually update this file every time we added a new test). And yeah, you're right; there definitely needs to be a command line option to only run tests associated with a certain profile. --Gilbert From bro at tracker.bro-ids.org Fri Sep 9 11:07:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 09 Sep 2011 18:07:50 -0000 Subject: [Bro-Dev] #378: Test for problem with optional record fields. In-Reply-To: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> References: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> Message-ID: <062.aaa85ebbdb1bfbe3d691ebbb6fd14f7f@tracker.bro-ids.org> #378: Test for problem with optional record fields. --------------------------------+---------------------- Reporter: robin | Owner: robin Type: Test Case Missing | Status: accepted Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta --------------------------------+---------------------- Comment (by jsiwek): In [eb3c5ee1a4361db6fa5cec53489081232ccb28de/bro]: {{{ #!CommitTicketReference repository="bro" revision="eb3c5ee1a4361db6fa5cec53489081232ccb28de" Fix indexing for record types with optional fields Addresses #378 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 9 11:16:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 09 Sep 2011 18:16:43 -0000 Subject: [Bro-Dev] #378: Test for problem with optional record fields. In-Reply-To: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> References: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> Message-ID: <062.6248180f155fa54653ebca05b7b166de@tracker.bro-ids.org> #378: Test for problem with optional record fields. ----------------------------+---------------------- Reporter: robin | Owner: robin Type: Merge Request | Status: accepted Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------------+---------------------- Changes (by jsiwek): * type: Test Case Missing => Merge Request Comment: In `topic/jsiwek/index-opt-record` It's the fix I described in comment:2 -- I don't think there's necessarily anything wrong with the way the index value's type didn't have the &optional attribute and changing that would involve transferring attributes from the set's index type to the value's type upon assignment which seems like it might have odd side-effects. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 9 11:31:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 09 Sep 2011 18:31:35 -0000 Subject: [Bro-Dev] #557: Remove malloc.c In-Reply-To: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> References: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> Message-ID: <062.e0e307a1de4eb6abd3d5e7f624c4dc7a@tracker.bro-ids.org> #557: Remove malloc.c ----------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: beta ----------------------+------------------------ Changes (by jsiwek): * status: assigned => closed * resolution: => fixed Comment: In [bf5b5218a9debc07c8787ccb219b2e1deced15f9/bro]: {{{ #!CommitTicketReference repository="bro" revision="bf5b5218a9debc07c8787ccb219b2e1deced15f9" Remove malloc.c (closes #557). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 9 13:04:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 09 Sep 2011 20:04:20 -0000 Subject: [Bro-Dev] #600: Coercion problem with log funcs In-Reply-To: <047.51c0e18a7925a83f11c8e3221c3807d2@tracker.bro-ids.org> References: <047.51c0e18a7925a83f11c8e3221c3807d2@tracker.bro-ids.org> Message-ID: <062.7f9974a683dd1c7349959c7635023864@tracker.bro-ids.org> #600: Coercion problem with log funcs ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by jsiwek): In [7ff2a3e115780f9b788e672f7f87ab3e20c13543/bro]: {{{ #!CommitTicketReference repository="bro" revision="7ff2a3e115780f9b788e672f7f87ab3e20c13543" Fix filter path_func to allow record argument as a subset of stream's columns. This required adding the ability for RecordVal::CoerceTo functions to optionally allow orphaning fields. The default is to not allow it, but now before writing to a log, the value of the stream's columns is coerced down, if necessary, before passing it on to the filter's path_func. Addresses #600. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 9 13:05:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 09 Sep 2011 20:05:44 -0000 Subject: [Bro-Dev] #600: Coercion problem with log funcs In-Reply-To: <047.51c0e18a7925a83f11c8e3221c3807d2@tracker.bro-ids.org> References: <047.51c0e18a7925a83f11c8e3221c3807d2@tracker.bro-ids.org> Message-ID: <062.907f4c5a1aa46c614816149a6e3c37ad@tracker.bro-ids.org> #600: Coercion problem with log funcs ----------------------------+------------------------ Reporter: robin | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: How does that fix in `topic/jsiwek/path-func-record-demote` look to you? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Sep 10 20:30:44 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 11 Sep 2011 03:30:44 -0000 Subject: [Bro-Dev] #604: topic/seth/ruby - broccoli-ruby binding support in broccoli Message-ID: <046.89a7c20dfe4965d53cd3b928817e2caa@tracker.bro-ids.org> #604: topic/seth/ruby - broccoli-ruby binding support in broccoli ---------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Broccoli | Version: Keywords: | ---------------------------+-------------------- The branch topic/seth/ruby in the broccoli repository adds support for the newly updated ruby bindings to the broccoli package directly. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Sep 10 20:32:04 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 11 Sep 2011 03:32:04 -0000 Subject: [Bro-Dev] #605: cmake warning when disabling python bindings Message-ID: <046.8b41a6db267ec257b10840da1b679fb9@tracker.bro-ids.org> #605: cmake warning when disabling python bindings ----------------------+-------------------- Reporter: seth | Owner: jsiwek Type: Problem | Status: new Priority: Normal | Milestone: Component: Broccoli | Version: Keywords: | ----------------------+-------------------- When building broccoli, if the python bindings are disabled with --disable-python the following cmake warning is seen: {{{ CMake Warning: The variable, 'PY_MOD_INSTALL_DIR', specified manually, was not used during the generation. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Sep 10 20:40:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 11 Sep 2011 03:40:03 -0000 Subject: [Bro-Dev] #606: broccoli and connection records Message-ID: <046.83cc834c3a71aa4f378a2b7bec3e66a4@tracker.bro-ids.org> #606: broccoli and connection records ----------------------+----------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Component: Broccoli | Version: Keywords: | ----------------------+----------------- There seems to be a problem with broccoli and connections records. I'm not sure if the problem has to do with optional fields or deeply nested records with the same field name used within a contained record or what, but here's what I've been seeing. If you run bro with the examples/test-connection-type.bro script from the broccoli-ruby repository and run ruby with the examples/test-connection- type.rb script (with broccoli debugging enabled) you will get the output from the block below. It looks like Bro is passing the full connection record but then broccoli fails when it starts trying to parse the c$conn record, but that record should still be nil at that point so it shouldn't be trying to parse anything. {{{ 59970 1315678634.139695 bro_parser.y/105 Parsing configuration from '/usr/local/bro/etc/broccoli.conf'. 59970 1315678634.139835 /tmp/tmp/bro/aux/broccoli/src/bro_config.c/263 Selecting configuration domain, name is (null) 59970 1315678634.139867 /tmp/tmp/bro/aux/broccoli/src/bro_config.c/263 Selecting configuration domain, name is (null) 59970 1315678634.140333 /tmp/tmp/bro/aux/broccoli/src/bro_openssl.c/196 Seeding PRNG from /dev/random, using 1024 bytes. 59970 1315678634.140435 /tmp/tmp/bro/aux/broccoli/src/bro_openssl.c/277 Loading OpenSSL error strings for debugging 59970 1315678634.142115 /tmp/tmp/bro/aux/broccoli/src/bro_config.c/263 Selecting configuration domain, name is (null) 59970 1315678634.142136 /tmp/tmp/bro/aux/broccoli/src/bro_config.c/263 Selecting configuration domain, name is (null) 59970 1315678634.142142 /tmp/tmp/bro/aux/broccoli/src/bro_config.c/263 Selecting configuration domain, name is (null) 59970 1315678634.142147 /tmp/tmp/bro/aux/broccoli/src/bro_openssl.c/304 use_ssl not used and host certificate not given -- not using SSL. 59970 1315678634.142184 /tmp/tmp/bro/aux/broccoli/src/bro.c/379 Connecting to host 127.0.0.1:47758 59970 1315678634.142281 /tmp/tmp/bro/aux/broccoli/src/bro_openssl.c/610 Connection established successfully. 59970 1315678634.142303 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/676 Setting raw buffer content for message, type now 1, buffer data: 0x1002f5b60 59970 1315678634.142315 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/561 Enqueing msg of type BRO_MSG_VERSION 59970 1315678634.142326 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/565 No queue yet. 59970 1315678634.143698 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/414 >>> Attempting write of BRO_MSG_VERSION 59970 1315678634.144030 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/581 Wrote int: 8/0x00000008 59970 1315678634.144164 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/581 Wrote int: 0/0x00000000 59970 1315678634.144172 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/460 Filling raw data into buffer 59970 1315678634.144179 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/525 Serialized message sized 16 bytes. 59970 1315678634.144184 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/581 Wrote int: 16/0x00000010 59970 1315678634.144190 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/568 Message serialized. 59970 1315678634.144196 /tmp/tmp/bro/aux/broccoli/src/bro_openssl.c/710 Sending 32 bytes: 0x00 0x00 0x00 0x08 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x10 0x00 0x00 0x00 0x07 0x00 0x00 0x03 0xe8 0x00 0x00 0x00 0x14 0x00 0x00 0x00 0x00 59970 1315678634.144332 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/380 <<< Sent 0/32 bytes. 59970 1315678634.144341 /tmp/tmp/bro/aux/broccoli/src/bro_openssl.c/710 Sending 32 bytes: 0x00 0x00 0x00 0x08 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x10 0x00 0x00 0x00 0x07 0x00 0x00 0x03 0xe8 0x00 0x00 0x00 0x14 0x00 0x00 0x00 0x00 59970 1315678634.144362 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/380 <<< Sent 0/32 bytes. 59970 1315678634.144369 /tmp/tmp/bro/aux/broccoli/src/bro_openssl.c/710 Sending 32 bytes: 0x00 0x00 0x00 0x08 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x10 0x00 0x00 0x00 0x07 0x00 0x00 0x03 0xe8 0x00 0x00 0x00 0x14 0x00 0x00 0x00 0x00 59970 1315678634.144385 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/380 <<< Sent 0/32 bytes. 59970 1315678634.144391 /tmp/tmp/bro/aux/broccoli/src/bro_openssl.c/710 Sending 32 bytes: 0x00 0x00 0x00 0x08 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x10 0x00 0x00 0x00 0x07 0x00 0x00 0x03 0xe8 0x00 0x00 0x00 0x14 0x00 0x00 0x00 0x00 59970 1315678634.144415 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/380 <<< Sent 32/32 bytes. 59970 1315678634.144453 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/572 Message sent. 59970 1315678634.144463 /tmp/tmp/bro/aux/broccoli/src/bro.c/169 Phase done to peer on 0x1002f5410, self now in HANDSHAKE stage. 59970 1315678634.344511 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/176 Read 197 bytes in 1/20 rounds. 59970 1315678634.344538 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/893 ----- Attempting to extract a message 59970 1315678634.344561 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/121 We have a chunk of size 8 59970 1315678634.344567 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/968 Received MSG_VERSION 59970 1315678634.344572 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/121 We have a chunk of size 17 59970 1315678634.344579 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/1008 Protocols compatible, we speak version 7 59970 1315678634.344585 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/1018 Data formats compatible, we speak version 20 59970 1315678634.344590 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/1022 Receiver cache size set to 3000 entries. 59970 1315678634.344598 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/1026 VERSION received, on 0x7fff5fbfd8a0, peer now in HANDSHAKE stage. 59970 1315678634.344604 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/191 Consuming 33 bytes in buffer. 59970 1315678634.344613 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/893 ----- Attempting to extract a message 59970 1315678634.344618 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/121 We have a chunk of size 8 59970 1315678634.344624 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/1034 Received MSQ_SERIAL 59970 1315678634.344629 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/121 We have a chunk of size 108 59970 1315678634.344635 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 105/0x69 59970 1315678634.344641 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/256 Processing serialized ID. 59970 1315678634.344654 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678634.344661 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678634.344667 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 36865/0x9001 59970 1315678634.344682 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344690 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 16/0x00000010 59970 1315678634.344696 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'peer_description' 59970 1315678634.344702 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 2/0x02 59970 1315678634.344707 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344713 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678634.344719 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 0/0x00000000 59970 1315678634.344725 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 0/0x00000000 59970 1315678634.344730 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 0/0x00000000 59970 1315678634.344736 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344742 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344752 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678634.344758 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678634.344764 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678634.344770 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678634.344775 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35329/0x8a01 59970 1315678634.344782 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344788 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 8/0x08 59970 1315678634.344794 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 4/0x04 59970 1315678634.344800 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344806 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678634.344811 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344817 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344823 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 2 in cache. 59970 1315678634.344830 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 2 of type 0x8a01 unserialized successfully. 59970 1315678634.344837 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678634.344843 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678634.344849 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 3/0x00000003 59970 1315678634.344854 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35841/0x8c01 59970 1315678634.344861 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344867 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678634.344872 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678634.344878 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344884 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678634.344890 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 2/0x1002f5c80 from cache. 59970 1315678634.344896 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678634.344903 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344909 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 2/0x02 59970 1315678634.344915 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 3 in cache. 59970 1315678634.344921 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 3 of type 0x8c01 unserialized successfully. 59970 1315678634.344927 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678634.344932 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678634.344938 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 4/0x00000004 59970 1315678634.344944 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 34822/0x8806 59970 1315678634.344951 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344957 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678634.344962 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678634.344968 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.344974 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678634.344980 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 2/0x1002f5c80 from cache. 59970 1315678634.344986 /tmp/tmp/bro/aux/broccoli/src/bro_val.c/443 Type in val has type tags 8/4 59970 1315678634.344992 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678634.345002 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 3/0x00000003 59970 1315678634.345009 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'bro' 59970 1315678634.345015 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 4 in cache. 59970 1315678634.345021 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 4 of type 0x8806 unserialized successfully. 59970 1315678634.345027 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 1 in cache. 59970 1315678634.345033 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 1 of type 0x9001 unserialized successfully. 59970 1315678634.345038 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/260 ID read successfully. 59970 1315678634.345044 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f5ba0 has non-zero refcount, not releasing 59970 1315678634.345049 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/191 Consuming 124 bytes in buffer. 59970 1315678634.345056 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/893 ----- Attempting to extract a message 59970 1315678634.345061 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/121 We have a chunk of size 8 59970 1315678634.345067 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/1124 Received MSG_CAPS 59970 1315678634.345073 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/121 We have a chunk of size 12 59970 1315678634.345079 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/1139 Skipping capabilities interpretation 59970 1315678634.345084 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/191 Consuming 28 bytes in buffer. 59970 1315678634.345090 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/893 ----- Attempting to extract a message 59970 1315678634.345096 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/121 We have a chunk of size 8 59970 1315678634.345101 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/1077 Phase done from peer on 0x1002f5410, no sync requested, peer now in RUNNING stage. 59970 1315678634.345107 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/191 Consuming 12 bytes in buffer. 59970 1315678634.345113 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/893 ----- Attempting to extract a message 59970 1315678634.345118 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/318 Checking for 4 bytes available, but have only 0 59970 1315678634.345124 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/109 Couldn't read chunk size 59970 1315678634.345131 /tmp/tmp/bro/aux/broccoli/src/bro_event_reg.c/572 Requesting event 'test_conn' 59970 1315678634.345146 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/687 Setting request content for message, type now 3 59970 1315678634.345153 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/561 Enqueing msg of type BRO_MSG_REQUEST 59970 1315678634.345159 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/565 No queue yet. 59970 1315678634.345165 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/414 >>> Attempting write of BRO_MSG_REQUEST 59970 1315678634.345171 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/581 Wrote int: 8/0x00000008 59970 1315678634.345177 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/581 Wrote int: 0/0x00000000 59970 1315678634.345182 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/495 Filling request into buffer 59970 1315678634.345188 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/525 Serialized message sized 10 bytes. 59970 1315678634.345193 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/581 Wrote int: 10/0x0000000a 59970 1315678634.345199 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/568 Message serialized. 59970 1315678634.345205 /tmp/tmp/bro/aux/broccoli/src/bro_openssl.c/710 Sending 26 bytes: 0x00 0x00 0x00 0x08 0x08 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x0a test_conn 0x00 59970 1315678634.345241 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/380 <<< Sent 26/26 bytes. 59970 1315678634.345248 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/572 Message sent. 59970 1315678634.345256 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/676 Setting raw buffer content for message, type now 1, buffer data: 0x1002f5fd0 59970 1315678634.345263 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/561 Enqueing msg of type BRO_MSG_CAPS 59970 1315678634.345268 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/565 No queue yet. 59970 1315678634.345273 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/414 >>> Attempting write of BRO_MSG_CAPS 59970 1315678634.345279 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/581 Wrote int: 8/0x00000008 59970 1315678634.345285 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/581 Wrote int: 0/0x00000000 59970 1315678634.345291 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/460 Filling raw data into buffer 59970 1315678634.345296 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/525 Serialized message sized 12 bytes. 59970 1315678634.345302 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/581 Wrote int: 12/0x0000000c 59970 1315678634.345308 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/568 Message serialized. 59970 1315678634.345313 /tmp/tmp/bro/aux/broccoli/src/bro_openssl.c/710 Sending 28 bytes: 0x00 0x00 0x00 0x08 0x11 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x0c 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 59970 1315678634.345335 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/380 <<< Sent 28/28 bytes. 59970 1315678634.345342 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/572 Message sent. 59970 1315678634.345348 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/561 Enqueing msg of type BRO_MSG_PHASE_DONE 59970 1315678634.345354 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/565 No queue yet. 59970 1315678634.345360 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/414 >>> Attempting write of BRO_MSG_PHASE_DONE 59970 1315678634.345365 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/581 Wrote int: 8/0x00000008 59970 1315678634.345371 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/568 Message serialized. 59970 1315678634.345377 /tmp/tmp/bro/aux/broccoli/src/bro_openssl.c/710 Sending 12 bytes: 0x00 0x00 0x00 0x08 0x0e 0x00 0x00 0x00 0x00 0x00 0x00 0x00 59970 1315678634.345395 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/380 <<< Sent 12/12 bytes. 59970 1315678634.345402 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/572 Message sent. 59970 1315678634.345408 /tmp/tmp/bro/aux/broccoli/src/bro.c/226 Phase done to peer on 0x1002f5410, no sync requested, self now in RUNNING stage. 59970 1315678634.345414 /tmp/tmp/bro/aux/broccoli/src/bro.c/255 Self now in RUNNING stage. 59970 1315678634.345420 /tmp/tmp/bro/aux/broccoli/src/bro.c/104 Self already in requested state. connected 59970 1315678636.035932 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/168 Reallocating buffer 59970 1315678636.036092 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/168 Reallocating buffer 59970 1315678636.036105 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/168 Reallocating buffer 59970 1315678636.036211 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/176 Read 12942 bytes in 4/20 rounds. 59970 1315678636.036359 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/893 ----- Attempting to extract a message 59970 1315678636.036533 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/121 We have a chunk of size 8 59970 1315678636.036701 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/1034 Received MSQ_SERIAL 59970 1315678636.036868 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/121 We have a chunk of size 12926 59970 1315678636.037040 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 101/0x65 59970 1315678636.037207 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/238 Processing serialized event. 59970 1315678636.037253 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 9/0x00000009 59970 1315678636.037262 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'test_conn' 59970 1315678636.037268 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/463 Read double: 1315678636.034540 59970 1315678636.037310 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678636.037386 /tmp/tmp/bro/aux/broccoli/src/bro_event.c/194 Reading 1 arguments for event test_conn 59970 1315678636.037617 /tmp/tmp/bro/aux/broccoli/src/bro_event.c/201 Reading parameter 1 59970 1315678636.037784 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.037950 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 5/0x00000005 59970 1315678636.038117 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 34826/0x880a 59970 1315678636.038298 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038480 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.038614 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038624 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038630 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 6/0x00000006 59970 1315678636.038636 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35335/0x8a07 59970 1315678636.038649 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038656 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 18/0x12 59970 1315678636.038661 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 7/0x07 59970 1315678636.038667 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038673 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038678 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038684 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038690 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 25/0x00000019 59970 1315678636.038696 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038701 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 25/0x00000019 59970 1315678636.038711 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038718 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.038723 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038729 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038735 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 7/0x00000007 59970 1315678636.038741 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35335/0x8a07 59970 1315678636.038747 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038754 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 18/0x12 59970 1315678636.038760 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 7/0x07 59970 1315678636.038765 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038771 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038777 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038782 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038788 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 4/0x00000004 59970 1315678636.038794 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038799 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 4/0x00000004 59970 1315678636.038805 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038811 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038817 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 8/0x00000008 59970 1315678636.038823 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35841/0x8c01 59970 1315678636.038829 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038835 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.038841 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038847 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038852 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 9/0x00000009 59970 1315678636.038858 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35329/0x8a01 59970 1315678636.038864 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038871 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 13/0x0d 59970 1315678636.038877 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 5/0x05 59970 1315678636.038882 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038888 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038893 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038900 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038906 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 9 in cache. 59970 1315678636.038912 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 9 of type 0x8a01 unserialized successfully. 59970 1315678636.038919 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678636.038925 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038931 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 21/0x15 59970 1315678636.038936 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 8 in cache. 59970 1315678636.038942 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 8 of type 0x8c01 unserialized successfully. 59970 1315678636.038948 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.038954 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038959 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.038965 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 9/0x00000009 59970 1315678636.038971 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 9/0x1002f64c0 from cache. 59970 1315678636.038978 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 6/0x00000006 59970 1315678636.038984 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'orig_h' 59970 1315678636.038990 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.038996 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039001 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 10/0x0000000a 59970 1315678636.039007 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35841/0x8c01 59970 1315678636.039014 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039020 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.039026 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039031 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039037 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 11/0x0000000b 59970 1315678636.039042 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35329/0x8a01 59970 1315678636.039049 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039055 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 12/0x0c 59970 1315678636.039060 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 2/0x02 59970 1315678636.039066 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039072 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039077 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039083 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039088 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 11 in cache. 59970 1315678636.039094 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 11 of type 0x8a01 unserialized successfully. 59970 1315678636.039100 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678636.039106 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039112 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 21/0x15 59970 1315678636.039118 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 10 in cache. 59970 1315678636.039124 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 10 of type 0x8c01 unserialized successfully. 59970 1315678636.039130 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.039135 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039141 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039146 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 11/0x0000000b 59970 1315678636.039152 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 11/0x1002f6720 from cache. 59970 1315678636.039158 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 6/0x00000006 59970 1315678636.039164 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'orig_p' 59970 1315678636.039170 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039176 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039181 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 12/0x0000000c 59970 1315678636.039187 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35841/0x8c01 59970 1315678636.039194 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039199 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.039205 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039211 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039216 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 9/0x00000009 59970 1315678636.039222 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 9/0x1002f64c0 from cache. 59970 1315678636.039228 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678636.039234 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039240 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 21/0x15 59970 1315678636.039246 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 12 in cache. 59970 1315678636.039251 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 12 of type 0x8c01 unserialized successfully. 59970 1315678636.039257 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.039263 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039268 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039274 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 9/0x00000009 59970 1315678636.039280 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 9/0x1002f64c0 from cache. 59970 1315678636.039286 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 6/0x00000006 59970 1315678636.039291 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'resp_h' 59970 1315678636.039297 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039303 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039309 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 13/0x0000000d 59970 1315678636.039315 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35841/0x8c01 59970 1315678636.039321 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039327 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.039332 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039338 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039344 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 11/0x0000000b 59970 1315678636.039349 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 11/0x1002f6720 from cache. 59970 1315678636.039355 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678636.039361 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039367 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 21/0x15 59970 1315678636.039373 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 13 in cache. 59970 1315678636.039379 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 13 of type 0x8c01 unserialized successfully. 59970 1315678636.039385 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.039390 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039396 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039401 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 11/0x0000000b 59970 1315678636.039407 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 11/0x1002f6720 from cache. 59970 1315678636.039413 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 6/0x00000006 59970 1315678636.039419 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'resp_p' 59970 1315678636.039425 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039431 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 7 in cache. 59970 1315678636.039437 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 7 of type 0x8a07 unserialized successfully. 59970 1315678636.039443 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678636.039449 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'id' 59970 1315678636.039455 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039461 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.039466 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039472 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039478 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 14/0x0000000e 59970 1315678636.039483 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35335/0x8a07 59970 1315678636.039490 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039496 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 18/0x12 59970 1315678636.039502 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 7/0x07 59970 1315678636.039508 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039513 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039519 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039525 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039530 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 4/0x00000004 59970 1315678636.039536 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039542 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 4/0x00000004 59970 1315678636.039548 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039554 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.039559 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039564 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039570 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 15/0x0000000f 59970 1315678636.039576 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35329/0x8a01 59970 1315678636.039583 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039589 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 3/0x03 59970 1315678636.039595 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 2/0x02 59970 1315678636.039600 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039606 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039612 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039617 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039623 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 15 in cache. 59970 1315678636.039629 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 15 of type 0x8a01 unserialized successfully. 59970 1315678636.039635 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 4/0x00000004 59970 1315678636.039641 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'size' 59970 1315678636.039647 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039652 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.039658 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039722 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039729 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 15/0x0000000f 59970 1315678636.039735 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 15/0x1002f6d20 from cache. 59970 1315678636.039742 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 5/0x00000005 59970 1315678636.039748 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'state' 59970 1315678636.039754 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039760 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.039766 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 16/0x00000010 59970 1315678636.039772 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35841/0x8c01 59970 1315678636.039778 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.039784 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.040651 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.040660 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.040666 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 15/0x0000000f 59970 1315678636.040672 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 15/0x1002f6d20 from cache. 59970 1315678636.040679 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678636.040685 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.040691 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.040697 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 16 in cache. 59970 1315678636.040703 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 16 of type 0x8c01 unserialized successfully. 59970 1315678636.040786 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.040795 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.040800 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.040806 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 15/0x0000000f 59970 1315678636.040812 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 15/0x1002f6d20 from cache. 59970 1315678636.040818 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 8/0x00000008 59970 1315678636.040824 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'num_pkts' 59970 1315678636.040878 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.040940 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.040948 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 17/0x00000011 59970 1315678636.040954 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35841/0x8c01 59970 1315678636.040961 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.040967 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.041048 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.041056 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.041062 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 15/0x0000000f 59970 1315678636.041068 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 15/0x1002f6d20 from cache. 59970 1315678636.041074 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678636.041173 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.041184 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.041190 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 17 in cache. 59970 1315678636.041197 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 17 of type 0x8c01 unserialized successfully. 59970 1315678636.041280 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.041287 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.041293 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.041341 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 15/0x0000000f 59970 1315678636.041471 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 15/0x1002f6d20 from cache. 59970 1315678636.041542 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 12/0x0000000c 59970 1315678636.041708 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'num_bytes_ip' 59970 1315678636.041747 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.041755 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 14 in cache. 59970 1315678636.041872 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 14 of type 0x8a07 unserialized successfully. 59970 1315678636.041900 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 4/0x00000004 59970 1315678636.041951 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'orig' 59970 1315678636.042049 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042055 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.042061 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042067 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042072 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 14/0x0000000e 59970 1315678636.042079 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 14/0x1002f6c20 from cache. 59970 1315678636.042085 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 4/0x00000004 59970 1315678636.042091 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'resp' 59970 1315678636.042098 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042103 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.042109 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042114 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042120 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 18/0x00000012 59970 1315678636.042126 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35329/0x8a01 59970 1315678636.042133 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042139 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 6/0x06 59970 1315678636.042145 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 3/0x03 59970 1315678636.042151 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042156 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042162 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042167 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042173 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 18 in cache. 59970 1315678636.042179 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 18 of type 0x8a01 unserialized successfully. 59970 1315678636.042186 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 10/0x0000000a 59970 1315678636.042192 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'start_time' 59970 1315678636.042198 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042204 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.042209 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042215 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042220 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 19/0x00000013 59970 1315678636.042226 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35329/0x8a01 59970 1315678636.042233 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042239 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 7/0x07 59970 1315678636.042245 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 3/0x03 59970 1315678636.042251 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042256 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042262 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042268 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042273 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 19 in cache. 59970 1315678636.042279 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 19 of type 0x8a01 unserialized successfully. 59970 1315678636.042285 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 8/0x00000008 59970 1315678636.042291 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'duration' 59970 1315678636.042298 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042303 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.042309 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042314 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042320 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 20/0x00000014 59970 1315678636.042326 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35333/0x8a05 59970 1315678636.042333 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042340 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 16/0x10 59970 1315678636.042345 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 7/0x07 59970 1315678636.042351 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042357 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042362 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042368 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042374 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042379 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.042385 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042391 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042495 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 21/0x00000015 59970 1315678636.042503 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35330/0x8a02 59970 1315678636.042528 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042536 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 19/0x13 59970 1315678636.042542 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 7/0x07 59970 1315678636.042547 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042553 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042558 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042564 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042570 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042575 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.042834 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042846 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042852 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678636.042858 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 2/0x1002f5c80 from cache. 59970 1315678636.042864 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678636.042870 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.042876 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.042881 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.042887 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678636.042893 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 2/0x1002f5c80 from cache. 59970 1315678636.042899 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 21 in cache. 59970 1315678636.043123 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 21 of type 0x8a02 unserialized successfully. 59970 1315678636.043132 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.043137 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 20 in cache. 59970 1315678636.043143 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 20 of type 0x8a05 unserialized successfully. 59970 1315678636.043150 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 7/0x00000007 59970 1315678636.043156 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'service' 59970 1315678636.043246 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.043252 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.043258 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.043264 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.043269 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678636.043275 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 2/0x1002f5c80 from cache. 59970 1315678636.043361 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 4/0x00000004 59970 1315678636.043369 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'addl' 59970 1315678636.043375 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.043381 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.043387 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.043451 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.043461 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 15/0x0000000f 59970 1315678636.043468 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 15/0x1002f6d20 from cache. 59970 1315678636.043474 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 3/0x00000003 59970 1315678636.043480 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'hot' 59970 1315678636.043486 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.043492 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.043498 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.043503 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.043509 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678636.043514 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 2/0x1002f5c80 from cache. 59970 1315678636.043717 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 7/0x00000007 59970 1315678636.043726 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'history' 59970 1315678636.043736 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.043742 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.043747 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.043753 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.043835 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678636.043844 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 2/0x1002f5c80 from cache. 59970 1315678636.043851 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 3/0x00000003 59970 1315678636.043857 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'uid' 59970 1315678636.043863 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.043869 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.043967 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 22/0x00000016 59970 1315678636.043976 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35841/0x8c01 59970 1315678636.043986 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.043993 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.044073 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044082 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044087 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 23/0x00000017 59970 1315678636.044093 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35335/0x8a07 59970 1315678636.044101 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044275 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 18/0x12 59970 1315678636.044283 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 7/0x07 59970 1315678636.044292 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044298 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044304 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044309 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044315 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 7/0x00000007 59970 1315678636.044321 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044327 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 7/0x00000007 59970 1315678636.044333 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044340 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044345 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 24/0x00000018 59970 1315678636.044351 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35841/0x8c01 59970 1315678636.044358 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044364 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.044370 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044376 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044381 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 18/0x00000012 59970 1315678636.044387 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 18/0x1002f7260 from cache. 59970 1315678636.044394 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678636.044400 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044406 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 21/0x15 59970 1315678636.044412 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 24 in cache. 59970 1315678636.044418 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 24 of type 0x8c01 unserialized successfully. 59970 1315678636.044424 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.044430 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044435 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044441 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 18/0x00000012 59970 1315678636.044447 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 18/0x1002f7260 from cache. 59970 1315678636.044453 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678636.044459 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'ts' 59970 1315678636.044465 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044471 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044477 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 25/0x00000019 59970 1315678636.044483 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35841/0x8c01 59970 1315678636.044489 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044495 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.044500 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044506 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044512 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678636.044518 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 2/0x1002f5c80 from cache. 59970 1315678636.044523 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678636.044529 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044535 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 21/0x15 59970 1315678636.044541 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 25 in cache. 59970 1315678636.044547 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 25 of type 0x8c01 unserialized successfully. 59970 1315678636.044553 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.044558 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044564 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044569 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678636.044575 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 2/0x1002f5c80 from cache. 59970 1315678636.044581 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 3/0x00000003 59970 1315678636.044587 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'uid' 59970 1315678636.044593 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044599 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044604 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 26/0x0000001a 59970 1315678636.044610 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35841/0x8c01 59970 1315678636.044617 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044623 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.044628 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044634 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044640 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 7/0x00000007 59970 1315678636.044646 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 7/0x1002f6300 from cache. 59970 1315678636.044651 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 1/0x00000001 59970 1315678636.044657 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044663 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 21/0x15 59970 1315678636.044669 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/442 Storing object 26 in cache. 59970 1315678636.044675 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/448 Object 26 of type 0x8c01 unserialized successfully. 59970 1315678636.044681 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.044686 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044691 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044697 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 7/0x00000007 59970 1315678636.044703 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/400 Returning object 7/0x1002f6300 from cache. 59970 1315678636.044709 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 2/0x00000002 59970 1315678636.044715 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/448 Read string: 'id' 59970 1315678636.044721 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044727 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044732 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 27/0x0000001b 59970 1315678636.044738 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35841/0x8c01 59970 1315678636.044744 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 0/0x00 59970 1315678636.044750 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/357 Unserializing a type, checking for name-only format. 59970 1315678636.044756 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044761 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/412 Read char: 1/0x01 59970 1315678636.044767 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/476 Read int: 28/0x0000001c 59970 1315678636.044773 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/502 Read short: 35338/0x8a0a 59970 1315678636.044779 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/109 Creation of object type 0x8a0a failed. 59970 1315678636.044784 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/431 Reading object 27 of type 0x8c01 FAILED. 59970 1315678636.044791 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/431 Reading object 23 of type 0x8a07 FAILED. 59970 1315678636.044797 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f7260 has non-zero refcount, not releasing 59970 1315678636.044803 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f7aa0 has non-zero refcount, not releasing 59970 1315678636.044809 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f5c80 has non-zero refcount, not releasing 59970 1315678636.044815 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f7c10 has non-zero refcount, not releasing 59970 1315678636.044821 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f6300 has non-zero refcount, not releasing 59970 1315678636.044827 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f7d80 has non-zero refcount, not releasing 59970 1315678636.044833 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/431 Reading object 22 of type 0x8c01 FAILED. 59970 1315678636.044840 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/431 Reading object 6 of type 0x8a07 FAILED. 59970 1315678636.044846 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f6300 has non-zero refcount, not releasing 59970 1315678636.044853 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f6c20 has non-zero refcount, not releasing 59970 1315678636.044859 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f6c20 has non-zero refcount, not releasing 59970 1315678636.044865 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f7260 has non-zero refcount, not releasing 59970 1315678636.044872 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f73b0 has non-zero refcount, not releasing 59970 1315678636.044879 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f7500 has non-zero refcount, not releasing 59970 1315678636.044885 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f5c80 has non-zero refcount, not releasing 59970 1315678636.044892 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f6d20 has non-zero refcount, not releasing 59970 1315678636.044899 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f5c80 has non-zero refcount, not releasing 59970 1315678636.044906 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/126 Object 0x1002f5c80 has non-zero refcount, not releasing 59970 1315678636.044913 /tmp/tmp/bro/aux/broccoli/src/bro_sobject.c/431 Reading object 5 of type 0x880a FAILED. 59970 1315678636.044920 /tmp/tmp/bro/aux/broccoli/src/bro_event.c/204 Couldn't read parameter val 1. 59970 1315678636.044928 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/211 Skipping 4 + 12926 59970 1315678636.044934 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/191 Consuming 12942 bytes in buffer. 59970 1315678636.044950 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/893 ----- Attempting to extract a message 59970 1315678636.044956 /tmp/tmp/bro/aux/broccoli/src/bro_buf.c/318 Checking for 4 bytes available, but have only 0 59970 1315678636.044962 /tmp/tmp/bro/aux/broccoli/src/bro_io.c/109 Couldn't read chunk size }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 11 05:49:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 11 Sep 2011 12:49:48 -0000 Subject: [Bro-Dev] #607: New segfault discovered when implementing duplicate notice suppression Message-ID: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> #607: New segfault discovered when implementing duplicate notice suppression ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Keywords: beta | ---------------------+-------------------- In the branch topic/seth/notice-suppression I discovered a new segfault. The testing/btest/scripts/base/frameworks/notice/suppression.bro script exhibits the problem. It's seems to be related to timers expiring at shutdown time that are trying to delete data. Here's a backtrace: {{{ Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000 0x00000001000049df in BaseList::operator[] (this=0x1003e4be0, i=0) at List.h:78 78 return entry[i]; (gdb) bt #0 0x00000001000049df in BaseList::operator[] (this=0x1003e4be0, i=0) at List.h:78 #1 0x000000010001a199 in ScopePList::operator[] (this=0x1003e4be0, i=0) at BroList.h:47 #2 0x000000010001a17c in global_scope () at /tmp/tmp/bro/src/Scope.cc:222 #3 0x00000001000251d5 in MutableVal::~MutableVal (this=0x101dc06c0) at /tmp/tmp/bro/src/Val.cc:648 #4 0x0000000100245b65 in RecordVal::~RecordVal (this=0x101dc06c0) at /tmp/tmp/bro/src/Val.cc:2807 #5 0x0000000100003c9d in Unref (o=0x101dc06c0) at Obj.h:219 #6 0x000000010010abc1 in EventMgr::~EventMgr (this=0x1003e59c0) at /tmp/tmp/bro/src/Event.cc:77 #7 0x000000010026406b in __tcf_1 () at /tmp/tmp/bro/src/Event.cc:12 #8 0x00007fff81952374 in __cxa_finalize () #9 0x00007fff8195228c in exit () #10 0x000000010002969f in start () at iostream:77 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 12 10:51:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 12 Sep 2011 17:51:45 -0000 Subject: [Bro-Dev] #608: broctl print times out if the table is too big Message-ID: <049.8e88348ff12d18f23c9143d4a84c4f7d@tracker.bro-ids.org> #608: broctl print times out if the table is too big ---------------------+--------------------- Reporter: aashish | Type: Problem Status: new | Priority: Normal Milestone: | Component: Bro Version: 1.5.4 | Keywords: ---------------------+--------------------- $ broctl print Drop::drop_info bro -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 12 11:21:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 12 Sep 2011 18:21:24 -0000 Subject: [Bro-Dev] #607: New segfault discovered when implementing duplicate notice suppression In-Reply-To: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> References: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> Message-ID: <061.111ee76ad94eb881fd9b706576b61a3c@tracker.bro-ids.org> #607: New segfault discovered when implementing duplicate notice suppression ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by jsiwek): In [a267c30fcdc48a602a57caaa3cf723009aad34aa/bro]: {{{ #!CommitTicketReference repository="bro" revision="a267c30fcdc48a602a57caaa3cf723009aad34aa" Fix crash on exit (addresses #607). MutableVal's destructor now checks if the global scope still exists before trying to remove an identifier from it. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 12 11:23:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 12 Sep 2011 18:23:32 -0000 Subject: [Bro-Dev] #607: New segfault discovered when implementing duplicate notice suppression In-Reply-To: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> References: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> Message-ID: <061.c27cf57b856e4dcca5094cfe78c84d51@tracker.bro-ids.org> #607: New segfault discovered when implementing duplicate notice suppression ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by jsiwek): That fix was committed to the `topic/seth/notice-suppression` branch, but the unit test needs a baseline added (and some other unit test baselines need to be updated). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 12 12:14:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 12 Sep 2011 19:14:14 -0000 Subject: [Bro-Dev] #609: Unit tests need to pass Message-ID: <047.430e076354ee4907df9d41917a8ed78b@tracker.bro-ids.org> #609: Unit tests need to pass ---------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: beta | ---------------------+-------------------- Just for the record, not all unit tests pass yet. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 12 12:17:39 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 12 Sep 2011 19:17:39 -0000 Subject: [Bro-Dev] #585: LogWriterAscii can write NULL bytes In-Reply-To: <048.d5296b9ea016f9129205b1fc4b0eccbe@tracker.bro-ids.org> References: <048.d5296b9ea016f9129205b1fc4b0eccbe@tracker.bro-ids.org> Message-ID: <063.c6c94195226a11e743b395e140af76e3@tracker.bro-ids.org> #585: LogWriterAscii can write NULL bytes ----------------------+------------------------ Reporter: gregor | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: BETA ----------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [83783c5ca75dc94e6c8b12fc6c3ee318862d292d/bro]: {{{ #!CommitTicketReference repository="bro" revision="83783c5ca75dc94e6c8b12fc6c3ee318862d292d" Bugfix for log writer. It didn't escape binary stuff in some situations. Closes #585. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 12 12:31:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 12 Sep 2011 19:31:33 -0000 Subject: [Bro-Dev] #610: topic/seth/syslog-analyzer-updates - Updates for syslog analyzer Message-ID: <046.cb3dee1a160d6374e0f15924076c0111@tracker.bro-ids.org> #610: topic/seth/syslog-analyzer-updates - Updates for syslog analyzer ---------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: beta | ---------------------------+-------------------- - Supports "Octet Stuffing" mode for Syslog over TCP (untested!). If someone has a tracefile with TCP syslog, I'd appreciate getting a few packets. - DPD support for syslog. Calls ProtocolConfirmation when detected and includes signatures for UDP and TCP syslog. - Removing newlines and nulls from EOL when syslog implementation has included those in the actual message. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 12 13:12:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 12 Sep 2011 20:12:12 -0000 Subject: [Bro-Dev] #610: topic/seth/syslog-analyzer-updates - Updates for syslog analyzer In-Reply-To: <046.cb3dee1a160d6374e0f15924076c0111@tracker.bro-ids.org> References: <046.cb3dee1a160d6374e0f15924076c0111@tracker.bro-ids.org> Message-ID: <061.89d544572b306a90cfaaf52751e1e35b@tracker.bro-ids.org> #610: topic/seth/syslog-analyzer-updates - Updates for syslog analyzer ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Comment (by robin): On Mon, Sep 12, 2011 at 19:31 -0000, you wrote: > - Supports "Octet Stuffing" mode for Syslog over TCP (untested!). This parts makes me uneasy. Let's try to test that before merging ... -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 12 13:17:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 12 Sep 2011 20:17:54 -0000 Subject: [Bro-Dev] #610: topic/seth/syslog-analyzer-updates - Updates for syslog analyzer In-Reply-To: <046.cb3dee1a160d6374e0f15924076c0111@tracker.bro-ids.org> References: <046.cb3dee1a160d6374e0f15924076c0111@tracker.bro-ids.org> Message-ID: <061.cd6bd54b164500c2cd26d0abe74af5bd@tracker.bro-ids.org> #610: topic/seth/syslog-analyzer-updates - Updates for syslog analyzer ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Comment (by seth): > This parts makes me uneasy. Let's try to test that before merging ... I'm getting a tracefile right now but I can't use it in a test. We'll have to get another tracefile for the test suite. Anyone want to generate one? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 12 13:42:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 12 Sep 2011 20:42:10 -0000 Subject: [Bro-Dev] #604: topic/seth/ruby - broccoli-ruby binding support in broccoli In-Reply-To: <046.89a7c20dfe4965d53cd3b928817e2caa@tracker.bro-ids.org> References: <046.89a7c20dfe4965d53cd3b928817e2caa@tracker.bro-ids.org> Message-ID: <061.e3463cc0dfa68aa9078f7720baac5a09@tracker.bro-ids.org> #604: topic/seth/ruby - broccoli-ruby binding support in broccoli ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Broccoli | Version: Resolution: | Keywords: ----------------------------+-------------------- Comment (by jsiwek): There's also a commit in `topic/jsiwek/ruby` of the `bro` repo that can be merged with this. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 08:24:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 13 Sep 2011 15:24:05 -0000 Subject: [Bro-Dev] #587: topic/jsiwek/cmake-rpath In-Reply-To: <048.aee02a39621ffba5ac8cc08b74fbd84e@tracker.bro-ids.org> References: <048.aee02a39621ffba5ac8cc08b74fbd84e@tracker.bro-ids.org> Message-ID: <063.314ff9386fffa990893801e56520412f@tracker.bro-ids.org> #587: topic/jsiwek/cmake-rpath -----------------------------+------------------------ Reporter: jsiwek | Owner: seth Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: assigned => closed * resolution: => Solved/Applied Comment: I applied this to broccoli-ruby. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 08:25:12 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 13 Sep 2011 15:25:12 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.e58a0ca7c920f464c676230a09e71270@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by seth): I committed another test fix to master this morning and it looks like it's working very well in live traffic right now. Gregor, could you test it too? Maybe we can get this ticket closed! -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 08:28:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 13 Sep 2011 15:28:29 -0000 Subject: [Bro-Dev] #487: Bug and missing feature in syslog analyzer In-Reply-To: <046.39317bf8c8ad376fb1f4afd399768ef7@tracker.bro-ids.org> References: <046.39317bf8c8ad376fb1f4afd399768ef7@tracker.bro-ids.org> Message-ID: <061.b5a485e90dd7767331f6afbe543f1199@tracker.bro-ids.org> #487: Bug and missing feature in syslog analyzer ----------------------+------------------------ Reporter: seth | Owner: seth Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by seth): This is addressed in the merge request in ticket #610 -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 08:29:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 13 Sep 2011 15:29:23 -0000 Subject: [Bro-Dev] #597: net_init and bro_init In-Reply-To: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> References: <046.ee31377937cd857ee1474df4d4758e41@tracker.bro-ids.org> Message-ID: <061.88f03c90c6ed14fa01fe0a4c32b64cac@tracker.bro-ids.org> #597: net_init and bro_init -----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Rejected | Keywords: beta -----------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Rejected Comment: I'm going to close this. We'll at least have the discussion archived here in the tracker still. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 08:30:08 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 13 Sep 2011 15:30:08 -0000 Subject: [Bro-Dev] #487: Bug and missing feature in syslog analyzer In-Reply-To: <046.39317bf8c8ad376fb1f4afd399768ef7@tracker.bro-ids.org> References: <046.39317bf8c8ad376fb1f4afd399768ef7@tracker.bro-ids.org> Message-ID: <061.2018ad4e1263cb6e5601f91c679b9e87@tracker.bro-ids.org> #487: Bug and missing feature in syslog analyzer -----------------------------+------------------------ Reporter: seth | Owner: seth Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 08:37:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 13 Sep 2011 15:37:14 -0000 Subject: [Bro-Dev] #568: Syslog analyzer should catch binpac exceptions In-Reply-To: <048.fa82a0608170a3c7dde2449bcee5d6d3@tracker.bro-ids.org> References: <048.fa82a0608170a3c7dde2449bcee5d6d3@tracker.bro-ids.org> Message-ID: <063.a073496a7d6d2ae2d29bba42483213dc@tracker.bro-ids.org> #568: Syslog analyzer should catch binpac exceptions -----------------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: Fixed in the topic/seth/syslog-analyzer-updates branch and will be merged into master from there so I'm closing this ticket. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 08:38:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 13 Sep 2011 15:38:28 -0000 Subject: [Bro-Dev] #588: Segfault in rotation code for logging framework In-Reply-To: <046.8c03ccb146ba869387339739e5ce79a3@tracker.bro-ids.org> References: <046.8c03ccb146ba869387339739e5ce79a3@tracker.bro-ids.org> Message-ID: <061.6d4db6e3d6588d24510a9dbf5cfea2a6@tracker.bro-ids.org> #588: Segfault in rotation code for logging framework -----------------------------+------------------------ Reporter: seth | Owner: Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: beta -----------------------------+------------------------ Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: I think we can close this ticket. I haven't seen nor heard of anyone experiencing this crash anymore. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 08:48:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 13 Sep 2011 15:48:52 -0000 Subject: [Bro-Dev] #611: Segfault in strftime BiF Message-ID: <046.220cfaf223e8017638bae915af2d1c77@tracker.bro-ids.org> #611: Segfault in strftime BiF ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: beta | ---------------------+-------------------- Here's the backtrace.... {{{ #0 0x0000003b41ca27f5 in __strftime_internal () from /lib64/libc.so.6 #1 0x0000003b41ca48b6 in strftime_l () from /lib64/libc.so.6 #2 0x000000000069d011 in BifFunc::bro_strftime (frame=0x6bd62b0, BiF_ARGS=0x6b99410) at bro.bif:2775 #3 0x000000000068d983 in BuiltinFunc::Call (this=0x27f5390, args=0x6b99410, parent=0x6bd62b0) at /bro/src/Func.cc:463 #4 0x000000000067e362 in CallExpr::Eval (this=0x27d99f0, f=0x6bd62b0) at /bro/src/Expr.cc:4649 #5 0x0000000000681b9c in eval_list (f=0x6bd62b0, l=0x27d8760) at /bro/src/Expr.cc:5488 #6 0x000000000067e2f1 in CallExpr::Eval (this=0x27d9c50, f=0x6bd62b0) at /bro/src/Expr.cc:4640 #7 0x0000000000681b9c in eval_list (f=0x6bd62b0, l=0x27d9cf0) at /bro/src/Expr.cc:5488 #8 0x000000000067e2f1 in CallExpr::Eval (this=0x27d9f20, f=0x6bd62b0) at /bro/src/Expr.cc:4640 #9 0x0000000000755b7b in ExprStmt::Exec (this=0x27d9fc0, f=0x6bd62b0, flow=@0x7fffb4876474) at /bro/src/Stmt.cc:369 #10 0x000000000075a47d in StmtList::Exec (this=0x27d76c0, f=0x6bd62b0, flow=@0x7fffb4876474) at /bro/src/Stmt.cc:1404 #11 0x000000000068cdec in BroFunc::Call (this=0x27da250, args=0x4e62bb0, parent=0x37fbbf0) at /bro/src/Func.cc:320 #12 0x000000000067e362 in CallExpr::Eval (this=0x27ed8e0, f=0x37fbbf0) at /bro/src/Expr.cc:4649 #13 0x0000000000759f7d in ReturnStmt::Exec (this=0x27ed980, f=0x37fbbf0, flow=@0x7fffb48766c4) at /bro/src/Stmt.cc:1344 #14 0x000000000075a47d in StmtList::Exec (this=0x279a110, f=0x37fbbf0, flow=@0x7fffb48766c4) at /bro/src/Stmt.cc:1404 #15 0x000000000068cdec in BroFunc::Call (this=0x27eda30, args=0x4fed380, parent=0x3804740) at /bro/src/Func.cc:320 #16 0x000000000067e362 in CallExpr::Eval (this=0x27a7f00, f=0x3804740) at /bro/src/Expr.cc:4649 #17 0x0000000000759f7d in ReturnStmt::Exec (this=0x27a7fa0, f=0x3804740, flow=@0x7fffb48769b4) at /bro/src/Stmt.cc:1344 #18 0x0000000000756405 in IfStmt::DoExec (this=0x27a80f0, f=0x3804740, v=0x4b398c0, flow=@0x7fffb48769b4) at /bro/src/Stmt.cc:484 #19 0x0000000000755baf in ExprStmt::Exec (this=0x27a80f0, f=0x3804740, flow=@0x7fffb48769b4) at /bro/src/Stmt.cc:373 #20 0x000000000075a47d in StmtList::Exec (this=0x27a7020, f=0x3804740, flow=@0x7fffb48769b4) at /bro/src/Stmt.cc:1404 #21 0x000000000068cdec in BroFunc::Call (this=0x27a81b0, args=0x7fffb4876af0, parent=0x0) at /bro/src/Func.cc:320 #22 0x00000000006d9929 in LogMgr::FinishedRotation (this=0x25dce80, writer=0x3942c20, new_name="notice-11-09-12_18.08.24.log", old_name="notice.log", open=1315868904.315407, close=5.8589466593042906e+17, terminating=false) at /bro/src/LogMgr.cc:1542 #23 0x00000000006deca0 in LogWriter::FinishedRotation (this=0x3942c20, new_name="notice-11-09-12_18.08.24.log", old_name="notice.log", open=1315868904.315407, close=5.8589466593042906e+17, terminating=false) at /bro/src/LogWriter.cc:157 #24 0x00000000006e0836 in LogWriterAscii::DoRotate (this=0x3942c20, rotated_path="notice-11-09-12_18.08.24", open=1315868904.315407, close=5.8589466593042906e+17, terminating=false) at /bro/src/LogWriterAscii.cc:303 #25 0x00000000006de918 in LogWriter::Rotate (this=0x3942c20, rotated_path="notice-11-09-12_18.08.24", open=1315868904.315407, close=5.8589466593042906e+17, terminating=false) at /bro/src/LogWriter.cc:95 #26 0x00000000006d938a in LogMgr::Rotate (this=0x25dce80, winfo=0x3a384a0) at /bro/src/LogMgr.cc:1499 #27 0x00000000006d8ce2 in RotationTimer::Dispatch (this=0x38628f0, t=5.8589466593042906e+17, is_expire=0) at /bro/src/LogMgr.cc:1409 #28 0x0000000000768266 in PQ_TimerMgr::DoAdvance (this=0x25e9b40, new_t=5.8589466593042906e+17, max_expire=300) at /bro/src/Timer.cc:164 #29 0x0000000000767ec8 in TimerMgr::Advance (this=0x25e9b40, arg_t=5.8589466593042906e+17, max_expire=300) at /bro/src/Timer.cc:107 #30 0x00000000006f0bb3 in expire_timers (src_ps=0x34eecf0) at /bro/src/Net.cc:310 #31 0x00000000006f0cc2 in net_packet_dispatch (t=1315878725.982353, hdr=0x34eed30, pkt=0x7f3d03ac5172
, hdr_size=0, src_ps=0x34eecf0, pkt_elem=0x0) at /bro/src/Net.cc:331 #32 0x00000000006f1035 in net_packet_arrival (t=1315878725.982353, hdr=0x34eed30, pkt=0x7f3d03ac5172
, hdr_size=0, src_ps=0x34eecf0) at /bro/src/Net.cc:416 #33 0x0000000000702cdf in PktSrc::Process (this=0x34eecf0) at /bro/src/PktSrc.cc:272 #34 0x00000000006f116d in net_run () at /bro/src/Net.cc:446 #35 0x00000000005ee221 in main (argc=17, argv=0x7fffb4877718) at /bro/src/main.cc:1011 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 13:31:59 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 13 Sep 2011 20:31:59 -0000 Subject: [Bro-Dev] #612: Segfault in identify_data BiF Message-ID: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> #612: Segfault in identify_data BiF ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: beta | ---------------------+------------------------ I had a crash in libmagic. This is very likely a very rare crash, since I had Bro running for quite a while and it's the first time I've seen a crash like that. Unfortunately, I can't get the arguments in the call to magic_buffer(). {{{ #0 0x0000000801631007 in memchr () from /lib/libc.so.7 #1 0x0000000800da08bc in ?? () from /usr/lib/libmagic.so.4 #2 0x0000000800da1da4 in file_softmagic () from /usr/lib/libmagic.so.4 #3 0x0000000800da196f in file_softmagic () from /usr/lib/libmagic.so.4 #4 0x0000000800daded8 in file_ascmagic_with_encoding () from /usr/lib/libmagic.so.4 #5 0x0000000800dae6e6 in file_ascmagic () from /usr/lib/libmagic.so.4 #6 0x0000000800dacc4e in file_buffer () from /usr/lib/libmagic.so.4 #7 0x0000000800da82f5 in magic_buffer () from /usr/lib/libmagic.so.4 #8 0x0000000000508337 in BifFunc::bro_identify_data (frame=Variable "frame" is not available. ) at bro.bif:3298 #9 0x00000000004fd13e in BuiltinFunc::Call (this=0x9ab400, args=0x3ca9ad60, parent=0x34b02958) at /home/gregor/projects/bro-master/src/Func.cc:463 #10 0x00000000004dabe3 in CallExpr::Eval (this=0x107c480, f=0x34b02958) at /home/gregor/projects/bro-master/src/Expr.cc:4649 #11 0x00000000004da382 in eval_list (f=0x34b02958, l=0x107c510) at /home/gregor/projects/bro-master/src/Expr.cc:5488 #12 0x00000000004dab14 in CallExpr::Eval (this=0x1081a58, f=0x34b02958) at /home/gregor/projects/bro-master/src/Expr.cc:4640 #13 0x00000000004ead1e in IndexExpr::Eval (this=0x1081db0, f=0x34b02958) at /home/gregor/projects/bro-master/src/Expr.cc:2892 #14 0x00000000004d9980 in AssignExpr::Eval (this=0x1081e48, f=0x34b02958) at /home/gregor/projects/bro-master/src/Expr.cc:2598 #15 0x00000000005af3a8 in ExprStmt::Exec (this=0x1081f80, f=0x34b02958, flow=@0x7fffffffe2ac) at /home/gregor/projects/bro-master/src/Stmt.cc:369 #16 0x00000000005ab111 in IfStmt::DoExec (this=0x1082098, f=0x34b02958, v=Variable "v" is not available. ) at /home/gregor/projects/bro-master/src/Stmt.cc:484 #17 0x00000000005af3c5 in ExprStmt::Exec (this=0x1082098, f=0x34b02958, flow=@0x7fffffffe2ac) at /home/gregor/projects/bro-master/src/Stmt.cc:373 #18 0x00000000005abb1b in StmtList::Exec (this=0x107b258, f=0x34b02958, flow=@0x7fffffffe2ac) at /home/gregor/projects/bro-master/src/Stmt.cc:1404 #19 0x00000000004fd5ae in BroFunc::Call (this=0x8e09b8, args=0x32375080, parent=0x0) at /home/gregor/projects/bro-master/src/Func.cc:320 #20 0x00000000004b38aa in EventHandler::Call (this=0x7d4898, vl=0x32375080, no_remote=Variable "no_remote" is not available. ) at /home/gregor/projects/bro-master/src/EventHandler.cc:73 #21 0x00000000004b3157 in EventMgr::Dispatch (this=Variable "this" is not available. ) at Event.h:46 #22 0x00000000004b3288 in EventMgr::Drain (this=0x793a40) at /home/gregor/projects/bro-master/src/Event.cc:119 #23 0x0000000000559353 in net_packet_dispatch (t=1315594012.6068211, hdr=0x126d658, pkt=0x8017806aa "", hdr_size=14, src_ps=0x126d618, pkt_elem=0x0) at /home/gregor/projects/bro- master/src/Net.cc:354 #24 0x00000000005684d8 in PktSrc::Process (this=0x126d618) at /home/gregor/projects/bro-master/src/PktSrc.cc:272 #25 0x00000000005595d5 in net_run () at /home/gregor/projects/bro- master/src/Net.cc:446 #26 0x0000000000468c7a in main (argc=19326872, argv=Variable "argv" is not available. ) at /home/gregor/projects/bro-master/src/main.cc:1011 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 13:34:35 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 13 Sep 2011 20:34:35 -0000 Subject: [Bro-Dev] #612: Segfault in identify_data BiF In-Reply-To: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> References: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> Message-ID: <063.0a997aedc2710c937731d239b989ed8f@tracker.bro-ids.org> #612: Segfault in identify_data BiF ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by gregor): Note that my Bro was running without port 80 traffic. (-f "not port 80") -- Ticket URL: Bro Tracker Bro Issue Tracker From gregor at icir.org Tue Sep 13 13:38:35 2011 From: gregor at icir.org (Gregor Maier) Date: Tue, 13 Sep 2011 13:38:35 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/seth/syslog-analyzer-updates: Handling binpac parsing failures in syslog analyzer. (5e95092) In-Reply-To: <201109131536.p8DFaJ4g013787@bro-ids.icir.org> References: <201109131536.p8DFaJ4g013787@bro-ids.icir.org> Message-ID: <4E6FBF4B.2070405@icir.org> > - interp->NewData(orig, data, data + len); > + try > + { > + interp->NewData(orig, data, data + len); > + } > + catch ( const binpac::Exception& e ) > + { > + ProtocolViolation(fmt("Syslog analyzer BinPAC exception: %s", e.c_msg())); > + } > } You might want to omit adding the e.c_msg() to the ProtocolViolation(). Especially for the syslog analyzer that often a dump of the packet contents that caused the parse error and it's generally not really useful (*). (If you want to keep it for debugging, I would do an ifdef and print through a weird or such) (*) That's how I found the format string bug in Reporter.cc cu Gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From bro at tracker.bro-ids.org Tue Sep 13 15:03:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 13 Sep 2011 22:03:48 -0000 Subject: [Bro-Dev] #611: Segfault in strftime BiF In-Reply-To: <046.220cfaf223e8017638bae915af2d1c77@tracker.bro-ids.org> References: <046.220cfaf223e8017638bae915af2d1c77@tracker.bro-ids.org> Message-ID: <061.1193edae436aed978650178e806cd75e@tracker.bro-ids.org> #611: Segfault in strftime BiF ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by jsiwek): Looks like the network_time got corrupted and the main way I think that would happen is if pcap_next() is returning a bad value. Were you using the libpcap from PF_RING when this happened? Looks like there's modifications done there to extend pcap_pkthdr for nanosecond accuracy that could be going wrong somehow. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 17:22:50 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 14 Sep 2011 00:22:50 -0000 Subject: [Bro-Dev] #611: Segfault in strftime BiF In-Reply-To: <046.220cfaf223e8017638bae915af2d1c77@tracker.bro-ids.org> References: <046.220cfaf223e8017638bae915af2d1c77@tracker.bro-ids.org> Message-ID: <061.64b2090a7e38900ae22851acd5d2bed6@tracker.bro-ids.org> #611: Segfault in strftime BiF ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by seth): > Were you using the libpcap from PF_RING when this happened? Yep, that's what I was using when the crash happened. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 18:34:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 14 Sep 2011 01:34:36 -0000 Subject: [Bro-Dev] #581: Add body length from http_stat to http.log In-Reply-To: <048.563f43cc45822dc0b2b6386e3aa94732@tracker.bro-ids.org> References: <048.563f43cc45822dc0b2b6386e3aa94732@tracker.bro-ids.org> Message-ID: <063.8f8058ca22ec7d042dc41b44d831283d@tracker.bro-ids.org> #581: Add body length from http_stat to http.log ----------------------+------------------------ Reporter: gregor | Owner: seth Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: beta ----------------------+------------------------ Changes (by seth): * owner: => seth * status: new => closed * resolution: => fixed Comment: In [af6c7c8b1a2886a7eadc296f4a1c2dc0f2d3f6cb/bro]: {{{ #!CommitTicketReference repository="bro" revision="af6c7c8b1a2886a7eadc296f4a1c2dc0f2d3f6cb" HTTP body size measurement added to http log. - The value of the content-length headers has now been removed but it could be added back locally at an installation by a user. - Added fields to indicate if some parsing interruption happened during the body transfer. - Closes #581 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 19:45:11 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 14 Sep 2011 02:45:11 -0000 Subject: [Bro-Dev] #519: policy/protocols/http/headers.bro only logs client headers In-Reply-To: <046.5777f3c379efa3c38e8a08a6979a91db@tracker.bro-ids.org> References: <046.5777f3c379efa3c38e8a08a6979a91db@tracker.bro-ids.org> Message-ID: <061.91518a4e0b5bebdb9e459583a5b030a9@tracker.bro-ids.org> #519: policy/protocols/http/headers.bro only logs client headers ----------------------+-------------------- Reporter: vern | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by seth): I'm ready to implement this but I need an example of the desired output or at least a concrete example of how the data would be further post processed for some analysis. There are too many options for how to do this. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 19:46:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 14 Sep 2011 02:46:16 -0000 Subject: [Bro-Dev] #519: policy/protocols/http/headers.bro only logs client headers In-Reply-To: <046.5777f3c379efa3c38e8a08a6979a91db@tracker.bro-ids.org> References: <046.5777f3c379efa3c38e8a08a6979a91db@tracker.bro-ids.org> Message-ID: <061.ae1a5c51823eff8ec861c6f385b03fdd@tracker.bro-ids.org> #519: policy/protocols/http/headers.bro only logs client headers ----------------------+-------------------- Reporter: vern | Owner: seth Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: beta ----------------------+-------------------- Changes (by seth): * owner: => seth * status: new => closed * resolution: => fixed Comment: In [f32b567c8558121d017f45662d767fbde7c50826/bro]: {{{ #!CommitTicketReference repository="bro" revision="f32b567c8558121d017f45662d767fbde7c50826" New script for logging header names and values. - Closes #519. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 20:01:26 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 14 Sep 2011 03:01:26 -0000 Subject: [Bro-Dev] #519: policy/protocols/http/headers.bro only logs client headers In-Reply-To: <046.5777f3c379efa3c38e8a08a6979a91db@tracker.bro-ids.org> References: <046.5777f3c379efa3c38e8a08a6979a91db@tracker.bro-ids.org> Message-ID: <061.a214dd88b74514fa76399ebe48bcbb6e@tracker.bro-ids.org> #519: policy/protocols/http/headers.bro only logs client headers ----------------------+---------------------- Reporter: vern | Owner: seth Type: Problem | Status: reopened Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+---------------------- Changes (by seth): * status: closed => reopened * resolution: fixed => Comment: Crap, that commit didn't close this ticket. It was only working toward closing the ticket. Reopening. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 13 21:42:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 14 Sep 2011 04:42:49 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.d4ac07cdfb7a9e6ad0217aa794cc5fba@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by gregor): > I committed another test fix to master this morning and it looks like it's > working very well in live traffic right now. Gregor, could you test it > too? Maybe we can get this ticket closed! Didn't help :-( cu Gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 14 05:54:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 14 Sep 2011 12:54:38 -0000 Subject: [Bro-Dev] #613: Problem with LogExpireInterval setting in BroControl Message-ID: <046.6b2381d7e3479123e85b6eccb468740e@tracker.bro-ids.org> #613: Problem with LogExpireInterval setting in BroControl ------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Keywords: | ------------------------+-------------------- It doesn't seem to work. It should be deleting logs, but the logs are still there. I also wonder if we should have this turned off by default? I'd rather not have people surprised when their logs start disappearing after a month (the default setting is 30 days). This can wait to be fixed after the beta's out I think. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 14 07:16:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 14 Sep 2011 14:16:05 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.babffaa487266e5a81a3260d33b727bb@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by robin): On Wed, Sep 14, 2011 at 04:42 -0000, you wrote: > Didn't help :-( Do we have a way to reproduce the effect offline yet? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 14 07:53:54 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 14 Sep 2011 14:53:54 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.3b873f7713afcb908c95e71666ad1f98@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by seth): > Do we have a way to reproduce the effect offline yet? Nope. I've been testing lots of different settings with different trace files. I get the sense that there is some particular behavior of some SSL connections that are causing it. The thing confusing to me though is that nothing is stored outside of the connection record so it seem to me that there would have to be an extremely large number of persistent connections to be able to consume the amount of memory that people are seeing used since the *should* (right?) be deleted when the connection is deleted. I don't think I'm storing references to that data anywhere that would cause it to not be deleted. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 14 11:28:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 14 Sep 2011 18:28:20 -0000 Subject: [Bro-Dev] #599: Capstats and pf_ring In-Reply-To: <046.c573030a1b3b313b9ea9fd908254c7b1@tracker.bro-ids.org> References: <046.c573030a1b3b313b9ea9fd908254c7b1@tracker.bro-ids.org> Message-ID: <061.a36b7f1b81feb10e3cf1f14b1d97919a@tracker.bro-ids.org> #599: Capstats and pf_ring -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Capstats | Version: Resolution: Solved/Applied | Keywords: beta -----------------------------+-------------------- Changes (by jsiwek): * status: new => closed * resolution: => Solved/Applied Comment: Seth confirmed this is fixed by [0a661845e6221ddf08de273ef10b91573902d6ac/capstats]. Nonblocking mode wasn't being set on the pcap handle as it should have been, and he was running in an environment where a worker node wasn't seeing any packets on the monitored interface. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 14 21:37:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 15 Sep 2011 04:37:36 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.fa5c877eaea60a5978ac00a90c0b2c0a@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by seth): Another fix in [d656e2a8c49445beeddee3ad996aecb581901995] (which is in master). I would appreciate any feedback on it. My experience so far has been a very dramatic reduction in memory use. Maybe *now* we can close this ticket... -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 04:10:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 15 Sep 2011 11:10:23 -0000 Subject: [Bro-Dev] #614: trace-summary failure Message-ID: <046.8a57d3a3cdb4d3c1f868ec96d4a81df3@tracker.bro-ids.org> #614: trace-summary failure ---------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: trace-summary | Version: Keywords: | ---------------------------+-------------------- I got this traceback in an email today. I have no clue what caused it, I doubt anyone was messing around with anything at the time that it happened and there were no crashes or anything around the time that it happened either. {{{ Traceback (most recent call last): File "/usr/local/bro/bin/trace-summary", line 854, in readConnSummaries(file) File "/usr/local/bro/bin/trace-summary", line 451, in readConnSummaries parseConnLine(line) File "/usr/local/bro/bin/trace-summary", line 512, in parseConnLine time = float(f[ts_idx]) UnboundLocalError: local variable 'ts_idx' referenced before assignment 0:00.07 real, 0.05 user, 0.00 sys, 0K total memory }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 04:27:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 15 Sep 2011 11:27:49 -0000 Subject: [Bro-Dev] #30: Drop logic doesn't pass reason to external script In-Reply-To: <057.0ad45d6f5bf6892f3fcea8ee22660335@tracker.bro-ids.org> References: <057.0ad45d6f5bf6892f3fcea8ee22660335@tracker.bro-ids.org> Message-ID: <072.2a252b21c83c1ee7d11fa6354a7f0d1b@tracker.bro-ids.org> #30: Drop logic doesn't pass reason to external script -----------------------+------------------------------- Reporter: rreitz@? | Owner: Type: Problem | Status: seen Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: branch-robin-work Resolution: | Keywords: -----------------------+------------------------------- Changes (by seth): * milestone: Bro1.6 => Bro1.7 Comment: Next release isn't going to have the code to react to things (react framework) so I'm bumping this back. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 07:50:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 15 Sep 2011 14:50:15 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.f90e7c5e26011f9b8aa571987e8f2b35@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by gregor): > Another fix in [d656e2a8c49445beeddee3ad996aecb581901995] (which is in > master). I would appreciate any feedback on it. My experience so far has > been a very dramatic reduction in memory use. Maybe *now* we can close > this ticket... Testing it right now. Will report once traffic ramps up for the day. However, it looks to me that the store is only ever access by a script in *policy*. So if only run with default scripts, that shouldn't make any difference, right? Yesterday (before the commit) I was also running my bro's without the normal ssl scripts (bare mode with everything but SSL) and just added a single ssl event so that the ssl analyzer runs but non of the default scripts for it. I could still see the memory usage, which would indicate that it's a problem in the core. From earlier test I'm also quite sure that it's not a real leak but rather large per connection memory usage that's freed after a connection finishes. Note though that all the fixes so far have helped a bit, but there's still more left that's causing memory usage. cu Gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 08:14:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 15 Sep 2011 15:14:24 -0000 Subject: [Bro-Dev] #614: trace-summary failure In-Reply-To: <046.8a57d3a3cdb4d3c1f868ec96d4a81df3@tracker.bro-ids.org> References: <046.8a57d3a3cdb4d3c1f868ec96d4a81df3@tracker.bro-ids.org> Message-ID: <061.cd5e5f54ef8258ec94681595352f737e@tracker.bro-ids.org> #614: trace-summary failure ----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: trace-summary | Version: Resolution: | Keywords: ----------------------------+-------------------- Comment (by jsiwek): See #602. This looks like it's fixed in trace-summary master branch for me, but just the broctl submodule for it needs to be updated. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 11:59:36 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 15 Sep 2011 18:59:36 -0000 Subject: [Bro-Dev] #615: Cluster manager crash Message-ID: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> #615: Cluster manager crash ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: beta | ---------------------+-------------------- Here is the backtrace... {{{ #0 0x00000035d5832905 in raise () from /lib64/libc.so.6 #1 0x00000035d58340e5 in abort () from /lib64/libc.so.6 #2 0x00000000005ed354 in SerializationFormat::ReadData (this=0x7fffc245db80, b=, count=4) at /bro/src/SerializationFormat.cc:63 #3 0x00000000005ed390 in BinarySerializationFormat::Read (this=0x7fffc245db80, str=0x7fffc245db40, len=0x7fffc245db48, tag=) at /bro/src/SerializationFormat.cc:186 #4 0x00000000005ed66c in BinarySerializationFormat::Read (this=, v=0x7fffc245dbc0, tag=) at /bro/src/SerializationFormat.cc:226 #5 0x00000000005d8008 in RemoteSerializer::ProcessLogWrite (this=0x25feb10) at /bro/src/RemoteSerializer.cc:2676 #6 0x00000000005dad32 in RemoteSerializer::Poll (this=0x25feb10, may_block=false) at /bro/src/RemoteSerializer.cc:1539 #7 0x00000000005dae5b in RemoteSerializer::Poll (this=0x25feb10, may_block=false) at /bro/src/RemoteSerializer.cc:1516 #8 0x00000000005db403 in RemoteSerializer::NextTimestamp (this=0x25feb10, local_network_time=0x7fffc245deb8) at /bro/src/RemoteSerializer.cc:1339 #9 0x0000000000598a49 in IOSourceRegistry::FindSoonest (this=0x8d8d50, ts=0x7fffc245df18) at /bro/src/IOSource.cc:63 #10 0x00000000005b7912 in net_run () at /bro/src/Net.cc:427 #11 0x00000000004f4d39 in main (argc=, argv=) at /bro/src/main.cc:1011 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 12:00:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 15 Sep 2011 19:00:13 -0000 Subject: [Bro-Dev] #615: Cluster manager crash In-Reply-To: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> References: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> Message-ID: <061.8e4358b17060e2e2278f3d93140d2dab@tracker.bro-ids.org> #615: Cluster manager crash ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Changes (by seth): * priority: Normal => High -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 14:12:17 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 15 Sep 2011 21:12:17 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.a3ef0023c446690ae58e1e02367677f1@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by jsiwek): In [8697615b81635ae1039b6c5887a665996e4c4d36/bro]: {{{ #!CommitTicketReference repository="bro" revision="8697615b81635ae1039b6c5887a665996e4c4d36" Memory leak fixes. Both related to Val lists constructed as arguments to events that were not freed because the event function was never called (e.g. no handlers). Addresses #574 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 14:20:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 15 Sep 2011 21:20:33 -0000 Subject: [Bro-Dev] #616: topic/jsiwek/leak-fixes Message-ID: <048.ef681d1f8461c6a6f4cb3f562fdabd24@tracker.bro-ids.org> #616: topic/jsiwek/leak-fixes ---------------------------+------------------------ Reporter: jsiwek | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------------+------------------------ The leaks were observed when running like `bro -b -i eth0 base/protocols/ssl`, easily repeatable by replaying traffic. May be related to #574 -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 14:26:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 15 Sep 2011 21:26:21 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.3dc83a44e0488060eefdcc070fa11ee2@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by jsiwek): > Yesterday (before the commit) I was also running my bro's without the normal ssl scripts (bare mode with everything but SSL) I definitely found some leaks running in bare-mode that didn't show up in the default mode because it's related to event handler arguments not getting unref'd because no handlers are called. I fixed everything valgrind showed me so far in `topic/jsiwek/leak-fixes`. Want to check that out and see if it helps? -- Ticket URL: Bro Tracker Bro Issue Tracker From gc355804 at ohio.edu Thu Sep 15 16:28:28 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 15 Sep 2011 19:28:28 -0400 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <063.3dc83a44e0488060eefdcc070fa11ee2@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> <063.3dc83a44e0488060eefdcc070fa11ee2@tracker.bro-ids.org> Message-ID: <4E728A1C.1050805@ohio.edu> I've been thinking about building a library interposer (in the spirit of http://developers.sun.com/solaris/articles/lib_interposers.html) that will track malloc / realloc / calloc by script line (pull the top frame out of g_frame_stack) and script file; I envision using SIGUSR1 and SIGUSR2 to dump and clear gathered statistics, respectively. I tried to do something similar with systemtap, but I wasn't able to keep bro running fast enough to process nontrivial volumes of live traffic. Haven't tried dtrace yet. If this sounds like something that'd be useful here, I can inject it into the top of the TODO. --Gilbert From bro at tracker.bro-ids.org Thu Sep 15 16:36:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 15 Sep 2011 23:36:21 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.0e5da872801fdcb96c864e8618a9c652@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by gregor): > I definitely found some leaks running in bare-mode that didn't show up in > the default mode because it's related to event handler arguments not > getting unref'd because no handlers are called. I fixed everything > valgrind showed me so far in `topic/jsiwek/leak-fixes`. Want to check > that out and see if it helps? That's not what causes my problems, because I see them in both normal and bare mode. I justed checked that. In general the C++ code should always check whether there's an event handler for a particular event before allocating val_list for it (both, to prevent leaks and to not waste time allocating stuff we never need). If it doesn't that's a bug. Do you know which event were causing the problem? We should try to fix the problem there too. cu gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 16:56:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 15 Sep 2011 23:56:13 -0000 Subject: [Bro-Dev] #610: topic/seth/syslog-analyzer-updates - Updates for syslog analyzer In-Reply-To: <046.cb3dee1a160d6374e0f15924076c0111@tracker.bro-ids.org> References: <046.cb3dee1a160d6374e0f15924076c0111@tracker.bro-ids.org> Message-ID: <061.ee32ae4c4e2bd7b51faeb840b7b98274@tracker.bro-ids.org> #610: topic/seth/syslog-analyzer-updates - Updates for syslog analyzer ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Comment (by robin): {{{ static bool Available() { return syslog_message && 1; } }}} I'm assuming the `&& 1` is for testing only, removing it. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 17:10:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 00:10:00 -0000 Subject: [Bro-Dev] #610: topic/seth/syslog-analyzer-updates - Updates for syslog analyzer In-Reply-To: <046.cb3dee1a160d6374e0f15924076c0111@tracker.bro-ids.org> References: <046.cb3dee1a160d6374e0f15924076c0111@tracker.bro-ids.org> Message-ID: <061.18fb539e476e5a3a8d699a6913776886@tracker.bro-ids.org> #610: topic/seth/syslog-analyzer-updates - Updates for syslog analyzer ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Comment (by robin): I'm not really happy with this code: - in `syslog-analyzer.pac`, the loop cutting of characters at the end looks overly complex and potentially dangerous (if length is zero). I'd fix it but without a trace, that's hard. - `syslog-tcp.pac` much of the code either isn't used or commented out (because "we are punting" I think). Can we get this in shape and implement both cases? I think you said there's a binpac problem, is that still the problem? Any workaround? - let's rename `AnalyzerTag::SYSLOG` to `AnalyzerTag::SYSLOG_UDP` (easy to do; mentioning here for reference) - I think we really need a trace for the test-suite here before we merge it in. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 17:10:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 00:10:47 -0000 Subject: [Bro-Dev] #610: topic/seth/syslog-analyzer-updates - Updates for syslog analyzer In-Reply-To: <046.cb3dee1a160d6374e0f15924076c0111@tracker.bro-ids.org> References: <046.cb3dee1a160d6374e0f15924076c0111@tracker.bro-ids.org> Message-ID: <061.9d3227292193b6838978ec93621bf22d@tracker.bro-ids.org> #610: topic/seth/syslog-analyzer-updates - Updates for syslog analyzer ---------------------+-------------------- Reporter: seth | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ---------------------+-------------------- Changes (by robin): * type: Merge Request => Task -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Sep 15 17:25:52 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 15 Sep 2011 17:25:52 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/parallel-btest: Adding group "comm" to communication tests for parallelizing execution with new btest version. (32cb6d6) In-Reply-To: <4E6A4DA2.4070309@ohio.edu> References: <201109081613.p88GD4m4024259@bro-ids.icir.org> <4E6926A5.9090006@ohio.edu> <20110909123244.GH9465@icir.org> <4E6A4535.7060709@ohio.edu> <20110909170758.GQ9465@icir.org> <4E6A4DA2.4070309@ohio.edu> Message-ID: <20110916002552.GG88926@icir.org> On Fri, Sep 09, 2011 at 13:32 -0400, you wrote: > Yeah, but a lot of alternative logging targets would only really use a > subset of the tests. Testing log rotation, for example, wouldn't make > sense when dealing with an SQL backend. Is it more tests that use the profile, or more that don't? I'd suspect the former, which makes listing all tests to be used with a profile quite cumbersome; and also error prone, because for each new test, one would need to add it to potentially a number of profiles. > So, how about this: in each profile directory, we add a 'tests' file. > This file contains a list of all the tests corresponding to a given > profile. How about doing both: we do two files "include" and "exclude", each listing tests. If the first exists, only those are run. If the second exists, all are run except those. And if both exists, "include except exclude" is run. Then each profile can decide on its own what makes more sense. > The exception here would be the default profile, for which the tests > file would list all tests that *would not* run (since e.g. some > SQL-specific tests might not make sense when dealing with vanilla > log files, but it would be tedious to manually update this file > every time we added a new test). I think the TEST-REQUIRES command is the better way to express that a test a SQL-specific. Otherwise, the information gets separated from the test itself. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Sep 15 17:26:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 00:26:52 -0000 Subject: [Bro-Dev] #592: topic/gilbert/profiles In-Reply-To: <048.50f5253509e35787f64b7ef3adde24a0@tracker.bro-ids.org> References: <048.50f5253509e35787f64b7ef3adde24a0@tracker.bro-ids.org> Message-ID: <063.f241d9be164592f1b15b44f33136c8fa@tracker.bro-ids.org> #592: topic/gilbert/profiles ---------------------+---------------------- Reporter: gclark | Owner: gclark Type: Task | Status: assigned Priority: Low | Milestone: Component: BTest | Version: Resolution: | Keywords: ---------------------+---------------------- Changes (by robin): * type: Merge Request => Task Comment: See email thread on bro-dev for more action items. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 17:28:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 00:28:07 -0000 Subject: [Bro-Dev] #378: Test for problem with optional record fields. In-Reply-To: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> References: <047.54dcb3c24337077399e16588c73b258d@tracker.bro-ids.org> Message-ID: <062.0896963fa7b17fbd8a64ff6071dcca34@tracker.bro-ids.org> #378: Test for problem with optional record fields. ----------------------------+-------------------- Reporter: robin | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: beta ----------------------------+-------------------- Changes (by robin): * status: accepted => closed * resolution: => fixed Comment: In [8d10a76eba32a0a09b8a2770680ceeacf5885cd7/bro]: {{{ #!CommitTicketReference repository="bro" revision="8d10a76eba32a0a09b8a2770680ceeacf5885cd7" Merge remote-tracking branch 'origin/topic/jsiwek/index-opt-record' * origin/topic/jsiwek/index-opt-record: Fix indexing for record types with optional fields Closes #378. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 17:28:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 00:28:07 -0000 Subject: [Bro-Dev] #557: Remove malloc.c In-Reply-To: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> References: <047.d9416191ab6a77114a37dd25d0191abc@tracker.bro-ids.org> Message-ID: <062.a9afc372017f4cc3a25f9ed69b879926@tracker.bro-ids.org> #557: Remove malloc.c ----------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: beta ----------------------+------------------------ Comment (by robin): In [2a2edb3eb44832c174377b110ccf7020c2c341a6/bro]: {{{ #!CommitTicketReference repository="bro" revision="2a2edb3eb44832c174377b110ccf7020c2c341a6" Merge remote-tracking branch 'origin/fastpath' * origin/fastpath: Remove malloc.c (closes #557). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 17:28:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 00:28:07 -0000 Subject: [Bro-Dev] #572: topic/jsiwek/filter-rotation In-Reply-To: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> References: <048.7c794b4e182e76013c28504d097ef303@tracker.bro-ids.org> Message-ID: <063.0956336c3df6ded94fc69bf23a157aaa@tracker.bro-ids.org> #572: topic/jsiwek/filter-rotation ----------------------------+------------------------ Reporter: jsiwek | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: beta ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [62581ede3177ada0e503ab7ac00e745af5ef417b/bro]: {{{ #!CommitTicketReference repository="bro" revision="62581ede3177ada0e503ab7ac00e745af5ef417b" Merge remote-tracking branch 'origin/topic/jsiwek/filter-rotation' * origin/topic/jsiwek/filter-rotation: Remove Log::rotation_control (addresses #572). Add an optional Log::RotationControl to Log::Filter records. Closes #572. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 17:28:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 00:28:07 -0000 Subject: [Bro-Dev] #604: topic/seth/ruby - broccoli-ruby binding support in broccoli In-Reply-To: <046.89a7c20dfe4965d53cd3b928817e2caa@tracker.bro-ids.org> References: <046.89a7c20dfe4965d53cd3b928817e2caa@tracker.bro-ids.org> Message-ID: <061.443405741e4dee3932d096c26f27091e@tracker.bro-ids.org> #604: topic/seth/ruby - broccoli-ruby binding support in broccoli ----------------------------+-------------------- Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Broccoli | Version: Resolution: fixed | Keywords: ----------------------------+-------------------- Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [c3217011cf5bbde9f0e03193c9d2d93571feeea3/bro]: {{{ #!CommitTicketReference repository="bro" revision="c3217011cf5bbde9f0e03193c9d2d93571feeea3" Merge remote-tracking branch 'origin/topic/jsiwek/ruby' * origin/topic/jsiwek/ruby: Add configure options for ruby/bindings integration. Closes #604. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 17:28:08 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 00:28:08 -0000 Subject: [Bro-Dev] #616: topic/jsiwek/leak-fixes In-Reply-To: <048.ef681d1f8461c6a6f4cb3f562fdabd24@tracker.bro-ids.org> References: <048.ef681d1f8461c6a6f4cb3f562fdabd24@tracker.bro-ids.org> Message-ID: <063.f99973fcc4cfd555e092def6336e9b23@tracker.bro-ids.org> #616: topic/jsiwek/leak-fixes ----------------------------+------------------------ Reporter: jsiwek | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [622f5d695341fdf36b04fea8a6f7fec382ae0d6f/bro]: {{{ #!CommitTicketReference repository="bro" revision="622f5d695341fdf36b04fea8a6f7fec382ae0d6f" Merge remote-tracking branch 'origin/topic/jsiwek/leak-fixes' * origin/topic/jsiwek/leak-fixes: Memory leak fixes. Closes #616. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 17:28:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 00:28:07 -0000 Subject: [Bro-Dev] #600: Coercion problem with log funcs In-Reply-To: <047.51c0e18a7925a83f11c8e3221c3807d2@tracker.bro-ids.org> References: <047.51c0e18a7925a83f11c8e3221c3807d2@tracker.bro-ids.org> Message-ID: <062.21b147a57475c07d2c03a2739b59dd7b@tracker.bro-ids.org> #600: Coercion problem with log funcs ----------------------------+------------------------ Reporter: robin | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: beta ----------------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [c2e432c5fae5f451a5c6854936128d95132e0464/bro]: {{{ #!CommitTicketReference repository="bro" revision="c2e432c5fae5f451a5c6854936128d95132e0464" Merge remote-tracking branch 'origin/topic/jsiwek/path-func-record-demote' * origin/topic/jsiwek/path-func-record-demote: Fix filter path_func to allow record argument as a subset of stream's columns. Conflicts: src/LogMgr.cc Closes #600. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 17:36:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 00:36:06 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.b91e769158eb7fb88bf4694328fe5605@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): I still see the error {{{ > pwd /Users/robin/bro/master/aux/broccoli > cmake --version cmake version 2.8.3 > git checkout topic/broccoli-manual-rest > git pull > ./configure [..] > make doc ( cd build && make doc ) [ 0%] [Doxygen] Generating HTML Broccoli API docs [?] Generating file index... Generating example index... Generating file member index... [100%] Built target broccolidoc make[4]: *** No rule to make target `broccolidoc', needed by `doc/CMakeFiles/doc'. Stop. make[3]: *** [doc/CMakeFiles/doc.dir/all] Error 2 make[2]: *** [doc/CMakeFiles/doc.dir/rule] Error 2 make[1]: *** [doc] Error 2 make: *** [doc] Error 2 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 17:39:01 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 00:39:01 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.eb0e5a63f754a8a7bbb025f1879d93be@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): The Doxygen docs look good now. -- Ticket URL: Bro Tracker Bro Issue Tracker From gc355804 at ohio.edu Thu Sep 15 17:40:38 2011 From: gc355804 at ohio.edu (Gilbert Clark) Date: Thu, 15 Sep 2011 20:40:38 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] topic/robin/parallel-btest: Adding group "comm" to communication tests for parallelizing execution with new btest version. (32cb6d6) In-Reply-To: <20110916002552.GG88926@icir.org> References: <201109081613.p88GD4m4024259@bro-ids.icir.org> <4E6926A5.9090006@ohio.edu> <20110909123244.GH9465@icir.org> <4E6A4535.7060709@ohio.edu> <20110909170758.GQ9465@icir.org> <4E6A4DA2.4070309@ohio.edu> <20110916002552.GG88926@icir.org> Message-ID: <4E729B06.4050209@ohio.edu> > How about doing both: we do two files "include" and "exclude", each > listing tests. If the first exists, only those are run. If the second > exists, all are run except those. And if both exists, "include except > exclude" is run. Then each profile can decide on its own what makes > more sense. > Makes sense. > I think the TEST-REQUIRES command is the better way to express that a > test a SQL-specific. Otherwise, the information gets separated from > the test itself. Cool. --Gilbert From bro at tracker.bro-ids.org Thu Sep 15 17:59:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 00:59:42 -0000 Subject: [Bro-Dev] #617: configure errors due to ruby Message-ID: <047.fc0412592c1a1428e0ab9071b234beab@tracker.bro-ids.org> #617: configure errors due to ruby ---------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: High | Milestone: Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ With the ruby stuff merged in, master now gives me this with `configure` on a machine presumably not having the right ruby pieces installed: {{{ CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: RUBY_INCLUDE_DIR (ADVANCED) used as include directory in directory /home/robin/bro/master/aux/broccoli/bindings/broccoli-ruby RUBY_LIBRARY (ADVANCED) linked by target "broccoli_ext" in directory /home/robin/bro/master/aux/broccoli/bindings/broccoli-ruby }}} (Work-around: `--disable-ruby` works) -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Sep 15 18:35:44 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 15 Sep 2011 18:35:44 -0700 Subject: [Bro-Dev] Installation thoughts Message-ID: <20110916013544.GJ88926@icir.org> Sorry for going back to the installation process but a few thoughts about "make install" (feel free to remind me about past discussions, I don't remember these very well anymore): - It currently installs the following binaries: adtrace bdcat bro broccoli-config broctl capstats ftwire2bro nfcollector rst trace-summary That's seem too much to me, I think we should skip the bro-aux programs by default. That would then leave us with just bro broctl capstats trace-summary broccoli-config We could add target "make install-aux" to get the rest. - It creates "$prefix/logs". I don't remember whether we discussed this already, but is that a good default location for logs? - I noticed that we still install scripts that are now obsolete: bro-logchk.pl hot-report mon-report mvlog That leaves us in into $prefix/scripts with: host-grep ip-grep host-to-addrs lock_file However, I'm almost thinking we should remove these as well; not sure it's worth shipping them. Or, if we really want to keep them, at least don't install by default (then perhaps as part of the "install-aux" target from above). - Was there a reason we install the spool pieces directly into $prefix/spool/ instead of into $prefix/spool/bro/? - test-all-policy.bro ends up in share/bro, which doesn't seem right. - Do we really need to install share/bro/site/local-proxy.bro? It's empty and unlikley that many people will want to edit it. - Thoughts about local.bro: * Why is "@load protocols/http/detect-webapps" commented out? We should add a comment explaining when one would want to include it. * "Requires that the Site::local_zones variable". We should add where/how to do that. Perhaps an empty definition right in local.bro? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Sep 15 18:40:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 01:40:13 -0000 Subject: [Bro-Dev] #618: broctl check rotates logs Message-ID: <047.2c46ff0234b524e0ef0a8917a71abd60@tracker.bro-ids.org> #618: broctl check rotates logs --------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Keywords: beta | --------------------+--------------------- When I start with an empty logs/* directory, and then do "broctl check" I get this: {{{ > ls logs 2011-09-15 > ls -al logs/* total 12 drwxr-xr-x 2 robin wheel 512 Sep 15 18:37 . drwxr-xr-x 3 robin wheel 512 Sep 15 18:37 .. -rw-r--r-- 1 robin wheel 326 Sep 15 18:37 communication.18:37:07-18:37:07.gz -rw-r--r-- 1 robin wheel 1011 Sep 15 18:37 loaded_scripts.18:37:07-18:37:07.gz -rw-r--r-- 1 robin wheel 138 Sep 15 18:37 packet_filter.18:37:07-18:37:07.gz -rw-r--r-- 1 robin wheel 162 Sep 15 18:37 reporter.18:37:07-18:37:07.gz }}} Rotation should be disabled with check. I thought we already had done that, but apparently not. #component: broctl -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Sep 15 18:52:23 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 15 Sep 2011 18:52:23 -0700 Subject: [Bro-Dev] PCAP_PF_RING_CLUSTER_ID Message-ID: <20110916015223.GL88926@icir.org> With my 3 node cluster on FreeBSD, all the workers are started with PCAP_PF_RING_CLUSTER_ID=21. Where is the 21 coming from? And shouldn't we restrict this to systems that actually have PF_RING, like via a configure check? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Thu Sep 15 18:59:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 01:59:46 -0000 Subject: [Bro-Dev] #619: Log rotations seem broken Message-ID: <047.81859bf695ec52d8dd6278a9176c26bd@tracker.bro-ids.org> #619: Log rotations seem broken ------------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: BroControl | Version: git/master Keywords: beta | ------------------------+------------------------ I did a fresh cluster install and I see only some (unimportant) logs rotated/archived: {{{ > ls logs current > broctl start starting manager ... starting proxy-1 ... starting worker-01 ... starting worker-02 ... starting worker-03 ... > ls logs/current communication.log http.log notice.log reporter.log software.log stdout.log conn.log known_certs.log packet_filter.log smtp.log ssl.log syslog.log dns.log loaded_scripts.log prof.log smtp_entities.log stderr.log weird.log > broctl stop stopping manager ... stopping proxy-1 ... stopping worker-01 ... stopping worker-02 ... stopping worker-03 ... > ls logs/ 2011-09-15/ current/ > ls logs/2011-09-15/ communication.18:56:17-18:56:38.gz notice.18:56:23-18:56:38.gz reporter.18:56:17-18:56:38.gz loaded_scripts.18:56:17-18:56:38.gz packet_filter.18:56:17-18:56:38.gz }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 15 19:00:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 02:00:29 -0000 Subject: [Bro-Dev] #617: configure errors due to ruby In-Reply-To: <047.fc0412592c1a1428e0ab9071b234beab@tracker.bro-ids.org> References: <047.fc0412592c1a1428e0ab9071b234beab@tracker.bro-ids.org> Message-ID: <062.55df3a9da55704857a22a40357249bf7@tracker.bro-ids.org> #617: configure errors due to ruby ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Changes (by robin): * cc: beta (added) * milestone: => Bro1.6 -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Thu Sep 15 19:19:53 2011 From: seth at icir.org (Seth Hall) Date: Thu, 15 Sep 2011 22:19:53 -0400 Subject: [Bro-Dev] PCAP_PF_RING_CLUSTER_ID In-Reply-To: <20110916015223.GL88926@icir.org> References: <20110916015223.GL88926@icir.org> Message-ID: On Sep 15, 2011, at 9:52 PM, Robin Sommer wrote: > With my 3 node cluster on FreeBSD, all the workers are started with > PCAP_PF_RING_CLUSTER_ID=21. Where is the 21 coming from? Heh, it's "bro" typed out on a phone number pad (276) represented as an 8bit int (wrapped around). In other words, it's a fairly arbitrary number. :) > And shouldn't > we restrict this to systems that actually have PF_RING, like via a > configure check? I thought about doing that, but it seemed somewhat superfluous since those environment variables will only be used if the pf_ring libpcap wrapper is used. It does seem like the right thing to do however. We could just set the default value to "0" which will cause the environment variables to not be set but it seems like just causing more effort for users without much benefit. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Fri Sep 16 08:04:02 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 15:04:02 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.be735a0e449ba22dcfa8a38697ab4129@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by jsiwek): > In general the C++ code should always check whether there's an event handler for a particular event before allocating val_list for it (both, to prevent leaks and to not waste time allocating stuff we never need). If it doesn't that's a bug. Do you know which event were causing the problem? We should try to fix the problem there too. The ones I was seeing were `Analyzer::ProtocolViolation()` creating the `protocol_confirmation` event when the DPD framework isn't loaded and thus that event has no handler body defined. (`procotol_violation` looks like it will behave similarly). Then there was `Reporter::Weird`s constructing `val_list`s when the notice framework wasn't loaded and there were no weird event handlers. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Fri Sep 16 08:18:49 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Fri, 16 Sep 2011 10:18:49 -0500 Subject: [Bro-Dev] Installation thoughts In-Reply-To: <20110916013544.GJ88926@icir.org> References: <20110916013544.GJ88926@icir.org> Message-ID: > - I noticed that we still install scripts that are now obsolete: > > bro-logchk.pl > hot-report > mon-report > mvlog > > That leaves us in into $prefix/scripts with: > > host-grep > ip-grep > host-to-addrs > lock_file > > However, I'm almost thinking we should remove these as well; not > sure it's worth shipping them. Or, if we really want to keep them, > at least don't install by default (then perhaps as part of the > "install-aux" target from above). Check http://tracker.bro-ids.org/bro/ticket/511 I think they're removed in the `topic/dist-cleanup` branch. Maybe those branches can be merged across the repos now -- I think just the COPYING normalization task wasn't finished yet. - Jon From bro at tracker.bro-ids.org Fri Sep 16 08:36:00 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 15:36:00 -0000 Subject: [Bro-Dev] #511: Misc distribution cleanup In-Reply-To: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> References: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> Message-ID: <062.f6e019da14f20c2b77450388c9cfd052@tracker.bro-ids.org> #511: Misc distribution cleanup ----------------------------+-------------------- Reporter: robin | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Changes (by robin): * type: Task => Merge Request -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Fri Sep 16 08:36:40 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 16 Sep 2011 08:36:40 -0700 Subject: [Bro-Dev] Installation thoughts In-Reply-To: References: <20110916013544.GJ88926@icir.org> Message-ID: <20110916153640.GA19393@icir.org> On Fri, Sep 16, 2011 at 10:18 -0500, you wrote: > I think they're removed in the `topic/dist-cleanup` branch. Ah, thanks, I missed that the scripts are part of that. I'll go ahead and merge. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Fri Sep 16 11:26:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 18:26:56 -0000 Subject: [Bro-Dev] #617: configure errors due to ruby In-Reply-To: <047.fc0412592c1a1428e0ab9071b234beab@tracker.bro-ids.org> References: <047.fc0412592c1a1428e0ab9071b234beab@tracker.bro-ids.org> Message-ID: <062.52673f3319c32124875443593c2d7b4c@tracker.bro-ids.org> #617: configure errors due to ruby ----------------------------+------------------------ Reporter: robin | Owner: Type: Merge Request | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: Fixed in `topic/jsiwek/opt-ruby-bindings`, commit [359c0bf5454ce3cfa9d7dcce54b54d1f9723154b/broccoli-ruby] -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 16 12:08:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 19:08:32 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.8e7aa3f14bc58f161ca17734b6137571@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): In [5b37f39082b93e1e2a0cfa9d90749e5f9004beb3/broccoli]: {{{ #!CommitTicketReference repository="broccoli" revision="5b37f39082b93e1e2a0cfa9d90749e5f9004beb3" Work around bug in CMake custom target dependencies. Addresses #563 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 16 12:11:04 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 16 Sep 2011 19:11:04 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.cacf8b38dd7767dd2c3edd28e161acdf@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by jsiwek): > I still see the error Should be fixed now (the bug in specifying custom target dependencies is fixed in at least CMake 2.8.5, which is why I didn't see the same error). -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Fri Sep 16 17:52:56 2011 From: seth at icir.org (Seth Hall) Date: Fri, 16 Sep 2011 20:52:56 -0400 Subject: [Bro-Dev] manager-child cpu utilization? Message-ID: <6E371FAC-03B4-4EC5-B1F2-5DAC1C5719B5@icir.org> I'm seeing what appears to be extremely high CPU utilization by the manager's child process in a rather large (23 worker?) cluster. I believe the deserialization is being done by the child process and I'm guessing that the child process is being overwhelmed by all of the deserialization it's having to do for the logging framework's log records. Any one have thoughts about if I'm correct or ways this could be optimized? I'm just worried this could become a problem because logs are written much faster at other sites and I can see other sites getting communication framework overload really easily. I think it's something that we need to keep a close eye on at least. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From mcholste at gmail.com Sat Sep 17 19:17:38 2011 From: mcholste at gmail.com (Martin Holste) Date: Sat, 17 Sep 2011 21:17:38 -0500 Subject: [Bro-Dev] manager-child cpu utilization? In-Reply-To: <6E371FAC-03B4-4EC5-B1F2-5DAC1C5719B5@icir.org> References: <6E371FAC-03B4-4EC5-B1F2-5DAC1C5719B5@icir.org> Message-ID: Yep, this is definitely happening. Our cluster's manager and its child, as well as the proxy and its child chew up about 20-25% CPU each, so that's almost a full CPU for the manager/child and proxy/child. On Fri, Sep 16, 2011 at 7:52 PM, Seth Hall wrote: > I'm seeing what appears to be extremely high CPU utilization by the manager's child process in a rather large (23 worker?) cluster. ?I believe the deserialization is being done by the child process and I'm guessing that the child process is being overwhelmed by all of the deserialization it's having to do for the logging framework's log records. > > Any one have thoughts about if I'm correct or ways this could be optimized? ?I'm just worried this could become a problem because logs are written much faster at other sites and I can see other sites getting communication framework overload really easily. ?I think it's something that we need to keep a close eye on at least. > > ?.Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From scampbell at lbl.gov Sat Sep 17 21:37:56 2011 From: scampbell at lbl.gov (Scott Campbell) Date: Sat, 17 Sep 2011 23:37:56 -0500 Subject: [Bro-Dev] manager-child cpu utilization? In-Reply-To: References: <6E371FAC-03B4-4EC5-B1F2-5DAC1C5719B5@icir.org> Message-ID: <4E7575A4.1000006@lbl.gov> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I ran strace on the manager parent and child process during a cycle of relatively high use and came up with the following sets of system calls: 98337 kill 143871 read 128700 select 8058 write 70 sec Child: 56193 read 1 rt_sigreturn 684399 select 333803 write 75 sec For the parent the kill() call looks like: 1316290215.456133 kill(10062, SIG_0) = 0 with 10062 being the child manager process. There is a select loop in RemoteSerializer.cc which may be a candidate to tune > // FIXME: Fine-tune this (timeouts, flush, etc.) struct timeval > small_timeout; small_timeout.tv_sec = 0; small_timeout.tv_usec = > io->CanWrite() || io->CanRead() ? 1 : 10; > > int a = select(max_fd + 1, &fd_read, &fd_write, &fd_except, > &small_timeout); > > if ( a == 0 ) ++timeouts; > > if ( selects % 100000 == 0 ) Log(fmt("selects=%ld canwrites=%ld > timeouts=%ld", selects, canwrites, timeouts)); > > if ( a < 0 ) // Ignore errors for now. continue; or ChunkedIO.cc : small_timeout.tv_usec = 50; This is just a (pair of) guesses - current version is 1.6-dev-1261 running on Linux golumn.nersc.gov 2.6.38.6-26.rc1.fc15.x86_64 #1 SMP Mon May 9 20:45:15 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux via PF_RING 5.01 and ixgbe+DNA driver. scott On 9/17/11 9:17 PM, Martin Holste wrote: > Yep, this is definitely happening. Our cluster's manager and its > child, as well as the proxy and its child chew up about 20-25% CPU > each, so that's almost a full CPU for the manager/child and > proxy/child. > > On Fri, Sep 16, 2011 at 7:52 PM, Seth Hall wrote: >> I'm seeing what appears to be extremely high CPU utilization by >> the manager's child process in a rather large (23 worker?) >> cluster. I believe the deserialization is being done by the >> child process and I'm guessing that the child process is being >> overwhelmed by all of the deserialization it's having to do for >> the logging framework's log records. >> >> Any one have thoughts about if I'm correct or ways this could be >> optimized? I'm just worried this could become a problem because >> logs are written much faster at other sites and I can see other >> sites getting communication framework overload really easily. I >> think it's something that we need to keep a close eye on at >> least. >> >> .Seth >> >> -- Seth Hall International Computer Science Institute (Bro) >> because everyone has a network http://www.bro-ids.org/ >> >> >> _______________________________________________ bro-dev mailing >> list bro-dev at bro-ids.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev >> > > _______________________________________________ bro-dev mailing > list bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFOdXWkK2Plq8B7ZBwRApFUAKDYqQtgpCtbOYmum2pCghxNwdHfEQCgm/D4 OhNCPaIEdX+G2HfBC5SROPM= =TJ2l -----END PGP SIGNATURE----- From bro at tracker.bro-ids.org Sun Sep 18 08:52:21 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 18 Sep 2011 15:52:21 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.e5ab85e5d44b57e1df99dc0b5eb05bc0@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by gregor): We now have a hotfix for that. Once the SSL connection is going encrypted (ssl_established), we disable the analyzer from script land. After that the analyzer doesn't do anything else anyways. If you want to see that "bad" behavior comment out the disable_analyzer() call in scripts/base/protocols/ssl/main.bro This solves (most of) the symptoms but not the root cause. Will open another ticket to address and track down the actual cause for 2.1 Note though, that when running in bare mode with with some ssl event handlers, the problem will re-appear disable_analyzer() won't be used then. cu Gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 18 08:53:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 18 Sep 2011 15:53:51 -0000 Subject: [Bro-Dev] #620: SSL analyzer memory usage -- find root cause Message-ID: <048.be0914f46337bb36376aa0f26a41a1d6@tracker.bro-ids.org> #620: SSL analyzer memory usage -- find root cause ---------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.7 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ See #574 While we got rid of the symptoms for #574, the root cause still persists. Find it an squish it. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 18 16:08:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 18 Sep 2011 23:08:07 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.850d61f572b293e1cbdfb26752b8867f@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by robin): > This solves (most of) the symptoms but not the root cause. Will open > another ticket to address and track down the actual cause for 2.1 I have a new piece for the puzzle. My guess is that this is causing the trouble: {{{ type SSLPDU(is_orig: bool) = record { records : SSLRecord(is_orig)[] &until($element <= 0); } &byteorder = bigendian; }}} What I believe is happening is that binpac builds the complete array in memory and doesn't release anything before the connection terminates. For long-lived heavy-volume SSL connections, that's not good. Indeed, this patch for the *generated* code gives me stable memory usage with Gregor's trace and ``SSL::disable_analyzer_after_detection=F``: {{{ --- build/src/ssl_pac.cc.org 2011-09-18 15:57:03.000000000 -0700 +++ build/src/ssl_pac.cc 2011-09-18 15:57:41.000000000 -0700 @@ -3098,7 +3098,9 @@ } if ( ! t_records__elem_parsing_complete ) goto need_more_data; - records_->push_back(records__elem_); + + //records_->push_back(records__elem_); + // Check &until( ( $element <= ((int) 0) ) ) if ( ( records__elem_ <= 0 ) ) { @@ -3106,6 +3108,8 @@ records__elem_ = 0; goto end_of_records; } + + delete records__elem_; records__elem_ = 0; } end_of_records: ; }}} Can you guys confirm that (1) this indeed fixes the problem, and (2) doesn't break anything else? If so, we should figure out how to teach binpac to generate the modified code. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 18 16:28:18 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 18 Sep 2011 23:28:18 -0000 Subject: [Bro-Dev] #511: Misc distribution cleanup In-Reply-To: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> References: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> Message-ID: <062.ea3fd28745e4be2e5719fe56abae45ec@tracker.bro-ids.org> #511: Misc distribution cleanup ---------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ---------------------+-------------------- Changes (by robin): * type: Merge Request => Task Comment: Merged, but I'm leaving this open for the remaining changes. I 'll go through the README and licensing information before we release. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 18 16:31:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 18 Sep 2011 23:31:07 -0000 Subject: [Bro-Dev] #617: configure errors due to ruby In-Reply-To: <047.fc0412592c1a1428e0ab9071b234beab@tracker.bro-ids.org> References: <047.fc0412592c1a1428e0ab9071b234beab@tracker.bro-ids.org> Message-ID: <062.41072160507564230e8164363fcac2b9@tracker.bro-ids.org> #617: configure errors due to ruby -----------------------------+------------------------ Reporter: robin | Owner: Type: Merge Request | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 18 20:54:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 19 Sep 2011 03:54:42 -0000 Subject: [Bro-Dev] #472: Cleanup scripts in bro-aux In-Reply-To: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> References: <047.3f213fc8d7000dd46ed9e7cf4dcc81a0@tracker.bro-ids.org> Message-ID: <062.823c53c77661c439f5fa319cf9530603@tracker.bro-ids.org> #472: Cleanup scripts in bro-aux -----------------------------+--------------------- Reporter: robin | Owner: Type: Task | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: preview -----------------------------+--------------------- Comment (by robin): In [c1ff52a963e6ecdba940738d870248b0202c63fb/bro-aux]: {{{ #!CommitTicketReference repository="bro-aux" revision="c1ff52a963e6ecdba940738d870248b0202c63fb" Merge remote-tracking branch 'origin/topic/dist-cleanup' * origin/topic/dist-cleanup: Updating README Add gen-mozilla-ca-list.rb script back to bro-aux Remove $Id$ tags Remove all bro-aux scripts, addresses #511, #472 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 18 20:54:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 19 Sep 2011 03:54:42 -0000 Subject: [Bro-Dev] #511: Misc distribution cleanup In-Reply-To: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> References: <047.7fad0a85cd281981d5bd58a78ff6a111@tracker.bro-ids.org> Message-ID: <062.f2e85c20fef3f71b035be7efad9b07b0@tracker.bro-ids.org> #511: Misc distribution cleanup ---------------------+-------------------- Reporter: robin | Owner: Type: Task | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ---------------------+-------------------- Comment (by robin): In [c1ff52a963e6ecdba940738d870248b0202c63fb/bro-aux]: {{{ #!CommitTicketReference repository="bro-aux" revision="c1ff52a963e6ecdba940738d870248b0202c63fb" Merge remote-tracking branch 'origin/topic/dist-cleanup' * origin/topic/dist-cleanup: Updating README Add gen-mozilla-ca-list.rb script back to bro-aux Remove $Id$ tags Remove all bro-aux scripts, addresses #511, #472 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 18 20:54:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 19 Sep 2011 03:54:53 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.c9e31a6db64ba123a2317b8074a92e15@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): In [1c55d525a843af481f674ada46d48b35e2d367f3/broccoli]: {{{ #!CommitTicketReference repository="broccoli" revision="1c55d525a843af481f674ada46d48b35e2d367f3" Merge remote-tracking branch 'origin/topic/broccoli-manual-rest' * origin/topic/broccoli-manual-rest: Work around bug in CMake custom target dependencies. Broccoli API Doxygen doc tweaks (addresses #563) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Sun Sep 18 21:20:23 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 18 Sep 2011 21:20:23 -0700 Subject: [Bro-Dev] manager-child cpu utilization? In-Reply-To: <6E371FAC-03B4-4EC5-B1F2-5DAC1C5719B5@icir.org> References: <6E371FAC-03B4-4EC5-B1F2-5DAC1C5719B5@icir.org> <4E7575A4.1000006@lbl.gov> <6E371FAC-03B4-4EC5-B1F2-5DAC1C5719B5@icir.org> <6E371FAC-03B4-4EC5-B1F2-5DAC1C5719B5@icir.org> Message-ID: <20110919042023.GL97650@icir.org> On Fri, Sep 16, 2011 at 20:52 -0400, Seth wrote: > I'm seeing what appears to be extremely high CPU utilization by the > manager's child process in a rather large (23 worker?) cluster. I > believe the deserialization is being done by the child process It's not actually, the child process just relays the raw data between the remote side and the main process. The latter does the (de-)serialization. Also, the logging does actually not use (most of) the normal serialization framework. It has its own code for (un-)marshalling its data, which is less complex. And I'm just realizing that once we have merged in the threaded logging, we could actually even shift the demarshalling to the threads I believe .. On Sat, Sep 17, 2011 at 21:17 -0500, Martin wrote: > Yep, this is definitely happening. Our cluster's manager and its > child, as well as the proxy and its child chew up about 20-25% CPU > each, so that's almost a full CPU for the manager/child and > proxy/child. Note that the proxy doesn't see the logging data, just &synchronized data passes through it. For the manager it's the opposite: it sees the logs, but not &synchronized. Seth, didn't you say we actually don't have much &synchronized at all anymore now? The communication performance with the new scripts is something I'm still planing to play with a bit more myself. Soon hopefully. On Sat, Sep 17, 2011 at 23:37 -0500, Scott wrote: > There is a select loop in RemoteSerializer.cc which may be a candidate > to tune Yes, this may indeed be something that could need a bit more tuning. But generally, I don't really see the child process becoming a bottleneck. High CPU there could also be just a sign of idle-looping a bit too much. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Sun Sep 18 21:27:28 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 18 Sep 2011 21:27:28 -0700 Subject: [Bro-Dev] PCAP_PF_RING_CLUSTER_ID In-Reply-To: References: <20110916015223.GL88926@icir.org> Message-ID: <20110919042728.GM97650@icir.org> On Thu, Sep 15, 2011 at 22:19 -0400, you wrote: > Heh, it's "bro" typed out on a phone number pad (276) represented as > an 8bit int (wrapped around). In other words, it's a fairly arbitrary > number. :) I see. :) But let's then please use something like 0 or -1 to indicate that it actually doesn't matter. Arbitrary values can be quite confusing for the non-initiated. > I thought about doing that, but it seemed somewhat superfluous since > those environment variables will only be used if the pf_ring libpcap > wrapper is used. It does seem like the right thing to do however. Yeah, for the same reason as above: I bet that otherwise somebody will scratch his head at some point to understand what effect PCAP_PF_RING_CLUSTER_ID has on his FreeBSD box. Cc'in the tracker. Tasks are: - If system doesn't have PF_RING support, don't set environment variable. - If system has PF_RING support, set ID to a default that indicates "not set". @component: BroControl @version: git/master @type: Task @milestone: Bro1.6 @keywords: beta -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Sun Sep 18 21:50:46 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 19 Sep 2011 04:50:46 -0000 Subject: [Bro-Dev] #621: Re: PCAP_PF_RING_CLUSTER_ID Message-ID: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> #621: Re: [Bro-Dev] PCAP_PF_RING_CLUSTER_ID ------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: Bro1.6 | Component: Bro Version: git/master | Keywords: beta ------------------------+--------------------- On Thu, Sep 15, 2011 at 22:19 -0400, you wrote: > Heh, it's "bro" typed out on a phone number pad (276) represented as > an 8bit int (wrapped around). In other words, it's a fairly arbitrary > number. :) I see. :) But let's then please use something like 0 or -1 to indicate that it actually doesn't matter. Arbitrary values can be quite confusing for the non-initiated. > I thought about doing that, but it seemed somewhat superfluous since > those environment variables will only be used if the pf_ring libpcap > wrapper is used. It does seem like the right thing to do however. Yeah, for the same reason as above: I bet that otherwise somebody will scratch his head at some point to understand what effect PCAP_PF_RING_CLUSTER_ID has on his FreeBSD box. Cc'in the tracker. Tasks are: - If system doesn't have PF_RING support, don't set environment variable. - If system has PF_RING support, set ID to a default that indicates "not set". -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 18 21:52:24 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 19 Sep 2011 04:52:24 -0000 Subject: [Bro-Dev] #622: Installation issues Message-ID: <047.9f791003db697f134f198a006352d1c5@tracker.bro-ids.org> #622: Installation issues ---------------------------+--------------------- Reporter: robin | Type: Problem Status: new | Priority: Normal Milestone: Bro1.6 Bro1.7 | Component: Bro Version: git/master | Keywords: beta ---------------------------+--------------------- Sorry for going back to the installation process but a few thoughts about "make install" (feel free to remind me about past discussions, I don't remember these very well anymore): - It currently installs the following binaries: adtrace bdcat bro broccoli-config broctl capstats ftwire2bro nfcollector rst trace-summary That's seem too much to me, I think we should skip the bro-aux programs by default. That would then leave us with just bro broctl capstats trace-summary broccoli-config We could add target "make install-aux" to get the rest. - It creates "$prefix/logs". I don't remember whether we discussed this already, but is that a good default location for logs? - I noticed that we still install scripts that are now obsolete: bro-logchk.pl hot-report mon-report mvlog That leaves us in into $prefix/scripts with: host-grep ip-grep host-to-addrs lock_file However, I'm almost thinking we should remove these as well; not sure it's worth shipping them. Or, if we really want to keep them, at least don't install by default (then perhaps as part of the "install-aux" target from above). - Was there a reason we install the spool pieces directly into $prefix/spool/ instead of into $prefix/spool/bro/? - test-all-policy.bro ends up in share/bro, which doesn't seem right. - Do we really need to install share/bro/site/local-proxy.bro? It's empty and unlikley that many people will want to edit it. - Thoughts about local.bro: * Why is "@load protocols/http/detect-webapps" commented out? We should add a comment explaining when one would want to include it. * "Requires that the Site::local_zones variable". We should add where/how to do that. Perhaps an empty definition right in local.bro? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 18 21:53:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 19 Sep 2011 04:53:52 -0000 Subject: [Bro-Dev] #622: Installation issues In-Reply-To: <047.9f791003db697f134f198a006352d1c5@tracker.bro-ids.org> References: <047.9f791003db697f134f198a006352d1c5@tracker.bro-ids.org> Message-ID: <062.35a2a83a0b51b4bead77a17b8e794c5c@tracker.bro-ids.org> #622: Installation issues ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Changes (by robin): * milestone: Bro1.6 Bro1.7 => Bro1.6 Old description: > Sorry for going back to the installation process but a few thoughts > about "make install" (feel free to remind me about past discussions, I > don't remember these very well anymore): > > - It currently installs the following binaries: > > adtrace bdcat bro broccoli-config > broctl capstats ftwire2bro nfcollector > rst trace-summary > > That's seem too much to me, I think we should skip the bro-aux > programs by default. That would then leave us with just > > bro broctl capstats trace-summary broccoli-config > > We could add target "make install-aux" to get the rest. > > - It creates "$prefix/logs". I don't remember whether we discussed > this already, but is that a good default location for logs? > > - I noticed that we still install scripts that are now obsolete: > > bro-logchk.pl > hot-report > mon-report > mvlog > > That leaves us in into $prefix/scripts with: > > host-grep > ip-grep > host-to-addrs > lock_file > > However, I'm almost thinking we should remove these as well; not > sure it's worth shipping them. Or, if we really want to keep them, > at least don't install by default (then perhaps as part of the > "install-aux" target from above). > > - Was there a reason we install the spool pieces directly into > $prefix/spool/ instead of into $prefix/spool/bro/? > > - test-all-policy.bro ends up in share/bro, which doesn't seem right. > > - Do we really need to install share/bro/site/local-proxy.bro? It's > empty and unlikley that many people will want to edit it. > > - Thoughts about local.bro: > > * Why is "@load protocols/http/detect-webapps" commented out? We > should add a comment explaining when one would want to include it. > > * "Requires that the Site::local_zones variable". We should add > where/how to do that. Perhaps an empty definition right in > local.bro? New description: Sorry for going back to the installation process but a few thoughts about "make install" (feel free to remind me about past discussions, I don't remember these very well anymore): - It currently installs the following binaries: adtrace bdcat bro broccoli-config broctl capstats ftwire2bro nfcollector rst trace-summary That's seem too much to me, I think we should skip the bro-aux programs by default. That would then leave us with just bro broctl capstats trace-summary broccoli-config We could add target "make install-aux" to get the rest. - It creates "$prefix/logs". I don't remember whether we discussed this already, but is that a good default location for logs? - Was there a reason we install the spool pieces directly into $prefix/spool/ instead of into $prefix/spool/bro/? - test-all-policy.bro ends up in share/bro, which doesn't seem right. - Do we really need to install share/bro/site/local-proxy.bro? It's empty and unlikley that many people will want to edit it. - Thoughts about local.bro: * Why is "@load protocols/http/detect-webapps" commented out? We should add a comment explaining when one would want to include it. * "Requires that the Site::local_zones variable". We should add where/how to do that. Perhaps an empty definition right in local.bro? -- -- Ticket URL: Bro Tracker Bro Issue Tracker From mcholste at gmail.com Mon Sep 19 06:54:20 2011 From: mcholste at gmail.com (Martin Holste) Date: Mon, 19 Sep 2011 08:54:20 -0500 Subject: [Bro-Dev] PCAP_PF_RING_CLUSTER_ID In-Reply-To: <20110919042728.GM97650@icir.org> References: <20110916015223.GL88926@icir.org> <20110919042728.GM97650@icir.org> Message-ID: But a cluster_id of 0 indicates the cluster is not being used, which it is, so it must be a positive integer. To keep it as simple as possible, I guess I'd go with 1, but since there may be other clusters running on the same system, there is value in picking a pseudo-random number like 21, as 1 is probably more common for someone to choose for the reason above. 21 has some "Bro" meaning. For instance, a system with Suricata will use cluster_id 99 by default. Also, I would argue that if you've installed PF_RING, you are "initiated," at least to some extent. I say this because you have to go out of your way to install it--it's not in any standard Linux distro. On Sun, Sep 18, 2011 at 11:27 PM, Robin Sommer wrote: > > On Thu, Sep 15, 2011 at 22:19 -0400, you wrote: > >> Heh, it's "bro" typed out on a phone number pad (276) represented as >> an 8bit int (wrapped around). ?In other words, it's a fairly arbitrary >> number. :) > > I see. :) But let's then please use something like 0 or -1 to indicate > that it actually doesn't matter. Arbitrary values can be quite > confusing for the non-initiated. > >> I thought about doing that, but it seemed somewhat superfluous since >> those environment variables will only be used if the pf_ring libpcap >> wrapper is used. ?It does seem like the right thing to do however. > > Yeah, for the same reason as above: I bet that otherwise somebody will > scratch his head at some point to understand what effect > PCAP_PF_RING_CLUSTER_ID has on his FreeBSD box. > > Cc'in the tracker. Tasks are: > > ? ?- If system doesn't have PF_RING support, don't set environment > ? ?variable. > > ? ?- If system has PF_RING support, set ID to a default that > ? ?indicates "not set". > > @component: BroControl > @version: git/master > @type: Task > @milestone: Bro1.6 > @keywords: beta > > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL ? ?* Fax ? +1 (510) 666-2956 * ? www.icir.org > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From robin at icir.org Mon Sep 19 08:29:12 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 19 Sep 2011 08:29:12 -0700 Subject: [Bro-Dev] PCAP_PF_RING_CLUSTER_ID In-Reply-To: References: <20110916015223.GL88926@icir.org> <20110919042728.GM97650@icir.org> Message-ID: <20110919152912.GB35703@icir.org> On Mon, Sep 19, 2011 at 08:54 -0500, you wrote: > But a cluster_id of 0 indicates the cluster is not being used, which > it is, so it must be a positive integer. To keep it as simple as > possible, I guess I'd go with 1, but since there may be other clusters > running on the same system, there is value in picking a pseudo-random > number like 21, as 1 is probably more common for someone to choose for > the reason above. 21 has some "Bro" meaning. Ah, I see. I misread "fairly arbitrary" as "doesn't matter", but I think I got it now. Ok, then let's stay with 21. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Mon Sep 19 08:30:05 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 19 Sep 2011 15:30:05 -0000 Subject: [Bro-Dev] #621: Re: PCAP_PF_RING_CLUSTER_ID In-Reply-To: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> References: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> Message-ID: <062.8b770550144b328e923dbd886cace35f@tracker.bro-ids.org> #621: Re: [Bro-Dev] PCAP_PF_RING_CLUSTER_ID ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by robin): Update: let's stay with 21, I misunderstood what it means. But let's still do the configure check please. -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Mon Sep 19 10:36:55 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 19 Sep 2011 12:36:55 -0500 Subject: [Bro-Dev] cmake repo/submodule Message-ID: <79DCEFA5-8606-43F0-96FE-090493838EB0@ncsa.illinois.edu> I'd like to reorganize all the CMake-related scripts of each repository (the cmake/ dirs) into it's own repository and then just have each repo use that as a submodule. The advantage of this is decreasing the burden of maintaining CMake script changes -- there's a lot of shared code between repos that are currently maintained separately. Disadvantages might be: 1) more submodule maintenance. I think we already understand well enough how do this that it's not going to be a problem, and maintaining submodules is much less of a burden than maintaining CMake scripts in each repo 2) initializing/updating at least this cmake module after cloning the parent repo will be required to configure/build it. This is only a problem for people that weren't cloning recursively, but to take care of it I'll just make the ./configure scripts do the cmake/ submodule initialization if it sees nothing's there. Anyone think of a reason not to go forward with that plan? (This could wait until release-after-next, but it's been nagging at me lately). - Jon From robin at icir.org Mon Sep 19 10:49:30 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 19 Sep 2011 10:49:30 -0700 Subject: [Bro-Dev] cmake repo/submodule In-Reply-To: <79DCEFA5-8606-43F0-96FE-090493838EB0@ncsa.illinois.edu> References: <79DCEFA5-8606-43F0-96FE-090493838EB0@ncsa.illinois.edu> Message-ID: <20110919174930.GC37474@icir.org> On Mon, Sep 19, 2011 at 12:36 -0500, you wrote: > I'd like to reorganize all the CMake-related scripts of each > repository (the cmake/ dirs) into it's own repository and then just > have each repo use that as a submodule. That's a very good thought. > 1) more submodule maintenance. > 2) initializing/updating at least this cmake module after cloning Both sound fine to me. > (This could wait until release-after-next, but it's been nagging at me lately). I'd rather do it now than later. How much work do you estimate this reorg to be? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Mon Sep 19 11:00:35 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 19 Sep 2011 13:00:35 -0500 Subject: [Bro-Dev] cmake repo/submodule In-Reply-To: <20110919174930.GC37474@icir.org> References: <79DCEFA5-8606-43F0-96FE-090493838EB0@ncsa.illinois.edu> <20110919174930.GC37474@icir.org> Message-ID: > I'd rather do it now than later. How much work do you estimate this > reorg to be? Probably just a day or two; mostly shuffling files around and testing that I didn't break anything. I'll start on it this afternoon. - Jon From bro at tracker.bro-ids.org Mon Sep 19 12:50:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 19 Sep 2011 19:50:55 -0000 Subject: [Bro-Dev] #622: Installation issues In-Reply-To: <047.9f791003db697f134f198a006352d1c5@tracker.bro-ids.org> References: <047.9f791003db697f134f198a006352d1c5@tracker.bro-ids.org> Message-ID: <062.eb460993d3130265d7a9a150ae421bcb@tracker.bro-ids.org> #622: Installation issues ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by jsiwek): In [531d428255652cce7d7255ec007dd2ade25e52a9/bro-aux]: {{{ #!CommitTicketReference repository="bro-aux" revision="531d428255652cce7d7255ec007dd2ade25e52a9" Adding "install-aux" target. (addresses #622) This target is only used to install the bro-aux programs when it's included as a sub-project of something else. When bro-aux is the main project, the regular install target will install the programs. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 20 06:40:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 20 Sep 2011 13:40:56 -0000 Subject: [Bro-Dev] #563: Check topic/broccoli-manual-rest In-Reply-To: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> References: <047.f233ee327abb14d7481311a0fa273fa2@tracker.bro-ids.org> Message-ID: <062.0bf56c038b43cab227120158fd530db4@tracker.bro-ids.org> #563: Check topic/broccoli-manual-rest -----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: closed Priority: Low | Milestone: Bro1.6 Component: Broccoli | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: assigned => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Tue Sep 20 08:24:47 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 20 Sep 2011 10:24:47 -0500 Subject: [Bro-Dev] cmake repo/submodule In-Reply-To: References: <79DCEFA5-8606-43F0-96FE-090493838EB0@ncsa.illinois.edu> <20110919174930.GC37474@icir.org> Message-ID: <313FF926-8A7A-4B56-BB1B-F296ED1C1A5C@ncsa.illinois.edu> On Sep 19, 2011, at 1:00 PM, Jonathan Siwek wrote: >> I'd rather do it now than later. How much work do you estimate this >> reorg to be? > > > Probably just a day or two; mostly shuffling files around and testing that I didn't break anything. I'll start on it this afternoon. It's done now in all the masters. A regular `git pull` alone at this point is not going to be enough for anyone to make their local clones work again, make sure to also do `git submodule update --recursive --init` - Jon From bro at tracker.bro-ids.org Tue Sep 20 09:02:15 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 20 Sep 2011 16:02:15 -0000 Subject: [Bro-Dev] #623: topic/seth/notice-suppression - Notice suppression and more Message-ID: <046.1bb40cc2293c862bb534f83287f00394@tracker.bro-ids.org> #623: topic/seth/notice-suppression - Notice suppression and more ---------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: beta | ---------------------------+-------------------- This is ready to merge and it has the fix for #607 in it (i'll close that ticket now). There are a few other fixes not related to notice suppression in it as well. - Duplicate notices are discovered with the new Notice::Info field $identifier. It's a string that is left up to the notice implementor to define which would indicate a fundamentally duplicate notice. The field is optional and if it's not included it's not possible for notice suppression to take place. - New events were created to give visibility into the notice framework's suppression activity. - event Notice::begin_suppression(n: Notice::Info) - event Notice::suppressed(n: Notice::Info) - event Notice::end_suppression(n: Notice::Info) - Worker raised notices are printed a single time by the manager now. - Cluster framework and notice framework integration cleaned up and implemented better and more completely. - The table tracking notice suppressions is now done with a table attribute instead of "manually" with scheduled events. - Two new notice tests. - Fix crash on exit (addresses #607). -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 20 09:02:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 20 Sep 2011 16:02:53 -0000 Subject: [Bro-Dev] #607: New segfault discovered when implementing duplicate notice suppression In-Reply-To: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> References: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> Message-ID: <061.28ee7576b72a46baa88f013563ba6aa4@tracker.bro-ids.org> #607: New segfault discovered when implementing duplicate notice suppression -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: beta -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 20 09:12:52 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 20 Sep 2011 16:12:52 -0000 Subject: [Bro-Dev] #614: trace-summary failure In-Reply-To: <046.8a57d3a3cdb4d3c1f868ec96d4a81df3@tracker.bro-ids.org> References: <046.8a57d3a3cdb4d3c1f868ec96d4a81df3@tracker.bro-ids.org> Message-ID: <061.32c2c11590bbd7677ba1dc585ad0abc9@tracker.bro-ids.org> #614: trace-summary failure -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: trace-summary | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by jsiwek): * status: new => closed * resolution: => Solved/Applied Comment: A fresh install doesn't give me this anymore: an update to broctl's trace- summary submodule fixed it. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 20 10:59:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 20 Sep 2011 17:59:47 -0000 Subject: [Bro-Dev] #621: Re: PCAP_PF_RING_CLUSTER_ID In-Reply-To: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> References: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> Message-ID: <062.42436db2509073e4fa476ce18b02d0ae@tracker.bro-ids.org> #621: Re: [Bro-Dev] PCAP_PF_RING_CLUSTER_ID ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by jsiwek): In [cc6ccf793efa8ad46e4f14864aac64a8f3ac2d80/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="cc6ccf793efa8ad46e4f14864aac64a8f3ac2d80" Add configure-time check for libpcap PF_RING support. This adds a configure time check for whether a found libpcap has PF_RING support and sets the right BroControl variable to either enable or disable starting Bro instances with an environment that will enable PF_RING use in libpcap. Addresses #621 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 20 11:00:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 20 Sep 2011 18:00:31 -0000 Subject: [Bro-Dev] #621: Re: PCAP_PF_RING_CLUSTER_ID In-Reply-To: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> References: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> Message-ID: <062.a82547daf60ffa5b30f9cad4c6dfc94e@tracker.bro-ids.org> #621: Re: [Bro-Dev] PCAP_PF_RING_CLUSTER_ID ----------------------------+------------------------ Reporter: robin | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: Replying to [comment:1 robin]: > Update: let's stay with 21, I misunderstood what it means. But let's still do the configure check please. In `topic/jsiwek/pfring-configure-check` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 20 11:34:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 20 Sep 2011 18:34:49 -0000 Subject: [Bro-Dev] #624: Bro won't shutdown if not seeing packets Message-ID: <046.06582d3d3a1ee7e1010ccc163ac53a7f@tracker.bro-ids.org> #624: Bro won't shutdown if not seeing packets ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Keywords: beta | ---------------------+-------------------- If appears that if a Bro instance is sniffing traffic but not actually seeing packets on the interface and it's instructed to shutdown with the control framework (terminate_communication() BiF) it won't actually shutdown. This is annoying in broctl if a node isn't seeing traffic because it makes the stop and restart commands take a long time since they wait until the timeout and have to kill the instances eventually -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 20 23:33:48 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 06:33:48 -0000 Subject: [Bro-Dev] #607: New segfault discovered when implementing duplicate notice suppression In-Reply-To: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> References: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> Message-ID: <061.2cd6ab7993d6da74848b44911e1c0564@tracker.bro-ids.org> #607: New segfault discovered when implementing duplicate notice suppression ----------------------+---------------------- Reporter: seth | Owner: Type: Problem | Status: reopened Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+---------------------- Changes (by seth): * status: closed => reopened * resolution: Solved/Applied => Comment: I have a crash from this fix. It only seems to happen on the manager in a cluster and it seems to be triggered very quickly after starting the cluster. {{{ #0 0x00000035d5832905 in raise () from /lib64/libc.so.6 #1 0x00000035d58340e5 in abort () from /lib64/libc.so.6 #2 0x00000000006d22ca in Reporter::InternalError (this=0x21ad8f0, fmt=0x7d95e0 "%s") at /src/Reporter.cc:102 #3 0x000000000071b72d in RemoteSerializer::InternalCommError (this=0x21e2160, msg=0x21a9fc0 "unexpected msg type 0") at /src/RemoteSerializer.cc:3073 #4 0x00000000007165be in RemoteSerializer::Poll (this=0x21e2160, may_block=false) at /src/RemoteSerializer.cc:1519 #5 0x0000000000715c0b in RemoteSerializer::NextTimestamp (this=0x21e2160, local_network_time=0x7fffd656c8a8) at /src/RemoteSerializer.cc:1337 #6 0x00000000006c50d9 in IOSourceRegistry::FindSoonest (this=0xa9efb0, ts=0x7fffd656c9b8) at /src/IOSource.cc:61 #7 0x00000000006f0f19 in net_run () at /src/Net.cc:425 #8 0x00000000005ee2bd in main (argc=14, argv=0x7fffd656cfb8) at /src/main.cc:1009 }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 21 07:02:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 14:02:31 -0000 Subject: [Bro-Dev] #625: Log rotation broken Message-ID: <046.1f1a58a7be641fbc924cc8183913059c@tracker.bro-ids.org> #625: Log rotation broken ---------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Keywords: beta | ---------------------+-------------------- For some reason, file rotation isn't happening anymore. I'm not sure when this regression was introduced but I'm definitely not seeing files rotate anymore. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 21 07:03:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 14:03:47 -0000 Subject: [Bro-Dev] #607: New segfault discovered when implementing duplicate notice suppression In-Reply-To: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> References: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> Message-ID: <061.414e2409ced231671e5d9d14cd2236d8@tracker.bro-ids.org> #607: New segfault discovered when implementing duplicate notice suppression -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: beta -----------------------------+-------------------- Changes (by seth): * status: reopened => closed * resolution: => Solved/Applied Comment: Nevermind the backtrace I filed. It's a communication system overload. I have some ideas of how to deal with it, but it's definitely unrelated to this ticket. I'm going to re-close this ticket now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 21 07:15:57 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 14:15:57 -0000 Subject: [Bro-Dev] #625: Log rotation broken In-Reply-To: <046.1f1a58a7be641fbc924cc8183913059c@tracker.bro-ids.org> References: <046.1f1a58a7be641fbc924cc8183913059c@tracker.bro-ids.org> Message-ID: <061.3d60e7e9b27ac897c811221280d80e1f@tracker.bro-ids.org> #625: Log rotation broken ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by jsiwek): Is it bro standalone, broctl standalone, or a broctl cluster that's not rotating for you? (i.e. same as #619 ?) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 21 07:17:42 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 14:17:42 -0000 Subject: [Bro-Dev] #625: Log rotation broken In-Reply-To: <046.1f1a58a7be641fbc924cc8183913059c@tracker.bro-ids.org> References: <046.1f1a58a7be641fbc924cc8183913059c@tracker.bro-ids.org> Message-ID: <061.ac877b95f3617e55c534c0be682cd58d@tracker.bro-ids.org> #625: Log rotation broken ------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: Duplicate | Keywords: beta ------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Duplicate Comment: Whoops. Duplicate of #619. Thanks Jon. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 21 07:18:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 14:18:14 -0000 Subject: [Bro-Dev] #625: Log rotation broken In-Reply-To: <046.1f1a58a7be641fbc924cc8183913059c@tracker.bro-ids.org> References: <046.1f1a58a7be641fbc924cc8183913059c@tracker.bro-ids.org> Message-ID: <061.4766fe62856e927ce3dc2767683452aa@tracker.bro-ids.org> #625: Log rotation broken ------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: Duplicate | Keywords: beta ------------------------+-------------------- Comment (by seth): It's a broctl cluster BTW. I haven't tested in standalone broctl or just plain bro. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 21 13:34:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 20:34:51 -0000 Subject: [Bro-Dev] #619: Log rotations seem broken In-Reply-To: <047.81859bf695ec52d8dd6278a9176c26bd@tracker.bro-ids.org> References: <047.81859bf695ec52d8dd6278a9176c26bd@tracker.bro-ids.org> Message-ID: <062.45c463f98cecc16422c29a14e3bd00d3@tracker.bro-ids.org> #619: Log rotations seem broken -------------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: | Keywords: beta -------------------------+------------------------ Comment (by jsiwek): In [39af63841bb767e7cddbf5e4f07e7502fefb497e/bro]: {{{ #!CommitTicketReference repository="bro" revision="39af63841bb767e7cddbf5e4f07e7502fefb497e" Fix broctl cluster log rotation (addresses #619) Before Log::rotation_control was removed in fe38c22d, it was used to lookup rotation inverval at the time of installing a rotation timer, and could fall back on the &default=default_rotation_inverval. But now since filter rotation information is instead transferred to a Writer when it's created, it also needed to be taught to fall back on the default rotation interval. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 21 13:35:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 20:35:27 -0000 Subject: [Bro-Dev] #618: broctl check rotates logs In-Reply-To: <047.2c46ff0234b524e0ef0a8917a71abd60@tracker.bro-ids.org> References: <047.2c46ff0234b524e0ef0a8917a71abd60@tracker.bro-ids.org> Message-ID: <062.8f1aaeb63049c900437aa4a5360031c4@tracker.bro-ids.org> #618: broctl check rotates logs ----------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by jsiwek): In [d497d86639f073575d0f9a755505ea22672ee1e4/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="d497d86639f073575d0f9a755505ea22672ee1e4" broctl check no longer rotates logs (addresses #618) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 21 13:35:28 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 20:35:28 -0000 Subject: [Bro-Dev] #596: broctl "restart" commands is reversed In-Reply-To: <046.ff424b35e40b3078b7bbffd077d23722@tracker.bro-ids.org> References: <046.ff424b35e40b3078b7bbffd077d23722@tracker.bro-ids.org> Message-ID: <061.9edec51e48d6d4c33b9bf574f42b672d@tracker.bro-ids.org> #596: broctl "restart" commands is reversed -------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta -------------------------+-------------------- Comment (by jsiwek): In [abb3f9ac9a084bad57177b8784dbcbb22aeda030/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="abb3f9ac9a084bad57177b8784dbcbb22aeda030" Stop and restart command now stop worker nodes first (addresses #596) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 21 13:37:14 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 20:37:14 -0000 Subject: [Bro-Dev] #618: broctl check rotates logs In-Reply-To: <047.2c46ff0234b524e0ef0a8917a71abd60@tracker.bro-ids.org> References: <047.2c46ff0234b524e0ef0a8917a71abd60@tracker.bro-ids.org> Message-ID: <062.ee4c3e6eab0e75a6eb92ec3477fe695d@tracker.bro-ids.org> #618: broctl check rotates logs ----------------------------+-------------------- Reporter: robin | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Changes (by jsiwek): * type: Problem => Merge Request Comment: In `topic/jsiwek/broctl-cluster-fixes` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 21 13:38:53 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 20:38:53 -0000 Subject: [Bro-Dev] #596: broctl "restart" commands is reversed In-Reply-To: <046.ff424b35e40b3078b7bbffd077d23722@tracker.bro-ids.org> References: <046.ff424b35e40b3078b7bbffd077d23722@tracker.bro-ids.org> Message-ID: <061.0b5268b1432799751acb02ce78991f06@tracker.bro-ids.org> #596: broctl "restart" commands is reversed ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Changes (by jsiwek): * type: Problem => Merge Request Comment: In `topic/jsiwek/broctl-cluster-fixes` -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 21 13:40:16 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 20:40:16 -0000 Subject: [Bro-Dev] #615: Cluster manager crash In-Reply-To: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> References: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> Message-ID: <061.3e868bd381c988e07e7be1db069deaae@tracker.bro-ids.org> #615: Cluster manager crash ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by jsiwek): Any more context to go with this or ideas how to reproduce? -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Wed Sep 21 13:42:04 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 21 Sep 2011 13:42:04 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/broctl] topic/jsiwek/broctl-cluster-fixes: broctl check no longer rotates logs (addresses #618) (d497d86) In-Reply-To: <201109212035.p8LKZUAJ007898@bro-ids.icir.org> References: <201109212035.p8LKZUAJ007898@bro-ids.icir.org> Message-ID: <20110921204204.GE13706@icir.org> On Wed, Sep 21, 2011 at 13:35 -0700, Jonathan Siwek wrote: > + cmd += " Log::default_rotation_interval=0secs" I'd prefer to move this setting into a script file called something like check.bro that only gets loaded when checking. That way, we can later add further tuning there if there's more that's particular to "check mode". (In fact, I believe we used to have such a check.bro but it may not exist anymore currently.) Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Wed Sep 21 14:12:23 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Wed, 21 Sep 2011 21:12:23 -0000 Subject: [Bro-Dev] #618: broctl check rotates logs In-Reply-To: <047.2c46ff0234b524e0ef0a8917a71abd60@tracker.bro-ids.org> References: <047.2c46ff0234b524e0ef0a8917a71abd60@tracker.bro-ids.org> Message-ID: <062.69bf06c84d2818fb0a4bd4d34477ee07@tracker.bro-ids.org> #618: broctl check rotates logs ----------------------------+-------------------- Reporter: robin | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Comment (by jsiwek): In [a453e07d3a95d67c83ef7d035d83af9dbbcc0904/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="a453e07d3a95d67c83ef7d035d83af9dbbcc0904" Give check command its own script for tuning options (addresses #618) }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 22 09:50:29 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 22 Sep 2011 16:50:29 -0000 Subject: [Bro-Dev] #626: Known module Info record conflicts Message-ID: <048.84bf7ecc0b805ee86009860897680327@tracker.bro-ids.org> #626: Known module Info record conflicts ---------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: | ---------------------+------------------------ I think `Known::Info` records defined in `protocols/conn/known-hosts.bro`, `protocols/conn/known-services.bro`, and `protocols/ssl/known-certs.bro` are clashing, should they instead be using different module namespaces? e.g. I'm trying to figure out how to make this work: {{{ $ bro -b policy/tuning/track-all-assets error: stream event's argument type does not match column record type error: stream event's argument type does not match column record type }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 22 12:18:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 22 Sep 2011 19:18:33 -0000 Subject: [Bro-Dev] #611: Segfault in strftime BiF In-Reply-To: <046.220cfaf223e8017638bae915af2d1c77@tracker.bro-ids.org> References: <046.220cfaf223e8017638bae915af2d1c77@tracker.bro-ids.org> Message-ID: <061.bb9f898da9fce24d05681d7be27b8d71@tracker.bro-ids.org> #611: Segfault in strftime BiF ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by jsiwek): In [73dcdbcfc5f9174da267ba9ec272bbc2bad612fe/bro]: {{{ #!CommitTicketReference repository="bro" revision="73dcdbcfc5f9174da267ba9ec272bbc2bad612fe" Fix PktSrc setting next_timestamp even when no packet available When pcap_next failed to return packet data, such as when a live packet capture interface has no traffic, the next_timestamp member would still be set to possibly uninitialized or meaningless data. Maybe addresses #611. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 22 19:07:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 02:07:37 -0000 Subject: [Bro-Dev] #618: broctl check rotates logs In-Reply-To: <047.2c46ff0234b524e0ef0a8917a71abd60@tracker.bro-ids.org> References: <047.2c46ff0234b524e0ef0a8917a71abd60@tracker.bro-ids.org> Message-ID: <062.4fb8a24cc23e9277394325abe038a2b7@tracker.bro-ids.org> #618: broctl check rotates logs ----------------------------+-------------------- Reporter: robin | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta ----------------------------+-------------------- Changes (by robin): * component: Bro => BroControl -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 22 19:14:34 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 02:14:34 -0000 Subject: [Bro-Dev] #621: Re: PCAP_PF_RING_CLUSTER_ID In-Reply-To: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> References: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> Message-ID: <062.08c60bd4287fddbff5ab48e8b0fac111@tracker.bro-ids.org> #621: Re: [Bro-Dev] PCAP_PF_RING_CLUSTER_ID ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: | Keywords: beta ----------------------------+------------------------ Changes (by robin): * owner: => jsiwek * status: new => assigned * component: Bro => BroControl Comment: I don't think that setting the default value should go into ``broctl.cfg.in`` as it's nothing the user is likely to change. ``BroControl/options.py`` seems the better place though we'll need a way to get configure values in there (and take care that the doc generation does something reasonable when showing the default value; probably just ``configuration dependent`` or so.) -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 22 20:38:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 03:38:49 -0000 Subject: [Bro-Dev] #619: Log rotations seem broken In-Reply-To: <047.81859bf695ec52d8dd6278a9176c26bd@tracker.bro-ids.org> References: <047.81859bf695ec52d8dd6278a9176c26bd@tracker.bro-ids.org> Message-ID: <062.a4b2c244de839d507b9412e7fade3204@tracker.bro-ids.org> #619: Log rotations seem broken -------------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: | Keywords: beta -------------------------+------------------------ Comment (by robin): In [fc491c087b5c420f20cace30efa9ff29a3bba6b3/bro]: {{{ #!CommitTicketReference repository="bro" revision="fc491c087b5c420f20cace30efa9ff29a3bba6b3" Merge branch 'master', remote-tracking branch 'origin/fastpath' * origin/fastpath: Fix PktSrc setting next_timestamp even when no packet available Fix lack of NUL-termination in to_upper/to_lower BIF's return val. Fixing unit tests and some minor bugs. Fix broctl cluster log rotation (addresses #619) * master: }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 22 20:38:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 03:38:49 -0000 Subject: [Bro-Dev] #623: topic/seth/notice-suppression - Notice suppression and more In-Reply-To: <046.1bb40cc2293c862bb534f83287f00394@tracker.bro-ids.org> References: <046.1bb40cc2293c862bb534f83287f00394@tracker.bro-ids.org> Message-ID: <061.bb56e78f874a202c1117c4e089281107@tracker.bro-ids.org> #623: topic/seth/notice-suppression - Notice suppression and more ----------------------------+-------------------- Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: fixed | Keywords: beta ----------------------------+-------------------- Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [c9b9bab4737599bc137d01da95bd527298b8240d/bro]: {{{ #!CommitTicketReference repository="bro" revision="c9b9bab4737599bc137d01da95bd527298b8240d" Merge remote-tracking branch 'origin/topic/seth/notice-suppression' * origin/topic/seth/notice-suppression: Updated a notice related baseline and added a necessary @load line. Notice suppression clean up and notice/cluster integrtion fixes. Updates for notice suppression to use the &create_expire attribute Small, mostly cosmetic updates and fixing a test. Fix crash on exit (addresses #607). Duplicate notice suppression. Closes #623. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 22 20:38:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 03:38:49 -0000 Subject: [Bro-Dev] #607: New segfault discovered when implementing duplicate notice suppression In-Reply-To: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> References: <046.17b605bc0e9fadd54fa80a7ce10830b8@tracker.bro-ids.org> Message-ID: <061.636131c5aba7f41f3fde348e2be176ad@tracker.bro-ids.org> #607: New segfault discovered when implementing duplicate notice suppression -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: beta -----------------------------+-------------------- Comment (by robin): In [c9b9bab4737599bc137d01da95bd527298b8240d/bro]: {{{ #!CommitTicketReference repository="bro" revision="c9b9bab4737599bc137d01da95bd527298b8240d" Merge remote-tracking branch 'origin/topic/seth/notice-suppression' * origin/topic/seth/notice-suppression: Updated a notice related baseline and added a necessary @load line. Notice suppression clean up and notice/cluster integrtion fixes. Updates for notice suppression to use the &create_expire attribute Small, mostly cosmetic updates and fixing a test. Fix crash on exit (addresses #607). Duplicate notice suppression. Closes #623. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 22 20:39:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 03:39:45 -0000 Subject: [Bro-Dev] #618: broctl check rotates logs In-Reply-To: <047.2c46ff0234b524e0ef0a8917a71abd60@tracker.bro-ids.org> References: <047.2c46ff0234b524e0ef0a8917a71abd60@tracker.bro-ids.org> Message-ID: <062.3b245b7581c5450f4cb6bb01f73f1991@tracker.bro-ids.org> #618: broctl check rotates logs ----------------------------+-------------------- Reporter: robin | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: fixed | Keywords: beta ----------------------------+-------------------- Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [b00e7016b423c89e6bc6035e7f6b45a24818294d/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="b00e7016b423c89e6bc6035e7f6b45a24818294d" Merge remote-tracking branch 'origin/topic/jsiwek/broctl-cluster-fixes' * origin/topic/jsiwek/broctl-cluster-fixes: Give check command its own script for tuning options (addresses #618) Stop and restart command now stop worker nodes first (addresses #596) broctl check no longer rotates logs (addresses #618) Note: I've added a process-trace.bro script in the same spirit as check.bro Closes #596. Closes #618. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 22 20:39:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 03:39:45 -0000 Subject: [Bro-Dev] #596: broctl "restart" commands is reversed In-Reply-To: <046.ff424b35e40b3078b7bbffd077d23722@tracker.bro-ids.org> References: <046.ff424b35e40b3078b7bbffd077d23722@tracker.bro-ids.org> Message-ID: <061.7d0753a393354471c5b77a04880aa560@tracker.bro-ids.org> #596: broctl "restart" commands is reversed ----------------------------+-------------------- Reporter: seth | Owner: robin Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: fixed | Keywords: beta ----------------------------+-------------------- Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [b00e7016b423c89e6bc6035e7f6b45a24818294d/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="b00e7016b423c89e6bc6035e7f6b45a24818294d" Merge remote-tracking branch 'origin/topic/jsiwek/broctl-cluster-fixes' * origin/topic/jsiwek/broctl-cluster-fixes: Give check command its own script for tuning options (addresses #618) Stop and restart command now stop worker nodes first (addresses #596) broctl check no longer rotates logs (addresses #618) Note: I've added a process-trace.bro script in the same spirit as check.bro Closes #596. Closes #618. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 22 21:23:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 04:23:38 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.2a1acfe81acf3fc562d8993275b93cb2@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by robin): This seems to fix it. I'm adding support for &transient arrays to binpac. Will still leave the ``disable_analyzer`` call in, seems that can only help. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 22 21:46:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 04:46:47 -0000 Subject: [Bro-Dev] #627: Crash in libmagic Message-ID: <047.ac2cfdb439b01aa3adf8209e0d016f6a@tracker.bro-ids.org> #627: Crash in libmagic ---------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Keywords: beta | ---------------------+------------------------ {{{ Core was generated by `bro'. Program terminated with signal 11, Segmentation fault. #0 0x000000080162c007 in memchr () from /lib/libc.so.7 #0 0x000000080162c007 in memchr () from /lib/libc.so.7 #1 0x0000000800d9b8bc in ?? () from /usr/lib/libmagic.so.4 #2 0x0000000800d9cda4 in file_softmagic () from /usr/lib/libmagic.so.4 #3 0x0000000800d9c96f in file_softmagic () from /usr/lib/libmagic.so.4 #4 0x0000000800da8ed8 in file_ascmagic_with_encoding () from /usr/lib/libmagic.so.4 #5 0x0000000800da96e6 in file_ascmagic () from /usr/lib/libmagic.so.4 #6 0x0000000800da7c4e in file_buffer () from /usr/lib/libmagic.so.4 #7 0x0000000800da32f5 in magic_buffer () from /usr/lib/libmagic.so.4 #8 0x00000000005068b7 in BifFunc::bro_identify_data (frame=) at bro.bif:3296 #9 0x00000000004fd0de in BuiltinFunc::Call (this=0x801a08a00, }}} No trace unfortunately. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 22 21:49:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 04:49:22 -0000 Subject: [Bro-Dev] #628: SaveTraces doesn't rotate Message-ID: <047.6606d3118ed306c827488fe7cefe653b@tracker.bro-ids.org> #628: SaveTraces doesn't rotate ------------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Keywords: beta | ------------------------+-------------------- With ``SaveTraces=1`` in broctl.cfg, the workers record traces but they are never rotated. The code for that seems to have been lost. We need to put that back in: it's supposed to delete the trace file every few minutes. Note that this used the old rotation framework, not immediately sure how to do that now ... -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 23 04:19:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 11:19:27 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.dc64fcd028f6c03f694e46429e09bbd9@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by seth): > This seems to fix it. I'm adding support for &transient arrays to binpac. > Will still leave the ``disable_analyzer`` call in, seems that can only > help. Hah, damn. I "fixed" the problem with existing binpac features on the flight home. &transient on arrays in binpac would be better though. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 23 08:10:43 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 15:10:43 -0000 Subject: [Bro-Dev] #627: Crash in libmagic In-Reply-To: <047.ac2cfdb439b01aa3adf8209e0d016f6a@tracker.bro-ids.org> References: <047.ac2cfdb439b01aa3adf8209e0d016f6a@tracker.bro-ids.org> Message-ID: <062.dcdc9de88e3ccbeab69a371742996d0a@tracker.bro-ids.org> #627: Crash in libmagic ------------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Duplicate | Keywords: beta ------------------------+------------------------ Changes (by gregor): * status: new => closed * resolution: => Duplicate Comment: Already reported in #612 -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 23 08:12:56 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 15:12:56 -0000 Subject: [Bro-Dev] #612: Segfault in identify_data BiF In-Reply-To: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> References: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> Message-ID: <063.224229d405110e65d99016fec4132f7c@tracker.bro-ids.org> #612: Segfault in identify_data BiF ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by gregor): The crash is maybe not so rare. Robin had one too (#627). IMHO it's not libmagic that's at fault but rather bro that passes an invalid pointer to it. We might have to run Bro with --enable-debug so we can get the call arguments to magic_buffer (or identify_data respectively) in the stack traces. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 23 08:21:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 15:21:06 -0000 Subject: [Bro-Dev] #612: Segfault in identify_data BiF In-Reply-To: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> References: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> Message-ID: <063.66a127a30b52d9184160fb3a9796c698@tracker.bro-ids.org> #612: Segfault in identify_data BiF ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by robin): On Fri, Sep 23, 2011 at 15:12 -0000, you wrote: > The crash is maybe not so rare. Robin had one too (#627). I saw it within minutes of starting up on campus, though only once and not again after restarting the worker. Fixing #628 could be very helpful to track this down. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 23 08:34:10 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 15:34:10 -0000 Subject: [Bro-Dev] #488: Bug in HTTP parser In-Reply-To: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> References: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> Message-ID: <062.ee7bda75cd1e09f8794db2aa018d464a@tracker.bro-ids.org> #488: Bug in HTTP parser ----------------------+------------------------ Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------+------------------------ Comment (by jsiwek): In [ee302681649df3d3f0398fed560639c1321da636/bro]: {{{ #!CommitTicketReference repository="bro" revision="ee302681649df3d3f0398fed560639c1321da636" Teach HTTP parser to derive content length of multipart/byteranges bodies. Addresses #488. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 23 09:41:22 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 16:41:22 -0000 Subject: [Bro-Dev] #621: Re: PCAP_PF_RING_CLUSTER_ID In-Reply-To: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> References: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> Message-ID: <062.bf42c5f5554fa541bf2a25abdd5ed97e@tracker.bro-ids.org> #621: Re: [Bro-Dev] PCAP_PF_RING_CLUSTER_ID ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: | Keywords: beta ----------------------------+------------------------ Comment (by jsiwek): In [72483b4c6442ceaa57e6c470ab939698ddac149c/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="72483b4c6442ceaa57e6c470ab939698ddac149c" Move configuration of PFRINGClusterID from broctl.cfg.in to options.py. Addresses #621. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 23 10:04:27 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 17:04:27 -0000 Subject: [Bro-Dev] #488: Bug in HTTP parser In-Reply-To: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> References: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> Message-ID: <062.afd861b81b255b9357dd840595ccd4e0@tracker.bro-ids.org> #488: Bug in HTTP parser ----------------------------+------------------------ Reporter: robin | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Changes (by jsiwek): * type: Problem => Merge Request Comment: ` topic/jsiwek/http-multipart-byteranges` fixes the problem with the parser not being able to distinguish the different multipart bodies and the test now passes. I didn't replace any other ASSERTs with weirds, think that task is ok to push back until later? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Fri Sep 23 13:06:02 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 23 Sep 2011 20:06:02 -0000 Subject: [Bro-Dev] #615: Cluster manager crash In-Reply-To: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> References: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> Message-ID: <061.d187c0842e06070b6e8bb93e0eb1aede@tracker.bro-ids.org> #615: Cluster manager crash ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by robin): Back to running a 26-node cluster, I'm seeing this crash roughly every 20-30 minutes now. The ``RemoteSerializer::ProcessLogWrite`` in there seems to suggest a problem with the new log-related code. I'll try it with a debug version next to see if that gives us useful core dump. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sat Sep 24 21:32:06 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 25 Sep 2011 04:32:06 -0000 Subject: [Bro-Dev] #628: SaveTraces doesn't rotate In-Reply-To: <047.6606d3118ed306c827488fe7cefe653b@tracker.bro-ids.org> References: <047.6606d3118ed306c827488fe7cefe653b@tracker.bro-ids.org> Message-ID: <062.f34db2cfe69d25ed63fcf0940748046c@tracker.bro-ids.org> #628: SaveTraces doesn't rotate ---------------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: Works for Me | Keywords: beta ---------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Works for Me Comment: This is done by scripts/policy/misc/trim-trace-file.bro which is loaded by default on the workers. It works fine for me with and without broctl. You can configure the rotation interval (default remains 5 minutes) with the TrimTraceFile::trim_interval variable. I'm going to close this ticket. If you are still having the problem, go ahead and reopen. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 25 15:07:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 25 Sep 2011 22:07:31 -0000 Subject: [Bro-Dev] #574: More exessive memory usage In-Reply-To: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> References: <048.a8c38567bae04a69ed948eb2d38bbce7@tracker.bro-ids.org> Message-ID: <063.e52bc6085d5c3a46a3b01f3c642a2341@tracker.bro-ids.org> #574: More exessive memory usage ----------------------+------------------------ Reporter: gregor | Owner: robin Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: beta ----------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [3b7806379cc9336b938a173014c42657ad653ee5/bro]: {{{ #!CommitTicketReference repository="bro" revision="3b7806379cc9336b938a173014c42657ad653ee5" Adding &transient attribute to top-level SSL pac array type. This avoids complete buffering of all SSL sessions in memory. Closes #574. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 25 15:07:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 25 Sep 2011 22:07:31 -0000 Subject: [Bro-Dev] #498: Efficiency problem with remote log flushing In-Reply-To: <047.837129432a53b3b6b387597dd321a7c9@tracker.bro-ids.org> References: <047.837129432a53b3b6b387597dd321a7c9@tracker.bro-ids.org> Message-ID: <062.280bb1f3813ff578a08ec3d662ed1221@tracker.bro-ids.org> #498: Efficiency problem with remote log flushing ----------------------+------------------------ Reporter: robin | Owner: robin Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: fixed | Keywords: beta ----------------------+------------------------ Changes (by robin): * owner: => robin * status: new => closed * resolution: => fixed Comment: In [d289db34db21c6cfdecc815111de453e297728c0/bro]: {{{ #!CommitTicketReference repository="bro" revision="d289db34db21c6cfdecc815111de453e297728c0" Communication fix and extension. - Removing unnecessary log flushing. Closes #498. - Adding new BiF disconnect() that shuts a connection to a peer down. - terminate_connection() now first flushes any still buffered log messages. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Sun Sep 25 15:09:28 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 25 Sep 2011 15:09:28 -0700 Subject: [Bro-Dev] manager-child cpu utilization? In-Reply-To: <4E7575A4.1000006@lbl.gov> References: <6E371FAC-03B4-4EC5-B1F2-5DAC1C5719B5@icir.org> <4E7575A4.1000006@lbl.gov> Message-ID: <20110925220928.GA90668@icir.org> Turns out that at least some of this was caused by the inefficiency described in #498. That's fixed now, and I see much lower utilization for the child process now. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Sun Sep 25 15:11:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sun, 25 Sep 2011 22:11:31 -0000 Subject: [Bro-Dev] #615: Cluster manager crash In-Reply-To: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> References: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> Message-ID: <061.08cd50458f9a51dba4a4ab1af4183039@tracker.bro-ids.org> #615: Cluster manager crash ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+-------------------- Comment (by robin): With the fix for #498, I'm not seeing this anymore right now. Will keep and eye on it. Please let me know if anybody still runs into this, otherwise we can close the ticket. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 25 17:14:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 26 Sep 2011 00:14:07 -0000 Subject: [Bro-Dev] #629: Cluster script load order Message-ID: <047.3262250a91e9b4a9f1aae8f971e72bcd@tracker.bro-ids.org> #629: Cluster script load order ------------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Keywords: beta | ------------------------+-------------------- {{{ > > broctl scripts manager 2>&1 | grep manager manager is ok. 3 /xa/bro/share/bro/base/frameworks/cluster/./nodes/manager.bro 4 /xa/bro/share/bro/site/local-manager.bro 0 /xa/bro/share/bro/broctl/nodes/manager.bro }}} That means that I can't override options set in ``broctl.nodes/manager.bro`` in ``local-manager.bro``, which in particular means I can't redefine the ``default_rotation_internval``. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 25 17:18:18 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 26 Sep 2011 00:18:18 -0000 Subject: [Bro-Dev] #630: Adding log rotation interval option to broctl.cfg Message-ID: <047.c5fa14817d6b5013bdabf7e5c3698edc@tracker.bro-ids.org> #630: Adding log rotation interval option to broctl.cfg -----------------------------+-------------------- Reporter: robin | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Keywords: beta | -----------------------------+-------------------- It would be good if the default log rotation interval could be changed in ``broctl.cfg``. That seems to be one the few basic options that people may want to change (I certainly don't like the 1hr default :). But that should only apply to the manager so we can't just add the it to ``auto/broctl-config.bro``. Do we need a separate ``auto/broctl-config- manager.bro``? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Sun Sep 25 19:01:13 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 26 Sep 2011 02:01:13 -0000 Subject: [Bro-Dev] #630: Adding log rotation interval option to broctl.cfg In-Reply-To: <047.c5fa14817d6b5013bdabf7e5c3698edc@tracker.bro-ids.org> References: <047.c5fa14817d6b5013bdabf7e5c3698edc@tracker.bro-ids.org> Message-ID: <062.4e218c271543d7e9fd4e8b0f5518ce31@tracker.bro-ids.org> #630: Adding log rotation interval option to broctl.cfg ------------------------------+-------------------- Reporter: robin | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Resolution: | Keywords: beta ------------------------------+-------------------- Comment (by seth): > It would be good if the default log rotation interval could be changed in > ``broctl.cfg``. That seems to be one the few basic options that people may > want to change (I certainly don't like the 1hr default :). Hah, to each his own. I agree though, it does make sense to have it as a broctl.cfg option. > But that should only apply to the manager so we can't just add the it to > ``auto/broctl-config.bro``. Do we need a separate ``auto/broctl-config- > manager.bro``? You can do the preprocessor-like trick with the cluster framework. It may make the most sense to just write out some "manager only" parts of the broctl-config file. Like this: {{{ @if ( Cluster::local_node_type() == Cluster::MANAGER ) ... @endif }}} I would rather avoid creating more autogenerated files if at all possible. Readability shouldn't be a problem anyway since the autogenerated scripts are never meant to be read by users anyway. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Sun Sep 25 19:25:51 2011 From: robin at icir.org (Robin Sommer) Date: Sun, 25 Sep 2011 19:25:51 -0700 Subject: [Bro-Dev] Leak Message-ID: <20110926022551.GB2372@icir.org> I'm pretty sure we still have a leak (or overagressive state-keeping) somewhere, my workers on the blades all went up to 4G of memory within a few hours this afternoon. Is that a "know problem" seen elsewhere as well? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Mon Sep 26 05:10:10 2011 From: seth at icir.org (Seth Hall) Date: Mon, 26 Sep 2011 08:10:10 -0400 Subject: [Bro-Dev] Leak In-Reply-To: <20110926022551.GB2372@icir.org> References: <20110926022551.GB2372@icir.org> Message-ID: <342998F2-E329-4A99-AE07-809C8032B287@icir.org> On Sep 25, 2011, at 10:25 PM, Robin Sommer wrote: > Is that a "know problem" seen elsewhere as well? I've seen this to a degree but I'm not currently seeing it. It's been a little difficult to figure out with the SSL issues and other problems we've had lately. I'll keep an eye on it where I can though. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From mcholste at gmail.com Mon Sep 26 06:39:21 2011 From: mcholste at gmail.com (Martin Holste) Date: Mon, 26 Sep 2011 08:39:21 -0500 Subject: [Bro-Dev] manager crash Message-ID: Got this: internal error: unknown msg type 101 in Poll() /usr/local/bro-git-20110925/share/broctl/scripts/run-bro: line 60: 4445 Aborted (core dumped) nohup $mybro $@ from revision v1.6-dev-1302-g827dcea I'll try with the latest. From mcholste at gmail.com Mon Sep 26 06:43:23 2011 From: mcholste at gmail.com (Martin Holste) Date: Mon, 26 Sep 2011 08:43:23 -0500 Subject: [Bro-Dev] Leak In-Reply-To: <342998F2-E329-4A99-AE07-809C8032B287@icir.org> References: <20110926022551.GB2372@icir.org> <342998F2-E329-4A99-AE07-809C8032B287@icir.org> Message-ID: I've never gotten better than 2 GB per worker, though all my traffic has a lot of SSL. On Mon, Sep 26, 2011 at 7:10 AM, Seth Hall wrote: > > On Sep 25, 2011, at 10:25 PM, Robin Sommer wrote: > >> Is that a "know problem" seen elsewhere as well? > > > I've seen this to a degree but I'm not currently seeing it. ?It's been a little difficult to figure out with the SSL issues and other problems we've had lately. ?I'll keep an eye on it where I can though. > > ?.Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From seth at icir.org Mon Sep 26 08:28:18 2011 From: seth at icir.org (Seth Hall) Date: Mon, 26 Sep 2011 11:28:18 -0400 Subject: [Bro-Dev] Leak In-Reply-To: References: <20110926022551.GB2372@icir.org> <342998F2-E329-4A99-AE07-809C8032B287@icir.org> Message-ID: On Sep 26, 2011, at 9:43 AM, Martin Holste wrote: > I've never gotten better than 2 GB per worker, though all my traffic > has a lot of SSL. Make sure you update to the commit that Robin just did over the weekend and update all of the submodules. git pull git submodule update --init --recursive cd build make make install The *real* SSL memory problem fix is finally made in binpac. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Mon Sep 26 08:50:01 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 26 Sep 2011 08:50:01 -0700 Subject: [Bro-Dev] manager crash In-Reply-To: References: Message-ID: <20110926155001.GE13358@icir.org> On Mon, Sep 26, 2011 at 08:39 -0500, you wrote: > internal error: unknown msg type 101 in Poll() > from revision v1.6-dev-1302-g827dcea Hmm... I would have suggested trying the latest logging fix, but that's already in there. :( In the past, we have seen this error in one of two cases: (1) communication overload, i.e., a node receives more messages than it can handle, usually noticable by extremely high CPU load; or (2) *another* node crashes for whatever reason and that then causes some peers to crash as well with this error. Does it look like one of these two? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From mcholste at gmail.com Mon Sep 26 08:57:50 2011 From: mcholste at gmail.com (Martin Holste) Date: Mon, 26 Sep 2011 10:57:50 -0500 Subject: [Bro-Dev] Leak In-Reply-To: References: <20110926022551.GB2372@icir.org> <342998F2-E329-4A99-AE07-809C8032B287@icir.org> Message-ID: Yep, did a new build this morning. I'll keep you posted. On Mon, Sep 26, 2011 at 10:28 AM, Seth Hall wrote: > > On Sep 26, 2011, at 9:43 AM, Martin Holste wrote: > >> I've never gotten better than 2 GB per worker, though all my traffic >> has a lot of SSL. > > > Make sure you update to the commit that Robin just did over the weekend and update all of the submodules. > > git pull > git submodule update --init --recursive > cd build > make > make install > > The *real* SSL memory problem fix is finally made in binpac. > > ?.Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro-ids.org/ > > From mcholste at gmail.com Mon Sep 26 08:59:00 2011 From: mcholste at gmail.com (Martin Holste) Date: Mon, 26 Sep 2011 10:59:00 -0500 Subject: [Bro-Dev] manager crash In-Reply-To: <20110926155001.GE13358@icir.org> References: <20110926155001.GE13358@icir.org> Message-ID: Well, there is high CPU and high volume almost all of the time, and all of the workers are still up and running, so this seems to be a volume issue. On Mon, Sep 26, 2011 at 10:50 AM, Robin Sommer wrote: > > On Mon, Sep 26, 2011 at 08:39 -0500, you wrote: > >> internal error: unknown msg type 101 in Poll() >> from revision v1.6-dev-1302-g827dcea > > Hmm... I would have suggested trying the latest logging fix, but > that's already in there. :( > > In the past, we have seen this error in one of two cases: (1) > communication overload, i.e., a node receives more messages than it > can handle, usually noticable by extremely high CPU load; or (2) > *another* node crashes for whatever reason and that then causes some > peers to crash as well with this error. > > Does it look like one of these two? > > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL ? ?* Fax ? +1 (510) 666-2956 * ? www.icir.org > From jsiwek at ncsa.illinois.edu Mon Sep 26 09:05:25 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 26 Sep 2011 11:05:25 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Updating some test baselines. (827dcea) In-Reply-To: <201109251209.p8PC9g6K011331@bro-ids.icir.org> References: <201109251209.p8PC9g6K011331@bro-ids.icir.org> Message-ID: <54B5D15A-9372-4BB6-BBBD-620A81BB3544@ncsa.illinois.edu> > - I see 5 broken tests currently. Me, too: doc.coverage ? failed This will fail whenever new bro scripts are added, but it's easy to fix: as the test comments indicate, run the `doc/scripts/genDocSourcesList.sh` script. Maybe later, if this is reliable enough, we can make it happen automatically on commits or something. scripts.bare-mode-coverage ? failed This is an indicator that there's some errors when loading individual scripts in bare-mode. It's not going to fully work until something is done about http://tracker.bro-ids.org/bro/ticket/544 (hot.conn.bro & scan.bro), but the diagnostic output is still useful to correct other errors. scripts.test-all-policy-coverage ? failed scripts.base.init-default-coverage ? failed Also get out of date when new scripts are added to either policy/ or base/ -- fixing them should be inside your domain? :) scripts.base.protocols.http.http-header-crlf ? failed I have a fix waiting merge in #488. So we're not far away from having everything pass. - Jon From robin at icir.org Mon Sep 26 09:06:27 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 26 Sep 2011 09:06:27 -0700 Subject: [Bro-Dev] manager crash In-Reply-To: References: <20110926155001.GE13358@icir.org> Message-ID: <20110926160627.GI13358@icir.org> On Mon, Sep 26, 2011 at 10:59 -0500, you wrote: > Well, there is high CPU and high volume almost all of the time, What's the typical CPU load on the manager with the fixes from the weekend applied? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Mon Sep 26 09:11:04 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 26 Sep 2011 09:11:04 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Updating some test baselines. (827dcea) In-Reply-To: <54B5D15A-9372-4BB6-BBBD-620A81BB3544@ncsa.illinois.edu> References: <201109251209.p8PC9g6K011331@bro-ids.icir.org> <54B5D15A-9372-4BB6-BBBD-620A81BB3544@ncsa.illinois.edu> Message-ID: <20110926161104.GJ13358@icir.org> On Mon, Sep 26, 2011 at 11:05 -0500, you wrote: > So we're not far away from having everything pass. Yippie. :) Let's put the notes re the tests that need updating when scripts change into something like "btest/NOTES.tests". That's a good reference on what to do. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From mcholste at gmail.com Mon Sep 26 09:11:21 2011 From: mcholste at gmail.com (Martin Holste) Date: Mon, 26 Sep 2011 11:11:21 -0500 Subject: [Bro-Dev] manager crash In-Reply-To: <20110926160627.GI13358@icir.org> References: <20110926155001.GE13358@icir.org> <20110926160627.GI13358@icir.org> Message-ID: The manager processes are among the lowest for CPU utilization. The workers are top. top - 11:06:43 up 12 days, 22:51, 1 user, load average: 6.30, 5.90, 5.46 Tasks: 195 total, 16 running, 179 sleeping, 0 stopped, 0 zombie Cpu(s): 1.7%us, 1.9%sy, 0.3%ni, 95.3%id, 0.0%wa, 0.1%hi, 0.7%si, 0.0%st Mem: 66176992k total, 6142712k used, 60034280k free, 155124k buffers Swap: 17358840k total, 0k used, 17358840k free, 1129988k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15361 root 20 0 315m 275m 134m R 57 0.4 18:44.79 bro 15357 root 20 0 317m 282m 137m S 55 0.4 16:32.71 bro 15363 root 20 0 312m 276m 137m R 47 0.4 16:55.25 bro 15355 root 20 0 314m 277m 137m R 43 0.4 18:22.26 bro 15360 root 20 0 308m 268m 134m R 43 0.4 16:30.32 bro 15359 root 20 0 326m 290m 137m R 39 0.5 20:11.75 bro 15364 root 20 0 321m 284m 137m R 39 0.4 17:54.01 bro 15362 root 20 0 309m 274m 134m R 37 0.4 18:31.64 bro 15193 root 20 0 82336 44m 3836 S 25 0.1 9:17.12 bro 15227 root 25 5 76420 25m 508 R 18 0.0 7:17.88 bro 15365 root 25 5 188m 145m 128m R 18 0.2 5:15.67 bro 15367 root 25 5 188m 145m 128m R 16 0.2 5:11.59 bro 15226 root 20 0 81832 43m 3812 S 14 0.1 9:01.03 bro 15368 root 25 5 188m 145m 128m S 14 0.2 5:11.62 bro 15372 root 25 5 188m 146m 128m S 14 0.2 5:26.85 bro 15369 root 25 5 188m 145m 128m S 12 0.2 5:14.59 bro 15370 root 25 5 188m 145m 128m R 12 0.2 5:12.91 bro 15371 root 25 5 188m 145m 128m S 12 0.2 5:19.80 bro 15194 root 25 5 76452 22m 508 R 10 0.0 7:03.27 bro 15366 root 25 5 188m 145m 128m R 4 0.2 5:18.45 bro On Mon, Sep 26, 2011 at 11:06 AM, Robin Sommer wrote: > > On Mon, Sep 26, 2011 at 10:59 -0500, you wrote: > >> Well, there is high CPU and high volume almost all of the time, > > What's the typical CPU load on the manager with the fixes from the > weekend applied? > > Robin > > -- > Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org > ICSI/LBNL ? ?* Fax ? +1 (510) 666-2956 * ? www.icir.org > From robin at icir.org Mon Sep 26 09:18:17 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 26 Sep 2011 09:18:17 -0700 Subject: [Bro-Dev] manager crash In-Reply-To: References: <20110926155001.GE13358@icir.org> <20110926160627.GI13358@icir.org> Message-ID: <20110926161817.GA52412@icir.org> On Mon, Sep 26, 2011 at 11:11 -0500, you wrote: > The manager processes are among the lowest for CPU utilization. The > workers are top. That's what I thought, and then normally I wouldn't attribute the crash to the manager's load. However, I forgot that this is all running on a single box where the manager may just not get sufficient cycles to process what it gets. If you see these crashes regularly, it may be worth playing with some scheduling parameters, like pinning the manager process to its own core. Robin PS: To be very clear, these 101 crashes should never happen at all; it's a bug somewhere in the communication code that has evaded detection for a while already. If anything, Bro should tear down the connection regularly if it can't deal with what it gets. -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Mon Sep 26 09:19:24 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 26 Sep 2011 11:19:24 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Updating some test baselines. (827dcea) In-Reply-To: <20110926161104.GJ13358@icir.org> References: <201109251209.p8PC9g6K011331@bro-ids.icir.org> <54B5D15A-9372-4BB6-BBBD-620A81BB3544@ncsa.illinois.edu> <20110926161104.GJ13358@icir.org> Message-ID: > re the tests that need updating when scripts > change into something like "btest/NOTES.tests". That's a good > reference on what to do. I think it's better if the test file itself has some comments about how to correct failures that might be fairly regular because it's the more obvious place to look for someone trying to fix a failure (they're going to look at what the test actually does and find the comments that way), plus if someone changes the test, they're more likely to remember to update the documentation that goes along with it if it's in the same file. - Jon From seth at icir.org Mon Sep 26 09:22:45 2011 From: seth at icir.org (Seth Hall) Date: Mon, 26 Sep 2011 12:22:45 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Updating some test baselines. (827dcea) In-Reply-To: <54B5D15A-9372-4BB6-BBBD-620A81BB3544@ncsa.illinois.edu> References: <201109251209.p8PC9g6K011331@bro-ids.icir.org> <54B5D15A-9372-4BB6-BBBD-620A81BB3544@ncsa.illinois.edu> Message-ID: <2F3A3416-10F2-449D-B232-41F183D849CD@icir.org> On Sep 26, 2011, at 12:05 PM, Jonathan Siwek wrote: > scripts.bare-mode-coverage ? failed > > This is an indicator that there's some errors when loading individual scripts in bare-mode. It's not going to fully work until something is done about http://tracker.bro-ids.org/bro/ticket/544 (hot.conn.bro & scan.bro), I'm working on those. > So we're not far away from having everything pass. Thanks for the notes and I agree with Robin that having those documented in a readme would be awesome. I suppose it should also be added to the developer documentation. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Mon Sep 26 09:24:37 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 26 Sep 2011 09:24:37 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Updating some test baselines. (827dcea) In-Reply-To: References: <201109251209.p8PC9g6K011331@bro-ids.icir.org> <54B5D15A-9372-4BB6-BBBD-620A81BB3544@ncsa.illinois.edu> <20110926161104.GJ13358@icir.org> Message-ID: <20110926162437.GB52412@icir.org> On Mon, Sep 26, 2011 at 11:19 -0500, you wrote: > I think it's better if the test file itself has some comments about I think both would be good. The NOTES files is more helpful to change things in advance, i.e., "I just added a script; which tests do I need to adapt?". It would be fine though to just list the relevant tests there, and leave the docs on what specficially to do in the test files itself so that we don't have two places to keep in sync. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Mon Sep 26 09:25:57 2011 From: seth at icir.org (Seth Hall) Date: Mon, 26 Sep 2011 12:25:57 -0400 Subject: [Bro-Dev] manager crash In-Reply-To: References: <20110926155001.GE13358@icir.org> Message-ID: <4962F21A-FA9F-4987-BB9C-65A76C55C5C9@icir.org> On Sep 26, 2011, at 11:59 AM, Martin Holste wrote: > Well, there is high CPU and high volume almost all of the time, and > all of the workers are still up and running, so this seems to be a > volume issue. Is your weird.log file oddly large? I've been seeing communications overload occasionally that is in part due to a lot of weird log messages. I'm becoming really tempted to turn off weird messages but measure some of them through the metrics framework to find where there might be issues (due to checksum offloading, async routing, load balancing problems etc, abundance of out of order traffic, etc). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From gregor at icir.org Mon Sep 26 09:51:04 2011 From: gregor at icir.org (Gregor Maier) Date: Mon, 26 Sep 2011 09:51:04 -0700 Subject: [Bro-Dev] manager crash In-Reply-To: <4962F21A-FA9F-4987-BB9C-65A76C55C5C9@icir.org> References: <20110926155001.GE13358@icir.org> <4962F21A-FA9F-4987-BB9C-65A76C55C5C9@icir.org> Message-ID: <4E80AD78.6060309@icir.org> On 9/26/11 9:25 , Seth Hall wrote: > > On Sep 26, 2011, at 11:59 AM, Martin Holste wrote: > >> Well, there is high CPU and high volume almost all of the time, and >> all of the workers are still up and running, so this seems to be a >> volume issue. > > > Is your weird.log file oddly large? I've been seeing communications overload occasionally that is in part due to a lot of weird log messages. I'm becoming really tempted to turn off weird messages but measure some of them through the metrics framework to find where there might be issues (due to checksum offloading, async routing, load balancing problems etc, abundance of out of order traffic, etc). FWIW, I often get tons of weirds from the DNS scripts. (The scripts appear to get confused with the number of answers to expect) cu gregor -- Gregor Maier Int. Computer Science Institute (ICSI) 1947 Center St., Ste. 600 Berkeley, CA 94704, USA http://www.icir.org/gregor/ From mcholste at gmail.com Mon Sep 26 09:58:25 2011 From: mcholste at gmail.com (Martin Holste) Date: Mon, 26 Sep 2011 11:58:25 -0500 Subject: [Bro-Dev] manager crash In-Reply-To: <4E80AD78.6060309@icir.org> References: <20110926155001.GE13358@icir.org> <4962F21A-FA9F-4987-BB9C-65A76C55C5C9@icir.org> <4E80AD78.6060309@icir.org> Message-ID: I disable weird logs through disable_stream(). On Mon, Sep 26, 2011 at 11:51 AM, Gregor Maier wrote: > On 9/26/11 9:25 , Seth Hall wrote: >> >> On Sep 26, 2011, at 11:59 AM, Martin Holste wrote: >> >>> Well, there is high CPU and high volume almost all of the time, and >>> all of the workers are still up and running, so this seems to be a >>> volume issue. >> >> >> Is your weird.log file oddly large? ?I've been seeing communications >> overload occasionally that is in part due to a lot of weird log messages. >> ?I'm becoming really tempted to turn off weird messages but measure some of >> them through the metrics framework to find where there might be issues (due >> to checksum offloading, async routing, load balancing problems etc, >> abundance of out of order traffic, etc). > > > FWIW, > I often get tons of weirds from the DNS scripts. (The scripts appear to get > confused with the number of answers to expect) > > cu > gregor > > -- > Gregor Maier > ? > Int. Computer Science Institute (ICSI) > 1947 Center St., Ste. 600 > Berkeley, CA 94704, USA > http://www.icir.org/gregor/ > From jsiwek at ncsa.illinois.edu Mon Sep 26 10:04:07 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 26 Sep 2011 12:04:07 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Updating some test baselines. (827dcea) In-Reply-To: <20110926162437.GB52412@icir.org> References: <201109251209.p8PC9g6K011331@bro-ids.icir.org> <54B5D15A-9372-4BB6-BBBD-620A81BB3544@ncsa.illinois.edu> <20110926161104.GJ13358@icir.org> <20110926162437.GB52412@icir.org> Message-ID: <7A552369-D933-4548-BC90-2747E23E469E@ncsa.illinois.edu> > I think both would be good. The NOTES files is more helpful to change > things in advance, i.e., "I just added a script; which tests do I need > to adapt?". Changing things in advance without actually running the tests to make sure it works as expected seems dangerous. I was thinking that part of the standard development process would be for one to run the test suite themselves before requesting a merge (or doing a merge) to master. And in that case, the question is answered by the act of just running the test suite. If running the test suite manually isn't part of the process, then I don't think any extra amount of documentation is going to help prevent new script commits resulting in breaking of tests (the automated NMI B&T is best used to catch platform-specific problems, not as a general way to catch problems after they've already been committed.) - Jon From seth at icir.org Mon Sep 26 10:24:00 2011 From: seth at icir.org (Seth Hall) Date: Mon, 26 Sep 2011 13:24:00 -0400 Subject: [Bro-Dev] manager crash In-Reply-To: <4E80AD78.6060309@icir.org> References: <20110926155001.GE13358@icir.org> <4962F21A-FA9F-4987-BB9C-65A76C55C5C9@icir.org> <4E80AD78.6060309@icir.org> Message-ID: <4E418BFB-266D-42F9-A8E9-877BE0570EC2@icir.org> On Sep 26, 2011, at 12:51 PM, Gregor Maier wrote: > FWIW, > I often get tons of weirds from the DNS scripts. (The scripts appear to get confused with the number of answers to expect) I'm planning on addressing that soon, those weirds aren't very good. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From seth at icir.org Mon Sep 26 10:25:36 2011 From: seth at icir.org (Seth Hall) Date: Mon, 26 Sep 2011 13:25:36 -0400 Subject: [Bro-Dev] manager crash In-Reply-To: References: <20110926155001.GE13358@icir.org> <4962F21A-FA9F-4987-BB9C-65A76C55C5C9@icir.org> <4E80AD78.6060309@icir.org> Message-ID: <46431EDF-B1F9-4BB1-A30A-DB07FD07344A@icir.org> On Sep 26, 2011, at 12:58 PM, Martin Holste wrote: > I disable weird logs through disable_stream(). Ah, that's right. Robin, if a stream is disabled that causes a remote logging host to stop trying to send the log to it's log-accepting peer, correct? I just want to verify that the stream is as disabled as I'm assuming it is. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From bro at tracker.bro-ids.org Mon Sep 26 12:27:20 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 26 Sep 2011 19:27:20 -0000 Subject: [Bro-Dev] #631: Special message for broctl locking when done by cron Message-ID: <046.60d83c62aa090d2979230708fe26b94e@tracker.bro-ids.org> #631: Special message for broctl locking when done by cron -----------------------------+-------------------- Reporter: seth | Owner: Type: Feature Request | Status: new Priority: Normal | Milestone: Bro1.7 Component: BroControl | Version: Keywords: | -----------------------------+-------------------- If the broctl lock is being held by the cron command it would be nice if the message that indicates a lock is already held would indicate if it is the cron command. If multiple people are working with broctl the person that gets a lock doesn't know if it's because of another user or because they happened to be trying to do something while the cron command is running. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Mon Sep 26 12:40:26 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 26 Sep 2011 12:40:26 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Updating some test baselines. (827dcea) In-Reply-To: <7A552369-D933-4548-BC90-2747E23E469E@ncsa.illinois.edu> References: <201109251209.p8PC9g6K011331@bro-ids.icir.org> <54B5D15A-9372-4BB6-BBBD-620A81BB3544@ncsa.illinois.edu> <20110926161104.GJ13358@icir.org> <20110926162437.GB52412@icir.org> <7A552369-D933-4548-BC90-2747E23E469E@ncsa.illinois.edu> Message-ID: <20110926194026.GA60403@icir.org> On Mon, Sep 26, 2011 at 12:04 -0500, Jonathan Siwek wrote: > Changing things in advance without actually running the tests to make > sure it works as expected seems dangerous. That's not what I wanted to say. :) Of course one still needs to run the tests before pushing things further upstream. But it's easier to first adapt the tests and then run them, as otherwise one gets the failures and then needs to decide one by one whether it's expected to break or a new problem. Better to get the expected stuff out of the way first. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From jsiwek at ncsa.illinois.edu Mon Sep 26 13:42:40 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Mon, 26 Sep 2011 15:42:40 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Updating some test baselines. (827dcea) In-Reply-To: <20110926194026.GA60403@icir.org> References: <201109251209.p8PC9g6K011331@bro-ids.icir.org> <54B5D15A-9372-4BB6-BBBD-620A81BB3544@ncsa.illinois.edu> <20110926161104.GJ13358@icir.org> <20110926162437.GB52412@icir.org> <7A552369-D933-4548-BC90-2747E23E469E@ncsa.illinois.edu> <20110926194026.GA60403@icir.org> Message-ID: <2CE191AC-D91E-4AA6-B0D8-FB54CA89092D@ncsa.illinois.edu> >> Changing things in advance without actually running the tests to make >> sure it works as expected seems dangerous. > > That's not what I wanted to say. :) Of course one still needs to run > the tests before pushing things further upstream. Ah, ok. > But it's easier to first adapt the tests and then run them Maybe it's just more my style, but I'd rather see what the failure looks like first before adapting anything. Otherwise I might be adapting something that didn't need adapting or adapting it in the wrong way such that I'll still have to fix it later. I'm still thinking that a NOTES file that maintains a list of tests that are expected to fail more often because they're coverage-type tests is going become out-of-date easily. Supporting evidence: I just noticed that the testing/btest/README is in need of significant updating because it tried to maintain a list of major test directories. Right now all the tests that are expected or more likely to break when new code is added have "coverage" somewhere in the name. What do you think about grouping them all in a single "coverage" directory to keep that distinction, but then also provide a canonical place for preemptive test-fixers to look? - Jon From robin at icir.org Mon Sep 26 13:48:42 2011 From: robin at icir.org (Robin Sommer) Date: Mon, 26 Sep 2011 13:48:42 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Updating some test baselines. (827dcea) In-Reply-To: <2CE191AC-D91E-4AA6-B0D8-FB54CA89092D@ncsa.illinois.edu> References: <201109251209.p8PC9g6K011331@bro-ids.icir.org> <54B5D15A-9372-4BB6-BBBD-620A81BB3544@ncsa.illinois.edu> <20110926161104.GJ13358@icir.org> <20110926162437.GB52412@icir.org> <7A552369-D933-4548-BC90-2747E23E469E@ncsa.illinois.edu> <20110926194026.GA60403@icir.org> <2CE191AC-D91E-4AA6-B0D8-FB54CA89092D@ncsa.illinois.edu> Message-ID: <20110926204842.GE54358@icir.org> On Mon, Sep 26, 2011 at 15:42 -0500, you wrote: > new code is added have "coverage" somewhere in the name. What do you > think about grouping them all in a single "coverage" directory to keep Ok, I like that. Robun -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Mon Sep 26 15:51:38 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 26 Sep 2011 22:51:38 -0000 Subject: [Bro-Dev] #411: Non-binpac HTTP analyzer fails on 1xx response codes In-Reply-To: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> References: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> Message-ID: <061.20cf9c42e0bd98aa73670d24ad277bf3@tracker.bro-ids.org> #411: Non-binpac HTTP analyzer fails on 1xx response codes ----------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------+-------------------- Comment (by jsiwek): In [64e821624b34dcb886599a5e478f0a386f26e569/bro]: {{{ #!CommitTicketReference repository="bro" revision="64e821624b34dcb886599a5e478f0a386f26e569" Fix handling of HTTP 1xx response codes (addresses #411). Changed the parser to not treat 1xx response codes as a final answer to an unanswered request -- a later response is still expected. The scripting layer will also not finish a request-reply pair when seeing 1xx's, instead it logs both the 1xx and final response messages with associated information of the current request as they're seen. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Mon Sep 26 15:53:40 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Mon, 26 Sep 2011 22:53:40 -0000 Subject: [Bro-Dev] #411: Non-binpac HTTP analyzer fails on 1xx response codes In-Reply-To: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> References: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> Message-ID: <061.6d055baa3ddee24dd3fd0479aa5bf409@tracker.bro-ids.org> #411: Non-binpac HTTP analyzer fails on 1xx response codes ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------------+-------------------- Changes (by jsiwek): * type: Problem => Merge Request Comment: See if you like those changes in `topic/jsiwek/http-1xx-replies` ? -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 27 11:11:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 27 Sep 2011 18:11:09 -0000 Subject: [Bro-Dev] #626: Known module Info record conflicts In-Reply-To: <048.84bf7ecc0b805ee86009860897680327@tracker.bro-ids.org> References: <048.84bf7ecc0b805ee86009860897680327@tracker.bro-ids.org> Message-ID: <063.a6235f3e8d0ac87ce725b77163a6d49b@tracker.bro-ids.org> #626: Known module Info record conflicts -----------------------------+------------------------ Reporter: jsiwek | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by jsiwek): * status: new => closed * resolution: => Solved/Applied Comment: Looks fixed in [43ea0ba182e7352544089efeb7e0a17a6e78d833/bro] -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Tue Sep 27 12:17:49 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Tue, 27 Sep 2011 19:17:49 -0000 Subject: [Bro-Dev] #411: Non-binpac HTTP analyzer fails on 1xx response codes In-Reply-To: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> References: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> Message-ID: <061.06553472fa8863bf084828e47c919cae@tracker.bro-ids.org> #411: Non-binpac HTTP analyzer fails on 1xx response codes ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------------+-------------------- Comment (by jsiwek): In [7af3977a5006002593e2b61c0c86265872b2cdcb/bro]: {{{ #!CommitTicketReference repository="bro" revision="7af3977a5006002593e2b61c0c86265872b2cdcb" Change logging of HTTP 1xx responses to occur in their own columns. Instead of as entirely new log lines (addresses #411). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From jsiwek at ncsa.illinois.edu Tue Sep 27 14:57:17 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Tue, 27 Sep 2011 16:57:17 -0500 Subject: [Bro-Dev] non-deterministic notice_policy order Message-ID: <5F146456-5FB5-4848-8689-05D2565B3913@ncsa.illinois.edu> I saw that the suppression-disable.bro test periodically failed because the order of elements in Notice::ordered_policy varies between runs (run `bro -e ''` multiple times, looking at notice_policy.log each time for differences). And I think the reason for that variance is because the table/set hashing is partly computed based on the memory address of PolicyItem$pred functions. If that's right, is there an idea of how to make hash computations for functions deterministic? - Jon From robin at icir.org Wed Sep 28 16:30:07 2011 From: robin at icir.org (Robin Sommer) Date: Wed, 28 Sep 2011 16:30:07 -0700 Subject: [Bro-Dev] non-deterministic notice_policy order In-Reply-To: <5F146456-5FB5-4848-8689-05D2565B3913@ncsa.illinois.edu> References: <5F146456-5FB5-4848-8689-05D2565B3913@ncsa.illinois.edu> Message-ID: <20110928233007.GB63322@icir.org> On Tue, Sep 27, 2011 at 16:57 -0500, you wrote: > If that's right, is there an idea of how to make hash computations for functions deterministic? How about printing out the function's code and hashing that when we're in "deterministic hashing mode" (i.e., have_random_seed() return true)? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From bro at tracker.bro-ids.org Wed Sep 28 17:12:45 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 29 Sep 2011 00:12:45 -0000 Subject: [Bro-Dev] #488: Bug in HTTP parser In-Reply-To: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> References: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> Message-ID: <062.715248b9bcdd354e5006246d5f90d4c9@tracker.bro-ids.org> #488: Bug in HTTP parser ----------------------------+------------------------ Reporter: robin | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): Yes, ok. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 28 17:41:08 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 29 Sep 2011 00:41:08 -0000 Subject: [Bro-Dev] #488: Bug in HTTP parser In-Reply-To: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> References: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> Message-ID: <062.9bd56d843e3ba4e0bb85d9d43c92b3ad@tracker.bro-ids.org> #488: Bug in HTTP parser ----------------------------+------------------------ Reporter: robin | Owner: Type: Merge Request | Status: new Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: ----------------------------+------------------------ Comment (by robin): I'm adding a bit more error handling, it's always good to catch things that aren't as expected. Also a note: the parsing creates a number of new string objects each time. In this case it doesn't really matter because the code won't be executed that often (relative to the rest of the HTTP code), but generally it's better to avoid that and also keep the style consistent with how other code is doing the parsing. The MIME code has some helpers already. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 28 17:52:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 29 Sep 2011 00:52:33 -0000 Subject: [Bro-Dev] #411: Non-binpac HTTP analyzer fails on 1xx response codes In-Reply-To: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> References: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> Message-ID: <061.83a76449fe818bbef37c84e023702a67@tracker.bro-ids.org> #411: Non-binpac HTTP analyzer fails on 1xx response codes ----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: ----------------------------+-------------------- Comment (by robin): In [4d6a90ce89875a0697c5f2e2c0d651a47af9e3dd/bro]: {{{ #!CommitTicketReference repository="bro" revision="4d6a90ce89875a0697c5f2e2c0d651a47af9e3dd" Merge remote-tracking branch 'origin/topic/jsiwek/http-1xx-replies' * origin/topic/jsiwek/http-1xx-replies: Change logging of HTTP 1xx responses to occur in their own columns. Fix handling of HTTP 1xx response codes (addresses #411). }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 28 17:52:33 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 29 Sep 2011 00:52:33 -0000 Subject: [Bro-Dev] #448: Broccoli does not support IPv6 addresses In-Reply-To: <047.24dcf4ee9ba479e14bf9dc723cf7b498@tracker.bro-ids.org> References: <047.24dcf4ee9ba479e14bf9dc723cf7b498@tracker.bro-ids.org> Message-ID: <062.ad0eb86be626e220bc54ea148543b8b0@tracker.bro-ids.org> #448: Broccoli does not support IPv6 addresses -----------------------+---------------------- Reporter: robin | Owner: kreibich Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.7 Component: Broccoli | Version: Resolution: fixed | Keywords: ipv6 -----------------------+---------------------- Changes (by robin): * status: new => closed * resolution: => fixed Comment: In [5f7aed6687560056562ed934d17fdff60e9c2c84/bro]: {{{ #!CommitTicketReference repository="bro" revision="5f7aed6687560056562ed934d17fdff60e9c2c84" Merge remote-tracking branch 'origin/topic/jsiwek/http-multipart- byteranges' * origin/topic/jsiwek/http-multipart-byteranges: Teach HTTP parser to derive content length of multipart/byteranges bodies. Closes #448. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 28 17:52:57 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 29 Sep 2011 00:52:57 -0000 Subject: [Bro-Dev] #621: Re: PCAP_PF_RING_CLUSTER_ID In-Reply-To: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> References: <047.4ecdb6cda6eb54b03d29fa894ad083a4@tracker.bro-ids.org> Message-ID: <062.3587f4e7d8662ee875864066115a02cd@tracker.bro-ids.org> #621: Re: [Bro-Dev] PCAP_PF_RING_CLUSTER_ID ----------------------------+------------------------ Reporter: robin | Owner: jsiwek Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: git/master Resolution: fixed | Keywords: beta ----------------------------+------------------------ Changes (by robin): * status: assigned => closed * resolution: => fixed Comment: In [f90d3eded266b4effbdd607f76768dd010c7f3b5/broctl]: {{{ #!CommitTicketReference repository="broctl" revision="f90d3eded266b4effbdd607f76768dd010c7f3b5" Merge remote-tracking branch 'origin/topic/jsiwek/pfring-configure-check' * origin/topic/jsiwek/pfring-configure-check: Move configuration of PFRINGClusterID from broctl.cfg.in to options.py. Add configure-time check for libpcap PF_RING support. Closes #621. }}} -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 28 17:54:09 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 29 Sep 2011 00:54:09 -0000 Subject: [Bro-Dev] #488: Bug in HTTP parser In-Reply-To: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> References: <047.adda44ca13a0bcadd2291eced501c487@tracker.bro-ids.org> Message-ID: <062.39f77537075cb173d04227bb1f911a8d@tracker.bro-ids.org> #488: Bug in HTTP parser -----------------------------+------------------------ Reporter: robin | Owner: Type: Merge Request | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: Solved/Applied | Keywords: -----------------------------+------------------------ Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Wed Sep 28 17:54:37 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 29 Sep 2011 00:54:37 -0000 Subject: [Bro-Dev] #448: Broccoli does not support IPv6 addresses In-Reply-To: <047.24dcf4ee9ba479e14bf9dc723cf7b498@tracker.bro-ids.org> References: <047.24dcf4ee9ba479e14bf9dc723cf7b498@tracker.bro-ids.org> Message-ID: <062.f70656ad6b4b901621873e011dc0d900@tracker.bro-ids.org> #448: Broccoli does not support IPv6 addresses -----------------------+---------------------- Reporter: robin | Owner: kreibich Type: Problem | Status: reopened Priority: Normal | Milestone: Bro1.7 Component: Broccoli | Version: Resolution: | Keywords: ipv6 -----------------------+---------------------- Changes (by robin): * status: closed => reopened * resolution: fixed => Comment: wrong ticket closes. Reopened. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 29 15:30:47 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 29 Sep 2011 22:30:47 -0000 Subject: [Bro-Dev] #411: Non-binpac HTTP analyzer fails on 1xx response codes In-Reply-To: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> References: <046.8834c8b3115bb22814256a680a5869d1@tracker.bro-ids.org> Message-ID: <061.d12cb832c34282244bdf0eb3251c7710@tracker.bro-ids.org> #411: Non-binpac HTTP analyzer fails on 1xx response codes -----------------------------+-------------------- Reporter: seth | Owner: Type: Merge Request | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 29 15:39:07 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 29 Sep 2011 22:39:07 -0000 Subject: [Bro-Dev] #632: process command does not produce logs in cluster mode Message-ID: <047.ea938b17f19ea4f32708e6af5c0b3a5d@tracker.bro-ids.org> #632: process command does not produce logs in cluster mode ------------------------+-------------------- Reporter: robin | Owner: Type: Problem | Status: new Priority: Normal | Milestone: Bro1.6 Component: BroControl | Version: Keywords: beta | ------------------------+-------------------- When using "process" with a cluster installation, there aren't any log files produced presumably because local logging is turned off. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 29 16:20:02 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 29 Sep 2011 23:20:02 -0000 Subject: [Bro-Dev] #615: Cluster manager crash In-Reply-To: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> References: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> Message-ID: <061.5f9e7b8cb501a998d63f52d875134bd5@tracker.bro-ids.org> #615: Cluster manager crash -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: beta -----------------------------+-------------------- Changes (by robin): * status: new => closed * resolution: => Solved/Applied Comment: Closing. If anybody still sees it, please reopen. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 29 16:21:51 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 29 Sep 2011 23:21:51 -0000 Subject: [Bro-Dev] #612: Segfault in identify_data BiF In-Reply-To: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> References: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> Message-ID: <063.5a5963f35f2fe62e5d5a58af645c80ec@tracker.bro-ids.org> #612: Segfault in identify_data BiF ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Changes (by robin): * priority: Normal => High Comment: I see this regularly but unfortunately can't reproduce with a trace recorded by crashing workers. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 29 16:32:32 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Thu, 29 Sep 2011 23:32:32 -0000 Subject: [Bro-Dev] #612: Segfault in identify_data BiF In-Reply-To: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> References: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> Message-ID: <063.c106fcb7e84a49ebbd56f37b4d58668b@tracker.bro-ids.org> #612: Segfault in identify_data BiF ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by gregor): Did you try to compile it with --enable-debug, so we can see what memory is passed to the libmagic function. Without --enable-debug the arguments are optimized away.... Do you know which analyzer / policy scripts cause the segfaults. Is it several or just one (I got my segfault from a non-HTTP analyzer but don't know which) cu Gregor -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 29 17:52:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 30 Sep 2011 00:52:55 -0000 Subject: [Bro-Dev] #612: Segfault in identify_data BiF In-Reply-To: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> References: <048.2145c0c1e4de755bc9fa18b2c7b0ac68@tracker.bro-ids.org> Message-ID: <063.a6b42e986f3aa233279f4a8993211fee@tracker.bro-ids.org> #612: Segfault in identify_data BiF ----------------------+------------------------ Reporter: gregor | Owner: Type: Problem | Status: new Priority: High | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by robin): > Did you try to compile it with --enable-debug, so we can see what memory > is passed to the libmagic function. Without --enable-debug the arguments > are optimized away.... Not yet, will do that next. First trying to see whether its stable otherwise now. -- Ticket URL: Bro Tracker Bro Issue Tracker From bro at tracker.bro-ids.org Thu Sep 29 18:31:31 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 30 Sep 2011 01:31:31 -0000 Subject: [Bro-Dev] #611: Segfault in strftime BiF In-Reply-To: <046.220cfaf223e8017638bae915af2d1c77@tracker.bro-ids.org> References: <046.220cfaf223e8017638bae915af2d1c77@tracker.bro-ids.org> Message-ID: <061.aeef2a9bb580e30d8fd7d071a7a9bcb2@tracker.bro-ids.org> #611: Segfault in strftime BiF -----------------------------+-------------------- Reporter: seth | Owner: Type: Problem | Status: closed Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: Resolution: Solved/Applied | Keywords: beta -----------------------------+-------------------- Changes (by seth): * status: new => closed * resolution: => Solved/Applied Comment: I haven't seen this crash anymore. -- Ticket URL: Bro Tracker Bro Issue Tracker From robin at icir.org Thu Sep 29 22:47:04 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 29 Sep 2011 22:47:04 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Fix for shutdown bug in http scripts. (012d8cf) In-Reply-To: <201109300125.p8U1PK5S005850@bro-ids.icir.org> References: <201109300125.p8U1PK5S005850@bro-ids.icir.org> Message-ID: <20110930054704.GA66615@icir.org> On Thu, Sep 29, 2011 at 18:25 -0700, Seth Hall wrote: > +++ b/testing/btest/scripts/base/protocols/http/100-continue.bro > +++ b/testing/btest/scripts/base/protocols/http/http-header-crlf.bro Not sure what's going in here but these two now fail for me? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From robin at icir.org Thu Sep 29 22:57:52 2011 From: robin at icir.org (Robin Sommer) Date: Thu, 29 Sep 2011 22:57:52 -0700 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Fixing a bunch of memory leaks. (91ed9ff) In-Reply-To: <201109300553.p8U5rPha026606@bro-ids.icir.org> References: <201109300553.p8U5rPha026606@bro-ids.icir.org> Message-ID: <20110930055752.GC54224@icir.org> On Thu, Sep 29, 2011 at 22:53 -0700, I wrote: > There was a big one in the logging code as well though. It looks like this fix has major impact: my workers' memory footprint is very stable right now. Let's see how that's going to develop ... Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Fri Sep 30 04:30:19 2011 From: seth at icir.org (Seth Hall) Date: Fri, 30 Sep 2011 07:30:19 -0400 Subject: [Bro-Dev] SSL paper using Bro Message-ID: <033AF1D4-1E28-4FAF-9487-D3F91D9F6838@icir.org> The authors are planning to present the paper at IMC but most importantly, they used Bro. :) http://www.net.in.tum.de/fileadmin/bibtex/publications/papers/imc-pkicrawl-2.pdf .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Fri Sep 30 06:26:53 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Fri, 30 Sep 2011 08:26:53 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Fix for shutdown bug in http scripts. (012d8cf) In-Reply-To: <20110930054704.GA66615@icir.org> References: <201109300125.p8U1PK5S005850@bro-ids.icir.org> <20110930054704.GA66615@icir.org> Message-ID: >> +++ b/testing/btest/scripts/base/protocols/http/100-continue.bro > >> +++ b/testing/btest/scripts/base/protocols/http/http-header-crlf.bro > > Not sure what's going in here but these two now fail for me? Me, too. The original worked for me somehow, but still didn't look quite right anymore either, so I changed it to check weird.log doesn't exist at all. Seth, do they pass for you now (maybe when you checked before, it was using a version of bro that hadn't been re-built with the parser changes?) ? - Jon From seth at icir.org Fri Sep 30 06:31:36 2011 From: seth at icir.org (Seth Hall) Date: Fri, 30 Sep 2011 09:31:36 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Fix for shutdown bug in http scripts. (012d8cf) In-Reply-To: References: <201109300125.p8U1PK5S005850@bro-ids.icir.org> <20110930054704.GA66615@icir.org> Message-ID: <338F7AED-4627-44CF-900A-913C6F33F656@icir.org> On Sep 30, 2011, at 9:26 AM, Jonathan Siwek wrote: > Me, too. The original worked for me somehow, but still didn't look quite right anymore either, so I changed it to check weird.log doesn't exist at all. Seth, do they pass for you now (maybe when you checked before, it was using a version of bro that hadn't been re-built with the parser changes?) ? Works fine now. Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Fri Sep 30 06:34:34 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Fri, 30 Sep 2011 08:34:34 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Fixing unit tests. (71dc6b6) In-Reply-To: <201109301254.p8UCsQgW026033@bro-ids.icir.org> References: <201109301254.p8UCsQgW026033@bro-ids.icir.org> Message-ID: > Fixing unit tests. Sorry for the direct master commit, but test fixes I did before kept lagging behind other new commits such that they were failing again by the time they were merged. Right now all the unit tests pass for me (except for notice.suppression-disable intermittently failing, which I'll try to fix later). Does everyone else see the same thing? - Jon From bro at tracker.bro-ids.org Fri Sep 30 06:37:03 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Fri, 30 Sep 2011 13:37:03 -0000 Subject: [Bro-Dev] #544: scan.bro and hot.conn.bro need updating In-Reply-To: <047.737dcb7555d7c558c1757654428e87e0@tracker.bro-ids.org> References: <047.737dcb7555d7c558c1757654428e87e0@tracker.bro-ids.org> Message-ID: <062.f74cd0db3e8ca9eca90540237c300708@tracker.bro-ids.org> #544: scan.bro and hot.conn.bro need updating ----------------------+------------------------ Reporter: robin | Owner: seth Type: Problem | Status: assigned Priority: Normal | Milestone: Bro1.6 Component: Bro | Version: git/master Resolution: | Keywords: beta ----------------------+------------------------ Comment (by jsiwek): Just noting here that these scripts were removed from the source tree for now and so I removed the documentation blacklist for them and also removed them from test-all-policy.bro -- Ticket URL: Bro Tracker Bro Issue Tracker From seth at icir.org Fri Sep 30 06:50:19 2011 From: seth at icir.org (Seth Hall) Date: Fri, 30 Sep 2011 09:50:19 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Fixing unit tests. (71dc6b6) In-Reply-To: References: <201109301254.p8UCsQgW026033@bro-ids.icir.org> Message-ID: <918D4D03-55F3-492D-8D8D-2FE6CA5D388E@icir.org> On Sep 30, 2011, at 9:34 AM, Jonathan Siwek wrote: >> Fixing unit tests. > > Sorry for the direct master commit, but test fixes I did before kept lagging behind other new commits such that they were failing again by the time they were merged. Right now all the unit tests pass for me (except for notice.suppression-disable intermittently failing, which I'll try to fix later). Does everyone else see the same thing? Woo! I see the same thing. It looks like the baseline for notice.suppression-disable is old, want me to just update it? (I haven't run the external tests either yet) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From jsiwek at ncsa.illinois.edu Fri Sep 30 07:04:37 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Fri, 30 Sep 2011 09:04:37 -0500 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Fixing unit tests. (71dc6b6) In-Reply-To: <918D4D03-55F3-492D-8D8D-2FE6CA5D388E@icir.org> References: <201109301254.p8UCsQgW026033@bro-ids.icir.org> <918D4D03-55F3-492D-8D8D-2FE6CA5D388E@icir.org> Message-ID: <12978B14-DFE3-475B-8847-AF3220A3D22B@ncsa.illinois.edu> > Woo! I see the same thing. It looks like the baseline for notice.suppression-disable is old, want me to just update it? What I see for that test is that sometimes it passes and sometimes not, and that was because the notice_policy.log ordering isn't deterministic (there's another thread in bro-dev about it). - Jon From jsiwek at ncsa.illinois.edu Fri Sep 30 07:07:24 2011 From: jsiwek at ncsa.illinois.edu (Jonathan Siwek) Date: Fri, 30 Sep 2011 09:07:24 -0500 Subject: [Bro-Dev] non-deterministic notice_policy order In-Reply-To: <20110928233007.GB63322@icir.org> References: <5F146456-5FB5-4848-8689-05D2565B3913@ncsa.illinois.edu> <20110928233007.GB63322@icir.org> Message-ID: <140F4C67-BAA9-4E9C-840E-3956962F63CD@ncsa.illinois.edu> > How about printing out the function's code and hashing that when we're > in "deterministic hashing mode" (i.e., have_random_seed() return > true)? Yeah, I think that would work as far as making the test work consistently, but I think we want a fix that will always be in effect so that the ordering of Notice::policy's PolicyItem's with equal priority are applied in a consistent order during actual operation? - Jon From seth at icir.org Fri Sep 30 07:19:07 2011 From: seth at icir.org (Seth Hall) Date: Fri, 30 Sep 2011 10:19:07 -0400 Subject: [Bro-Dev] [Bro-Commits] [git/bro] master: Fixing unit tests. (71dc6b6) In-Reply-To: <12978B14-DFE3-475B-8847-AF3220A3D22B@ncsa.illinois.edu> References: <201109301254.p8UCsQgW026033@bro-ids.icir.org> <918D4D03-55F3-492D-8D8D-2FE6CA5D388E@icir.org> <12978B14-DFE3-475B-8847-AF3220A3D22B@ncsa.illinois.edu> Message-ID: <6D7213CB-572E-488E-A76F-B3F1CD5BAD4A@icir.org> On Sep 30, 2011, at 10:04 AM, Jonathan Siwek wrote: >> Woo! I see the same thing. It looks like the baseline for notice.suppression-disable is old, want me to just update it? > > What I see for that test is that sometimes it passes and sometimes not, and that was because the notice_policy.log ordering isn't deterministic (there's another thread in bro-dev about it). Ah, of course. I forgot about that thread. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From robin at icir.org Fri Sep 30 07:27:00 2011 From: robin at icir.org (Robin Sommer) Date: Fri, 30 Sep 2011 07:27:00 -0700 Subject: [Bro-Dev] non-deterministic notice_policy order In-Reply-To: <140F4C67-BAA9-4E9C-840E-3956962F63CD@ncsa.illinois.edu> References: <5F146456-5FB5-4848-8689-05D2565B3913@ncsa.illinois.edu> <20110928233007.GB63322@icir.org> <140F4C67-BAA9-4E9C-840E-3956962F63CD@ncsa.illinois.edu> Message-ID: <20110930142700.GC94019@icir.org> On Fri, Sep 30, 2011 at 09:07 -0500, you wrote: > Yeah, I think that would work as far as making the test work > consistently, but I think we want a fix that will always be in effect > so that the ordering of Notice::policy's PolicyItem's with equal > priority are applied in a consistent order during actual operation? Makes sense. How about (always) hashing function name and location information? Robin -- Robin Sommer * Phone +1 (510) 722-6541 * robin at icir.org ICSI/LBNL * Fax +1 (510) 666-2956 * www.icir.org From seth at icir.org Fri Sep 30 07:28:25 2011 From: seth at icir.org (Seth Hall) Date: Fri, 30 Sep 2011 10:28:25 -0400 Subject: [Bro-Dev] non-deterministic notice_policy order In-Reply-To: <140F4C67-BAA9-4E9C-840E-3956962F63CD@ncsa.illinois.edu> References: <5F146456-5FB5-4848-8689-05D2565B3913@ncsa.illinois.edu> <20110928233007.GB63322@icir.org> <140F4C67-BAA9-4E9C-840E-3956962F63CD@ncsa.illinois.edu> Message-ID: On Sep 30, 2011, at 10:07 AM, Jonathan Siwek wrote: >> How about printing out the function's code and hashing that when we're >> in "deterministic hashing mode" (i.e., have_random_seed() return >> true)? > > Yeah, I think that would work as far as making the test work consistently, but I think we want a fix that will always be in effect so that the ordering of Notice::policy's PolicyItem's with equal priority are applied in a consistent order during actual operation? I would have expected that to be true already due to the seeding of the random number generator. Here's the code that writes the notice policy to the logging stream... local tmp: table[count] of set[PolicyItem] = table(); for ( pi in policy ) { if ( pi$priority < 0 || pi$priority > 10 ) Reporter::fatal("All Notice::PolicyItem priorities must be within 0 and 10"); if ( pi$priority !in tmp ) tmp[pi$priority] = set(); add tmp[pi$priority][pi]; } local rev_count = vector(10,9,8,7,6,5,4,3,2,1,0); for ( i in rev_count ) { local j = rev_count[i]; if ( j in tmp ) { for ( pi in tmp[j] ) { pi$position = |ordered_policy|; ordered_policy[|ordered_policy|] = pi; Log::write(Notice::POLICY_LOG, pi); } } } All it's doing is taking the unordered set of PolicyItems and turning it into a vector ordered by the $priority field. It's surprising to me that due to the seeding we're seeing the order of the log change. It's the index value of the vector changing that's causing the difference in notice.log on the line just before the Log::write call. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro-ids.org/ From mcholste at gmail.com Fri Sep 30 12:30:42 2011 From: mcholste at gmail.com (Martin Holste) Date: Fri, 30 Sep 2011 14:30:42 -0500 Subject: [Bro-Dev] #615: Cluster manager crash In-Reply-To: <061.5f9e7b8cb501a998d63f52d875134bd5@tracker.bro-ids.org> References: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> <061.5f9e7b8cb501a998d63f52d875134bd5@tracker.bro-ids.org> Message-ID: This is still occurring regularly for me, even after today's code update. On Thu, Sep 29, 2011 at 6:20 PM, Bro Tracker wrote: > #615: Cluster manager crash > -----------------------------+-------------------- > ?Reporter: ?seth ? ? ? ? ? ?| ? ? ?Owner: > ? ? ?Type: ?Problem ? ? ? ? | ? ? Status: ?closed > ?Priority: ?High ? ? ? ? ? ?| ?Milestone: ?Bro1.6 > ?Component: ?Bro ? ? ? ? ? ? | ? ?Version: > Resolution: ?Solved/Applied ?| ? Keywords: ?beta > -----------------------------+-------------------- > Changes (by robin): > > ?* status: ?new => closed > ?* resolution: ? => Solved/Applied > > > Comment: > > ?Closing. If anybody still sees it, please reopen. > > -- > Ticket URL: > Bro Tracker > Bro Issue Tracker > > _______________________________________________ > bro-dev mailing list > bro-dev at bro-ids.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > From bro at tracker.bro-ids.org Fri Sep 30 18:32:55 2011 From: bro at tracker.bro-ids.org (Bro Tracker) Date: Sat, 01 Oct 2011 01:32:55 -0000 Subject: [Bro-Dev] #615: Cluster manager crash In-Reply-To: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> References: <046.aca24e97ce3580f474951460ead41b9e@tracker.bro-ids.org> Message-ID: <061.f0f3e83d97f4df07e36e61675595e3e7@tracker.bro-ids.org> #615: Cluster manager crash ----------------------+---------------------- Reporter: seth | Owner: Type: Problem | Status: reopened Priority: High | Milestone: Bro1.6 Component: Bro | Version: Resolution: | Keywords: beta ----------------------+---------------------- Changes (by seth): * status: closed => reopened * resolution: Solved/Applied => Comment: Martin has reported still seeing crash on his manager. Reopening. -- Ticket URL: Bro Tracker Bro Issue Tracker