From edthoma at sandia.gov Mon May 7 14:17:32 2007 From: edthoma at sandia.gov (Thomas, Eric D.) Date: Mon, 07 May 2007 14:17:32 -0700 Subject: [Bro] Continuous memory growth without tcp.bro Message-ID: Without tcp.bro the memory footprint of the Bro process increases until it reaches the memory limit and Bro dies. I conducted the following tests: 1. bro -i eth0 profiling.bro 2. bro -i eth0 -f "tcp" profiling.bro 3. bro -i eth0 -f "tcp profiling.bro tcp.bro 4. bro -i eth0 profiling.bro tcp.bro Only test 4 didn't result in outrageous memory usage. Obviously the capture-filter preventing the processing of data packets is the reason. But I'm curious, what exactly does Bro store when processing data packets that causes such a memory bloat? I would have figured the processing of data packets only results in updating pre-existing connection state objects. Is there any way to prevent this bloat without modifying the source code? Thanks, Eric From jmzhou.ml at gmail.com Tue May 8 14:14:31 2007 From: jmzhou.ml at gmail.com (jmzhou.ml at gmail.com) Date: Tue, 8 May 2007 14:14:31 -0700 (PDT) Subject: [Bro] binpac accumulated patch Message-ID: Hi there, The attached patch is for binpac in bro 1.2.1. It fixes a problem in handling the ``refine'' keyword under flowunit mode. We must defer the ``MarkIncrementalInput'' till all ``refine'' declarations have been processed. The patch also includes two previous patches: . data dependency between record fields . format string for module operation Regards, Jimmy ____________________________________________________________ The future is not set. There is no fate but what we make for ourselves. - Terminator II, Judgment Day ------------------------------------------------------------ -------------- next part -------------- diff -ruN bro-1.2.1.orig/src/binpac/pac_common.h bro-1.2.1/src/binpac/pac_common.h --- bro-1.2.1.orig/src/binpac/pac_common.h 2006-07-26 15:02:40.000000000 -0700 +++ bro-1.2.1/src/binpac/pac_common.h 2007-04-18 11:09:16.000000000 -0700 @@ -105,6 +105,11 @@ if ( pc ) \ for ( ct::iterator i = (pc)->begin(); i != (pc)->end(); ++i ) +#define foreach_rev(i, ct, pc) \ + if ( pc ) \ + for ( ct::iterator i = (pc)->end(); i-- != (pc)->begin(); ) + + #define delete_list(ct, pc) \ { \ foreach(delete_list_i, ct, pc) \ diff -ruN bro-1.2.1.orig/src/binpac/pac_decl.cc bro-1.2.1/src/binpac/pac_decl.cc --- bro-1.2.1.orig/src/binpac/pac_decl.cc 2006-07-26 15:02:40.000000000 -0700 +++ bro-1.2.1/src/binpac/pac_decl.cc 2007-05-08 13:59:22.000000000 -0700 @@ -9,6 +9,7 @@ #include "pac_output.h" #include "pac_param.h" #include "pac_record.h" +#include "pac_flow.h" #include "pac_type.h" #include "pac_utils.h" @@ -70,6 +71,8 @@ if ( ! decl_list_ ) return; + get_flow_decl()->MarkIncrementalInput(); + foreach(i, DeclList, decl_list_) { Decl *decl = *i; diff -ruN bro-1.2.1.orig/src/binpac/pac_expr.def bro-1.2.1/src/binpac/pac_expr.def --- bro-1.2.1.orig/src/binpac/pac_expr.def 2006-07-26 15:02:39.000000000 -0700 +++ bro-1.2.1/src/binpac/pac_expr.def 2007-04-18 11:06:47.000000000 -0700 @@ -14,7 +14,7 @@ EXPR_DEF(EXPR_MINUS, 2, "%s - %s") EXPR_DEF(EXPR_TIMES, 2, "%s * %s") EXPR_DEF(EXPR_DIV, 2, "%s / %s") -EXPR_DEF(EXPR_MOD, 2, "%s % %s") +EXPR_DEF(EXPR_MOD, 2, "%s %% %s") EXPR_DEF(EXPR_BITNOT, 1, "~%s") EXPR_DEF(EXPR_BITAND, 2, "%s & %s") EXPR_DEF(EXPR_BITOR, 2, "%s | %s") diff -ruN bro-1.2.1.orig/src/binpac/pac_field.cc bro-1.2.1/src/binpac/pac_field.cc --- bro-1.2.1.orig/src/binpac/pac_field.cc 2006-07-26 15:02:39.000000000 -0700 +++ bro-1.2.1/src/binpac/pac_field.cc 2007-04-19 09:23:59.000000000 -0700 @@ -12,6 +12,7 @@ decl_id_ = current_decl_id; field_id_str_ = strfmt("%s:%s", decl_id()->Name(), id_->Name()); attrs_ = 0; + prepared_ = false; } Field::~Field() @@ -73,6 +74,11 @@ void Field::Prepare(Env *env) { + /* prevent infinite recursion */ + if ( prepared_ ) + return ; + prepared_ = true; + if ( type_ ) { if ( anonymous_field() ) diff -ruN bro-1.2.1.orig/src/binpac/pac_field.h bro-1.2.1/src/binpac/pac_field.h --- bro-1.2.1.orig/src/binpac/pac_field.h 2006-07-26 15:02:40.000000000 -0700 +++ bro-1.2.1/src/binpac/pac_field.h 2007-04-19 09:24:05.000000000 -0700 @@ -74,6 +74,7 @@ protected: FieldType tof_; int flags_; + bool prepared_; ID* id_; Type *type_; const ID* decl_id_; diff -ruN bro-1.2.1.orig/src/binpac/pac_flow.cc bro-1.2.1/src/binpac/pac_flow.cc --- bro-1.2.1.orig/src/binpac/pac_flow.cc 2006-10-12 14:13:12.000000000 -0700 +++ bro-1.2.1/src/binpac/pac_flow.cc 2007-05-08 13:57:27.000000000 -0700 @@ -15,6 +15,20 @@ #include "pac_varfield.h" +static FlowDecl* g_flow_decl = NULL; + +FlowDecl* get_flow_decl() +{ + return g_flow_decl; +} + +void set_flow_decl(FlowDecl* decl) +{ + assert (NULL == g_flow_decl); + g_flow_decl = decl; +} + + FlowDecl::FlowDecl(ID *id, ParamList *params, AnalyzerElementList *elemlist) @@ -24,6 +38,8 @@ conn_decl_ = 0; flow_buffer_var_field_ = 0; AddElements(elemlist); + + set_flow_decl (this); } FlowDecl::~FlowDecl() @@ -55,6 +71,14 @@ "flow should be defined in only a connection declaration"); } +void FlowDecl::MarkIncrementalInput() + { + if ( dataunit_->type() != AnalyzerDataUnit::FLOWUNIT ) + return; + + dataunit_->data_type()->MarkIncrementalInput(); + } + void FlowDecl::ProcessDataUnitElement(AnalyzerDataUnit *dataunit_elem) { if ( dataunit_ ) @@ -66,7 +90,7 @@ if ( dataunit_->type() == AnalyzerDataUnit::FLOWUNIT ) { - dataunit_->data_type()->MarkIncrementalInput(); + //dataunit_->data_type()->MarkIncrementalInput(); flow_buffer_var_field_ = new PrivVarField( flow_buffer_id->clone(), diff -ruN bro-1.2.1.orig/src/binpac/pac_flow.h bro-1.2.1/src/binpac/pac_flow.h --- bro-1.2.1.orig/src/binpac/pac_flow.h 2006-07-26 15:02:39.000000000 -0700 +++ bro-1.2.1/src/binpac/pac_flow.h 2007-05-08 14:01:13.000000000 -0700 @@ -3,6 +3,8 @@ #include "pac_analyzer.h" +FlowDecl* get_flow_decl(); + class FlowDecl : public AnalyzerDecl { public: @@ -12,6 +14,7 @@ void Prepare(); void set_conn_decl(ConnDecl *c) { conn_decl_ = c; } + void MarkIncrementalInput(); static ParameterizedType *flow_buffer_type(); diff -ruN bro-1.2.1.orig/src/binpac/pac_type.cc bro-1.2.1/src/binpac/pac_type.cc --- bro-1.2.1.orig/src/binpac/pac_type.cc 2006-07-26 15:02:40.000000000 -0700 +++ bro-1.2.1/src/binpac/pac_type.cc 2007-04-19 09:29:56.000000000 -0700 @@ -305,9 +305,40 @@ } } + /* prepare parameter fields first as they may be dependency */ foreach (i, FieldList, fields_) { Field *f = *i; + if (PARAM_FIELD == f->tof()) + f->Prepare(env); + } + + foreach (i, FieldList, fields_) + { + Field *f = *i; + Expr* req; + if (NULL == f->type() /* can be a padding field */ || + NULL == (req = f->type()->attr_requires())) + { + /* there is no &requires dependents, good! */ + f->Prepare(env); + continue; + } + + /* there are dependents on some &let fields :-( */ + ASSERT(Expr::EXPR_CALLARGS == req->expr_type()); + + FieldList::iterator newi = i+1; + for (; newi != fields_->end(); ++ newi) + { + /* not a field of &let, ignore it */ + Field* newf = *newi; + if (LET_FIELD != newf->tof()) + continue; + + if (req->HasReference (newf->id())) + newf->Prepare(env); + } f->Prepare(env); } } @@ -367,7 +398,7 @@ void Type::GenCleanUpCode(Output* out_cc, Env* env) { - foreach (i, FieldList, fields_) + foreach_rev (i, FieldList, fields_) { Field *f = *i; if ( f->tof() != CASE_FIELD ) diff -ruN bro-1.2.1.orig/src/binpac/pac_type.h bro-1.2.1/src/binpac/pac_type.h --- bro-1.2.1.orig/src/binpac/pac_type.h 2006-10-17 15:46:32.000000000 -0700 +++ bro-1.2.1/src/binpac/pac_type.h 2007-04-19 09:28:18.000000000 -0700 @@ -155,6 +155,7 @@ Expr *attr_if_expr() const { return attr_if_expr_; } // TODO: generate the length expression automatically. Expr *attr_length_expr() const { return attr_length_expr_; } + Expr *attr_requires() const { return attr_requires_; } bool attr_refcount() const { return attr_refcount_; } bool attr_transient() const { return attr_transient_; } From hhoffman at ip-solutions.net Wed May 9 13:20:28 2007 From: hhoffman at ip-solutions.net (Harry Hoffman) Date: Wed, 9 May 2007 16:20:28 -0400 (EDT) Subject: [Bro] enabling dpd results in run-time error Message-ID: <25575.207.106.111.3.1178742028.squirrel@www.ip-solutions.net> Hi All, running bro-1.2.1 under CentOS 5.0. I'm attempting to enable dpd via brolite.bro. When I change the: const use_dpd = F; to const use_dpd = T; bro fails to start with the following errors: /usr/local/bro/policy/http-request.bro, line 34: run-time error: error compiling pattern /((((((((((((((((((((^?.*(etc\/(passwd|shadow|netconfig)))|(^?.*(IFS[ \t]*=)))|(^?.*(nph-test-cgi\?)))|(^?.*((%0a|\.\.)\/(bin|etc|usr|tmp))))|(^?.*(\/Admin_files\/order\.log)))|(^?.*(\/carbo\.dll)))|(^?.*(\/cgi-bin\/(phf|php\.cgi|test-cgi))))|(^?.*(\/cgi-dos\/args\.bat)))|(^?.*(\/cgi-win\/uploader\.exe)))|(^?.*(\/search97\.vts)))|(^?.*(tk\.tgz)))|(^?.*(ownz)))|(^?.*(viewtopic\.php.*%.*\(.*\()))|(^?.*(sshd\.(tar|tgz).*)))|(^?.*([aA][dD][oO][rR][eE][bB][sS][dD].*)))|(^?.*(shv4\.(tar|tgz).*)))|(^?.*(lrk\.(tar|tgz).*)))|(^?.*(lyceum\.(tar|tgz).*)))|(^?.*(maxty\.(tar|tgz).*)))|(^?.*(rootII\.(tar|tgz).*)))|(^?.*(invader\.(tar|tgz).*))/ /usr/local/bro/policy/http-request.bro, line 42: run-time error: error compiling pattern /((^?.*(.*\/c\+dir))|(^?.*(.*cool.dll.*)))|(^?.*(.*Admin.dll.*Admin.dll.*))/ /usr/local/bro/policy/http-request.bro, line 48: run-time error: error compiling pattern /^?.*(\/cgi-bin\/(phf|php\.cgi|test-cgi))/ /usr/local/bro/policy/http-request.bro, line 50: run-time error: error compiling pattern /^?.*(wwwroot|WWWROOT)/ /usr/local/bro/policy/http-reply.bro, line 111: run-time error: error compiling pattern /^?.*(^ )/ /usr/local/bro/policy/hot-ids.bro, line 15: run-time error: error compiling pattern /^?.*((y[o0]u)(r|ar[e3])([o0]wn.*))/ /usr/local/bro/policy/ftp.bro, line 43: run-time error: error compiling pattern /((((((((((((((((((((((^?.*(.*(etc\/|master\.)?(passwd|shadow|s?pwd\.db)))|(^?.*(.*snoop\.(tar|tgz).*)))|(^?.*(.*bnc\.(tar|tgz).*)))|(^?.*(.*datapipe.*)))|(^?.*(.*ADMw0rm.*)))|(^?.*(.*newnick.*)))|(^?.*(.*sniffit.*)))|(^?.*(.*neet\.(tar|tgz).*)))|(^?.*(.*\.\.\..*)))|(^?.*(.*ftpscan.txt.*)))|(^?.*(.*jcc.pdf.*)))|(^?.*(.*\.[Ff]rom.*)))|(^?.*(.*sshd\.(tar|tgz).*)))|(^?.*(.*\/rk7.*)))|(^?.*(.*rk7\..*)))|(^?.*(.*[aA][dD][oO][rR][eE][bB][sS][dD].*)))|(^?.*(.*[tT][aA][gG][gG][eE][dD].*)))|(^?.*(.*shv4\.(tar|tgz).*)))|(^?.*(.*lrk\.(tar|tgz).*)))|(^?.*(.*lyceum\.(tar|tgz).*)))|(^?.*(.*maxty\.(tar|tgz).*)))|(^?.*(.*rootII\.(tar|tgz).*)))|(^?.*(.*invader\.(tar|tgz).*))/ /usr/local/bro/policy/ftp.bro, line 48: run-time error: error compiling pattern /(^?.*(.*\.rhosts))|(^?.*(.*\.forward))/ /usr/local/bro/policy/ftp.bro, line 51: run-time error: error compiling pattern /^?.*([Ee][Xx][Ee][Cc].*)/ /usr/local/bro/policy/ftp.bro, line 63: run-time error: error compiling pattern /^?.*(,0,0)/ /usr/local/bro/policy/ftp.bro, line 154: run-time error: error compiling pattern /^?.*((\/|[A-Za-z]:[\\\/]).*)/ /usr/local/bro/policy/ftp.bro, line 349: run-time error: error compiling pattern /^?.*([\x00-\x7f])/ /usr/local/bro/policy/ftp.bro, line 462: run-time error: error compiling pattern /^?.*([Ee][Xx][Ee][Cc])/ /usr/local/bro/policy/ftp.bro, line 527: run-time error: error compiling pattern /^?.*(\"([^\"]|\"\")*(\/|\\)([^\"]|\"\")*\")/ /usr/local/bro/policy/ftp.bro, line 545: run-time error: error compiling pattern /^?.*(((\/)+([^\/]|\\\/)+)?((\/)+\.\.(\/)+))/ /usr/local/bro/policy/ftp.bro, line 555: run-time error: error compiling pattern /^?.*((\/){2,})/ /usr/local/bro/policy/ftp.bro, line 700: run-time error: error compiling pattern /^?.*([\x80-\xff]{3})/ /usr/local/bro/policy/ftp.bro, line 735: run-time error: error compiling pattern /^?.*(USER|PASS|ACCT)/ /usr/local/bro/policy/portmapper.bro, line 310: run-time error: error compiling pattern /^?.*(^\[)/ /usr/local/bro/policy/portmapper.bro, line 311: run-time error: error compiling pattern /^?.*(\]$)/ /usr/local/bro/policy/login.bro, line 66: run-time error: error compiling pattern /((((((((((((((((((((((((((((((((^?.*(rewt))|(^?.*(eggdrop)))|(^?.*(\/bin\/eject)))|(^?.*(oir##t)))|(^?.*(ereeto)))|(^?.*((shell|xploit)_?code)))|(^?.*(execshell)))|(^?.*(ff\.core)))|(^?.*(unset[ \t]+(histfile|history|HISTFILE|HISTORY))))|(^?.*(neet\.tar)))|(^?.*(r0kk0)))|(^?.*(su[ \t]+(daemon|news|adm))))|(^?.*(\.\/clean)))|(^?.*(rm[ \t]+-rf[ \t]+secure)))|(^?.*(cd[ \t]+\/dev\/[a-zA-Z]{3})))|(^?.*(solsparc_lpset)))|(^?.*(\.\/[a-z]+[ \t]+passwd)))|(^?.*(\.\/bnc)))|(^?.*(bnc\.conf)))|(^?.*(\"\/bin\/ksh\")))|(^?.*(LAST STAGE OF DELIRIUM)))|(^?.*(SNMPXDMID_PROG)))|(^?.*(snmpXdmid for solaris)))|(^?.*(\"\/bin\/uname)))|(^?.*(gcc[ \t]+1\.c)))|(^?.*(>\/etc\/passwd)))|(^?.*(lynx[ \t]+-source[ \t]+.*(packetstorm|shellcode|linux|sparc))))|(^?.*(gcc.*\/bin\/login)))|(^?.*(#define NOP.*0x)))|(^?.*(printf\(\"overflowing)))|(^?.*(exec[a-z]*\(\"\/usr\/openwin)))|(^?.*(perl[ \t]+.*x.*[0-9][0-9][0-9][0-9])))|(^?.*(ping.*-s.*%d))/ /usr/local/bro/policy/login.bro, line 72: run-time error: error compiling pattern /^?.*([ \t]*(cd|pushd|more|less|cat|vi|emacs|pine)[ \t]+((['"]?\.\.\.)|(["'](\.*)[ \t])))/ /usr/local/bro/policy/login.bro, line 75: run-time error: error compiling pattern /^?.*(No such file or directory)/ Any ideas why? I've search the lists and google but nothing is coming up. Also, checked the configure.log to see if perhaps I missed something there. Cheers, Harry From jones at tacc.utexas.edu Wed May 9 13:27:15 2007 From: jones at tacc.utexas.edu (William L. Jones) Date: Wed, 9 May 2007 15:27:15 -0500 Subject: [Bro] enabling dpd results in run-time error In-Reply-To: <25575.207.106.111.3.1178742028.squirrel@www.ip-solutions.net> References: <25575.207.106.111.3.1178742028.squirrel@www.ip-solutions.net> Message-ID: Try the following: cd /usr/local/bro cp policy/sigs/dpd.sig site/dpd.sig Bill Jones > -----Original Message----- > From: bro-bounces at ICSI.Berkeley.EDU [mailto:bro-bounces at ICSI.Berkeley.EDU] > On Behalf Of Harry Hoffman > Sent: Wednesday, May 09, 2007 3:20 PM > To: bro at ICSI.Berkeley.EDU > Subject: [Bro] enabling dpd results in run-time error > > Hi All, > > running bro-1.2.1 under CentOS 5.0. > > I'm attempting to enable dpd via brolite.bro. When I change the: > const use_dpd = F; > to > const use_dpd = T; > > bro fails to start with the following errors: > > /usr/local/bro/policy/http-request.bro, line 34: run-time error: error > compiling pattern > /((((((((((((((((((((^?.*(etc\/(passwd|shadow|netconfig)))|(^?.*(IFS[ > \t]*=)))|(^?.*(nph-test- > cgi\?)))|(^?.*((%0a|\.\.)\/(bin|etc|usr|tmp))))|(^?.*(\/Admin_files\/ord er > \.log)))|(^?.*(\/carbo\.dll)))|(^?.*(\/cgi-bin\/(phf|php\.cgi|test- > cgi))))|(^?.*(\/cgi-dos\/args\.bat)))|(^?.*(\/cgi- > win\/uploader\.exe)))|(^?.*(\/search97\.vts)))|(^?.*(tk\.tgz)))|(^?.*(ow nz > )))|(^?.*(viewtopic\.php.*%.*\(.*\()))|(^?.*(sshd\.(tar|tgz).*)))|(^?.*( [a > A][dD][oO][rR][eE][bB][sS][dD].*)))|(^?.*(shv4\.(tar|tgz).*)))|(^?.*(lrk \. > (tar|tgz).*)))|(^?.*(lyceum\.(tar|tgz).*)))|(^?.*(maxty\.(tar|tgz).*)))| (^ > ?.*(rootII\.(tar|tgz).*)))|(^?.*(invader\.(tar|tgz).*))/ > /usr/local/bro/policy/http-request.bro, line 42: run-time error: error > compiling pattern > /((^?.*(.*\/c\+dir))|(^?.*(.*cool.dll.*)))|(^?.*(.*Admin.dll.*Admin.dll. *) > )/ > /usr/local/bro/policy/http-request.bro, line 48: run-time error: error > compiling pattern /^?.*(\/cgi-bin\/(phf|php\.cgi|test-cgi))/ > /usr/local/bro/policy/http-request.bro, line 50: run-time error: error > compiling pattern /^?.*(wwwroot|WWWROOT)/ > /usr/local/bro/policy/http-reply.bro, line 111: run-time error: error > compiling pattern /^?.*(^ )/ > /usr/local/bro/policy/hot-ids.bro, line 15: run-time error: error > compiling pattern /^?.*((y[o0]u)(r|ar[e3])([o0]wn.*))/ > /usr/local/bro/policy/ftp.bro, line 43: run-time error: error compiling > pattern > /((((((((((((((((((((((^?.*(.*(etc\/|master\.)?(passwd|shadow|s?pwd\.db) )) > |(^?.*(.*snoop\.(tar|tgz).*)))|(^?.*(.*bnc\.(tar|tgz).*)))|(^?.*(.*datap ip > e.*)))|(^?.*(.*ADMw0rm.*)))|(^?.*(.*newnick.*)))|(^?.*(.*sniffit.*)))|(^ ?. > *(.*neet\.(tar|tgz).*)))|(^?.*(.*\.\.\..*)))|(^?.*(.*ftpscan.txt.*)))|(^ ?. > *(.*jcc.pdf.*)))|(^?.*(.*\.[Ff]rom.*)))|(^?.*(.*sshd\.(tar|tgz).*)))|(^? .* > (.*\/rk7.*)))|(^?.*(.*rk7\..*)))|(^?.*(.*[aA][dD][oO][rR][eE][bB][sS][dD ]. > *)))|(^?.*(.*[tT][aA][gG][gG][eE][dD].*)))|(^?.*(.*shv4\.(tar|tgz).*)))| (^ > ?.*(.*lrk\.(tar|tgz).*)))|(^?.*(.*lyceum\.(tar|tgz).*)))|(^?.*(.*maxty\. (t > ar|tgz).*)))|(^?.*(.*rootII\.(tar|tgz).*)))|(^?.*(.*invader\.(tar|tgz).* )) > / > /usr/local/bro/policy/ftp.bro, line 48: run-time error: error compiling > pattern /(^?.*(.*\.rhosts))|(^?.*(.*\.forward))/ > /usr/local/bro/policy/ftp.bro, line 51: run-time error: error compiling > pattern /^?.*([Ee][Xx][Ee][Cc].*)/ > /usr/local/bro/policy/ftp.bro, line 63: run-time error: error compiling > pattern /^?.*(,0,0)/ > /usr/local/bro/policy/ftp.bro, line 154: run-time error: error compiling > pattern /^?.*((\/|[A-Za-z]:[\\\/]).*)/ > /usr/local/bro/policy/ftp.bro, line 349: run-time error: error compiling > pattern /^?.*([\x00-\x7f])/ > /usr/local/bro/policy/ftp.bro, line 462: run-time error: error compiling > pattern /^?.*([Ee][Xx][Ee][Cc])/ > /usr/local/bro/policy/ftp.bro, line 527: run-time error: error compiling > pattern /^?.*(\"([^\"]|\"\")*(\/|\\)([^\"]|\"\")*\")/ > /usr/local/bro/policy/ftp.bro, line 545: run-time error: error compiling > pattern /^?.*(((\/)+([^\/]|\\\/)+)?((\/)+\.\.(\/)+))/ > /usr/local/bro/policy/ftp.bro, line 555: run-time error: error compiling > pattern /^?.*((\/){2,})/ > /usr/local/bro/policy/ftp.bro, line 700: run-time error: error compiling > pattern /^?.*([\x80-\xff]{3})/ > /usr/local/bro/policy/ftp.bro, line 735: run-time error: error compiling > pattern /^?.*(USER|PASS|ACCT)/ > /usr/local/bro/policy/portmapper.bro, line 310: run-time error: error > compiling pattern /^?.*(^\[)/ > /usr/local/bro/policy/portmapper.bro, line 311: run-time error: error > compiling pattern /^?.*(\]$)/ > /usr/local/bro/policy/login.bro, line 66: run-time error: error compiling > pattern > /((((((((((((((((((((((((((((((((^?.*(rewt))|(^?.*(eggdrop)))|(^?.*(\/bi n\ > /eject)))|(^?.*(oir##t)))|(^?.*(ereeto)))|(^?.*((shell|xploit)_?code)))| (^ > ?.*(execshell)))|(^?.*(ff\.core)))|(^?.*(unset[ > \t]+(histfile|history|HISTFILE|HISTORY))))|(^?.*(neet\.tar)))|(^?.*(r0kk 0) > ))|(^?.*(su[ > \t]+(daemon|news|adm))))|(^?.*(\.\/clean)))|(^?.*(rm[ \t]+-rf[ > \t]+secure)))|(^?.*(cd[ > \t]+\/dev\/[a-zA-Z]{3})))|(^?.*(solsparc_lpset)))|(^?.*(\.\/[a-z]+[ > \t]+passwd)))|(^?.*(\.\/bnc)))|(^?.*(bnc\.conf)))|(^?.*(\"\/bin\/ksh\")) )| > (^?.*(LAST > STAGE OF DELIRIUM)))|(^?.*(SNMPXDMID_PROG)))|(^?.*(snmpXdmid for > solaris)))|(^?.*(\"\/bin\/uname)))|(^?.*(gcc[ > \t]+1\.c)))|(^?.*(>\/etc\/passwd)))|(^?.*(lynx[ \t]+-source[ > \t]+.*(packetstorm|shellcode|linux|sparc))))|(^?.*(gcc.*\/bin\/login)))| (^ > ?.*(#define > NOP.*0x)))|(^?.*(printf\(\"overflowing)))|(^?.*(exec[a- > z]*\(\"\/usr\/openwin)))|(^?.*(perl[ > \t]+.*x.*[0-9][0-9][0-9][0-9])))|(^?.*(ping.*-s.*%d))/ > /usr/local/bro/policy/login.bro, line 72: run-time error: error compiling > pattern /^?.*([ \t]*(cd|pushd|more|less|cat|vi|emacs|pine)[ > \t]+((['"]?\.\.\.)|(["'](\.*)[ \t])))/ > /usr/local/bro/policy/login.bro, line 75: run-time error: error compiling > pattern /^?.*(No such file or directory)/ > > > Any ideas why? I've search the lists and google but nothing is coming up. > Also, checked the configure.log to see if perhaps I missed something > there. > > Cheers, > Harry > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro From hhoffman at ip-solutions.net Wed May 9 13:30:32 2007 From: hhoffman at ip-solutions.net (Harry Hoffman) Date: Wed, 9 May 2007 16:30:32 -0400 (EDT) Subject: [Bro] followup: enabling dpd results in run-time error Message-ID: <25830.207.106.111.3.1178742632.squirrel@www.ip-solutions.net> sorry, just found the answer on the wiki. I must have missed it the first time around Cheers, Harry From richih at net.in.tum.de Thu May 10 02:35:44 2007 From: richih at net.in.tum.de (Richard Hartmann) Date: Thu, 10 May 2007 11:35:44 +0200 (CEST) Subject: [Bro] Activating a scanner within a scanner? Message-ID: Hi all, as some of you will know, I am writing a SIP and (soonish) a RTCP analyzer in BinPAC. I have a question regarding the analyzation of protocols which are inlined into other protocols. The relation of SIP to SDP could losely be described as that of a HTML header and body. When the Content-Length field in the SIP packet is non-zero, there is a SDP payload that also needs to be parsed. I am not sure if it would make more sense to hook another analyzer into the SIP analyzer or to just parse the SDP payload within my SIP analyzer. Another consideration would be how to write the SDP analyzer in a way that accounts for both for standalone detection and as a plugin for my SIP analyzer (working on packets vs working on data i feed it directly). Any feedback is appreciated, Richard From christian at whoop.org Thu May 10 10:21:37 2007 From: christian at whoop.org (Christian Kreibich) Date: Thu, 10 May 2007 10:21:37 -0700 Subject: [Bro] followup: enabling dpd results in run-time error In-Reply-To: <25830.207.106.111.3.1178742632.squirrel@www.ip-solutions.net> References: <25830.207.106.111.3.1178742632.squirrel@www.ip-solutions.net> Message-ID: <1178817698.2591.126.camel@strangepork> On Wed, 2007-05-09 at 16:30 -0400, Harry Hoffman wrote: > sorry, just found the answer on the wiki. > > I must have missed it the first time around Thanks for checking the wiki. I put up the article just the other day because this is becoming a FAQ. Cheers, Christian -- ________________________________________________________________________ http://www.icir.org/christian http://www.whoop.org From robin at icir.org Thu May 10 15:23:10 2007 From: robin at icir.org (Robin Sommer) Date: Thu, 10 May 2007 15:23:10 -0700 Subject: [Bro] Activating a scanner within a scanner? In-Reply-To: References: Message-ID: <20070510222310.GB12457@icir.org> On Thu, May 10, 2007 at 11:35 +0200, you wrote: (Disclaimer: I've no idea about SIP/SDP). > I am not sure if it would make more sense to hook another analyzer into > the SIP analyzer or to just parse the SDP payload within my SIP > analyzer. How closely are the two coupled? Is there information which needs to be passed between the two? If the coupling is not too tight, I'd say separating the two looks nicer, as it shows that semantically they are seperat. The outer analyzer can instantiate an instance of the inner if necessary (i.e., if there's a body) and then call it's DeliverStream() method for the data. > Another consideration would be how to write the SDP analyzer in a way > that accounts for both for standalone detection and as a plugin for my > SIP analyzer (working on packets vs working on data i feed it directly). Not sure I understand that. Does this mean SDP can be encapulated inside SIP but does not have to? Robin -- Robin Sommer * Phone +1 (510) 931-5555 * robin at icir.org LBNL/ICSI * Fax +1 (510) 666-2956 * www.icir.org From richih at net.in.tum.de Fri May 11 07:10:01 2007 From: richih at net.in.tum.de (Richard Hartmann) Date: Fri, 11 May 2007 16:10:01 +0200 (CEST) Subject: [Bro] Activating a scanner within a scanner? In-Reply-To: <20070510222310.GB12457@icir.org> References: <20070510222310.GB12457@icir.org> Message-ID: > How closely are the two coupled? Is there information which needs to > be passed between the two? > Not sure I understand that. Does this mean SDP can be encapulated > inside SIP but does not have to? SIP may contain SDP, whereas SDP is mainly used in SIP. SDP could be used for the description of other sessions, but from what I could find, its main use is definately for SIP. The only information about SDP in SIP is the Content-Length field, so I know that when I see a Content-Length != 0, there will be SDP encapsulated into my SIP packet. > If the coupling is not too tight, I'd say separating the two looks > nicer, as it shows that semantically they are seperat. The outer > analyzer can instantiate an instance of the inner if necessary > (i.e., if there's a body) and then call it's DeliverStream() method > for the data. SIP is UDP-based. Would this work for DeliverPacket, as well? A related problem I have is that SIP uses a blank line to show where the SIP part ends and where the SDP part (if any) starts. What would be a clean way to handle this situation? Ideally, I would want to try two regular expressions. If the first one matches, I know I am still in the SIP part. As soon as that regular expression does not match any more, I could look for a blank line and either set a status flag or invoke the secondary analyzer. The problem is that from what I understand about BinPAC, it is impossible to look at a line and just try to match. Either your whole type/record matches or the whole rest of the packet is discarded. Unfortunately, I can not simply look for a magic last item in the SIP part as the order of the keywords may be completely random. I tried looking at the HTTP analyzer, because this protocol uses a newline to show when the header is finished, but to no avail. Any help on this question would be greatly appreciated, as I am simply unable to solve this problem. For future versions on BinPAC, expanding the case statement so that one could match the contents of a field against a regular expression and/or text would probably help in a lot of situations. What I have in mind is along the lines of fork : case foo of { REGEXP_1 -> bar: REGEXP_1; REGEXP_2 -> baz: REGEXP_23; "string" -> qux: bytestring &restofdata; default -> quux: "hello"; }; Any help appreciated :) Richard From aashish at uiuc.edu Fri May 11 08:49:45 2007 From: aashish at uiuc.edu (Aashish Sharma) Date: Fri, 11 May 2007 10:49:45 -0500 Subject: [Bro] login.bro policy run-time error Message-ID: <20070511154945.GA17221@uiuc.edu> Hello All : I came across the following run-time error which resulted in a crash of bro. Just through should bring this to the attention: 1178868745.638454 /usr/local/bro/policy/login.bro, line 608 (c): run-time error, value used but not set I don't have packet traces so would be difficult to pin point exactly what was going on. Let me know if you need any other details. Thanks Aashish From richih at net.in.tum.de Fri May 11 10:38:22 2007 From: richih at net.in.tum.de (Richard Hartmann) Date: Fri, 11 May 2007 19:38:22 +0200 (CEST) Subject: [Bro] Activating a scanner within a scanner? In-Reply-To: References: <20070510222310.GB12457@icir.org> Message-ID: To elaborate on options to parse newlines which I can think of in the order I would prefer to be able to do this: 1) Define a type like type SIP_Headers = SIP_Header[] &until(foo); where foo would be some way to tell the parser 'until you hit a blank line' 2) Use a case statement to decide how to continue parsing of the data 3) Try to parse parts of the remaining buffer and continue with another branch in the parsing tree if that first attempt fails 4) If all else fails, I will probably try to just grab the first letter of the current line and decide based on what is in that field. This would be an extremely kludgy (think several strings starting with the same letter, resulting in even more branching) and hacky (just ugly, hard to read and generally bad style) Best regards, Richard From vern at icir.org Sun May 13 23:43:44 2007 From: vern at icir.org (Vern Paxson) Date: Sun, 13 May 2007 23:43:44 -0700 Subject: [Bro] login.bro policy run-time error In-Reply-To: <20070511154945.GA17221@uiuc.edu> (Fri, 11 May 2007 10:49:45 CDT). Message-ID: <200705140643.l4E6hiOo040283@jaguar.icir.org> > 1178868745.638454 /usr/local/bro/policy/login.bro, line 608 (c): run-time error, value used but not set > > I don't have packet traces so would be difficult to pin point exactly what was going on. Let me know if you need any other details. Unfortunately it's hard to see how to track this one down without a trace. There's nothing at that line in the script that points to an obvious problem. Vern From robin at icir.org Tue May 15 18:37:27 2007 From: robin at icir.org (Robin Sommer) Date: Tue, 15 May 2007 18:37:27 -0700 Subject: [Bro] Activating a scanner within a scanner? In-Reply-To: References: <20070510222310.GB12457@icir.org> Message-ID: <20070516013727.GJ20205@icir.org> On Fri, May 11, 2007 at 16:10 +0200, you wrote: > SIP may contain SDP, whereas SDP is mainly used in SIP. SDP could be > used for the description of other sessions, [...] > The only information about SDP in SIP is the Content-Length field, Ok, so then I would go for a seperate analyzer. > SIP is UDP-based. Would this work for DeliverPacket, as well? Yes. The "packet" is actually just a chunk of data, with the actual interpretation being left to the analyzer. > I tried looking at the HTTP analyzer, because this protocol uses a > newline to show when the header is finished, but to no avail. If I understand you correctly, you should be able to do this in the same way as the HTTP analyer does it. I thinkt this is the relevant type from http-protocol.pac: type HTTP_Headers = HTTP_Header[] &until($input.length() == 0); Does something similar work for you? Robin -- Robin Sommer * Phone +1 (510) 931-5555 * robin at icir.org LBNL/ICSI * Fax +1 (510) 666-2956 * www.icir.org From rpang at cs.princeton.edu Thu May 17 10:40:33 2007 From: rpang at cs.princeton.edu (Ruoming Pang) Date: Thu, 17 May 2007 13:40:33 -0400 Subject: [Bro] Activating a scanner within a scanner? In-Reply-To: <20070516013727.GJ20205@icir.org> References: <20070510222310.GB12457@icir.org> <20070516013727.GJ20205@icir.org> Message-ID: > > I tried looking at the HTTP analyzer, because this protocol uses a > > newline to show when the header is finished, but to no avail. > > If I understand you correctly, you should be able to do this in the > same way as the HTTP analyer does it. I thinkt this is the relevant > type from http-protocol.pac: > > type HTTP_Headers = HTTP_Header[] &until($input.length() == 0); > > Does something similar work for you? Richard, >From what you described, it should work for you. Please keep us updated if it doesn't. To provide a bit of background, $input in the "until" clause corresponds to a line without the trailing CRLF, because HTTP_Header is defined with a &oneline attribute. By the way, this means $input can also be used in more fancy conditions, such as ($input == "foo"). Ruoming From muscletot at gmail.com Wed May 23 16:27:15 2007 From: muscletot at gmail.com (Mike Wood) Date: Wed, 23 May 2007 16:27:15 -0700 Subject: [Bro] howto append a subnet addr to a set via cmd line... Message-ID: <7f3e32390705231627ldb90bcdk94895cd22c09188f@mail.gmail.com> I would like to have in myscript.bro const SOMENETS : set[subnet] &redef; and then run bro 'SOMENETS += 1.2.3.0/8' myscript.bro but I get an error , line 1 (1.2.3.0/8): error, bad initializer Any chance this will work w/ slightly different syntax on the cmd line...? Thanks, Mike From vern at icir.org Wed May 23 23:10:37 2007 From: vern at icir.org (Vern Paxson) Date: Wed, 23 May 2007 23:10:37 -0700 Subject: [Bro] howto append a subnet addr to a set via cmd line... In-Reply-To: <7f3e32390705231627ldb90bcdk94895cd22c09188f@mail.gmail.com> (Wed, 23 May 2007 16:27:15 PDT). Message-ID: <200705240610.l4O6AbDx090397@jaguar.icir.org> > bro 'SOMENETS += 1.2.3.0/8' myscript.bro += doesn't support additions to sets. Instead use: bro -e 'add SOMENETS[1.2.3.0/8]' myscript.bro - Vern From kiesling at ICSI.Berkeley.EDU Fri May 25 15:46:28 2007 From: kiesling at ICSI.Berkeley.EDU (Tobias Kiesling) Date: Fri, 25 May 2007 15:46:28 -0700 Subject: [Bro] ssl binpac analyzer -- patches In-Reply-To: <2e4d286d0705251544u4bc2d931hf7288693b7c8ba04@mail.gmail.com> References: <2e4d286d0705251540m48d76dc0h4b34facc7aa2849e@mail.gmail.com> <2e4d286d0705251544u4bc2d931hf7288693b7c8ba04@mail.gmail.com> Message-ID: <2e4d286d0705251546w76039027ndf6b750d2dd01f27@mail.gmail.com> Hi all, I have been working on the development of an ssl analyzer using binpac, which is now finished. In the process of development, I have found and fixed some bugs in binpac, as well as added an additional feature to binpac. Attached you can find the patches for binpac in the bro-1.2.1 distribution (binpac-[1-8].patch) with a file documenting the patches ( binpac-patch-doc.txt). I would like to ask you to have a look over those patches (especially Ruoming). All of the patches should be applied with patch -p1 in the main bro directory. Cheers, Tobias -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20070525/1d259a5b/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: binpac-1.patch Type: application/octet-stream Size: 1009 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20070525/1d259a5b/attachment.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: binpac-2.patch Type: application/octet-stream Size: 2884 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20070525/1d259a5b/attachment-0001.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: binpac-3.patch Type: application/octet-stream Size: 1402 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20070525/1d259a5b/attachment-0002.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: binpac-4.patch Type: application/octet-stream Size: 539 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20070525/1d259a5b/attachment-0003.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: binpac-5.patch Type: application/octet-stream Size: 2166 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20070525/1d259a5b/attachment-0004.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: binpac-6.patch Type: application/octet-stream Size: 1094 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20070525/1d259a5b/attachment-0005.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: binpac-7.patch Type: application/octet-stream Size: 750 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20070525/1d259a5b/attachment-0006.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: binpac-8.patch Type: application/octet-stream Size: 6161 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20070525/1d259a5b/attachment-0007.obj -------------- next part -------------- binpac fixes ---------------- numbers of issues below correspond to the patch numbers (1) correct calculation of minimal header size in pac_expr.cc - problem: EXPR_CALLARGS and EXPR_CASE not considered for the calculation of minimal header size - solution: added two cases in switch stmt of Expr::MinimalHeaderSize for EXPR_CALLARGS and EXPR_CASE (2) ensure parsing of fields first referenced in a case expression or let field with an &if attribute - problem: in cases where the if expression evaluates to false or the proper case does not occur, fields get not parsed at all - solution: force evaluation of all IDs referenced in a let field with if attribute or a case expression before the body of the corresponding switch stmt or the if stmt - added public method Expr::ForceIDEval, properly called before generating the code of a field with if attribute or the case expression (3) properly assert the use of fields with an if attribute - problem: the use of fields with an if attribute was not asserted in all cases and asserted in the wrong way in some others due to the corresponding BINPAC_ASSERT only called upon parsing the field - solution: perform BINPAC_ASSERT upon calling the fields accessor function - moved BINPAC_ASSERT statement from LetField::GenEval to Type::GenPubDecls (4) incremental input with records with a non-negative StaticSize - problem: incremental input with records with a StaticSize >= 0 cannot be performed due to necessary length attribute, leading to an invalid call of GenBoundaryCheck in RecordType::DoGenParseCode - solution: added a check for incremental input in RecordType::DoGenParseCode before calling GenBoundaryCheck (5) empty type with incremental input - problem: with an empty type and incremental input, although the Parse function is created, it is never called, leading to problems, if additional actions are to be performed when encountering that empty type - solution: generate call to Parse of empty type in Type::GenParseBuffer (6) parsing loop in flow ParseBuffer (while(true)) - problem: while(true) leads to problems after parsing of a type is complete; at this time, it is unexpected that parsing continues, even if no data is available in the flow buffer - solution: check if data is available before starting a new parsing cycle - added a method data_available to FlowBuffer - changed while(true) in FlowDecl::GenCodeFlowUnit to while(flow_buffer_->data_available()) (7) initialization of flow buffer in CaseType with bufferable fields in cases - problem: initialization of buffer occurs in every Parse call, regardless if it was initialized before or not; initialization is correct only on first such occurence - solution: check to buffer_state is to be created always when buffering_state_id is in environment in Type::GenBufferConfig - changed condition from buffering_state_var_field_ to env->GetDataType(buffering_state_id) (8) allowing init and cleanup code to be redefined, as well as addition of code to FlowEOF calls in analyzer and flow - problem 1: when refining an analyzer or flow definition, additional init and cleanup code was not allowed, if these were already defined before; this leads to problems when adding new members, as these cannot be initialized and destroyed properly - solution: allow init and cleanup code to be specified more than once - changed deifnitions and usage of constructor_helper and destructor_helper to allow for lists of constructor and destructor helpers (similar to member declarations) in pac_analyzer.h and pac_analyzer.cc - problem 2: in some cases, it is desirable to execute code when encountering the end of the input stream, which is not possible in binpac - solution: added a %eof binpac primitive similar to %init, which adds code to the FlowEOF function of an analyzer or a flow From robin at icir.org Fri May 25 16:16:00 2007 From: robin at icir.org (Robin Sommer) Date: Fri, 25 May 2007 16:16:00 -0700 Subject: [Bro] ssl binpac analyzer -- patches In-Reply-To: <2e4d286d0705251546w76039027ndf6b750d2dd01f27@mail.gmail.com> References: <2e4d286d0705251540m48d76dc0h4b34facc7aa2849e@mail.gmail.com> <2e4d286d0705251544u4bc2d931hf7288693b7c8ba04@mail.gmail.com> <2e4d286d0705251546w76039027ndf6b750d2dd01f27@mail.gmail.com> Message-ID: <20070525231559.GB18305@icir.org> On Fri, May 25, 2007 at 15:46 -0700, Tobias Kiesling wrote: > binpac-patch-doc.txt). I would like to ask you to have a look over those > patches (especially Ruoming). Ruoming, if you can do quickly skim over the patches to see if they're fine, I'll apply the patches to my branch and also integrate Tobias' new SSL analyzer. Robin -- Robin Sommer * Phone +1 (510) 931-5555 * robin at icir.org LBNL/ICSI * Fax +1 (510) 666-2956 * www.icir.org From rpang at cs.princeton.edu Fri May 25 18:19:28 2007 From: rpang at cs.princeton.edu (Ruoming Pang) Date: Fri, 25 May 2007 21:19:28 -0400 Subject: [Bro] ssl binpac analyzer -- patches In-Reply-To: <20070525231559.GB18305@icir.org> References: <2e4d286d0705251540m48d76dc0h4b34facc7aa2849e@mail.gmail.com> <2e4d286d0705251544u4bc2d931hf7288693b7c8ba04@mail.gmail.com> <2e4d286d0705251546w76039027ndf6b750d2dd01f27@mail.gmail.com> <20070525231559.GB18305@icir.org> Message-ID: Yes, I'm looking at them now. Tobias: Thanks for sharing the patches and providing detailed explanations! Ruoming On 5/25/07, Robin Sommer wrote: > > On Fri, May 25, 2007 at 15:46 -0700, Tobias Kiesling wrote: > > > binpac-patch-doc.txt). I would like to ask you to have a look over those > > patches (especially Ruoming). > > Ruoming, if you can do quickly skim over the patches to see if > they're fine, I'll apply the patches to my branch and also integrate > Tobias' new SSL analyzer. > > Robin > > -- > Robin Sommer * Phone +1 (510) 931-5555 * robin at icir.org > LBNL/ICSI * Fax +1 (510) 666-2956 * www.icir.org > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > From ju at klipix.org Sat May 26 12:21:57 2007 From: ju at klipix.org (Julien Desfossez) Date: Sat, 26 May 2007 15:21:57 -0400 Subject: [Bro] include icmp6.h problem Message-ID: <465888D5.5090107@klipix.org> Hello, I'm developping support for ICMPv6 in Bro and I have a problem with an include I can't solve. I have just added "#include " in net_util.h and I get errors not (apparently) related to this change : if g++ -DHAVE_CONFIG_H -I. -I. -I.. -I. -I../src/binpac/lib -I../src -I. -I.. -Ilibedit -I../linux-include -O -W -Wall -Wno-unused -I../linux-include -g -O2 -MT dns_pac.o -MD -MP -MF ".deps/dns_pac.Tpo" -c -o dns_pac.o dns_pac.cc; \ then mv -f ".deps/dns_pac.Tpo" ".deps/dns_pac.Po"; else rm -f ".deps/dns_pac.Tpo"; exit 1; fi ../src/Analyzer.h:100: warning: 'class Analyzer::OutputHandler' has virtual functions but non-virtual destructor ../src/dns_pac.h:328: error: expected ',' or '...' before '.' token ../src/dns_pac.h:424: error: expected ';' before '.' token ../src/dns_pac.h:425: error: expected `;' before 'uint16' ../src/dns_pac.h:453: error: expected ';' before '.' token ../src/dns_pac.h:454: error: expected `;' before 'uint16' dns_pac.cc:535: error: expected ',' or '...' before '.' token dns_pac.cc: In constructor 'binpac::DNS::DNS_rdata::DNS_rdata(binpac::DNS::DNS_message*, binpac::uint16)': dns_pac.cc:547: error: request for member 'icmp6_type' in 'rr_hdr', which is of non-class type 'binpac::uint16' dns_pac.cc:548: error: argument of type 'binpac::uint16 (binpac::DNS::DNS_rdata::)()const' does not match 'binpac::uint16' dns_pac.cc: In member function 'int binpac::DNS::DNS_rdata::Parse(const binpac::uint8*, const binpac::uint8*, binpac::DNS::ContextDNS*, int)': dns_pac.cc:614: error: 'rr_hdr' was not declared in this scope dns_pac.cc: In member function 'int binpac::DNS::DNS_rr::Parse(const binpac::uint8*, const binpac::uint8*, binpac::DNS::ContextDNS*, int)': dns_pac.cc:779: error: 'rr_hdr' was not declared in this scope dns_pac.cc: In member function 'Val* binpac::DNS::DNS_Flow::build_dns_answer(binpac::DNS::DNS_rr*)': dns_pac.cc:1244: error: 'class binpac::DNS::DNS_rr' has no member named 'rr_hdr' dns_pac.cc: In member function 'Val* binpac::DNS::DNS_Flow::build_edns_additional(binpac::DNS::DNS_rr*)': dns_pac.cc:1280: error: 'class binpac::DNS::DNS_rr' has no member named 'rr_hdr' dns_pac.cc: In member function 'bool binpac::DNS::DNS_Flow::process_dns_rr(binpac::DNS::DNS_rr*)': dns_pac.cc:1311: error: 'class binpac::DNS::DNS_rr' has no member named 'rr_hdr' dns_pac.cc:1320: error: 'class binpac::DNS::DNS_rr' has no member named 'rr_hdr' I have made this modification in the bro-1.2.1 sources without any other changes. I have the same problem on GNU/Linux and OpenBSD 4. Does anyone have an idea how to solve/debug this problem ? Thanks a lot ! Julien Desfossez From vern at icir.org Sat May 26 12:38:49 2007 From: vern at icir.org (Vern Paxson) Date: Sat, 26 May 2007 12:38:49 -0700 Subject: [Bro] include icmp6.h problem In-Reply-To: <465888D5.5090107@klipix.org> (Sat, 26 May 2007 15:21:57 EDT). Message-ID: <200705261938.l4QJcnXi091065@jaguar.icir.org> > I have just added "#include " in net_util.h and I get > errors not (apparently) related to this change : > ... > ../src/dns_pac.h:328: error: expected ',' or '...' before '.' token > ... > Does anyone have an idea how to solve/debug this problem ? My first suggestion would be to compile using g++ -E to get the output from the preprocessor (sent to stdout). Often a bizarre problem like this after including a header is because the header introduces a #define that conflicts with a name used elsewhere in a different context. Vern From ju at klipix.org Sat May 26 14:50:10 2007 From: ju at klipix.org (Julien Desfossez) Date: Sat, 26 May 2007 17:50:10 -0400 Subject: [Bro] include icmp6.h problem In-Reply-To: <200705261938.l4QJcnXi091065@jaguar.icir.org> References: <200705261938.l4QJcnXi091065@jaguar.icir.org> Message-ID: <4658AB92.9020304@klipix.org> >> I have just added "#include " in net_util.h and I get >> errors not (apparently) related to this change : >> ... >> ../src/dns_pac.h:328: error: expected ',' or '...' before '.' token >> ... >> Does anyone have an idea how to solve/debug this problem ? >> > > My first suggestion would be to compile using g++ -E to get the output > from the preprocessor (sent to stdout). Often a bizarre problem like this > after including a header is because the header introduces a #define that > conflicts with a name used elsewhere in a different context. > Indeed it was a #define that conflicted with another variable : rr_type. The problem now is that I need to include icmp6.h but it conflicts with dns_pac. What do you think is the best practice to solve this issue : rename the variable in dns_pac, include a modified version of icmp6.h with Bro... ? Thanks, Julien Desfossez From vern at icir.org Sat May 26 15:00:51 2007 From: vern at icir.org (Vern Paxson) Date: Sat, 26 May 2007 15:00:51 -0700 Subject: [Bro] include icmp6.h problem In-Reply-To: <4658AB92.9020304@klipix.org> (Sat, 26 May 2007 17:50:10 EDT). Message-ID: <200705262200.l4QM0pES092691@jaguar.icir.org> > What do you think is the best practice to solve this issue : rename the > variable in dns_pac, include a modified version of icmp6.h with Bro... ? The first question is whether the icmp6.h include really needs to be in net_util.h, or if it could be confined to the source file(s) that use the definitions. If it does, then following the include with #undef rr_type (and a comment as to why) is probably the way to go, ugly as it is. Vern From ju at klipix.org Sat May 26 15:33:56 2007 From: ju at klipix.org (Julien Desfossez) Date: Sat, 26 May 2007 18:33:56 -0400 Subject: [Bro] include icmp6.h problem In-Reply-To: <200705262200.l4QM0pES092691@jaguar.icir.org> References: <200705262200.l4QM0pES092691@jaguar.icir.org> Message-ID: <4658B5D4.8080607@klipix.org> >> What do you think is the best practice to solve this issue : rename the >> variable in dns_pac, include a modified version of icmp6.h with Bro... ? >> > > The first question is whether the icmp6.h include really needs to be in > net_util.h, or if it could be confined to the source file(s) that use the > definitions. > Actually I just need it in Sessions.cc (for the switch on the ICMP type) so I will use the #undef solution : it's less ugly than keeping a modified version of icmp6.h. Thanks for your help ! Julien Desfossez From vern at icir.org Sat May 26 15:54:00 2007 From: vern at icir.org (Vern Paxson) Date: Sat, 26 May 2007 15:54:00 -0700 Subject: [Bro] include icmp6.h problem In-Reply-To: <4658B5D4.8080607@klipix.org> (Sat, 26 May 2007 18:33:56 EDT). Message-ID: <200705262254.l4QMs0td093190@jaguar.icir.org> > Actually I just need it in Sessions.cc (for the switch on the ICMP type) > so I will use the #undef solution : it's less ugly than keeping a > modified version of icmp6.h. I think you misunderstood me. If you just need it in Sessions.cc, then just include it in Sessions.cc rather than in net_util.h (and include it after DNS-binpac.h, with a comment, if that's the only conflict). A little better than using an #undef. Vern From ju at klipix.org Sat May 26 15:59:32 2007 From: ju at klipix.org (Julien Desfossez) Date: Sat, 26 May 2007 18:59:32 -0400 Subject: [Bro] include icmp6.h problem In-Reply-To: <200705262254.l4QMs0td093190@jaguar.icir.org> References: <200705262254.l4QMs0td093190@jaguar.icir.org> Message-ID: <4658BBD4.3080009@klipix.org> > just include it in Sessions.cc rather than in net_util.h (and include it > after DNS-binpac.h, with a comment, if that's the only conflict). A little > better than using an #undef. > Indeed it works ! I was sure I had tested it before... Moreover, it's clean now :-) Thanks again, Julien Desfossez From jmzhou.ml at gmail.com Tue May 29 11:56:30 2007 From: jmzhou.ml at gmail.com (jmzhou.ml at gmail.com) Date: Tue, 29 May 2007 11:56:30 -0700 (PDT) Subject: [Bro] ssl binpac analyzer -- patches In-Reply-To: <4658AB92.9020304@klipix.org> References: <200705261938.l4QJcnXi091065@jaguar.icir.org> <4658AB92.9020304@klipix.org> Message-ID: Some comments about the patch set Patch 4: I can confirm that this is a bug. I had the same problem and same fix at my side. Patch 5: I think the new call (DoGenParseCode) in pac_type.cc should be replaced by GenParseCode3. Imagine that a record field (identifier f1) is empty type and there are other fields (identifier f2, f3, ...) after field f1. If you call DoGenParseCode, the identifier f1 will not be evaluated like in GenParseCode3 (pac_type.cc:754-755). This will result in re-entrance of RecordDataField::GenParseCode (pac_record.cc:420) because when f2->prev()-GenParseCode (pac_record.cc:428) is called, env->Evaluated( id()) will return false. This is an infinite loop - till at some point of binpac there will be an assertion failure. Patch 6: I think there is another issue here: if some buffering request has been sent to the flow buffer, and the flow buffer is not ready with partial data, we will end up with entering the loop - a waste of CPU cycles. From kiesling at ICSI.Berkeley.EDU Tue May 29 17:21:40 2007 From: kiesling at ICSI.Berkeley.EDU (Tobias Kiesling) Date: Tue, 29 May 2007 17:21:40 -0700 Subject: [Bro] ssl binpac analyzer -- patches In-Reply-To: References: <200705261938.l4QJcnXi091065@jaguar.icir.org> <4658AB92.9020304@klipix.org> Message-ID: <2e4d286d0705291721u2c555dbblc9bd4fb400701ce2@mail.gmail.com> On 5/29/07, jmzhou.ml at gmail.com wrote: > > Some comments about the patch set > > Patch 4: I can confirm that this is a bug. I had the same problem and > same fix at my side. > > Patch 5: I think the new call (DoGenParseCode) in pac_type.cc should be > replaced by GenParseCode3. Imagine that a record field (identifier f1) > is empty type and there are other fields (identifier f2, f3, ...) after > field f1. If you call DoGenParseCode, the identifier f1 will not be > evaluated like in GenParseCode3 (pac_type.cc:754-755). This will result > in re-entrance of RecordDataField::GenParseCode (pac_record.cc:420) > because > when f2->prev()-GenParseCode (pac_record.cc:428) is called, > env->Evaluated( > id()) will return false. This is an infinite loop - till at some point of > binpac there will be an assertion failure. I think I can see your point and also that it might generally be more safe to call GenParseCode3 instead (I attached the updated patch file). > Patch 6: I think there is another issue here: if some buffering request > has been sent to the flow buffer, and the flow buffer is not ready with > partial data, we will end up with entering the loop - a waste of CPU > cycles. Maybe you are right, but what would you suggest as change? Do you want to check whether the buffer is ready before entering the loop? But then it has to be ensured that the buffer is properly initialized in any case. At the moment I cannot see all the consequences of such a change. And do you think that the impact on performance is really relevant? Thanks for the comments, Tobias _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20070529/02bd855a/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: binpac-5.patch Type: application/octet-stream Size: 2166 bytes Desc: not available Url : http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20070529/02bd855a/attachment.obj From jmzhou.ml at gmail.com Wed May 30 09:08:02 2007 From: jmzhou.ml at gmail.com (jmzhou.ml at gmail.com) Date: Wed, 30 May 2007 09:08:02 -0700 (PDT) Subject: [Bro] ssl binpac analyzer -- patches In-Reply-To: <2e4d286d0705291721u2c555dbblc9bd4fb400701ce2@mail.gmail.com> References: <200705261938.l4QJcnXi091065@jaguar.icir.org> <4658AB92.9020304@klipix.org> <2e4d286d0705291721u2c555dbblc9bd4fb400701ce2@mail.gmail.com> Message-ID: On Tue, 29 May 2007, Tobias Kiesling wrote: > On 5/29/07, jmzhou.ml at gmail.com wrote: > >> Patch 6: I think there is another issue here: if some buffering request >> has been sent to the flow buffer, and the flow buffer is not ready with >> partial data, we will end up with entering the loop - a waste of CPU >> cycles. > > > Maybe you are right, but what would you suggest as change? Do you want to > check whether the buffer is ready before entering the loop? But then it has > to be ensured that the buffer is properly initialized in any case. At the > moment I cannot see all the consequences of such a change. And do you think > that the impact on performance is really relevant? The cost can be a little bit expensive if there are many layers of parsing. You end up with many unnecessary calls to parsing functions and condition jumps. One possible approach is like this: . add a new boolean member have_pending_request to FlowBuffer, initialized as false. . set have_pending_request to true in call NewFrame and NewLine. . reset have_pending_request to false in call DiscardData. . change the while loop condition to: while (flow_buffer->data_available() && (!flow_buffer->have_pending_request() || flow_buffer->ready())) Analysis: 1. The first time, data_available = true, !have_pending_request = true, we enter into the loop. Good. 2. All data are consumed, data_available = false, we do not enter into the loop. Good. 3. A request is not satisfied because of partial data: data_available = true, !have_pending_request = false, ready = false, we do not enter into the loop. Good. 4. Parsing is forced to stop because of exception. data_available = false. We do not enter into the loop. Good. 5. Parsing current dataunit has finished, we still have residual data - wrong data? data_available = true, !have_pending_request = false, ready = true (from last parsing). We enter into the loop, and start a new round of parsing. As expected. Good. So far so good. :-) Jimmy ____________________________________________________________ The future is not set. There is no fate but what we make for ourselves. - Terminator II, Judgment Day ------------------------------------------------------------ From robin at icir.org Wed May 30 18:57:18 2007 From: robin at icir.org (Robin Sommer) Date: Wed, 30 May 2007 18:57:18 -0700 Subject: [Bro] Continuous memory growth without tcp.bro In-Reply-To: References: Message-ID: <20070531015718.GD8954@icir.org> On Mon, May 07, 2007 at 14:17 -0700, Thomas, Eric D. wrote: > 1. bro -i eth0 profiling.bro So, I've (finally) tried this and Bro runs fine for me; memory stays stable after some time. Which Bro version are you using? Can you send me prof.log? Robin -- Robin Sommer * Phone +1 (510) 931-5555 * robin at icir.org LBNL/ICSI * Fax +1 (510) 666-2956 * www.icir.org From kiesling at ICSI.Berkeley.EDU Wed May 30 22:11:07 2007 From: kiesling at ICSI.Berkeley.EDU (Tobias Kiesling) Date: Wed, 30 May 2007 22:11:07 -0700 Subject: [Bro] ssl binpac analyzer -- patches In-Reply-To: References: <200705261938.l4QJcnXi091065@jaguar.icir.org> <4658AB92.9020304@klipix.org> <2e4d286d0705291721u2c555dbblc9bd4fb400701ce2@mail.gmail.com> Message-ID: <2e4d286d0705302211l48fe0853y92213923e405b6b2@mail.gmail.com> Jimmy, your suggestion sounds good. I implemented the changes and tested them with my ssl analyzer. Everything worked as expected. :-) Attached you can find the patch for these changes (to be applied with -p0 in the bro directory, after applying all of the other binpac patches that I provided before). Could you test this further with whatever binpac analyzers you have got? Anyone else any comments on this issue (Ruoming)? Tobias On 5/30/07, jmzhou.ml at gmail.com wrote: > > On Tue, 29 May 2007, Tobias Kiesling wrote: > > > On 5/29/07, jmzhou.ml at gmail.com wrote: > > > >> Patch 6: I think there is another issue here: if some buffering request > >> has been sent to the flow buffer, and the flow buffer is not ready with > >> partial data, we will end up with entering the loop - a waste of CPU > >> cycles. > > > > > > Maybe you are right, but what would you suggest as change? Do you want > to > > check whether the buffer is ready before entering the loop? But then it > has > > to be ensured that the buffer is properly initialized in any case. At > the > > moment I cannot see all the consequences of such a change. And do you > think > > that the impact on performance is really relevant? > > The cost can be a little bit expensive if there are many layers of > parsing. > You end up with many unnecessary calls to parsing functions and condition > jumps. One possible approach is like this: > > . add a new boolean member have_pending_request to FlowBuffer, initialized > as false. > > . set have_pending_request to true in call NewFrame and NewLine. > > . reset have_pending_request to false in call DiscardData. > > . change the while loop condition to: > while (flow_buffer->data_available() && > (!flow_buffer->have_pending_request() || flow_buffer->ready())) > > Analysis: > > 1. The first time, data_available = true, !have_pending_request = true, > we enter into the loop. Good. > > 2. All data are consumed, data_available = false, we do not enter into > the loop. Good. > > 3. A request is not satisfied because of partial data: data_available = > true, !have_pending_request = false, ready = false, we do not enter into > the loop. Good. > > 4. Parsing is forced to stop because of exception. data_available = > false. > We do not enter into the loop. Good. > > 5. Parsing current dataunit has finished, we still have residual data - > wrong data? data_available = true, !have_pending_request = false, ready = > true (from last parsing). We enter into the loop, and start a new round of > parsing. As expected. Good. > > So far so good. :-) > > Jimmy > > ____________________________________________________________ > The future is not set. There is no fate but what we make > for ourselves. - Terminator II, Judgment Day > ------------------------------------------------------------ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.ICSI.Berkeley.EDU/pipermail/bro/attachments/20070530/036d759d/attachment.html -------------- next part -------------- Index: src/binpac/lib/binpac_buffer.cc =================================================================== --- src/binpac/lib/binpac_buffer.cc (revision 14) +++ src/binpac/lib/binpac_buffer.cc (working copy) @@ -32,6 +32,7 @@ data_seq_at_orig_data_end_ = 0; eof_ = false; + have_pending_request_ = false; buffer_n_ = 0; @@ -119,6 +120,7 @@ mode_ = LINE_MODE; frame_length_ = 0; chunked_ = false; + have_pending_request_ = true; if ( state_ == FRAME_0 ) ResetLineState(); MarkOrCopyLine(); @@ -130,6 +132,7 @@ mode_ = FRAME_MODE; frame_length_ = frame_length; chunked_ = chunked; + have_pending_request_ = true; MarkOrCopyFrame(); } @@ -148,6 +151,7 @@ { mode_ = UNKNOWN_MODE; message_complete_ = false; + have_pending_request_ = false; orig_data_begin_ = orig_data_end_ = 0; buffer_n_ = 0; Index: src/binpac/lib/binpac_buffer.h =================================================================== --- src/binpac/lib/binpac_buffer.h (revision 14) +++ src/binpac/lib/binpac_buffer.h (working copy) @@ -82,6 +82,8 @@ bool eof() const { return eof_; } void set_eof(); + bool have_pending_request() const { return have_pending_request_; } + protected: // Reset the buffer for a new message void NewMessage(); @@ -136,6 +138,7 @@ int data_seq_at_orig_data_end_; bool eof_; + bool have_pending_request_; }; typedef FlowBuffer *flow_buffer_t; Index: src/binpac/pac_flow.cc =================================================================== --- src/binpac/pac_flow.cc (revision 14) +++ src/binpac/pac_flow.cc (working copy) @@ -277,9 +277,11 @@ env_->RValue(begin_of_data), env_->RValue(end_of_data)); - out_cc->println("while ( %s->data_available() )", + out_cc->println("while ( %s->data_available() && ", env_->LValue(flow_buffer_id)); out_cc->inc_indent(); + out_cc->println("( !%s->have_pending_request() || %s->ready() ) )", + env_->LValue(flow_buffer_id), env_->LValue(flow_buffer_id)); out_cc->println("{"); // Generate a new dataunit if necessary From jmzhou.ml at gmail.com Thu May 31 08:26:07 2007 From: jmzhou.ml at gmail.com (jmzhou.ml at gmail.com) Date: Thu, 31 May 2007 08:26:07 -0700 (PDT) Subject: [Bro] ssl binpac analyzer -- patches In-Reply-To: <2e4d286d0705302211l48fe0853y92213923e405b6b2@mail.gmail.com> References: <200705261938.l4QJcnXi091065@jaguar.icir.org> <4658AB92.9020304@klipix.org> <2e4d286d0705291721u2c555dbblc9bd4fb400701ce2@mail.gmail.com> <2e4d286d0705302211l48fe0853y92213923e405b6b2@mail.gmail.com> Message-ID: Yes, the changes work well at my side. Some problems of binpac: . Split binpac out of bro source tree. I think to make binpac standalone makes testing/developing much easier. One can develop new analyzers and test them with dedicated .pcap files. . Binpac does not support SunRPC over TCP now. There are four extra bytes prepended in RPC packets. Either TCP layer decoder should take care of these extra bytes, or the RPC decoder has to do something with it. In addition, &exportsourcedata is used in RPC/UDP decoder based on datagram mode. It is not supported by flowunit mode. This means, we cannot simply change the decoder from datagram mode to flowunit mode for RPC/TCP. Finally, a Ref call is missing in the NewCall function when a call already exists, and a Unref call is not correctly called in FinishCall. The Ref/Unref calls also need to be re-written. These changes are necessary to properly support retransmission. The problem in the original code is: if you get two identical calls followed by two identical replies, the decoder will fail on the second reply. I met with this problem at my site. I give the rough idea of the fix below: function NewCall (xid: uint32, call: RPC_Call): bool %{ if (find_orig_call (xid)) { handle_mismatch_between_current_call_and_previous_call; orig_call->msg()->Ref(); return false; } else { insert_new_call_into_call_table (xid); new_call->msg()->Ref(); } %} function FinishCall (call: RPC_Call): void %{ xid = call->msg()->xid(); /* save it because msg may be freed below */ int refcount = Unref(call->msg()); if (refcount <= 0) erase_xid_from_call_table; %} class RefCount { int Ref() { return ++count;} int Unref() {return -- count;} ... } inline int Unref (RefCount* x) { int refcount = 0; if (x) { refcount = x->Unref(); if (refcount <= 0) delete x; } return refcount; } Regards, Jimmy On Wed, 30 May 2007, Tobias Kiesling wrote: > Jimmy, > > your suggestion sounds good. I implemented the changes and tested them with > my ssl analyzer. Everything worked as expected. :-) > > Attached you can find the patch for these changes (to be applied with -p0 in > the bro directory, after applying all of the other binpac patches that I > provided before). Could you test this further with whatever binpac analyzers > you have got? > > Anyone else any comments on this issue (Ruoming)? > > Tobias > > > On 5/30/07, jmzhou.ml at gmail.com wrote: >> >> On Tue, 29 May 2007, Tobias Kiesling wrote: >> >> > On 5/29/07, jmzhou.ml at gmail.com wrote: >> > >> >> Patch 6: I think there is another issue here: if some buffering request >> >> has been sent to the flow buffer, and the flow buffer is not ready with >> >> partial data, we will end up with entering the loop - a waste of CPU >> >> cycles. >> > >> > >> > Maybe you are right, but what would you suggest as change? Do you want >> to >> > check whether the buffer is ready before entering the loop? But then it >> has >> > to be ensured that the buffer is properly initialized in any case. At >> the >> > moment I cannot see all the consequences of such a change. And do you >> think >> > that the impact on performance is really relevant? >> >> The cost can be a little bit expensive if there are many layers of >> parsing. >> You end up with many unnecessary calls to parsing functions and condition >> jumps. One possible approach is like this: >> >> . add a new boolean member have_pending_request to FlowBuffer, initialized >> as false. >> >> . set have_pending_request to true in call NewFrame and NewLine. >> >> . reset have_pending_request to false in call DiscardData. >> >> . change the while loop condition to: >> while (flow_buffer->data_available() && >> (!flow_buffer->have_pending_request() || flow_buffer->ready())) >> >> Analysis: >> >> 1. The first time, data_available = true, !have_pending_request = true, >> we enter into the loop. Good. >> >> 2. All data are consumed, data_available = false, we do not enter into >> the loop. Good. >> >> 3. A request is not satisfied because of partial data: data_available = >> true, !have_pending_request = false, ready = false, we do not enter into >> the loop. Good. >> >> 4. Parsing is forced to stop because of exception. data_available = >> false. >> We do not enter into the loop. Good. >> >> 5. Parsing current dataunit has finished, we still have residual data - >> wrong data? data_available = true, !have_pending_request = false, ready = >> true (from last parsing). We enter into the loop, and start a new round of >> parsing. As expected. Good. >> >> So far so good. :-) >> >> Jimmy >> >> ____________________________________________________________ >> The future is not set. There is no fate but what we make >> for ourselves. - Terminator II, Judgment Day >> ------------------------------------------------------------ >> >> > ____________________________________________________________ The future is not set. There is no fate but what we make for ourselves. - Terminator II, Judgment Day ------------------------------------------------------------ From bltierney at lbl.gov Thu May 31 13:19:25 2007 From: bltierney at lbl.gov (Brian Tierney) Date: Thu, 31 May 2007 15:19:25 -0500 Subject: [Bro] Registration for the Bro Hands-on Workshop now open Message-ID: <465F2DCD.7070507@lbl.gov> Registration for the Bro Hands-on Workshop, July 23-25, at the San Diego Supercomputer Center (SDSC) is now open. See: http://www.bro-ids.org/bro-workshop-2007/Bro07-workshop.html Space is limited, so if you wish to attend please register as soon as possible. Registration deadline is July 8. The workshop will start with a half day Bro overview and tutorial that is free and open to anyone that signs up. This will be followed by 2 day hands-on workshop that is only open to a smaller number registrants ($50 registration fee). You must have a Unix-based laptop with a working Bro installation to attend this part of the workshop. If you have written any custom Bro policy scripts or Bro log file analysis tools that you would like to present, please let us know. If enough people register soon we will look into reserving a block of hotel rooms. From rpang at cs.princeton.edu Thu May 31 18:09:10 2007 From: rpang at cs.princeton.edu (Ruoming Pang) Date: Thu, 31 May 2007 21:09:10 -0400 Subject: [Bro] ssl binpac analyzer -- patches In-Reply-To: <2e4d286d0705302211l48fe0853y92213923e405b6b2@mail.gmail.com> References: <200705261938.l4QJcnXi091065@jaguar.icir.org> <4658AB92.9020304@klipix.org> <2e4d286d0705291721u2c555dbblc9bd4fb400701ce2@mail.gmail.com> <2e4d286d0705302211l48fe0853y92213923e405b6b2@mail.gmail.com> Message-ID: Tobias and Jimmy, Thanks to you both for looking into this. I will take a look at this now. Ruoming On 5/31/07, Tobias Kiesling wrote: > Jimmy, > > your suggestion sounds good. I implemented the changes and tested them with > my ssl analyzer. Everything worked as expected. :-) > > Attached you can find the patch for these changes (to be applied with -p0 in > the bro directory, after applying all of the other binpac patches that I > provided before). Could you test this further with whatever binpac analyzers > you have got? > > Anyone else any comments on this issue (Ruoming)? > > Tobias > > > > On 5/30/07, jmzhou.ml at gmail.com wrote: > > On Tue, 29 May 2007, Tobias Kiesling wrote: > > > > > On 5/29/07, jmzhou.ml at gmail.com wrote: > > > > > >> Patch 6: I think there is another issue here: if some buffering request > > >> has been sent to the flow buffer, and the flow buffer is not ready with > > >> partial data, we will end up with entering the loop - a waste of CPU > > >> cycles. > > > > > > > > > Maybe you are right, but what would you suggest as change? Do you want > to > > > check whether the buffer is ready before entering the loop? But then it > has > > > to be ensured that the buffer is properly initialized in any case. At > the > > > moment I cannot see all the consequences of such a change. And do you > think > > > that the impact on performance is really relevant? > > > > The cost can be a little bit expensive if there are many layers of > parsing. > > You end up with many unnecessary calls to parsing functions and condition > > jumps. One possible approach is like this: > > > > . add a new boolean member have_pending_request to FlowBuffer, initialized > > as false. > > > > . set have_pending_request to true in call NewFrame and NewLine. > > > > . reset have_pending_request to false in call DiscardData. > > > > . change the while loop condition to: > > while (flow_buffer->data_available() && > > (!flow_buffer->have_pending_request() || > flow_buffer->ready())) > > > > Analysis: > > > > 1. The first time, data_available = true, !have_pending_request = true, > > we enter into the loop. Good. > > > > 2. All data are consumed, data_available = false, we do not enter into > > the loop. Good. > > > > 3. A request is not satisfied because of partial data: data_available = > > true, !have_pending_request = false, ready = false, we do not enter into > > the loop. Good. > > > > 4. Parsing is forced to stop because of exception. data_available = > false. > > We do not enter into the loop. Good. > > > > 5. Parsing current dataunit has finished, we still have residual data - > > wrong data? data_available = true, !have_pending_request = false, ready = > > true (from last parsing). We enter into the loop, and start a new round of > > parsing. As expected. Good. > > > > So far so good. :-) > > > > Jimmy > > > > > ____________________________________________________________ > > The future is not set. There is no fate but what we make > > for ourselves. - Terminator II, Judgment Day > > > ------------------------------------------------------------ > > > > > > > _______________________________________________ > Bro mailing list > bro at bro-ids.org > http://mailman.ICSI.Berkeley.EDU/mailman/listinfo/bro > > From rpang at cs.princeton.edu Thu May 31 18:18:01 2007 From: rpang at cs.princeton.edu (Ruoming Pang) Date: Thu, 31 May 2007 21:18:01 -0400 Subject: [Bro] ssl binpac analyzer -- patches In-Reply-To: References: <200705261938.l4QJcnXi091065@jaguar.icir.org> <4658AB92.9020304@klipix.org> <2e4d286d0705291721u2c555dbblc9bd4fb400701ce2@mail.gmail.com> <2e4d286d0705302211l48fe0853y92213923e405b6b2@mail.gmail.com> Message-ID: On 5/31/07, jmzhou.ml at gmail.com wrote: > Yes, the changes work well at my side. > > Some problems of binpac: > > . Split binpac out of bro source tree. That's actually happening. Please stay tuned. Also, if you have recommendation on a fast regexp library with BSD-like license, please let me know. Note that we do not need perl-like captures, but only the longest match. > I think to make binpac standalone > makes testing/developing much easier. One can develop new analyzers and > test them with dedicated .pcap files. I agree likewise. In fact, one way to significantly improve testing in binpac is to make a (proof-of-concept) script when an issue arises. Such as in this case... By keeping this scripts around we can make sure that old problems do not surface again. > . Binpac does not support SunRPC over TCP now. There are four extra bytes > prepended in RPC packets. Either TCP layer decoder should take care of > these extra bytes, or the RPC decoder has to do something with it. In > addition, &exportsourcedata is used in RPC/UDP decoder based on datagram > mode. It is not supported by flowunit mode. This means, we cannot simply > change the decoder from datagram mode to flowunit mode for RPC/TCP. The way I imagine doing this is to consider RPC on TCP a trivial lower-level protocol than RPC on UDP. For each RPC-on-TCP message, the analyzer calls the datagram mode RPC analyzer's NewData() routine. What do you think? > Finally, a Ref call is missing in the NewCall function when a call already > exists, and a Unref call is not correctly called in FinishCall. This is a good point. Thanks for pointing it out! Ruoming